Data analytics flounder when inputs are “ugly”!
Qualis® is a new class of software designed to enhance the accuracy of analytics in modern digital platforms—what we call a Data Refinery & Distribution Platform. It facilitates a systematic transition from legacy "boiling-the-ocean" implementations to harmonized and curated firm-wide data.
Qualis eliminates the wasteful actions of expensive last-mile resources—informal, inconsistent, and repetitive "massaging" prior to analysis by data scientists in silos. Qualis delivers a data-centric solution that allows you to rapidly build targeted "Apps" to solve not only the problems of today, but also those of the future. Qualis' data hub is designed to integrate seamlessly with existing investments in data warehouse, marts or lakes, if any.
While Qualis is sophisticated enough to be a comprehensive data platform, it is commonly deployed first to reap immediate ROI by solving a single pressing business problem—do you have a functioning historical security master?
Eliminate Siloed, Repetitive Scrubbing
In recent years, firms have invested heavily in digital transformation initiatives such as EDM, Data Warehouse, Data Marts, and Data Lakes. However, analysts continue to waste resources massaging firm-wide data to suit their individual needs. This informal, widespread manipulation creates data silos and results in the loss of lineage, rendering it unacceptable for regulatory compliance.
There are three common drivers (Before above). First, business units look at data differently depending on need1. A single version of the truth is no longer sufficient. Data scientists are being forced to massage that single truth using expertise at hand—Excel, MATLAB, SAS, R etc. Second, units shield themselves from interruptions caused by monolithic legacy initiatives by erecting data silos2. Third, users develop a higher level of confidence in massaged data from peer silos3 rather than the least-common-denominator truth. This leads to faulty analytics and the loss of lineage.
Qualis addresses the problem in multiple dimensions. First, it permits a single version of the truth to be expressed in as many variant views as necessary to better suit specific business needs (vertical chevrons, After above). Second, it features a distribution component (horizontal chevrons, After above) that delivers point-in-time firm-wide snapshots (that are shielded from one another) to business units, data marts, and data lakes.
The refinement process forces data through four distinct stages.
First, the data is passed through an integrity process—normalize, apply waterfall hierarchies, map namespaces, validate, and cross-check—to establish firm-wide TRUTH. Second (optional), using that truth, custom Apps solve complex business problems (such as historical security master) and generate additional, enriched content. Third, use-specific Views with full end-to-end lineage are then created from the established truth. Finally, isolated (hardware) Snapshots of these views are distributed when needed for research, analytics, trading, compliance, reporting, and to other data stores such as data lakes.
[ETL component (not depicted here) is automated and almost all machine-generated.]
Qualis provides an elastic platform to build targeted Apps to solve complex business problems in unique ways. Apps can expand or contract as necessary to remedy deficiencies in enterprise data infrastructure. Here are some of the more popular Apps:
- Excel at its best ("NIXEL")
Microsoft Excel is one of the most popular and effective analytical tools ever. However, overreliance on interactive Excel processes is a wide-spread concern. The "London Whale" debacle is a good example.
"...volatility appeared lower than it should have because of issues with this model that can, in part, be traced back to how Excel was used."Qualis can extract institutional knowledge from critical spreadsheets into a structured environment with proper controls, yet also fully support the use of Excel via worksheet-like views without diminishing worker productivity.
- Historical Security Master
Commercial EDM software tackles, at best, Ephemeral Security Master. Accurate analytics, however, demand permanent, historical asset identifiers. Qualis Apps can algorithmically identify, generate, and maintain unique firm-wide security identifiers over extended (30+ years) histories with comprehensive cross-references to other vendor identification schemes.
- Cross-investment Risk Exposure and AUM
When one fund invests in another, cross-investments occur. Possible collapse of any one fund could rapidly create a vicious circle of declining performance across the organization. Qualis can continuously and accurately assess risk across the firm, effectively managing exposures in time.
With deeply nested cross-investments routinely crossing regional, asset-class, liquidity, market, and currency boundaries, it is a daunting task to prevent the double-counting of Assets Under Management (AUM) for risk assessment and regulatory reporting. Qualis can continuously analyze shareholder information to unravel deeply nested cross-investments and accurately assess firm-wide risk exposures along each nested path. Firm-wide AUM can then be algorithmically calculated with indisputable lineage without double-counting assets.
- System Consolidation/Migration
M&A activity, corporate actions, budget constraints, collaboration needs, or even obsolescence can create a need for line-of-business applications to be replaced. While vendors may assist with software migration, historical data from retiring systems is often discarded, severely limiting the look-back horizon for future analysis.
Qualis can serve as the data hub, sourcing historical data from older systems and continuously feeding the newer systems. Custom Apps can implement necessary conversion logic to ensure accurate transition. This robust, automated, repeatable, and scalable solution delivers accurate history, sparing the user the agony of historical data migration.
Qualis does not restrict depth of history.
- Firm-wide Data Integrity Architecture—BCBS-239
While warehouses, data marts, EDMs and data lakes have simplified data management, they have created data silos with little ability to ensure that assumptions are consistent across the firm. For example, are cash, accruals, and derivatives modeled consistently across the organization?
Regulatory standards such as BCBS-239 developed after the financial crisis of 2008 specifically require firms to implement comprehensive "Data architecture and IT infrastructure" to address the manner in which firm-wide risk is estimated. Given that the approach to risk management is so unique to each firm, no shrink-wrap software can meet this requirement effectively.
Qualis can meet these requirements. Custom Apps can enforce rigid accounting-system-like algorithmic checks on firm-wide data to ensure that every "credit" has an equal and opposite "debit." Discrepancies, if any, are highlighted with full lineage so that analysts can rectify the variance at the source or provide necessary explanations.
This level of strict, custom-built integrity enforcement across the firm, while meeting regulatory requirements for firm-wide "Data architecture and IT infrastructure," can also prevent potentially damaging investment decisions due to inaccurate assessment of firm-wide risk.
- Systematic Transition
Stop boiling the ocean!
While Qualis is sophisticated enough to be a comprehensive data platform, it is commonly deployed first to reap immediate ROI by solving a single pressing business problem. You can start to transition your digital infrastructure methodically without abandoning prior investments abruptly.
- Elastic Functionality
Eliminate repetitive, informal massaging of data in data silos!
With a built-in asynchronous workflow engine and the ability to break long-running tasks into smaller, parallel jobs, apps can be as simple or as sophisticated as necessary. Implementations originally designed to address limited gaps in enterprise data systems frequently grow to replace entire legacy components such as EDW.
- Unlimited Time-series/History
Stop purging history just to process current operating window!
Databases are carefully structured to handle limitless chronological growth. Depth of history has no impact on active intake/processing. Every variation—as-is, relational, enriched, and transformed—of aggregated content is readily available in its entirety.
- Split System "Silo Buster" Architecture
Shield your analysts from source system hiccups—leading cause of data silos—never exposing "half-baked" content!
There are two internal sub-systems, one operating 24/7 for intake (supply-side) and another for uninterrupted distribution (demand-side).
Intake is always on and does not impose any order of delivery on source systems. Source data is routinely handled out of sequence. Distribution system is separate from intake and uses hardware technologies to deliver point-in-time snapshots of data when, where, and how needed for direct consumption.
- Client Self-sufficiency
Absolve your data scientists from repetitive “cleansing” of inputs!
The refinery, being self-discovering, self-healing, and event-driven, is fully capable of handling challenges of not just today but well into the unforeseen future. Since more than 80% of the logic is machine-generated, consistency is guaranteed, resulting in easier maintenance.
Qualis takes a business friendly approach to lineage so it is easy to quickly identify sources of inconsistencies for rapid resolution. Lineage is not just limited to data. Self-discovering algorithms unravel lineage of logic automatically.
- Small IT Footprint
Stop wasting expensive data scientist resources on resolving data challenges!
Qualis is built on a single technology platform—Microsoft SQL Server Database Server—and requires no additional skills. SQL talent is widely available making it easy to recruit and retain.
- Last forever!
Future-proof your data infrastructure with elastic apps!
Qualis is carefully constructed with a goal of a 10-year minimum lifespan. It requires no third-party components or immature add-ons that negatively impact longevity. Installations almost always exceed design life. One installation was retired after 18 years of service!