TOP DATA OBSERVABILITY & DATA QUALITY TRENDS TO PUT ON YOUR RADAR
- DQaaS or DQSaaS
- Hybrid Usage & Island Hopping.
- Machine Learning has hit the DQ space.
- From patchwork to a fundamental capability.
- From a technology to a business driven framework.
TREND #1. DQaaS or DQSaaS
Data quality as a service (DQaaS or DQSaaS) is an emerging trend that involves outsourcing a subset of data quality management functionality to third-party cloud application providers. DQaaS providers offer tools and services to monitor and improve data quality, reducing the workload on in-house data teams. In general, SaaS is provided in a cloud-based or hosted model. We see two types of DQaaS:
- One type basically providing a complete set of data quality functionalities (equivalent to traditional on-premise offerings) that run on cloud platforms. Clients typically order them on demand from cloud enabled vendors and often use them on a subscription basis (1 year or longer).
- The other type of DQaaS is basically based on on-demand online services to i.e. validate and verify addresses or other relevant data assets. These micro data quality services are typically used on a pay per usage / service-call basis.
TREND #2. Hybrid Usage & Island Hopping
Until recently, Data Quality was primarily applied in an operational/transactional context. DQ in an analytics context was also already implemented in quiet some organization, but what is relative new is the enhanced DQ usage in a number of other data related initiatives such as MDM, DG, Data Engineering and AI/ML. What we also see as part of that move, is a shift from an island DQ usage tailored towards one specific initiative to an organization-wide DQ usage. The organization-wide DQ approach provides a number of benefits ranging from more consistent data quality up to enhanced collaboration, more efficient data processing and reuse.
TREND #3. Machine Learning has hit the DQ space
We don’t need to explain you that AI/ML is hot and all over the place. The AI/ML examples that you read about might not always seem to be relevant to you. However, DQ is one of those areas where AI/ML can really deliver substantial added value. Machine learning algorithms can be used to automatically identify data quality issues and correct them. For example, machine learning models can be trained to detect duplicates, correct spelling errors, and identify missing data. Next to more automated error resolution a lot of DQ applications are expanding DQ to provide insights by discovering relationships, patterns and trends.
We see both custombuild applications (Python and other open source libraries) and Data Quality platforms coming with embedded AI/ML functionality to bring DQ automation to the next level.
TREND #4. From patchwork to a fundamental & companywide capability
As Data Governance (DG) maturity in organizations grows, we also see that they address Data Quality as a fundamental integrated data capability. Instead of the so called one-off usage for one specific case (data migration, CRM, etc), a lot of companies make DQ a structural component that they can reuse continuously for all their data related initiatives. The benefit of this wider embedded approach is that it eases organizations to demonstrate DQ to be a profit vs a cost component.
TREND #5. From a technology to a business driven framework
As we mentioned in the previous trend and as maturity increases, we are also observing a shift in the approach from an it-driven towards a business-driven perspective. The main reason for this is that organizations require data quality to be seamlessly integrated into their business processes for optimal results.
Conclusion
In conclusion, data quality management is a critical component of data management, and there are many exciting trends and technologies emerging in this area. From data profiling to machine learning, organizations have many tools and techniques available to improve data quality and drive business growth.
