What is an ML Data Catalog?
An ML Data Catalog automatically classifies and curates the metadata by combining the basic meta harvested from physical data sources with data profiles and the business context of data using a combination of statistical, NLP, and rule-based algorithms.
As businesses aspire to reach their data intelligence goals, it is essential to acknowledge the significant role that smart tools play in easing the data discovery process, improving trust in data quality, and even bolstering data security. It is no surprise that businesses are investing significantly in intelligent tools that can help them discover transformative insights, after all, a healthy data management program lays a strong foundation for remediation, growth, improvement, and sustenance. William Binney has rightly said that “data is not intelligence.” So, how does one become intelligent through our data? One possible way is to employ the use of a smart data catalog.
When data catalogs were initially introduced, they were a simple metadata management tool employed by IT teams. With the maturation of the business environment and the introduction and adoption of new technology on a global scale, a new fad, big data posed unique challenges. A new requirement for a flexible, intelligent, and adaptable tool ushered in the age of machine learning data catalogs.
What is a Machine Learning Data Catalog?
According to Forrester Research, MLDCs are machine learning (ML)-powered metadata catalogs that maintain traits of data within a data fabric for activation within systems of insights. Simply put, leveraging machine learning and helps provide intelligent metadata management. MLDC monitors existing interactions and governance capabilities in an organization and optimizes to ensure maximum utility to all stakeholders. Employing Artificial intelligence-based algorithms can aid in automatically curating, cataloging, and classifying organization data assets.
Key AI features of an ML Data Catalog
1. Automatically classify datasets and data elements: It should be able to automatically classify the data based on universal, industry-specific, or custom entities. Classic examples are Names, Phone Numbers, email addresses, Social Security numbers, and Credit Card Numbers. It should do this by not looking at exact or fuzzy column names, but also inspecting the actual values and patterns in the data. For example, a field called “Contact Info” containing data that looks like an email should be tagged as “Email.” Furthermore, it should also be able to classify data on its sensitivity posture further. e.g., PII data elements, sensitive data. Even at the table or dataset level, it should be able to automatically classify datasets by Types like Reference, Master, Transactional data, and the Entity Types like Customers, Products, Shipments, Orders
Watch automatic classification in action >>
2. Automatically recommend a new business glossary with business rules. It should automatically recommend new semantic custom entities and associated business rules. E.g., a data field called campaign type in a dataset has enumerated values as “Discounts,” “Bundles” for most of the values, but a minimal set of records as “Misc”, an ML Data Catalog should recommend a new business term called “Campaign Type” with a business Rules or acceptable values only as “Discounts” and “Bundles”.
Watch recommendation features in action >>
3. Automated Data Quality Rules and Alerts: It should be able to compute the data quality of datasets automatically. E.g., if a data field has 90% of data as Numerical, but 10% of data is String, and 2% of values are empty, it should automatically be able to establish that the data type for this field should be a Number and this data should always be populated. Based on this rule, it should be able to compute the data quality score. A more advanced example would be to automatically detect alerts on significant shifts in data that might impact data pipelines and analytics accuracy. E.g., if the volume of data coming into your data lake from a third party is unexpectedly lower by 25%, it should be able to detect changes based on history.
Watch automated data quality alerts in action >>
4. Learn from Human Curation efforts: A true ML Data Catalog should be able to learn from the human curation efforts to continue improving its data classification and data linking accuracy. For example, if a physical data field called Segment Size containing values like Small, Medium, Large was linked to a business term called “Customer Segment”. The Catalog should be able to learn from this such that when a new data field called “Company Size” or “Customer Size” containing similar values is cataloged, it will automatically be linked to “Customer Segment”.
Let's chat about your Data Initiatives
& how we can assist
Benefits of an ML Data Catalog
Speed up Data discovery:
A configured MLDC can automatically scan through organization databases, data lakes, BI Reports, and applications. It harvests the metadata, profiles the data, classifies it on multiple characteristics, and links it to other data and your data domains. Semantic tagging allows MLDC to have a search-like function that can drastically speed up the discovery process.
Improve Data Quality:
An MLDC can automatically set rules based on data profile and alert when data changes unexpectedly. It can track inconsistencies and even recommend improvements to data quality.
Launch a Lean, and Sustainable Data Governance Program:
Automatic metadata harvesting, data classification, and prioritization. Launch a lean and sustainable Data Governance program aligned to your business priorities.
Automate Data lineage:
An MLDC can outline the lifecycle of a particular asset and use ownership information to provide intelligent insights
Increase Data Security and Data Privacy compliance:
Automatic PII identification and masking algorithms are often at work when scanning data. MLDC helps to continue to maintain GDPR, HIPAA, and other data privacy compliances requirements.
In addition, a machine learning data catalog should organize and consolidate data into a single source of truth. It should allow business users a seamless experience to search and access the data they need. It should catalog, categorize, and curate data automatically and allow for collaboration capabilities across teams. It should also enable user access management (UAM) for better data governance.
