5 levels of metadata management maturity in a modern enterprise

Metadata management (MM) discipline has been around for decades. But in the recent years, people and organizations have become much more interested in the subject as digitalization and exponential data growth have been rampant. Being good at managing metadata suddenly became a business necessity.

This article explores five levels of MM maturity in order to provide insight into the current state of MM practice and the potential for future progress. The intent is to start a conversation about both the challenges and solutions to modern MM practices, and show a framework for measuring improvement at both an individual organization and industry level.

What is metadata? A quick reference.

Simply, metadata is data about data.

It is information about one or more aspects of the data that helps to describe and define it.

It can also summarize some basic information and provide additional insight into the characteristics of the data described.

Metadata can include:

• Names
• Descriptions
• Definitions
• Meaning & purpose
• Classifications
• Ownership / role / access permissions information
• Relationships to other data assets
• Lineage (the path of data flows throughout systems and processes)
• Version information
• Location
• Technical attributes
• Quality scores & statistics
• Other attributes

Metadata management refers to the discipline of managing metadata systematically throughout its life cycle.

Why do organizations need to manage metadata?

Metadata management is basically managing the knowledge about data assets. It is the discipline of managing metadata systematically throughout its life cycle.

Benefits of managing metadata are many and they increase as the MM management posture matures:

1. Increased trust in company’s data
2. Better compliance and increased security at lower costs
3. Well defined roles and responsibilities in relation to data assets
4. Lower costs and increased agility for any data-related disciplines: data engineering, analytics, data science, etc.
5. Reduced costs for employee on-boarding, training and knowledge transfer

Organizations that don’t have a well-developed metadata management posture don’t really “know” their data. They are not well-positioned to maximize value extraction from their data.

Metadata management maturity levels

Level 1 – Unaware

At this level, the organization has no awareness of metadata management as a discipline. “We don’t know that we don’t know.”

Metadata is likely called “documentation” and is being organized sporadically, if at all. Metadata maintenance is usually a time-consuming, manual, error-prone process. Metadata is not in a central repository and is usually a total mystery to non-technical users as well.

Hopefully, this article will help to advance from this stage.

Level 2 – Initial

At this level, the organization has some awareness of metadata management.

Organizations at this stage start realizing that metadata management is a necessity. They might also discover the MM field, perhaps start looking into possible solutions.

At the 2nd maturity level, organizations start getting to know the basic concepts of MM and how it can help them. They don’t have any tools or methodologies to deal with metadata at this point. There is no central metadata repository or anything of that sort.

Yet, organizations may use simple tools that they already have in the organization – spreadsheets, documents, wiki pages, databases, etc. – to identify, collect, organize and maintain metadata.

Level 3 – Fragmented

At the third level of maturity, an organization recognizes the importance of metadata management in improving efficiency and effectiveness. However, business units or IT silos make decisions about their own metadata management systems and responsibilities without considering the broader organizational needs. This fragments metadata management.

In many cases data elements can be organized using basic folder structures, or spreadsheets, but they are not cataloged or categorized according to any enterprise standards. Companies might promote some guidelines to comply with, but they do it on a unit-, not enterprise- level.

Organization is likely to have multiple metadata repositories, that it tries to organize – one or more “Document Management Systems” (DMS), several business glossaries, internal wiki, code documentation scattered throughout git repositories, spreadsheets with data dictionaries and data mappings all over the place.

This is common for organizations that don’t have enterprise-level data management / data governance programs. Each organizational unit typically manages their data separately and/or there are no clear boundaries of who’s responsible for what.

At this stage the organization begins to understand that metadata management requires a holistic, coordinated effort across various business units. Data is being shared across organizational boundaries, but there is no standardized approach to MM.

At this level introduction of automation for metadata scraping and management can occur, but it will again be fragmented.

Some advanced organizations might introduce quality metrics for metadata at this stage.

Most medium and large organizations I know are at this maturity level.

Level 4 – Enterprise

At this level of maturity, there is a recognition of the importance of metadata management at the highest levels of the organization. Likely, there is a Chief Data Officer or an equivalent role and an enterprise-level Data Management team.

Organizations at this stage adopt metadata management tools and procedures that are implemented throughout the entire organization. More and more people across the organization recognize the strategic importance of metadata. The organization becomes much better at using a common language.

Metadata is managed in a centralized fashion, although federated approaches are also possible with standardization.

Automation of MM becomes commonplace. Metadata is discovered and scraped from the source systems mostly without human intervention.

At this level automatic or semi-automatic metadata enrichment is also likely to be present.

A lot of lineage information is captured, some automatically, some manually. Organizations become noticeably better at understanding their data flows and tracing dependencies.

Metadata quality is measured systematically and often reported at the C-Level. There are ongoing efforts to reach and sustain metadata completeness.

I know several companies that I would categorize being at this level.

Level 5 – Strategic

Metadata is regarded as the most important strategic data in the organization and is treated as such.

All organizational metadata is stored in one common repository. Publishing and consuming metadata by all processes and systems is mandatory. Universal metadata formats exist to facilitate these integrations. When external data is brought in, its metadata is being conformed to internal metadata standards.

Essentially, the business is coded and described in metadata, and it is possible to make changes to business processes by changing metadata.

Metadata standards are very strict, and quality is scrupulously managed. Metadata enrichment is often a well-established and automated process.

It looks like at this level of maturity algorithms interact with metadata more than humans.

I know only a handful of companies that are this high end, and all of them are startups or small firms in the tech space.

I doubt it is possible for large companies to achieve this level of sophistication with metadata management on the enterprise level – it would mean a re-engineering of the entire business. But it’s probably possible to instill metadata-driven operations in the specific units of large enterprises.

Metadata Management Maturity Levels Side-by-side

Summary

My deep belief is that strong metadata management culture is the foundation of a data-centric digital enterprise. Metadata management maturity determines not only how well the organization knows its data, but also how good it is at processing it and extracting value. It is the bedrock of trust in data assets.

Without strong MM posture, midsize and large organizations are unlikely to comply with rapidly advancing privacy and data-related regulations. Also, the costs for all data-related disciplines such as data engineering, data science and analytics are likely to be much higher for organizations that are not good at handling metadata.

I wrote this article to raise awareness about the importance of metadata management, and to help those of you who are starting your metadata journey. Building MM capabilities and culture is a critical first step towards building a strong data-centric business.