2.4: Metadata, Data Governance, and Data Stewardship

2.4.1 Metadata: Data About Data

Metadata is commonly defined as "data about data." It is the information that describes the context, content, and structure of actual data records. Think of metadata as the documentation or properties of your data which make it easier to understand, find, and use.

For a simple example, consider a data file (like a spreadsheet). The data might be sales figures for each month, but the metadata would include things like the file name, creation date, the person who created it, the columns' definitions, units of measure, etc. In a relational database, metadata includes the schema information: table names, column names and data types, relationships between tables (primary/foreign keys), and so on.

Metadata is often categorized into types such as: - Descriptive metadata: Information that helps identify and discover data, like titles, authors, keywords, or abstracts. For a dataset, descriptive metadata could be a summary or keywords that facilitate search. For a photo, it might be the caption or tags. - Structural metadata: Details about how data is organized, e.g. how pages are ordered in a book or how files are structured in a database or data lake. It can describe hierarchies or groupings in the data. - Administrative metadata: Information for managing data, such as who has access, data ownership, rights and permissions, and data retention policy. This often includes audit logs or info needed for preservation. - Technical metadata: Low-level details like file format, size, encoding, data types, and technical requirements for using the data. For example, an image's technical metadata might include resolution, color depth, and camera settings. - Semantic metadata: This relates to the meaning of data – for instance, business definitions or ontologies that clarify what each data element represents. (Semantic metadata might say that "Customer_ID" refers to a unique identifier for a customer in the CRM system, linking it to the concept of customer.) - Provenance metadata (or lineage): Information about where data originated and how it has moved or transformed over time. This is key for tracing outputs back to inputs (important in analytics to trust results).

Here in Figure 2.4.1, we have a hub-and-spoke diagram illustrating six types of metadata — data about data — and their roles in making data discoverable, usable, and governable. At the center is a dark teal rounded rectangle labeled "METADATA — Data About Data" with the subtitle "Makes data discoverable, usable, governable." Six categories branch out from the center, arranged in two rows of three. The top row contains: "Descriptive" (titles, authors, keywords — helps discover data), "Structural" (organization, hierarchy — file relationships), and "Administrative" (access rights, ownership — retention policies). The bottom row contains: "Technical" (file format, size, encoding — data types), "Semantic" (business definitions, ontologies — meaning of fields), and "Provenance" (origin, lineage — transformation history). Each category is displayed in a light-bordered card with a gold circular icon, connected to the central hub by lines. The diagram conveys that metadata encompasses multiple complementary dimensions, each serving a distinct purpose in helping people find, understand, and manage data.

Figure \(\PageIndex{1}\): This diagram identifies six types of metadata — information that describes other data — showing what each type does and how it helps people find, understand, and manage data. (CC BY 4.0; Felix Amoruwa) Access to a detaiiled description of Figure \(\PageIndex{1}\)

 

The importance of metadata cannot be overstated: - It makes data discoverable. With rich metadata, analysts can find relevant datasets without combing through raw content. For example, a data catalog might allow searching by metadata fields like "datasets related to customer feedback from 2022". - It provides context. Metadata describes assumptions or definitions, so analysts know, for instance, that "revenue" in one dataset is in USD and includes international sales, whereas in another dataset it might be only domestic. Without this context, one could misinterpret data and make errors. - It supports data quality and integrity. As IBM notes, metadata enhances data accessibility and supports functions like data lineage, quality checks, and compliance. For instance, metadata can store validation rules for a field (like "Gender can only be M/F/O"), which ties into enforcing data validity (quality). - It is critical for automation and data governance. Tools like data catalogs, governance platforms, or AI algorithms rely on metadata to manage data at scale [15]. If you have thousands of data assets, you need metadata to categorize them, set policies, and track usage.

Example: In a university setting, consider a research data repository. Each dataset submitted by researchers has metadata: author names, dataset title, summary of what it contains, the methodology of data collection, file format, etc. This metadata allows other researchers to search the repository (e.g., "climate data 2020 California") and also to understand how to use the data properly (units, variables, any caveats). Without metadata, the datasets would be almost unusable beyond their creator.

Example: Within a corporation, a data catalog tool might display a table's metadata: "Table Name: sales_transactions; Columns: Date (YYYY-MM-DD), Store_ID (integer, link to Stores table), Product_ID, Quantity, Sale_Amount (USD)" and so on. It may also list the steward or owner of that data, update frequency, and sample values. This metadata saves an analyst time in figuring out if the table is what they need and how to join it with other data.

Metadata is often stored in a metadata repository or data catalog. Modern data governance emphasizes being "metadata-driven" – meaning tasks like impact analysis (what reports will break if I change this field's definition?) or compliance (find all datasets containing personal emails) are made possible via metadata.

One of the reasons metadata is key to governance is explained by Gartner research: companies not using a metadata-driven approach can spend up to 40% more on data management. This is because without good metadata, people manually chase down info or fix mistakes that could have been avoided if data context was known upfront.

In the era of big data, metadata is sometimes the only way to impose order on chaos. For example, a data lake might contain raw semistructured logs; adding tags/metadata about what system or product those logs come from, and what each field means, turns the data lake from a swamp into a more usable resource.

In summary, metadata is the glue that holds a data ecosystem together. As the Carnegie Mellon Metadata Guide succinctly put it, while calling metadata "data about data" is easy, it's more useful to see it as the lifeline that gives coherence, context, and meaning to data. Without metadata, data is an orphan – with metadata, data becomes information.

2.4.2 Data Governance: Policies and Oversight for Data

Data Governance is the overarching framework of policies, processes, and standards that ensure an organization's data is managed properly – meaning data is high quality, secure, private, and used in line with business goals and regulations. It is a strategic function, often involving senior management, that answers questions like: Who can access what data? Who is responsible for data quality? What are the rules for data retention and compliance? How do we resolve data-related conflicts or issues?

In simpler terms, data governance is the "law and order" for data in an organization. It provides the laws (policies and standards) and assigns order (roles and responsibilities) to manage data as an asset.

Key components of a data governance program include: - Data governance council or committee: A governing body, often cross-departmental, that sets the direction and makes decisions. This might include executives (Chief Data Officer, CIO) and representatives from business units. - Policies and Standards: Formal documentation on data management practices. For example, a policy might state that all critical data must have an assigned owner and steward, or that customer personal data must be encrypted and only stored in certain systems. Standards could include naming conventions for data fields, approved data formats, or quality thresholds. - Roles and Responsibilities: Defining who does what. Common roles: - Data Owners – accountable for data, typically a business function head who "owns" the data content and its usage. - Data Stewards – (covered in next section) responsible for day-to-day management and quality of data under guidance from owners. - Data Custodians – IT roles that manage the technical environment of data (databases, storage), ensuring availability and security, but not deciding the content. - Governance team – individuals who coordinate the governance process, such as a Data Governance Manager or data office members. - Processes: How policies are implemented and enforced. This includes processes for data quality monitoring, data issue resolution (e.g., if a department finds a data error, how is it escalated and fixed?), change management for data schema changes, access request workflows, etc. - Metrics and Monitoring: Governance also means measuring how well data management is doing. For example, tracking data quality metrics, audit results, or the percentage of datasets with complete metadata.

A well-known framework in data governance is to ensure data is “available, usable, integrable, and secure” throughout its life cycle. Availability means people can get the data when needed (with proper access controls), usability means it's understandable and fit for purpose (which ties to quality and metadata), integrability implies different data can connect (common standards, master data), and security covers privacy and protection.

Example: In financial services, data governance is often driven by regulatory requirements like BCBS 239 (a banking regulation on risk data aggregation) or GDPR for privacy. A bank might establish a data governance program to inventory all critical risk data, assign owners to each dataset, define data quality checks (accuracy, completeness thresholds), and ensure lineage documentation (so any risk report can be traced to source data). The governance team might meet monthly to review data quality reports (e.g., “we are at 98% completeness for counterparty data, short of the 100% goal, what is the issue and plan?”). They also approve any new data usage to ensure compliance (e.g., can we use customer transaction data for a new marketing use-case under our privacy policy?).

Example: A university with research data might form a data governance board to handle issues of data sharing and security. Policies may dictate how student data is protected, who can access academic records, and standards for data anonymization for research. Data governance would ensure, for instance, that any cloud service used for storing student data complies with privacy laws (FERPA in the US).

A strong data governance yields benefits: - It helps maintain data quality by formalizing stewardship and accountability. - It improves consistency because standards reduce disparate practices (e.g., everyone uses the same country codes list, instead of one system using USA vs. another using US). - It ensures compliance with laws (like GDPR, HIPAA) and reduces risk of data breaches or misuse by controlling access. - It fosters trust internally; users know where data came from and that it went through proper checks.

On the other hand, poor or nonexistent governance often leads to the classic "data swamp": lots of data, no documentation (lack of metadata), conflicting versions, and no clear owner to fix issues. A scenario illustrating poor governance: different departments each create their own report of "total sales", and each uses slightly different definitions, leading to multiple answers and internal confusion. Good governance would have a single source of truth or at least definitions so that differences are known and intentional.

It's often said data governance is not just about control, but enabling data democratization in a safe manner. Rather than blocking access, governance provides a framework so more people can use data confidently. Modern governance emphasizes a balance between control and freedom: implement guardrails (security, quality) while encouraging usage.

2.4.3 Data Stewardship: Operational Management of Data Assets

If data governance is the rule-setting and high-level oversight, Data Stewardship is the day-to-day practice of implementing those rules and managing data on the ground [17]. Data Stewards are individuals or teams who are responsible for the content and context of data – they ensure data is properly collected, labeled, stored, and maintained in accordance with governance policies. They are often the go-to people for questions about a dataset.

Key duties of data stewards include: - Data Quality Management: They monitor data quality metrics for their domain, investigate issues (e.g., why did completeness drop this month?), and coordinate data cleansing or corrections. For example, a customer data steward might regularly run de-duplication processes and fix records where addresses are missing or invalid. - Metadata Management: Stewards document the data – definitions of fields, business rules, allowable values – effectively curating the metadata so that others can understand and use the data. They might maintain a data dictionary for their subject area. - Access Control and Support: They often handle data access requests for their domain, making sure only authorized users get access and explaining the data to new users (how to interpret it, caveats). They enforce proper use as per governance rules (e.g., if someone tries to use data inappropriately, a steward might flag that). - Liaison Role: Data stewards sit at the junction of business and IT. They work with IT (data custodians) to implement changes or improvements (like adding a new field or adjusting a data pipeline to fix an issue). They also work with business users to ensure the data meets business needs and to gather requirements for data enhancements. - Policy Enforcement: While governance sets policies, stewards execute them. For instance, if a policy says "all customer records must be reviewed for accuracy quarterly," the steward will carry that out. If governance says "if a customer requests their data to be deleted (right to be forgotten), we must do so in 30 days," the steward coordinates that deletion across systems.

In terms of hierarchy, often data stewards are assigned for specific data domains or business areas – e.g., a finance data steward, a product data steward, a customer data steward. They might report to a Chief Data Officer or to their line-of-business but have a dotted line to the governance council.

Using the earlier analogy, if data governance are the lawmakers and police chiefs, the data stewards are the on-the-ground officers ensuring the laws are followed and guiding people (data users) to adhere.

Figure \(\PageIndex{2}\): This layered diagram shows how data governance sets the rules and policies while data stewardship carries them out day-to-day, with metadata supporting both levels. (CC BY 4.0; Felix Amoruwa) Access to a detailed description of Figure \(\PageIndex{2}\)

Example: Imagine an e-commerce company. Data governance policy dictates that product data must have certain fields populated before a product can be listed online (to ensure completeness and consistency). The Product Data Steward is responsible for checking newly created product entries to ensure they have all required info (price, weight, dimensions, category, etc.), and if not, working with the product team to fill the gaps. They might also maintain the controlled vocabularies for product categories (ensuring people use the right category names). When a new category is introduced, they update documentation and communicate it to analysts. If an analyst finds that some product records have negative inventory due to a system glitch, the steward investigates and coordinates a fix with IT, and maybe updates a data validation rule to prevent it recurring.

Example: In a healthcare context, a hospital may have a Data Steward for patient data. This person ensures that patient records are consistently formatted, that merges of duplicate patient entries are handled, and that metadata like code dictionaries (for diagnoses, procedures) are up to date and correctly used. They might lead training for data entry staff on common data quality issues, thus improving accuracy at source.

Data stewardship is increasingly supported by tools (data quality dashboards, catalog systems where stewards can approve definitions or annotate data). However, it fundamentally is a human role because it requires understanding meaning and context, not just technical criteria. For instance, an outlier in data might be flagged by a system, but a steward uses business knowledge to judge if it's an error or a legitimate rare event.

The relationship between governance and stewardship is collaborative. A common question: Is data stewardship part of data governance? Generally, yes – stewardship functions operate within the governance framework. Governance sets the strategy (the "what" and "why"), stewardship is about execution (the "how"). They are distinct but interdependent. Neglecting one undermines the other: Great policies mean little if no one enforces or operationalizes them (no stewardship), and doing a lot of cleaning tasks is reactive if no policies address root causes (no governance).

To highlight: Atlan (2023) describes data governance as determining "who has access to data, who is responsible for accuracy, and how data is stored/protected," whereas data stewardship is ensuring data quality, handling data requests, and enforcing policies day-to-day. Another analogy given is governance is the strategy (like laws for road safety) and stewardship is the tactics (traffic cops enforcing those laws).

Finally, let's consider the tie back to Data Quality: Data stewards are often the champions of data quality improvement initiatives. If data quality KPIs are not met (say accuracy is 95% but target is 99%), the stewardship team will dive in, perhaps launch a data cleansing project or propose a process change (like adding a validation at entry). They serve as data quality guardians under the mandate of governance.

2.4.4 Bringing It Together: Governance and Stewardship in Practice

It's valuable to illustrate how metadata, governance, and stewardship come together through a brief case study:

Case Study (Financial Institution): A large bank undertakes a project to improve its customer 360 view and comply with new data privacy laws. The Chief Data Officer (CDO) leads the data governance council to set objectives: unify customer data from various product lines, ensure quality (especially no duplicate customer profiles), and enforce privacy preferences.

• Metadata efforts: The bank deploys a data catalog to inventory all customer-related datasets. Each dataset is documented with metadata: source system, data owner, data steward, field definitions, update frequency. They classify metadata for privacy (tag certain fields as sensitive PII). This allows quick responses to privacy requests (like finding all places a customer's data exists).

• Governance policies: The council issues policies such as: all systems must use the master customer ID; any new customer data source must integrate with the master data management (MDM) system; and define rules like "when merging records, use last update timestamp to keep latest info".

• Data Stewardship: The Customer Data Steward (with a small team) gets to work implementing these. They run data matching algorithms to find duplicate customer records across savings, credit card, and mortgage systems. They coordinate with IT to merge them under a single ID in the MDM hub (fixing uniqueness issues). When policy says to capture customer consent for marketing, they ensure a new field "Marketing_Opt_In" is added everywhere and properly populated (completeness). They also create a data quality dashboard tracking accuracy (e.g., valid email or phone rate) and completeness (e.g., percent of customers with updated addresses). They periodically report these metrics back to the governance council.

• Outcome: Over time, the bank sees a reduction in returned mail (because addresses are now more complete and up-to-date – timeliness and accuracy improved). Customer satisfaction rises because they no longer get mis-targeted (e.g., receiving offers for services they already have, due to duplicate profiles). And when a new privacy law requires providing customers a copy of all their data, the bank can quickly aggregate it thanks to the metadata and single customer ID governance put in place. This case demonstrates the synergy of clear governance directives and active stewardship in enhancing data as a strategic asset.

In summary, metadata, governance, and stewardship form a critical trio in data management: - Metadata provides the information backbone that makes data usable and controllable. - Data Governance provides the policy and high-level management to ensure data is treated as a valuable asset and risk is managed. - Data Stewardship provides the skilled human touch to enforce standards, improve quality, and guide the organization in daily data practices.

These are especially crucial as organizations scale up their data operations (more data sources, more users, more regulations). Without such frameworks, the likelihood of data chaos, quality collapse, or compliance failures increases greatly. With them, organizations can trust their data and use it to drive innovation and competitive advantage safely.