Nobody files a ticket saying, “Our architecture has an abstraction problem.” They file tickets saying that the data is wrong, missing, or late. Then, engineering spends two weeks chasing a data quality issue that does not actually exist. 

That is more or less what was happening to us. 

We had an on-premises MongoDB instance registered as the golden source for enterprise reference data. Codes, classifications, identity lookups. The unglamorous shared data that quietly underpins customer onboarding, regulatory reporting and about a dozen other things people care about only when they break. It was well-maintained and authoritative, properly registered as the single source of truth. The team that owned it was rightfully proud of it. 

However, every time an analytics team or a downstream product group needed something from it, the experience was miserable. They would reverse-engineer the operational schema, write one-off queries against nested JSON they only half-understood, track down whoever had institutional knowledge of the collection structure, wait — and repeat the whole ritual three months later when the requirement changed slightly. 

I watched this for months before the actual diagnosis clicked. The data was fine. The problem was that we were asking an operational store to moonlight as an analytical platform, and it was bad at that second job. Not because of any flaw in the system, but because that was never what it was built for. 

The Mismatch Nobody Named 

Operational stores optimize for correctness and lifecycle management. They are designed for transactional access patterns: Lookups by key, controlled updates and schema structures that mirror how applications interact with the data. MongoDB did all of this well. 

What analytics teams actually need is something fundamentally different. Stable shapes, documented fields, a predictable refresh cadence and a way to evaluate whether a dataset is fit for purpose without reverse-engineering someone else’s schema. Those are not the same requirements. Conflating them is how you end up with a system that is technically perfect and practically useless. 

The symptoms were predictable. Customizing analytical views in MongoDB required complex aggregation pipelines. Onboarding a new analytical consumer meant manual query design and data restructuring every time. At least four teams had independently extracted roughly the same reference data, each with its own refresh logic, its own interpretation of field semantics and its own private definition of ‘current’. Classification terms lacked centralized governance. Multiple conflicting versions of the same data existed across collections. 

None of these were data quality problems. They were consumption problems dressed up as data problems. 

What We Did Instead 

So, we stopped asking people to consume reference data directly from MongoDB. We started treating each dataset as a data product: Something with an owner, a definition, quality gates, governed access and a proper path to publication. 

The architecture separated three concerns that were previously tangled together. 

• MongoDB remained the authoritative source. 
  No ‘rip and replace’. Authority stayed where it belonged, with the team that understood the data’s life cycle and had been maintaining it correctly for years. The temptation to declare a new system the source of truth is always lurking in modernization projects. It is almost always wrong. 

• A single delivery pipeline replaced the fragmented extractions. 
  Change events flowed from MongoDB through Kafka Connect into an AWS pipeline with explicit Landing, Bronze and Silver layers, all backed by Apache Iceberg tables on S3. Airflow orchestrated the pipeline, with schema validation firing before anything touched the landing zone. One path, one set of rules, one model that anyone can reason about. 

• Publication became a pipeline stage, not an afterthought. 
  Data reaching the Silver layer got published into the enterprise data marketplace — UDM Marketplace, our internal governed data catalogue — with metadata, quality signals, standards compliance and subscription behavior already attached. Consumers could find datasets, evaluate fitness for purpose and subscribe without needing to know what S3 path to ask about or which Slack channel to beg in. 

Note: The figure above shows the full pipeline — MongoDB as the authoritative source, events flowing through Kafka into Landing, Bronze and Silver layers on S3, Athena providing the query surface and the UDM Marketplace as the publication endpoint. Airflow orchestrates everything; DPPS UI (our internal data pipeline operations portal) gives operational visibility. 

What ‘Data Product’ Actually Forced Us to Decide 

‘Data product’ is one of those phrases that can mean almost anything. In our case, we made it mean something specific: A dataset that could not be published until it had a named owner, a data dictionary, business and technical metadata, operational metadata, a data source design, documented audit expectations, quality gates, a standards and guardrails document and a governed route into the marketplace. Not optional. Not encouraged. Required. 

That framing immediately surfaced questions that should have been answered years earlier. What is the actual boundary of this product? Which attributes matter to consumers, and which are operational plumbing that nobody outside the owning team cares about? What does ‘current’ mean for this dataset, and how would a consumer know if something had gone stale? How do you discover it without filing a ticket and waiting for someone to guide you toward the right S3 path? 

These were not governance overhead. Answering them was the architecture. The technical pipeline was almost secondary. 

Three Decisions That Shaped Everything Else 

• We kept MongoDB as the golden source. 
  The reference data store had a registered golden source designation and was correctly maintained by its owning team. We were building a delivery and consumption layer, not replacing a foundation. In a regulated environment, preserving the authority chain matters. You do not want ambiguity about which system is the source of record. 

• We built one delivery model instead of tolerating four. 
  The polite word for multiple independent extractions with divergent semantics is ‘decentralized’. The honest word is ‘chaos’. A single pipeline with consistent schema validation, consistent error handling and a consistent path from source to publication replaced all of that. When every product travels the same route, new team members learn the pattern once rather than discovering bespoke extraction logic for each dataset. 

• We treated publication as a first-class pipeline stage. 
  The enterprise data marketplace, where products are listed with metadata, documentation, quality scores and subscription capabilities, is the point at which a dataset stops being an engineering artifact and becomes something a consumer can evaluate and use independently. Getting data to S3 is not the finish line. Getting it to the marketplace, properly described and governed, is. 

On Self-Service: Storage is not the Product 

I have seen teams land data in S3, declare victory on self-service and then spend six months confused about why nobody is using it. The answer is almost always the same. ‘The data is in S3’ is not a product. It is a location. People need to know that the data exists, understand what it means, judge whether it is fit for their purpose and know who to contact when something looks wrong. 

The marketplace addressed this more than any individual pipeline component. It turned a scattered set of S3 paths into a governed catalog of subscribable products with documentation, quality signals and clear ownership. The publishing service assigns a Kitemark quality score to each product, and compliance with all active standards at deployment time is mandatory, not aspirational. Consumption occurs exclusively via the marketplace, and subscription functionality is enabled and tested as part of the publication process. 

That last point matters more than it sounds. When you force consumption through a governed channel with subscription behavior, you get visibility into who is using what. That visibility is what makes it possible to assess the impact of a schema change before it breaks something downstream. 

The Boring Stuff Turned out to be the Hard Stuff 

I kept waiting for the hard problems to show up in the pipeline itself. Kafka connector configuration, Iceberg table compaction, Athena partition tuning. All of these needed attention, and all of them got sorted. But the gap between ‘a pipeline that works’ and ‘a platform people trust’ came from things I used to dismiss as housekeeping. Naming conventions aligned to enterprise data model standards. Audit column standards are applied consistently to every Silver table. Documentation templates that someone would actually use. Ownership that was real rather than nominal. 

There is something uncomfortable to admit here. Shared enterprise data tends to fail socially before it fails technically. The Kafka connector will be fine. What corrodes is the shared understanding of what ‘authoritative’ means in practice: Whether a given dataset is the real one or a copy someone made eighteen months ago and forgot to deprecate. 

No amount of Iceberg optimization fixes that. You have to fix it at the layer where consumers decide whether to trust a dataset, which is the product layer. 

If I had Five Minutes With Someone Starting This Work 

Separate truth, transport and consumption. That’s it. 

MongoDB holds truth. The pipeline moves it reliably: Capturing change events, validating schemas, preserving arrival fidelity. The product layer — Silver tables, queryable through Athena and discoverable through the marketplace — makes it consumable by people who do not need to know how MongoDB organizes its collections. 

When those three things are genuinely separate, an enormous amount of organizational friction disappears. Producers stop getting pulled into ad hoc reporting. Consumers stop reverse-engineering operational intent. The ops team can evolve the MongoDB schema without breaking downstream jobs. A new team that needs country codes or currency classifications can find them in the marketplace, read the documentation, subscribe and be done with it. 

Not glamorous, but in a large organization, ‘teams can reliably find and use shared reference data’ is worth more than almost any clever pipeline optimization.