Making Data Products Rebuildable

As a distributed system grew, related data began to live in more than one place for good reasons. MongoDB supported operational workflows; Tinybird made analytical questions fast and useful. Different teams owned different parts of the record and shaped each store around the work it needed to do.

The ambiguity appeared when teams needed to understand those records together. Treating either team's database as the shared source of truth would couple one product to another team's schema, availability, and operating decisions. A useful query could quietly become a fragile production dependency.

I co-developed the high-level direction with senior engineering and infrastructure partners: persist canonical data in S3 first, then build purpose-specific materializations for the systems and queries that need it.

That changed the role of MongoDB, Tinybird, and future stores. They could remain excellent at their own query shapes without competing to be authoritative. The durable record remained authoritative underneath them, while teams could rebuild, replace, or extend each downstream projection.

The canonical layer kept a versioned history for each shared entity, including deletions, while audit and recovery metadata stayed separate from the original payload. That let teams share different kinds of data without forcing every domain into one global schema.

I designed and built the implementation end to end: the event flow, the S3, SNS, and SQS infrastructure in Terraform, Rust consumers, and the live materialization and replay paths used across several data products.

The durable write was the commit point. Downstream delivery was deliberately at least once and eventually consistent, so consumers had to be idempotent and tolerant of reprocessing. In return, a slow or failing projection could retry or catch up, and teams could replace it without blocking producers.

Live and rebuild workloads followed separate paths. Teams could control replays independently, so a backfill or recovery did not have to compete with current ingestion. The same mechanism onboarded a new consumer and reconstructed a stale or corrupted projection. Recovery was not a separate emergency mechanism; it was a routine property of the platform.

I also built a Go SDK that let a partner team publish its data through a small interface. The team did not need to learn the storage conventions, delivery machinery, or replay model to participate safely.

Teams could discard and reconstruct incorrect or outdated projections. They could add new consumers without changing the canonical store, and each operational or analytical system could use a shape suited to its own work.

I applied the same discipline to the meaning of those views. An analytical data product needed to state which entities it included, what time window its metrics covered, and whether its state fields were current or historical. Without that contract, a technically valid view could quietly combine historical activity with current metadata and give users the wrong interpretation.

The SDK lowered the cost for other teams to contribute data, which made shared analysis practical without exporting the complexity of the platform to every adopter.

The design did not promise instant cross-system read-after-write behavior or strict global ordering. It chose brief projection lag and retry tolerance in exchange for producer independence, isolated consumer failures, and deterministic recovery.

The important result was not simply that data landed in S3. Fragmented cross-team dependencies became a durable shared platform: authoritative at the source, explicit in meaning, flexible in consumption, and recoverable as downstream needs changed.