Data Contracts (Herror & Wider)

  • #data
  • #ai
  • #web
  • #fragments

, 3 min read

https://blog.chaoticgood.computer/content/annotations/highlights/martinfowler.com/Data-Contracts

From martinfowler.com

Annotations

Quote

Data contracts are a way to describe the contents and guarantees of datasets. They apply practices from API design to data by treating them as data-providing services - such as data products - whose interfaces we want to define as clearly as possible.

This is probably the writing on the wall that I need to polish & start advertising Flowthru, since it’s exactly this process for .NET/C#.

I was a bit shocked to find out that this wasn’t something that already existed in .NET (despite its somewhat lackluster OSS community compared to, say, Rust/JS).

Quote

The specification of the Open Data Contract Standard is publicly available on GitHub, where it has over 1000 stars. The standard comes with a JSON schema to help validate YAML files in terms of standard compliance.

On a cursory glance, I think Flowthru’s catalog system handles… maybe 40% of this, at a glance? It gets some of the fundamentals (catalog items are objects, with all that entails. I suppose they are technically versioned alongside the software? Schema is a gimme. Quality is… somewhat checked at preflight, although I can already see it can do way better.

Ownership and (maybe?) support is technically handle-able via CODEOWNERS, but obviously not to the extent I’d prefer. Terms of use… is interesting, but not possibly applicable. SLA would go into Preflight. “Servers” is handled by the internal medium abstraction. Security is non-existent (gimme a break – the framework is in beta. Not saying it’s an afterthought, but… my plan is to think about it after, y’know?). AI context… probably also not applicable? Pricing is a likely non-factor, although interesting in light of recent discussions I’ve had both at work and during consulting.

Extensibility is a wildcard that I dare not touch,.

Quote

The Story of Versioning and Breaking Changes One of the main goals of data contracts is to find breaking changes before consumers are affected, and offer graceful evolution processes. For that, data contracts follow the SemVer versioning scheme. Major Version Change: Breaking change such as removing a column or renaming it Minor Version Change: Non-breaking change such as adding a new column Patch Version Change: Small change such as fixing a typo or updating the support channels

I have… thoughts on this.

In Flowthru, I’ve found that this is largely the easy part: if a column changes names, that’s obviously going to break some stuff downstream. That’s pretty much deterministically lintable, even without running a downstream pipeline.

Data qualities (say: this column is normalized) is what gets tricky. They touch on this a bit above with the pre-flight checks, but that is a genuinely difficult thing to try and test. It’s not a breaking change, but it could be.

In the econometrics world, when a survey changes (say, the wording of a question changes), that’s kind of a breaking change. Even if the literal, material properties of the answer haven’t changed (it’s still, say, a multiple choice), that’s something that makes comparing before-and-after an apples-to-oranges comparison. I suppose the choice of when to tick up semver is ultimately the onus of the data owner, with an understanding of what “breaking” means to downstream consumers, but it definitely fuzzes the line a bit.

Around the Web

No discussion yet. Be the first to reply.

On Bluesky

No discussion yet. Be the first to reply.

On Hacker News

No discussion yet. Be the first to reply.