The Semantic Layer Is Not a Layer Anymore
We spent the last decade making data more open. Now we need to make meaning open too.
There is a lot of excitement around semantic layers right now, and I am glad there is. But I am also worried that we are about to repeat a familiar mistake: standardize the bottom of the stack while quietly rebuilding lock-in above it.
We did this with data platforms. Open table formats gave us a standard wheel. Then every vendor added its own catalog, governance model, access layer and control plane — the proprietary bolts that make the wheel difficult to move anywhere else.
The same thing is beginning to happen with semantics.
Snowflake has Semantic Views. Databricks has Metric Views. Google has LookML, Dataplex and the Open Knowledge Format. AWS is developing a context layer around knowledge graphs and Iceberg. Cube is evolving an independent semantic runtime. dbt has MetricFlow. Microsoft, Salesforce and the BI vendors have their own semantic models.
All of this activity tells us something important: the semantic layer is no longer a thin abstraction between a warehouse and a dashboard. It is becoming the place where a company connects business meaning, executable calculation, ownership, policy, context, performance and delivery.
That is not really a layer anymore. It is an emerging semantic control plane: the missing architecture between a consumer asking a business question and a replaceable engine calculating the answer.
Underneath that phrase are five concerns it has to hold together:
Business meaning — definitions, entities, dimensions, metrics, terminology and ownership.
Execution — joins, aggregation logic, query compilation, federation and engine selection.
Governance — access, certification, lineage, policy, accountability and change control.
Knowledge and context — the decisions, examples and relationships that explain why the model looks the way it does.
Performance — caching, pre-aggregation, materialization, freshness and cost-aware routing.
Most products implement some of these well and gesture at the rest. That is the trap: not one control plane, but several partial ones, each authoritative only inside a single tool.
The fifth concern is not quite a peer of the others. Performance is where meaning, governance and execution meet, so it never gets its own box. It keeps reappearing — in the cache key, in the router, in the materialization. Watch for it.
There is a reason this is happening now rather than five years ago. For a decade the consumer of a metric was a person looking at a dashboard, and a person can repair an ambiguous number by asking someone. The consumer is increasingly an agent, and an agent cannot. That changes what these interfaces have to carry — and it is why they are being built, quickly, right now.
There is a second reason the interface matters now. Business questions will increasingly cross the boundary between structured and unstructured data. An agent may calculate governed net_revenue from Iceberg tables, then retrieve the policies, contracts, emails or research documents that explain why it changed. Those steps may run through entirely different execution systems: a SQL engine for the number and a lexical, vector, hybrid or multimodal retrieval runtime for the evidence. OpenSearch already combines lexical and semantic retrieval through configurable search pipelines, while LanceDB supports structured filters alongside vector, full-text and multimodal search. The semantic control plane has to resolve the same governed concepts across both, enforce policy across both, and return a composite receipt: source snapshots and conformance evidence for the calculation; corpus and index versions, retrieval configuration and citations for the supporting evidence.
The question is not which product owns all five. It is whether the interfaces between them are open. If they are not, we end up with portable data and immovable meaning.
The hardest data problem is rarely the query
Across science, engineering and enterprise data, I keep seeing the same pattern: the difficult part is often not getting an answer. It is agreeing on the question.
What is an active customer?
Which date determines monthly recurring revenue?
Do refunds belong to the original transaction month or the month in which the refund happened?
Which exchange rate applies?
What exactly does churn mean when one customer uses several products?
The SQL is usually solvable. The organizational ambiguity is not.
These decisions are often hidden in places that were never designed to become enterprise infrastructure: a dashboard filter, a finance spreadsheet, an analyst’s query, a slide deck, a Slack thread, or the memory of the one person who remembers why the definition changed three years ago.
Now the consumer of the answer is increasingly not a person. An agent cannot ask the one person who remembers; it infers, and it infers fast. Worse, agents compose. A query feeds a decision that feeds the next query. A number that is stale, out of policy or built on last quarter’s definition does not surface as one wrong cell in a dashboard — it propagates through a chain of reasoning and arrives as a confident recommendation.
It is tempting to answer that a capable enough model can simply read the dbt project, the Slack thread and the finance spreadsheet and work the definition out for itself. Often it can. But an inferred definition is one that nobody approved and nobody can audit. Being right most of the time is not a governance model. At machine speed it is a way to be wrong at scale, with citations.
A semantic system becomes strategically useful when it turns that invisible negotiation into an explicit, governed asset: a business definition with an owner; an executable calculation with valid dimensions and join paths; policies, lineage and lifecycle state; and enough context to explain when the metric should — and should not — be used.
That, to me, is the real opportunity: not better YAML, but better organizational memory that can actually be executed.
One metric, two engines, two contracts
Imagine I am an AWS customer. Finance defines net_revenue once. An interactive dashboard queries it through Athena. A batch pipeline calculates it through EMR Spark. Both read the same Iceberg tables through the same Glue catalog.
Do they return the same number?
To answer that, we have to separate two things that usually travel under one word.
The semantic contract defines the business object: which transactions count as revenue, how refunds are attributed, which currency conversion applies, which dimensions are valid, which date determines the reporting period. This is the level at which Apache Ossie operates. It tells a runtime what net_revenue means.
The execution contract defines what must not vary while a runtime calculates it: timestamp coercion, decimal precision, rounding, overflow, NULL handling, and whether distinct counts are exact or approximate. A model can be complete at the first level and still leave the second wide open.
A portable plan is not yet a portable answer. Both contracts have to hold before two engines can be trusted to agree.
The second contract sounds abstract until you notice that someone has already built one. GoogleSQL, previously named ZetaSQL, is open-sourced under Apache 2.0 and combines three things: a language definition, a reference implementation, and a compliance framework. It pins the deterministic semantics that quietly break portability — name resolution, type checking, implicit casting, date and timestamp behavior — while allowing engines to implement different feature subsets. Its test framework even distinguishes results it pins from non-deterministic ones it does not require to match exactly. It is not itself a database or query engine; it is the contract that BigQuery, Spanner, F1, Bigtable, Dremel and Procella are held to.
So execution behavior can be declared, and compliance with it can be tested. GoogleSQL proves both. What it does not do is cross a vendor boundary.
Substrait crosses that boundary, and shows why crossing it is not enough on its own. It is a vendor-neutral representation of the relational plan — the algebra between a SQL string and an execution engine — and engines including DuckDB, DataFusion, Velox, Arrow’s Acero and Spark via Gluten can exchange it. Its function registry even names the operations that break portability: integer overflow, rounding, division by zero, decimal precision, null handling. But those behaviors are options rather than obligations. The specification is explicit that if a plan omits one, the consuming engine “is free to choose any supported behavior”.
Substrait therefore carries the plan between the two contracts while guaranteeing neither. Two Substrait-compatible engines can execute the same plan and return different numbers, both legitimately.
Lining the three up makes the hole obvious:
The last row is the requirement, not an existing artifact. Every column has already been solved by somebody — but no single row satisfies all four at once, and nothing yet combines them. That empty row is the argument of this article.
The plan is portable. The answer is not yet portable.
Return to Athena and Spark. They do not need the same optimizer, compiler or even the same plan format. But if net_revenue is one governed metric across both, the semantic declaration has to pin the business choices, and a conformance profile has to pin the observable execution behavior. Then the same golden cases can be run against each runtime. The implementations stay different; the contract becomes testable.
This turns openness from an abstract cross-vendor appeal into a first-party product requirement. You cannot promise a customer that net_revenue means the same thing in Athena and Spark until you have declared what it means precisely enough to test. And once you have done that, you have built most of the open contract anyway.
Semantics without execution is documentation. Execution without semantics is optimized disagreement.
Where should a metric be materialized?
Ask a practical question and the architecture answers back. Two users request net revenue by customer segment last quarter. One may see all regions while the other may see only Switzerland. One dashboard may use finance-approved definition 4.2.0 while another still points to 4.1.0. The Iceberg table may have advanced to a new snapshot. A daily aggregate may be fresh enough for the dashboard but not for a regulatory report.
So what is the cache key? It cannot just be the SQL string, or even the metric name. And “should this live in Arrow or in Iceberg?” turns out to be three questions wearing one coat: what format, who owns the artifact, and how validity is decided.
Format follows lifetime. Apache Arrow is an uncompressed, random-access, CPU-native columnar layout built for data in flight; Parquet is compressed and encoded for data at rest. That is not an architectural preference, it is a placement rule: Arrow for seconds to minutes, Parquet and Iceberg for hours to months.
Cube is the clearest worked example, because it does both. Cube Store’s documented stack is “Rust, Apache Arrow, DataFusion, Parquet, RocksDB.” Streaming rows land as in-memory Arrow chunks that are immediately queryable, then compact to Parquet roughly every minute, because “all persistent data is stored as Parquet files.” One engine, both tiers, and the tier is chosen by age rather than by taste. Cube also gives hot and cold explicit names: rollup_lambda with union_with_source_data unions a batch rollup with live source data, and partition_granularity with an incremental refresh_key rebuilds only recent partitions.
Its matching logic is the part worth stealing. A pre-aggregation is selected by reasoning about measures, dimensions, granularity and additivity — not by comparing query text. Members of a view are resolved to the underlying cube members, so a rollup defined on cubes serves queries written against views. A materialization is valid for a request because of what it means.
And then the uncomfortable part. Search the whole of Cube’s documentation and the word “iceberg” does not appear; neither does “delta lake.” Cube Store writes Parquet files, but they are governed by its own internal metastore — schema, table, index, partition, chunk — with no manifest, no snapshot, no catalog. They are not a table any other engine can read.
That is not a criticism of Cube. It is the strongest evidence in this article. The most portability-minded semantic runtime in the ecosystem still had to invent a private materialization store, because there is no open standard for what a semantic aggregate is.
What would an open one need? Not a new invention, it turns out. Iceberg’s View specification is published, and a Materialized View specification has been proposed alongside it — opened in August 2024, still under discussion, still not part of the published spec. It defines a storage-table holding the precomputed results and a refresh-state record carried in the storage table’s snapshot summary: the view version the refresh ran against, when it started, and the state of every source table and view it depends on. A change to the definition produces a new version id, and snapshots recorded against an older one are no longer valid.
That is most of the manifest this article has been asking for, arriving from the storage layer. What it does not carry is the semantic half: which governed metric this implements, at which grain, under which policy scope, and until when it may be reused. The ask is therefore smaller and more concrete than “someone should design a manifest.” It is: extend a specification that has been waiting two years, and help land it.
Which leaves the question of who decides that a given materialization may answer a given request. The cleanest way to think about it mirrors the two contracts:
The materialization declares a guarantee. The request declares a requirement. The router matches them.
Format follows lifetime. Validity follows declaration.
This is why freshness: PT1H belongs in the request and not in the artifact. A forty-minute-old rollup is simultaneously valid for the dashboard and invalid for the regulatory report. Hot and cold are therefore not storage decisions at all — they are routing policy over declared guarantees. The same three tiers fall out naturally: an Arrow result bound to one policy scope and produced seconds ago; an Iceberg materialization with a manifest declaring metric version, grain, policy scope and freshness; and the source tables underneath, authoritative and slow. Engine-native caches, indexes and rewrites stay inside the chosen engine, where they should stay competitive.
An open request and receipt can carry that evidence without dictating how any runtime optimizes:
request: # what the consumer asks for — no engine, by design
metric: https://semantic.example.com/finance/metric/net-revenue
definition_version: 4.2.0
dimensions: [customer_segment]
time_range: {start: 2026-01-01, end: 2026-03-31}
freshness: PT1H
policy_scope: urn:company:policy:finance-aggregate-access-v2
receipt: # evidence of how it was produced — engine recorded, not requested
materialization: semantic.net_revenue_daily
format: iceberg
source_snapshots: {orders: 9384920384, refunds: 2837293912}
engine: athena
conformance_profile: open-semantics-0.1The asymmetry is deliberate. The request is prescriptive: it says what the consumer needs without naming a runtime. The receipt is descriptive: it records that Athena produced this answer, from these snapshots, through this materialization, under this conformance profile. If a consumer genuinely needs to influence engine selection, it should express a requirement — a certification level, a latency target, a cost ceiling — and let the router choose. Naming Athena in the request would bypass the router and recreate exactly the coupling the contract is meant to remove.
The receipt matters more to an agent than to a dashboard. A person reading a chart brings context to it and will often notice when a number looks wrong. An agent eight steps into a chain will not. The receipt is the only machine-readable way to tell it when not to trust an answer: this came from a forty-minute-old aggregate, under this policy scope, against definition 4.2.0, from an engine at this conformance level. An agent that can read that can decide to re-run against source data before it files a regulatory number. An agent that cannot will simply be confident.
Arrow carries that payload well, and can even hold the receipt as schema metadata — but it enforces none of it. The control plane does. An engine-neutral semantic cache is Arrow for the payload and an enforced receipt for the meaning.
A fast answer can still be wrong, stale or unauthorized. Performance is therefore part of semantics and governance, not a separate optimization problem.
The graph is the contract, not necessarily the database
net_revenue is related to gross sales, refunds, currencies, customers, accounting policies, source tables, owners, quality tests, dashboards and physical materializations. A flat list of metrics cannot express this well. The natural model is a graph.
A metric is a connected object, not a row in a glossary.
Requiring a graph does not mean requiring one graph database. The graph should be the logical contract: stable identities, typed relationships, provenance, versioning and machine-readable traversal. Its representations can stay federated across an executable semantic model, a catalog, a policy system, linked documents and open tables.
This is where Google’s Open Knowledge Format matters. OKF uses Markdown, YAML frontmatter and links to give organizational knowledge a surface that humans and machines can both read. Version 0.2 adds trust signals: provenance sources, generated and verified markers, a stale_after date, lifecycle status and attested computations. (Google Cloud)
OKF should not become the query compiler. Ossie should not contain pages of policy prose. Iceberg should not become an ontology format. They should point to the same governed concept.
An open semantic control plane needs MDM for meaning.
Traditional MDM gives a customer, product or supplier a governed identity across systems. Semantic MDM does the same for concepts such as net_revenue, active_customer or churn: one stable identity connecting the definition, calculation, owner, policies, context, materializations and downstream uses.
This does not require one central “golden model.” The representations can remain distributed across Ossie, OKF, catalogs, policy systems and platforms. What becomes authoritative is the identity, its relationships and its lifecycle.
The golden record becomes a golden identity.
That identity is the spine of the control plane. It is what lets a router know that an Ossie metric, an OKF document, a policy, an Iceberg materialization and a dashboard all refer to the same governed concept, without forcing them into one system.
Open data — and open SQL — did not open meaning
Iceberg opened the data. It did not open the meaning.
Apache Iceberg separated analytical tables from a single engine, giving every client a shared representation of schemas, snapshots and transactional metadata. Its REST Catalog protocol went further and standardized the catalog boundary itself — discovering namespaces and tables, loading metadata and snapshot references, committing changes.
That boundary is worth dwelling on, because it is where the proprietary bolts usually go. It does not have to be that way: Lakekeeper is an Apache-2.0 implementation of the Iceberg REST Catalog, written in Rust, that adds access control, credential vending and audit — authenticating through any OIDC provider and delegating authorization to engines such as OpenFGA. An open catalog can therefore carry governance too, not just a table listing. In a semantic control plane that is exactly the interface to physical state: a router can locate a materialization, obtain scoped credentials for it, pin the snapshot it read, and record all of that in the receipt without coupling to a different catalog client per engine.
What no catalog can tell you is that semantic.net_revenue_daily implements net_revenue version 4.2.0, at which grain, for whom, until when.
Two more rungs are already open. GoogleSQL aligned expression semantics across an engine family. Substrait opened the plan between SQL and the runtime. Storage, expressions, plans — three rungs, and not one of them knows what an active customer is, whether revenue is additive across time, which join path Finance approved, or whether refunds belong to the transaction month. Each also stops at a boundary: at the edge of the GoogleSQL family, where BigQuery meets Dataproc Spark, the question returns immediately.
The platforms are converging on the rung above from different directions. AWS Context is announced as a governed organizational context layer backed by a knowledge graph, with context metadata projected into Iceberg; as of this writing it is described as coming soon rather than generally available. Google approaches organizational knowledge through OKF, while Dataplex provides the managed catalog and governance role. Ossie addresses open analytical semantic interchange. Cube shows what an independent runtime looks like.
These are not competing answers to one narrow feature. They are complementary parts of an architecture whose connective tissue is still missing.
The missing middle — and how to build it
An open semantic control plane is not another universal platform inserted between every consumer and every database. It is a set of interoperable contracts that lets specialized systems participate in one governed path from question to answer.
It is a control plane because it decides which governed object a request refers to, which policy applies, which materialization is valid and which runtime should execute it. It is open when identities, requests, relationships, manifests and receipts can cross product boundaries — even if the resolver, optimizer, graph index or policy engine behind them is managed and proprietary.
The control plane decides; the data plane computes. Follow one engine-free net_revenue request down through identity, meaning, policy and routing — across the boundary into materialization and execution — and one portable receipt back up.
Follow the request through it.
The consumer asks for meaning, not an engine. A dashboard, application or AI agent requests the governed
net_revenueidentity, dimensions, time range and service requirements through the open semantic query contract. It does not ask for Athena SQL or Spark SQL.The registry resolves the governed identity. One stable identifier, its version, its lifecycle state and its owner — the spine that connects every other representation.
The control plane assembles the concept. It retrieves the executable metric and join rules from Ossie, follows links to OKF context and business decisions, and obtains the applicable ownership, approval and access policy.
The router decides how the request may be answered. It validates dimensions and joins, evaluates the caller’s policy scope, checks freshness and cost, and matches the requirement against the guarantees declared by available materializations. Only then does it select a runtime, handing it native SQL or a Substrait plan under a declared conformance profile. Some requests will fan out across runtimes — Athena or Spark for the governed calculation, and a search or multimodal retrieval engine for the evidence — before the control plane assembles one answer with both kinds of provenance.
The data plane provides portable evidence. Iceberg tables and snapshots expose the source state, reached through an open catalog; a semantic aggregate manifest states which metric version, grain and policy scope a materialization implements. The engine optimizes freely without changing the contract the consumer sees.
The result returns with a receipt. An Arrow payload carrying the resolved definition, policy decision, source snapshots, materialization, engine and conformance profile. If Athena and Spark disagree, the organization compares evidence instead of rediscovering the business definition from two SQL strings.
Step four deserves more attention than it usually gets, because agentic workloads change what it is for.
A human asks one question at a time. An agent fans out into dozens of subagents, each asking something small and specific. Why would a query over a pre-aggregated table need a distributed cluster, when an in-process engine such as DuckDB — which reads Iceberg tables directly, and gains full access by attaching to an Iceberg REST catalog — answers it in milliseconds? At that volume, cluster startup is not overhead; it is the entire latency budget, and the cost curve stops being an abstraction.
So engine selection stops being a deployment decision and becomes a per-request one. And it is the same decision as materialization matching: if a valid aggregate exists at the right version, grain, freshness and policy scope, a single-node engine will serve it; if none does, the request falls through to the source tables and needs something larger. Which materialization and which engine are one question, not two.
Which is exactly why the execution contract cannot stay optional. A router that chooses freely between engines that disagree about decimals, NULLs and time zones will return different answers to the same question depending on load. Cost-aware routing without conformance is not optimization. It is nondeterminism with a budget.
Two things follow. The router needs machine-readable capability and cost metadata for each runtime — GoogleSQL’s own caveat that engines may implement different feature subsets becomes an eligibility rule, not a footnote. And pushing execution to an engine running beside the agent means policy can no longer be inherited from the deployment; it has to be vended per request, which is precisely what an open catalog’s credential vending is for.
Today’s platforms usually collapse these steps into one stack: semantic model, policy system, router, materialization logic and engine all move together. That fusion is the lock-in. The open version pulls them apart at the seams so each can evolve, or be replaced, without redefining the business.
No single product has to implement every box. An independent runtime might provide the router, an open catalog might vend credentials and enforce access, OKF files might carry institutional knowledge, Ossie might express analytical semantics and Iceberg might carry materializations. What matters is that each component can exchange governed identities and evidence through open interfaces.
So how does this actually start? Not with a grand alliance or a universal ontology. It starts with a test.
Conformance cannot decide what net_revenue ought to mean. That remains a business decision. It can test whether two runtimes faithfully execute the same declared meaning. The distinction matters more than it sounds. Suppose a semantic language can express a sum over orders but cannot state that the fiscal calendar is 4-4-5, that refunds are attributed to the original sale month, and that distinct customer counts must be exact rather than approximate. Athena and Spark can then produce different numbers while both passing every available test. That is not only an engine problem. It is a hole in the declaration format.
So the corpus drives the specification. Every golden case that conforming engines can still interpret differently identifies a missing piece of the declaration language. The tests make ambiguity visible before customers discover it in two executive dashboards.
In practical order, the industry needs three deliverables.
A golden conformance corpus and reference adapters. This tests the seam between meaning and execution — the arrow between steps 4 and 5 above. GoogleSQL already proves the pattern works one rung down, including the crucial habit of separating results the contract pins from those it deliberately leaves open. Adapters should emit native SQL and, where supported, Substrait plans, so a failure can be localized to the declaration, the translation or the engine. Start with the cases that routinely break portability: time zones, fiscal calendars, NULLs, decimals, distinct counts, many-to-many joins, semi-additive measures, late-arriving data and policy-dependent results.
An Ossie-aligned request, result and receipt contract. This is steps 1 and 6 — the consumer boundary. Let consumers ask for governed semantic objects without starting from vendor-specific SQL, and return the definition version, policy decision, source snapshots, materialization, freshness and engine behind the result. The materialization half of this is the smallest ask in the article: extend the proposed Iceberg materialized-view refresh state with semantic identity, grain and policy scope, and land it.
An identity and registry protocol. This is step 2, and it gives one concept one identity across all five concerns. Stable identifiers, delegated namespaces, versions and lifecycle states for metrics, dimensions, policies, owners, documents, datasets and materializations — linking the executable Ossie object to its OKF knowledge and to local platform representations.
Two of these are components in the diagram. The first is not, because it is not a component: it is the test that the arrows between meaning and engine can be trusted. That is also why it comes first. The corpus creates evidence and demand; the request contract turns that demand into an integration target; the registry becomes useful precisely because real implementations now need to resolve the same concepts across systems.
Nobody needs permission to begin. A foundation, an independent vendor or a single cloud team can publish difficult cases unilaterally. AWS can test Athena against Spark and Redshift. Google can test BigQuery against Dataproc. An independent runtime can test itself against both. Buyers can then ask vendors to show their results.
On Monday, a team could write ten hard net_revenue cases, express them as far as today’s model allows, run them through two engines, and publish both the results and the unresolved ambiguities. That would do more for an open semantic architecture than another diagram of a universal platform.
That is the architectural promise: one concept, one governed identity, multiple representations — and replaceable runtimes underneath.
Humans could route around a missing contract by asking someone. Agents cannot, and they will not ask.
The incentive is already inside the platforms
This is not an argument against managed platforms. Google, AWS, Snowflake, Databricks and others should differentiate on runtimes, optimizers, graph indexing, governance experiences, developer tooling and AI interfaces. Independent vendors should differentiate on portability and cross-platform execution.
The products already embody different bets about where semantics should live. Snowflake Semantic Views and Databricks Metric Views bind governed metrics closely to their platforms’ data, optimizers and access-control systems. Cube Core separates the semantic runtime and exposes SQL, REST and GraphQL interfaces with aggregate-aware pre-aggregations. Each bet can stay differentiated. The open requirement is only that governed identity, requests and result evidence survive the runtime.
The direction of travel is already visible in the materialization layer, and it is uneven. Snowflake can publish dynamic tables as Iceberg that Spark and Trino read directly, provided Snowflake remains the catalog. Databricks offers external Iceberg access to materialized views, currently in preview. Both are opening the same seam from opposite sides, at different speeds, with different constraints. That is what an unsettled interface looks like shortly before it settles.
There is a sharper version of this worth naming. The agentic moment is currently the best argument any vendor has for a proprietary semantic layer: our agent works best on our semantics, our context, our governance. That pitch is not even wrong — it is genuinely easier to make an agent reliable inside a single platform. But it means the semantic control plane is being closed right now, under the banner of AI, and largely with good intentions. That is why these contracts matter this year rather than in three.
The self-interest is straightforward: internal coherence, customer trust, and the chance to shape a standard rather than inherit someone else’s.
The precedents point the same way. Iceberg shows how consumers demanding cross-engine access can open a storage interface. GoogleSQL shows how a definition, a reference implementation and a compliance corpus can align behavior across several engines. Neither standardized the engine. Both standardized enough shared contract to make interoperability testable.
Meaning will follow a similar path. BI tools, applications and AI agents will not want a different semantic integration for every runtime, just as they did not want a different storage integration for every warehouse. The first platforms to publish cross-engine conformance and portable receipts will not be giving away their execution advantage. They will be turning an internal QA burden into ecosystem leadership.
Do not standardize the engine. Standardize the contract and the evidence.
Open source lets you inspect a component. Open standards let you replace it.
Open data was the first step
Technology will not make people agree. We still have to do the hard, messy human work of defining the business.
But once we agree, the architecture should make that agreement explicit, governed, executable, testable and portable — from the first consumer request to the final result receipt.
We spent the last decade making data more open. The next step is to make meaning open too.
Because the engine will change.
The platform will change.
The AI model will change.
The business should not have to redefine itself every time they do.
Further reading
Semantics and knowledge
Apache Ossie — open interchange for analytical semantics
Open Knowledge Format specification and the v0.2 spec in Google’s knowledge-catalog repo
Google Cloud: How the Open Knowledge Format can improve data sharing
Open formats and contracts
Apache Iceberg, the table spec, the view spec and the REST Catalog specification
Lakekeeper — an open-source Iceberg REST Catalog with access control and credential vending (source)
Apache Parquet — the on-disk cousin of Arrow
Apache Arrow, the columnar format, Arrow Flight and ADBC
Substrait, its function extensions and who uses it
DuckDB’s Iceberg extension — an in-process engine reading Iceberg and attaching to REST catalogs
GoogleSQL: language definition, reference implementation and compliance tests
Products referenced