Serving One Delta Lake Copy to Four Data Platforms

At a previous job, I worked on delivering on-chain and market data to US institutional clients. Real-time metrics went out through an API, but some clients also wanted large volumes of historical data, for backtesting and similar work. We kept that history as Parquet files under Hive-partitioned S3 paths, with no table format, and gave clients read access to those paths. For clients who wanted to query it with SQL right away, we also copied the same data into BigQuery tables as a Query-as-a-Service offering and granted them read access there.

This setup had a problem from day one. Because it was nothing more than a pile of files, a client reading while we were writing, or while the schema was changing, could end up with inconsistent data. That is exactly the problem open table formats such as Delta Lake and Iceberg exist to solve.

As the client base grew, a different problem took over. Some clients worked on AWS directly, but quite a few were on BigQuery, Databricks, or Snowflake, and with a data estate of this size we could not afford one more copy for every platform we added. We needed to move to an open table format in a way that let several platforms read the same copy. And since the consumers were external clients rather than teams inside the company, picking a table format also meant picking which client platforms we would be compatible with.

This post walks through how we merged the S3 Parquet plus BigQuery setup into a single Delta Lake copy and served it to clients on four platforms through Delta Sharing and UniForm. For how each platform feature behaves, I checked the official documentation as of October 2026, so details may differ from what we ran at the time.

The old setup and its problems

The limits of handing out files

Handing out raw files had three problems.

• Schema evolution: Nothing recorded the schema, so readers had to infer it file by file. Adding a column meant old and new files had different schemas, and since every engine merges them differently, the new column could simply be missing on the client side.
• No transactions: If a write landed on a path while a client was reading it, the client could read a half-updated state.
• Duplicate storage: The same data lived twice, once in S3 and once in BigQuery.

A table format fixes the first two. Iceberg and Delta Lake track the schema and the list of files as metadata and make each commit an atomic write. Choosing a table format or a processing engine, however, raised a different question.

Institutional clients use many platforms

Institutional clients, financial institutions in particular, often run on SaaS cloud data platforms such as Databricks, BigQuery, and Snowflake. So choosing a table format was not purely a technical call; we also had to think about how easy it would be for clients to read.

We needed every client to pull the data comfortably from their own platform while storing as few duplicates as possible.

Alternatives we considered

Iceberg

In the industry, the mainstream open-source, engine-neutral table format was Iceberg. Fortunately, the SaaS platforms above were moving beyond their own optimized managed storage (native tables) toward supporting open table formats. Snowflake, BigQuery, and Databricks' Spark engine can all read external Iceberg tables, so at first the front-runner was Iceberg tables in the Glue catalog, written by EMR Spark.

The closer we looked, though, the more two things bothered us. First, with Iceberg as the source of truth, we could not give Databricks clients a Delta table. Spark can read and write Iceberg tables, but letting Databricks clients pick the data up as a Delta table in their own workspace only works when the source is Delta. In the other direction, a table written in Delta Lake can carry Iceberg metadata alongside it through UniForm, so Iceberg clients can read it too. At the time there was no way to do the reverse and make an Iceberg table readable as Delta.¹ Second, for a small data engineering team, running and learning EMR Spark cost far more than Databricks. The company had many analysts, so the sheer usability of Databricks notebooks was attractive, and with job compute and warm-up used well, Databricks was not more expensive than EMR.

Delta Lake and UniForm

So in the end we decided to go with the Databricks platform. Databricks' Delta UniForm generates Iceberg metadata whenever it writes a Delta table. The data files (Parquet), which make up nearly all of the storage, exist only once, yet they can be read both as Delta and as Iceberg.

Clients only read, and all writes happen in our internal pipeline, so giving up Iceberg writes cost us little. On top of that, Delta Sharing made it easy to deliver data to Databricks clients inside the platform itself.

Architecture

Real-time data lands in an operational database as it arrives, and we replicate it with CDC to use it for analytics. Spark jobs, each defined for its own domain, then load the data into Delta tables registered in Unity Catalog (INSERT plus UPSERT). Given the nature of blockchain data, the UPSERTs only rewrote the last n intervals, like a sliding window, and because the tables were clustered, we never rewrote a whole table.

• Delta Sharing: Clients on Databricks get the Delta table shared as is. We managed access with Databricks' governance features.
• UniForm + Iceberg REST Catalog: Clients on AWS directly or on Snowflake read the tables as Iceberg on their own platform through the Iceberg REST Catalog that Unity Catalog exposes. Clients on BigQuery read the Iceberg tables through BigQuery Omni.

Enabling UniForm

UniForm is turned on through a Delta table's TBLPROPERTIES. The table must be registered in Unity Catalog, and you need Databricks Runtime 14.3 LTS or later.

CREATE TABLE market.history.exchange_flows (
  asset STRING,
  exchange STRING,
  ts TIMESTAMP,
  value DOUBLE
)
TBLPROPERTIES (
  'delta.columnMapping.mode' = 'id',
  'delta.enableIcebergCompatV2' = 'true',
  'delta.universalFormat.enabledFormats' = 'iceberg'
);

UniForm only works on tables with column mapping enabled. Turning UniForm on when you create a new table enables column mapping along with it, and Databricks recommends id mode, as above, for compatibility. To enable it on an existing table, you set column mapping to name mode at the same time, on Databricks Runtime 15.4 LTS or later (Databricks, 2026).

ALTER TABLE market.history.exchange_flows SET TBLPROPERTIES (
  'delta.columnMapping.mode' = 'name',
  'delta.enableIcebergCompatV2' = 'true',
  'delta.universalFormat.enabledFormats' = 'iceberg'
);

Constraints we accepted

Read-only Iceberg

A Delta table with UniForm enabled is read-only to Iceberg clients. Our clients never needed to write, so this was not an issue.

Freshness lag

Delta Sharing clients can see new data as soon as a commit finishes. Clients reading through Iceberg, however, hit two separate delays before they see it.

The first delay is the UniForm conversion. The compute that finished the Delta commit generates the Iceberg metadata asynchronously after the commit. Only one conversion runs at a time on a given compute, so commits that arrive while a conversion is in progress are picked up together by the next one. As a result, there is no guarantee that Delta versions and Iceberg snapshots map one to one (Databricks, 2026).

The second delay comes from how each consumer refreshes its catalog view. Clients that talk to the Iceberg REST Catalog directly, such as Spark, fetch the latest metadata location from the catalog when they open a table. Iceberg's Spark catalog does cache loaded tables for 30 seconds by default, though, so within a single session a new snapshot can show up that much later (Apache Iceberg, 2026). Snowflake and BigQuery, on the other hand, register the table definition on their side, so a new snapshot only becomes visible after they re-read the metadata. In Snowflake, if you enable AUTO_REFRESH when creating the table, it polls the catalog every 30 seconds by default. The option is off by default, and tables without it have to be refreshed by hand with ALTER ICEBERG TABLE ... REFRESH (Snowflake, 2026c, 2026b, 2026a). On the BigQuery side, we used BigQuery Omni to register the Iceberg tables on S3 as BigLake tables (Google Cloud, 2026). A table registered this way points at one specific metadata.json file, so it keeps reading the old snapshot until that path is updated, even after new snapshots appear (Google Cloud, 2026b). We therefore ran a separate job that periodically found the latest metadata.json path and wrote it into the BigQuery table definition, which means data queried through BigQuery lags by up to that interval.²

What clients consuming the data through the data lake wanted was to backfill historical data, not real-time freshness, so this lag was acceptable. When conversion falls behind, you can trigger it manually with MSCK REPAIR TABLE <table> SYNC METADATA, which runs synchronously.

Dependence on Databricks

Both writes and data governance are tied to Databricks (Unity Catalog). We judged the platform's usability to be excellent, and we accepted this dependence as the price of not running EMR ourselves.

Results

• We consolidated hundreds of tables into Delta Lake and could cut hundreds of terabytes of data that had been stored twice, in S3 and in BigQuery.
• Write transactions, schema changes, and the like became simple Spark SQL statements against an abstract table-format interface.
• Clients on Snowflake and BigQuery could connect to the data directly from their own platform, which gave us a sales channel for data beyond the API.

Looking back

There was a lot to like, from governance on Unity Catalog to getting rid of duplicate data. What I regret is not putting SLAs (service level agreements) in place at the same time, since they are the data quality measure institutional clients care about, along with a data quality platform to manage them. We were planning to build out DataHub back then, but Unity Catalog keeps adding quality and lineage features, so I suspect it alone may be enough going forward.

References

• Apache Iceberg. (2026). Spark configuration. https://iceberg.apache.org/docs/latest/spark-configuration/
• Databricks. (n.d.). What is Apache Iceberg in Databricks? Retrieved October 22, 2026, from https://docs.databricks.com/aws/iceberg
• Databricks. (2026a). Access Databricks data using external systems. https://docs.databricks.com/aws/en/external-access/
• Databricks. (2026b). Read Delta Lake tables with Iceberg clients. https://docs.databricks.com/aws/en/delta/uniform
• Databricks. (2026c). Unity Catalog credential vending for external system access. https://docs.databricks.com/aws/en/external-access/credential-vending
• Google Cloud. (2026a). About cross-cloud data access. https://docs.cloud.google.com/lakehouse/docs/about-cross-cloud-data-access
• Google Cloud. (2026b). Create Amazon S3 BigLake tables. https://docs.cloud.google.com/bigquery/docs/omni-aws-create-external-table
• Google Cloud. (2026c). Create Apache Iceberg external tables. https://docs.cloud.google.com/bigquery/docs/iceberg-external-tables
• Microsoft. (2026). Access Azure Databricks tables from Apache Iceberg clients. https://learn.microsoft.com/en-us/azure/databricks/external-access/iceberg
• Snowflake. (2026a). ALTER ICEBERG TABLE … REFRESH. https://docs.snowflake.com/en/sql-reference/sql/alter-iceberg-table-refresh
• Snowflake. (2026b). Automatically refresh Apache Iceberg tables. https://docs.snowflake.com/en/user-guide/tables-iceberg-auto-refresh
• Snowflake. (2026c). CREATE ICEBERG TABLE (Iceberg REST catalog). https://docs.snowflake.com/en/sql-reference/sql/create-iceberg-table-rest