# Serving One Delta Lake Copy to Four Data Platforms

> How I merged data kept twice in S3 Parquet and BigQuery into one Delta Lake copy, served through Delta Sharing to Databricks and as Iceberg via UniForm.

- Canonical URL: https://jerrylog.dev/en/blog/delta-uniform-multi-platform-serving
- Date: 2026-10-17
- Author: Jerry
- Language: en
- Topic: Software Engineering / Data Engineering
- Tags: delta-lake, iceberg, uniform, delta-sharing, databricks, data-platform

---

At a previous job, I worked on delivering on-chain and market data to US institutional clients. Real-time metrics went out through an API, but some clients also wanted large volumes of historical data, for backtesting and similar work. We kept that history as Parquet files under Hive-partitioned S3 paths, with no table format, and gave clients read access to those paths. For clients who wanted to query it with SQL right away, we also copied the same data into BigQuery tables as a Query-as-a-Service offering and granted them read access there.

![A SQL editor querying a BigQuery table in the Query-as-a-Service offering](https://jerrylog.dev/assets/images/delta-uniform/query-as-a-service.png)

This setup had a problem from day one. Because it was nothing more than a pile of files, a client reading while we were writing, or while the schema was changing, could end up with inconsistent data. That is exactly the problem open table formats such as Delta Lake and Iceberg exist to solve.

As the client base grew, a different problem took over. Some clients worked on AWS directly, but quite a few were on BigQuery, Databricks, or Snowflake, and with a data estate of this size we could not afford one more copy for every platform we added. We needed to move to an open table format in a way that let several platforms read the same copy. And since the consumers were external clients rather than teams inside the company, picking a table format also meant picking which client platforms we would be compatible with.

This post walks through how we merged the S3 Parquet plus BigQuery setup into a single Delta Lake copy and served it to clients on four platforms through Delta Sharing and UniForm. For how each platform feature behaves, I checked the official documentation as of October 2026, so details may differ from what we ran at the time.

## The old setup and its problems

### The limits of handing out files

Handing out raw files had three problems.

> **Diagram:** Before: the same data stored twice, in S3 Parquet and BigQuery. After: a single Delta Lake copy.
> 
> Before, the same data was stored twice, as S3 Parquet and as a BigQuery replica, and served to customers from both. After, a single Delta Lake table is served through two paths: Delta Sharing and UniForm (Iceberg).

- **Schema evolution**: Nothing recorded the schema, so readers had to infer it file by file. Adding a column meant old and new files had different schemas, and since every engine merges them differently, the new column could simply be missing on the client side.
- **No transactions**: If a write landed on a path while a client was reading it, the client could read a half-updated state.
- **Duplicate storage**: The same data lived twice, once in S3 and once in BigQuery.

A table format fixes the first two. Iceberg and Delta Lake track the schema and the list of files as metadata and make each commit an atomic write. Choosing a table format or a processing engine, however, raised a different question.

### Institutional clients use many platforms

Institutional clients, financial institutions in particular, often run on SaaS cloud data platforms such as Databricks, BigQuery, and Snowflake. So choosing a table format was not purely a technical call; we also had to think about how easy it would be for clients to read.

| Client platform | How they want to read it                             |
| --------------- | ---------------------------------------------------- |
| **AWS direct**  | Read tables on S3 with Spark, Athena, and the like   |
| **Databricks**  | Read it as a table in their own workspace            |
| **Snowflake**   | Attach it as an external table and query it with SQL |
| **BigQuery**    | Attach it as an external table and query it with SQL |

We needed every client to pull the data comfortably from their own platform while storing as few duplicates as possible.

## Alternatives we considered

### Iceberg

In the industry, the mainstream open-source, engine-neutral table format was Iceberg. Fortunately, the SaaS platforms above were moving beyond their own optimized managed storage (native tables) toward supporting open table formats. Snowflake, BigQuery, and Databricks' Spark engine can all read external Iceberg tables, so at first the front-runner was Iceberg tables in the Glue catalog, written by EMR Spark.

The closer we looked, though, the more two things bothered us.
First, with Iceberg as the source of truth, we could not give Databricks clients a Delta table. Spark can read and write Iceberg tables, but letting Databricks clients pick the data up as a Delta table in their own workspace only works when the source is Delta. In the other direction, a table written in Delta Lake can carry Iceberg metadata alongside it through UniForm, so Iceberg clients can read it too. At the time there was no way to do the reverse and make an Iceberg table readable as Delta.[^iceberg]
Second, for a small data engineering team, running and learning EMR Spark cost far more than Databricks. The company had many analysts, so the sheer usability of Databricks notebooks was attractive, and with job compute and warm-up used well, Databricks was not more expensive than EMR.

### Delta Lake and UniForm

So in the end we decided to go with the Databricks platform. Databricks' Delta UniForm generates Iceberg metadata whenever it writes a Delta table. The data files (Parquet), which make up nearly all of the storage, exist only once, yet they can be read both as Delta and as Iceberg.

Clients only read, and all writes happen in our internal pipeline, so giving up Iceberg writes cost us little. On top of that, Delta Sharing made it easy to deliver data to Databricks clients inside the platform itself.

| Option                                   | Pros                                                                              | Cons                                           | Verdict  |
| ---------------------------------------- | --------------------------------------------------------------------------------- | ---------------------------------------------- | -------- |
| **Keep the status quo**                  | No extra development cost                                                         | Duplicate data, no schema or transaction story | Rejected |
| **Native Iceberg** (EMR Spark)           | Engine-neutral                                                                    | Heavy EMR operations load for a small team     | Rejected |
| **Delta Lake + Delta Sharing + UniForm** | One copy to write, readable on all four platforms, light Databricks operations    | No Iceberg writes, Databricks lock-in          | Adopted  |

## Architecture

> **Diagram:** One Delta table served through two paths: Delta Sharing and the Iceberg REST Catalog
> 
> The internal pipeline writes a single copy of the data files in Delta format, and that copy is read through two kinds of metadata: the Delta log and Iceberg metadata. Through Unity Catalog, Databricks customers receive it via Delta Sharing and customers on Iceberg-compatible platforms via the Iceberg REST Catalog.

Real-time data lands in an operational database as it arrives, and we replicate it with CDC to use it for analytics. Spark jobs, each defined for its own domain, then load the data into Delta tables registered in Unity Catalog (INSERT plus UPSERT). Given the nature of blockchain data, the UPSERTs only rewrote the last n intervals, like a sliding window, and because the tables were clustered, we never rewrote a whole table.

- **Delta Sharing**: Clients on Databricks get the Delta table shared as is. We managed access with Databricks' governance features.
- **UniForm + Iceberg REST Catalog**: Clients on AWS directly or on Snowflake read the tables as Iceberg on their own platform through the Iceberg REST Catalog that Unity Catalog exposes. Clients on BigQuery read the Iceberg tables through BigQuery Omni.

### Enabling UniForm

UniForm is turned on through a Delta table's `TBLPROPERTIES`. The table must be registered in Unity Catalog, and you need Databricks Runtime 14.3 LTS or later.

```sql
CREATE TABLE market.history.exchange_flows (
  asset STRING,
  exchange STRING,
  ts TIMESTAMP,
  value DOUBLE
)
TBLPROPERTIES (
  'delta.columnMapping.mode' = 'id',
  'delta.enableIcebergCompatV2' = 'true',
  'delta.universalFormat.enabledFormats' = 'iceberg'
);
```

UniForm only works on tables with column mapping enabled. Turning UniForm on when you create a new table enables column mapping along with it, and Databricks recommends `id` mode, as above, for compatibility. To enable it on an existing table, you set column mapping to `name` mode at the same time, on Databricks Runtime 15.4 LTS or later (Databricks, 2026).

```sql
ALTER TABLE market.history.exchange_flows SET TBLPROPERTIES (
  'delta.columnMapping.mode' = 'name',
  'delta.enableIcebergCompatV2' = 'true',
  'delta.universalFormat.enabledFormats' = 'iceberg'
);
```

## Constraints we accepted

### Read-only Iceberg

A Delta table with UniForm enabled is read-only to Iceberg clients. Our clients never needed to write, so this was not an issue.

### Freshness lag

Delta Sharing clients can see new data as soon as a commit finishes. Clients reading through Iceberg, however, hit two separate delays before they see it.

The first delay is the UniForm conversion. The compute that finished the Delta commit generates the Iceberg metadata asynchronously after the commit. Only one conversion runs at a time on a given compute, so commits that arrive while a conversion is in progress are picked up together by the next one. As a result, there is no guarantee that Delta versions and Iceberg snapshots map one to one (Databricks, 2026).

The second delay comes from how each consumer refreshes its catalog view. Clients that talk to the Iceberg REST Catalog directly, such as Spark, fetch the latest metadata location from the catalog when they open a table. Iceberg's Spark catalog does cache loaded tables for 30 seconds by default, though, so within a single session a new snapshot can show up that much later (Apache Iceberg, 2026). Snowflake and BigQuery, on the other hand, register the table definition on their side, so a new snapshot only becomes visible after they re-read the metadata. In Snowflake, if you enable `AUTO_REFRESH` when creating the table, it polls the catalog every 30 seconds by default. The option is off by default, and tables without it have to be refreshed by hand with `ALTER ICEBERG TABLE ... REFRESH` (Snowflake, 2026c, 2026b, 2026a). On the BigQuery side, we used BigQuery Omni to register the Iceberg tables on S3 as BigLake tables (Google Cloud, 2026). A table registered this way points at one specific `metadata.json` file, so it keeps reading the old snapshot until that path is updated, even after new snapshots appear (Google Cloud, 2026b). We therefore ran a separate job that periodically found the latest `metadata.json` path and wrote it into the BigQuery table definition, which means data queried through BigQuery lags by up to that interval.[^bq]

[^iceberg]: As of October 2026, Databricks offers managed Iceberg tables, which you create directly in Unity Catalog and can both read and write (Databricks Runtime 16.4 LTS or later) (Databricks, n.d.).

[^bq]: As of October 2026, Google recommends against Iceberg external tables that point at a `metadata.json` file in most cases. For tables managed by a remote catalog such as Unity Catalog, it suggests connecting through cross-cloud data access in the Lakehouse runtime catalog. Lakehouse is the new name for BigLake, and this feature attaches to the remote catalog's Iceberg REST endpoint and syncs the metadata periodically. It is still in Preview (Google Cloud, 2026c, 2026a).

> **Diagram:** When new data becomes visible to each kind of client after a Delta commit
> 
> Once the internal pipeline commits to the Delta table, Delta Sharing customers can read the new data right away. Customers reading through Iceberg see it only after UniForm generates the Iceberg metadata, and Snowflake and BigQuery customers only after their engine re-reads that metadata. The freshness lag splits into two stretches: UniForm conversion lag and consumer engine refresh lag.

What clients consuming the data through the data lake wanted was to backfill historical data, not real-time freshness, so this lag was acceptable. When conversion falls behind, you can trigger it manually with `MSCK REPAIR TABLE <table> SYNC METADATA`, which runs synchronously.

### Dependence on Databricks

Both writes and data governance are tied to Databricks (Unity Catalog). We judged the platform's usability to be excellent, and we accepted this dependence as the price of not running EMR ourselves.

## Results

- We consolidated hundreds of tables into Delta Lake and could cut hundreds of terabytes of data that had been stored twice, in S3 and in BigQuery.
- Write transactions, schema changes, and the like became simple Spark SQL statements against an abstract table-format interface.
- Clients on Snowflake and BigQuery could connect to the data directly from their own platform, which gave us a sales channel for data beyond the API.

## Looking back

There was a lot to like, from governance on Unity Catalog to getting rid of duplicate data. What I regret is not putting SLAs (service level agreements) in place at the same time, since they are the data quality measure institutional clients care about, along with a data quality platform to manage them. We were planning to build out DataHub back then, but Unity Catalog keeps adding quality and lineage features, so I suspect it alone may be enough going forward.

## References

- Apache Iceberg. (2026). Spark configuration. https://iceberg.apache.org/docs/latest/spark-configuration/
- Databricks. (n.d.). What is Apache Iceberg in Databricks? Retrieved October 22, 2026, from https://docs.databricks.com/aws/iceberg
- Databricks. (2026a). Access Databricks data using external systems. https://docs.databricks.com/aws/en/external-access/
- Databricks. (2026b). Read Delta Lake tables with Iceberg clients. https://docs.databricks.com/aws/en/delta/uniform
- Databricks. (2026c). Unity Catalog credential vending for external system access. https://docs.databricks.com/aws/en/external-access/credential-vending
- Google Cloud. (2026a). About cross-cloud data access. https://docs.cloud.google.com/lakehouse/docs/about-cross-cloud-data-access
- Google Cloud. (2026b). Create Amazon S3 BigLake tables. https://docs.cloud.google.com/bigquery/docs/omni-aws-create-external-table
- Google Cloud. (2026c). Create Apache Iceberg external tables. https://docs.cloud.google.com/bigquery/docs/iceberg-external-tables
- Microsoft. (2026). Access Azure Databricks tables from Apache Iceberg clients. https://learn.microsoft.com/en-us/azure/databricks/external-access/iceberg
- Snowflake. (2026a). ALTER ICEBERG TABLE … REFRESH. https://docs.snowflake.com/en/sql-reference/sql/alter-iceberg-table-refresh
- Snowflake. (2026b). Automatically refresh Apache Iceberg tables. https://docs.snowflake.com/en/user-guide/tables-iceberg-auto-refresh
- Snowflake. (2026c). CREATE ICEBERG TABLE (Iceberg REST catalog). https://docs.snowflake.com/en/sql-reference/sql/create-iceberg-table-rest