Skip to content
CHDC Data Lake

Nessie

What it is for

Section titled “What it is for”

A table format such as Iceberg needs a catalog: the answer to “which tables exist and which version of each is current?”. Nessie is that catalog. It also keeps a commit history like git, so every change to a table is recorded. Spark and Trino both talk to it, and the catalog is named lake in both.

Nessie stores only small pointers in PostgreSQL (database nessie). The data stays in the lake bucket.

Where it lives

Section titled “Where it lives”
Software /opt/nessie/nessie-quarkus-runner.jar, run with Java 21 (-Xmx2g)
Config /etc/lake/nessie/application.properties
Service nessie (step 05)
Ports 19120 (API and UI), 9000 (management), both 127.0.0.1
Store PostgreSQL database nessie (JDBC2 version store)
Warehouse s3://lake/warehouse

Check it

Section titled “Check it”
Terminal window
systemctl is-active nessie
curl -s http://127.0.0.1:19120/api/v2/config | head -c 300        # native API
curl -s http://127.0.0.1:19120/iceberg/v1/config | head -c 300    # Iceberg REST view

How to use it

Section titled “How to use it”

Open the web UI through the tunnel: http://localhost:29120. You can:

  • browse the branch main and its tables (each lake table appears as bronze.<name>, silver.<name>, gold.<name>),
  • read the commit log to see who changed which table and when,
  • compare references.

From a shell:

Terminal window
curl -s http://127.0.0.1:19120/api/v2/trees | head -c 600              # branches and tags
curl -s "http://127.0.0.1:19120/api/v2/trees/main/entries" | head -c 800   # tables on main

Everything uses the branch main. A separate dev branch for notebooks is an idea for later.

Configure it

Section titled “Configure it”

quarkus.http.host=127.0.0.1 keeps it local. S3 keys and the endpoint come from cluster.env through step 05. After changing them re-run step 05.

Things that go wrong

Section titled “Things that go wrong”
Symptom Likely cause
Trino or Spark say the catalog is unreachable systemctl status nessie; it takes about 15 s to start
createView is not supported Expected: Trino cannot store views on a Nessie catalog. That is why the platform keeps view definitions in its registry
After a restore, tables vanish The nessie database must be restored together with the lake bucket

Back up the nessie database (see PostgreSQL). Without it the Parquet files exist but no one knows which belong to which table.