One flow, three layers
A file is stored untouched, then written to bronze (as received), silver (cleaned and typed) and gold (one distinct set). Every row can be traced back to its file.
CHDC Data Lake
Files in. Clean, queryable, traceable data out. Built from open-source parts on plain servers, installed by scripts, no containers.
Everything is a script
A server is built by numbered, re-runnable install scripts and one settings file. No hand-edited servers, no Docker, no Kubernetes.
Three kinds of people
Vendors upload files by web page or push API. Analysts run SQL, save views and publish them as live APIs. Admins run the platform.
Open-source parts
SeaweedFS, PostgreSQL, Nessie, Iceberg, Spark, Trino, Keycloak, NiFi and JupyterHub, each documented here: what it does, where it lives and how to use it.
Where to go next
Section titled “Where to go next”ArchitectureThe parts, the roles and how a file travels through them.
Using the platformStep-by-step guides for vendors, analysts and admins.
ApplicationsThe API, the web app and the worker we wrote.
Server apps, one by oneEach installed program: purpose, files, ports, how to use it.
ScriptsEvery install, deploy and helper script, in order.
OperationsSettings, rebuilding a server, local development, troubleshooting.