Developer helpers

These run on a developer machine. None of them is installed on a server.

dev/tunnel.sh: reach the server’s services

Services listen on the server’s localhost only. The tunnel forwards each one to a port on your machine: local port = remote port + 10000, so nothing clashes with software you already run.

cp dev/local.env.example dev/local.env     # once; git-ignored
bash dev/tunnel.sh                         # leave it running; Ctrl+C closes everything

dev/local.env holds STAGING (an ssh host or user@host), an optional KEY, and TUNNEL_FORWARDS (local:remote pairs). Default forwards:

Service	Local	Remote
PostgreSQL	15432	5432
SeaweedFS S3	18333	8333
Nessie	29120	19120
Trino	18080	8080
PostgREST	13000	3000
NiFi	18443	8443
Spark master UI	18081	8081
Spark worker UI	18083	8083
Keycloak	18180	8180
SeaweedFS master / filer UI	19333 / 18888	9333 / 8888
JupyterHub	18000	8000

It uses ssh -N with keep-alives and ExitOnForwardFailure, so a port already in use stops it instead of silently half-working. Spark’s port 7077 is not tunnelled on purpose: a Spark driver on your machine cannot work through tunnels (executors must connect back to it).

Never open server ports to the internet to avoid the tunnel; in a data centre the equivalent is a VPN plus firewall rules and TLS.

local.env: ports for your machine

The repo-root local.env (copy local.env.example) overrides cluster.env’s hosts and ports with the tunnel ports, no secrets. Load order for the apps: cluster.env, then local.env, then real environment variables. Values are written out in full (no ${VAR}).

dev/push-smoke.sh: smoke test of the push API

CHDCP_KEY=chdcp_... bash dev/push-smoke.sh [base-url] [project-code] [folder-code]

Tests a key, creates a project and folder, sends a tiny CSV, lists and reads it, and prints PASS or FAIL for each step. A vendor-wide key runs every step; a project key skips “make project”; a folder key skips both. It leaves a small test file behind (the push API cannot delete files), so use a test vendor.

API helper scripts (apps/api/scripts/)

Script	What
seed-dev-users.js	Test logins dev-vendor, dev-analyst, dev-admin in Keycloak (password from DEV_USER_PASSWORD)
smoke-login.js	Logs each test user in through the real code + PKCE flow and calls the API
apply-sql.js <file>	Applies one migration (servers use install step 14)
seed-stress-data.js [--vendors N] [--clean]	Many fake vendors, logins, projects and folders, all tagged stress_NNN, to check pages at volume; --clean removes them

Tests you can run

Where	Command
API	cd apps/api && npm test
Web	cd apps/web && npm test
Worker	cd apps/worker && python -m unittest discover -s tests
lakestatus	python -m unittest discover -s infra/lakestatus
JupyterHub config	python infra/jupyterhub/test_jupyterhub.py

Each uses fakes for the servers, so none needs the tunnel.