Start with lakestatus (which service is red?), then journalctl -u <unit> -n 100 --no-pager.
| Symptom |
Cause |
Fix |
$S3_ENDPOINT (or any variable) is empty |
cluster.env not loaded in this shell |
set -a; source /opt/lake/cluster.env; set +a |
| Connection string fails to parse |
The password contains /, + or = (base64) |
Use openssl rand -hex 16 for every secret |
install-all.sh refuses to start |
A CHANGE_ME is still in cluster.env |
Fill in the names it prints |
A bash script fails with strange \r errors |
The file has Windows (CRLF) line ends |
Every file in the repo is LF (.gitattributes); re-save with LF |
| Symptom |
Cause |
Fix |
| NiFi: “Invalid SNI” |
Opened by IP |
Use https://localhost:18443/nifi |
| NiFi controller services are disabled after a restart |
Known behaviour |
Enable them again |
NiFi restarted by itself during an apt run |
needrestart |
NEEDRESTART_MODE=l (step 18 sets it) |
| PostgREST rejects every token |
jwt.secret has a trailing newline |
Regenerate with openssl rand -base64 48 and tee (no extra newline), restart |
| PostgREST does not serve a new table |
Schema cache is old |
sudo systemctl kill -s SIGUSR1 postgrest |
Spark job: NoClassDefFoundError ... awssdk |
AWS SDK v2 bundle jar missing |
Add the jar whose version the Hadoop POM names (aws-java-sdk-v2.version) |
Spark cannot read s3a:// after re-running step 07 |
Step 07 rewrites spark-defaults.conf |
Re-run step 07.5 |
| Trino: “createView is not supported” |
Nessie catalogs cannot store views |
Expected: the platform keeps view definitions in its registry |
| Trino is slow to answer after a restart |
It takes 30-45 s to start |
Wait for /v1/info |
| Symptom |
Cause |
Fix |
| Keycloak rejects the redirect |
App opened at 127.0.0.1 |
Use http://localhost:5173 |
| Token rejected by the API |
Issuer mismatch (token issued through another address) |
Use one address consistently; set a real hostname in production |
| JupyterHub login gives 401 from Keycloak |
Token issuer differs from the address the hub calls |
The hub reads the user from the ID token (already configured); make sure the roles mapper puts roles in the ID token |
| A vendor user sees no data |
Login exists in Keycloak but has no registry row |
Create the vendor login in the web app, not in Keycloak |
| Symptom |
Cause |
Fix |
File stays uploaded |
Worker is down, or Spark has no worker |
systemctl status lake-worker spark-worker |
File stuck in processing |
Job died; or the worker restarted |
The worker re-queues such files on start; or Process again in the web app (offered after 10 minutes) |
File failed with a column message |
Columns differ from the folder’s earlier files |
Fix the file, or make a new folder for the new layout |
| File stays in Archiving or Deleting |
The server still runs an old worker without remove_file.py |
Deploy and re-run step 15; nothing is lost |
| Upload returns 502 |
Storage down or full |
df -h /data, journalctl -u seaweed |
| Upload returns 413 |
File over UPLOAD_MAX_MB |
Raise the limit or split the file |
| Symptom |
Cause |
Fix |
| Gold has fewer rows than bronze |
Expected: silver drops exact duplicates in a file; gold drops duplicates across files |
None |
A phone number starts with ' in an exported CSV |
The CSV formula guard |
Expected for text starting with = + - @; real numbers are not touched |
| A published API answers 503 |
The view is blocked (owner lost access) or switched off |
Restore the owner’s access, or switch the API on |
| A view build fails |
See the error in Run history; the last good gold keeps serving |
Fix the SQL and Update view |