Skip to content
CHDC Data Lake

NiFi

What it is for

Section titled “What it is for”

Apache NiFi is a visual tool for moving data between systems. In the lake it is for the pull side only: reading from vendor databases, APIs and SFTP servers on a schedule. It is not in the upload path; vendors who send files use the web page or the Push API.

State today: installed and tested with a simple file-to-S3 flow. No production flow has been built yet.

Where it lives

Section titled “Where it lives”
Software /opt/nifi-<version> with symlink /opt/nifi (NiFi 2.x)
Service nifi (step 09), runs nifi.sh run as lake
Port 8443 (HTTPS)
Heavy data /data/nifi/<repository> (symlinked into /opt/nifi)
Heap NIFI_HEAP in cluster.env (4 GB on staging)
Login NIFI_USER / NIFI_PASSWORD (12+ characters), single-user mode
Logs /opt/nifi/logs/nifi-app.log

Check it

Section titled “Check it”
Terminal window
systemctl is-active nifi
grep -m1 "NiFi has started" /opt/nifi/logs/nifi-app.log

Startup is slow (a minute or more). Open https://localhost:18443/nifi through the tunnel and accept the certificate warning (NiFi makes its own certificate).

How to use it

Section titled “How to use it”

NiFi works with processors connected by relationships on a canvas. A flow is built by dragging processors in, setting their properties and connecting them.

A first flow: write a file to the raw bucket

Section titled “A first flow: write a file to the raw bucket”

A file-to-S3 flow like this was tested on staging; check the property names against your NiFi version.

  1. Add GetFile (or ListSFTP + FetchSFTP) as the source.
  2. Add PutS3Object. Create its AWSCredentialsProviderControllerService with the S3 access key and secret. Set Endpoint Override URL to http://127.0.0.1:8333, Bucket raw, enable path-style access, and an Object Key such as nifi/${now():format('yyyy/MM/dd')}/${filename}.
  3. Connect source > PutS3Object, enable the controller service, start both.
Section titled “The recommended production pattern (a design, not yet built)”

Have NiFi post pulled files to the lake’s own front door so they get the registry row, audit trail and worker automatically:

QueryDatabaseTable / InvokeHTTP (pull)  ->  ConvertRecord (to CSV)  ->  InvokeHTTP POST
     /api/v1/push/projects/<p>/folders/<f>/files?name=<file>.csv     with   Authorization: Bearer <push key>

Use a folder-scoped push key per flow. Two flows are planned: one flow, many tables (ListDatabaseTables > QueryDatabaseTable incremental > CSV > push) and a date-range export API pull (InvokeHTTP > push).

Configure it

Section titled “Configure it”

Version, checksum, heap and login are in cluster.env (NIFI_VERSION, NIFI_SHA512, NIFI_HEAP, NIFI_USER, NIFI_PASSWORD). Re-run step 09 after changing them (the heap is edited in conf/bootstrap.conf).

Things that go wrong

Section titled “Things that go wrong”
Symptom Likely cause
Invalid SNI Opened by IP; use localhost
Controller services disabled after a restart Known: re-enable them (secrets for them are kept, the enabled state is not)
NiFi restarted by itself during an apt run needrestart restarts services using updated libraries. Step 18 sets NEEDRESTART_MODE=l to prevent it
Slow or out-of-memory Raise NIFI_HEAP and re-run step 09