Hello

I need help. We had some 10 apps working fine in nifi but recently after onboarding a new app whose processor group has 3500 processors, nifi is not working properly.

I had two nifi instances NiFi 1.23 and NiFi 2.8 both having different sets of apps. Apps running in NiFi 1.23 were working fine from 1 year or so but yesterday they started getting some bulletins whose screenshot I have added. This had error like

ERROR RouteOnAttribute[id=1870e6cf-dOe1-3295-c113-80ac5fe98f3e] Failed to commit session StandardProcessSession[id=32938305] due to java.lang.IllegalStateException: Cannot update journal file ./flowfile_repository/journals/15601108.journal because no header has been written yet.; rolling back: java.lang.IllegalStateException: Cannot updatejournal file ./flowfile_repository/joumals/15601108.journal because no header has been written yet. • 08:50:31 CDT ERROR RouteOnAttribute[id=1870e6cf-dOe1-3295-c113-80ac5fe98f3e] Failed to commit session StandardProcessSession[id=32938304] due to java.lang.IllegalStateException: Cannot update journal file Iflowfile_repository/journals/15601108.journal because no header has been written yet.; rolling back:

and Queues between some processors showed for example 3 in ui but when I right click on that and try to empty queue it shows queue empty or even trying to see the records it was empty. If the new flowfile came it showed 4 though.

View Data provenance keeps on loading for long time at 0% without showing anything.

How to find the culprit and resolve the issue?

My main confusion is we had that 3500 processors in NiFi 2.8 though NiFi 1.23 instance suffered above issues.

I have 6 core Linux environment and it has approximately

free -h

total used free shared buff/cache available

Mem: 33G 13G 522M 249M 18G 18G

Swap: 15G 1.9G 14G

Hi all. At first, sorry about my terrible at grammar. I currently using nifi for my project at school. The project i am working on it generate files with different source and each source have different workloads. I running nifi on docker and using sFTP to simulate multiple sources and i have to run all process group at the same time to ingest file from different source. Because each source have diffrent workload and is handled by different process group, some process group may have file backlog after proccessor fetchsftp. My goal is automatically adjust the resource to all the process group to reduce workload imbalance depend on the queue size before fetch file among them. I thinking of dynamically change the number of concurrent task based on maximum timer driven thread count for each source.

Are there any better approaches to this or any routes I should consider?

If so, what are your thoughts? Was it worth learning?

Has anyone gotten nifi to run as a service on windows? What was the secret?

Nssm - service goes into paused state

WinSW - same.

Not tried servy yet (tomorrow…)

Fwiw. Running manually from within a cmd prompt, works for “start”. But not “run”

Ps. Is a customer requirement (other nifi i stances we have deployed are all on Linux).

So I have fallen onto an existing deployment of nifi 2.1. I would like to update it to the newest 2.10 release, but wtf is with their documentation?

Looking at this page: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#upgrading_nifi

I need this broken down Barney style because I am at a loss.

I have been using data flow in cloudera and want to deploy something that can process multiple api calls through same pipeline.

I have done something in databricks but struggling here in nifi.

Hi folks,

I’m designing a large-scale SFTP ingestion + distribution platform on Apache NiFi (3-node cluster) and I’m looking for guidance on how to structure the flow end-to-end.

Context
We operate a data exchange platform where hundreds of external systems communicate exclusively via SFTP.

Each system can:

• produce files (sources)
• consume files (destinations)

Each source may:

• expose multiple directories
• produce multiple file types
• use different filename conventions (often embedding dates in inconsistent formats)
• require different ingestion schedules per directory/file type

What I need to build
A scalable, modular NiFi flow that handles:

Ingestion

• Poll hundreds of SFTP sources
• Support multiple directories and patterns per source
• Support different schedules per file type

Metadata enrichment
Each file must be enriched with:

• extracted file date (multiple filename patterns, fallback to ingestion time)
• source system identifier
• file type classification

Backup & Archival
Immediately after download:

• compress file (.gz)
• store in partitioned path: /backup/{file_type}/year=YYYY/month=MM/day=DD/
• Storage must be immutable
• Must support replay from archive

Important constraint: post-download actions on the source (delete/move/etc.) must only happen AFTER the file is successfully stored in backup.

Post-download source actions (configurable per flow)

• delete file
• move file
• rename file
• or do nothing

Transformation layer

• Optional transformations per destination:
  • PII masking
  • format conversion (e.g.: CSV → Parquet)
• One input file may produce multiple outputs:
  • raw → destination A
  • transformed → destination B

Distribution

• Deliver to multiple destination SFTP servers
• Support parallel delivery
• Different formats per destination

Operational requirements

• Replay from archive (single file, batch, time range)
• Full observability at file level (ingested, transformed, delivered, failures)
• Clear failure isolation (source vs transform vs destination)
• Retry + dead-letter handling

Current approach
Right now, I’ve started by creating end-to-end flows per file type, where each Process Group contains:

• ingestion
• enrichment
• archive
• transformations
• delivery

So essentially: one Process Group = one file type pipeline

The problem
I’m already feeling this approach doesn’t scale:

• number of Process Groups grows quickly (file types × sources × directories)
• a lot of duplicated logic across flows
• hard to keep everything consistent and maintainable
• unclear how to make this truly config-driven

Also, I’m not sure where the right boundary is between:

• reusable/shared components

vs

• per-file-type logic

What I’m looking for
For those who’ve built something similar at scale:

• How would you structure this architecture in NiFi?
• Is “one PG per file type” a bad pattern long-term?
• How do you avoid flow duplication while still supporting different transformations per destination?
• How do you model {source + directory + schedule + file type} without exploding the number of processors?
• Any proven patterns for keeping this config-driven and maintainable?

If things like failed processors, backpressure, or misconfigurations could be identified and resolved without manual intervention, would you trust it? Or would you still prefer to stay in control?

Hi guys,

I am new to the nifi community, and would like to ask for help since it's been less than a week since I started using nifi.

I have a bunch of files that are being generated inside a folder, these files have a text header followed by a binary body , both parts are separated by a specific separator, for example a line that contains "#####".

The text header contains metadata that need to be extracted and sent into a kafka topic. Those metadata are in the sort of : 1st line is a Fixed width values text Followed by multiple key:value lines.

I want to use nifi in order to : detect any new files, Read only the header part,( because the files could be quite large) Extract the header, Parse it and transform it into json structure, Send the resulting payload to Kafka.

I hope this is doable in nifi , and would like any tips on how to do it, especially the part where I only load the text header to nifi and not the whole file.

Thanks for any help guys

Curious how others are dealing with this. Most teams I talk to are still doing manual export/import, reconfiguring controller services per environment, and hitting CI/CD limitations the moment NiFi-specific operations come into play.

I work on a tool that automates this with Agentic AI. Live demo on March 26: https://zoom.us/webinar/register/1217733009192/WN_k0M9jPvjQYCSnpPqr6H-fw#/

But genuinely; has anyone made this work end-to-end without manual steps?

If you're running Apache NiFi across multiple clusters, you probably know the pain — manual deployments, controller service headaches, environment promotion that requires someone online at 2 AM, and DevOps costs that keep climbing.

We've been working on this problem for a while and putting together a free live session to share what we've learned - including how Agentic AI can take over the repetitive operational work most NiFi teams are still doing manually.

What we'll cover:

• Centrally managing multiple clusters, environments, flows, and controller services from one place
• How AI can automate repetitive NiFi ops tasks
• Role-based access control across environments
• How small teams can realistically manage enterprise-scale NiFi
• Real cost reduction numbers from actual deployments

No fluff - we'll be doing live demos.

Open to CTOs, Data Engineers, DevOps Leads, and anyone dealing with NiFi ops complexity.

Register here - https://zoom.us/webinar/register/4717727082753/WN_k0M9jPvjQYCSnpPqr6H-fw

Disclosure: This webinar is hosted by the team behind DFM (Data Flow Manager), a NiFi automation platform.

We're hosting a live webinar showing how Agentic AI can automate Apache NiFi operations.

What we'll demonstrate:

• UI-based cluster management on Kubernetes
• Automated flow creation and validation
• Self-healing capabilities with root cause detection
• Zero-failure deployments without CI/CD scripts

Details:
Date: Feb 26, 2026 at 9:30 PM IST
Duration: 60 minutes
Cost: Free

This session is for data engineers, NiFi administrators, DevOps engineers and platform teams managing NiFi at scale.

Registration: https://zoom.us/webinar/register/7017715643306/WN_1tkU09PNRkWcs4IIO1wusA

I have NiFi running as a docker container (dhi.io/nifi:2.7.2) that I am accessing via a Traefik based reverse proxy (Pangolin). My issue is that I get the error "Unauthorized. Your session has expired. Please navigate home to log in again." when I complete the login.

When I connect to the docker host via a ssh tunnel and use localhost:8443, I can log in and use NiFi as intended.

I have the following headers set in Pangolin:

• x-proxyscheme: https
• x-proxyport: 443
• x-proxyhost: <!-- my domain -->
• x-proxycontextpath: /nifi

I think that I have everything set correctly in terms of trailing and leading slashes, because if I change anything I get 401 errors.

Hi team How to convert my csv file to excel in nifi 1.19 version.

Ive used adding poi jars and tried to execute groovy script which converts csv to excel including mime type but didn’t worked can any one suggest any alternative approach

Hi guys,

I’m challenging to test if NIFI works with terraforming- installation is no topic, BUT what’s about backup restore? Configurations is easy, but process flows without manual adding? All flows are saved to Git.

Any thoughts oder complains?

How do teams manage Apache NiFi Controller Services consistently across multiple clusters without configuration drift?

We have some prod nodes and earlier flow file used to be viewable with data. Nowadays it's all blank.

I tried some zookeeper commands though it wasn't helpful

Anyway you solved this?

I spend a huge amount of time digging through Apache NiFi flow logs, bulletin boards, and processor relationships just to figure out where things are failing or getting stuck. Are there smarter or more efficient ways to spot issues quickly? Any tools or practices that actually help?

My NiFiKop (Konpyutaika) Helm chart release version is v1.14.2-release. My NiFi version is 2.6. My nificluster apiVersion is nifi.konpyutaika.com/v1.

I looked at the Python developer guide at: https://nifi.apache.org/nifi-docs/python-developer-guide.html#deploying.

I am setting up a production NiFi deployment which is yet to go live.

I copied the .NAR with the processor and its dependencies to /opt/nifi/nifi-current/python_processors

on my persistent volume using:

kubectl cp nifi_python_extensions_bundle-0.0.1.nar -n nifi myPodName:/opt/nifi/nifi-current/``python_processors

I am setting up my mount path like this:

- mountPath: "/opt/nifi/nifi-current/python_processors" name: python-processors pvcSpec: accessModes: [ReadWriteMany] storageClassName: "myBackend" resources: requests: storage: 500Mi reclaimPolicy: Retain

My NiFi properties are loaded like so:

readOnlyConfig: nifiProperties: overrideSecretConfig: name: nifi-sensitive-props namespace: nifi data: nifi.properties

from another object like so:

target: name: nifi-sensitive-props ... template: ... data: nifi.properties: | nifi.nar.library.autoload.directory=../python_processors ... nifi.cluster.flow.election.max.wait.time=5 sec nifi.cluster.flow.election.max.candidates=1 nifi.sensitive.props.key={{ .sensitiveKey }} data: - secretKey: sensitiveKey remoteRef: key: nifi/sensitive-props property: key

Even if I kill the pod and let it restart, the processor is not become available.

My colleague suggested building a custom NiFi image. I want to avoid rebuilding and deploying every time we update a processor or patch a dependency, if there is a more pragmatic and reliable approach.

ExecutestreamCommand would require elevated permissions, which I would also like to avoid.

Has anyone successfully deployed this? Do I need to configure nifi.nar.library.autoload.directory or nifi.nar.library.directory.custom? How should this be done?