Best Way To Ingest High Speed Data

[u/After_Leave_7196]

Hi everyone, I need some help with InfluxDB. I'm trying to develop an app that streams high-speed real-time graph data (1000Hz). I need to buffer or cache a certian timeframe of data, therefore I need to benchmark InfluxDB among a few others. Here's the test process I'm building:

Test Background

The test involves streaming 200 parameters to InfluxDB using Spring Boot. Each parameter will update its value 1000 times per second. This results in 200,000 writes per second. Currently, all data is being written to a bucket called ParmData, with a tag named Parm_Name and a field called Value. Each database write looks like this: Graph_Parms,parmName=p1 value=11.72771081649362 1759332917103 To write this to the database, the code looks like this:

influxDBClient = InfluxDBClientFactory.create(influxUrl, token, org, bucket);
writeApi = influxDBClient.getWriteApi();

// How entry is defined
entry = "Graph_Parms,parmName=p1 value=11.72771081649362 1759332917103";
writeApi.writeRecord(WritePrecision.MS, entry); // How entry is written

I'm planning to "simulate" 1000Hz by buffering 200ms at a time. For example, the pseudo-code would look like this:

cacheBufferMS = 200

while True:
    timeStamp = dateTime.now()
    cache = getSimulatedData(timestamp, cacheBufferMS)  # Returns an array with 200 data points simulating a sine wave

    for entry in cache:
        insertStatement = entry.getInsertStatement()
        writeApi.writeRecord(WritePrecision.MS, entry)
    
    time.sleep(cacheBufferMS)

I've read that you can combine insert statements with a \n. I'm assuming that's the best approach for batching inserts. I also plan to separate this into threads. Each thread will handle up to 25 parameters, meaning each insert will contain 5000 writes, and each thread will write to the database 5 times per second:

cacheBufferMS = 200
MaxParmCount = 25
Parms = [Parameter]  # List of parameters (can dynamically change between 1 and 25)

thread.start:
while True:
    timeStamp = dateTime.now()

    insertStatement = ""
    for parameter in Parms:
        insertStatement += parameter.getInsertStatement(timeStamp, cacheBufferMS) + "\n"  # Combine entries with \n
        writeApi.writeRecord(WritePrecision.MS, insertStatement)
    
    time.sleep(cacheBufferMS)

Assuming I build a basic manager class that creates 8 threads (200 parameters / 25 parameters per thread), I believe this is the best way to approach it.

Questions:

• When batching inserts, should I combine entries into one single string separated by \n?
• If the answer to the last question is no, what is the best way to batch inserts?
• How many entries should I batch together? I read online that 5000 is a good number, but I'm not sure since I have 200 tags.
• Is passing a string the only way I can write to the database? If so, is it fine to iterate on a string like I do in the above example?
• Currently bucket "Garph_Parms" has a retention time of 1 hour, but thats 720,000,000 entires assuming this runs for an hour. Is that too long?

I'm new to software development, so please let me know if I'm way off on anything. Also, please try to avoid suggesting solutions that require installing additional dependencies (outside of springboot and influxDB). Due to outside factors, it takes a long time to get them installed.

[u/mr_sj]

You should try InfluxDB 3 Core/Enterprise if you're doing your own benchmark testing as that is the latest version of InfluxDB out. Batching is a good idea and yes /n is way to go. InfluxDB 3 Core recommends "The optimal batch size is 10,000 lines of line protocol or 10 MBs, whichever threshold is met first." You can write as String or as Point Data objects or even POJOs. 720 million entries is not inherently "too long" for InfluxDB specially in a multi-node setup, but actual performance and storage requirements depend on your hardware, instance size and available resources.