If multiple Python processes write data into InfluxDB, only parts are written

Question

I have multiple Python processes, running on a server (RPI5) by cron, which read data from web API's, and write it then into a common InfluxDB database - on the same bucket.

However, some of the data is lost. The code to write into Influx is:

influxdb_client = InfluxDBClient(url=url, token=token, org=org) ... def f(df): write_api = influxdb_client.write_api() ... record = [] for i in range(df.shape[0]): point = Point(measurement).tag("location", ...).time(...) for col in list(df.columns): value = df.loc[i, col] point = point.field(col, value) record += [point] write_api.write(bucket=bucket, org=org, record=record) ... ## Let df be a data.frame with 20-500 rows, and 10-20 columns. f(df)

What could be the reason of this issue? Some problem with asynchronous/synchronous?

Thx

Have a number of issues with this, have you checked the logs for errors, how do you actually know data is missing ? . defining a function f() has taking no arguments then calling it with an argument should throw a TypeError ... , is this actual code ... — ticktalk
– ticktalk, Commented Jul 5, 2024 at 6:25
Ok, I missed the "df" argument in f(df), when simplifying the code for the example. The data loss is shown in Influx Data Explorer, but also read clients. I struggle between SYCHRONOUS api and ASYNC. Both are not really working as expected for this parallel use case. — lambruscoAcido
– lambruscoAcido, Commented Jul 5, 2024 at 10:16
maybe InfluxDB is not "thread safe". maybe when you use it with many processes then it can't control it and finally some processes delete values from other processes. Maybe you should have only one process which access InfluxDB and other should use queue to send data to this process. — furas
– furas, Commented Jul 5, 2024 at 12:39
Given I used first MongoDB as a database, I am sort of disappomited about InfluxDB. Mongo had no struggles at all with parallel processes writing and reading. Sigh. Adding now async or queue stuff makes the code more complex, which I wanted to avoid. — lambruscoAcido
– lambruscoAcido, Commented Jul 8, 2024 at 7:14

Suyash · Accepted Answer · 2024-08-09 12:37:34Z

1

Your reasoning seems correct, modify the write_api() to use the synchronous mode and also add error handling logic to the code and try again.

answered Aug 9, 2024 at 12:37

Suyash

3602 silver badges11 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

lambruscoAcido Over a year ago

Thanks. I will give it a try. As told, hoped it would be easier.

Collectives™ on Stack Overflow

If multiple Python processes write data into InfluxDB, only parts are written

1 Answer 1

1 Comment

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Related