Skip to content

Pandas outputs warning when calling dataframe.append in flux_csv_parser._prepare_data_frame #613

Open
#698
Open
Pandas outputs warning when calling dataframe.append in flux_csv_parser._prepare_data_frame#613
#698
Labels
bugSomething isn't working

Description

@jonashoechst
jonashoechst
opened on Sep 19, 2023

Specifications

  • Client Version: 1.36.0
  • InfluxDB Version: 2.7

Code sample to reproduce problem

query_api.query_data_frame(query)

Expected behavior

The query should be executed without warnings.

Actual behavior

Pandas outputs a FutureWarning:

/usr/local/lib/python3.11/site-packages/influxdb_client/client/flux_csv_parser.py:256: FutureWarning:
The behavior of array concatenation with empty entries is deprecated. In a future version, this will no longer exclude empty items when determining the result dtype. To retain the old behavior, exclude the empty entries before the concat operation.

This is related to the this code snippet:

https://github.com/influxdata/influxdb-client-python/blob/aa6e5c4aad1aaf004195c175ebe424ac6040c2e3/influxdb_client/client/flux_csv_parser.py#L243C9-L256

Additional info

In the recent pandas version handling of concatenation of empty dataframes changed, see pandas-dev/pandas#39122. This should be handled else wise to reflect the changes in pandas.

Activity

  1. jonashoechst
    added
    bugSomething isn't working
    on Sep 19, 2023
  2. bednar
    assigned
    Sciator
    on Oct 18, 2023

  3. Sciator commented on Oct 26, 2023

    @Sciator
    Sciator
    on Oct 26, 2023
    Contributor

    I wasn't able to replicate this issue. I have tried to run this with 2.0.3 and 2.1.1 pandas with no warning.
    Could you provide more information (query, data etc.) ?

  4. bednar commented on Oct 26, 2023

    @bednar
    bednar
    on Oct 26, 2023 via email
    Contributor

  5. ivankudibal commented on Nov 21, 2023

    @ivankudibal
    ivankudibal
    on Nov 21, 2023

    pls, it would be great if we have data and the query to reproduce, before we can go on with this issue

  6. bednar
    unassigned
    Sciator
    on Jan 3, 2024
  7. bednar
    assigned
    karel-rehor
    on Jul 24, 2024
  8. bednar
    unassigned
    karel-rehor
    on Sep 5, 2024

  9. oskarwilliams commented on Oct 11, 2024

    @oskarwilliams
    oskarwilliams
    on Oct 11, 2024

    I am seeing this issue as well, do you still need a query to repro this?

  10. bednar commented on Oct 11, 2024

    @bednar
    bednar
    on Oct 11, 2024
    Contributor

    Hi @oskarwilliams, the query and possibly an example of your data in the database would be helpful… Thanks.

  11. dave-cz commented on Nov 7, 2024

    @dave-cz
    dave-cz
    on Nov 7, 2024

    Python 3.12
    Pandas 2.2.3
    InfluxDB 2.7.5
    influxdb_client 1.47.0

    Query results from InfluxDB Data Explorer
    query(1).csv

    Code:

    query_api.query_data_frame(query, data_frame_index=['time'])

    query:

    from(bucket: "model")
|> range(start: 2024-10-27T23:00:00Z, stop: 2024-11-03T22:00:00.000001Z)
|> filter(fn: (r) => r["_measurement"] == "sp_el.J612")
|> aggregateWindow(every: 1h, fn: last, timeSrc: "_start", timeDst: "time", createEmpty: true)
|> pivot(rowKey: ["time"], columnKey: ["_measurement", "_field"], valueColumn: "_value")`

    flux_csv_parser.py l.258
    df = pd.concat([self._data_frame.astype(_temp_df.dtypes), _temp_df])

    >> self._data_frame

Empty DataFrame
Columns: [result, table, _start, _stop, sp_el.J612_value]
Index: []

    >> _temp_df
                            result  table                    _start                            _stop  sp_el.J612_value
time                                                                                                                  
2024-10-27 23:00:00+00:00  _result      0 2024-10-27 23:00:00+00:00 2024-11-03 22:00:00.000001+00:00               NaN
2024-10-28 00:00:00+00:00  _result      0 2024-10-27 23:00:00+00:00 2024-11-03 22:00:00.000001+00:00               NaN
...
2024-10-29 22:00:00+00:00  _result      0 2024-10-27 23:00:00+00:00 2024-11-03 22:00:00.000001+00:00               NaN
2024-10-29 23:00:00+00:00  _result      0 2024-10-27 23:00:00+00:00 2024-11-03 22:00:00.000001+00:00               2.0
2024-10-30 00:00:00+00:00  _result      0 2024-10-27 23:00:00+00:00 2024-11-03 22:00:00.000001+00:00               2.0
...
2024-10-31 22:00:00+00:00  _result      0 2024-10-27 23:00:00+00:00 2024-11-03 22:00:00.000001+00:00               2.0
2024-10-31 23:00:00+00:00  _result      0 2024-10-27 23:00:00+00:00 2024-11-03 22:00:00.000001+00:00               NaN
...
2024-11-03 22:00:00+00:00  _result      0 2024-10-27 23:00:00+00:00 2024-11-03 22:00:00.000001+00:00               NaN

    _temp_df.to_csv('temp_df.csv')

    temp_df.csv

    WARNING    warnings                  C:\my_project\venv\Lib\site-packages\influxdb_client\client\flux_csv_parser.py:258: FutureWarning:
The behavior of array concatenation with empty entries is deprecated. In a future version, this will no longer exclude empty items when determining the result dtype. To retain the old behavior, exclude the empty entries before the concat operation.

    I see

    >>> df.equals(_temp_df)
True

    so possible solution can look like

    if self._data_frame.empty:
    df = _temp_df
else:
    df = pd.concat([self._data_frame.astype(_temp_df.dtypes), _temp_df])

    but I'm not sure about consequences across all codes

  12. bednar
    assigned
    karel-rehor
    on Nov 13, 2024

  13. djhoese commented on Jul 23, 2025

    @djhoese
    djhoese
    on Jul 23, 2025

    Any updates on this? I'm also seeing this, but it looks like the last comment provided detailed information.

  14. bednar
    unassigned
    karel-rehor
    on Sep 25, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions