Skip to content

Fix SFTP pattern timestamp comparisons with non-UTC settings - #73386

Draft
mifuha wants to merge 1 commit into
apache:mainfrom
mifuha:fix/sftp-pattern-timezone
Draft

mifuha wants to merge 1 commit into
apache:mainfrom
mifuha:fix/sftp-pattern-timezone

Conversation

@mifuha

@mifuha mifuha commented Sep 19, 2026

Copy link
Copy Markdown

When SFTPTrigger uses file_pattern with newer_than, different process and Airflow timezones can make it choose the wrong files. A file modified at 12:00 UTC can incorrectly pass a 14:00 UTC threshold. Reversing the timezone settings can reject a file that is new enough.

This fix reads the modification timestamp directly as UTC. It keeps the existing inclusive comparison and whole-second precision.

The regression tests run the real trigger in fresh processes with different timezone settings and mocked SFTP calls. They cover both wrong decisions, UTC controls, equal timestamps, a non-hour timezone offset, fractional seconds and missing modification times.

Related: #73382. The single-file trigger and synchronous sensor paths still use the existing string-based hook methods and remain affected. This PR leaves those interfaces unchanged, so the issue should stay open.

Checks run with locked dependencies:

  • Both mismatch cases failed before the fix; the UTC controls passed.
  • All 55 focused SFTP tests passed after the fix on Python 3.12.3.
  • All 11 timezone regression cases passed on Python 3.10.21 and 3.12.3.
  • The regular pre-commit checks and commit-message check passed.
  • git diff --check passed.

The manual provider check could not finish because the local environment reported bus and I/O errors while Docker was exporting its CI image. The full provider suite, live SFTP tests and compatibility matrix were not run locally.

@mifuha


Was generative AI tooling used to co-author this PR?
  • Yes — Codex (GPT-6)

AI-assisted: following the guidelines. I used Codex to help investigate the issue, write the fix and tests.

Different process and Airflow timezones can make a file modified at
12:00 UTC pass a 14:00 UTC threshold, or reject a file that is new enough.
Pattern matching should compare absolute modification times regardless
of either timezone setting.

Related: apache#73382
@boring-cyborg

boring-cyborg Bot commented Sep 19, 2026

Copy link
Copy Markdown

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

@mifuha
mifuha marked this pull request as ready for review September 19, 2026 19:09
@mifuha
mifuha marked this pull request as draft September 19, 2026 19:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant