Skip to content

feat: implement the FileAutomation 1.0 roadmap - #109

Merged
JE-Chen merged 136 commits into
devfrom
feat/universal-storage-layer
Oct 11, 2026
Merged

JE-Chen merged 136 commits into
devfrom
feat/universal-storage-layer

Conversation

@JE-Chen

@JE-Chen JE-Chen commented Oct 8, 2026

Copy link
Copy Markdown
Member

Implements the FileAutomation 1.0 roadmap (#107) on top of dev. Nothing was removed: every FA_* action, facade name and command-line flag of 0.0.x still works. docs/source/Eng/usage/migration.rst lists the few behaviours that changed.

What is in it

  • Universal storage layer: File / Storage over one StorageBackend contract and one URI syntax. Twelve backends: local, in-memory, S3, Azure Blob, Google Drive, OneDrive, Dropbox, SFTP, FTP/FTPS, WebDAV, SMB, fsspec. An 88-case contract suite that every backend passes; copy_between now runs on the layer.
  • Extras: the base install carries no cloud SDK and no GUI toolkit; each backend, Parquet and the GUI is an extra ([all] for everything).
  • IntegrityMonitor 2.0: baselines and verification of a tree in any backend, six kinds of change, watching, opt-in remediation.
  • Pipeline runtime: retry, timeout, cancellation, conditions, idempotency, checkpoint and resume, dry run, history, YAML/JSON definitions.
  • Scheduler v2: cron with a time zone, manual, file, event and pipeline triggers; a record of every run.
  • Events, notification router, audit schema v2: the router and the audit trail are opt-in.
  • Semantic MCP tools: fourteen tools, read-only and confined to named roots by default, with a dry run.
  • UI 2.0 on an application layer (automation_file.app), which the web UI renders too.
  • CLI: storage, integrity, pipeline and audit subcommands.
  • Docs in three languages: storage, events, integrity, audit, pipelines, scheduler, MCP, GUI, application layer, public API policy, migration, deployment, integration tests.

Problems fixed on the way

  • A move between two views of one store (two roots, two prefixes) overwrote the file and then deleted it.
  • WebDAVClient could be sent, credentials included, to another host by an absolute path or a redirect.
  • ActionACL and the MCP --allowed-actions list did not check actions nested in the arguments of another action.
  • FA_storage_verify could not fail a pipeline task; it takes strict=True now.

Decisions made here, open to change before 1.0

  • OneDrive is the eleventh remote backend of the contract; Box stays action-only.
  • The public API and deprecation policy (docs/source/Eng/usage/api_policy.rst).
  • A MINOR or MAJOR release is made by writing X.Y.0 in stable.toml and dev.toml in the pull request to main (scripts/stable_release.py).
  • tzdata is a base dependency on Windows, for time zones in the scheduler.

Checks

Locally, on Windows with Python 3.14: 5763 passed and 257 skipped with every extra, 3879 passed and 137 skipped with the base dependencies only, ruff check, ruff format --check and mypy automation_file clean.

Not verified: please read

  • .github/workflows/integration.yml and tests/integration/ have never run. There is no Docker on the development machine, so this pull request is their first run. The container options, the Samba and FTP ones most of all, may need adjusting. These jobs do not gate publishing.
  • No storage adapter has met a real service; all were tested against stand-ins.
  • The unit tests have not run on Linux or macOS, nor on Python 3.10 to 3.13.
  • UI 2.0 has only run on Qt's offscreen platform, never on a real display.
  • The Sphinx build of the new pages was not run.

progress.md lists what remains open; docs/updates/2026-10.md has one entry per piece of work (U-20261008-01 to -32).

JE-Chen and others added 30 commits May 19, 2023 17:56
Update stable version and Add dependabot
Add back print execute event and update dev version
Add CLI mode & Update dev and stable version
Rename package and update version
Cron with a time zone, manual, file, event and pipeline triggers; action-list and pipeline targets; a record of every run with seven states; overlap protection, timeouts and cancellation.
One plain-Python service per navigation entry, a sidebar GUI built on them with a pipeline editor, and a web UI that renders the same services. The older tabs stay under Advanced.
@codacy-production

codacy-production Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Not up to standards ⛔

🔴 Issues 3 medium · 1 minor

Alerts:
⚠ 4 issues (≤ 0 issues of at least minor severity)

Results:
4 new issues

Category Results
UnusedCode 1 minor
ErrorProne 3 medium

View in Codacy

🟢 Metrics 9000 complexity · 81 duplication

Metric Results
Complexity 9000
Duplication 81

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

JE-Chen and others added 20 commits October 8, 2026 16:04
MinIO from quay.io, a writable FTP root, Samba on port 445 and a replace that removes the target first, a 400 from Apache for a path below a file, a versioning name too long to ask about on macOS, the package job on the locked tools, and equality tests written so that a scanner does not read them as self-comparisons.
File-level pylint disables in the test modules for the rules pytest idioms trip, each with its reason; markers with reasons where a scanner reads a test digest or a constant SQL statement as a risk; nine long lines wrapped; one variable renamed. The S3 integration job uses S3Mock, since MinIO's image can no longer be pulled.
…orage-layer

Integrate FileAutomation development into main
…on-1.0-roadmap

docs: define FileAutomation 1.0 roadmap and architecture
@JE-Chen
JE-Chen merged commit 169e8fa into dev Oct 11, 2026
1 of 2 checks passed
@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants