Repository navigation
feat(workspace): use filesystem storage without S3 - #13012
Merged
Merged
Conversation
Sync filesystem publication and isolate materialized tool files from immutable blob inodes. Reject symlink paths within the blob tree. Opt shared PostgreSQL participants into revision-lock coordination for ingest, reference publication, and purge checks and deletion. Preserve legacy ingest order when the explicit option is disabled. Verify process publication and interruption, injected failures, and shared-schema PostgreSQL races across independent connection pools. Assisted-by: nib:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Honor immediate caller cancellation and give delayed PostgreSQL cleanup an independent 30-second budget for revision-lock acquisition and reference checks. Preserve blobs when cleanup cannot acquire the lock. Cover cancellation, delayed cleanup, and pool release. Convert publication tests to Ginkgo/Gomega while retaining independent-process child bootstrapping. Assisted-by: nib:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Allow Files participants to share immutable blobs without S3. Require PostgreSQL metadata and reject invalid explicit roots at startup. Keep default storage and model staging unchanged. Document the volume, permissions and coordinated backend-switching contract. Assisted-by: nib:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Cancel the owned startup context and close the auth pool when explicit shared Files startup fails. Preserve the caller context and legacy startup behavior. Cover early validation and late Files failure with PostgreSQL-backed cleanup assertions. Assisted-by: nib:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Use configured S3 for every workspace participant, including owners without distributed services. Otherwise retain DataPath/chats/blobs through the direct filesystem adapter, without another path setting. Derive PostgreSQL publication and purge coordination from the blob backend so non-distributed owners also protect shared filesystem bytes. Keep shared mount and metadata identity an explicit operator contract. Remove the opt-in probe and document storage selection without migration. Verify automatic selection with red/green regressions, S3 constructor spies, focused startup tests and blob race tests. Exercise shared files and PostgreSQL coordination against an owned PostgreSQL 16 fixture. Assisted-by: nib:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Own the startup child context for configured S3 as well as PostgreSQL metadata. Cancel background work, shut down opened resources and close the auth pool on fatal initialization errors without canceling the caller context or changing legacy JSON/SQLite without S3. Regress missing-bucket failures using temporary SQLite and no-auth owners through the production cleanup helper. Verify pool closure, owned-context cancellation and resource shutdown. Assisted-by: nib:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
mudler-agent
changed the title
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| if err := s.stage("directory sync"); err != nil { | ||
| return err | ||
| } | ||
| f, err := os.Open(path) |
| if err := s.safe(p); err != nil { | ||
| return nil, err | ||
| } | ||
| f, err := os.Open(p) |
| if err != nil { | ||
| return "", 0, err | ||
| } | ||
| defer func() { f.Close(); os.Remove(f.Name()) }() |
| if err != nil { | ||
| return "", 0, err | ||
| } | ||
| defer func() { f.Close(); os.Remove(f.Name()) }() |
Mark best-effort cleanup errors explicitly and inspect the config type without copying its atomic fields. Document why the validated blob paths do not permit file inclusion. Keep publication, synchronization and PostgreSQL coordination unchanged. Assisted-by: nib:gpt-6.1-sol Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
This was referenced Nov 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Allow distributed workspace Files to use the existing filesystem backend when S3 is not configured. No new flag or workspace path is needed.
Workspace blob selection is automatic for frontends, the add-on owner and optional chat workers:
DataPath/chats/blobsdirectory through the direct workspace filesystem adapter.Distributed deployments without S3 must mount the same backing data directory on every Files participant and use shared PostgreSQL metadata. Equal path strings do not prove shared storage; this is an operator responsibility. No authoritative filesystem blobs are placed in temporary scratch.
Global model/input staging and the generic distributed ObjectStore remain unchanged. Standalone JSON/SQLite workspace behavior remains supported. The owner does not need a different distributed role.
Notes for Reviewers
This revision removes the custom path and opt-in configuration from the earlier version of this PR. The safety fixes remain:
PostgreSQL filesystem uploads hold the workspace revision lock through ingestion, serializing metadata writers during slow uploads. Input readers must honor cancellation while blocked. SQLite/JSON behavior is preserved; S3 does not use filesystem coordination.
Documentation covers shared metadata versus shared blob bytes, same-node PVC placement, coherent RWX requirements and compatible service permissions. Existing filesystem blob paths stay unchanged. This PR does not copy or migrate existing data. A backend change on populated metadata still requires a separate copy/verification procedure and consistent selection across participants.
Verification
Commands run with
GOPROXY=off GOTOOLCHAIN=local:With
LOCALAI_TEST_POSTGRES_URLset to a new disposable PostgreSQL 16 fixture:Root verification passes: blob 30 specs, non-PostgreSQL workspace 137 specs, config/CLI and application regressions, PostgreSQL application 3 selected specs, PostgreSQL shared-filesystem 14 selected specs, and all PostgreSQL workspace 142 selected specs. No selected specs skipped at runtime; other specs were excluded by focus. Disposable containers were removed. Independent final review passes and independently reran application/config/CLI/blob unit tests and diff checks.
Genuine regression failures were observed before implementation, including automatic selection, owner S3 validation and standalone S3 startup cleanup.
Limits: no full build/lint, real RWX storage, live S3, UI or full-server startup validation. S3 tests use spies plus constructor initialization against a fake endpoint. Existing local protobuf generators and a prebuilt real UI bundle satisfy ignored compile prerequisites; no generated artifacts enter this PR. Real volume identity and power-loss behavior remain deployment checks. No infrastructure changes or deployment included.
Signed commits
docs/content/features/chats.mdanddistributed-mode.md).