Opening a cloud collection: stop paying a round trip per folder

Two findings, both classification rather than algorithm.

**A File-Provider folder was being walked as if it were local.**
`ResumableTreeWalk` already knows it is latency-bound — a source declares how
many listings it can usefully have in flight, `RemoteTreeSource` says six, and
the default is one because a real directory listing is a syscall over a warm
cache that gains nothing from overlapping. A vault on iCloud Drive, Dropbox or
any other Files provider has a *file path*, so it reaches `LocalTreeSource` and
inherited that serial default — but every listing there is an XPC round trip
into the provider's extension, and a network fetch behind it for a folder not
yet enumerated. N folders, N latencies, end to end. That is where the minutes
were.

`LocalTreeSource` now reports six for a provider-backed root, detected by path
(`/Library/Mobile Documents/`, `/Library/CloudStorage/`) rather than by asking
the coordinator — the question is asked once per scan and must not itself be a
round trip. An ordinary folder still reports one, which matters: routing width-1
through the concurrency window once made the local walk about four times slower,
which is why `theWalkIsCompetitiveWithTheEnumeratorOnARealisticVault` exists. A
test measures the overlap on a synthetic tree with a provider's latency profile.

**The direct-API path listed one folder at a time when it did not have to.**
Six in flight is still N/6 round trips for N folders, and every provider can
return a whole subtree in paginated batches — Dropbox `list_folder` with
`recursive: true`, Graph `/delta`, Drive's folder query, Box recursed
server-side. Dropbox's was already implemented and already in use: it is how
`changes(since: nil,)` obtains its cursor. The initial open simply never asked
for it.

`RemoteStore.listRecursively` is that call, defaulting to nil so a provider
gains it one at a time and nothing regresses meanwhile. `RecursiveListingCache`
fetches once and answers every `children(of:)` from memory.

Deliberately a cache rather than a replacement for the walk: building the tree
straight from a recursive listing would discard checkpointing, incremental
publishing, per-directory fault isolation and resumption, all of which are worth
more on a large interrupted sync than the walk's own bookkeeping costs. The walk
is untouched; only its expensive step is made free. Measured: twenty folders
cost twenty-one listings before and one recursive request after, finding the
same hundred files.

472 app tests, 408 editor.

Co-Authored-By: Claude <claude-opus-5> <noreply@anthropic.com>

Author: ChristineTham

HelloNotes/Core/Remote/DropboxStore.swift

@@ -80,6 +80,26 @@ final class DropboxStore: NSObject, RemoteStore, @unchecked Sendable {
         return all
     }
 
+    /// Every entry under `path`, from one recursive `list_folder` plus its
+    /// continuations — the same request `changes(since: nil,)` already makes to
+    /// obtain a cursor, asked here for the entries alone.
+    ///
+    /// A vault of three hundred folders costs three or four requests this way
+    /// rather than three hundred listings six at a time.
+    func listRecursively(path: String) async throws -> [RemoteEntry]? {
+        var data = try await sendAuthed {
+            Self.listFolderRequest(path: path, token: $0, recursive: true)
+        }
+        var page = try Self.parseListFolderPage(data)
+        var all = page.entries
+        while page.hasMore, let cursor = page.cursor {
+            data = try await sendAuthed { Self.listFolderContinueRequest(cursor: cursor, token: $0) }
+            page = try Self.parseListFolderPage(data)
+            all += page.entries
+        }
+        return all
+    }
+
     /// Dropbox's delta: a recursive `list_folder` issues a cursor, and
     /// `list_folder/continue` returns everything that changed after it —
     /// including deletions, which a plain re-list can only infer by absence.

HelloNotes/Core/Remote/RemoteMirror.swift

@@ -326,7 +326,13 @@ final class RemoteMirror {
         var updated = manifest
         var seen = Set<String>()
         var outcome = RemoteSyncOutcome()
-        let source = RemoteTreeSource(store: store, remoteRoot: remoteRoot, cacheRoot: cacheRoot)
+        // One recursive listing for the whole tree where the provider has one,
+        // consulted by every `children(of:)` below. The walk is unchanged — it
+        // simply stops paying a round trip per folder. See
+        // `RecursiveListingCache`.
+        let source = RemoteTreeSource(
+            store: store, remoteRoot: remoteRoot, cacheRoot: cacheRoot,
+            prefetch: RecursiveListingCache(store: store, root: remoteRoot))
 
         // Counted once and then kept, rather than recounted per directory.
         // `entries.values.count(where:)` walked the *whole* manifest on every

HelloNotes/Core/Remote/RemoteStore.swift

@@ -60,6 +60,23 @@ protocol RemoteStore: AnyObject, Sendable {
     /// design rather than polling the tree.
     func changes(since cursor: String?, path: String) async throws -> RemoteChangeSet?
 
+    /// Every entry under `path`, at every depth, in as few round trips as the
+    /// provider allows — or `nil` when it has no such call wired up here.
+    ///
+    /// **This is the difference between one request and one per folder.** The
+    /// walk is latency-bound: `RemoteTreeSource` overlaps six listings precisely
+    /// because each is a round trip the app spends idle, and six-at-a-time is
+    /// still N/6 latencies for N folders. Every provider can return a whole
+    /// subtree in paginated batches instead — Dropbox `list_folder` with
+    /// `recursive: true`, Graph `/delta`, Drive `files.list` with a folder
+    /// query, Box `/folders/:id/items` recursed server-side — turning a few
+    /// hundred round trips into a few.
+    ///
+    /// `nil` is not a failure: the caller falls back to walking directory by
+    /// directory, so a provider gains this one at a time and nothing regresses
+    /// while it does.
+    func listRecursively(path: String) async throws -> [RemoteEntry]?
+
     /// A cursor marking "everything up to now", **without** fetching any data.
     ///
     /// Taken at the end of a full sync so the *first* refresh can already use
@@ -73,6 +90,7 @@ protocol RemoteStore: AnyObject, Sendable {
 }
 
 extension RemoteStore {
+    func listRecursively(path: String) async throws -> [RemoteEntry]? { nil }
     func changes(since cursor: String?, path: String) async throws -> RemoteChangeSet? { nil }
     func latestCursor(path: String) async throws -> String? { nil }
 }

HelloNotes/Core/Remote/RemoteTreeSource.swift

@@ -38,8 +38,27 @@ struct RemoteTreeSource: TreeSource {
     /// 429 finishes later than one that never asked for it.
     var listingConcurrency: Int { 6 }
 
+    /// Answers every listing from one recursive fetch, when the provider can do
+    /// one. Shared by reference so the `struct` stays a value the walk can copy.
+    var prefetch: RecursiveListingCache? = nil
+
     func children(of directory: String) async throws -> DirectoryListing {
+        // **One request for the tree, rather than one per folder.**
+        //
+        // The walk stays exactly as it is — frontier, checkpoints, progress,
+        // per-directory fault isolation, resumption — and simply stops paying a
+        // network latency for each step. Overlapping six listings still leaves
+        // N/6 round trips for N folders; this leaves a handful for the whole
+        // tree. A provider without the call returns nil and nothing changes.
+        if let prefetch, let cached = await prefetch.children(of: remotePath(forRelative: directory)) {
+            return listing(from: cached)
+        }
         let entries = try await store.list(path: remotePath(forRelative: directory))
+        return listing(from: entries)
+    }
+
+    /// One directory's entries, as the walk's own shape.
+    private func listing(from entries: [RemoteEntry]) -> DirectoryListing {
         var listing = DirectoryListing()
         listing.children.reserveCapacity(entries.count)
         for entry in entries {
@@ -81,3 +100,62 @@ struct RemoteTreeSource: TreeSource {
         return remotePath(forRelative: relative)
     }
 }
+
+/// One recursive listing, fetched once and then answered from memory.
+///
+/// **Why a cache rather than replacing the walk.** The obvious use of a
+/// recursive listing is to build the tree from it directly and skip
+/// `ResumableTreeWalk` entirely — and that would throw away the checkpointing,
+/// the incremental publishing, the per-directory fault isolation and the
+/// resumption that make a large, interrupted sync survivable. All of that is
+/// worth more than the walk's own bookkeeping costs. So the walk is left intact
+/// and only its *expensive* step is made free.
+///
+/// Failure is not fatal by design: a provider that has no recursive call, or one
+/// whose call fails, leaves `entries` nil and every listing falls through to
+/// `store.list` exactly as before. The attempt is made once — a provider that
+/// refused will refuse again, and retrying per directory would be slower than
+/// never having tried.
+actor RecursiveListingCache {
+    private let store: RemoteStore
+    private let root: String
+    private var byDirectory: [String: [RemoteEntry]]?
+    private var attempted = false
+
+    init(store: RemoteStore, root: String) {
+        self.store = store
+        self.root = root
+    }
+
+    /// The entries directly inside `path`, or nil when there is no prefetch to
+    /// answer from.
+    func children(of path: String) async -> [RemoteEntry]? {
+        if !attempted {
+            attempted = true
+            byDirectory = await fetch()
+        }
+        return byDirectory?[normalise(path)] ?? (byDirectory == nil ? nil : [])
+    }
+
+    private func fetch() async -> [String: [RemoteEntry]]? {
+        // `try?` flattens the double optional, so "threw" and "not supported"
+        // arrive the same way — which is what we want: both mean walk instead.
+        guard let entries = try? await store.listRecursively(path: root) else { return nil }
+        // Group by parent. A recursive listing arrives flat and in no
+        // particular order, and the walk asks for one directory at a time.
+        var grouped: [String: [RemoteEntry]] = [:]
+        for entry in entries {
+            let parent = normalise((entry.path as NSString).deletingLastPathComponent)
+            grouped[parent, default: []].append(entry)
+        }
+        return grouped
+    }
+
+    /// Providers disagree about the trailing slash and about the case of the
+    /// root; the walk asks with whatever `remotePath(forRelative:)` produced.
+    private func normalise(_ path: String) -> String {
+        var p = path
+        while p.hasSuffix("/") { p.removeLast() }
+        return p.lowercased()
+    }
+}

HelloNotes/Core/ResumableTreeWalk.swift

@@ -356,6 +356,47 @@ nonisolated enum ResumableTreeWalk {
 /// A folder on disk, listed one level at a time.
 nonisolated struct LocalTreeSource: TreeSource {
     let root: URL
+
+    /// How many listings to keep in flight.
+    ///
+    /// **A folder on a File Provider is not a local folder wearing a path.**
+    /// The default of 1 is right for a real directory — a listing there is a
+    /// syscall over a warm cache and overlapping them buys nothing. It is
+    /// exactly wrong for iCloud Drive or any other Files provider, where
+    /// `contentsOfDirectory` is an XPC round trip into the provider's extension
+    /// and, for a folder it has not enumerated yet, a network fetch behind that.
+    /// Those are latencies with the app doing nothing at all, so a tree of N
+    /// folders cost N of them laid end to end — which is how adding a large
+    /// cloud vault came to take minutes.
+    ///
+    /// The classification was the bug: such a vault reaches
+    /// `LocalTreeSource` because it *has* a file path, inherited the serial
+    /// default meant for warm-cache syscalls, and so was walked one directory
+    /// at a time. `RemoteTreeSource` had already worked this out for the
+    /// direct-API providers and overlaps six.
+    ///
+    /// Nothing changes for an ordinary folder, which is the point: the width is
+    /// 1 unless the root is demonstrably provider-backed.
+    var listingConcurrency: Int { Self.isProviderBacked(root) ? 6 : 1 }
+
+    /// Whether `url` lives behind a file provider rather than on the disk.
+    ///
+    /// Two shapes, both by path, and deliberately not by asking the provider —
+    /// the question is asked once per scan and must not itself be a round trip:
+    ///
+    ///   * `…/Library/Mobile Documents/…` — iCloud Drive, including another
+    ///     app's ubiquity container, which is where an Obsidian vault lives.
+    ///   * `…/Library/CloudStorage/…` — every other Files provider on macOS
+    ///     (Dropbox, Google Drive, OneDrive, Box).
+    ///
+    /// `FileManager.isUbiquitousItem` would answer the first case and not the
+    /// second, and answers it by asking the coordinator — so it is both
+    /// narrower and more expensive than looking at the path.
+    static func isProviderBacked(_ url: URL) -> Bool {
+        let path = url.path
+        return path.contains("/Library/Mobile Documents/")
+            || path.contains("/Library/CloudStorage/")
+    }
     /// When false, non-Markdown files are dropped during the listing rather than
     /// collected and filtered later — the difference between reading and
     /// discarding a hundred thousand names, and ignoring them.