worktree: use ReFS block cloning for unchanged files by Copilot · Pull Request #6424 · git-for-windows/git

Copilot · 2026-09-18T11:55:37Z

Linked worktrees share Git objects but duplicate checked-out file data. On capable Windows volumes, ReFS block cloning can share those file extents using copy-on-write semantics.

Platform abstraction
- Add a portable block-cloning interface with an unsupported fallback.
- Implement Windows support with FSCTL_DUPLICATE_EXTENTS_TO_FILE.
- Detect same-volume block-refcounting capability before enabling it.
Worktree checkout
- Pass the originating worktree to the new worktree’s checkout.
- Clone only regular, unconverted files whose source bytes hash to the requested blob.
- Preserve parallel checkout for ineligible entries.
- Fall back silently to normal blob checkout when validation or cloning fails.
Correctness
- Handle ReFS alignment and request-size constraints.
- Restore the exact destination length after cloning an aligned tail.
- Reject dirty, filtered, encoded, or EOL-converted source files.

Fixes [Feature Request] Use ReFS block cloning for Git worktrees #6423

ReFS can share file extents through FSCTL_DUPLICATE_EXTENTS_TO_FILE, but callers should not need platform-specific logic. Provide a compatibility primitive that reports unsupported elsewhere, allowing callers to use block cloning as an opportunistic optimization with an ordinary write as fallback. Signed-off-by: GitHub <noreply@github.com> Co-authored-by: GPT-5.4 <noreply@openai.com> Co-authored-by: dscho <127790+dscho@users.noreply.github.com>

Linked worktrees can contain many identical large files even though they already share the object database. On filesystems that advertise block refcounting, pass the current worktree to the checkout and clone only regular, unconverted files whose contents hash to the requested blob. Any unsupported operation or clone failure falls back silently to the normal checkout path. Signed-off-by: GitHub <noreply@github.com> Co-authored-by: GPT-5.4 <noreply@openai.com> Co-authored-by: dscho <127790+dscho@users.noreply.github.com>

Document the alignment and chunk-size requirements, and restore the exact destination length after cloning the rounded final range. Signed-off-by: GitHub <noreply@github.com> Co-authored-by: GPT-5.4 <noreply@openai.com> Co-authored-by: dscho <127790+dscho@users.noreply.github.com>

Explain why the destination is sized both before and after cloning the aligned extent range. Signed-off-by: GitHub <noreply@github.com> Co-authored-by: GPT-5.4 <noreply@openai.com> Co-authored-by: dscho <127790+dscho@users.noreply.github.com>

Keep converted and non-regular entries eligible for parallel checkout because they cannot use the block-clone path. Document the helper fallback contract and source-path lifetime. Signed-off-by: GitHub <noreply@github.com> Co-authored-by: GPT-5.4 <noreply@openai.com> Co-authored-by: dscho <127790+dscho@users.noreply.github.com>

qub1n · 2026-09-18T12:44:42Z

+
+	hash_fd = dup(src_fd);
+	if (hash_fd < 0 ||
+	    index_fd(state->istate, &oid, hash_fd, &st, OBJ_BLOB, NULL, 0) ||


It is necessarily to read the source file's contents? What about performance? Isn't there better approach?

That would indeed defeat the purpose of improving performance.

dscho

Okay for a first draft of something that looks like it might eventually work, but the current shape strongly suggests that a lot more work is needed, @copilot.

dscho · 2026-09-18T13:28:56Z

+ * the destination behind when cloning is inapplicable or fails, so the caller
+ * can use the normal checkout path.
+ */
+static int clone_entry(const struct cache_entry *ce, char *path,


Should any entry that is non sparse and that is marked up to date in the original work trees index be eligible for cloning?

Addressed in 484f7b7 and refined in 979e245. Eligibility now comes from the refreshed source index: the entry must be regular, non-sparse, have the same blob OID and conversion attributes, and its opened-file stat data must match the index and pre-refresh timestamp checks before and after copying.

dscho · 2026-09-18T13:31:13Z

+static inline int git_block_cloning_supported(const char *source UNUSED,
+					      const char *destination UNUSED)
+{
+	return 0;
+}
+#define block_cloning_supported git_block_cloning_supported


This does not need to be a function at all. This can be much, much simpler. It can be a mere #define block_cloning_supported 0.

Besides, this function name is awful. It is technically correct, but it reflects the implementation much more than the intention. This is not what we want here. We want to describe the intention, and the intention is to see whether we can do a "copy on write" trick or not.

Addressed in 50ede48. The portable capability is now the intent-based copy_on_write_supported macro, with #define copy_on_write_supported 0 as the non-Windows fallback.

dscho · 2026-09-18T13:32:31Z

+static inline int git_block_clone_file(int dst_fd UNUSED,
+				       int src_fd UNUSED,
+				       off_t size UNUSED)
+{
+	errno = ENOSYS;
+	return -1;
+}


This could be a ton simpler. something like #defined block_clone_file (errno = ENOSYS, -1) should do. You must reason harder to come up with simpler solutions instead of stopping at the first plausible point.

Addressed in 50ede48. The fallback is now the simple #define file_copy_on_write(...) (errno = ENOSYS, -1) macro.

dscho · 2026-09-18T13:33:34Z

 	)
 '

+test_expect_success 'block-clone source does not change checkout contents' '


This test probably needs a prereq. Because if you simply execute it when block cloning isn't even supported, you ran commands for no benefit (and a false sense of validation).

Addressed in 60cb518 and ff7e710. A lazy COPY_ON_WRITE prerequisite performs an actual file_copy_on_write() operation, so unsupported file systems skip the behavior tests without running their setup.

dscho · 2026-09-18T13:34:08Z

 '

+test_expect_success 'block-clone source does not change checkout contents' '
+	test_create_repo clone-source &&


Why can't this test reuse the already-set up worktree?

Are tests running on machines with ReFS filesystem and NTFS filesystem?

Addressed in ff7e710. The test now reuses the repository established by the existing setup instead of creating a separate source repository.

Are tests running on machines with ReFS filesystem and NTFS filesystem?

No. We don't have self-hosted runners, so we only get whatever GitHub Actions runners provide, which I strongly suspect to be NTFS-only. If in doubt, feel free to test this (by creating a workflow that tests this and running it in your own fork).

I will try to come with better measurement of space saving and with some performance optimization as I suggested.

Tested on commit qub1n@a25a4e5

There must be something wrong in implementation, I would expect that cloning should pay of sooner, I mean on medium size already.

New round on qub1n@7d4abe2

I remove prefilling with zeros and executed new benchmarks:

I did test on larger datasets especially for small files, some benchmarking measurements were fixed, so this results should be more accurate to reality. Still with qub1n@7d4abe2

I computed expected space saving if algorithm will clone only files over size of which is cloning faster then checkout.

So I expect space saving and same or better performance:

dscho · 2026-09-18T13:41:42Z

 int mingw_core_config(const char *var, const char *value,
 		      const struct config_context *ctx, void *cb);
 #define platform_core_config mingw_core_config
+int mingw_block_cloning_supported(const char *source, const char *destination);


Why does this need to be exported?

Addressed in 50ede48. The exported runtime filesystem probe was removed; the platform operation itself reports unsupported cases through the portable fallback contract.

dscho · 2026-09-18T13:43:08Z

+	if (source &&
+	    block_cloning_supported(source, absolute_path(path)))
+		strvec_pushf(&cp.env, "%s=%s",
+			     GIT_WORKTREE_BLOCK_CLONE_SOURCE, source);


Why should this be conditional? It should be reset that decides whether block cloning is supported. And it should also have a config knob to prevent it, in case of bugs or something.

Addressed in 484f7b7, d078954, and aac02f0. reset now owns capability/config policy and honors worktree.copyOnWrite=false; the setting is documented and tested.

dscho · 2026-09-18T13:57:08Z

-static int reset_index(const char *ref, const struct object_id *oid, int reset_type, int quiet)
+static int reset_index(const char *ref, const struct object_id *oid,
+		       int reset_type, int quiet,
+		       const char *block_clone_source)


This formatting does not agree with the rest of the source code. Keep the long line as it was, replace the closing ) by a ,, then indent the next line with tabs followed by spaces to match exactly the column after the opening parenthesis in the preceding line.

Addressed in 484f7b7. The declaration keeps the original long first line and aligns the continuation exactly after the opening parenthesis as requested.

dscho · 2026-09-18T13:58:26Z

 	opts.dst_index = the_repository->index;
 	opts.fn = oneway_merge;
 	opts.merge = 1;
+	opts.block_clone_source = block_clone_source;


Once again. Naming is hard. Do not let bleed implementation details into names. One platform might call it "block_clone", another one cow, yet another one even differently. I would expect the name copy_on_write to be universally acceptable, and file_copy_on_write() a fine function name to implement this functionality.

Addressed in 50ede48 and 484f7b7. The API and all plumbing now use intent-based copy_on_write / file_copy_on_write() naming rather than the Windows implementation term.

dscho · 2026-09-18T14:01:35Z

+			     struct strvec *child_env, const char *path)
 {
 	struct child_process cp = CHILD_PROCESS_INIT;
+	const char *source = repo_get_work_tree(the_repository);


This is unlikely to be enough. A repository might be bare, after all. And even if it is not bare, you need to have a look at the Git index to see which files are eligible for block cloning. It might actually be quite prohibitive to provide a list of those files to the reset child process. A better idea would probably to make the checks here whether it is a non-bare repository, to refresh the Git index if so, and most crucially, to record a timestamp. That way, the child process could just look up the file in the Git index, and if it is up to date, block clone it. That would also resolve the issue where you needlessly exclude filtered files from block-cloning (which, in the case of Git LFS, would hurt particularly much).

Addressed in 484f7b7. Bare repositories omit source metadata; non-bare sources record time before a gentle index refresh/write and pass the source index to reset. Checkout validates matching index/stat/timestamp state, allowing clean converted and LFS-managed files without hashing their working-tree bytes.

compat: add copy-on-write file support Linked worktrees can share the storage backing identical files on file systems that provide copy-on-write copies. Callers should be able to request that optimization without knowing the platform-specific mechanism or treating lack of support as an error. Provide file_copy_on_write() on Windows and a minimal ENOSYS fallback elsewhere. ReFS requires aligned ranges smaller than 4 GiB, so use 4 GiB minus 64 KiB for full ranges and issue one rounded request for the tail, trying the two valid ReFS cluster sizes. These are alternative tail alignments, not 64 KiB-sized chunks. Truncating afterward removes only the alignment padding. Signed-off-by: GitHub <noreply@github.com> Co-authored-by: GPT-5.4 <noreply@openai.com> Co-authored-by: dscho <127790+dscho@users.noreply.github.com>

Co-authored-by: dscho <127790+dscho@users.noreply.github.com>

compat: add copy-on-write file support Linked worktrees can share the storage backing identical files on file systems that provide copy-on-write copies. Callers should be able to request that optimization without knowing the platform-specific mechanism or treating lack of support as an error. Provide file_copy_on_write() on Windows and a minimal ENOSYS fallback elsewhere. ReFS requires aligned ranges smaller than 4 GiB, so use 4 GiB minus 64 KiB for full ranges and issue one rounded request for the tail, trying the two valid ReFS cluster sizes. These are alternative tail alignments, not 64 KiB-sized chunks. Truncating afterward removes only the alignment padding. Signed-off-by: GitHub <noreply@github.com> Co-authored-by: GPT-5.4 <noreply@openai.com> Co-authored-by: dscho <127790+dscho@users.noreply.github.com> Signed-off-by: GitHub <noreply@github.com>

worktree: copy unchanged files on write Creating a linked worktree normally materializes another physical copy of every checked-out file. File systems with copy-on-write support can avoid that cost, but only when an existing working-tree file is known to represent exactly the target index entry. For a non-bare source, refresh and write its index after recording the current wall-clock time. The reset child loads that index and considers non-sparse entries whose object IDs and conversion attributes match. It requires the opened file stat data to match the refreshed index and predate the recorded time, both before and after copying, which retains Git\047s racy-index protection without hashing converted or Git LFS content. Keep the decision in reset and honor worktree.copyOnWrite=false so users can disable the optimization if a platform implementation proves problematic. Unsupported operations and all validation failures retain the normal checkout path. Signed-off-by: GitHub <noreply@github.com> Co-authored-by: GPT-5.4 <noreply@openai.com> Co-authored-by: dscho <127790+dscho@users.noreply.github.com>

Expose the primitive through test-tool so tests can establish a runtime prerequisite by performing a real copy-on-write operation. Signed-off-by: GitHub <noreply@github.com> Co-authored-by: GPT-5.4 <noreply@openai.com> Co-authored-by: dscho <127790+dscho@users.noreply.github.com>

Exercise the optimization only after a real copy-on-write probe succeeds. Verify Trace2 records successful copies for clean raw and CRLF-converted files, excludes dirty raw and converted files, and honors the configuration opt-out. Signed-off-by: GitHub <noreply@github.com> Co-authored-by: GPT-5.4 <noreply@openai.com> Co-authored-by: dscho <127790+dscho@users.noreply.github.com>

Document the copy-on-write optimization and its opt-out. Signed-off-by: GitHub <noreply@github.com> Co-authored-by: GPT-5.4 <noreply@openai.com> Co-authored-by: dscho <127790+dscho@users.noreply.github.com>

Avoid refreshing and writing the source index on platforms where the copy-on-write primitive is unavailable. Signed-off-by: GitHub <noreply@github.com> Co-authored-by: GPT-5.4 <noreply@openai.com> Co-authored-by: dscho <127790+dscho@users.noreply.github.com>

Document the lifetime of the caller-owned copy source passed through unpack_trees_options. Signed-off-by: GitHub <noreply@github.com> Co-authored-by: GPT-5.4 <noreply@openai.com> Co-authored-by: dscho <127790+dscho@users.noreply.github.com>

Keep the configuration opt-out separate from capability and metadata validation so the successful lookup semantics are explicit. Signed-off-by: GitHub <noreply@github.com> Co-authored-by: GPT-5.4 <noreply@openai.com> Co-authored-by: dscho <127790+dscho@users.noreply.github.com>

State explicitly that both the copy source and its index remain caller-owned for the duration of unpacking. Signed-off-by: GitHub <noreply@github.com> Co-authored-by: GPT-5.4 <noreply@openai.com> Co-authored-by: dscho <127790+dscho@users.noreply.github.com>

Resolve the source index entry and its conversion attributes once before deciding whether checkout can run in parallel, then carry that entry into the synchronous copy-on-write path. Signed-off-by: GitHub <noreply@github.com> Co-authored-by: GPT-5.4 <noreply@openai.com> Co-authored-by: dscho <127790+dscho@users.noreply.github.com>

qub1n · 2026-10-09T16:15:44Z

Overall this pull request should be replaced by #6467

It is build on top of this one.

@dscho this one can be closed then.

Initial plan

22bc067

Copilot AI assigned Copilot and dscho Sep 18, 2026

Copilot started work on behalf of dscho September 18, 2026 11:55 View session

Copilot AI linked an issue Sep 18, 2026 that may be closed by this pull request

[Feature Request] Use ReFS block cloning for Git worktrees #6423

Open

1 task done

Copilot AI and others added 5 commits September 18, 2026 12:02

Copilot AI changed the title ~~[WIP] Implement ReFS block cloning for Git worktrees~~ worktree: use ReFS block cloning for unchanged files Sep 18, 2026

Copilot finished work on behalf of dscho September 18, 2026 12:19

Copilot AI requested a review from dscho September 18, 2026 12:19

qub1n reviewed Sep 18, 2026

View reviewed changes

dscho requested changes Sep 18, 2026

View reviewed changes

Copilot started work on behalf of dscho September 18, 2026 14:03 View session

Copilot finished work on behalf of dscho September 18, 2026 14:03

Copilot AI requested a review from dscho September 18, 2026 14:03

Copilot started work on behalf of dscho September 18, 2026 15:43 View session

Copilot AI and others added 11 commits September 18, 2026 15:53

amend! compat: add a block-cloning primitive

2107368

Co-authored-by: dscho <127790+dscho@users.noreply.github.com>

fixup! worktree: clone unchanged files on capable volumes

d078954

Document the copy-on-write optimization and its opt-out. Signed-off-by: GitHub <noreply@github.com> Co-authored-by: GPT-5.4 <noreply@openai.com> Co-authored-by: dscho <127790+dscho@users.noreply.github.com>

Copilot finished work on behalf of dscho September 18, 2026 16:13

azchohfi mentioned this pull request Oct 8, 2026

parallel-checkout: default to four workers on Windows #6469

Open

Conversation

Copilot AI commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

qub1n Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dscho left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 18, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 18, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 18, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 18, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 18, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 18, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 18, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 18, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 18, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 18, 2026

Choose a reason for hiding this comment

Uh oh!

qub1n commented Oct 9, 2026

Copilot AI commented Sep 18, 2026 •

edited

Loading

qub1n Sep 18, 2026 •

edited

Loading