Make all the codegen we do incremental by weswigham · Pull Request #64331 · microsoft/TypeScript

Wesley Wigham (weswigham) · 2026-09-19T00:50:34Z

This PR adds incremental cache wrappers around all of the codegen we perform in the repo, which, experimentally, can cut up to 80% of the time off of some generate subtasks if the cached results are reusable. Very useful if you have an agent repeatedly running them just in case it modified a file that affects codegen.

Copilot

Copilot review overview

🟡 Changes recommended

Stringer can reuse stale output for package-wide changes, direct Go generation can lose formatter resolution, and duplicate broken test files remain.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 · 1 · 1

Open (3)

Use repository-local dprint instead of relying on PATH · New
Cache ignores other package files, causing stale generated String tables · New
Remove obsolete root-level generator test copies · New

What changed in this PR

Adds hash-based incremental caching and force-regeneration support across repository code generators.

Changes:

Introduces shared generated-file caching and generator utilities.
Wraps Go and TypeScript generators with cache-aware entry points.
Adds extensive code-generation integration tests.

File	Description
`tsc/internal/vfs/vfsmatch/vfsmatch.go`	Uses cached stringer wrapper.
`tsc/internal/vfs/vfs.go`	Uses cached moq wrapper.
`tsc/internal/testutil/projecttestutil/projecttestutil.go`	Migrates mock generation.
`tsc/internal/stringutil/generate.go`	Consolidates Unicode formatting.
`tsc/internal/stringutil/_scripts/generate-unicode-data.mts`	Caches Unicode outputs.
`tsc/internal/project/project.go`	Migrates stringer generation.
`tsc/internal/ls/autoimport/export.go`	Migrates stringer generation.
`tsc/internal/diagnostics/diagnostics.go`	Uses cached diagnostics generator.
`tsc/internal/core/tristate.go`	Migrates stringer generation.
`tsc/internal/core/scriptkind.go`	Migrates stringer generation.
`tsc/internal/core/languagevariant.go`	Migrates stringer generation.
`tsc/internal/core/compileroptions.go`	Migrates stringer generation.
`tsc/internal/checker/types.go`	Migrates stringer generation.
`tsc/internal/bundled/bundled.go`	Uses cached bundle generator.
`tsc/internal/ast/kind_generated.go`	Updates generated directive.
`tools/scripts/tsc/tsconfig.json`	Extends shared generator config.
`tools/scripts/tsc/schema.ts`	Reuses shared repository root.
`tools/scripts/tsc/generate.ts`	Propagates forced regeneration.
`tools/scripts/tsc/generate-ts-ast.ts`	Caches TypeScript AST outputs.
`tools/scripts/tsc/generate-go-ast.ts`	Caches Go AST outputs.
`tools/scripts/tsc/generate-encoder.ts`	Caches protocol encoder outputs.
`tools/scripts/generatedFile.test.mts`	Adds duplicate root-level tests.
`tools/scripts/generatedFile.mts`	Adds duplicate cache implementation.
`tools/scripts/gen/utils.mts`	Adds shared generator utilities.
`tools/scripts/gen/tsconfig.json`	Configures generator type checking.
`tools/scripts/gen/generateStringer.mts`	Adds cached stringer wrapper.
`tools/scripts/gen/generateMoq.mts`	Adds cached moq wrapper.
`tools/scripts/gen/generateDiagnostics.mts`	Adds cached diagnostics generation.
`tools/scripts/gen/generatedFile.test.mts`	Tests incremental generation.
`tools/scripts/gen/generatedFile.mts`	Implements generated-output caching.
`tools/scripts/gen/generateBundled.mts`	Adds cached library bundling.
`tools/scripts/gen/generateAPI.mts`	Adds cached API generation.
`packages/typescript/scripts/generateSync.ts`	Caches sync API outputs.
`packages/typescript/scripts/generate.ts`	Coordinates package generators.
`packages/typescript/package.json`	Uses unified package generator.
`Herebyfile.mjs`	Adds force handling and cached tasks.

Files not reviewed (1)

tsc/internal/ast/kind_generated.go: Generated file

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

Copilot

Copilot review overview

🟢 Approval recommended

The cache implementation validates inputs and outputs, handles failed or concurrent input changes safely, supports forced regeneration, and includes broad integration coverage.

Review effort: Balanced
Findings: None

Resolved since last review (3)

Use repository-local dprint instead of relying on PATH
Cache ignores other package files, causing stale generated String tables
Remove obsolete root-level generator test copies

Files not reviewed (1)

tsc/internal/ast/kind_generated.go: Generated file

…asks - go:generate directives only for compat

Jake Bailey (jakebailey)

We're doing a bunch of discussion off thread on Teams; I'm sort of leaning back to "maybe we should just move code generation into scripts and leave go generate for convenience" to avoid this getting too complicated and avoid the go generate overhead... Then just have hereby generate as "the thing to run" and it's incremental and that's that.

I do think that this is maybe a bit too complicated. But maybe it's whatever and we could just later future changes on it?

Jake Bailey (jakebailey) · 2026-09-21T18:28:09Z

 npx hereby generate      # Regenerate compiler sources and bundled assets
+npx hereby generate:all  # Also regenerate AST, LSP, APIs, localization, and vendored files


I think these might be out of date, if hereby generate now does everything for you, right?

Why would we want :all at all?

hereby generate does everything go generate would find, hereby generate:all does all generate-prefixed tasks, which is a handful more.

I don't think back compat really matters, is there any reason we wouldn't always run generate:all and therefore it should just be generate, and the Go code is relegated to generate:go?

I can rename 'em if you'd like. Just left 'em like that because npx hereby generate just called out to go generate before.

I think renaming is a good idea. When I introduced hereby generate, the only code we had to generate was go generate generated.

I pushed the rename for ya.

Wesley Wigham (weswigham) · 2026-09-21T18:28:56Z

We're doing a bunch of discussion off thread on Teams; I'm sort of leaning back to "maybe we should just move code generation into scripts and leave go generate for convenience" to avoid this getting too complicated and avoid the go generate overhead... Then just have hereby generate as "the thing to run" and it's incremental and that's that.

Yep - I just pushed those changes (and a merge commit fix) - I think I actually raced Andrew Branch (@andrewbranch) 's review there, so I'll re-request. I'm still collecting metrics, but it's promising thus far - cold generate calls are 5-10% faster, and warm ones are like 30%+ faster (which cascades into making validate faster, too). go generate must have pretty big overhead that we really didn't need.

Jake Bailey (jakebailey) · 2026-09-21T20:04:34Z

I think this is looking good, my only suggestion would be that the logging doesn't show much what's going on, like:

Generated codegen outputs.
Generated codegen outputs.
Generated codegen outputs.
Generated codegen outputs.
Generated codegen outputs.
Formatted 1 file.
Generated codegen outputs.
Generated codegen outputs.
Formatted 1 file.
Generated codegen outputs.
Generated codegen outputs.
Generated codegen outputs.
Generated codegen outputs.
Formatted 1 file.
Generated codegen outputs.
Generated codegen outputs.
Formatted 1 file.
Generated codegen outputs.
Generated codegen outputs.
Formatted 2 files.
Generated Unicode tables.
Formatted 1 file.

Compared to go generate listing packages, or otherwise things showing the commands being run (which herebyfile tries to do as much as possible).

Wesley Wigham (weswigham) · 2026-09-21T20:18:12Z

That's pretty fair - the logging got simplified as I merged helpers from the first version and it got a wee bit too simple.

Wesley Wigham (weswigham) · 2026-09-21T21:04:02Z

Jake Bailey (@jakebailey) There ya go, output of the cache layer's a bit more verbose.

Make all the codegen we do incremental

64cc937

Wesley Wigham (weswigham) requested review from Andrew Branch (andrewbranch) and Jake Bailey (jakebailey) and a balanced review from Copilot September 19, 2026 00:50

github-project-automation Bot added this to PR Backlog Sep 19, 2026

typescript-automation Bot assigned Wesley Wigham (weswigham) Sep 19, 2026

typescript-automation Bot added Author: Team For Uncommitted Bug PR for untriaged, rejected, closed or missing bug labels Sep 19, 2026

github-project-automation Bot moved this to Not started in PR Backlog Sep 19, 2026

Copilot started reviewing on behalf of Wesley Wigham (weswigham) September 19, 2026 00:51 View session

Copilot AI reviewed Sep 19, 2026

View reviewed changes

Comment thread tools/scripts/gen/utils.mts

Comment thread tools/scripts/gen/generateStringer.mts Outdated

Comment thread tools/scripts/generatedFile.test.mts Outdated

Wesley Wigham (weswigham) added 2 commits September 18, 2026 18:25

Man, I thought I removed these ages ago

a9819a9

Keep the commands we cache in the go:generate line

2607bb1

Wesley Wigham (weswigham) requested a balanced review from Copilot September 19, 2026 02:51

Copilot started reviewing on behalf of Wesley Wigham (weswigham) September 19, 2026 02:51 View session

Copilot AI reviewed Sep 19, 2026

View reviewed changes

Make herebyfile responsible for defining and offloading all codegen t…

4603f1b

…asks - go:generate directives only for compat

Andrew Branch (andrewbranch) approved these changes Sep 21, 2026

View reviewed changes

github-project-automation Bot moved this from Not started to Needs merge in PR Backlog Sep 21, 2026

Jake Bailey (jakebailey) reviewed Sep 21, 2026

View reviewed changes

Comment thread Herebyfile.mjs Outdated

Merge branch 'main' into make-codegen-incremental

072ae95

Wesley Wigham (weswigham) force-pushed the make-codegen-incremental branch from ed9b879 to 072ae95 Compare September 21, 2026 18:29

Wesley Wigham (weswigham) requested a review from Andrew Branch (andrewbranch) September 21, 2026 18:33

Rename tasks

193f7e3

Jake Bailey (jakebailey) reviewed Sep 21, 2026

View reviewed changes

Comment thread .github/workflows/ci.yml Outdated

PR feedback

699651e

Jake Bailey (jakebailey) reviewed Sep 21, 2026

View reviewed changes

Comment thread tools/scripts/gen/cache.mts Outdated

Reuse run for logging

1ecd9e0

Jake Bailey (jakebailey) reviewed Sep 21, 2026

View reviewed changes

Comment thread Herebyfile.mjs Outdated

Extract run from Herebyfile

b9bca35

Jake Bailey (jakebailey) approved these changes Sep 21, 2026

View reviewed changes

Comment thread tools/scripts/gen/cache.mts Outdated

Format skipped commands the same way run formats executed ones

e006814

Jake Bailey (jakebailey) approved these changes Sep 21, 2026

View reviewed changes

Wesley Wigham (weswigham) enabled auto-merge September 21, 2026 22:02

Wesley Wigham (weswigham) added this pull request to the merge queue Sep 21, 2026

Merged via the queue into microsoft:main with commit 1ac2d5a Sep 21, 2026
28 checks passed

Wesley Wigham (weswigham) deleted the make-codegen-incremental branch September 21, 2026 22:36

github-project-automation Bot moved this from Needs merge to Done in PR Backlog Sep 21, 2026

		npx hereby generate # Regenerate compiler sources and bundled assets
		npx hereby generate:all # Also regenerate AST, LSP, APIs, localization, and vendored files

Conversation

Wesley Wigham (weswigham) commented Sep 19, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟢 Approval recommended

Uh oh!

Jake Bailey (jakebailey) left a comment

Choose a reason for hiding this comment

Uh oh!

Jake Bailey (jakebailey) Sep 21, 2026

Choose a reason for hiding this comment

Uh oh!

Wesley Wigham (weswigham) Sep 21, 2026

Choose a reason for hiding this comment

Uh oh!

Jake Bailey (jakebailey) Sep 21, 2026

Choose a reason for hiding this comment

Uh oh!

Wesley Wigham (weswigham) Sep 21, 2026

Choose a reason for hiding this comment

Uh oh!

Jake Bailey (jakebailey) Sep 21, 2026

Choose a reason for hiding this comment

Uh oh!

Wesley Wigham (weswigham) Sep 21, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Wesley Wigham (weswigham) commented Sep 21, 2026

Uh oh!

Uh oh!

Jake Bailey (jakebailey) commented Sep 21, 2026

Uh oh!

Wesley Wigham (weswigham) commented Sep 21, 2026

Uh oh!

Wesley Wigham (weswigham) commented Sep 21, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants