Skip to content

ci: make the .NET binding buildable from a tend session - #6345

Open
prql-bot wants to merge 6 commits into
mainfrom
skills/dotnet-sandbox-35429855401
Open

prql-bot wants to merge 6 commits into
mainfrom
skills/dotnet-sandbox-35429855401

Conversation

@prql-bot

@prql-bot prql-bot commented Sep 19, 2026

Copy link
Copy Markdown
Collaborator

Changes to prqlc/bindings/dotnet/ had to be pushed unverified and judged from CI, because dotnet in a tend session fails in two ways that don't name their own cause. This makes the whole test-dotnet job runnable in-session: one sandbox_setup precondition, and one running-tend section for the part that can't be preinstalled.

A NuGet mutex that needs a writable /tmp. NuGet.Common.Migrations.MigrationRunner opens a named Mutex whose backing directory is a hard-coded /tmp path that no environment variable moves, and the sandbox's /tmp is read-only — so dotnet build, restore, test and new all abort with mkdtemp("/tmp/.dotnet.XXXXXX") == nullptr; errno == EROFS, while dotnet --info and --version succeed because neither reaches the runner. The runner skips the mutex entirely once the marker for its migration count exists, so sandbox_setup creates ${XDG_DATA_HOME:-$HOME/.local/share}/NuGet/Migrations/1 and no session has to know any of this. 1 is current as of SDK 10.0.400; a later SDK bumps it and needs the new number in .config/tend.yaml.

MSBuild's worker nodes. socket(AF_UNIX, …) is refused sandbox-wide with EPERM at creation — socketpair still works, so only cross-process Unix-socket IPC is lost. A multi-node build dies in the node handshake and prints Build FAILED. with 0 Error(s) and no diagnostic; -m:1 keeps it in one process and makes real errors visible again. A single-project build succeeds either way, which is why a quick probe misses it. That can't be preinstalled, so the skill section carries it along with the end-to-end recipe.

Verified from a fresh sandbox — ~/.dotnet holding tools/ only, no ~/.nuget, /tmp read-only throughout: with the marker alone, dotnet new console restores, dotnet build … -m:1 succeeds, and dotnet test … -m:1 reports Passed! - Failed: 0, Passed: 10. Deleting only the marker brings the EROFS straight back; re-creating only the marker takes it away again.

How this arrived at the marker, after two reversals

Earlier commits on this branch added (78d862f), removed (358ca86) and restored (ec3ef9f) a with-writable-tmp.sh wrapper that ran dotnet under unshare --map-root-user --mount with a tmpfs over /tmp. The wrapper worked, but it made the build run as uid 0 and relocated every absolute path in its output to $TMPDIR/with-writable-tmp-checkout/…, and it left one result unexplained: a session had reported a fully green unwrapped build and test after clearing ~/.dotnet and ~/.nuget.

The marker explains it. It lives under ~/.local/share, in neither of those two directories, so clearing both left it in place — and that session had already run wrapped dotnet commands, which write it into the real home. Nothing was misread; the wrapper was simply a heavier way to satisfy a precondition a touch satisfies, and it's gone from this PR.

The failure also moves rather than disappearing, which is worth knowing when diagnosing it: once ~/.dotnet/<version>.dotnetFirstUseSentinel exists, the same mkdtemp surfaces as a generic MSB4018 from NuGet.targets under RestoreTask instead of a bare IOException from the first-run configurer. Matching on the original message misses that. NUGET_MIGRATIONS_DISABLED=1 does not suppress the mutex.

Newtonsoft.Json 9.0.1 reaches PrqlCompiler.Tests transitively through Microsoft.NET.Test.Sdk and restore flags it as NU1903, high severity. That predates this PR; the new section notes it so a future session doesn't chase it, and fixing it is a separate concern.

The .github/workflows/tend-*.yaml edits are the generated mirror of the new sandbox_setup lines — the same edit the nightly tend/update-workflows regen would make, applied now so the config isn't inert until then.

#6335 adds a section to the same skill file ~50 lines away, so the two merge cleanly in either order.

No changelog entry — this changes agent instructions and CI config, not anything user-facing.

The SDK's first-run Mutex needs a writable /tmp, which the sandbox doesn't
have, and MSBuild's worker nodes fail silently because AF_UNIX binds are
refused. Add a wrapper that runs a command with a private tmpfs at /tmp, and
record both constraints plus the full test-dotnet recipe.

@prql-bot prql-bot left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The /tmp premise doesn't reproduce. From this session, with /tmp still read-only, the whole test-dotnet job runs green without the wrapper — -m:1 is the only thing it needs.

Before concluding that, I cleared everything that could have been masking a first-run failure: ~/.dotnet, ~/.nuget, and both projects' bin/ and obj/. The configurer genuinely re-ran on the next command — it reprinted the Welcome to .NET 10.0! banner and recreated 10.0.400.dotnetFirstUseSentinel — so this is the cold path the new section describes, not a warm one:

$ dotnet build prqlc/bindings/dotnet -m:1          # no wrapper
Build succeeded.
$ cp target/debug/libprqlc_c.* prqlc/bindings/dotnet/PrqlCompiler{,.Tests}/bin/Debug/net10.0/
$ dotnet test prqlc/bindings/dotnet -m:1           # no wrapper
Passed!  - Failed: 0, Passed: 10, Skipped: 0, Total: 10

dotnet new console succeeds outside the wrapper too. None of the four commands the section says abort produced mkdtemp("/tmp/.dotnet.XXXXXX") == nullptr; errno == EROFS, and touch /tmp/... still fails with Read-only file system, so this isn't a sandbox that quietly gained a writable /tmp.

The -m:1 half reproduces exactly as described, and deterministically:

$ dotnet build prqlc/bindings/dotnet               # no -m:1, no wrapper
Build FAILED.
    0 Warning(s)
    0 Error(s)                                     # exit 1

I can't reconcile the EROFS this description reports against a sandbox that was the same one ten minutes earlier, so I'm not claiming the authoring session misread it. What I can say is that nothing in this job needs a writable /tmp now, and that the silent node failure alone accounts for every symptom I can still reproduce.

That matters because the wrapper isn't free. Merging it tells every future .NET session to run builds under unshare --map-root-user --mount, which changes the build's effective uid to 0 and relocates the checkout to a bind path — MSBuildStartupDirectory and every absolute path in the build output come out as $TMPDIR/with-writable-tmp-checkout/... rather than the real checkout. That's a real behavior change to carry for a constraint I can't observe.

So I'm dropping with-writable-tmp.sh and the /tmp section, and keeping the -m:1 guidance and the reproduction recipe — the parts that hold up under a clean re-run. The script stays reachable in this PR's history if the EROFS ever reappears.

Full verification sequence

Each step ran from the repo root in this session, with /tmp read-only throughout:

  1. touch /tmp/probeRead-only file system. Confirms the sandbox constraint is still in place.
  2. Wrapper path, as the PR documents it: dotnet build prqlc/bindings/dotnet -m:1 through with-writable-tmp.shBuild succeeded. The wrapper itself works as described, and its namespace grants nothing new (mount -o remount,rw refused; writing outside the checkout still EROFS).
  3. rm -rf ~/.dotnet ~/.nuget prqlc/bindings/dotnet/*/{bin,obj}, then dotnet build prqlc/bindings/dotnet -m:1 with no wrapper → first-run banner printed, Restored both projects, Build succeeded.
  4. cargo build -p prqlc-c, copy libprqlc_c.* into both bin/Debug/net10.0/, then dotnet test prqlc/bindings/dotnet -m:1 with no wrapper → Passed! - Failed: 0, Passed: 10.
  5. Control, no wrapper and no -m:1, twice from a cleaned bin//obj/Build FAILED. / 0 Error(s) / exit 1 both times; adding -m:1 back → Build succeeded.
  6. dotnet new console in a scratch directory, no wrapper → Restore succeeded.

The Newtonsoft.Json 9.0.1 NU1903 advisory the description mentions shows up on every one of these restores, so it's independent of any of this.

The read-only /tmp constraint the wrapper worked around does not reproduce: from a cleared ~/.dotnet, ~/.nuget and bin/obj, the whole test-dotnet job runs green without it (10 passed). MSBuild's silent node failure is the only real blocker, and -m:1 alone clears it.
@prql-bot prql-bot changed the title ci: make the .NET binding buildable from a tend session ci: document how to build and test the .NET binding from a tend session Sep 19, 2026
358ca86 dropped with-writable-tmp.sh on the premise that a read-only /tmp no
longer blocks the SDK. It does. From a fresh sandbox with no dotnetFirstUseSentinel:

  dotnet new console        -> mkdtemp("/tmp/.dotnet.lDmpY9") == nullptr; EROFS
  dotnet build … -m:1       -> same failure, in DotnetFirstTimeUseConfigurer
  + sentinel, dotnet build  -> MSB4018 from NuGet.targets, same mkdtemp, in RestoreTask
  wrapped build + test      -> Build succeeded; Passed! Failed: 0, Passed: 10

The middle row is what makes the constraint look gone: one wrapped run writes
~/.dotnet/<version>.dotnetFirstUseSentinel into the real home, outside the
tmpfs, so a later unwrapped command clears the configurer and fails further in.
The section now names that trap.

Also corrects the socket finding: socket(AF_UNIX, …) is refused at creation with
EPERM, not at bind, and socketpair() still succeeds — so only cross-process
Unix-socket IPC is lost, which is why MSBuild's worker nodes are what breaks.
The review that dropped the wrapper reports clearing ~/.dotnet before its green
unwrapped run, so the persisted sentinel doesn't account for it. Say what is
reproducible — EROFS from a fresh sandbox, and the failure moving into
RestoreTask once the sentinel exists — and leave the contrary result named and
unexplained rather than attributed.
@prql-bot

Copy link
Copy Markdown
Collaborator Author

Reversed the wrapper removal in d0c1001, after re-running the constraint at this branch's head.

From a fresh sandbox — ~/.dotnet holding only 10.0.400.toolpath.sentinel, ~/.nuget absent — the command this section had been left recommending fails:

$ dotnet build prqlc/bindings/dotnet -m:1
... mkdtemp("/tmp/.dotnet.i1m5Ci") == nullptr; errno == EROFS;
   at Microsoft.DotNet.Configurer.DotnetFirstTimeUseConfigurer.Configure()

dotnet new console fails identically. Creating ~/.dotnet/10.0.400.dotnetFirstUseSentinel gets past the configurer and then fails on the same mkdtemp inside RestoreTask, surfacing as MSB4018 from NuGet.targets(198,5). Through the wrapper, the build succeeds and dotnet test … -m:1 reports Passed! - Failed: 0, Passed: 10.

That doesn't reconcile with the green unwrapped run above, which reports clearing ~/.dotnet and ~/.nuget first — so a persisted sentinel doesn't explain it either. The section and the description now name that as unexplained rather than resolving it. What decides it is the asymmetry: keeping the wrapper costs bind paths in absolute build output, while dropping it leaves the documented recipe failing outright for the next session that follows it.

Corrected in the same push: socket(AF_UNIX, …) is refused at creation with EPERM, not at bind — socketpair still succeeds, so only cross-process Unix-socket IPC is lost, which is why MSBuild's worker nodes are the thing that breaks.

@prql-bot prql-bot left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The /tmp constraint is real — EROFS reproduces here from a fresh sandbox, so restoring the wrapper in ec3ef9f was the right call over 358ca86. But the wrapper isn't what the constraint needs, and the result this section records as unreconciled turns out to be explained.

The whole /tmp dependency is one named mutex. NuGet.Common.Migrations.MigrationRunner.Run opens a Mutex called NuGet-Migrations, backed by a hard-coded /tmp path — but only when the marker for its migration count is absent. That marker lives at ${XDG_DATA_HOME:-~/.local/share}/NuGet/Migrations/1, which is in neither ~/.dotnet nor ~/.nuget. Creating it removes the constraint outright: with /tmp still read-only and no wrapper, dotnet new, build and test all run green.

That also reconciles the contrary result. The earlier session cleared ~/.dotnet and ~/.nuget — the two directories this section still names — and the marker, sitting outside both, survived. Its green unwrapped run wasn't a misread; it was a sandbox that had already been through a wrapped dotnet command, which writes the marker into the real home.

So the wrapper buys nothing over a touch, and it isn't free: unshare --map-root-user --mount runs the build as uid 0 and relocates every absolute path in the output to $TMPDIR/with-writable-tmp-checkout/…. I've pushed the replacement — the marker moves to sandbox_setup in .config/tend.yaml, where this repo already handles "the sandbox is missing a precondition every session needs", so no session has to remember it. -m:1, the recipe and the NU1903 note stay; with-writable-tmp.sh goes.

This is not 358ca86 again. That commit dropped the wrapper on an unexplained green run and left the documented recipe failing for the next session. This one names the mechanism, replaces it, and runs the whole test-dotnet job to Passed! - Failed: 0, Passed: 10 without it.

One caveat now in the skill: 1 is NuGet's migration count as of SDK 10.0.400. A later SDK bumps it, the marker stops matching and the EROFS returns — self-announcing, but it needs the new number in .config/tend.yaml.

Controlled runs

All from this session, /tmp read-only throughout (touch /tmp/probeRead-only file system before and after).

sandbox state command result
fresh: ~/.dotnet holds tools/ only, no ~/.nuget, no marker dotnet new console mkdtemp("/tmp/.dotnet.wZHyxR") == nullptr; errno == EROFS, in DotnetFirstTimeUseConfigurer
after one wrapped dotnet build unwrapped dotnet build … -m:1 Build succeeded.
marker deleted; sentinel and ~/.nuget kept unwrapped dotnet build … -m:1 MSB4018 from NuGet.targets(198,5), same mkdtemp, inside RestoreTask
touch …/NuGet/Migrations/1, nothing else changed unwrapped dotnet build … -m:1 Build succeeded.
marker only; ~/.dotnet sentinels deleted, ~/.nuget moved away unwrapped dotnet new console Restore succeeded.
marker only unwrapped cargo build -p prqlc-c → copy libprqlc_c.*dotnet test … -m:1 Passed! - Failed: 0, Passed: 10

Rows 3 and 4 are the pair that isolates it: deleting only the marker brings the failure back, and re-creating only the marker takes it away. Row 5 shows it covers the first-run configurer too, not just RestoreTask.

The section's description of how the failure moves is exactly right and worth keeping — once ~/.dotnet/<version>.dotnetFirstUseSentinel exists the EROFS surfaces as a generic MSB4018 rather than a bare IOException, so a probe matching on the original message misses it.

Two negatives, so they don't get re-derived: NUGET_MIGRATIONS_DISABLED=1 does not suppress the mutex (still EROFS), and the -m:1 finding reconfirms unchanged — unwrapped dotnet build without it gives Build FAILED. with 0 Warning(s), 0 Error(s) and exit 1.

with-writable-tmp.sh itself is sound — shellcheck-clean, correct argument passing and mount ordering, and it does what it claims. The objection is only that the job doesn't need it.

The read-only /tmp only blocks dotnet because
NuGet.Common.Migrations.MigrationRunner opens a named Mutex backed by a
hard-coded /tmp path, and it does that only when the marker for its migration
count is absent. Creating that marker lets the whole test-dotnet job run
unwrapped, so with-writable-tmp.sh and its uid-0 namespace and bind paths go.

The marker lives outside ~/.dotnet and ~/.nuget, which is why clearing those two
left an earlier session with a green unwrapped run it could not explain.

sandbox_setup creates it, so no session has to know.
@prql-bot prql-bot changed the title ci: document how to build and test the .NET binding from a tend session ci: make the .NET binding buildable from a tend session Sep 19, 2026

@prql-bot prql-bot left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-m:1 isn't accepted by every dotnet command, and dotnet new — one of the four the .config/tend.yaml comment names — is where it breaks: dotnet new console -o a -m:1 exits 127 with '-m:1' is not a valid option. A session that follows Pass -m:1 to every dotnet command literally hits an unexplained failure in the one place this section exists to prevent one. build, test and restore all accept it, and dotnet new doesn't need it — its post-create restore is a single project. I'm pushing that narrowing to the branch.

The marker mechanism itself holds up from a sandbox the authoring session never touched, which is the check this branch's two reversals actually call for. This session started with no ~/.local/share/NuGet/Migrations, no ~/.nuget, ~/.dotnet holding tools/ only, /tmp read-only, and no dotnet command yet run. dotnet new console aborted with exactly the documented mkdtemp("/tmp/.dotnet.XXXXXX") == nullptr; errno == EROFS out of MigrationRunner.Run inside DotnetFirstTimeUseConfigurer; a bare mkdir -p plus touch of the marker — nothing else — took it away, and the documented recipe then ran end to end to Passed! - Failed: 0, Passed: 10 with /tmp still read-only.

Independent reproduction

Every row from this review session, /tmp read-only throughout (touch /tmp/probe-reviewRead-only file system).

state command result
fresh sandbox, no marker dotnet new console -o app mkdtemp("/tmp/.dotnet.iqTQpg") == nullptr; errno == EROFS at MigrationRunner.RunDotnetFirstTimeUseConfigurer.Configure
marker created, nothing else dotnet new console -o app Restore succeeded.
marker only dotnet build prqlc/bindings/dotnet Build FAILED. / 0 Warning(s) / 0 Error(s) / exit 1
marker only dotnet build prqlc/bindings/dotnet -m:1 Build succeeded., plus the two NU1903 warnings
marker only cargo build -p prqlc-ccpdotnet test prqlc/bindings/dotnet -m:1 Passed! - Failed: 0, Passed: 10
scratch single-project app dotnet build (no -m:1) Build succeeded. — confirms why a quick probe misses the node failure

The AF_UNIX claim reproduces directly too: socket(AF_UNIX, SOCK_STREAM) raises PermissionError/EPERM while socketpair() and socket(AF_INET, SOCK_STREAM) both succeed, so the section's account of what is and isn't lost is exact.

All eight .github/workflows/tend-*.yaml files carry the two new sandbox_setup lines, matching the eight that declare sandbox_setup at all. The skill hunk lands ~50 lines below #6335's, so the two are independent.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant