Repository navigation
feat!: export shared object products through persistent depots - #105
Merged
Merged
Conversation
`[paths] depot` (`--depot`, `NNNOTES_PATHS_DEPOT`) names one directory for everything a configuration reads and computes: - `downloads/`: the game files as downloaded, under their own names (catalogs, decrypted bundles, raw CDN files, files read from the APK, one directory per catalog source); - `cas/`, `ac/`, `inputs/`, `catalogs/`, `runs/` and `costs/`: the results of the stages by content, their task results, input identities, catalog versions and run manifests. The `depot` command group replaces the `store` group and adds `depot gc`, which removes the runs, results and contents that no kept run reaches (`--keep N` latest runs per selection, `--dry-run`). Task inputs locate their bytes with the locator kinds `depot`, `download` (a path in `downloads/`), `file` and `catalog`. The catalog API speaks of downloads: `Catalog(catalog_bytes, download_dir)`, `Catalog.downloaded`, `Catalog.downloaded_raw`, `Catalog.file_name`, `jp.Source.download_dir`, and `jp.open_catalog` takes the downloads directory. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The evaluation script reads and writes a depot (`--base DEPOT`, `--depot D`) like the commands do. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…oads Every command that reads the depot holds a lease in <depot>/leases/ while it runs. When the last running command ends, the bundles and raw files of downloads/ used least recently are removed until the rest fit [depot] downloads: a size (500MB, 20GiB), all, or 0 (unset). Catalog files stay. The catalog marks a file as used when it hands it out, and a removed file is downloaded again when a command needs its bytes. A command that must run alone holds leases/.exclusive: trimming the downloads, and depot gc, which exits with an error while another command uses the depot. Commands that start meanwhile wait for it. bundle pull copies the decrypted bundles of its keys to -o DIR. BREAKING CHANGE: downloaded bundles and raw files are removed when the last command ends unless [depot] downloads keeps them, and bundle pull requires -o DIR. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The depot records the identity of each input name and of each hashed file as one file, under inputs/<kind>/ and inputs/paths/, written through a temporary file and a rename. Identities recorded at the same time by the threads that fetch and hash inputs are all kept, so a depot that no longer holds a downloaded file plans its tasks from the recorded identity without downloading the file again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
[depot] remote gives the depot a remote shared by the machines that use it: s3://<bucket>/<prefix> in S3-compatible object storage (the remote extra, through boto3) or file:///<directory>, with [depot] endpoint, access_key_id, secret_access_key and signing_region. The remote holds the same keys as the depot directory: cas/, ac/, inputs/<kind>/, runs/. The depot directory is the remote's cache. A content, result, input identity or run it lacks is read from the remote and kept, and a content read from the remote is checked against its id. What the depot adds goes to the remote as well: the bytes of a result before the result, a run manifest before the pointer to it. A result that differs from the remote's is a conflict. The results of the tasks of the latest run read from the remote are known to be there without a request for each. depot gc --remote collects the remote by its runs; its results and contents younger than a day stay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The atoms (texture decoding, PNG and ASTC encoding, sprite images, mesh arrays, glTF, file sniffing and the bundle reader) are the pure functions Unity objects are converted with, and live in nnnotes.unity.atoms. Object ids (nnnotes.unity.ids) and media types (nnnotes.core.media) sit below them; nnnotes.pipeline.contract keeps object_id, parse_object_id, media_type and MEDIA_TYPES among its names. Atom ids, and with them task keys, are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nnnotes.unity.convert holds the converters, one per Unity class: the bundle's object view (names, typetrees, scripts, the object JSON of a typetree) in objects.py and a module per group of classes. Artifacts go out through an emit function the caller gives, so the library knows nothing of tasks or the depot. unity.export runs the converters over one bundle and keeps the items, the fallbacks and the provenance. The typetree exporter takes its material and controller documents and the streamed clip decoding from the library. canon/1 and its hashes are nnnotes.core.canon; nnnotes.pipeline.contract keeps exporting them. clip.json/2 writes a clip's bindings as stored: pathCrc, attributeCrc, typeID with its class, customType, the pptr, int and serializeReference flags, and for a binding that names a script its reference and its script table entry. It resolves the attributes of Transform bindings, which need no hierarchy, and leaves paths and other attributes to the hierarchy that plays the clip. Events keep every field. The clip's name is `name`. controller.json/2 drops the `controller` field, which repeated `name`. BREAKING CHANGE: clip.json and controller.json documents have the forms above; the converter versions are clip.json/2 and controller.json/2. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every object a converter takes has a `json` artifact or is contained in another object's. A Texture2D, Mesh, TextAsset or Font adds its typetree JSON without the bytes its other artifact holds (the pixels, the vertex, index and compressed mesh data, the stored text, the font file). A sprite's document and a prefab's are their `json`. A sprite's json holds its decoded geometry (positions, triangles, stored texture coordinates) and the render data its image was cut from: the sprite's own or its atlas' entry, with texture, rects, uvTransform, downscale and settings. Each item lists in `refs` the ids of the objects its documents reference, other bundles' objects included and the objects of its own prefab left out, so the objects reachable from an object follow from the items alone. The original layout writes a secondary artifact whose role is its extension at `<path>.<ext>` (a texture's json at x.png.json). BREAKING CHANGE: the `meta` and `prefab` roles are `json`; the converters are tex.png/2, sprite.png/2, mesh.glb/2, prefab.json/2, data/2 and font/2. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With `[paths] apk` set, the player's own Unity files under `assets/bin/Data/` of the APK are subjects of unity.census and unity.export: `apk/data.unity3d`, `apk/datapack.unity3d` and `apk/Resources/unity default resources`. The census always covers them; everything and `--select bundle:apk/*` export them. A serialized file outside a bundle is loaded under the name references into it use, so object ids such as `unity_builtin_extra:12` and `unity default resources:10` name objects these tasks export, and link.addresses maps their files to their subjects. These files store no type trees. MonoBehaviours are read with the type trees nnnotes ships for the game's classes (nnnotes.unity.scripttypes), checked against the stored type hash and the object's size; the export task's context carries the trees its objects use. A MonoBehaviour without one gives its header with `unsupported.typetree`, and PlayerSettings gives its fields up to `m_ActiveColorSpace`, the part its type tree reads right. generic.json is version 2. The CRI stages take their `boot` input from the built-in inputs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lding it The census (document nnnotes.census/2, stage version 3) reads every object's typetree and records two new fields per object: `refs`, the object ids its references name, and `unit`, the root GameObject of the hierarchy whose prefab holds it. The objects reachable from a request therefore follow from censuses alone, before anything is converted; they are a superset of the references in the unity.export documents. Shipped MonoBehaviour type trees are now selected by the type hash the file stores for an object. The census of a built-in file of the APK reads its MonoBehaviours with them and names them in its key (atom `typetrees`). BREAKING CHANGE: census documents are nnnotes.census/2, and the unity.export context keys shipped type trees by type hash. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A stage can now name a batch per task (Stage.batch). The tasks of a stage that share a batch form one job: one worker runs them one after another, so what they share (such as a loaded bundle) is loaded once, and calls Stage.end_batch after the last of them to drop it. A task without a batch is a job alone. Scheduling, memory admission and the plan's estimates work on jobs: a job's CPU is the sum of its tasks', its peak the largest of theirs. When a worker dies inside a job, the task it was running fails and the job's remaining tasks run in another worker. The run log names each task's job. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
unity.export (version 2) runs one task per unit: a prefab hierarchy (a root GameObject with the objects its prefab holds) or an object outside any hierarchy, named by the stable address of its root object, `<bundle>:<pathId>`. The unit table (nnnotes.pipeline.units) comes from the censuses: a run exports the units of the selected bundles and every unit their objects reference, and `--only-class` selects units by the class of their root object instead of entering the task keys. - The tasks of one bundle form one batch: a worker loads the bundle once and shares its object index, hierarchies, atlases and last decoded textures across them. A unit's result does not depend on the other units of the batch or their order. - A task key covers the bundle, the unit's root, the converters of its classes and the script entries and shipped type trees its objects use, so a converter or script change runs again only the units holding it. - MonoScript and AssetBundle objects get the typetree JSON like any other class. - The original layout roots objects without a container path at `_bundles/<bundle>` from the unit table; cri.audio, cri.movie, sprite.crop and view.voices find bundle-held content through it. BREAKING CHANGE: unity.export task ids are `unity.export:<bundle>:<pathId>`. The bundle-level artifacts `scripts.json` and `bundle.json`, the link.artifacts stage and the nnnotes.artifacts/1 schema are removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A product stage (nnnotes.pipeline.assemble.ProductStage) makes one task
per subject from root objects: the objects of the catalog keys it gives
for the subject, through the address table, and the object ids it names.
Its task reaches the units holding its roots and every unit an item of a
reached unit references, through the unity.export results; a sprite whose
atlas is in another bundle takes its image from sprite.crop.
- The task's input is the object table of what it reaches
(nnnotes.objects/1): each object's class, name, item status and files,
the object holding a contained one, and the referenced objects no census
holds. Its key therefore follows exactly the files it can read.
- The result holds the product's own files as `<task id>#<path>`, its
object table as `<task id>#objects` and one item for the subject.
- The unit request takes the fact `roots` ({keys, objects}):
`assemble.request` gives it from the products asked for, so a run
exports the units their roots reach and nothing else of their bundles.
- layout.product_entries and layout.product_manifest place products for
a player: every file at `assets/<sha256>.<ext>` and a manifest
(nnnotes.product/1) per product listing its files and its objects'
files.
- view.voices reads the address table through the same cached reader.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The census lists the container of a ResourceManager (`resourcePaths`: the paths Resources.Load finds objects at, in stored order); unity.census is at version 4. The address table (nnnotes.addresses/2, link.addresses version 2) adds `resources`, each path with the object ids it names, and gives those objects stable addresses as container paths of the built-in file holding the ResourceManager. AddressIndex.resource looks a path up ignoring case. A request's roots and a product stage take resource paths besides catalog keys and object ids (`resources`), so a product can start from the objects a runtime loads by name, such as the materials of a model's masks. BREAKING CHANGE: address tables are nnnotes.addresses/2 with a required `resources` table, and census documents carry `resourcePaths`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An Assembly names, besides the objects its files use, the roles it keeps of an object of which it uses some files only (`roles`), such as a shader's programs for one graphics API. The product's object table, and so its manifest and layout, hold only those files. Naming an object the table lacks or a role the object has no file for is an error of the stage. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nnnotes.unity.bindings.resolve gives, for the bindings of an AnimationClip document, the path and the attribute each names in a prefab.json hierarchy played by the Animator on a given GameObject: paths relative to that GameObject by their crc32, attributes among the field paths of the bound component (a MonoBehaviour's by its script), the properties of a renderer's materials for a material binding, and the engine property names. A binding that no name, or several, fit stays unresolved; several fitting paths are listed. The binding name tables, field path listing and the material attribute bit layout live in this module, and the typetree exporter uses them from there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two product stages make a Live2D model from the unity.export results of the objects its prefab reaches, never from a bundle: - live2d.model (version 2) is the model the player reads: model.json (format 3) names the prefab, the moc, the canvas, the atlas pages, the shaders, the mask materials loaded by Resources.Load path, MotionSync and the bindings of every clip named in the model's hierarchy. Its object table drops the MonoScripts and keeps, of a shader, the parsed form, the variant index and the GLSL ES 3.00 programs the drawn materials use. - live2d.cubism is the model as Cubism SDK files: moc3, atlas pages, physics3, exp3, motion3 and model3. `export assets --layout product` places the products of the selection: each product stage makes the products of the keys the selected bundles hold, and without an object layout the run exports only what the products reach. `export live2d MODEL -o OUT` runs the same pipeline for one model and writes its Cubism SDK files, or with `--player` the player's model in the product layout. The Cubism format conversions live in nnnotes.formats.cubism, the choice of the player's shader program in formats.shader.player_program, and the model ids in content.live2d; the site's model export uses them from there. BREAKING CHANGE: `export live2d` writes the Cubism SDK files only (no prefab JSON, no motions/_fades.json) and runs through the depot; live2d.model tasks take the object table of the model instead of its bundles. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A product manifest named every object of its object table, including the Transforms and components contained in a prefab's file. Those entries carried no files, and the references a product reads from other objects' files name objects with files, so the manifest now lists only those. An object contained in a prefab is named by the prefab's JSON, whose nodes and components carry their path ids. For a Live2D model this drops the hierarchy's objects (several thousand entries) from models/<name>.json, so a player fetches and parses only the entries it resolves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A path id is a 64-bit integer. Written as a JSON number, a reader that parses
numbers as doubles (JavaScript's JSON.parse) rounds most of them, so the
object a reference names could not be looked up by its id there. Object JSON
now writes every path id as its decimal string: the pathId of a {"$ref"}, and
the pathId and transformPathId of a prefab's nodes and components. The object
id "<file>:<pathId>" is the reference's file and pathId as written.
The converters whose documents hold references move to a new version
(sprite.png/3, prefab.json/3, material.json/2, clip.json/3,
controller.json/3, mono.json/3, font/3, shader.json/2, generic.json/3), so
their results are made again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two product stages make a spot from the unity.export results of the objects its prefabs reach, never from a bundle: - room.gltf (subject a background key Spot/<place>/Background/<place>, manifest rooms/<place>.json) writes room.glb and room.json: every mesh the background prefab draws, baked into the prefab's space from the Mesh objects' local glTF, with materials translated from each Material's document and its Shader's parsed form, and textures from the Texture2D images and settings. What draws nothing is listed in room.json: nullMaterialSubmeshes, nullMeshFilters and emptyMeshes (a mesh without vertices or triangles). - spot.files (subject a situation key Spot/<place>/<situation>/<situation>, manifest spots/<situation>.json) writes spot.json and the Spine files under spine/. Its input `master` is the spot's record of MasterHomeSpot, MasterText and MasterStoryHomeSpotTapTalkEpisode, made at planning. A product stage that reads master tables names them in MASTER; the planning fact "master" is set when one is in the pipeline, and `export assets --layout product` gives such a stage no subjects without master data. content.prefabs reads a prefab document as a hierarchy (parents, world matrices, activity, in-prefab references) for the Live2D, room and spot assemblers. The story host keeps its bundle reader of a spot in content.hostspot. `export spot SPOT_ID -o OUT` and `export room KEY -o OUT` run the pipeline for the spot's two products, or the one room, and write their files; they take the options of `export live2d`. BREAKING CHANGE: `export room` writes OUT/room.glb and OUT/room.json and draws only the meshes of the background prefab's hierarchy; `export spot` writes no shaders/ and spot.json has no `resources`; room.json lists empty meshes as emptyMeshes; the spine.skeleton stage is removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The address table (nnnotes.addresses/3) gains a `builtins` table: the
objects of the APK's built-in files by class ("GraphicsSettings") and
MonoBehaviours by their script's class ("MonoBehaviour:<class>"),
without MonoScripts and with a hierarchy listed by its root only.
AddressIndex.builtin(selector) reads it.
A product stage can name these selectors as roots
(ProductStage.builtins); the request fact "roots" carries them, so the
units of settings assets that the game reads without a catalog key or a
Resources.Load path are exported with the products that use them.
BREAKING CHANGE: link.addresses is at version 3 and writes
nnnotes.addresses/3; the roots fact has a "builtins" list.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cri.audio and cri.movie tell raw files apart by their first bytes (ACB,
AWB, USM). Those bytes were kept in memory only, so after the depot
trimmed its downloads a later plan could not classify a raw file whose
identity it already knew and stopped with the file pending.
The first bytes are now recorded in the depot per content
("magic:<sha256>"); a raw file without the record is read through its
locators, fetching it through the catalog when no local copy is left.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The web audio formats (WEB_AUDIO, the default format), the bit-exact encode into one of them (web_encode) and the MP4 encoder delay reader (mp4_priming) are in nnnotes.formats.encoders, below the pipeline, so product stages encode audio the same way the site does. The site and its command read them from there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two product stages make what a live plays besides its chart, each shared by every live that plays it: - live.music (subject: a music id) writes music.json, the BGM sound with its cue structure, and each layer of the BGM in a web format (parameter `format`, default AAC with its encoder delay; `flac` keeps the decoded file). Its manifest is music/<id>.json. - live.sounds (subject: `default` or `NoteSePatternId=<ids>`) writes sounds.json, the note and live sounds with the category volumes, the ACF's buses, categories and REACT, the note and live sound tables and the timings, and each sound's decoded FLAC. Its roots include the sound settings of the APK's built-in files; it reads the ACF of the APK. Its manifest is sounds/<subject>.json. Product stages gain what these need: cue sheets (the ACB a catalog key loads, held in a bundle or a raw file, with the files cri.audio made of it as inputs and the cri.audio tasks as dependencies), files of the APK (the fact "apk", inputs "apk:<path>"), product files that are content the depot already holds, and `select(keys, env)` with the planning facts. content.liveaudio is a set of functions over master rows, ACB and ACF bytes and the settings' documents, which the live directory export uses as well. BREAKING CHANGE: ProductStage.select takes the planning environment; liveaudio.note_se_settings, acf_info and category_volumes take master rows, ACF bytes and the settings' documents. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ct table The object table of a product task (nnnotes.objects/1) now lists, under `keys`, the root objects of each catalog key the stage gives for the subject. Assemblers read them with Objects.key(key), so a product with several keys finds each object by the key the game loads it by. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tself A SpriteAtlas references every sprite it packs, and each of those sprites references the atlas, whose textures hold its image. Following both edges made one sprite of a large atlas bring every other sprite of it into a request and into a product's object table. An atlas reached through a reference now adds its other references (its textures) but not the sprites it packs; an atlas a request or a product starts from still reaches all of them. The rule applies to the units a request exports (units.build) and to what a product reaches (assemble.reach). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…roducts Three product stages make what a live spawns and animates per note, each shared by every live that loads it: - live.notes (subject `default`, manifest notes/default.json): the note view, line, pair line and groove loop prefabs, the lane effect settings and the note settings (option defaults and ranges, live settings, the display time, the lane geometry, the default skin and effect set, and the names of every skin and effect set by option value). - live.noteskin (a MasterLiveNoteSkin asset name, noteskins/<name>.json): the LiveNoteSkinAsset, with the flick views whose arrows its units set up. - live.noteeffect (a MasterLiveNoteEffectSkin asset name or its Light variant, noteeffects/<name>.json): the LiveNoteEffectAssetSettings and the effect prefabs it references. Each document names objects by id and carries `bindings`, the clips each Animator of its prefabs plays (its controller's clips, a LoopAnimationTimeSync's loop, a skin's arrow loop on the flick views) named from the Animator's node, and `shaders`, the shader table. The product keeps every object but the MonoScripts, and of a shader its parsed form, variant index and GLSL ES 3.00 programs. content.prefabs gains the helpers these use (controller_clips, clip_bindings, shader_table); copies of one shader in several bundles are one entry of the shader table. The `product` layout selects the three stages from the keys a selection holds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A scene bundle's container names a scene without an object; its scene hashes name the scene's serialized file. census.scenes reads them: a scene is the objects of that file outside a hierarchy (the root GameObjects and the scene's settings). link.addresses (version 4) lists them as the objects of the scene's key and gives them stable addresses under the scene's path, so a product can start from a scene key such as EmbScene/Live. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A material names its shader by id, so a product may hold different shaders of one name: a bundle's copy of a shader beside the copy in the APK's built-in files, each compiled on its own. The name table of content.prefabs.shader_table now gives what a lookup by name (Shader.Find) finds: the copies of one shader by the first id; of different shaders of a name, the one of `found` (the shaders the lookup finds first, which a product passes as GraphicsSettings' always-included shaders); else the name names none of them. Every shader keeps its files either way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
content.prefabs gains key_object, the one object of a catalog key of a class (a prefab by its GameObject, a MonoBehaviour by its script class), and component_ref, the component a field names: in the same hierarchy by path, or in another hierarchy of the table by a $ref to a component the table lists as contained in that hierarchy's root. The note highway uses key_object, and livenotes.animator_bindings takes further clips an Animator plays besides its controller's and its LoopAnimationTimeSyncs'. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nichinichisou0609
force-pushed
the
release/0.3.0-review
branch
from
October 18, 2026 13:12
ac182fd to
b0454bb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A persistent depot provides content-addressed task results, input identities, catalog versions and retained runs with optional S3-compatible remote storage. Product exports assemble Live2D, rooms, spots, live charts, stories and UI from shared Unity object records with versioned conversion tasks. Standalone episode documents, shaders and audio reuse the same conversion tasks and retained layouts.
Named site builds use the product graph, preserve regional topics and publish assets, manifests and indexes in dependency order. Remote run publication shares completed local results and their stored dependencies. Collection retains recent results through a configurable grace period; verification can inspect remote-only content. Command records retain accessed content, and source downloads follow configured budgets and leases.
PNG and glyph task identities include the native compression backend linked by Pillow.
The 0.3.0 package includes command references, depot and format documentation, and shared product schemas. Site builds validate player 0.2.x compatibility and publish chart metadata and requested language references. The deck extra pins ournotes-sim 0.0.12.
Validation: full synthetic tests, Ruff, Python type checks, import contracts, strict documentation and 25 shared schema comparisons. Golden records cover all 65 registered output and site stages, including product assemblers and master-data views. The coverage check verifies stage identity and nonempty fixtures. CI uses pinned font and conversion dependencies for golden records and tests Python 3.11–3.14.
S3 collection reports per-object deletion errors. Protocol tests cover continuation tokens, the 1,000-object batch limit and partial failure responses. English and Chinese setup guides document the package versions, complete player package, depot retention and static hosting.