Skip to content

Week 7 - M1: Finalize packfiles + storage performance baseline #44

Open

Description

@aman-a-shah

Steps

  1. Finish the packfile and transfer-hardening work; check the implementation against docs/packfiles.md and make the document describe the actual format and packing entry point.
  2. Close remaining gaps in packed-only, loose-only, and mixed lookup using fresh store instances.
    • Loose lookup first; corrupt loose objects must not silently fall back to a packed copy
    • Missing -> ObjectNotFoundError; corrupt -> ObjectCorruptError
    • has_object and duplicate writes must preserve those distinctions
  3. Finalize validation of pack/index versions, counts, offsets, record boundaries, types, lengths, compression, and hashes. Published packs must preserve the exact bytes of blobs, trees, and commits.
  4. Verify temporary-file cleanup and pack-before-index publication. Interrupted publication must leave earlier objects readable; incomplete/unpublished packs must not become visible to readers.
  5. With M4, measure packing time, object-read time, and stored bytes on the shared moderately large fixture.
    • Record fixture size, object counts, environment, and commands
    • Compare loose and packed storage separately; report retained loose bytes separately from pack/index bytes
    • Include fresh-store reads so a warm in-memory cache does not hide the initial cost
  6. Profile repeated lookups and fix a measured storage bottleneck if one is found. Any index cache must discover newly published packs and preserve corruption checks.
  7. Add a small regression for each fix; reuse existing pack and transfer tests. Leave pruning, delta compression, and format redesign outside this issue.
  8. Document reproduction commands and remaining performance findings for Week 8 in the PR; scripts/quality-check.sh -> linked PR -> teammate review.

Tests

  • Fresh-instance reads of blobs, nested trees, and two-parent commits from multiple packs
  • Packed-only and mixed stores; duplicate requests and overlapping objects across packs
  • Empty/binary/Unicode payloads retain the original type, bytes, and hash
  • Invalid index/record, truncated data, bad zlib, wrong hash, failed publication
  • Existing store instance can read a newly published pack; old objects remain readable
  • Shared performance fixture plus a small correctness case in the normal test suite; no wall-clock CI threshold

Done when

  • Packfile implementation and format/usage documentation agree
  • Packed and mixed reads preserve validation and publication guarantees
  • Reproducible storage baseline recorded; any optimization has before/after evidence
  • Regression and integration checks pass
  • Quality check green; PR open; review requested
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

module-1Object storageweek-7Week 7 - optimization and polish

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions