storage: map_file - read a file straight from memory-mapped flash

[lynt-smitka]

storage.map_file(f) lets use an asset straight from the CIRCUITPY drive's flash, without
copying it into RAM. FatFs fragmented file is not a problem: the function returns one read-only
memoryview per contiguous cluster run, in file order, and the caller composes them. A bitmap, a
sample bank or a MIDI track that does not fit in RAM stays an ordinary file on the drive. This
started as picogame.xip_map for the picogame's backgrounds loading. The same works for
audiocore.RawSample, synthio.MidiTrack and synthio.Note wavetables, which keep their buffer
for as long as they exist, so I moved the function in storage.

In a test with a FAT image written by Linux, 16 % of new files were contiguous after every other
file had been deleted and 58 % after random deletes, and a fragmented file had 2, at most 3 runs. So
the function returns every run. A record inside one run is a slice of that run's memoryview. A
record that straddles a run boundary is copied.

def pieces(runs, size):
    """Yield the file in pieces aligned to `size`-byte records: the largest slice of a run that
    holds whole records (no copy), or one copied record that a run boundary cut in two.
    The file must be a whole number of records."""
    carry = bytearray()                              # the start of a record cut by a run boundary
    for run in runs:
        if carry:                                    # finish the cut record from this run
            take = min(size - len(carry), len(run))
            carry += run[:take]
            run = run[take:]
            if len(carry) < size:
                continue                             # this run ended too: keep carrying
            yield carry
            carry = bytearray()
        whole = len(run) // size                     # whole records in this run: a slice
        if whole:
            yield run[:whole * size]
        carry += run[whole * size:]                  # a cut record, if any, carries over

# a background as row bands, one bitmap per piece (this is how the strips in the table were loaded)
with open("/assets/level.bin", "rb") as f:
    runs = storage.map_file(f)
bands = [picogame.Bitmap(piece, WIDTH, len(piece) // STRIDE, format=picogame.PAL8, palette=PALETTE)
         for piece in pieces(runs, STRIDE)]

# a bank of sixteen 8 KB sound effects (16-bit mono), played through a mixer
with open("/sfx.bin", "rb") as f:
    runs = storage.map_file(f)
effects = [audiocore.RawSample(memoryview(piece)[start:start + EFFECT_BYTES].cast("h"), sample_rate=22050)
           for piece in pieces(runs, EFFECT_BYTES) for start in range(0, len(piece), EFFECT_BYTES)]
mixer.voice[0].play(effects[0])

Opening a file for reading already builds its FatFs cluster-link map, so the function only reads
that map. On RP2 the last written 4 KB flash sector can still sit in a RAM copy, so the function
flushes it to flash first and the memoryviews see what was last written. A port hook maps a drive
block to a flash address and says how far the mapping stays contiguous, so a run is split where it
is not. raspberrypi returns the execute-in-place address. espressif maps each drive partition with
esp_partition_mmap on first use and reports the seam of an extended drive, the default on 8 MB and
larger flash. Other ports raise NotImplementedError. Errors: EINVAL for a closed file or one not
open for reading only, EOPNOTSUPP on another mount, EIO for a corrupt chain, MemoryError if
open() could not allocate the map.

Measured with the 18 background strips (92 KB) of a picogame game, heap used:

board	map_file	open().read()
PicoPad (RP2040)	1,872 B	97,904 B
Feather ESP32-S3 TFT	1,872 B	114,288 B

Flash: +0.5 kB

The memoryviews are not copies of the file. When the file is rewritten they show the new bytes, and
a mix of old and new while the write runs. On RP2 AudioOut and I2SOut copy a whole
audiocore.RawSample into a staging buffer, so a mapped sample saves no RAM there. Through an
audiomixer.Mixer the voice reads the memoryview in place. displayio.Bitmap always allocates its
own pixel buffer, so a mapped image still has to be copied into one. Letting a Bitmap use the mapped
bytes directly, through the existing construct_from_buffer with read_only, is a follow-up.

[lynt-smitka]

Trying to turn off a module on the two boards that are tight on flash
(bluemicro840, datalore_ip_m4) so CI can get through. They are 16 and 20 bytes
over. I will look into it properly after CI runs.

[tannewt]

One big question about what happens when the file is modified.

[tannewt]

Can we invalidate the view when it is modified and raise an exception instead of returning junk? I think littlefs might not place a file over itself when writing. Even flash will be all 1s temporarily during a write.

[lynt-smitka]

I went with refusing the write instead: RawSample and MidiTrack keep the raw pointer, so an invalidated view wouldn't help them.

The new commit makes open() for writing and os.remove() on a mapped file raise EACCES until the next reload (rename still works). Non-FAT files now raise EOPNOTSUPP rather than TypeError.

Is EACCES the errno you'd want, or EROFS?