Skip to content

feat: Add Database#serialize and Database#deserialize methods - #685

Merged
flavorjones merged 8 commits into
sparklemotion:mainfrom
Maaarcocr:new-main
Oct 13, 2026
Merged

flavorjones merged 8 commits into
sparklemotion:mainfrom
Maaarcocr:new-main

Conversation

@Maaarcocr

Copy link
Copy Markdown
Contributor

Add support for sqlite3_serialize() and sqlite3_deserialize() which
allow serializing an in-memory database to a binary string and
restoring it later.

These methods are useful for:

  • Creating snapshots of in-memory databases
  • Transferring database contents between connections
  • Backing up and restoring database state

The methods are only available when SQLite is compiled without
SQLITE_OMIT_DESERIALIZE (requires SQLite >= 3.23.0).

@flavorjones

Copy link
Copy Markdown
Member

@Maaarcocr The test suite failure seems legit. Let me know if you need assistance getting unblocked!

@Maaarcocr

Copy link
Copy Markdown
Contributor Author

@Maaarcocr The test suite failure seems legit. Let me know if you need assistance getting unblocked!

sorry for the delay, getting to this only now. the issue I have is that I can't currently replicate. I run the test with no issues on my machine. could it be an OS issue? or a sqlite version issue?

@flavorjones

Copy link
Copy Markdown
Member

@Maaarcocr It was a SQLite version issue. The basic CI job builds against Ubuntu 24.04's system SQLite 3.45.1, where select 1 succeeds after deserializing garbage, so test_deserialize_invalid_data failed. The vendored SQLite 3.54.0 raises instead, which is likely why it passed for you.

Older SQLite has a worse problem, too: before 3.54.0, sqlite3_deserialize() returns SQLITE_OK while a statement is unfinished, and the next step of that statement reads freed memory. SQLite 3.54.0 returns SQLITE_BUSY instead. So Database#deserialize is now defined only when the loaded SQLite is 3.54.0 or later, and that test now skips on the basic job.

The other changes I pushed on top of yours:

  • I padded the deserialize buffer with 20 zeroed bytes, as the sqlite3_deserialize() docs advise. That also fixed deserialize(""), which raised NoMemoryError because sqlite3_malloc64(0) returns NULL.
  • serialize now raises SQLite3::Exception for an unknown database name, instead of NoMemoryError: failed to allocate -1 bytes.
  • The deserialize docs now say when it raises BusyException, that it detaches a file-backed database from its file, that a WAL-mode image can't be queried afterward, and that deserializing untrusted data is as risky as opening an untrusted database file.
  • I added a CHANGELOG entry and fixed two -Wformat warnings.

@flavorjones
flavorjones enabled auto-merge (squash) October 13, 2026 21:42
Maaarcocr and others added 8 commits October 13, 2026 18:24
Add support for sqlite3_serialize() and sqlite3_deserialize() which
allow serializing an in-memory database to a binary string and
restoring it later.

These methods are useful for:
- Creating snapshots of in-memory databases
- Transferring database contents between connections
- Backing up and restoring database state

The methods are only available when SQLite is compiled without
SQLITE_OMIT_DESERIALIZE (requires SQLite >= 3.23.0).
SQLite's docs advise padding the buffer in case a malformed image makes
SQLite read past its end. A comment explains why the padding is only
defensive here, and why it also makes `deserialize("")` work.
The `Database#deserialize` docs said a transaction made it raise
`BusyException`. That holds only for SQLite 3.54.0 or later, and only
for an open read or write transaction or a backup; older versions free
memory that an unfinished statement still uses. Correct that, and
document the WAL-mode trap, the detach from a file-backed database, and
the risk of untrusted data.
`Database#serialize` raised `NoMemoryError: failed to allocate -1 bytes`
for an unknown database name, or for "temp" before any temporary table
existed. Raise `SQLite3::Exception` naming the database instead.

Also test that `deserialize("")` produces an empty database. It raised
`NoMemoryError` until "Pad the deserialize buffer with 20 zeroed bytes".
Before SQLite 3.54.0, `sqlite3_deserialize()` returned `SQLITE_OK` while
a statement was unfinished, and the next step of that statement read
freed memory. Check the loaded version before defining the method.
`PRId64` expects `long` on LP64 Linux, but `sqlite3_int64` is
`long long`. Format with `%lld` and a cast instead.
@flavorjones
flavorjones disabled auto-merge October 13, 2026 22:27
@flavorjones
flavorjones merged commit 7ad6fc0 into sparklemotion:main Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants