Proposal: encrypt temporary files with a random key per file

[mtrossbach]

Hi Ulrich,

after the temp_store change in #271 I got curious about what actually ends up on disk when temporary data does go to a file, and one thing led to another. This is a proposal for option 1 from #148: every temporary file gets its own random key, behind a compile-time switch.

Type: Feature / Enhancement (security)

What ends up on disk today

With PRAGMA temp_store=FILE, or a build that sets SQLITE_TEMP_STORE to 0 or 1, temporary data is written in plaintext. To see how much, I put a small VFS below the SQLite3 Multiple Ciphers VFS and searched every write for a marker string from the table data. It showed up in temporary tables, in sorting, in DISTINCT, in statement journals and during VACUUM. A VACUUM, and a rekey that changes the cipher scheme, even write a full copy of the database to a temporary file. As you wrote in #148, keeping everything in memory isn't always possible when memory is tight.

The idea

SQLite opens temporary files without a name, and the xOpen documentation guarantees that such a file is temporary and deleted when it is closed. So every file that arrives without a name gets its own random 256-bit key and is encrypted with ChaCha20. The key comes from chacha20_rng, exists only in memory while the file is open and is wiped when it closes, so nobody has to know or store it. That covers the temporary and transient databases, their journals, the sorter files and the statement journals.

Main databases, journals and WAL files stay untouched, and no SQLite patches are needed. Everything sits in src/sqlite3mc_vfs.c: one new section of about 240 lines plus a few lines in the I/O methods. SQLITE3MC_ENCRYPT_TEMP_FILES turns it on and off at compile time.

How it works

ChaCha20 as a stream cipher keeps the data the same length, so SQLite can still read and write any byte range. The file is divided into blocks: temporary and transient databases use their page size, everything else 4 KiB. Each block is stored with an 8-byte counter value in front of it, so a block sits 8 bytes further into the file for each block before it, and the VFS moves the offsets accordingly.

That counter is the nonce. It counts up for every block the file writes and comes back from the file when the block is read, so no state per block has to be kept in memory. A keystream is therefore never used twice, not after a truncate and not after a failed write. Appends continue the keystream of the last block, so journals, whose records never line up with the blocks, cause the same I/O as without encryption; read-modify-write is left for real overwrites in the middle of a block. A block that was never written holds the counter value 0 and reads as zeros, which keeps the gaps sparse that a temporary database gets when pages are written in the order they leave the cache.

What stays in memory per open file is the key, the size, the counter, the last block's counter value with the part of its keystream that is used, and a 4 KiB work buffer. Nothing of it grows with the file.

If a write fails, the bytes it should have written read as zeros afterwards, as if it had never happened, and whatever it left behind the end of the file is removed right away. If the write reached into the part of the block that was already valid, that block reports an I/O error until it is written completely again, so SQLite never gets half a block that was encrypted with the previous counter value. Memory mapping is disabled for these files; SQLite then simply uses xRead.

What it doesn't do, and what it costs

It protects confidentiality, not integrity, which matches the state today. Temporary files of unencrypted databases are encrypted as well, because the VFS can't tell which connection a temporary file belongs to; you estimated 4-6 SQLite patches for that in #148.

The cost only appears when temporary data really goes to disk, and then it is one pass of ChaCha20 over every byte that is written or read. On an Apple M4 with temp_store=FILE: a temporary table took about 1.6 times as long as with plaintext temporary files, a large sort about 1.7 times, CREATE INDEX and VACUUM 6 to 8 percent longer. Keeping temporary data in memory is still faster than both.

Open questions

• Do you like the approach at all?
• Should the switch be on or off by default, and would you like a runtime switch as well, as in option 1?
• I picked ChaCha20 because it runs in constant time everywhere and is already part of the project. AES-256-CTR would work the same way and could use the hardware AES code, but where there are no AES instructions, in WebAssembly for example, it falls back to table-based AES, which isn't constant time and is exposed to cache-timing attacks. I can add it as a compile-time option.
• The stress test is in the PR as test/tempfiletest.c. I'd be glad to add the plaintext check as well, because it would catch a regression after a SQLite update.

How I tested it

Most of the work went into the testing, so here is the short version. The stress test below is part of the PR as test/tempfiletest.c.

The plaintext check from above runs for temporary tables, sorting, DISTINCT, statement journals, VACUUM and a rekey, on macOS and on Linux with gcc and clang, and the marker no longer appears anywhere. A differential test runs the same 12 steps once with temp_store=FILE and once with temp_store=MEMORY and compares the results, also with four sorter threads.

The main test is a stress test on a single temporary file: random unaligned writes, appends, reads and truncates against a model of the file, with a VFS underneath that checks every write on its own. The bytes on disk have to match the keystream for the expected nonce, and no keystream byte may be used twice. The same test runs with disk errors injected at random points, and wrong data without an error never came out of it.

At SQL level I let one write, half a write, all following writes, a read, a truncate or one memory allocation fail at every position of a workload, and checked afterwards that integrity_check is ok and the database is in the state before or after the failed statement. To see whether the tests really catch something, I broke the new code on purpose in nine different ways; every one of them made a test fail.

The sanitizers for memory, leaks, undefined behaviour and threads report nothing, the SQL tests with PRAGMA temp_store=FILE on every connection and cryptotest pass, and there are no new compiler warnings.

Best regards,
Markus

[utelle]

Hi Markus,

after the temp_store change in #271 I got curious about what actually ends up on disk when temporary data does go to a file,

And this is really an important question. Even if one follows the recommendation to set temp_store=2, so that memory is used for temporary data, there can still be situations where temporary data goes to a file (VACUUM being one example).

This is a proposal for option 1 from #148: every temporary file gets its own random key, behind a compile-time switch.

In many situations it is difficult or even impossible to detect to which connection or database temporary data belong and to determine the cipher scheme in use. Therefore using a random key is the best option one has.

The situation is made even more difficult by the fact that SQLite usually starts with temporary data in memory and spills over to disk only if the cache overflows. As a consequence temporary database pages don't have reserved bytes (which are important for most of SQLite3MC's cipher schemes).

What ends up on disk today

With PRAGMA temp_store=FILE, or a build that sets SQLITE_TEMP_STORE to 0 or 1, temporary data is written in plaintext. To see how much, I put a small VFS below the SQLite3 Multiple Ciphers VFS and searched every write for a marker string from the table data. It showed up in temporary tables, in sorting, in DISTINCT, in statement journals and during VACUUM. A VACUUM, and a rekey that changes the cipher scheme, even write a full copy of the database to a temporary file.

Especially the latter is a strong argument to implement encryption for temporary files.

The idea

SQLite opens temporary files without a name, and the xOpen documentation guarantees that such a file is temporary and deleted when it is closed. So every file that arrives without a name gets its own random 256-bit key and is encrypted with ChaCha20. The key comes from chacha20_rng, exists only in memory while the file is open and is wiped when it closes, so nobody has to know or store it. That covers the temporary and transient databases, their journals, the sorter files and the statement journals.

Right. Actually, there are only 2 types of temporary files: files that hold database pages, and files that hold rollback journals.

Main databases, journals and WAL files stay untouched, and no SQLite patches are needed.

Correct.

Everything sits in src/sqlite3mc_vfs.c: one new section of about 240 lines plus a few lines in the I/O methods. SQLITE3MC_ENCRYPT_TEMP_FILES turns it on and off at compile time.

IMHO we should try to avoid encrypting all temporary files unconditionally due to the overhead in processing time. If a database connection doesn't use encryption it shouldn't have to encrypt temporary files. Yes, that means to add patches to SQLite, because the VFS alone has no means to detect whether to use encryption for temporary files or not.

How it works

ChaCha20 as a stream cipher keeps the file length unchanged, so SQLite can still read and write any byte range. The file is divided into blocks of 4 KiB; databases with smaller pages use their page size as block size, so writing one page never rewrites another.

While it is possible to process the file without regard to its internal structure, it might be more effective to take the structure into account.

Your two concerns in #148 both come down to the nonce. Each block has a generation number, kept in memory together with how far the block is written, 12 bytes per block. The nonce is the block number plus that generation. A write that appends to a block continues its keystream and touches nothing else, every other write re-encrypts the block with the next generation. A keystream is therefore never used twice, not after a truncate and not after a failed write. Journal records never line up with the block boundaries, but they only append, so they cause the same I/O as without encryption. Read-modify-write is left for real overwrites in the middle of a block.

Most likely the logic will become simpler, if the structure of the temporary file is taken into account.

What it doesn't do, and what it costs

It protects confidentiality, not integrity, which matches the state today. Temporary files of unencrypted databases are encrypted as well, because the VFS can't tell which connection a temporary file belongs to; you estimated 4-6 SQLite patches for that in #148.

I made a few experiments and I think we can get away with 2 small patches to solve this question. Plus 1 small patch to determine the page size of temporary database files.

The cost only appears when temporary data really goes to disk, and then it is one pass of ChaCha20 over every byte that is written or read. On an Apple M4 with temp_store=FILE: a temporary table took about 1.6 times as long as with plaintext temporary files, a large sort about 1.7 times,

That sounds like a huge overhead, but I guess the encryption itself is only a few percent while most of the time is required to read from and write to the temporary file.

CREATE INDEX and VACUUM 6 to 8 percent longer.

This is more in the range I would expect for added encryption.

Open questions

• Do you like the approach at all?

In principle, yes. However, as mentioned above I think we should use the knowledge about the content structure. Most likely that would make the process more efficient.

• Should the switch be on or off by default, and would you like a runtime switch as well, as in option 1?

I guess the optimum would be to activate encryption only where it is really needed. I will apply the necessary patches to SQLite to accomplish that.

• I picked ChaCha20 because it runs in constant time everywhere and is already part of the project. AES-256-CTR would work the same way and could use the hardware AES code, but where there are no AES instructions, in WebAssembly for example, it falls back to table-based AES, which isn't constant time and is exposed to cache-timing attacks. I can add it as a compile-time option.

I'm pretty sure that AES is the slowest cipher, even with hardware support (which is actually not available for WebAssembly). ChaCha20 is a well-known stream cipher and we already have its implementation in SQLite3MC. So, it is the natural choice.

If we find better options in the future, we can change the cipher scheme any time later on, because we are talking about temporary files after all.

• My tests are standalone C programs that include sqlite3mc.c. I'd be glad to add some of them to the repository, the plaintext check in particular, because it would catch a regression after a SQLite update.

Yes, that is an excellent idea.

[utelle]

My concern with the current approach is the amount of memory required for holding administrative data per block. With the next release we will define SQLITE_TEMP_STORE=2 (if it isn't explicitly set to a different value). That is, temporary data will be held in memory. However, there are scenarios where memory is a scarce resource, so that users are forced to request temporary data to reside on disk. In such situations, it is counterproductive to use potentially large amounts of main memory for administrative data.

Therefore I would propose the following modification: use a 64-bit write operation counter for each temporary file, increase the counter for each block write, prepend the counter value to the block buffer and write the combination to disk - this increases the file size and it requires to adjust the offset within the file according to the number of the block, but it saves us from having to keep the counter for each block in memory.

Only the last block should be incomplete, if at all. With the counter prepended to the data buffer, the last buffer can be written incompletely, so that the file size tells us how many bytes are valid in the last block. Thus, it should be sufficient to keep the actual file size in memory, instead of number of used bytes per block.

For temporary or transient databases (flags SQLITE_OPEN_TEMP_DB or SQLITE_OPEN_TRANSIENT_DB) the actual page size should be used to determine the block size (counter size + pagesize). This way incomplete blocks never occur. In my local environment I already applied the necessary patches to SQLite, so that the page size is available.

In case of flag SQLITE_OPEN_TEMP_JOURNAL use a block size of (counter size + 4 kB) - as you already do.

[mtrossbach]

Done, I rebuilt it that way.

Each block now has its counter value in front of it, 8 bytes. The VFS shifts the offsets in the file accordingly. The counter is the nonce, and it is read back from the file, so I don't have to keep anything per block in memory. Temporary and transient databases use their page size as block size, the rest 4 KiB. For now the page size comes from the first write, so the PR builds without your patch. When yours lands, it can come from there.

A bit more than the file size stayed in memory, though. For the last block I keep its counter value and how much of its keystream is used, 16 bytes. Otherwise every append rewrites the partly filled block, and a journal writes about twice as much as it does today. In an early version that was 54.7 MB for 21.8 MB of data. There is also a short list of blocks whose write failed. Reading one of them returns an I/O error, because otherwise SQLite would get back half a block that is still encrypted with the old counter value.

A block nobody ever wrote has the counter value 0 and reads as zeros. Temporary databases write their pages in the order they fall out of the cache, so the files have holes, and they stay sparse.

If a write fails, whatever it left behind the end of the file is removed right away now. Otherwise it would look like a normal block as soon as the file grows past it again. The stress test found that.

Something else from testing, in case it helps with your patches. The sorter files for ORDER BY and CREATE INDEX are opened with SQLITE_OPEN_TEMP_JOURNAL, but they don't hold pages or journal records, only sorted records written in chunks. In the plaintext check they are the worst of the lot.

And the overflow pages you asked about. When a row doesn't fit on one page, the rest of it lands in overflow pages. When SQLite reads such a value, it sometimes takes the piece it needs straight from the file, past the pager. Your codec check keeps that away from encrypted databases, but a temporary database has no codec, so there a read can start in the middle of a page. With blocks that is no problem. If it ever becomes one, the VFS can stop reporting SQLITE_IOCAP_SUBPAGE_READ for these files and SQLite reads whole pages again.

The cost is the same as before. On an M4 with temp_store=FILE, a sort and a temporary table take about 1.6 times as long as with plaintext temporary files, CREATE INDEX and VACUUM 3 to 8 percent longer. I reran everything on macOS and Linux, including the runs with disk errors in the middle of a block. The stress test checks the counter values on disk now as well: they must not go backwards, and no keystream byte may be used twice.

If you would like it different anywhere, just say so.

[utelle]

Done, I rebuilt it that way.

Great!!! Thank you very much!

Each block now has its counter value in front of it, 8 bytes. The VFS shifts the offsets in the file accordingly. The counter is the nonce, and it is read back from the file, so I don't have to keep anything per block in memory. Temporary and transient databases use their page size as block size, the rest 4 KiB.

If we use the page size for temporary and transient databases, I guess we will have to change this member of the mcTempCipher struct

uint8_t buffer[MC_TEMP_HEADER + MC_TEMP_BLOCK];

MC_TEMP_BLOCK has the value 4096, but page size could be up to 64 kB. I would propose to dynamically allocate the buffer.

For now the page size comes from the first write, so the PR builds without your patch. When yours lands, it can come from there.

I will create a separate branch for implementing the handling of temporary files. And I will then apply the PR to that branch and adjust how the buffer size is set.

A bit more than the file size stayed in memory, though. For the last block I keep its counter value and how much of its keystream is used, 16 bytes. Otherwise every append rewrites the partly filled block, and a journal writes about twice as much as it does today.

Yes, that is the drawback of the block prefix approach that for blocks in the middle of the file one has to read, modify, and write the block.

We will have to analyze the impact that will have. As said this problem will not occur for temporary or transient databases, because only full blocks are read and written, but for temporary journals this could be really bad depending on how SQLite writes it. If data are appended only, it may be tolerable.

Something else from testing, in case it helps with your patches. The sorter files for ORDER BY and CREATE INDEX are opened with SQLITE_OPEN_TEMP_JOURNAL, but they don't hold pages or journal records, only sorted records written in chunks. In the plaintext check they are the worst of the lot.

Yes, I found out about that, too, recently. Unfortunately, SQLite uses SQLITE_OPEN_TEMP_JOURNAL as a generic flag for temporary data that is not a database. So, we probably have no chance to find out reliably with which type of file we actually have to deal.

And the overflow pages you asked about. When a row doesn't fit on one page, the rest of it lands in overflow pages. When SQLite reads such a value, it sometimes takes the piece it needs straight from the file, past the pager. Your codec check keeps that away from encrypted databases, but a temporary database has no codec, so there a read can start in the middle of a page.

Is that really so? I thought such reads start always at the beginning of a page, but read the page only partially.

With blocks that is no problem. If it ever becomes one, the VFS can stop reporting SQLITE_IOCAP_SUBPAGE_READ for these files and SQLite reads whole pages again.

We will have to carefully check whether this can be a serious issue or not.

The cost is the same as before. On an M4 with temp_store=FILE, a sort and a temporary table take about 1.6 times as long as with plaintext temporary files,

I still don't understand why this takes 60 percent longer.

CREATE INDEX and VACUUM 3 to 8 percent longer. I reran everything on macOS and Linux, including the runs with disk errors in the middle of a block. The stress test checks the counter values on disk now as well: they must not go backwards, and no keystream byte may be used twice.

It would be good to have this test program for further analysis of the runtime behaviour.

If you would like it different anywhere, just say so.

At first glance it looks good, but I will contact you, if I have questions or would like to have adjustments.

[mtrossbach]

Good points, thanks. The buffer is dynamic now. The block size also follows the page size for every size up to 64 kB, not just below 4 kB as before. While testing that, the stress test with 8 and 16 kB pages found a leftover: the zero buffer for filling up the last block was still 4 kB. So your hint caught a real bug.

You are right about the overflow pages, I had that wrong. The direct read starts at the beginning of the page and only reads part of it, "data is required from the start of this overflow page" is even in the comment above it. So nothing starts in the middle of a page there.

About the read-modify-write: I counted the bytes that reach the temporary files with and without encryption. Sort 77.8 against 77.9 MiB, temporary table 21.8 against 21.8, CREATE INDEX 20.8 against 20.9, VACUUM 43.5 against 43.6. So no extra I/O. Journals only append, and an append continues the keystream of the last block, which is what the 16 bytes per file are for. What does change is that every write now sits 8 bytes off an alignment boundary.

The 60 percent bothered me too, so I measured it. I built the same code with the cipher call turned into a no-op, everything else unchanged. Sort: 134 ms without encryption, 139 ms with the block layout but no cipher, 229 ms with the cipher. Temporary table: 100, 107, 176. So the layout, the offsets and the copies cost 4 to 7 percent, the rest is ChaCha20 itself.

I also timed the cipher inside the VFS. Per run the sort pushes 155.7 MiB through the temporary file and spends 88 ms of the 95 ms difference on encryption, the temporary table 87.4 MiB and 52 ms, CREATE INDEX 41.7 MiB and 24 ms, VACUUM 85.2 MiB and 48 ms. That is about 1.8 GB/s, which is what ChaCha20 does on this machine. The temporary files never leave the page cache in this benchmark, so there is no disk time for the cipher to hide behind. On real storage the share would be a lot smaller.

The stress test is in the PR now, as test/tempfiletest.c next to cryptotest, with the same four lines in Makefile.am and a step in the CI workflow. It takes the number of operations, a seed and a few modes, and it brings its own VFS that checks every write on the way to the disk. Without the encryption of temporary files it just checks the model, so it does no harm in a build that has it switched off. Drop the CI step if you would rather run it by hand.

[utelle]

Good points, thanks. The buffer is dynamic now. The block size also follows the page size for every size up to 64 kB, not just below 4 kB as before.

For temporary and transient databases we will soon just take the known page size immediately. Regarding other temporary files I'm not so sure whether changing the page size from 4 kB to anything else is really a useful step to take. For rollback journals the "optimal" block size would be "pagesize + 8" (after a 24 byte file header). I don't know whether temporary index or sort files have an "optimal" block size.

You are right about the overflow pages, I had that wrong. The direct read starts at the beginning of the page and only reads part of it, "data is required from the start of this overflow page" is even in the comment above it. So nothing starts in the middle of a page there.

Thanks for confirming.

About the read-modify-write: I counted the bytes that reach the temporary files with and without encryption. Sort 77.8 against 77.9 MiB, temporary table 21.8 against 21.8, CREATE INDEX 20.8 against 20.9, VACUUM 43.5 against 43.6. So no extra I/O. Journals only append, and an append continues the keystream of the last block, which is what the 16 bytes per file are for. What does change is that every write now sits 8 bytes off an alignment boundary.

I think the latter really doesn't matter for journal files, because pages aren't aligned anyway, due to the page record format.

The 60 percent bothered me too, so I measured it. I built the same code with the cipher call turned into a no-op, everything else unchanged. Sort: 134 ms without encryption, 139 ms with the block layout but no cipher, 229 ms with the cipher. Temporary table: 100, 107, 176. So the layout, the offsets and the copies cost 4 to 7 percent, the rest is ChaCha20 itself.

I hadn't expected that ChaCha20 would be that slow. However, you said you tested on Apple M4. That is, ChaCha20 could benefit from hardware acceleration, but it isn't enabled by default, because the implementation is not yet officially included in libsodium. Most likely performance will be better using PRAGMA mc_chacha20_hwaccel=max;.

I also timed the cipher inside the VFS. Per run the sort pushes 155.7 MiB through the temporary file and spends 88 ms of the 95 ms difference on encryption, the temporary table 87.4 MiB and 52 ms, CREATE INDEX 41.7 MiB and 24 ms, VACUUM 85.2 MiB and 48 ms. That is about 1.8 GB/s, which is what ChaCha20 does on this machine. The temporary files never leave the page cache in this benchmark, so there is no disk time for the cipher to hide behind. On real storage the share would be a lot smaller.

I haven't checked yet whether your test application enables hardware acceleration. If not, and if enabling it makes a big difference, we should make a decision whether to enable hardware acceleration for ChaCha20 by default despite of the fact that the implementation is not yet officially part of libsodium.

The stress test is in the PR now, as test/tempfiletest.c next to cryptotest, with the same four lines in Makefile.am and a step in the CI workflow. It takes the number of operations, a seed and a few modes, and it brings its own VFS that checks every write on the way to the disk. Without the encryption of temporary files it just checks the model, so it does no harm in a build that has it switched off. Drop the CI step if you would rather run it by hand.

Thanks a lot.

I merged your PR into the new branch temp-crypt. I will commit my own modifications (SQLite patches and so on) later today.

[utelle]

I committed my changes and additions to branch temp-crypt, see commit 1e0e406.

I did not remove the symbol SQLITE3MC_ENCRYPT_TEMP_FILES, so that it is possible to disable encryption for all temporary files. The patches I added allow to check whether a connection uses at least one encrypted database. If that is not the case the tempCipher is removed, because for plain databases it makes no sense to encrypt temporary files.

I also left the mechanism to determine the blocksize on the first write operation for temporary and transient databases as is, although the member pageSize of the file handle will have the correct page size, too.

[utelle]

While checking where I have to patch the SQLite amalgamation to intercept the opening of temporary files, I found out that subjournals (flag SQLITE_OPEN_SUBJOURNAL) can also be opened without a filename. However, I'm not sure under which circumstances that happens. So, for now I didn't add any checks for specific open flags - all files without a valid name will be treated as temporary files.