diff --git a/CHANGELOG.md b/CHANGELOG.md index 5b61a0f..8933631 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,14 @@ ## [Unreleased] +- Run Litestream commands without a shell and raise on command failures and timeouts. +- Support parsed JSON output from Litestream 0.5 commands with `json: true`. +- Bundle Litestream 0.5.17 release archives and verify their SHA-256 checksums during packaging. +- Replace the removed `generations`, `snapshots`, and `wal` wrappers and rake tasks with `ltx` and `status`. +- Use daemon-free `databases`, `status`, and `ltx` calls for dashboard data. +- Show local status and LTX files in the dashboard. +- Generate Litestream 0.5 configuration with a single replica and global snapshot settings. +- Document restoration and migration from Litestream 0.3 backups. + ## [0.14.0] - 2025-06-14 - Change async behaviour of replicate and other commands ([@hschne](https://github.com/fractaledmind/litestream-ruby/pull/62)) diff --git a/Gemfile.lock b/Gemfile.lock index 51f11ba..d3a5def 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -204,7 +204,6 @@ GEM rubocop (>= 1.48.1, < 2.0) rubocop-ast (>= 1.30.0, < 2.0) ruby-progressbar (1.13.0) - rubyzip (2.3.2) securerandom (0.4.1) sqlite3 (2.6.0-arm64-darwin) sqlite3 (2.6.0-x86_64-linux-gnu) @@ -244,7 +243,6 @@ DEPENDENCIES minitest (~> 5.0) rails rake (~> 13.0) - rubyzip standard (~> 1.3) BUNDLED WITH diff --git a/README.md b/README.md index e5c3056..0ce676a 100644 --- a/README.md +++ b/README.md @@ -85,16 +85,22 @@ The gem streamlines the configuration process by providing a default configurati The default configuration file looks like this if you only have one SQLite database: ```yaml +snapshot: + interval: 24h + retention: 24h + dbs: - path: storage/production.sqlite3 - replicas: - - type: s3 - path: storage/production.sqlite3 - bucket: $LITESTREAM_REPLICA_BUCKET - access-key-id: $LITESTREAM_ACCESS_KEY_ID - secret-access-key: $LITESTREAM_SECRET_ACCESS_KEY + replica: + type: s3 + path: storage/production.sqlite3 + bucket: $LITESTREAM_REPLICA_BUCKET + access-key-id: $LITESTREAM_ACCESS_KEY_ID + secret-access-key: $LITESTREAM_SECRET_ACCESS_KEY ``` +In Litestream 0.5, snapshot interval and retention are global settings rather than per-replica settings. + This is the default for Amazon S3. The full range of possible replica types (e.g. other S3-compatible object storage servers) are covered in Litestream's [replica guides](https://litestream.io/guides/#replica-guides). The gem also provides a default initializer file at `config/initializers/litestream.rb` that allows you to configure various variables referenced in the configuration file in Ruby. By providing a Ruby interface to these environment variables, you can use your preferred method of storing secrets. For example, the default generated file uses Rails' encrypted credentials to store your secrets. @@ -179,7 +185,7 @@ You can restore any replicated database at any point using the gem's provided `l > [!NOTE] > During the restoration process, you need to prevent any interaction with ActiveRecord/SQLite, such as from a running `rails server` or `rails console` instance. If there is any interaction, Rails might regenerate the production database and prevent restoration via litestream. If this happens, you might get a "cannot restore, output path already exists" error. -1. Rename the production (`production.sqlite3`, `production.sqlite3-shm`, and `production.sqlite3-wal`) databases (**recommended**) or alternatively delete. To delete the production databases locally, you can run the following at your own risk: +1. Rename the production database and its SQLite sidecar files (**recommended**) or alternatively delete them. To delete the production databases locally, you can run the following at your own risk: ```shell # DANGEROUS OPERATION, consider renaming database files instead bin/rails db:drop DISABLE_DATABASE_ENVIRONMENT_CHECK=1 @@ -215,26 +221,19 @@ You can forward arguments in whatever order you like, you simply need to ensure -if-replica-exists Returns exit code of 0 if no backups found. --parallelism NUM - Determines the number of WAL files downloaded in parallel. - Defaults to 8 - --replica NAME - Restore from a specific replica. - Defaults to replica with latest data. - --generation NAME - Restore from a specific generation. - Defaults to generation with latest data. - --index NUM - Restore up to a specific WAL index (inclusive). - Defaults to use the highest available index. +-txid TXID + Restore through a specific 16-character hexadecimal transaction ID. -timestamp TIMESTAMP Restore to a specific point-in-time. Defaults to use the latest available backup. +-json + Print the restore result as JSON. + +-dry-run + Print the restore plan without writing the database. + -config PATH Specifies the configuration file. Defaults to /etc/litestream.yml @@ -387,79 +386,59 @@ bin/rails litestream:databases This will return a list of databases and their configured replicas: ``` -path replicas +path replica /Users/you/Code/your-app/storage/production.sqlite3 s3 ``` -You can also list the generations of a specific database: +You can inspect local replication status for all databases or one specific database: ```shell -bin/rails litestream:generations -- --database=storage/production.sqlite3 +bin/rails litestream:status -- --database=storage/production.sqlite3 ``` -This will list all generations for the specified database, including stats about their lag behind the primary database and the time range they cover: +This reads local state and does not require a running Litestream process: ``` -name generation lag start end -s3 a295b16a796689f3 -156ms 2024-04-17T00:01:19Z 2024-04-17T00:01:19Z +database status local_txid wal_size +/Users/you/Code/your-app/storage/production.sqlite3 ok 000000000000000a 128 kB ``` -You can list the snapshots available for a database: +You can list all remote LTX files, including the level-9 snapshot: ```shell -bin/rails litestream:snapshots -- --database=storage/production.sqlite3 +bin/rails litestream:ltx -- --database=storage/production.sqlite3 --level=all ``` -This command lists snapshots available for that specified database: +The command returns the compaction level, transaction range, byte size, and creation time: ``` -replica generation index size created -s3 a295b16a796689f3 1 4645465 2024-04-17T00:01:19Z -``` - -Finally, you can list the wal files available for a database: - -```shell -bin/rails litestream:wal -- --database=storage/production.sqlite3 -``` - -This command lists wal files available for that specified database: - -``` -replica generation index offset size created -s3 a295b16a796689f3 1 0 2036 2024-04-17T00:01:19Z +level min_txid max_txid size created +0 0000000000000008 000000000000000a 1013 2026-09-08T03:16:43Z ``` ### Running commands from Ruby In addition to the provided rake tasks, you can also run Litestream commands directly from Ruby. The gem provides a `Litestream::Commands` module that wraps the Litestream CLI commands. This is particularly useful for the introspection commands, as you can use the output in your Ruby code. -The `Litestream::Commands.databases` method returns an array of hashes with the "path" and "replicas" keys for each database: +Pass `json: true` to return the 0.5 CLI's JSON values with string keys. The `Litestream::Commands.databases` method returns each database path and replica type: ```ruby -Litestream::Commands.databases -# => [{"path"=>"/Users/you/Code/your-app/storage/production.sqlite3", "replicas"=>"s3"}] +Litestream::Commands.databases(json: true) +# => [{"path"=>"/Users/you/Code/your-app/storage/production.sqlite3", "replica"=>"s3"}] ``` -The `Litestream::Commands.generations` method returns an array of hashes with the "name", "generation", "lag", "start", and "end" keys for each generation: +The `status` method returns local state without connecting to the replica or requiring a daemon: ```ruby -Litestream::Commands.generations('storage/production.sqlite3') -# => [{"name"=>"s3", "generation"=>"5f4341bc3d22d615", "lag"=>"3s", "start"=>"2024-04-17T19:48:09Z", "end"=>"2024-04-17T19:48:09Z"}] +Litestream::Commands.status("storage/production.sqlite3", json: true) +# => [{"database"=>"storage/production.sqlite3", "status"=>"ok", "local_txid"=>"000000000000000a", "wal_size"=>"128 kB"}] ``` -The `Litestream::Commands.snapshots` method returns an array of hashes with the "replica", "generation", "index", "size", and "created" keys for each snapshot: +The `ltx` method lists remote LTX files. An unreplicated database returns an empty array: ```ruby -Litestream::Commands.snapshots('storage/production.sqlite3') -# => [{"replica"=>"s3", "generation"=>"5f4341bc3d22d615", "index"=>"0", "size"=>"4645465", "created"=>"2024-04-17T19:48:09Z"}] -``` - -The `Litestream::Commands.wal` method returns an array of hashes with the "replica", "generation", "index", "offset","size", and "created" keys for each wal: - -```ruby -Litestream::Commands.wal('storage/production.sqlite3') -# => [{"replica"=>"s3", "generation"=>"5f4341bc3d22d615", "index"=>"0", "offset"=>"0", "size"=>"2036", "created"=>"2024-04-17T19:48:09Z"}] +Litestream::Commands.ltx("storage/production.sqlite3", json: true, "--level" => "all") +# => [{"level"=>0, "min_txid"=>"0000000000000008", "max_txid"=>"000000000000000a", "size"=>1013, "timestamp"=>"2026-09-08T03:16:43Z"}] ``` You can also restore a database programmatically using the `Litestream::Commands.restore` method, which returns the path to the restored database: @@ -479,14 +458,29 @@ The full set of commands available to the `litestream` executable are covered in ```shell litestream databases [arguments] -litestream generations [arguments] DB_PATH|REPLICA_URL +litestream info [arguments] +litestream list [arguments] +litestream ltx [arguments] DB_PATH +litestream register [arguments] litestream replicate [arguments] -litestream restore [arguments] DB_PATH|REPLICA_URL -litestream snapshots [arguments] DB_PATH|REPLICA_URL +litestream reset [arguments] +litestream restore [arguments] DB_PATH +litestream start [arguments] +litestream status [arguments] [DB_PATH] +litestream stop [arguments] +litestream sync [arguments] +litestream unregister [arguments] litestream version -litestream wal [arguments] DB_PATH|REPLICA_URL ``` +### Upgrading from 0.3 + +Litestream 0.5 changes each database from a `replicas:` list to a single `replica:` map and moves retention and snapshot settings into the root-level `snapshot:` block. Existing `replicas:` configuration files must be edited by hand before upgrading. The `databases` command and Ruby method now return a `"replica"` key instead of `"replicas"`. + +The removed `generations`, `snapshots`, and `wal` command methods and rake tasks now raise an error pointing to `ltx`, which replaces all three forms of remote backup introspection. + +The 0.5 `restore` command auto-detects both 0.3 and LTX backups and prefers whichever backup is newer. Existing 0.3 `generations/` objects are left in place; you can delete them after the LTX history covers your full retention window. To roll back, reinstall a 0.14.x release of this gem, which bundles Litestream 0.3. + ### Using in development By default, if you install the gem and configure via `puma.rb` or `Procfile`, Litestream will not start in development. diff --git a/app/views/litestream/processes/show.html.erb b/app/views/litestream/processes/show.html.erb index d8c8e31..e9cc0f0 100644 --- a/app/views/litestream/processes/show.html.erb +++ b/app/views/litestream/processes/show.html.erb @@ -56,63 +56,51 @@
-
- <% database['generations'].each do |generation| %> -
- - <%= generation['generation'] %> - (<%= generation['lag'] %> lag) - +
+ <% if database['error'] %> +

<%= database['error'] %>

+ <% else %> +
+
Status
+
<%= database.dig('status', 'status') %>
+
Local TXID
+
<%= database.dig('status', 'local_txid') %>
+
WAL size
+
<%= number_to_human_size database.dig('status', 'wal_size') %>
+
-
-
Start
-
- - - -
+
+ <% if database['ltx'] == [] %> +

No LTX files yet

+ <% else %> + + + + + + + + + + -
End
-
- - - -
- -
-
Snapshots
-
-
LevelMin TXIDMax TXIDSizeCreated
- - - - - - - - - - <% generation['snapshots'].each do |snapshot| %> - - - - - - <% end %> - -
Created atIndexSize
- - - - - <%= snapshot['index'] %> - - <%= number_to_human_size snapshot['size'] %> -
- - -
-
+ + <% database['ltx'].each do |ltx| %> + + <%= ltx['level'] %> + <%= ltx['min_txid'] %> + <%= ltx['max_txid'] %> + <%= number_to_human_size ltx['size'] %> + + + + + + + <% end %> + + + <% end %> <% end %>
diff --git a/lib/litestream.rb b/lib/litestream.rb index 159faa9..075d2d3 100644 --- a/lib/litestream.rb +++ b/lib/litestream.rb @@ -107,20 +107,18 @@ def replicate_process end def databases - databases = Commands.databases + databases = Commands.databases(json: true) databases.each do |db| - generations = Commands.generations(db["path"]) - snapshots = Commands.snapshots(db["path"]) - db["path"] = db["path"].gsub(Rails.root.to_s, "[ROOT]") - - db["generations"] = generations.map do |generation| - id = generation["generation"] - replica = generation["name"] - generation["snapshots"] = snapshots.select { |snapshot| snapshot["generation"] == id && snapshot["replica"] == replica } - .map { |s| s.slice("index", "size", "created") } - generation.slice("generation", "name", "lag", "start", "end", "snapshots") + path = db["path"] + begin + db["status"] = Commands.status(path, json: true).first + db["ltx"] = Commands.ltx(path, :json => true, "--level" => "all") + rescue Commands::CommandFailedException => error + db["error"] = error.message end + + db["path"] = path.gsub(Rails.root.to_s, "[ROOT]") end end diff --git a/lib/litestream/commands.rb b/lib/litestream/commands.rb index 0d86aad..4bbcb6c 100644 --- a/lib/litestream/commands.rb +++ b/lib/litestream/commands.rb @@ -1,3 +1,5 @@ +require "json" +require "open3" require_relative "upstream" module Litestream @@ -20,6 +22,9 @@ module Commands # raised when a litestream command fails CommandFailedException = Class.new(StandardError) + # raised when a litestream command times out + CommandTimeoutException = Class.new(CommandFailedException) + module Output class << self def format(data) @@ -47,7 +52,10 @@ def executable(exe_path: DEFAULT_DIR) litestream_install_dir = ENV["LITESTREAM_INSTALL_DIR"] if litestream_install_dir if File.directory?(litestream_install_dir) - warn "NOTE: using LITESTREAM_INSTALL_DIR to find litestream executable: #{litestream_install_dir}" + unless @litestream_install_dir_noted + warn "NOTE: using LITESTREAM_INSTALL_DIR to find litestream executable: #{litestream_install_dir}" + @litestream_install_dir_noted = true + end exe_path = litestream_install_dir exe_file = File.expand_path(File.join(litestream_install_dir, "litestream")) else @@ -100,6 +108,8 @@ def executable(exe_path: DEFAULT_DIR) def replicate(async: false, **argv) cmd = prepare("replicate", argv) run_replicate(cmd, async: async) + rescue CommandFailedException + raise rescue raise CommandFailedException, "Failed to execute `#{cmd.join(" ")}`" end @@ -114,35 +124,48 @@ def databases(**argv) execute("databases", argv) end - def generations(database, **argv) - raise DatabaseRequiredException, "database argument is required for generations command, e.g. litestream:generations -- --database=path/to/database.sqlite" if database.nil? + def ltx(database, **argv) + raise DatabaseRequiredException, "database argument is required for ltx command, e.g. litestream:ltx -- --database=path/to/database.sqlite" if database.nil? - execute("generations", argv, database) + execute("ltx", argv, database) end - def snapshots(database, **argv) - raise DatabaseRequiredException, "database argument is required for snapshots command, e.g. litestream:snapshots -- --database=path/to/database.sqlite" if database.nil? + # Litestream 0.5 filters `status` by the absolute database path, while + # `ltx` and `restore` match the path as written in the config file, so a + # relative path is expanded here to keep the documented + # `--database=storage/production.sqlite3` form working. + def status(database = nil, **argv) + execute("status", argv, database && File.expand_path(database)) + end - execute("snapshots", argv, database) + def generations(*) + raise CommandFailedException, "`generations` was removed in Litestream 0.5; use `ltx` (see README, Upgrading from 0.3)" end - def wal(database, **argv) - raise DatabaseRequiredException, "database argument is required for wal command, e.g. litestream:wal -- --database=path/to/database.sqlite" if database.nil? + def snapshots(*) + raise CommandFailedException, "`snapshots` was removed in Litestream 0.5; use `ltx` (see README, Upgrading from 0.3)" + end - execute("wal", argv, database) + def wal(*) + raise CommandFailedException, "`wal` was removed in Litestream 0.5; use `ltx` (see README, Upgrading from 0.3)" end private def execute(command, argv = {}, database = nil, tabled_output: true) - cmd = prepare(command, argv, database) - results = run(cmd, tabled_output: tabled_output) - - if Array === results && results.one? && results[0]["level"] == "ERROR" - raise CommandFailedException, "Failed to execute `#{cmd.join(" ")}`; Reason: #{results[0]["error"]}" + argv = argv.stringify_keys + timeout = argv.delete("timeout") + output = if argv.delete("json") + argv["-json"] = nil + :json + elsif tabled_output + :table else - results + :raw end + + cmd = prepare(command, argv, database) + run(cmd, output: output, timeout: timeout) end def prepare(command, argv = {}, database = nil) @@ -154,18 +177,71 @@ def prepare(command, argv = {}, database = nil) args = { "--config" => Litestream.config_path.to_s - }.merge(argv.stringify_keys).to_a.flatten.compact - cmd = [executable, command, *args, database].compact + }.merge(argv.stringify_keys).to_a.flatten.compact.map(&:to_s) + cmd = [executable, command, *args, database].compact.map(&:to_s) puts cmd.inspect if ENV["DEBUG"] cmd end - def run(cmd, tabled_output:) - stdout = `#{cmd.join(" ")}`.chomp - return stdout unless tabled_output + # Runs the command without a shell and returns its parsed stdout. A non-zero + # exit raises with stderr. With a timeout, the command runs in its own + # process group and is killed (TERM, then KILL) when the deadline passes. + def run(cmd, output:, timeout: nil) + stdin, stdout, stderr, wait_thread = Open3.popen3(*cmd, pgroup: true) + stdin.close + stdout_reader = Thread.new { stdout.read } + stderr_reader = Thread.new { stderr.read } + + # The readers finish when the last process holding the pipes exits, so + # waiting on them covers descendants the direct child may have left behind. + unless wait_thread.join(timeout) && stdout_reader.join(timeout) && stderr_reader.join(timeout) + kill_process_group("TERM", wait_thread.pid) + wait_thread.join(1) + kill_process_group("KILL", wait_thread.pid) + wait_thread.join + [stdout_reader, stderr_reader].each(&:join) + raise CommandTimeoutException, "Failed to execute `#{cmd[1]}`: timed out after #{timeout} seconds" + end + + status = wait_thread.value + unless status.success? + raise CommandFailedException, "Failed to execute `#{cmd[1]}` (exit status #{status.exitstatus}): #{stderr_reader.value.strip[0, 500]}" + end + + case output + when :json then parse_json(cmd, stdout_reader.value) + when :table then parse_table(stdout_reader.value) + else stdout_reader.value + end + ensure + [stdout_reader, stderr_reader].each { |reader| reader&.join } + [stdin, stdout, stderr].each { |io| io&.close unless io&.closed? } + end + + def kill_process_group(signal, pid) + Process.kill(signal, -pid) + rescue Errno::ESRCH + end + + # Two opt-in restore skips (-if-db-not-exists when the output exists, + # -if-replica-exists with no backups) exit 0 and print one logfmt line on + # stdout instead of JSON. They come back as {"skipped" => true, "message" => ...}. + def parse_json(cmd, stdout) + stdout = stdout.strip + return if stdout.empty? + return JSON.parse(stdout) if stdout.start_with?("{", "[") + + skipped = stdout.match(/\Atime=\S+ level=\S+ msg=(?:"([^"]*)"|(\S+))/) + return {"skipped" => true, "message" => skipped[1] || skipped[2]} if skipped + + raise CommandFailedException, "Unexpected output from `#{cmd[1]}`: #{stdout.lines.first.to_s.strip[0, 200]}" + end + + def parse_table(stdout) + keys, *rows = stdout.strip.split("\n").map { _1.split(/\s+/) } + return [] unless keys - keys, *rows = stdout.split("\n").map { _1.split(/\s+/) } rows.map { keys.zip(_1).to_h } end @@ -177,6 +253,14 @@ def run_replicate(cmd, async:) IO.popen(cmd, err: [:child, :out]) do |io| io.each_line { |line| puts line } end + status = $? + # `replicate` runs until it is signalled, so a signal is how it is + # meant to end. A non-zero exit is litestream failing to start or + # dying on its own, which is invisible today: the task exits 0 and a + # supervisor sees a clean stop rather than a crash to restart. + if status && !status.success? && !status.signaled? + raise CommandFailedException, "Failed to execute `replicate` (exit status #{status.exitstatus})" + end end end end diff --git a/lib/litestream/generators/litestream/templates/config.yml.erb b/lib/litestream/generators/litestream/templates/config.yml.erb index 43009db..f789f8a 100644 --- a/lib/litestream/generators/litestream/templates/config.yml.erb +++ b/lib/litestream/generators/litestream/templates/config.yml.erb @@ -9,13 +9,18 @@ # `replicate` command. # # For more details, see: https://litestream.io/reference/config/ +# Retention and snapshot settings are global in Litestream 0.5. +snapshot: + interval: 24h + retention: 24h + dbs: <%- production_sqlite_databases.each do |database| -%> - path: <%= database %> - replicas: - - type: s3 - bucket: $LITESTREAM_REPLICA_BUCKET - path: <%= database %> - access-key-id: $LITESTREAM_ACCESS_KEY_ID - secret-access-key: $LITESTREAM_SECRET_ACCESS_KEY + replica: + type: s3 + bucket: $LITESTREAM_REPLICA_BUCKET + path: <%= database %> + access-key-id: $LITESTREAM_ACCESS_KEY_ID + secret-access-key: $LITESTREAM_SECRET_ACCESS_KEY <%- end -%> diff --git a/lib/litestream/upstream.rb b/lib/litestream/upstream.rb index 0029efd..c25bead 100644 --- a/lib/litestream/upstream.rb +++ b/lib/litestream/upstream.rb @@ -1,14 +1,25 @@ module Litestream module Upstream - VERSION = "v0.3.13" + VERSION = "0.5.17" # rubygems platform name => upstream release filename NATIVE_PLATFORMS = { "aarch64-linux" => "litestream-#{VERSION}-linux-arm64.tar.gz", - "arm64-darwin" => "litestream-#{VERSION}-darwin-arm64.zip", + "arm64-darwin" => "litestream-#{VERSION}-darwin-arm64.tar.gz", "arm64-linux" => "litestream-#{VERSION}-linux-arm64.tar.gz", - "x86_64-darwin" => "litestream-#{VERSION}-darwin-amd64.zip", - "x86_64-linux" => "litestream-#{VERSION}-linux-amd64.tar.gz" + "x86_64-darwin" => "litestream-#{VERSION}-darwin-x86_64.tar.gz", + "x86_64-linux" => "litestream-#{VERSION}-linux-x86_64.tar.gz" } + + CHECKSUMS = { + "litestream-0.5.17-linux-x86_64.tar.gz" => "cfb371176d164437ae869f8351cfde49bd1804ae71c61923f75c9cba9c9c006d", + "litestream-0.5.17-linux-arm64.tar.gz" => "f8ca4a050095c1efbda2c4365172e61bf9d955ea0d9ac42f448b52e51819baa5", + "litestream-0.5.17-darwin-x86_64.tar.gz" => "891875af09db152e93a4b31a8a79f538ce7ce702c132803cfe0a831e7cb1b7db", + "litestream-0.5.17-darwin-arm64.tar.gz" => "e211f68ff7658d19f193f2914417afdf8f89a053ff8f263e5d6b3b1d3bbc7b08" + } + + def self.download_url(filename) + "https://github.com/benbjohnson/litestream/releases/download/v#{VERSION}/#{filename}" + end end end diff --git a/lib/tasks/litestream_tasks.rake b/lib/tasks/litestream_tasks.rake index 387701b..0cf4945 100644 --- a/lib/tasks/litestream_tasks.rake +++ b/lib/tasks/litestream_tasks.rake @@ -32,30 +32,35 @@ namespace :litestream do puts Litestream::Commands::Output.format(Litestream::Commands.databases(**options)) end - desc "List all generations for a database or replica, for example `rake litestream:generations -- -database=storage/production.sqlite3`" - task generations: :environment do + desc "List all LTX files for a database or replica, for example `rake litestream:ltx -- -database=storage/production.sqlite3`" + task ltx: :environment do options = parse_argv_options database = options.delete(:"--database") || options.delete(:"-database") - puts Litestream::Commands::Output.format(Litestream::Commands.generations(database, **options)) + puts Litestream::Commands::Output.format(Litestream::Commands.ltx(database, **options)) end - desc "List all snapshots for a database or replica, for example `rake litestream:snapshots -- -database=storage/production.sqlite3`" - task snapshots: :environment do + desc "Show replication status, for example `rake litestream:status -- -database=storage/production.sqlite3`" + task status: :environment do options = parse_argv_options database = options.delete(:"--database") || options.delete(:"-database") - puts Litestream::Commands::Output.format(Litestream::Commands.snapshots(database, **options)) + puts Litestream::Commands::Output.format(Litestream::Commands.status(database, **options)) end - desc "List all wal files for a database or replica, for example `rake litestream:wal -- -database=storage/production.sqlite3`" - task wal: :environment do - options = parse_argv_options - database = options.delete(:"--database") || options.delete(:"-database") + desc "Explain the removal of the generations command" + task generations: :environment do + abort "`generations` was removed in Litestream 0.5; use `ltx` (see README, Upgrading from 0.3)" + end - puts Litestream::Commands::Output.format( - Litestream::Commands.wal(database, **options) - ) + desc "Explain the removal of the snapshots command" + task snapshots: :environment do + abort "`snapshots` was removed in Litestream 0.5; use `ltx` (see README, Upgrading from 0.3)" + end + + desc "Explain the removal of the wal command" + task wal: :environment do + abort "`wal` was removed in Litestream 0.5; use `ltx` (see README, Upgrading from 0.3)" end private diff --git a/litestream.gemspec b/litestream.gemspec index e004350..e19400f 100644 --- a/litestream.gemspec +++ b/litestream.gemspec @@ -34,7 +34,6 @@ Gem::Specification.new do |spec| spec.add_dependency "railties", rails_version end spec.add_development_dependency "rails" - spec.add_development_dependency "rubyzip" # For more information and examples about making a new gem, check out our # guide at: https://bundler.io/guides/creating_gem.html diff --git a/rakelib/package.rake b/rakelib/package.rake index f1a047f..c4d134a 100644 --- a/rakelib/package.rake +++ b/rakelib/package.rake @@ -58,12 +58,13 @@ require "rubygems/package" require "rubygems/package_task" require "open-uri" +require "digest" +require "stringio" require "zlib" -require "zip" require_relative "../lib/litestream/upstream" def litestream_download_url(filename) - "https://github.com/benbjohnson/litestream/releases/download/#{Litestream::Upstream::VERSION}/#{filename}" + Litestream::Upstream.download_url(filename) end LITESTREAM_RAILS_GEMSPEC = Bundler.load_gemspec("litestream.gemspec") @@ -77,11 +78,13 @@ Litestream::Upstream::NATIVE_PLATFORMS.each do |platform, filename| LITESTREAM_RAILS_GEMSPEC.dup.tap do |gemspec| exedir = File.join(gemspec.bindir, platform) # "exe/x86_64-linux" exepath = File.join(exedir, "litestream") # "exe/x86_64-linux/litestream" + checksum_path = "#{exepath}.sha256" + expected_checksum = Litestream::Upstream::CHECKSUMS.fetch(filename) exepaths << exepath # modify a copy of the gemspec to include the native executable gemspec.platform = platform - gemspec.files += [exepath, "LICENSE-DEPENDENCIES"] + gemspec.files += [exepath, checksum_path, "LICENSE-DEPENDENCIES"] # create a package task gem_path = Gem::PackageTask.new(gemspec).define @@ -89,28 +92,35 @@ Litestream::Upstream::NATIVE_PLATFORMS.each do |platform, filename| task "gem:#{platform}" => [gem_path] directory exedir - file exepath => [exedir] do + validation_task = "validate-litestream-#{platform}" + task validation_task do + checksum_matches = File.exist?(checksum_path) && File.read(checksum_path).strip == expected_checksum + FileUtils.rm_f([exepath, checksum_path]) unless checksum_matches + end + + file exepath => [exedir, validation_task] do release_url = litestream_download_url(filename) warn "Downloading #{exepath} from #{release_url} ..." - # lazy, but fine for now. URI.open(release_url) do |remote| # standard:disable Security/Open - if release_url.end_with?(".zip") - Zip::File.open_buffer(remote) do |zip_file| - zip_file.extract("litestream", exepath) - end - elsif release_url.end_with?(".gz") - Zlib::GzipReader.wrap(remote) do |gz| - Gem::Package::TarReader.new(gz) do |reader| - reader.seek("litestream") do |file| - File.binwrite(exepath, file.read) - end + archive = remote.read + actual_checksum = Digest::SHA256.hexdigest(archive) + unless actual_checksum == expected_checksum + raise "Checksum mismatch for #{filename}: expected #{expected_checksum}, got #{actual_checksum}" + end + + Zlib::GzipReader.wrap(StringIO.new(archive)) do |gz| + Gem::Package::TarReader.new(gz) do |reader| + reader.seek("litestream") do |file| + File.binwrite(exepath, file.read) end end end end FileUtils.chmod(0o755, exepath, verbose: true) + File.write(checksum_path, "#{expected_checksum}\n") end + file checksum_path => [exepath] end end diff --git a/test/controllers/test_processes_controller.rb b/test/controllers/test_processes_controller.rb index 3f15a6e..af4f732 100644 --- a/test/controllers/test_processes_controller.rb +++ b/test/controllers/test_processes_controller.rb @@ -5,17 +5,12 @@ class Litestream::TestProcessesController < ActionDispatch::IntegrationTest stubbed_process = {pid: "12345", status: "sleeping", started: DateTime.now} stubbed_databases = [ {"path" => "[ROOT]/storage/test.sqlite3", - "replicas" => "s3", - "generations" => [ - {"generation" => SecureRandom.hex, - "name" => "s3", - "lag" => "23h59m59s", - "start" => "2024-05-02T11:32:16Z", - "end" => "2024-05-02T11:33:10Z", - "snapshots" => [ - {"index" => "0", "size" => "4145735", "created" => "2024-05-02T11:32:16Z"} - ]} - ]} + "replica" => "s3", + "status" => {"database" => "storage/test.sqlite3", "status" => "ok", "local_txid" => "000000000000000a", "wal_size" => 131072}, + "ltx" => [ + {"level" => 0, "min_txid" => "0000000000000008", "max_txid" => "000000000000000a", "size" => 1013, "timestamp" => "2026-09-08T03:16:43Z"} + ]}, + {"path" => "[ROOT]/storage/error.sqlite3", "replica" => "s3", "error" => "replica unavailable"} ] Litestream.stub :replicate_process, stubbed_process do Litestream.stub :databases, stubbed_databases do @@ -28,10 +23,13 @@ class Litestream::TestProcessesController < ActionDispatch::IntegrationTest assert_select "time", stubbed_process[:started].to_formatted_s(:db) end - assert_select "#databases li", 1 do + assert_select "#databases li", 2 + assert_select "#databases li:first-child" do assert_select "h2 code", stubbed_databases[0]["path"] - assert_select "details##{stubbed_databases[0]["generations"][0]["generation"]}" + assert_select "tbody tr", 1 + assert_select "td", text: "000000000000000a" end + assert_select "#databases li:last-child .text-red-600", "replica unavailable" end end end diff --git a/test/dummy/config/litestream.yml b/test/dummy/config/litestream.yml index 53280aa..50c2e46 100644 --- a/test/dummy/config/litestream.yml +++ b/test/dummy/config/litestream.yml @@ -9,12 +9,17 @@ # `replicate` command. # # For more details, see: https://litestream.io/reference/config/ +# Retention and snapshot settings are global in Litestream 0.5. +snapshot: + interval: 24h + retention: 24h + dbs: - path: storage/test.sqlite3 - replicas: - - type: s3 - bucket: $LITESTREAM_REPLICA_BUCKET - path: test - endpoint: http://localhost:9000 - access-key-id: $LITESTREAM_ACCESS_KEY_ID - secret-access-key: $LITESTREAM_SECRET_ACCESS_KEY + replica: + type: s3 + bucket: $LITESTREAM_REPLICA_BUCKET + path: test + endpoint: http://localhost:9000 + access-key-id: $LITESTREAM_ACCESS_KEY_ID + secret-access-key: $LITESTREAM_SECRET_ACCESS_KEY diff --git a/test/generators/test_install.rb b/test/generators/test_install.rb index ad7cda9..6805f0c 100644 --- a/test/generators/test_install.rb +++ b/test/generators/test_install.rb @@ -22,6 +22,10 @@ def after_teardown assert_match "- path: storage/test.sqlite3", content assert_match "- path: storage/queue.sqlite3", content assert_match "- path: storage/errors.sqlite3", content + assert_match "snapshot:", content + assert_match "interval: 24h", content + assert_match "retention: 24h", content + assert_match "replica:", content assert_match "bucket: $LITESTREAM_REPLICA_BUCKET", content assert_match "access-key-id: $LITESTREAM_ACCESS_KEY_ID", content assert_match "secret-access-key: $LITESTREAM_SECRET_ACCESS_KEY", content diff --git a/test/litestream/test_commands.rb b/test/litestream/test_commands.rb index 9b96224..1a7c4c7 100644 --- a/test/litestream/test_commands.rb +++ b/test/litestream/test_commands.rb @@ -1,4 +1,6 @@ require "test_helper" +require "fileutils" +require "tmpdir" class TestCommands < ActiveSupport::TestCase def run @@ -161,6 +163,36 @@ def test_replicate_does_not_set_env_var_from_config_when_env_vars_already_set assert_equal "original_key", ENV["LITESTREAM_ACCESS_KEY_ID"] assert_equal "original_access", ENV["LITESTREAM_SECRET_ACCESS_KEY"] end + + # These run a real child, and Litestream.replicate_process reads the global + # $? straight after backticks that its own tests stub out, so a failed or + # signalled status left here surfaces over there. Leave a clean one behind. + def teardown + super + system("true") + end + + def test_replicate_in_process_raises_when_litestream_exits_non_zero + error = assert_raises Litestream::Commands::CommandFailedException do + Litestream::Commands.send(:run_replicate, ["/bin/sh", "-c", "echo boom; exit 3"], async: false) + end + + assert_match "exit status 3", error.message + end + + def test_replicate_in_process_does_not_raise_when_litestream_is_signalled + Litestream::Commands.send(:run_replicate, ["/bin/sh", "-c", "kill -TERM $$"], async: false) + end + + def test_replicate_reports_the_exit_status_rather_than_the_command + error = assert_raises Litestream::Commands::CommandFailedException do + Litestream::Commands.stub :prepare, ["/bin/sh", "-c", "exit 3"] do + Litestream::Commands.replicate + end + end + + assert_match "exit status 3", error.message + end end class TestRestoreCommand < TestCommands @@ -204,7 +236,7 @@ def test_restore_with_string_option assert_equal "--config", argv[0] assert_match Regexp.new("dummy/config/litestream.yml"), argv[1] assert_equal "--parallelism", argv[2] - assert_equal 10, argv[3] + assert_equal "10", argv[3] assert_equal "db/test.sqlite3", argv[4] end Litestream::Commands.stub :run, stub do @@ -441,410 +473,276 @@ def test_databases_read_from_custom_configured_litestream_config_path end end - class TestGenerationsCommand < TestCommands - def test_generations_with_no_options + class TestLtxCommand < TestCommands + def test_ltx_with_no_options stub = proc do |cmd| executable, command, *argv = cmd assert_match Regexp.new("exe/test/litestream"), executable - assert_equal "generations", command - assert_equal 3, argv.size - assert_equal "--config", argv[0] - assert_match Regexp.new("dummy/config/litestream.yml"), argv[1] - assert_equal "db/test.sqlite3", argv[2] + assert_equal "ltx", command + assert_equal ["--config", Rails.root.join("config/litestream.yml").to_s, "db/test.sqlite3"], argv end - Litestream::Commands.stub :run, stub do - Litestream::Commands.generations("db/test.sqlite3") - end - end - def test_generations_with_boolean_option - stub = proc do |cmd| - executable, command, *argv = cmd - assert_match Regexp.new("exe/test/litestream"), executable - assert_equal "generations", command - assert_equal 4, argv.size - assert_equal "--config", argv[0] - assert_match Regexp.new("dummy/config/litestream.yml"), argv[1] - assert_equal "--if-db-not-exists", argv[2] - assert_equal "db/test.sqlite3", argv[3] - end Litestream::Commands.stub :run, stub do - Litestream::Commands.generations("db/test.sqlite3", "--if-db-not-exists" => nil) + Litestream::Commands.ltx("db/test.sqlite3") end end - def test_generations_with_string_option + def test_ltx_with_options stub = proc do |cmd| - executable, command, *argv = cmd - assert_match Regexp.new("exe/test/litestream"), executable - assert_equal "generations", command - assert_equal 5, argv.size - assert_equal "--config", argv[0] - assert_match Regexp.new("dummy/config/litestream.yml"), argv[1] - assert_equal "--parallelism", argv[2] - assert_equal 10, argv[3] - assert_equal "db/test.sqlite3", argv[4] - end - Litestream::Commands.stub :run, stub do - Litestream::Commands.generations("db/test.sqlite3", "--parallelism" => 10) + assert_equal ["--config", "CONFIG", "--level", "all", "-json", "db/test.sqlite3"], cmd.drop(2) end - end - def test_generations_with_config_option - stub = proc do |cmd| - executable, command, *argv = cmd - assert_match Regexp.new("exe/test/litestream"), executable - assert_equal "generations", command - assert_equal 3, argv.size - assert_equal "--config", argv[0] - assert_equal "CONFIG", argv[1] - assert_equal "db/test.sqlite3", argv[2] - end Litestream::Commands.stub :run, stub do - Litestream::Commands.generations("db/test.sqlite3", "--config" => "CONFIG") - end - end - - def test_generations_sets_replica_bucket_env_var_from_config_when_env_var_not_set - Litestream.replica_bucket = "mybkt" - - Litestream::Commands.stub :run, nil do - Litestream::Commands.generations("db/test.sqlite3") - end - - assert_equal "mybkt", ENV["LITESTREAM_REPLICA_BUCKET"] - assert_nil ENV["LITESTREAM_ACCESS_KEY_ID"] - assert_nil ENV["LITESTREAM_SECRET_ACCESS_KEY"] - end - - def test_generations_sets_replica_key_id_env_var_from_config_when_env_var_not_set - Litestream.replica_key_id = "mykey" - - Litestream::Commands.stub :run, nil do - Litestream::Commands.generations("db/test.sqlite3") - end - - assert_nil ENV["LITESTREAM_REPLICA_BUCKET"] - assert_equal "mykey", ENV["LITESTREAM_ACCESS_KEY_ID"] - assert_nil ENV["LITESTREAM_SECRET_ACCESS_KEY"] - end - - def test_generations_sets_replica_access_key_env_var_from_config_when_env_var_not_set - Litestream.replica_access_key = "access" - - Litestream::Commands.stub :run, nil do - Litestream::Commands.generations("db/test.sqlite3") - end - - assert_nil ENV["LITESTREAM_REPLICA_BUCKET"] - assert_nil ENV["LITESTREAM_ACCESS_KEY_ID"] - assert_equal "access", ENV["LITESTREAM_SECRET_ACCESS_KEY"] - end - - def test_generations_sets_all_env_vars_from_config_when_env_vars_not_set - Litestream.replica_bucket = "mybkt" - Litestream.replica_key_id = "mykey" - Litestream.replica_access_key = "access" - - Litestream::Commands.stub :run, nil do - Litestream::Commands.generations("db/test.sqlite3") + Litestream::Commands.ltx("db/test.sqlite3", "--config" => "CONFIG", "--level" => "all", :json => true) end - - assert_equal "mybkt", ENV["LITESTREAM_REPLICA_BUCKET"] - assert_equal "mykey", ENV["LITESTREAM_ACCESS_KEY_ID"] - assert_equal "access", ENV["LITESTREAM_SECRET_ACCESS_KEY"] end - def test_generations_does_not_set_env_var_from_config_when_env_vars_already_set - ENV["LITESTREAM_REPLICA_BUCKET"] = "original_bkt" - ENV["LITESTREAM_ACCESS_KEY_ID"] = "original_key" - ENV["LITESTREAM_SECRET_ACCESS_KEY"] = "original_access" - - Litestream.replica_bucket = "mybkt" - Litestream.replica_key_id = "mykey" - Litestream.replica_access_key = "access" - - Litestream::Commands.stub :run, nil do - Litestream::Commands.generations("db/test.sqlite3") + def test_ltx_requires_database + error = assert_raises Litestream::Commands::DatabaseRequiredException do + Litestream::Commands.ltx(nil) end - assert_equal "original_bkt", ENV["LITESTREAM_REPLICA_BUCKET"] - assert_equal "original_key", ENV["LITESTREAM_ACCESS_KEY_ID"] - assert_equal "original_access", ENV["LITESTREAM_SECRET_ACCESS_KEY"] + assert_match "database argument is required for ltx command", error.message end end - class TestSnapshotsCommand < TestCommands - def test_snapshots_with_no_options + class TestStatusCommand < TestCommands + def test_status_with_no_database stub = proc do |cmd| executable, command, *argv = cmd assert_match Regexp.new("exe/test/litestream"), executable - assert_equal "snapshots", command - assert_equal 3, argv.size - assert_equal "--config", argv[0] - assert_match Regexp.new("dummy/config/litestream.yml"), argv[1] - assert_equal "db/test.sqlite3", argv[2] + assert_equal "status", command + assert_equal ["--config", Rails.root.join("config/litestream.yml").to_s], argv end - Litestream::Commands.stub :run, stub do - Litestream::Commands.snapshots("db/test.sqlite3") - end - end - def test_snapshots_with_boolean_option - stub = proc do |cmd| - executable, command, *argv = cmd - assert_match Regexp.new("exe/test/litestream"), executable - assert_equal "snapshots", command - assert_equal 4, argv.size - assert_equal "--config", argv[0] - assert_match Regexp.new("dummy/config/litestream.yml"), argv[1] - assert_equal "--if-db-not-exists", argv[2] - assert_equal "db/test.sqlite3", argv[3] - end Litestream::Commands.stub :run, stub do - Litestream::Commands.snapshots("db/test.sqlite3", "--if-db-not-exists" => nil) + Litestream::Commands.status end end - def test_snapshots_with_string_option + def test_status_expands_a_relative_database_path stub = proc do |cmd| - executable, command, *argv = cmd - assert_match Regexp.new("exe/test/litestream"), executable - assert_equal "snapshots", command - assert_equal 5, argv.size - assert_equal "--config", argv[0] - assert_match Regexp.new("dummy/config/litestream.yml"), argv[1] - assert_equal "--parallelism", argv[2] - assert_equal 10, argv[3] - assert_equal "db/test.sqlite3", argv[4] + assert_equal File.expand_path("db/test.sqlite3"), cmd.last end + Litestream::Commands.stub :run, stub do - Litestream::Commands.snapshots("db/test.sqlite3", "--parallelism" => 10) + Litestream::Commands.status("db/test.sqlite3") end end - def test_snapshots_with_config_option + def test_status_with_database_and_options stub = proc do |cmd| - executable, command, *argv = cmd - assert_match Regexp.new("exe/test/litestream"), executable - assert_equal "snapshots", command - assert_equal 3, argv.size - assert_equal "--config", argv[0] - assert_equal "CONFIG", argv[1] - assert_equal "db/test.sqlite3", argv[2] + assert_equal ["--config", "CONFIG", "--no-expand-env", "-json", File.expand_path("db/test.sqlite3")], cmd.drop(2) end + Litestream::Commands.stub :run, stub do - Litestream::Commands.snapshots("db/test.sqlite3", "--config" => "CONFIG") + Litestream::Commands.status("db/test.sqlite3", "--config" => "CONFIG", "--no-expand-env" => nil, :json => true) end end + end - def test_snapshots_sets_replica_bucket_env_var_from_config_when_env_var_not_set - Litestream.replica_bucket = "mybkt" + class TestRemovedCommands < TestCommands + %w[generations snapshots wal].each do |command| + define_method(:"test_#{command}_raises_with_migration_message") do + error = assert_raises Litestream::Commands::CommandFailedException do + Litestream::Commands.public_send(command, "db/test.sqlite3") + end - Litestream::Commands.stub :run, nil do - Litestream::Commands.snapshots("db/test.sqlite3") + assert_equal "`#{command}` was removed in Litestream 0.5; use `ltx` (see README, Upgrading from 0.3)", error.message end - - assert_equal "mybkt", ENV["LITESTREAM_REPLICA_BUCKET"] - assert_nil ENV["LITESTREAM_ACCESS_KEY_ID"] - assert_nil ENV["LITESTREAM_SECRET_ACCESS_KEY"] end + end - def test_snapshots_sets_replica_key_id_env_var_from_config_when_env_var_not_set - Litestream.replica_key_id = "mykey" + class TestRunner < ActiveSupport::TestCase + def setup + @tmpdir = Dir.mktmpdir + @litestream_install_dir = ENV["LITESTREAM_INSTALL_DIR"] + @executable = File.join(@tmpdir, "litestream") + File.write(@executable, <<~SH) + #!/bin/sh + shift + while [ "$#" -gt 0 ]; do + case "$1" in + --table) + printf 'path replica\n/tmp/app.sqlite3 s3\n' + exit 0 + ;; + --json-object) + printf '{"txid":"abc123"}\n' + printf 'time=2026-01-01T00:00:00Z level=INFO msg="restore complete"\n' >&2 + exit 0 + ;; + --json-array) + printf '[{"path":"/tmp/app.sqlite3","replica":"s3"}]\n' + exit 0 + ;; + --json-empty-list) + printf '[]\n' + exit 0 + ;; + --json-empty) + exit 0 + ;; + --skip-db) + printf 'time=2026-01-01T00:00:00Z level=INFO msg="database already exists, skipping"\n' + exit 0 + ;; + --skip-replica) + printf 'time=2026-01-01T00:00:00Z level=INFO msg="no matching backups found"\n' + exit 0 + ;; + --fail) + printf 'Error: database not found in config\n' >&2 + exit 7 + ;; + --echo) + shift + printf '%s\n' "$1" + exit 0 + ;; + --sleep) + shift + printf '%s\n' "$$" > "$1" + sleep 30 + exit 0 + ;; + --sleep-in-child) + shift + (trap '' TERM; printf '%s\n' "$$" > "$1"; sleep 30) & + exit 0 + ;; + esac + shift + done + SH + File.chmod(0o755, @executable) + Litestream.config_path = File.join(@tmpdir, "litestream.yml") + end - Litestream::Commands.stub :run, nil do - Litestream::Commands.snapshots("db/test.sqlite3") + def teardown + if @litestream_install_dir + ENV["LITESTREAM_INSTALL_DIR"] = @litestream_install_dir + else + ENV.delete("LITESTREAM_INSTALL_DIR") end + Litestream::Commands.remove_instance_variable(:@litestream_install_dir_noted) if Litestream::Commands.instance_variable_defined?(:@litestream_install_dir_noted) + Litestream.config_path = nil + FileUtils.remove_entry(@tmpdir) + end - assert_nil ENV["LITESTREAM_REPLICA_BUCKET"] - assert_equal "mykey", ENV["LITESTREAM_ACCESS_KEY_ID"] - assert_nil ENV["LITESTREAM_SECRET_ACCESS_KEY"] + def test_successful_table_output + assert_equal [{"path" => "/tmp/app.sqlite3", "replica" => "s3"}], run_with_fake { Litestream::Commands.databases("--table" => nil) } end - def test_snapshots_sets_replica_access_key_env_var_from_config_when_env_var_not_set - Litestream.replica_access_key = "access" + def test_successful_json_object + assert_equal({"txid" => "abc123"}, run_with_fake { Litestream::Commands.restore("db.sqlite3", json: true, "--json-object": nil) }) + end - Litestream::Commands.stub :run, nil do - Litestream::Commands.snapshots("db/test.sqlite3") - end + def test_successful_json_array + expected = [{"path" => "/tmp/app.sqlite3", "replica" => "s3"}] - assert_nil ENV["LITESTREAM_REPLICA_BUCKET"] - assert_nil ENV["LITESTREAM_ACCESS_KEY_ID"] - assert_equal "access", ENV["LITESTREAM_SECRET_ACCESS_KEY"] + assert_equal expected, run_with_fake { Litestream::Commands.databases(json: true, "--json-array": nil) } end - def test_snapshots_sets_all_env_vars_from_config_when_env_vars_not_set - Litestream.replica_bucket = "mybkt" - Litestream.replica_key_id = "mykey" - Litestream.replica_access_key = "access" - - Litestream::Commands.stub :run, nil do - Litestream::Commands.snapshots("db/test.sqlite3") - end + def test_empty_json_list + assert_equal [], run_with_fake { Litestream::Commands.databases("json" => true, "--json-empty-list" => nil) } + end - assert_equal "mybkt", ENV["LITESTREAM_REPLICA_BUCKET"] - assert_equal "mykey", ENV["LITESTREAM_ACCESS_KEY_ID"] - assert_equal "access", ENV["LITESTREAM_SECRET_ACCESS_KEY"] + def test_empty_json_output + assert_nil run_with_fake { Litestream::Commands.databases("json" => true, "--json-empty" => nil) } end - def test_snapshots_does_not_set_env_var_from_config_when_env_vars_already_set - ENV["LITESTREAM_REPLICA_BUCKET"] = "original_bkt" - ENV["LITESTREAM_ACCESS_KEY_ID"] = "original_key" - ENV["LITESTREAM_SECRET_ACCESS_KEY"] = "original_access" + def test_if_db_not_exists_skip_output + expected = {"skipped" => true, "message" => "database already exists, skipping"} - Litestream.replica_bucket = "mybkt" - Litestream.replica_key_id = "mykey" - Litestream.replica_access_key = "access" + assert_equal expected, run_with_fake { Litestream::Commands.restore("db.sqlite3", json: true, "--skip-db": nil) } + end - Litestream::Commands.stub :run, nil do - Litestream::Commands.snapshots("db/test.sqlite3") - end + def test_if_replica_exists_skip_output + expected = {"skipped" => true, "message" => "no matching backups found"} - assert_equal "original_bkt", ENV["LITESTREAM_REPLICA_BUCKET"] - assert_equal "original_key", ENV["LITESTREAM_ACCESS_KEY_ID"] - assert_equal "original_access", ENV["LITESTREAM_SECRET_ACCESS_KEY"] + assert_equal expected, run_with_fake { Litestream::Commands.restore("db.sqlite3", json: true, "--skip-replica": nil) } end - end - class TestWalCommand < TestCommands - def test_wal_with_no_options - stub = proc do |cmd| - executable, command, *argv = cmd - assert_match Regexp.new("exe/test/litestream"), executable - assert_equal "wal", command - assert_equal 3, argv.size - assert_equal "--config", argv[0] - assert_match Regexp.new("dummy/config/litestream.yml"), argv[1] - assert_equal "db/test.sqlite3", argv[2] - end - Litestream::Commands.stub :run, stub do - Litestream::Commands.wal("db/test.sqlite3") + def test_nonzero_exit_raises_with_status_and_stderr + error = assert_raises(Litestream::Commands::CommandFailedException) do + run_with_fake { Litestream::Commands.databases("--fail" => nil) } end - end - def test_wal_with_boolean_option - stub = proc do |cmd| - executable, command, *argv = cmd - assert_match Regexp.new("exe/test/litestream"), executable - assert_equal "wal", command - assert_equal 4, argv.size - assert_equal "--config", argv[0] - assert_match Regexp.new("dummy/config/litestream.yml"), argv[1] - assert_equal "--if-db-not-exists", argv[2] - assert_equal "db/test.sqlite3", argv[3] - end - Litestream::Commands.stub :run, stub do - Litestream::Commands.wal("db/test.sqlite3", "--if-db-not-exists" => nil) - end + assert_includes error.message, "databases" + assert_includes error.message, "exit status 7" + assert_includes error.message, "Error: database not found in config" end - def test_wal_with_string_option - stub = proc do |cmd| - executable, command, *argv = cmd - assert_match Regexp.new("exe/test/litestream"), executable - assert_equal "wal", command - assert_equal 5, argv.size - assert_equal "--config", argv[0] - assert_match Regexp.new("dummy/config/litestream.yml"), argv[1] - assert_equal "--parallelism", argv[2] - assert_equal 10, argv[3] - assert_equal "db/test.sqlite3", argv[4] - end - Litestream::Commands.stub :run, stub do - Litestream::Commands.wal("db/test.sqlite3", "--parallelism" => 10) - end - end + def test_argument_with_space_is_passed_intact + output = run_with_fake { Litestream::Commands.restore("db.sqlite3", "--echo" => "argument with space") } - def test_wal_with_config_option - stub = proc do |cmd| - executable, command, *argv = cmd - assert_match Regexp.new("exe/test/litestream"), executable - assert_equal "wal", command - assert_equal 3, argv.size - assert_equal "--config", argv[0] - assert_equal "CONFIG", argv[1] - assert_equal "db/test.sqlite3", argv[2] - end - Litestream::Commands.stub :run, stub do - Litestream::Commands.wal("db/test.sqlite3", "--config" => "CONFIG") - end + assert_equal "argument with space\n", output end - def test_wal_sets_replica_bucket_env_var_from_config_when_env_var_not_set - Litestream.replica_bucket = "mybkt" + def test_timeout_kills_and_reaps_child + pid_file = File.join(@tmpdir, "pid") - Litestream::Commands.stub :run, nil do - Litestream::Commands.wal("db/test.sqlite3") + assert_raises(Litestream::Commands::CommandTimeoutException) do + run_with_fake { Litestream::Commands.databases(**{"timeout" => 0.1, "--sleep" => pid_file}) } end - assert_equal "mybkt", ENV["LITESTREAM_REPLICA_BUCKET"] - assert_nil ENV["LITESTREAM_ACCESS_KEY_ID"] - assert_nil ENV["LITESTREAM_SECRET_ACCESS_KEY"] + pid = wait_for_pid(pid_file) + assert_raises(Errno::ECHILD) { Process.wait(pid, Process::WNOHANG) } end - def test_wal_sets_replica_key_id_env_var_from_config_when_env_var_not_set - Litestream.replica_key_id = "mykey" + def test_timeout_kills_a_descendant_that_outlives_the_direct_child + pid_file = File.join(@tmpdir, "pid") - Litestream::Commands.stub :run, nil do - Litestream::Commands.wal("db/test.sqlite3") + assert_raises(Litestream::Commands::CommandTimeoutException) do + run_with_fake { Litestream::Commands.databases(**{"timeout" => 0.1, "--sleep-in-child" => pid_file}) } end - assert_nil ENV["LITESTREAM_REPLICA_BUCKET"] - assert_equal "mykey", ENV["LITESTREAM_ACCESS_KEY_ID"] - assert_nil ENV["LITESTREAM_SECRET_ACCESS_KEY"] + pid = wait_for_pid(pid_file) + sleep 0.05 + assert_raises(Errno::ESRCH) { Process.kill(0, pid) } end - def test_wal_sets_replica_access_key_env_var_from_config_when_env_var_not_set - Litestream.replica_access_key = "access" - - Litestream::Commands.stub :run, nil do - Litestream::Commands.wal("db/test.sqlite3") - end + def test_integer_option_values_are_passed_as_strings + output = run_with_fake { Litestream::Commands.restore("/tmp/app.sqlite3", "--echo" => 10) } - assert_nil ENV["LITESTREAM_REPLICA_BUCKET"] - assert_nil ENV["LITESTREAM_ACCESS_KEY_ID"] - assert_equal "access", ENV["LITESTREAM_SECRET_ACCESS_KEY"] + assert_equal "10\n", output end - def test_wal_sets_all_env_vars_from_config_when_env_vars_not_set - Litestream.replica_bucket = "mybkt" - Litestream.replica_key_id = "mykey" - Litestream.replica_access_key = "access" + def test_install_dir_note_is_printed_once + ENV["LITESTREAM_INSTALL_DIR"] = @tmpdir - Litestream::Commands.stub :run, nil do - Litestream::Commands.wal("db/test.sqlite3") + _stdout, stderr = capture_io do + Litestream::Commands.executable + Litestream::Commands.executable end - assert_equal "mybkt", ENV["LITESTREAM_REPLICA_BUCKET"] - assert_equal "mykey", ENV["LITESTREAM_ACCESS_KEY_ID"] - assert_equal "access", ENV["LITESTREAM_SECRET_ACCESS_KEY"] + assert_equal 1, stderr.scan("NOTE: using LITESTREAM_INSTALL_DIR").size end - def test_wal_does_not_set_env_var_from_config_when_env_vars_already_set - ENV["LITESTREAM_REPLICA_BUCKET"] = "original_bkt" - ENV["LITESTREAM_ACCESS_KEY_ID"] = "original_key" - ENV["LITESTREAM_SECRET_ACCESS_KEY"] = "original_access" + private - Litestream.replica_bucket = "mybkt" - Litestream.replica_key_id = "mykey" - Litestream.replica_access_key = "access" + def run_with_fake(&block) + Litestream::Commands.stub(:executable, @executable, &block) + end - Litestream::Commands.stub :run, nil do - Litestream::Commands.wal("db/test.sqlite3") + def wait_for_pid(pid_file) + deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + 5 + until File.exist?(pid_file) && !File.read(pid_file).strip.empty? + flunk "fake litestream never wrote its pid" if Process.clock_gettime(Process::CLOCK_MONOTONIC) > deadline + sleep 0.01 end - - assert_equal "original_bkt", ENV["LITESTREAM_REPLICA_BUCKET"] - assert_equal "original_key", ENV["LITESTREAM_ACCESS_KEY_ID"] - assert_equal "original_access", ENV["LITESTREAM_SECRET_ACCESS_KEY"] + pid = File.read(pid_file).to_i + assert_operator pid, :>, 0 + pid end end class TestOutput < ActiveSupport::TestCase def test_output_formatting_generates_table_with_data data = [ - {path: "/storage/database.db", replicas: "s3"}, - {path: "/storage/another-database.db", replicas: "s3"} + {path: "/storage/database.db", replica: "s3"}, + {path: "/storage/another-database.db", replica: "s3"} ] result = Litestream::Commands::Output.format(data) @@ -853,23 +751,23 @@ def test_output_formatting_generates_table_with_data assert_equal 3, lines.length assert_includes lines[0], "path" - assert_includes lines[0], "replicas" + assert_includes lines[0], "replica" assert_includes lines[1], "/storage/database.db" assert_includes lines[2], "/storage/another-database.db" end def test_output_formatting_generates_formatted_table data = [ - {path: "/storage/database.db", replicas: "s3"}, - {path: "/storage/another-database.db", replicas: "s3"} + {path: "/storage/database.db", replica: "s3"}, + {path: "/storage/another-database.db", replica: "s3"} ] result = Litestream::Commands::Output.format(data) lines = result.split("\n") - replicas_pos = lines[0].index("replicas") - assert_equal replicas_pos, lines[1].index("s3") - assert_equal replicas_pos, lines[2].index("s3") + replica_pos = lines[0].index("replica") + assert_equal replica_pos, lines[1].index("s3") + assert_equal replica_pos, lines[2].index("s3") end end end diff --git a/test/litestream/test_upstream.rb b/test/litestream/test_upstream.rb new file mode 100644 index 0000000..062aefa --- /dev/null +++ b/test/litestream/test_upstream.rb @@ -0,0 +1,16 @@ +require "test_helper" + +class TestUpstream < ActiveSupport::TestCase + def test_every_native_platform_has_a_checksum + Litestream::Upstream::NATIVE_PLATFORMS.each_value do |filename| + assert Litestream::Upstream::CHECKSUMS.key?(filename), "missing checksum for #{filename}" + end + end + + def test_release_urls_use_version_tag_and_unprefixed_filenames + Litestream::Upstream::NATIVE_PLATFORMS.each_value do |filename| + assert_equal "https://github.com/benbjohnson/litestream/releases/download/v0.5.17/#{filename}", Litestream::Upstream.download_url(filename) + refute filename.start_with?("litestream-v") + end + end +end diff --git a/test/tasks/test_litestream_tasks.rb b/test/tasks/test_litestream_tasks.rb index 344c3ec..79dfd31 100644 --- a/test/tasks/test_litestream_tasks.rb +++ b/test/tasks/test_litestream_tasks.rb @@ -9,6 +9,8 @@ def setup Rake::Task["litestream:replicate"].reenable Rake::Task["litestream:restore"].reenable Rake::Task["litestream:databases"].reenable + Rake::Task["litestream:ltx"].reenable + Rake::Task["litestream:status"].reenable Rake::Task["litestream:generations"].reenable Rake::Task["litestream:snapshots"].reenable Rake::Task["litestream:wal"].reenable @@ -138,129 +140,102 @@ def test_databases_task_with_arguments_without_separator end end - class TestGenerationsTask < TestLitestreamTasks - def test_generations_task_with_only_database_using_single_dash + class TestLtxTask < TestLitestreamTasks + def test_ltx_task_with_only_database_using_single_dash ARGV.replace ["--", "-database=db/test.sqlite3"] fake = Minitest::Mock.new fake.expect :call, nil, ["db/test.sqlite3"] - Litestream::Commands.stub :generations, fake do - Rake.application.invoke_task "litestream:generations" + Litestream::Commands.stub :ltx, fake do + Rake.application.invoke_task "litestream:ltx" end fake.verify end - def test_generations_task_with_only_database_using_double_dash + def test_ltx_task_with_only_database_using_double_dash ARGV.replace ["--", "--database=db/test.sqlite3"] fake = Minitest::Mock.new fake.expect :call, nil, ["db/test.sqlite3"] - Litestream::Commands.stub :generations, fake do - Rake.application.invoke_task "litestream:generations" + Litestream::Commands.stub :ltx, fake do + Rake.application.invoke_task "litestream:ltx" end fake.verify end - def test_generations_task_with_arguments - ARGV.replace ["--", "-database=db/test.sqlite3", "--if-db-not-exists"] + def test_ltx_task_with_arguments + ARGV.replace ["--", "-database=db/test.sqlite3", "--level=all"] fake = Minitest::Mock.new - fake.expect :call, nil, ["db/test.sqlite3"], "--if-db-not-exists": nil - Litestream::Commands.stub :generations, fake do - Rake.application.invoke_task "litestream:generations" + fake.expect :call, nil, ["db/test.sqlite3"], "--level": "all" + Litestream::Commands.stub :ltx, fake do + Rake.application.invoke_task "litestream:ltx" end fake.verify end - def test_generations_task_with_arguments_without_separator + def test_ltx_task_with_arguments_without_separator ARGV.replace ["-database=db/test.sqlite3"] fake = Minitest::Mock.new fake.expect :call, nil, [nil] - Litestream::Commands.stub :generations, fake do - Rake.application.invoke_task "litestream:generations" + Litestream::Commands.stub :ltx, fake do + Rake.application.invoke_task "litestream:ltx" end fake.verify end end - class TestSnapshotsTask < TestLitestreamTasks - def test_snapshots_task_with_only_database_using_single_dash + class TestStatusTask < TestLitestreamTasks + def test_status_task_with_database ARGV.replace ["--", "-database=db/test.sqlite3"] fake = Minitest::Mock.new fake.expect :call, nil, ["db/test.sqlite3"] - Litestream::Commands.stub :snapshots, fake do - Rake.application.invoke_task "litestream:snapshots" + Litestream::Commands.stub :status, fake do + Rake.application.invoke_task "litestream:status" end fake.verify end - def test_snapshots_task_with_only_database_using_double_dash - ARGV.replace ["--", "--database=db/test.sqlite3"] + def test_status_task_without_database + ARGV.replace ["--"] fake = Minitest::Mock.new - fake.expect :call, nil, ["db/test.sqlite3"] - Litestream::Commands.stub :snapshots, fake do - Rake.application.invoke_task "litestream:snapshots" + fake.expect :call, nil, [nil] + Litestream::Commands.stub :status, fake do + Rake.application.invoke_task "litestream:status" end fake.verify end - def test_snapshots_task_with_arguments - ARGV.replace ["--", "-database=db/test.sqlite3", "--if-db-not-exists"] + def test_status_task_with_arguments + ARGV.replace ["--", "--database=db/test.sqlite3", "--json"] fake = Minitest::Mock.new - fake.expect :call, nil, ["db/test.sqlite3"], "--if-db-not-exists": nil - Litestream::Commands.stub :snapshots, fake do - Rake.application.invoke_task "litestream:snapshots" + fake.expect :call, nil, ["db/test.sqlite3"], "--json": nil + Litestream::Commands.stub :status, fake do + Rake.application.invoke_task "litestream:status" end fake.verify end - def test_snapshots_task_with_arguments_without_separator - ARGV.replace ["-database=db/test.sqlite3"] + def test_status_task_with_arguments_without_separator + ARGV.replace ["--json"] fake = Minitest::Mock.new fake.expect :call, nil, [nil] - Litestream::Commands.stub :snapshots, fake do - Rake.application.invoke_task "litestream:snapshots" + Litestream::Commands.stub :status, fake do + Rake.application.invoke_task "litestream:status" end fake.verify end end - class TestWalTask < TestLitestreamTasks - def test_wal_task_with_only_database_using_single_dash - ARGV.replace ["--", "-database=db/test.sqlite3"] - fake = Minitest::Mock.new - fake.expect :call, nil, ["db/test.sqlite3"] - Litestream::Commands.stub :wal, fake do - Rake.application.invoke_task "litestream:wal" - end - fake.verify - end - - def test_wal_task_with_only_database_using_double_dash - ARGV.replace ["--", "--database=db/test.sqlite3"] - fake = Minitest::Mock.new - fake.expect :call, nil, ["db/test.sqlite3"] - Litestream::Commands.stub :wal, fake do - Rake.application.invoke_task "litestream:wal" - end - fake.verify - end - - def test_wal_task_with_arguments - ARGV.replace ["--", "-database=db/test.sqlite3", "--if-db-not-exists"] - fake = Minitest::Mock.new - fake.expect :call, nil, ["db/test.sqlite3"], "--if-db-not-exists": nil - Litestream::Commands.stub :wal, fake do - Rake.application.invoke_task "litestream:wal" - end - fake.verify - end + class TestRemovedTasks < TestLitestreamTasks + %w[generations snapshots wal].each do |command| + define_method(:"test_#{command}_task_aborts_with_migration_message") do + _out, err = capture_io do + error = assert_raises(SystemExit) do + Rake.application.invoke_task "litestream:#{command}" + end + assert_equal 1, error.status + end - def test_wal_task_with_arguments_without_separator - ARGV.replace ["-database=db/test.sqlite3"] - fake = Minitest::Mock.new - fake.expect :call, nil, [nil] - Litestream::Commands.stub :wal, fake do - Rake.application.invoke_task "litestream:wal" + assert_equal "`#{command}` was removed in Litestream 0.5; use `ltx` (see README, Upgrading from 0.3)\n", err end - fake.verify end end end diff --git a/test/test_litestream.rb b/test/test_litestream.rb index 5c1f41d..991ad6c 100644 --- a/test/test_litestream.rb +++ b/test/test_litestream.rb @@ -85,4 +85,55 @@ def test_replicate_process_ps assert_equal info[:started].class, DateTime end end + + def test_databases_loads_status_and_ltx_and_isolates_command_failures + database_path = Rails.root.join("storage/test.sqlite3").to_s + failing_path = Rails.root.join("storage/failing.sqlite3").to_s + databases = [ + {"path" => database_path, "replica" => "file"}, + {"path" => failing_path, "replica" => "file"} + ] + status = [{"database" => database_path, "status" => "ok", "local_txid" => "000000000000000a", "wal_size" => 131_072}] + ltx = [{"level" => 0, "min_txid" => "0000000000000008", "max_txid" => "000000000000000a", "size" => 1013, "timestamp" => "2026-09-08T03:16:43Z"}] + databases_stub = proc do |**options| + assert_equal({json: true}, options) + databases.map(&:dup) + end + status_stub = proc do |path, **options| + assert_equal({json: true}, options) + raise Litestream::Commands::CommandFailedException, "status unavailable" if path == failing_path + + status + end + ltx_stub = proc do |path, **options| + assert_equal database_path, path + assert_equal({:json => true, "--level" => "all"}, options) + ltx + end + + Litestream::Commands.stub :databases, databases_stub do + Litestream::Commands.stub :status, status_stub do + Litestream::Commands.stub :ltx, ltx_stub do + result = Litestream.databases + + assert_equal "[ROOT]/storage/test.sqlite3", result[0]["path"] + assert_equal status.first, result[0]["status"] + assert_equal ltx, result[0]["ltx"] + assert_equal "status unavailable", result[1]["error"] + assert_equal "[ROOT]/storage/failing.sqlite3", result[1]["path"] + end + end + end + end + + def test_databases_propagates_non_command_errors + databases = [{"path" => Rails.root.join("storage/test.sqlite3").to_s, "replica" => "file"}] + + Litestream::Commands.stub :databases, databases do + Litestream::Commands.stub :status, proc { raise ArgumentError, "unexpected data" } do + error = assert_raises(ArgumentError) { Litestream.databases } + assert_equal "unexpected data", error.message + end + end + end end