Using TarWriter to stream backup

Dev
,
1

While doing research, I found that this class is already available in Discourse via require 'rubygems/package': http://ruby-doc.org/stdlib-2.0.0/libdoc/rubygems/rdoc/Gem/Package/TarWriter.html

Using this should allow Discourse to take backups without having more than double the required disk space available, by streaming the entire archive to disk through an in-process tar and an in-process gzip.

Usage should look like the following:

destination = File.open(target_filename, "wb") gz_stream = Zlib::GzipWriter.new(destination, 5) @tar_writer = Gem::Package::TarWriter.new(gz_stream) log "Archiving data dump..." FileUtils.cd(File.dirname(@dump_filename)) do @tar_writer.add_file "dump.sql.gz", 0644 do |tf| File.open(@dump_filename) do |df| IO.copy_stream(df, tf) end end end rel_directory = File.join(Rails.root, "public") upload_directory = File.join(rel_directory, "uploads", @current_db) log "Archiving uploads..." last_progress = Time.now files_since_progress = 0 Dir[File.join(upload_directory, "**/*")].each do |file| stat = File.stat(file) relative = file.delete_prefix(rel_directory) if stat.directory? @tar_writer.mkdir relative, stat.mode else files_since_progress += 1 if files_since_progress > 100 or (last_progress < 15.seconds.ago) log "Archiving #{file}" files_since_progress = 0 last_progress = Time.now end @tar_writer.add_file relative, stat.mode do |tf| File.open(file, "rb") { |df| IO.copy_stream(df, tf) } end end end log "Finishing up archive..." @tar_writer.close gz_stream.close destination.close remove_tmp_directory

The above code does not have:

  • proper error reporting
  • progress indicators
5 Likes
2

Not sure, I thought @sam already had some ideas on this?

3

I am open to optimising this stuff, but we have to be careful around compatibility.

As a general approach I would like us

backup db as sql > stream to gzip > stream to tar
backup files > stream to cheapest gzip > stream to same tar

This can be done by shelling out or by using a Ruby gem

4 Likes
4

I think that any solution relying on tar --append is going to be wonky and prone to extra copies.

I should mention that this came up when I was thinking about a “more fully-featured user data checkout”.

The user_visits and topic_timings tables should absolutely be part of the user data checkout (see: new EU regulations, “request access to [the personal data]”), but that means writing a zip file with multiple files in it, not just a single csv. So I thought about, “how does the backup system do this?”

3 Likes