Backup solutions
Public cloud
Need to have an off-site backup solution as well, and cloud storage is convenient.
Some options, with prices in USD:
Platform | Storage ($/GB/mth) | Request ($/1000) | Retrieval ($/GB) | Transfer ($/GB) | Remarks |
---|---|---|---|---|---|
Amazon S3 Glacier Deep Archive | .002 | .06 (PUT) / .0004 (GET) / .03 (GET:Retrieve) | .005 | .12 | FAQ. Each object incurs an additional 32 KB metadata in S3 Glacier and 8KB in S3 Standard. |
Amazon S3 Standard | .025 | .005 (PUT) / .0004 (GET) | - | .12 | Prorated, need to perform COPY to S3 Standard bucket before data egress. |
Google Cloud Filestore | .20 | free (<2M), .003 (<1B, GET/PUT/LIST) | - | .12 | API Gateway. Internet egress rates. |
Microsoft Azure Blob Storage Archive | .00099 | .013 (PUT) / .65 (GET/LIST) | .02 | free (<5G), .08 (<10T) | Rehydration. Internet egress. |
Backblaze B2 | .005 | free (PUT) / .0004 (GET) / .004 (LIST) | - | .01 | Contract architecture requires repeated dispatch requests. |
SpiderOak One Backup | .0052 (annual) | - | - | - | Curious why this backend not supported by rclone. |
Some tips for zero-knowledge backups.
Private backend tools
"May be more cost-effective to deploy hardware at another physical location - to check electricity usage."
rclone
Link: https://rclone.org/
Strengths:
- Supports
crypt
which acts as an additional encryption layer to sync to remotes. According to Hacker News, seems trustworthy and reliable. Will be useful for data storage on untrusted environments. - Supports many cloud, local and remote backends.
Downloading rclone
is quite straightforward, using curl https://rclone.org/install.sh | sudo bash
which checks for the current version of rclone
. Note that the apt repositories for rclone are currently stuck at version 1.50.2-2.
Need to first configure remotes with rclone config
, during which the sync backend can be specified. Gave a hand at sftp
.
Attempted to perform a backup of the Dokuwiki directory, yielding a SFTP failure. See error messages below:
$ rclone sync -P /dokuwiki/ nas:/dokuwiki/ ... 2021-08-09 22:33:51 ERROR : data/cache/a/a41386c9523185547b948bab29f8724f.repo: Failed to copy: Update ReadFrom failed: sftp: "Failure" (SSH_FX_FAILURE) 2021-08-09 22:33:51 ERROR : data/cache/a/a4146da96d57160590dfd1faae419088.media.600x231.png: Failed to copy: Update ReadFrom failed: sftp: "Failure" (SSH_FX_FAILURE) 2021-08-09 22:33:51 ERROR : data/cache/a/a449e6e40ea628ef213ea69daa88689e.media.90x67.png: Failed to copy: Update ReadFrom failed: sftp: "Failure" (SSH_FX_FAILURE) 2021-08-09 22:33:51 ERROR : data/cache/a/a45e71078c542b6f00dd1b05d4741d48.repo: Failed to copy: Update ReadFrom failed: sftp: "Failure" (SSH_FX_FAILURE) Transferred: 29.820Mi / 88.080 MiByte, 34%, 6.867 MiByte/s, ETA 8s Errors: 735 (retrying may help) Checks: 3925 / 3925, 100% Transferred: 1 / 6088, 0% Elapsed time: 4.0s Transferring: * data/cache/a/a128707d9…dia.4032x2240.crop.jpg:100% /745.498Ki, 0/s, - * data/cache/a/a184ab518…a0e981e0d8fb0.metadata: 0% /10, 0/s, - * data/cache/a/a4146da96…dia.2867x1083.crop.png:100% /201.676Ki, 0/s, - * data/cache/a/a45e71078…f00dd1b05d4741d48.repo:100% /1.497Ki, 0/s, -^C
Since SFTP transfer is the primary go-to for low cost backup (see the full list of available backends), rclone
is currently not effective at the moment.
Some drawbacks of rclone
include: (1) no snapshot-based backup. Note this idiomatic command.
rsync
Strengths:
- Fundamentally simple at its core: rsync algorithm written in Python.
- Designed to work over shaky network connections, as opposed to regular
scp
. Supports transfer resumption. - Often recommended over
scp
, and has a verification check, see SO for a comparison.
Possible strategies:
rysnc -avhlAXEWP --itemize-changes --stats --info=progress2 /source/ /destination
rsnapshot
Kopia
Borgmatic
restic
Link: https://restic.net/ / https://github.com/restic/restic
Main drawback seems to be its inability to safely handle very large datasets (GB~PB), as well as memory usage and file corruption.
Strengths:
tar + netcat/scp
Incremental dumps using tar
: https://www.gnu.org/software/tar/manual/html_node/Incremental-Dumps.html
Duplicity
Looks like Duplicity doesn't support backups from remote: https://serverfault.com/questions/543039/backup-a-remote-directory-with-duplicity
duplicity /volume1/local/test scp://backup//volume1/remote/test
Others
- Encrypted backup strategy...? Delta backup of encrypted data
- Alternative workflows? others
Wrestling with Kopia for a while now... The full keyfile and known hosts need to be supplied via command line (why?), and in the end the following error appeared: ERROR can't connect to storage: unable to open SFTP storage: error establishing connecting: unable to create sftp client: ssh: subsystem request failed