Need to have an off-site backup solution as well, and cloud storage is convenient.
Some options, with prices in USD:
| Platform | Storage ($/GB/mth) | Request ($/1000) | Retrieval ($/GB) | Transfer ($/GB) | Remarks |
|---|---|---|---|---|---|
| Amazon S3 Glacier Deep Archive | .002 | .06 (PUT) / .0004 (GET) / .03 (GET:Retrieve) | .005 | .12 | FAQ. Each object incurs an additional 32 KB metadata in S3 Glacier and 8KB in S3 Standard. |
| Amazon S3 Standard | .025 | .005 (PUT) / .0004 (GET) | - | .12 | Prorated, need to perform COPY to S3 Standard bucket before data egress. |
| Google Cloud Filestore | .20 | free (<2M), .003 (<1B, GET/PUT/LIST) | - | .12 | API Gateway. Internet egress rates. |
| Microsoft Azure Blob Storage Archive | .00099 | .013 (PUT) / .65 (GET/LIST) | .02 | free (<5G), .08 (<10T) | Rehydration. Internet egress. |
| Backblaze B2 | .005 | free (PUT) / .0004 (GET) / .004 (LIST) | - | .01 | Contract architecture requires repeated dispatch requests. |
| SpiderOak One Backup | .0052 (annual) | - | - | - | Curious why this backend not supported by rclone. |
Some tips for zero-knowledge backups.
"May be more cost-effective to deploy hardware at another physical location - to check electricity usage."
Link: https://rclone.org/
Strengths:
crypt which acts as an additional encryption layer to sync to remotes. According to Hacker News, seems trustworthy and reliable. Will be useful for data storage on untrusted environments.
Downloading rclone is quite straightforward, using curl https://rclone.org/install.sh | sudo bash which checks for the current version of rclone. Note that the apt repositories for rclone are currently stuck at version 1.50.2-2.
Need to first configure remotes with rclone config, during which the sync backend can be specified. Gave a hand at sftp.
Attempted to perform a backup of the Dokuwiki directory, yielding a SFTP failure. See error messages below:
$ rclone sync -P /dokuwiki/ nas:/dokuwiki/ ... 2021-08-09 22:33:51 ERROR : data/cache/a/a41386c9523185547b948bab29f8724f.repo: Failed to copy: Update ReadFrom failed: sftp: "Failure" (SSH_FX_FAILURE) 2021-08-09 22:33:51 ERROR : data/cache/a/a4146da96d57160590dfd1faae419088.media.600x231.png: Failed to copy: Update ReadFrom failed: sftp: "Failure" (SSH_FX_FAILURE) 2021-08-09 22:33:51 ERROR : data/cache/a/a449e6e40ea628ef213ea69daa88689e.media.90x67.png: Failed to copy: Update ReadFrom failed: sftp: "Failure" (SSH_FX_FAILURE) 2021-08-09 22:33:51 ERROR : data/cache/a/a45e71078c542b6f00dd1b05d4741d48.repo: Failed to copy: Update ReadFrom failed: sftp: "Failure" (SSH_FX_FAILURE) Transferred: 29.820Mi / 88.080 MiByte, 34%, 6.867 MiByte/s, ETA 8s Errors: 735 (retrying may help) Checks: 3925 / 3925, 100% Transferred: 1 / 6088, 0% Elapsed time: 4.0s Transferring: * data/cache/a/a128707d9…dia.4032x2240.crop.jpg:100% /745.498Ki, 0/s, - * data/cache/a/a184ab518…a0e981e0d8fb0.metadata: 0% /10, 0/s, - * data/cache/a/a4146da96…dia.2867x1083.crop.png:100% /201.676Ki, 0/s, - * data/cache/a/a45e71078…f00dd1b05d4741d48.repo:100% /1.497Ki, 0/s, -^C
Since SFTP transfer is the primary go-to for low cost backup (see the full list of available backends), rclone is currently not effective at the moment.
Some drawbacks of rclone include: (1) no snapshot-based backup. Note this idiomatic command.
Strengths:
scp. Supports transfer resumption.scp, and has a verification check, see SO for a comparison.Possible strategies:
rysnc -avhlAXEWP --itemize-changes --stats --info=progress2 /source/ /destinationLink: https://restic.net/ / https://github.com/restic/restic
Main drawback seems to be its inability to safely handle very large datasets (GB~PB), as well as memory usage and file corruption.
Strengths:
Incremental dumps using tar: https://www.gnu.org/software/tar/manual/html_node/Incremental-Dumps.html
Looks like Duplicity doesn't support backups from remote: https://serverfault.com/questions/543039/backup-a-remote-directory-with-duplicity
duplicity /volume1/local/test scp://backup//volume1/remote/test