S3 has become so prevalent as a data storage service that its API has become something of a de facto standard for data storage. While Amazon promotes Cloud Drive as the consumer-focused product based on S3, S3 itself is quite accessible to end users through various clients. Many of those clients are proprietary, but there are two that I use that are free and open source.
The first is part of the s3tools suite, and is called
s3cmd. Given your access key and secret key,
s3cmd gives you full access to the S3 service, allow creation, modification and deletion of S3 buckets. It even supports an interface for syncing entire directories to S3 in one command.
The other tool, duplicity, is frequently used as the basis for encrypted, incremental backup. One of its features is that it supports S3 urls for the destination of the backup. A single duplicity invocation will compress, encrypt and backup any directory you specify to S3.
In addition to supporting S3, a bit of tuning enables these tools to support DreamHost’s DreamObjects service as well, which provides an S3-compatible interface, but is backed by the open-source Ceph object store/block storage/file system, rather than Amazon’s proprietary store.
In the case of
s3cmd, simply tweak a new
.s3cfg file by:
- Adding your DreamObjects
secret_keyto the file in the appropriate spots, and
- Replacing instaces of the host
You can then invoke
s3cmd as usual (assuming your
.s3cfg is located in your home directory).
If you would like to use both S3 and Dreamhost objects, you can rename your
.s3cfg setup for Dreamhost to something like
.dhcfg and then alias
alias dh='s3cmd -c ~/.dhcfg'
s3cmd as usual accesses your S3 account, and using
dh accesses your DreamObjects account.
In the case of duplicity, the customization is even easier. Assuming your DreamHost access key and secret key are in the
AWS_SECRET_ACCESS_KEY variables, respectively, you can invoke duplicity like so:
duplicity --allow-source-mismatch /path/to/stuff/to/backup s3://objects.dreamhost.com//<bucket-name>/backup-directory