[MLUG] s3backer

[Jeremy]

Just wanted to give you a heads up about this cool program I have been 
playing with. It makes a virtual disk that is hosted on amazon s3.  
http://code.google.com/p/s3backer/

I'll just paste their description:

"*s3backer* is a filesystem that contains a single file backed by the 
Amazon Simple Storage Service <http://aws.amazon.com/s3> (Amazon S3). As 
a filesystem, it is very simple: it provides a single normal file having 
a fixed size. Underneath, the file is divided up into blocks, and the 
content of each block is stored in a unique Amazon S3 object. In other 
words, what *s3backer* provides is really more like an S3-backed virtual 
hard disk device, rather than a filesystem.

In typical usage, a normal filesystem is mounted on top of the file 
exported by the *s3backer* filesystem using a loopback mount (or disk 
image mount on Mac OS X).

This arrangement has several benefits compared to more complete S3 
filesystem implementations:

    * By not attempting to implement a complete filesystem, which is a
      complex undertaking and difficult to get right, *s3backer* can
      stay very lightweight and simple. Only three HTTP operations are
      used: GET, PUT, and DELETE. All of the experience and knowledge
      about how to properly implement filesystems that already exists
      can be reused.

    * By utilizing existing filesystems, you get full UNIX filesystem
      semantics. Subtle bugs or missing functionality relating to hard
      links, extended attributes, POSIX locking, etc. are avoided.

    * The gap between normal filesystem semantics and Amazon S3
      ``eventual consistency'' is more easily and simply solved when one
      can interpret S3 objects as simple device blocks rather than
      filesystem objects (see below).

    * When storing your data on Amazon S3 servers, which are not under
      your control, the ability to encrypt data becomes a critical
      issue. *s3backer* supports secure encryption and authentication.
      Alternately, the encryption capability built into the Linux
      loopback device can be used.

    * Since S3 data is accessed over the network, local caching is also
      very important for performance reasons. Since *s3backer* presents
      the equivalent of a virtual hard disk to the kernel, most of the
      filesystem caching can be done where it should be: in the kernel,
      via the kernel's page cache. However *s3backer* also includes its
      own internal block cache for increased performance, using
      asynchronous worker threads to take advantage of the parallelism
      inherent in the network."

Basically you get a network hosted disk that is mounted locally through 
loopback. It works great (I am hosting videos that are played straight 
off the drive to the web), is relatively simple to set up and use, and 
of course it is cloud-hosted so you can use it on more than one system. 
The cost of S3 is very reasonable, it is a compelling solution for 
backups, web storage, etc.

Jeremy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/mlug-listserv.mlug.ca/attachments/20111015/5222ec4c/attachment.htm>