I've found a very nice distributed storage solution: Wuala. This is exactly what I've been wanting in a number of respects. You trade storage on your local filesystem for storage on the cloud. Everything is encrypted. It's written in Java. It comes with a nice user interface but I've set it up to run as a daemon on Linux. I access the files through an NFS share. One drawback is that you can copy a file to a shared folder and, if the file is particularly large, you won't see it right away. The file gets uploaded to the cloud in the background and you don't see it in the shared folder until it's fully uploaded. That makes Wuala better suited for backups than for interactive applications unless I can build a caching app on top of it. They have a REST interface but say that it's still in early development.

I still have some questions but I haven't gone through all the FAQs and documentation yet.

They says it's based on open-source but it doesn't appear to be entirely open source itself. The storage scheme is like a file system which is ok for some applications but it would be difficult to build an interactive application on top of it. Amazon's S3 is better for that.

Now if S3 would allow trading storage and encrypt or if Wuala would allow for a more object-oriented schema and synchronous transfers, all would be perfect.