ODIN: Secure P2P on the Web
Posted by code_monkey_steve, Mon Sep 25 09:11:00 UTC 2006
A couple weeks back, I ran across OFF, the Ownerless File Format, and was hooked pretty much immediately—the idea of “ownerless” content really piqued my interest. When I went to download and build the OFF client, however, I discovered that the *nix build was pretty much non-existant, and rather than spend the trouble porting a C++ code base to Linux, I decided that this would make for an excellent Rails web app. Thus was born the idea for the Ownerless Data Interchange Network (ODIN).
The first thing you need to understand for this to make sense is that the magic of OFF is all in the block encoding which turns meaningful bits into ownerless, quasi-random blocks identified by their SHA1 hashes. Once this transformation is performed, there is no way for anyone to know the content of a block on the network, as it has no meaning until it’s combined with other blocks (and, in that way, may actually “belong” to multiple files, simultaneously).
The thing that first screamed “Rails app!” to me is the fact that, once the content has been OFF-encoded into blocks, any transport mechanism can be used to move these blocks around, allowing for a network-agnostic peer-to-peer system. Using HTTP, I could write my own smart, caching web peer (with combined web UI) in Rails, but since it’s using HTTP, I could also use dumb web server file servers as mass block stores (no CGI required!). Eventually, I could add (trivial) support to talk to the existing OFF network (also HTTP, AFAIK). If I watned to get really crazy, I could make the app a 3-way network bridge with BitTorrent (which also uses SHA1-identified data blocks).
This is an extension of earlier musings about extending HTTP (or coming up with some other standard) for enumerating mirrors of a particular web resource, something like a per-directory MIRROR file (in XML). A local web-proxy/daemon would cache the status, speed, and locality of various mirrors, and attempt to provide the best location for a particular resource (Use Case #1: Linux RPM repositories). Throw in resource discovery, and you have a basic peer-to-peer network, all over HTTP. While this project assumes that each peer is serving OFF-encoded blocks, there’s no reason why this assumption couldn’t be refactored-out, leaving a generic HTTP P2P system.
To date I’ve already completed most of the functionality of ODIN: fast block encoding (more on that later), file insertion and extraction, basic file transfer and peer discovery. I’ll post more when I have something to show-off. ;)