It is clear we will need a tool / protocol on top of IPFS to coordinate IPFS nodes together. This issue will track design goals, constraints, proposals, and the progress.

What to Coordinate

Things worth coordinating between IPFS nodes:

• collaborative pin sets -- back up large pin sets together, to achieve redundancy and capacity constraints (including RAID-style modes).
• authentication graphs -- trust models, like PKIs or hierarchical auth of control.
• bitswap guilds -- the ability to band together into efficient data trade networks
• application servers -- afford redundancy guarantees to hosted protocols / APIs

and more (expand this!)

Consensus

Many of these require consensus, and thus we'll likely bundle a simple (read: FAST!) consensus protocol with ipfs-cluster. this could be RAFT (etcd) or Paxos, and does not require byzantine consensus. Though having byzantine consensus would be useful for massive untrusted clusters-- though this approaches Filecoin and is a very different use case altogether.

cluster == a virtualized ipfs node

One goal is to represent a virtualized IPFS node sharded across other nodes. This makes for a very nice modular architecture where one can plug ipfs nodes into clusters, and clusters into larger clusters (hierarchies). This makes cluster a bit harder to design, but much, much more useful. Backing up of massive data (like all of archive.org or all of wikimedia, or all scientific data ever produced) would thus become orders of magnitude simpler to reason about.

The general idea here is to make ipfs-cluster provide an API that matches the standard ipfs node API, (i.e. with an identity, being able to be connected to, and providing the ipfs core methods).