On Wed, Jun 04, 2014 at 11:21:33AM +0200, u-codalist-rcma_at_aetey.se wrote:
> First of all: I do _not_ suggest rewriting Coda from scratch :)

Actually I would love a rewrite of (parts of) Coda from scratch.

But replication wouldn't be a priority on my list. If you look at
Coda's development history replication was added first to improve
resilience to failing servers, and disconnected operation was added
later.

- Once you have client caching and disconnected operation in your client
  you can actually handle temporary server failures much easier.

- The current replication doesn't really help much for scalability
  because clients still have to get attributes from all servers to
  detect conflicts, and when a file changes all servers have to break
  callbacks with all clients that have ever looked at the file.

- Replication isn't helping the wide-area-network case either. Yes
  clients will try to fetch file data from the 'quickest to respond'
  server, but attributes still have to come from all servers and
  server-server resolution is very expensive consisting of several
  roundtrips and files/directory data getting shipped back and forth to
  gain consensus so servers really need to be located somewhat close to
  each other.

It would be quite interesting to drop all the server replication and
resolution. No more server-server conflicts to cause problems for
reintegration etc.

If I were to rewrite a Coda server I probably would look at how some of
the web application are scaled. Have a tcp/udp level load balancer that
redirects clients to one or more Coda server apps which themselves are
fairly lightweight, synchronizing callbacks through a pubsub backend and
storing file data in an S3-style store. A lot of the smarts in Coda is
already handled by the client, not having to deal with replication and
resolution should make the servers even simpler. Mostly reintegration
and callbacks.

Maybe at some point the server to client file transfers could even be
done directly from S3, or from an http server. Especially now with 10GB+
networking and things like tcp offloading, sending bulk data using small
UDP packets with SFTP just doesn't seem to make much sense to me anymore.

> Given that Coda still has some undesired implementation artifacts like
> dependency on ipv4 addresses and certain arbitrary limitations which
> can not be lifted without relatively intrusive changes, this may open a
> possibility to improve the replication/history properties at the same time.
> 
> Orifs is written in c++ so "friendly" to Coda. The license looks to
> be the shortest of the BSD kind.

C++ doesn't necessarily make it friendly, the biggest issues that I know
of are Coda's dependence on userspace cooperative threading (LWP), which
in turn makes it incompatible with things like C++ exception handling
and kernel level threads which is what you are quite likely going to
find in a lot of non trivial code. It prevented us from successfully
integrating OpenLDAP with the servers, etc.

Jan