Coda File System

Re: Come-and-go system with data replication

From: Stephen J. Turnbull <>
Date: Tue, 28 Oct 2003 22:07:56 +0900
>>>>> "josef" == josef schwarz <> writes:

    josef> No, but the point is that I want it to be more
    josef> flexible. The system shall not be dependant that the absent
    josef> server comes back, another server should take over his
    josef> position. So that's not possible with Coda, is it?

Depends on what you want.  If you want failover for file service, it's
built in.

However, Coda does assume a fixed list of servers, and a fixed list of
replication groups (which are each a list of servers that handles a
given set of volumes).  Those are kept on the SCM.  If the SCM goes
down, then you have problems communicating that information.  I don't
recall offhand exactly what problems that causes, but IIRC running
clients and servers just keep running until you can bring the SCM back
up.  IIRC the only time you need to restart servers (assuming normal
operation) is if you need to change the address of the SCM.

ISTR that if you have scm at x.y.z.w and server1 and server2, if scm
dies the death, but you have a backup copy of the SCM-only files (a
couple of itty-bitty ones), you can just shutdown server2, reconfigure
it with the SCM's IP address x.y.z.w and hostname (which it should be
getting from the DNS anyway), copy the SCM files to it, and bring it
back up as the SCM.  Now, there might even be DNS hacks to accomplish
this without taking over the old IP, I don't know.

Not ideal (especially if server2 was doing non-Coda duty), but hardly
terrifyingly unstable.

    josef> I am just thinking in terms of Peer-to-peer, with a node

You mean Napster/Gnutella?  If not, what do you mean?

    josef> being client and server at the same time (which is good for
    josef> Come-and-go behaviour).

What do you mean by "come-and-go behaviour"?  Napster?  Gnutella?
mDNS?  DHCP?  Or just that if a particular server is not available,
the client tries another one?

As for a node being client and server at the same time, that's
possible (I do it because the only fixed-address box I had that was
powerful enough to be a server was my main workstation), but it's not
the designed way for coda to work.

Coda is intended to provide reliable file service for weakly connected
clients.  This means (1) failover to a different server when a server
is down or the network is partitioned, and (2) caching of files in
case the client ends up partitioned from all servers.  Clients are
expected to be in that state a lot (mobile units, whatever).  If
that's not what you want to accomplish, maybe you want a different DFS.

    josef> With Coda, this is hardly possible, however, due to the
    josef> fact that a server needs to be restarted when another one
    josef> is added - that's right, isn't it?

That depends on your definition of "added."  There are some cases
where you need to restart, but they involve a configuration change.
So if you buy a new box and install a coda server on it, you'll need
to add it to volume storage groups so clients know which servers serve
which volumes.  This may require that the SCM at least be restarted, I

If "added" means that a known server returns to service, then you're
wrong: Coda deals with network partitions and server crashes
automatically, and performs crude load-balancing if parts of the
network are poorly connected.

    josef> It really seems that the set-up with IP addresses was no
    josef> good idea.

It wasn't, but not because you have IP addresses rather than domain
names.  AFAIK correctly configured IP addresses will work fine (at
least I've done that with earlier 5.x.y coda), but you have other
problems here.

    >> 'cfs lv /coda/' and 'cfs lv /coda/'

These are two separate realms.  A realm is a set of servers all
serving a single group of volumes.  The name of the realm is more or
less arbitrary AFAIK, but is normally taken as the address of the SCM.
If the clients are supposed to perceive them as a single realm (set of
available volumes), you're going to run into trouble.

You are not supposed to refer to different servers in the same realm
as different realms.  This will almost certainly cause problems as the
clients get confused about where to push the data from a given volume.

    josef> The reason that the two clients seem to be in different
    josef> subnets is that the underlying vpn infrastructure requires
    josef> it.  But this really should not matter, since the
    josef> appropriate files are adapted, and it is working basically.

AFAIK subnetting doesn't matter, as long as the addresses refer to
different hosts.  If you actually have a multihomed host with
addresses on multiple nets, and try to make coda know about that
(including via the DNS) you will have problems, as coda doesn't handle
multihomed hosts gracefully yet.

    >> ... you have a "fancy" configuration !

    josef> :) What's so fancy if I want to use IP addresses instead of
    josef> hostnames?

I wouldn't say it's fancy, but it's really unclear what you expect
from coda when you vary just about everything from the example in the
HOWTO, and don't tell us if you tried anything closer to that
"standard beginner's setup".

If you follow the instructions in the HOWTO and set up a simple
single-server realm, with the client on the same host, then you can at
least verify that your hardware is unbroken, the kernel is correctly
compiled, the kernel module is loaded, etc, etc.  Have you done that?
(FWIW, my guess is that either the kernel module isn't loaded or it's
incompatible with either venus or the kernel, from the complaints
about upcalls.)

If that doesn't work, it's quite possible that your problem is not in
Coda, but in an OS quirk, network configuration, hardware, etc.  (I'd
still bet on a Coda issue, but at least this way you've narrowed the
field.)  And if it's Coda, then probably your current problem doesn't
have anything to do with the replication feature, but with something
else in Coda.

If it _does_ work, then you can say that replication does have
something to do with it.  At least you've narrowed the field of
possible problems.

    josef> Thanks for advice, I think I have never read that much in
    josef> mailing list archives and got so little actual help out of
    josef> it than with the one of Coda.  And it really makes me
    josef> thinking about giving up, when I find messages like this
    josef> one, where a similar but easier configuration did still not
    josef> work after a whole year...


Yeah, that's the guy who wanted Win2000's cache on Unix.  If the
Win2000 cache is exactly what he wants, cool.  But Coda is not really
designed to work that way.  If you're trying to do something that Coda
is not designed for, it's probably going to be hard and inconvenient.

    josef> I know that Coda is nothing for beginners,

Hm.  I just followed the directions posted on the web site and pretty
much everything was fine until I ran out of RVM on my server.  Then
the fun began....

    josef> I would not consider me as a beginner, however -

That's not a good way to think about it.  Coda semantics are not the
same as Win2k's cache, they're not the same as NFS or SMB or AFS or
Gnutella.  If you think you know what you're doing, you are going to
lose big (unless you get very lucky, and don't lose at all).

Your posts are full of assumptions about what should be easy, what's
obvious, what's desirable, how things are implemented, etc.  You wo
Received on 2003-10-28 08:10:45