On 5 Dec 1998, Greg Troxel wrote:

> Just for grins, I built the 4.7.4 source under FreeBSD 2.2.6.  The
> only hitch was that coda.h didn't end up including <sys/time.h>, and
> therefore timespec wasn't defined.  Other than that, it built.
> Because of my later lossage, I didn't try to figure out the twisty
> maze of ifdefs and determine how to fix this without breaking anything
> else.  I then tried the 4.7.4 venus on a system with a 4.6.5
> (patch-4.4.?) kernel.  It gave me some message about can't mount /coda
> or can't open cfs0.
> 
> Before I try any further, is there any reason to expect this to work?
> Or do I need the 4.7.4 kernel version?

Because of the plethora of versions of FreeBSD, we are currently
concentrating on -stable and -current, and will probably drop support at
some point for -stable.  I have successfully built and used Coda on many
versions of 2.2-, but because 2.2- is a moving target (as is Coda) I
believe we only test on 2.2.7 and 2.2-stable, and with the release of
2.2.8 presumably 2.2.7 will fall off the list.  I believe a 4.7.4 userland
should be able to talk to a 4.6.5 kernel, but because a number of bugfixes
have been added to the BSD kernel code recently, I recommend running the
newest Coda kernel code you can find.  Under 3.0-CURRENT of FreeBSD, the
Coda kernel code is in the base distribution (and can be retrieved using
cvsup as part of the kernel code) so that makes life a lot easier.

> On another note:
> I had another client kernel crash.  It was in cfs_fbsd.c:cvref, and
> hit the panic with usecount <=0.   I think this had to do with killing
> venus while the network communication was wedged, but I can't
> reproduce it at will.  Should I be using 4.7.4 kernel/user code on the
> clients?  It seems like it has changed a bit and perhaps debugging the
> 4.6.5 stuff is not useful.

4.7.x code included many bugfixes and race conditions concerning the death
of venus.  However, whether it is useful to debug 4.6.5 is probably a
question for Bob Baron, as he is the active kernel developer for FreeBSD.

> I tried to run coda across a firewall not administered by me (server
> out, client in).  I found that the venus to codasrv messages went out
> and the replies came back, but that the codasrv-se to venus-se
> messages were blocked.  While I know I need to fix the firewall to
> treat these as part of the same 'connection', I found that this wedged
> the client particularly badly.  I expected venus to give me 'operation
> timed out' after a while, but it seemed not to (simple ls of /coda
> which has two dirs project and home - nothing fancy).  tcpdump showed
> that the basic rpc request/response were getting through, but not the
> -se stuff.  So it might be that this isn't interpreted as failure, but
> causes continuous retries.  
> 
> I don't know if it is possible to make the client send a udp datagram
> from venus-se to codasrv-se when expecting side effect packets; this
> would feel kludgy but probably cause coda to work across more
> firewalls, as not everyone is fortunate enough to be able to cause
> theirs to be changed.  I haven't looked into this.

Coda currently does not behave well when clients are behind a firewall.
This is especially the case when outgoing connections from the firewall
all appear to come from the same host (that is, it is not IP transparent).
This is because the Coda server builds a 'callback connection' to each
client, and attempts to maintain one per client IP address that connects
to it.  This connection is used to indicate changes to files on the server
that the client has in its cache, and needless to say, incorrect
functioning of this (the effective limit of one client per firewall :) is
not good.

On the other hand, having the firewall in the other direction (hiding a
single server) probably would work ok.  As you point out, NATs and
firewalls often require a priming of their UDP packet forwarding, rather
than letting unexpected UDP packets in (especially if it is not clear
where the packet would go :).  Indeed, changes could be made to RPC2/Coda
to make sure a packet or two is always sent to or from the server first,
but it depends on what your firewall arrangement is as to which would be
needed.  My feeling is that in the short term, firewall interoperability
in Coda will be pretty poor based on a number of the underlying design
assumptions (client IP == one cache manager) and the need to use an RPC
package based on UDP.  The use of encryption also prohibits firewall
involvement, as it makes it difficult for the firewall to modify packets
as they pass through the firewall so as to hide information (for example,
the client IP address is included in the bind packet for the RPC
connection, and communication to an RPC client is made using that IP
address, not the address where it actually came from).

There used to be some strange failure conditions when the main RPC
connection succeeded and the side effects consistently failed -- have you
observed any?  I tried running a client from behind a NAT for a while, and
while this often worked, the side effect behavior sometimes resulted in
the client crashing due to a failed assertion.

There are a few feasible long term solutions to the firewall problem.
Most of them involve modifying the Coda servers to support the idea of
multiple cache managers on a single client IP.  This, in combination with
not using fixed port numbers on the client should provide most of what we
need, in combination with connection priming.  Another interesting thing
to try might be to have a Coda application proxy on the firewall that
speaks the Coda protocol on one side (the outside) and looks like a single
client, but speaks a lightly modified version of the RPC protocol on the
inside to work around the encryption concerns, and proxies the data
requests to the correct clients as needed.  This would not be so easy to
write, I suspect.

Having the server behind a firewall is likely to be an unfortunate
experience, and may not be easily solvable.  For now, it is a fair
assumption that one IP address per server (at minimum) is required. 

> The good news veiled in this message is that coda is already working
> well enough for me that I'm expanding the user community to about 6
> from 1.2 people.

This is great news!

  Robert N Watson 

robert@fledge.watson.org              http://www.watson.org/~robert/
PGP key fingerprint: 03 01 DD 8E 15 67 48 73  25 6D 10 FC EC 68 C1 1C

Carnegie Mellon University            http://www.cmu.edu/
TIS Labs at Network Associates, Inc.  http://www.tis.com/
SafePort Network Services             http://www.safeport.com/