There isn't any, you could try to modify the rpc2_Decrypt and
  rpc2_Encrypt functions rpc2/rpc2-src/secure.c to wrap around
  cryptographic functions provided by openssl (libcrypto).

I believe that in order to get to a system with good
confidentiality/integrity properties (besides fixing defects such as
buffer overflows, etc.), it is important to think about key
management.  Right now, Coda has it's own Kerberos-like system, which
is only moderately easier to set up than Kerberos.  My thoughts have
been to ditch Coda's own authentication system, and use Kerberos for
all Coda authentication.  But, rather than using raw Kerberos, I'd use
GSSAPI.  The hard part is probably dealing with the rpc2
Decrypt/Encrypt - I haven't read this code, but now that Jan has
pointed out where it is, I realize it's more localized than I thought.

GSSAPI has the concept of "wrap" and "unwrap", which is a transform of
plaintext into an opaque blob which has integrity
protection/authentication and optional confidentiality protection.
Unfortunately, such transforms do not in general preserve size, and I
suspect that transforms providing authentication will generally
increase size.  So the places that call rpc2_ApplyE may need to
change, unless we can expand the buffer size in place somehow with a
copyout/copyback.

In rpc2a.c, the home-grown security handshake could be replaced by the
GSSAPI context establishment routines (which do something quite
similar behind the scenes).  Perhaps the gssapi handshake could be
done out-of-band, using a wrapper around rpc2 packets.  That way, a
received gssapi packet would be processed by gssapi to establish a
security context if it were that kind of message, and passed to rpc2
for processing if it is an already-authenticated rpc2 packet.

Note that encryption does not provide authentication, and there have
been attacks on systems involving replay of encrypted messages.
It may be that authentication is handled separately, though - I have
not read all the code.

  I believe that using real encryption will probably break
  retransmitting SFTP packets because some encrypted fields in the
  header are modified in the encrypted packet instead of a a decrypted
  version and re-encrypting it.

Using gss_wrap and gss_unwrap should make this clear pretty quickly,
and I suspect this will lead to rapid fixing of some latent defects.

THe other hard part about changing to GSSAPI is keeping track of the
bindings between "security contexts" (really session keys, but opaque)
and coda userids.  RPC2 traffic between a venus and a codasrv is on
behalf of venus itself sometimes, and sometimes on behalf of various
users.  This is perhaps not too hard, though, as rpc2 already takes a
'struct CEntry', which could store a 'const gss_ctx_id_t
context_handle' instead/additionally.

I've rambled about this before - sorry if I'm being repetitive, but I
hope this message has some more concrete details.

  Best bet for the actual encryption routines is openssl, it is already
  widely used and installed on most systems that have f.i. openssh.

If you just change xor to 3des or aes, I agree.  This library is
widely available.  But I wanted to point out the Kerberos/GSSAPI

        Greg Troxel <gdt_at_ir.bbn.com>