The Demand-Attach FileServer (DAFS) has resulted in many changes to how
many things on AFS fileservers behave. The most sweeping changes are
probably in the volume package, but significant changes have also been
made in the SYNC protocol, the vnode package, salvaging, and a few
miscellaneous bits in the various fileserver processes.

This document serves as an overview for developers on how to deal with
these changes, and how to use the new mechanisms. For more specific
details, consult the relevant doxygen documentation, the code comments,
and/or the code itself.

 - The salvageserver

The salvageserver (or 'salvaged') is a new OpenAFS fileserver process in
DAFS. This daemon accepts salvage requests via SALVSYNC (see below), and
salvages a volume group by fork()ing a child, and running the normal
salvager code (it enters vol-salvage.c by calling SalvageFileSys1).

Salvages that are initiated from a request to the salvageserver (called
'demand-salvages') occur automatically; whenever the fileserver (or
other tool) discovers that a volume needs salvaging, it will schedule a
salvage on the salvageserver without any intervention needed.

When scheduling a salvage, the vol id should be the id for the volume
group (the RW vol id). If the salvaging child discovers that it was
given a non-RW vol id, it will send the salvageserver a SALVSYNC LINK
command, and will exit. This will tell the salvageserver that whenever
it receives a salvage request for that vol id, it should schedule a
salvage for the corresponding RW id instead.

 - FSSYNC/SALVSYNC

The FSSYNC and SALVSYNC protocols are the protocols used for
interprocess communication between the various fileserver processes.
FSSYNC is used for querying the fileserver for volume metadata,
'checking out' volumes from the fileserver, and a few other things.
SALVSYNC is used to schedule and query salvages in the salvageserver.

FSSYNC existed prior to DAFS, but it encompasses a much larger set of
commands with the advent of DAFS. SALVSYNC is entirely new to DAFS.

 -- SYNC

FSSYNC and SALVSYNC are both layered on top of a protocol called SYNC.
SYNC isn't much a protocol in itself; it just handles some boilerplate
for the messages passed back and forth, and some error codes common to
both FSSYNC and SALVSYNC.

SYNC is layered on top of TCP/IP, though we only use it to communicate
with the local host (usually via a unix domain socket). It does not
handle anything like authentication, authorization, or even things like
serialization. Although it uses network primitives for communication,
it's only useful for communication between processes on the same
machine, and that is all we use it for.

SYNC calls are basically RPCs, but very simple. The calls are always
synchronous, and each SYNC server can only handle one request at a time.
Thus, it is important for SYNC server handlers to return as quickly as
possible; hitting the network or disk to service a SYNC request should
be avoided to the extent that such is possible.

SYNC-related source files are src/vol/daemon_com.c and
src/vol/daemon_com.h

 -- FSSYNC

 --- server

The FSSYNC server runs in the fileserver; source is in
src/vol/fssync-server.c.

As mentioned above, FSSYNC handlers should finish quickly when
servicing a request, so hitting the network or disk should be avoided.
In particular, you absolutely cannot make a SALVSYNC call inside an
FSSYNC handler; the SALVSYNC client wrapper routines actively prevent
this from happening, so even if you try to do such a thing, you will not
be allowed to. This prohibition is to prevent deadlock, since the
salvageserver could have made the FSSYNC request that you are servicing.

When a client makes a FSYNC_VOL_OFF or NEEDVOLUME request, the
fileserver offlines the volume if necessary, and keeps track that the
volume has been 'checked out'. A volume is left online if the checkout
mode indicates the volume cannot change (see VVolOpLeaveOnline_r).

Until the volume has been 'checked in' with the ON, LEAVE_OFFLINE, or
DONE commands, no other program can check out the volume.

Other FSSYNC commands include abilities to query volume metadata and
stats, to force volumes to be attached or offline, and to update the
volume group cache. See doc/arch/fssync.txt for documentation on the
individual FSSYNC commands.

 --- clients

FSSYNC clients are generally any OpenAFS process that runs on a
fileserver and tries to access volumes directly. The volserver,
salvageserver, and bosserver all qualify, as do (sometimes) some
utilities like vol-info or vol-bless. For issuing FSSYNC commands
directly, there is the debugging tool fssync-debug.  FSSYNC client code
is in src/vol/fssync-client.c, but it's not very interesting.

Any program that wishes to directly access a volume on disk must check
out the volume via FSSYNC (NEEDVOLUME or OFF commands), to ensure the
volume doesn't change while the program is using it. If the program
determines that the volume is somehow inconsistent and should be
salvaged, it should send the FSSYNC command FORCE_ERROR with reason code
FSYNC_SALVAGE to the fileserver, which will take care of salvaging it.

 -- SALVSYNC

The SALVSYNC server runs in the salvageserver; code is in
src/vol/salvsync-server.c. SALVSYNC clients are just the fileserver, the
salvageserver run with the -client switch, and the salvageserver worker
children. If any other process notices that a volume needs salvaging, it
should issue a FORCE_ERROR FSSYNC command to the fileserver with the
FSYNC_SALVAGE reason code.

The SALVSYNC protocol is simpler than the FSSYNC protocol. The commands
are basically just to create, cancel, change, and query salvages. The
RAISEPRIO command increases the priority of a salvage job that hasn't
started yet, so volumes that are accessed more frequently will get
salvaged first. The LINK command is used by the salvageserver worker
children to inform the salvageserver parent that it tried to salvage a
readonly volume for which a read-write clone exists (in which case we
should just schedule a salvage for the parent read-write volume).

Note that canceling a salvage is just for salvages that haven't run
yet; it only takes a salvage job off of a queue; it doesn't stop a
salvageserver worker child in the middle of a salvage.

 - The volume package

 -- refcounts

Before DAFS, the Volume struct just had one reference count, vp->nUsers.
With DAFS, we know have the notion of an internal/lightweight reference
count, and an external/heavyweight reference count. Lightweight refs are
acquired with VCreateReservation_r, and released with
VCancelReservation_r. Heavyweight refs are acquired as before, normally
with a GetVolume or AttachVolume variant, and releasing the ref with
VPutVolume.

Lightweight references are only acquired within the volume package; a vp
should not be given to e.g. the fileserver code with an extra
lightweight ref. A heavyweight ref is generally acquired for a vp that
will be given to some non-volume-package code; acquiring a heavyweight
ref guarantees that the volume header has been loaded.

Acquiring a lightweight ref just guarantees that the volume will not go
away or suddenly become unavailable after dropping VOL_LOCK. Certain
operations like detachment or scheduling a salvage only occur when all
of the heavy and lightweight refs go away; see VCancelReservation_r.

 -- state machine

Instead of having a per-volume lock, each vp always has an associated
'state', that says what, if anything, is occurring to a volume at any
particular time; or if the volume is attached, offline, etc. To do the
basic equivalent of a lock -- that is, ensure that nobody else will
change the volume when we drop VOL_LOCK -- you can put the volume in
what is called an 'exclusive' state (see VIsExclusiveState).

When a volume is in an exclusive state, no thread should modify the
volume (or expect the vp data to stay the same), except the thread that
put it in that state. Whenever you manipulate a volume, you should make
sure it is not in an exclusive state; first call VCreateReservation_r to
make sure the volume doesn't go away, and then call
VWaitExclusiveState_r. When that returns, you are guaranteed to have a
vp that is in a non-exclusive state, and so can me manipulated. Call
VCancelReservation_r when done with it, to indicate you don't need it
anymore.

Look at the definition of the VolState enumeration to see all volume
states, and a brief explanation of them.

 -- VLRU

See: Most functions with VLRU in their name in src/vol/volume.c.

The VLRU is what dictates when volumes are detached after a certain
amount of inactivity. The design is pretty much a generational garbage
collection mechanism. There are 5 queues that a volume can be on the
VLRU (VLRUQueueName in volume.h). 'Candidate' volumes haven't seen
activity in a while, and so are candidates to be detached. 'New' volumes
have seen activity only recently; 'mid' volumes have seen activity for
awhile, and 'old' volumes have seen activity for a long while. 'Held'
volumes cannot be soft detached at all.

Volumes are moved from new->mid->old if they have had activity recently,
and are moved from old->mid->new->candidate if they have not had any
activity recently. The definition of 'recently' is configurable by the
-vlruthresh fileserver parameter; see VLRU_ComputeConstants for how they
are determined. Volumes start at 'new' on attachment, and if any
activity occurs when a volume is on 'candidate', it's moved to 'new'
immediately.

Volumes are generally promoted/demoted and soft-detached by
VLRU_ScannerThread, which runs every so often and moves volumes between
VLRU queues depending on their last access time and the various
thresholds (or soft-detaches them, in the case of the 'candidate'
queue). Soft-detaching just means the volume is taken offline and put
into the preattached state.

 --- DONT_SALVAGE

The dontSalvage flag in volume headers can be set to DONT_SALVAGE to
indicate that a volume probably doesn't need to be salvaged. Before
DAFS, volumes were placed on an 'UpdateList' which was periodically
scanned, and dontSalvage was set on volumes that hadn't been touched in
a while.

With DAFS and the VLRU additions, setting dontSalvage now happens when a
volume is demoted a VLRU generation, and no separate list is kept. So if
a volume has been idle enough to demote, and it hasn't been accessed in
SALVAGE_INTERVAL time, dontSalvage will be set automatically by the VLRU
scanner.

 -- Vnode

Source files: src/vol/vnode.c, src/vol/vnode.h, src/vol/vnode_inline.h

The changes to the vnode package are largely very similar to those in
the volume package. A Vnode is put into specific states, some of which
are exclusive and act like locks (see VnChangeState_r,
VnIsExclusiveState). Vnodes also have refcounts, incremented and
decremented with VnCreateReservation_r and VnCancelReservation_r like
you would expect. I/O should be done outside of any global locks; just
the vnode is 'locked' by being put in an exclusive state if necessary.

In addition to a state, vnodes also have a count of readers. When a
caller gets a vnode with a read lock, we of course must wait for the
vnode to be in a nonexclusive state (VnWaitExclusive_r), then the number
of readers is incremented (VnBeginRead_r), but the vnode is kept in a
non-exclusive state (VN_STATE_READ).

When a caller gets a vnode with a write lock, we must wait not only for
the vnode to be in a nonexclusive state, but also for there to be no
readers (VnWaitQuiescent_r), so we can actually change it.

VnLock still exists in DAFS, but it's almost a no-op. All we do for DAFS
in VnLock is set vnp->writer to the current thread id for a write lock,
for some consistency checks later (read locks are actually no-ops).
Actual mutual exclusion in DAFS is done by the vnode state machine and
the reader count.

 - viced state serialization

See src/viced/serialize_state.* and ShutDownAndCore in
src/viced/viced.c

Before DAFS, whenever a fileserver restarted, it lost all information
about all clients, what callbacks they had, etc. So when a client with
existing callbacks contacted the fileserver, all callback information
needed to be reset, potentially causing a bunch of unnecessary traffic.
And of course, if the client does not contact the fileserver again, it
could not get sent callbacks it should get sent.

DAFS now has the ability to save the host and CB data to a file on
shutdown, and restore it when it starts up again. So when a fileserver
is restarted, the host and CB information should be effectively the same
as when it shut down. So a client may not even know if a fileserver was
restarted.

Getting this state information can be a little difficult, since the host
package data structures aren't necessarily always consistent, even after
H_LOCK is dropped. What we attempt to do is stop all of the background
threads early in the shutdown process (set fs_state.mode -
FS_MODE_SHUTDOWN), and wait for the background threads to exit (or be
marked as 'tranquil'; see the fs_state struct) later on, before trying
to save state. This makes it a lot less likely for anything to be
modifying the host or CB structures by the time we try to save them.

 - volume group cache

See: src/vol/vg_cache* and src/vol/vg_scan.c

The VGC is a mechanism in DAFS to speed up volume salvages. Pre-VGC,
whenever the salvager code salvaged an individual volume, it would need
to read all of the volume headers on the partition, so it knows what
volumes are in the volume group it is salvaging, so it knows what
volumes to tell the fileserver to take offline. With demand-salvages,
this can make salvaging take a very long time, since the time to read in
all volume headers can take much more time than the time to actually
salvage a single volume group.

To prevent the need to scan the partition volume headers every single
time, the fileserver maintains a cache of which volumes are in what
volume groups. The cache is populated by scanning a partition's volume
headers, and is started in the background upon receiving the first
salvage request for a partition (VVGCache_scanStart_r,
_VVGC_scan_start).

After the VGC is populated, it is kept up to date with volumes being
created and deleted via the FSSYNC VG_ADD and VG_DEL
commands. These are called every time a volume header is created,
removed, or changed when using the volume header wrappers in vutil.c
(VCreateVolumeDiskHeader, VDestroyVolumeDiskHeader,
VWriteVolumeDiskHeader). These wrappers should always be used to
create/remove/modify vol headers, to ensure that the necessary FSSYNC
commands are called.

 -- race prevention

In order to prevent races between volume changes and VGC partition scans
(that is, someone scans a header while it is being written and not yet
valid), updates to the VGC involving adding or modifying volume headers
should always be done under the 'partition header lock'. This is a
per-partition lock to conceptually lock the set of volume headers on
that partition. It is only read-held when something is writing to a
volume header, and it is write-held for something that is scanning the
partition for volume headers (the VGC or partition salvager). This is a
little counterintuitive, but it is what we want.  We want multiple
headers to be written to at once, but if we are the VGC scanner, we want
to ensure nobody else is writing when we look at a header file.

Because the race described above is so rare, vol header scanners don't
actually hold the lock unless a problem is detected. So, what they do is
read a particular volume header without any lock, and if there is a
problem with it, they grab a write lock on the partition vol headers,
and try again. If it still has a problem, the header is just faulty; if
it's okay, then we avoided the race.

Note that destroying vol headers does not require any locks, since
unlink()s are atomic and don't cause any races for us here.

 - partition and volume locking

Previously, whenever the volserver would attach a volume or the salvager
would salvage anything, the partition would be locked
(VLockPartition_r). This unnecessarily serializes part of most volserver
operations. It also makes it so only one salvage can run on a partition
at a time, and that a volserver operation cannot occur at the same time
as a salvage. With the addition of the VGC (previous section), the
salvager partition lock is unnecessary on namei, since the salvager does
not need to scan all volume headers.

Instead of the rather heavyweight partition lock, in DAFS we now lock
individual volumes. Locking an individual volume is done by locking a
certain byte in the file /vicepX/.volume.lock. To lock volume with ID
1234, you lock 1 byte at offset 1234 (with VLockFile: fcntl on unix,
LockFileEx on windows as of the time of this writing). To read-lock the
volume, acquire a read lock; to write-lock the volume, acquire a write
lock.

Due to the potentially very large number of volumes attached by the
fileserver at once, the fileserver does not keep volumes locked the
entire time they are attached (which would make volume locking
potentially very slow). Rather, it locks the volume before attaching,
and unlocks it when the volume has been attached. However, all other
programs are expected to acquire a volume lock for the entire duration
they interact with the volume. Whether a read or write lock is obtained
is determined by the attachment mode, and whether or not the volume in
question is an RW volume (see VVolLockType()).

These locks are all acquired non-blocking, so we can just fail if we
fail to acquire a lock. That is, an errant process holding a file-level
lock cannot cause any process to just hang, waiting for a lock.

 -- re-reading volume headers

Since we cannot know whether a volume is writable or not until the
volume header is read, and we cannot atomically upgrade file-level
locks, part of attachment can now occur twice (see attach2 and
attach_volume_header). What occurs is we read the vol header, assuming
the volume is readonly (acquiring a read or write lock as necessary).
If, after reading the vol header, we discover that the volume is
writable and that means we need to acquire a write lock, we read the vol
header again while acquiring a write lock on the header.

 -- verifying checkouts

Since the fileserver does not hold volume locks for the entire time a
volume is attached, there could have been a potential race between the
fileserver and other programs. Consider when a non-fileserver program
checks out a volume from the fileserver via FSSYNC, then locks the
volume. Before the program locked the volume, the fileserver could have
restarted and attached the volume. Since the fileserver releases the
volume lock after attachment, the fileserver and the other program could
both think they have control over the volume, which is a problem.

To prevent this non-fileserver programs are expected to verify that
their volume is checked out after locking it (FSYNC_VerifyCheckout).
What this does is ask the fileserver for the current volume operation on
the specific volume, and verifies that it matches how the program
checked out the volume.

For example, programType X checks out volume V from the fileserver, and
then locks it. We then ask the fileserver for the current volume
operation on volume V. If the programType on the vol operation does not
match (or the PID, or the checkout mode, or other things), we know the
fileserver must have restarted or something similar, and we do not have
the volume checked out like we thought we did.

If the program determines that the fileserver may have restarted, it
then must retry checking out and locking the volume (or return an
error).