Serving a cluster¶
One node mounts; a cluster shares. lith serve nfs turns a single node into a
read-only NFSv3 gateway: it holds the index and the cache and serves N
compute nodes, so the cluster reads a dataset from S3 through one node
instead of N.
# on the gateway (c8gd.4xlarge with local NVMe for the disk cache)
lith serve nfs s3://bucket/prefix --index-file dataset.idx \
--disk-cache 100GB --disk-path /mnt/nvme/cache --listen :2049
# on each compute node
sudo mount -t nfs -o vers=3,proto=tcp,port=2049,mountport=2049,nolock,hard,\
rsize=1048576,wsize=1048576,nconnect=4,actimeo=600 <gateway-ip>:/ /mnt/data
--disk-cache is not optional for a gateway: size it to the working set so a
re-run and a restart serve from local disk instead of re-fetching S3.
Running the gateway as a container (Kubernetes, systemd) with the health probes and volumes wired? See Running in a container — that is the image's main use.
The one rule: does the cluster share data?¶
The gateway is a funnel — every client's bytes cross one node's NIC. That shapes the entire decision:
- Distinct data per node (each node reads a different object — a per-sample
BAM, one shard each): give each node its own
lith mount. N nodes on N NICs read in parallel; a shared cache has nothing to share. The gateway would only serialize them behind one NIC. - Shared data across nodes (all nodes read the same reference, index, or model — the bwa/STAR index, a shared Zarr, a foundation-model checkpoint): use the gateway. It fetches the shared dataset from S3 once and serves it to all N nodes, and the second job is free.
Measured, N=8, ledger-backed¶
| workload | gateway | independent lith mount |
takeaway |
|---|---|---|---|
distinct (8 CRAMs, one per node, flagstat) |
90 s, 6,762 GET | 72 s, 7,183 GET | independent wins — no sharing to exploit |
| shared (8 nodes read the same 8.88 GB bwa index) | 36 s, 1,139 GET / 9.1 GB | 13 s, 8,561 GET / 71.3 GB | gateway moves 1/8 the S3 traffic; independent is faster cold |
| shared, second run | 0.4 s, 0 GET | (re-fetches 71 GB) | the free second job |
On shared data the gateway reads the dataset from S3 once (1,139 GETs / 9.1 GB) where eight independent mounts read it eight times (8,561 GETs / 71.3 GB) — the "N nodes, one node's S3 traffic" claim, quantified at ~1/8. It is not faster cold (the funnel: one NIC vs eight), but it is far cheaper on S3 traffic and rate-limit pressure, and a re-run costs zero S3.
Sizing the gateway¶
Size it by aggregate demand, not one client's: the gateway's NIC and the
disk cache are shared by all N nodes. A c8gd.4xlarge (NVMe, ~15 Gbps) serves a
handful of nodes reading a shared working set comfortably; scale the instance up
with N and the working set. The gateway's own read path is concurrent and has
no FUSE hop — a single stream reads faster than a FUSE mount (1,463 MB/s
loopback) — but aggregate cold throughput is still bounded by the one NIC.
The gateway now detects the NIC, like the mount¶
Until recently lith serve nfs derived nothing from the device: it set the NIC rate only from
an explicit --nic-gbps, never derived an in-flight budget, and passed no first-byte-latency
seed. A stock export therefore ran with no bytes-in-flight bound at all (InflightBytes: 0
disables byte gating) and its coalesce gap pinned to the 256 KiB floor instead of the
device figure — roughly 1 MiB at a 25 Gbps baseline. A mount could not reach either state
(#393).
It now resolves the NIC through the same chain the mount uses (ethtool → cache →
ec2:DescribeInstanceTypes → size estimate → 10 Gbps fallback), derives
--inflight-bytes from the baseline, and carries the mount's two NIC diagnostics — the
per-source failure reasons (#317) and the
peak-vs-baseline warning (#239). An export is
more exposed than a mount to a wrong NIC figure, because every client's bytes cross the one
link, so a figure 12× low or 2× high is multiplied across all of them.
If you tuned a gateway around the old unbounded behaviour, this is a restricting change: an
export that previously had no byte gating now acquires a NIC-derived one. Pass
--inflight-bytes explicitly to keep a specific figure.
The readahead divisor counts readers, not mounts¶
Until recently windowBlocks() divided the prefetch budget by the number of registered
clients — anyone who had sent a MOUNT within --client-idle, five minutes by default, whether
or not they ever read a byte. Measured locally with no client reading anything: 1 mount gave 96
blocks of readahead, 8 gave 61, 48 gave 10, and 246 reached the 2-block floor. A 48× collapse
from registration alone.
Measured externally on a real gateway, 47 clients that mounted and read nothing against one streaming client:
| mounted clients | streaming throughput |
|---|---|
| 1 | 1267–1300 MB/s |
| 8 | 1305–1231 MB/s (no cost) |
| 48 | 519–558 MB/s — 2.4× slower |
Bytes were identical in every arm — 3.777 GB, 458 GETs, prefetch issued = used — so it was purely window depth and nothing was being wasted. No byte or request counter could see it, which is the same signature the FUSE-side version of this bug had (#301).
The divisor now counts paths read within --read-idle (30 s), using a per-path timestamp
the read path was already maintaining. One streaming client among 48 mounted ones gets the
whole budget again (#337).
It still divides — the fix changed the divisor's input, not whether there is a share. Removing the rationing outright costs 13.8–22.4× wall clock with 81–92 % of prefetch evicted unread, and replacing the share with a byte-exact total on the same aggregate regressed concurrent readers 11× (#309).
Why this direction is the safe one: a reader must have mounted, so active read streams can never exceed registered clients, so the change can only make a window larger. It cannot starve anyone — starvation is what #309 did. The opposite risk, over-commitment, is bounded by the tier-pressure gate that now defaults on for every export (#389); the two changes compose, and this one would have been materially riskier before that landed.
The gateway exports the blockstore's gauges too¶
Until recently an export published lith_prefetch_pressure_held_total — the admission gate's
hold count — while publishing neither lith_prefetch_pressure, the quantity it gates on, nor
lith_prefetch_committed_bytes and lith_prefetch_unread_resident_bytes, the tier figures that
make it readable. The counters that did appear made the gateway look instrumented, so the gap
was invisible until an external cell tried to answer whether the pressure gate's measured band
transfers to a gateway and found a gateway's peak pressure could not be read at all
(#337).
All four are now registered on both commands. The readahead-window gauges stay FUSE-only on purpose: they describe per-handle shares, and the gateway rations per client instead.
Several mounts on one host: --mem-cache is per-daemon¶
--mem-cache defaults to 25 % of system RAM for each mount, so mounts on one host add
up rather than sharing. A node serving five prefix-scoped mounts — one per input collection,
which is the normal shape for a model run — defaults to a ~250 % cap on the box — 25 % of RAM each, and each mount's footprint is
~2× its own tier, because the tier is live Go heap and at the default GOGC=100 the
collector grows the heap to about twice the live set (measured: next_gc at 2.03 × the tier). On a
30 GB head node where the application itself wanted 27 GB, that forced the operator to bound
every mount explicitly; on a 768 GB compute node the default would reserve 192 GB apiece
(#242).
Nothing is wrong with 25 % for a single mount. If you run more than one on a host, set
--mem-cache explicitly so the mounts sum to a sane fraction — and remember it bounds the
tier, not the process: the footprint is the tier plus outstanding prefetch
(#314), so leave headroom for both. There is
no cross-daemon awareness; each mount sizes itself in ignorance of the others.
Concurrency under high GETATTR rate (fixed, #244)¶
Through v1.1.1 a shared gateway could return NFS3ERR_STALE on GETATTR under
concurrent readers (clean to ~32, ~15% from 48 up) — enough to abort a
tightly-coupled MPI job at file open, since a job has no retry at nf90_open.
The cause was an upstream go-nfs-client bug: a variable-length handle was read
with a single Read that ignored short reads, so a handle straddling a
buffer/segment boundary under load decoded with a zeroed tail and was rejected as
STALE. Bytes were never wrong — a staled read simply failed rather than returning
bad data. Fixed by bumping go-nfs-client to the io.ReadFull version (v1.1.2).
Validated where it matters: a 96-rank GCHP MPI job across two nodes completed
144/144 timesteps against one shared gateway with zero STALE (where v1.1.1
aborted 4 s in), and a standalone O_DIRECT ladder that failed 6.6–21.2% at
48–128 concurrent readers on v1.1.1 ran 0 failures across 7,360 reads on
v1.1.2.
So choosing the gateway vs per-node mounts is now purely the funnel decision
above, not a concurrency limit: use the gateway when many nodes read a shared
dataset (one node's S3 traffic, free second job); use per-node lith mounts when
nodes read distinct data (N NICs in parallel — a shared cache has nothing to
share).
When the gateway dies¶
Reads are served over NFS, so the client's mount options decide behavior:
hard(recommended for data integrity): clients block while the gateway is down and resume transparently when it restarts. No error, no corruption — the correct default.soft: clients get a clean I/O error after the timeout, so a job fails fast instead of hanging.
Restart is cheap: the gateway reloads the index from the file (no re-listing) and serves the working set from its on-disk cache — a restart mid-job re-serves with ~0 S3 GETs.
Versus EFS and FSx Lustre¶
The gateway replaces a managed shared filesystem for read-only S3 datasets, without one's setup:
- EFS must be hydrated from S3 first — measured ~4.5 min to copy an 8.88 GB index (~32 MB/s), during which compute idles, and its shared read was the slowest of the three. Then you pay EFS storage + throughput and delete it.
- FSx Lustre has a 1.2 TiB minimum (you pay for 1.2 TiB to use 9 GB) and lazy-loads from S3 on first read (slow); its Linux client is also unavailable on some platforms (arm64 Ubuntu).
- The gateway hydrates nothing — it serves directly from S3 on the first read, has no filesystem minimum, and is a single process to start and stop.
For a shared, read-only S3 dataset across a cluster, the gateway is the cheapest and simplest of the three; for read-write scratch or POSIX-complete semantics, EFS/FSx remain the right tools.
Limits¶
NFSv3 only (no NFSv4 state/delegations/ACLs); read-only (every mutating op is
NFS3ERR_ROFS); file handles are (index-content-sha, inode) — stable across a
restart against the same index, STALE against a rebuilt one; per-path (not
per-client) sequential state, a limitation of go-nfs's stateless read path that
does not affect throughput.
No rpcbind. The gateway never registers with portmap/rpcbind; clients mount
with an explicit port (port=/mountport=), as the recipe above shows. That is
the contract, not a toggle.
The readahead window is a registration-based share, not a measured fair share. The prefetch budget is divided by the number of currently-mounted (not-idle) clients, which bounds total in-flight prefetch by the global budget. Because go-nfs's read path carries no per-operation client identity, "active" means "mounted and not idle since its last MOUNT" — a client streaming after its idle expiry is not in the denominator. It is a sound global bound, not a per-client guarantee.
Security boundary: the network perimeter, not per-client S3 authorization.
The gateway advertises AUTH_UNIX (client-asserted, unverified uid/gid, with
squash) and AUTH_NULL, and does not check client identity — any host that can
reach the port gets read access to the export. Put the gateway on a trusted
network (security group / subnet) and treat reachability as the authorization
boundary; there is no Kerberos and no per-client S3-credential mapping.