Knobs

Every flag, grouped by the trade it makes. Defaults are chosen to be right for most workloads; each line says what the default is and why. The canonical, never-drifting list is always lith mount --help, lith index build --help, and lith bench --help — this page groups those and explains them.

Bytes vs latency

How much lith fetches per miss. Bigger reads amortize round-trips but waste bandwidth on bytes you won't use.

flag	default	why
--block-size	8MiB	The fill/readahead unit — a run of 1 MiB cache chunks coalesced into one range GET. 8 MiB balances round-trip amortization against over-fetch.
--max-range	64MiB	Caps how many contiguous blocks coalesce into a single GET, so one fill can't monopolize a connection.
--small-file	4MiB	Files at or below this are fetched whole on first read — one GET beats seeking within a tiny object.
--parts-max	auto	Files up to this are fetched whole as concurrent block-sized range parts — but only once the access pattern proves it tiles (#229): a sequential reader establishes within the first couple of reads and then gets the whole-file parts fetch (what lets a 31 MB granule beat aws s3 cp); a reader that only wants a slice (a hyperslab, a footer probe) never establishes and is served byte-exact instead of pulling the whole object. The size threshold is auto — derived from NIC baseline × first-byte latency, clamped to [--small-file, 64MiB] (#220): a fat pipe would fetch whole cheaply, a thin pipe keeps a sub-file read byte-exact. An explicit size overrides; 0 disables.
--bgzf-whole-file-max	512MiB	For a bgzf-family data file (BAM/CRAM/VCF.gz/BCF) that has a coordinate-index sibling (.bai/.tbi/.csi/.crai), files at or below this are fetched whole through the parts path when the index opens — a region-indexed scan (e.g. tabix over many regions) touches nearly every block, so the whole-file working set makes cold ≈ warm (v0.3, #107). Above it, tier-2 prefetches only the index-resolved slices/chunks. 0 disables whole-file prefetch. The handle keeps its adaptive readahead window either way — the plan only adds ranges.

Cold-start cost of "prove it first" (#229). lith will not fetch broadly (whole-file parts, or wide readahead) until a handle's reads show they tile — the first couple of reads are served precise. This is what keeps a hyperslab or a footer probe from pulling a whole object. The cost lands on the opposite case: a cold sequential read of a mid-size file (a cat/cp off the mount) pays roughly one extra round-trip of latency for its first block, because that block is demand-fetched in chunks before the pattern establishes and the whole-file fetch kicks in. It is cold-only (a warm re-read is unaffected), byte-identical, and shrinks with file size (the first block is a smaller fraction of a larger file). The block-0 coalescing gap that would erase even this is tracked in #233.

RAM vs re-fetch

How much lith keeps in memory (and optionally on disk) so a re-read is free instead of another GET.

flag	default	why
--mem-cache	25 % of the memory budget	The in-memory tier — not the process footprint. The tier is ordinary Go heap, so the garbage collector sizes itself around it: at the default GOGC=100 the heap target is about twice the live set, which is why 25 % is the default (its target is ~half the box, leaving room for the app, the page cache, and a burst's transient garbage) and why a tier above ~50 % of RAM cannot work without a memory limit. That ceiling is arithmetic from GOGC's definition, not a chosen constant — and it is now measured: next_gc came in at 2.03 × the tier in three arms that differed 3× in outstanding prefetch, with RSS within 1 % of next_gc every time. A 20 GB tier on a 33 GB box was OOM-killed at RSS 30.56 GB — before the tier was even full (#314); the mount and the gateway now warn at startup when 2 × tier ≥ RAM. If you need a larger tier, set GOMEMLIMIT so the collector works harder instead of the kernel killing the process. Raise it for re-read-heavy work with spare RAM, within that ceiling. Per-daemon, and it compounds with the doubling above: the default is 25 % of RAM for each mount, and each mount's footprint is ~2× its own tier — so five mounts on one host default to a ~250 % cap, not 125 %. Measured: a 10 GB tier produced 20.4 GB RSS with next_gc at 2.03 × the tier in three independent arms. If you run more than one mount on a box, set --mem-cache explicitly so they sum to a sane fraction (#242). Values below one 1 MiB chunk disable the tier entirely and the mount warns; between one chunk and 64 MiB the tier now scales its shard count to stay usable, where it used to accept nothing (#307). Size it at ~2.5× your distinct working set, not 1× — see below.
--memory-limit	auto	Soft memory limit for the Go runtime: a size, auto, or off (#314). Why it exists. The memory tier is live Go heap, and at the default GOGC=100 the collector lets the heap grow to about twice the live set — measured at next_gc = 2.03 × the tier, with RSS within 1% of it. So a tier over half the memory budget is an OOM-kill on the way to filling it. auto engages only in that case, setting a limit at 90% of the budget, and changes nothing for a tier that fits: at the default 25% the collector's target is ~50% of the budget, far under any limit. Its boundary is exactly the startup warning's (2 × tier ≥ budget), so the two cannot disagree about which configurations are at risk. With a limit in force the collector works harder instead of the kernel killing the process — the Go runtime caps that extra GC work at about half the process's CPU. The 90% is the Go team's guidance, not a measurement here — leave 5–10% headroom for memory the runtime does not account for (stacks, runtime structures, the kernel's FUSE buffers) — and its exact value matters less than it looks: it only ever applies where the alternative is an OOM-kill. The budget is the smaller of system RAM and the cgroup memory limit. Inside a container /proc/meminfo reports the host's RAM, so before this a container limited to 4 GiB on a 256 GiB host defaulted to a 64 GiB tier — sixteen times its allowance — and the startup warning, comparing against the same figure, stayed silent. This matters most for lith serve nfs, which is what the container image runs. A GOMEMLIMIT already in the environment is honoured as-is under auto; an explicit --memory-limit size wins over both. A limit at or below --mem-cache warns at startup, because the tier is live heap the collector cannot reclaim and it would run continuously against a target it cannot meet. The limit in force is logged at startup and exported by the Go collector as go_gc_gomemlimit_bytes. Measured on the configuration that was OOM-killed before this existed (a 20 GB tier on a 33 GB box): it survives in both reps, RSS plateauing at 28.8 GB against a 29.7 GB limit, with no measurable wall cost — but that workload is link-bound (16 concurrent streams), which is where extra GC work hides. Concurrent GC CPU was not captured, so treat it as free on a streaming workload rather than free in general; a CPU-bound reader with a tier this large would pay something. Either way the comparison it is making is against the process being killed.
--wire-ttfb	off	Diagnostic. Splits a first-byte latency into its three intervals, all labelled conn="new" / conn="reused" (#350): lith_s3_conn_acquire_seconds (getting a connection), lith_s3_request_write_seconds (getting the request onto it), and lith_s3_endpoint_ttfb_seconds (the only part that is genuinely S3 answering), with lith_s3_wire_ttfb_seconds as their sum. Seven mechanisms for an in-mount 60–100 ms first byte have been refuted by measurement — request concurrency (twice), bytes in flight, box CPU, above-the-wire, connection setup, allocation — and the Go runtime shows ~1 ms of STW per burst with not one of ~31k scheduler waits over 0.9 ms. So the delay is inside the wire interval, and these three are what it is made of. It forces the plain *http.Client path, so do not leave it on in production.
--prefetch-sibling-coverage	off	Experimental. Counts a hole in a handle's read stream as covered when the bytes in it were already demanded through lith (#316). lith opens with FOPEN_KEEP_CACHE, so every descriptor on an inode shares the kernel page cache: a sibling's reads are served by the kernel and never reach lith, each handle's own stream is punctate with coverage ≈ 1/N, and from four concurrent readers of one object the #221 coverage gate holds every handle provisional so nothing prefetches. Measured 243× slower than one reader alone, with byte amplification of 1.001 — the cleanest byte count of any cell and the slowest run, so no byte or request counter can see it. Measured with this flag at the default --prefetch-coverage-min: 16 of 16 readers establish in both reps, wall 67 s → 2.9 s, and coverage_held falls from ~1000 to 16 — the same as O_DIRECT. The holes are unchanged (median 0.95–0.96 of reads still have one in front of them), so it discounts them rather than removing them. It is also the most byte-efficient of the three ways to get there: 0.67 GB fetched for 0.54 GB read, against 1.89 GB for --prefetch-coverage-min 0.05 and 0.86–1.53 GB for O_DIRECT, because the window does not run as far past the reader's stop. The predicate is "already demanded", not "resident", and that is the whole fix. A random walk's holes are not resident at all and must not be discounted — #232 establishes there is no lever there and that prefetching for one adds bytes without moving the wall clock. lith's own unread over-fetch is resident but undemanded, and must not count either, or a strided reader would establish on regions its own window swept (#222's shape). Hole alignment cannot tell those apart — all three are 128 KiB-aligned, which is why the earlier proposal to key on alignment was wrong. Two conditions, covering different hole sizes. A hole is discounted only when another descriptor is open on the same object and its bytes were already demanded. The first is what keeps a lone strided reader out, and it is necessary because lith fetches a whole 1 MiB chunk to serve a 128 KiB read — so for holes smaller than a chunk a reader's own demand fetches make its own holes look demanded. The second holds the band between a chunk and a block, which nothing else covers. Holes wider than --block-size never reach either condition: the byte-gap gate forces Random before coverage is consulted.
--prefetch-pressure-max	derived: 0.85 where the evidence gate is off, off where it is on	Drops a prefetch dispatch when outstanding prefetch commitment already exceeds this fraction of the memory tier's realized capacity (#313). 0 (the default) decides from the evidence gate, which covers the complementary case: engaged at 0.85 where that gate is off — cross-region, or where the bucket's region cannot be determined — and off where it is on. In-region each handle is already bounded by its own consumed bytes, and this gate was measured never to bind there (peak pressure 0.50, zero holds; forcing the evidence gate off restores the collapse exactly — 3033 evictions, spread 3.94, committed 1.40 × tier). A negative value forces it off, including on a mount where the default would engage. lith serve nfs engages it on every export, because the gateway has no evidence gate at all — internal/nfs carries its own per-path detector and never bounds a window by what a reader has consumed, so this is an export's only admission bound. The band below was measured on a FUSE mount; the gateway inherits it by the arithmetic (same blockstore, same committed-bytes counter, same tier) rather than by its own measurement (#337). Why 0.85 and not 1.0: an evicted unread prefetch is a byte that was fetched and discarded, and this default engages only where the evidence gate does not — cross-region, where those bytes are billed egress rather than merely wasted request budget. So the trade is ~10 % wall against ~100 discarded-byte events, on exactly the mounts where a discarded byte has a price. In-region, where wall clock would be the only term, the gate is off entirely. One interaction to know about: the gate is sized against the tier, while the quantity it limits is independently bounded by --prefetch-budget, which defaults to half the tier — that is why one handle's steady-state pressure is ~0.50 and the gate stays silent until a concurrent start makes the divisor lag. Raise --prefetch-budget to or above 0.85 × --mem-cache and a single handle at full budget sits at the threshold, so the gate holds dispatches continuously and the mount throttles itself. The mount warns when the two knobs compose that way. On a c8gn.48xlarge the gate took the two former collapse cells from 19.6k and 25.5k unread evictions to zero, 5.4× and 6.3× faster. Measured band 0.85–1.0, 16 readers with the gate forced off: 1.0 gives 41 s against 116 s unbounded (2.8×) with 71–119 evictions left; 0.85 reaches 0–4 evictions for ~10% more wall; 0.5 over-throttles by 55% and buys nothing further. Above 1.0 it cannot work — it admits more unread bytes than the tier holds, so the tier still evicts; at 1.3 the gate fired and held pressure at exactly 1.300 as designed and evictions only fell from ~2900 to ~2100. It bounds bytes, not fairness: reader spread stays 2.3–3.1 at every threshold against ~1.05 with the evidence gate on, so it limits the collapse rather than fixing it.
--no-region-check	off (the check runs)	Suppresses the mount-time warning that the bucket's region differs from this instance's region. The check costs one IMDS lookup and is advisory — a cross-region mount is legitimate, it just must not be accidental. Measured on the report that prompted it: the same TB-scale sequential read ran at 1.52 MiB/s cross-region against 107–146 MB/s in-region (~70–96×), and the only symptom was slowness, which is indistinguishable from every other cause. It is worse on a random-fault workload, which already pays a round trip per miss (#232, #362). Silent when either region is unknown — IMDS blocked, or a custom --endpoint with no AWS region.
--prefetch-budget	50 % of --mem-cache	Bytes prefetch may hold un-demanded. Bounding it to half the tier stopped concurrent readers thrashing a small cache (#55).
--disk-cache	0 (off)	An on-disk second tier. Worth it only on fast local NVMe for working sets larger than RAM that you re-read; never on EBS/EFS/NFS.
--disk-path	$TMPDIR/lith-cache	Where the disk tier lives. Point it at your instance-store mount or /dev/shm.
--disk-writers	4	Write-behind workers that persist chunks off the read path, so disk writes never stall a reader.

--mem-cache is the tier, not the process footprint

In-flight prefetch is not charged against the cache it is about to land in, so resident memory is the tier plus the outstanding burst. Measured additive to 0.2%: a mount asked for a 20 GB tier on a 33.0 GB box and the kernel killed it at 30.56 GB RSS (20 + 13.078 = 33.08 GB against a 33.02 GB MemTotal). At stock defaults on that same box the footprint is 8.256 + 13.078 = 21.3 GB for a flag that reads 8.256.

The burst is bounded by concurrent readers × --max-readahead × --block-size, and because --max-readahead derives from the NIC while the tier derives from RAM, it grows with the link rather than with the memory available to hold it. Budget for both, or lower --max-readahead (#314, #313).

Sizing --mem-cache: budget ~2.5× your distinct working set, not 1×. The memory tier is 64 independently-evicting shards, with a chunk assigned by a hash of its key — so capacity is divided 64 ways and a shard evicts while its neighbours sit idle. Hash skew means the busiest shard holds well above the mean, and a cache that "just fits" by total bytes will re-fetch anyway. Measured by replaying a real workload's trace through the production tier (3.4 GiB distinct across 204 objects):

--mem-cache	headroom vs working set	re-fetched
24–32 GB	6.7×	nothing — eviction never fires
8 GB	2.2×	5.0 MiB
6 GB	1.7×	48.9 MiB
4 GB	1.1×	2.9 GiB (plus 251 prefetched chunks discarded unread)
2 GB	0.6×	16.5 GiB — more than double the run's entire S3 traffic

So the cliff is not at 1.0× but a little above 2×, and it is steep: between 1.7× and 1.1× the re-fetch cost rises by 60×. A pooled cache would hold this working set at 1.0×; the ~2.2× is what sharding costs. If you are tuning against a bill, measure the distinct bytes your job touches and multiply by 2.5.

Prefetch depth vs burst credits

How aggressively lith reads ahead. Deeper prefetch fills a fat NIC but can drain a burst-credit box's credits faster than it helps.

flag	default	why
--max-readahead	1.5 × the bandwidth-delay product	Sequential readahead window in blocks. The 1.5× is empirical, measured on a c8gd.16xlarge (#56); a raw-BDP window left the NIC underfed.
--inflight-bytes	2 × NIC baseline × 100 ms	Total bytes in flight to S3. NIC bandwidth is detected ethtool → EC2 DescribeInstanceTypes baseline → a fixed fallback, so burst-credit classes like c8g are now sized from their baseline automatically (#79); the result is cached in nic.json next to the index.
--nic-gbps	detect	Override the detected NIC bandwidth (Gbps) — it sizes --inflight-bytes, the readahead window, and the device-derived --parts-max/--coalesce-gap. Detection tries ethtool → cache → ec2:DescribeInstanceTypes → an IMDS size estimate → a 10 Gbps fallback. On AWS ParallelCluster and other least-privilege HPC roles, pass this explicitly: ENA reports no ethtool speed and the generated node role omits ec2:DescribeInstanceTypes, so lith falls to the IMDS size estimate (approximate) — --nic-gbps gives the exact baseline, and lith doctor WARNs when the NIC is undetected (#79, #237). Every source that fails now says why, in the startup log and in lith doctor (#317). Previously only the winner was reported, so a node whose DescribeInstanceTypes call had been denied looked identical to one that never asked — and the denial is one IAM action away from an exact figure. The reasons appear at WARN when the winning figure was not measured (fallback or imds-estimate) and at DEBUG/INFO behind a measured one. doctor now WARNs on the size estimate too, where it used to be an INFO inside an all-PASS report. Network-optimized instance families get no estimate at all. The estimate table is keyed on size, with no family dimension, so c8gn.48xlarge — rated 600 Gbps across two network cards — resolved through "48xlarge" to 50, a confident figure 12× low. For any family carrying the n attribute (c5n, c7gn, c8gn, m5n, m6idn, i3en, im4gn, …) lith now reports the NIC as undetected and WARNs, rather than reporting a number it cannot stand behind. That makes the figure lower (the 10 Gbps fallback) and the signal correct: pass --nic-gbps on these boxes. Families whose network far exceeds the table without an n — hpc7g/hpc7a, p4d/p5, trn1, and the flat metal entry — are a known residual; --nic-gbps is the answer there too. Pass the sustained baseline, not the "Up to N Gigabit" figure on the instance page — that is the peak. Overstating it (e.g. 15 for a c7g.4xlarge whose baseline is 7.5) over-sizes the readahead window and fetches bytes that are never read, for no speed gain: 654 MB fetched against 382 MB for the same 110 MB of data, with an indistinguishable wall clock — 42 % more bytes for zero milliseconds (#239). The mount itself now warns, not just doctor — the mistake is made on the mount command line, and doctor is opt-in. It fires only when --nic-gbps is set, naming the baseline to use instead. doctor also prints both figures (7.5 Gbps baseline (peak 15.0 — --nic-gbps wants the baseline, not this)), where it used to print one unlabelled number. The warning only compares against a figure lith actually measured — ethtool, DescribeInstanceTypes, or a cache of one. It stays silent against the size estimate and the fallback, because those are assumptions: on a c8gn.48xlarge the estimate reads 50 Gbps against a real 600, so "your 600 is well above the detected 50" would have the operator right and lith wrong (#317).
--s3-concurrency	128	Hard cap on concurrent S3 requests. 128 won on both cold sequential and stride in the tuning grid (#40).
--prefetch-concurrency	--s3-concurrency	Sub-cap on concurrent prefetch fills; lower it to reserve request slots for demand reads under heavy prefetch.
--sibling-readahead	16	On a directory walked in key order, prefetch this many following small siblings whole (v0.2, #63) — the chunked-store path. 0 disables.
--sibling-window	4	How close in index order two successive opens must be to count as "walking" the directory.

Index

The namespace snapshot that makes metadata free.

The s3://bucket/prefix you pass to mount is the root of the filesystem: the mount shows only what lives under that prefix, with the prefix stripped from every path. An index whose own build root is that prefix — or any parent of it — can serve the mount, so one wide index backs many prefix mounts concurrently, with no rebuild. A prefix the index cannot cover (narrower than, or disjoint from, its root), or one with no keys under it, is rejected rather than mounted empty. lith index inspect prints the index's root:.

flag	default	why
--index-file	(auto-build)	Load a prebuilt index. Without it, lith mount auto-builds from the bucket, bounded by --auto-index-limit. Prebuild for large buckets and reuse across nodes — one whole-bucket or parent-prefix index can back many prefix mounts at once.
--auto-index-limit	5000000	Max keys lith mount will auto-index; above this, build explicitly with lith index build first.
lith index build --inventory	—	Build from an S3 Inventory manifest instead of a ListObjectsV2 pass — far cheaper for very large buckets.
lith index build --shard	—	List sub-prefixes concurrently (repeatable) to speed a build over a wide namespace.
lith index build --page-size	1000	ListObjectsV2 page size.
lith index refresh	—	Re-list the bucket and rewrite the index when keys have changed (lith never mutates the bucket; the index is a private cache).
lith index build --keys	—	Build from an explicit key list (one key per line; optional \t<size>\t<mtime>), for buckets that are GET-public but deny ListObjectsV2 (Common Crawl cc-index, nyc-tlc). Keys lacking size/mtime are HeadObject-ed.
lith index build --keys-from-manifest	—	Same, but the key list is fetched from an s3:// URL or local path (.gz transparent).
lith index build --keys-allow-missing	off	Skip keys that HeadObject reports 403/404 instead of failing the build.

Access

Authentication, endpoint, and how files present to the OS.

flag	default	why
--no-sign-request	off	Anonymous requests, for public buckets (RODA).
--requester-pays	off	Add the requester-pays header when the bucket requires it.
--endpoint	(AWS)	Point at an S3-compatible store.
--path-style	off	Path-style addressing for stores that need it.
--region	(resolved)	Resolved from the bucket if empty.
--allow-other	off	Let other users access the mount (needs user_allow_other in /etc/fuse.conf). Security: files are world-readable (0444/0555) with no per-object access control, so this exposes the entire mounted subtree to every local user regardless of the bucket's S3 ACLs — enable it only where that is acceptable.
--uid / --gid	your uid/gid	Owner reported for every file.
--exec	off	Report files as mode 0555 instead of 0444 (executables on the mount).

Credentials on a long-running mount

lith takes credentials from the AWS SDK's default chain — the same order the AWS CLI uses — and signs with whatever that resolves. What matters for a mount that runs for days is whether those credentials renew themselves, and that depends on the source, not on lith:

source	renews itself?	on a long mount
EC2 instance role, IRSA / EKS Pod Identity, ECS task role	yes	nothing to do
SSO profile, assume-role profile (AWS_PROFILE)	yes	an expired SSO session needs aws sso login; the mount picks it up without remounting
credential_process that returns an Expiration	yes	nothing to do
long-term access key (no session token)	does not expire	nothing to do
temporary keys in ~/.aws/credentials (written by saml2aws, gimme-aws-creds, a cron job…)	no, but lith re-reads the file every 5 minutes	keep the helper rotating the file; no remount
temporary keys in environment variables (AWS_SESSION_TOKEN exported)	no, and cannot	the mount fails every uncached read with EIO when the token expires

The last row is the one to avoid. A running process cannot be handed a new environment, so exported STS credentials — 15 minutes to 12 hours — put a hard deadline on the mount. lith warns at startup when it sees them, and lith doctor reports them as a warning rather than a pass. Use any of the sources above it instead: the simplest for an interactive user is a profile in ~/.aws/config with credential_process or role_arn, selected with AWS_PROFILE.

The file row is lith's own refresh path. The SDK reads a static credentials file once and keeps the result, so before lith re-read it, a helper rotating the file was never seen by a running mount. The 5 minutes is a third of the shortest session STS issues, so a rotated file is picked up with at least two-thirds of even a 15-minute session left.

If S3 does reject a request as signed with expired credentials, lith logs it at ERROR — at most once a minute, with the remedy for the source in use — rather than leaving it indistinguishable from any other failure. RequestExpired is deliberately not treated as expiry: it means the request's timestamp is stale, which is clock skew.

Observability

flag	default	why
--metrics	off	Serve Prometheus metrics only (/metrics) on an address (e.g. :9101): cache hits by tier, S3 bytes/requests, prefetch accuracy, uncovered misses, FUSE op latency, and the realized readahead window with its divisor (see below). No pprof (see --pprof).
--pprof	off	Serve Go net/http/pprof handlers on an address (e.g. 127.0.0.1:6060). Security: pprof exposes the process argv (/cmdline) and an on-demand CPU/goroutine profiling DoS (/profile, /trace) with no auth — bind it to localhost and never expose it to an untrusted network. Enabling it also turns on block/mutex profiling.
--pf-trace	off	Diagnostic (#262). One CSV row per read recording what the access-pattern detector saw and decided: fh,pid,key,off,len,blk,gap,path,state_before,state_after,window,dispatched,peak_window, preceded by a # header line with the config that produced it (block size, max-readahead, parts-max, small-file, coverage gate, evidence ratio) so traces are self-describing and comparable across runs. Group by fh: lith builds one prefetcher per open, so the handle — not the key — is the unit that makes decisions, and concurrent handles on one object otherwise interleave indistinguishably. path marks which read path served the row (window = the prefetcher was driven; parts = a whole-file parts fetch was already in flight; footer = a byte-exact projection plan replaced the window), so reads the detector never saw are marked rather than silently dropped — they are a large, non-random share of the bytes on a mount where #229 whole-fetches most large objects. LITH_PF_TRACE is the older env-var spelling; the flag wins. Unbounded, and serialized under one mutex — but measured at +0.5–1.6% wall on a 48-rank MPI job over ~60k traced reads, so the lock is negligible below roughly 10^5 reads/run and the traced run describes an untraced one. Budget for the file instead: one row per read (~3.5–4.1 MB for 60k rows).
--timeline-csv	off	Diagnostic (#70/#95): write a per-chunk demand-read timeline — join-wait, in-flight fill depth, and prefetch-dispatch→open lag — to this CSV on unmount. Opt-in; no effect on the read path when unset.

--max-readahead is an upper bound, not the window — watch lith_readahead_window_blocks. --prefetch-budget / --block-size caps it too, and the mount warns at startup when it does (#297).

It is also capped by how many readers are streaming at once: the depth is clamp(--prefetch-budget/--block-size / streaming-handles, 2, --max-readahead), where streaming-handles counts the handles the detector is actually prefetching for.

That divisor used to count open file descriptors, and that was #301. Per descriptor, across processes, including descriptors never read — so a job holding 256 files open drove its own and every other reader's readahead to the floor of 2 blocks whatever the flag said, at 6–10× the wall clock on bytes and requests differing by 0.4% and 3%. No byte or request counter could see it. Both production mounts of the workload that found it were running there, by way of 48 ranks × ~6 files they did not account for. Fixed by changing what the divisor counts: idle descriptors now contribute nothing.

The division itself is deliberate, and removing it was tried and reverted. Enforcing --prefetch-budget byte-exactly on the measured total instead — which bounds the same aggregate, to within 2.4% — made concurrent readers 5.66× slower. A share is an allocation discipline, not just a ceiling: 16 × 30 blocks covers sixteen readers shallowly, while 2 × 223 + 14 × 0 commits the same total and covers two, and first-come admission produces the second. Removing the rationing outright is worse again: 13.8–22.4× wall and 4.2–5.2× the bytes, with 81–92% of prefetch evicted unread.

The obvious fix was tried and reverted, which is worth knowing before you propose it. Admitting prefetch byte-exactly against --prefetch-budget where the quantity is measured removes the over-charge cleanly in isolation — 2.8× faster at 64 held descriptors and 4.5× at 256, identical bytes and identical GETs — and it bounds the same aggregate to within 2.4% of what the divisor realizes. On concurrent readers it was 5.66× slower. The division is not just a total, it is an allocation discipline: 16 × 30 blocks covers sixteen readers shallowly, while 2 × 223 + 14 × 0 commits the same total and covers two, and byte-exact first-come admission produces the second. So the repair changed what the divisor counts, not whether it divides — measured at 9.26× on the shape that found it (255 never-read descriptors took a streamer from 25.74 s to 2.78 s).

The mount logs all three bounds on outstanding prefetch with the binding one named (#298). One handle's window commitment (--max-readahead × --block-size), the mount-wide --prefetch-budget, and --inflight-bytes are derived from three unrelated quantities — an empirical multiple of the bandwidth-delay product, a fraction of RAM, and NIC × latency — and in the shipping default they disagree by 1.5×, with the smallest winning. That is why raising --max-readahead from 223 to 492 once measured +3%: both configurations were already against a ceiling neither of them set. Check the binding field before tuning any of them.

lith_readahead_window_blocks reports the realized depth and lith_open_handles the divisor; you need both, because a window of 2 could be a tight budget or a crowded mount and only the divisor distinguishes them.

The mount now tells you where the floor is. The prefetch bounds line reports full_window_descriptors — how many concurrent open descriptors can each hold the whole window — and floor_at_descriptors, the count past which everyone is at 2 blocks. At the shipping default on a 33 GB box those are 2 and 165, so a job with 288 descriptors open is well past it. And because that count includes descriptors other processes hold, which no startup line can predict, a WARN is also emitted the first time a share actually reaches the floor. If you see it, --prefetch-budget is the lever: it is the numerator of every reader's share.

Diagnosing a concurrent-start thrash: read lith_prefetch_unread_resident_bytes against --mem-cache. lith_prefetch_committed_bytes is charged at dispatch, before the prefetch and S3 semaphores, so it counts resident-unread plus queued-for-a-slot plus on-the-wire — and --inflight-bytes bounds only the last of those. At rest the two gauges are equal to the byte; in flight they differ by however much prefetch is queued or on the wire, which on one measured box reached 26.9 GB. Only the resident part can evict anything, so the collapse condition is resident-unread approaching the tier's capacity: at that point every arriving chunk must evict an unread one, there being nothing else left to take. Measured, that separates clean runs (≤ 0.84 of tier) from collapsed ones (≥ 0.9996) with a margin of 0.16, where the committed ratio separates them by 0.002 (#313, #320).

The window is also only a proxy for what --prefetch-budget bounds. The budget is about bytes prefetched and still resident unread; the per-handle window times the handle count is a stand-in for it. lith_prefetch_committed_bytes is the real quantity and lith_prefetch_budget_bytes its limit — measured but not enforced, because the thing that enforced them byte-exactly is the change that was reverted. Against the old open-descriptor divisor the proxy over-charged by exactly that count (tightness 1/N, matched to within 6.2% from N=1 to N=256): at 256 descriptors a mount was charged 4295 MB, held 17.8 MB, and was throttled ~9.5× for it. Two cautions when reading these. Do not compare committed bytes against window × handles: an estimator built that way returns the standing window by construction, so the comparison is an identity rather than a check. And committed bytes are not released when a handle closes, so on a mount whose working set fits the memory tier the figure ratchets upward — which is the second reason byte-exact enforcement could not ship (#301).

committed, not resident: the counter increments at dispatch, so it includes bytes whose GET is still in flight. That is the right quantity for admission control and the wrong one for a memory limit. The mount also warns at startup when the configured depth is already unreachable (#297) — but that check is single-handle, so it cannot see the divisor, which is dynamic.

Reading lith_prefetch_used_total / lith_prefetch_issued_total. It is a chunk-touch rate, not a byte rate: a 64 KiB demand read marks the whole 1 MiB chunk used. So the ratio overstates byte follow-through, and overstates it most for the low-coverage readers where prefetch is least useful — a measured case reported 89% by this ratio while at most ~25% of the prefetched bytes were ever read (#256). Use it to compare like with like, not as an efficiency.

And lith_prefetch_issued_total is blind to the failure it looks relevant to. Three independent measurements have now found it identical across arms whose wall clock differs 3×: in one, 57045 issues in all eight cells. Evicted blocks come back as uncovered demand reads, not as new prefetch issues, so the counter that moves is lith_prefetch_uncovered_total — 256 in the fast arms against ~5150 in the slow ones — alongside lith_prefetch_evicted_unread_total. If you are diagnosing a thrash, read those two and ignore issued (#313).

mmap with a random access pattern is the one shape lith cannot help, and the cost is round trips rather than bytes. A program that mmaps an object and touches scattered pages gets one synchronous page fault per miss, and a single-threaded fault stream is serial by construction — there is nothing for lith to overlap. Measured on a real mount: 3000 random 4 KiB touches over an 892 MB object took 326 s, which is ~109 ms per fault, or one round trip each (#232).

The bytes are a separate and smaller story, and most of them are not lith's choice. Measured offline, lith is faithful to the read it is handed — 0.8× the requested bytes at the kernel's own 128 KiB readahead size — but its floor is one 64 KiB extent, so a 4 KiB read costs 64 KiB (15.3×) and a 4 KiB touch costs whatever the kernel decided to ask for. On the reported run that was ~220 KiB per touch, i.e. ~55× against what the program wanted, of which only ~3.4× was lith's extent rounding. Reducing lith's granularity would therefore address a small fraction of the bytes and none of the wall clock. If you have this shape, O_DIRECT bypasses the page cache and was measured restoring normal behaviour on a related issue (#316).

A handle that never prefetches is not necessarily a scattered walk. Check lith_prefetch_low_coverage_total. Concurrent readers of one object have each other's reads served from the shared kernel page cache and never reach lith, so each handle sees a punctate offset stream, the coverage gate classifies it as a walk, and nothing is prefetched. Measured at 243× slower than a single reader with byte amplification of 1.001 — the cleanest byte count in the whole gate (#316). If that counter is climbing while lith_streaming_handles sits at zero and many readers share a file, this is the shape you are in.

"Low coverage", not "scattered" — a correction. Through most of #256 this page and lith's own comments described the losing case as small scattered reads. A key-level fit over two mounts of one workload showed that was reading the wrong column. The two mounts had the same read size (104 vs 122 KiB median), both were sub-chunk on every single read, and both walked their objects monotonically end to end — so sequential was the correct classification and the detector was not fooled. What separated them was how much of each object the workload ever touched: 16% against 65%. Coverage — not read size, not read order, not reader count, not interleaving — is what predicts the over-fetch (ρ −0.69 on both arms, against a permutation null centred at +0.17). The practical consequence: when judging whether a reader will over-fetch, measure union(bytes read) / object size, not the read size. For a byte-level answer, lith_fill_bytes_total{kind="demand"|"whole"} splits demand fetches from readahead exactly, and --pf-trace plus an offline replay gives per-handle byte follow-through.

Experimental

Off by default; enable only if you have measured a win on your own workload.

flag	default	why
--footer-tier2	false	Experimental, clustered projections only. Byte-precise projection fetch for footer-family containers (Parquet/ORC/Arrow/zip) — a touched Parquet row group's projected column chunks, or a read zip entry + the next few in directory order (#108). Measured (sessions 29–43): it wins on bytes for a clustered projection (adjacent columns — ~10× fewer bytes than whole-file streaming) but loses on wall-clock for a spread projection, where the achievable floor is the reader's own footprint (pyarrow fetches ~the same bytes) and a whole-file stream is faster on a fat pipe (a handful of GETs at line rate vs a round-trip per column region). So it is off by default and stays experimental. Leave off unless you have a clustered projection where bytes-saved is the goal. Tier 1 (footer + head prefetch on open) always runs regardless.
--prefetch-coverage-min	0 (use the characterized 0.5)	Experimental (#316). The #221 coverage threshold: a handle establishes readahead only once its trailing 16 reads cover at least this fraction of their own byte span, and below it the handle is held provisional and prefetches nothing. The default comes from one characterization (streams ≥ 0.89, scattered walks ≤ 0.07) and does not account for concurrent readers of one object — see below.
--readahead-evidence-ratio	0 — decide from the region pair	Bound a committed readahead window to this multiple of the bytes the handle has actually read, so a single reader that wants a slice of a large object stops prefetching the whole thing. The default decides for you from the REGION PAIR — ratio 2 when the bucket is in this instance's region, off when it is not, and off when neither can be determined (off EC2, IMDS blocked, or a custom --endpoint). One IMDS lookup, no extra S3 call. Pass a positive value to force a ratio, or a negative value to force the gate off. This row described a latency-derived policy until recently, and that policy is gone (#349): first-byte latency is downstream of the gate's own decision — gate off → deeper burst → higher latency → gate stays off — so the same workload came out at 9.1×, 5.1× and 1.05× over-fetch on three identical cells. A mount's amplification has to be reproducible from its configuration, and the region pair is; a measured latency is not. The ratio was 4 until a 2-vs-4 sweep across nine shapes found 2 free or better everywhere (#407), including the shape it was originally set on: #284's single-variable NetCDF-4 read went 59.8 → 51.4 MB and six concurrent readers 358.6 → 308.3 MB, 0.86× bytes in both reps, with ratio 4 reproducing the historical figures to 0.1 MB as the control. The six-reader cell stayed exactly 6× the single-reader cell at both ratios, so the ratio does not interact with the per-handle divisor. netcdf4_hyperslab, webdataset, concurrent_handles and a single-stream dd were byte-identical at either value. Ratio 1 is not better than 2 — measured identical on all three walk shapes, because by then every window is at the 2-block dispatch floor and the ratio has stopped being the binding constraint. Takes 54× over-fetch to 2.28× on the shape that motivated it; costs a fixed ~0.07–0.18 s on a fast consumer reading most of an object. See below.
--readahead-evidence-per-run	off	Experimental. Makes the evidence gate measure the current run's consumption instead of the handle's lifetime total (#407). --readahead-evidence-ratio bounds a window by the bytes a handle has read — but that figure is cumulative and never reset. On a scattered-but-locally-sequential walk (a GRIB .idx field sweep, an HDF5 chunk walk, a FITS cutout's row bands) every run is 0.5–2 MB, so by the third run the handle has earned a window no single run can justify, and it re-places that bet at every re-establishment. An external shape sweep measured the consequence directly: 48 chunks prefetched and 2 used on GRIB, 71/11 on HEMCO, 47/13 on FITS — and a peak window of 5 blocks that this arithmetic reproduces exactly (11.9 MB consumed × 4 ÷ 8 MiB = 5.69 → 5). Those figures were measured at ratio 4, which is no longer the default — and at the current in-region default of 2 this flag does nothing on that shape. 11.9 MB at ratio 2 over an 8 MiB block earns 2.84 blocks, which floors to the same 2 blocks a 0.5–2 MB run earns, so both bases return the identical window. Discrimination starts at a lifetime consumption of 3 × block ÷ ratio = 12.6 MB at the defaults, and the measured shape sits 5.2% below it — so a sweep of that shape is inert by arithmetic and reports nothing about this flag in either direction. Above the boundary the cumulative basis grows without bound (4 blocks at 20 MB, 23 at 100 MB) while a short run stays pinned at 2, which is the case the flag exists for. A run ends at a seek out of an established pattern, a scattered landing, or a de-establishment. A genuine sequential copy never reaches any of those, so its run is the whole file and this is a no-op for it — that is the witness three cold-start fixes were refuted against (#231), and the reason the reset sits at run boundaries rather than on a timer. Necessary and not sufficient. The dispatch floor is 2 blocks — 16 MiB at the default block size — so a 1 MB run still floors far above what it wants. Closing the rest needs --readahead-subblock-dispatch, below, which bounds the same residual in bytes. The flag exists so the two halves can be measured separately, because the one precedent for this pair (#222) found each half alone made bytes worse and only both together gave 16×.
--readahead-subblock-dispatch	off	Experimental. Allows a readahead dispatch shorter than one block when that is all the handle's evidence earns (#407, arm B). The evidence gate is denominated in blocks and floors at 2, so the smallest window any established handle can be given is 16 MiB at the default 8 MiB block — while a GRIB .idx field sweep, an HDF5 chunk walk and a FITS cutout's row bands all read 0.5–2 MB per run. This converts that residual from blocks into bytes: a handle that has earned less than the floor is dispatched a contiguous span of exactly what it earned, placed immediately after the read that earned it. At ratio 2 a 1 MB run asks for 2 MB instead of 16 MiB. It only engages for a handle that has finished an established run, which a sequential copy never does. That precondition — evidence, not a threshold — is why it cannot reach a cold streaming reader, which #233 says should fetch more rather than less. Today the 2-block floor is what protects that reader, so an arm that removes the floor has to carry the protection itself. It engages for a handle that has finished an established run or has collapsed to Random at least once. The second route is what makes it reach real workloads: an external 2×2 on v1.14.0 found the arm inert on every measured shape, because the finished-run route alone describes a fixture and not the tools — a GRIB handle establishes once, in its last 12 of 90 reads, and finishes no runs at all, while 67 of its 90 reads are random→random. A plain sequential copy satisfies neither route, which is what keeps #233's cold streaming reader out: the #221 coverage gate holds a copy's first reads provisional as Cold, and it never passes through Random. Measured on a scatter-then-stream witness — a handle that reads scattered and then streams, which is the shape that makes this risky — the cost to the stream is zero and does not scale with stream length: the span fires once at establishment and the window ramps back. At this site the basis is always the current run, not --readahead-evidence-per-run's choice, because at such a handle's establishment its lifetime consumption is evidence about scattered reading rather than contiguous appetite. So the two flags are independent, and this one does not need the other. Measured externally on v1.15.0, two arms, two reps, default reproducing the previous release exactly as the control:
--demand-precise-nontiling	off	Experimental. Serves a demand read precisely — its 64 KiB extents rather than its whole 1 MiB chunk — when the coverage signal has judged this handle's access scattered, even though its block-space state is still Cold (#408). The shape it is for. A ZIP read of a 10 MB archive — central directory, then 20 random 20 KiB entries — fetches the whole archive. The byte-exact demand lane exists and these reads are well under its threshold (32 KiB against 1 MiB), but the lane keys on Random or Strided, which are denominated in blocks — and a 10 MB object over 8 MiB blocks is 1.25 blocks. Measured, 20 of 22 reads land in the same block, 1 adjacent, 0 further, so the handle never leaves Cold and the lane never engages. Its trailing coverage meanwhile is 0.056 against the 0.50 threshold with the window full: the block-space machine has nothing to say and the byte-space signal is emphatic. With this on, that shape fetches 3.3 MB instead of 9.4 MB. ★ It is off by default because it trades bytes for round trips, and the arithmetic favours the round trips. The same shape's fills go 9 → 16: whole-chunk fetching amortizes faults for a scattered reader whose entries cluster — 20 entries across 10 chunks means two often share one fetch. So 6.2 MB saved buys 5–20 ms of transfer while 7 extra round trips cost 210–990 ms at a 30–141 ms first byte. Enable it only where egress cost dominates wall clock: in practice a cross-region or cross-cloud mount reading archives it touches sparsely. What it cannot reach. A tiling stream, whose coverage is ~1 — and that guard is the pre-existing #233 witness rather than something invented for this, which a revert sweep confirmed catches a version that drops the window-full requirement. Nor a handle with a committed window: the coverage judgement and establishment are mutually exclusive by construction, so nothing is prefetching for a handle this matches and a precise read cannot land on top of a whole-block prefetch — #222's measured-worse case.
--coalesce-gap	0 (derived)	Largest gap between two projection/demand fill ranges still merged into one range GET (the byte-precise fill path; only active with --footer-tier2). 0 derives it from the device: NIC baseline × measured first-byte latency ÷ usable concurrency, clamped to [256 KiB, 64 MiB] — a round-trip's worth of bytes amortized across the concurrent requests in flight (#124/#31). Set a fixed size to override the derivation.

The Go runtime is on the scrape as of #350. go_gc_duration_seconds, go_sched_latencies_seconds, go_memstats_alloc_bytes_total, go_goroutines and process_cpu_seconds_total are registered alongside lith's own series. They were absent for the project's whole life, which is why five hypotheses for an in-mount first-byte latency were refuted one at a time without the runtime ever being on the list — it could not be looked at. Note that a wire-level first-byte measurement does not rule the runtime out: httptrace's callback fires on the transport's goroutine, so it is immune to the fill goroutine's scheduling and not to a runtime-wide pause or assist.

Reading lith_readahead_evidence_ratio. It reports the evidence-gate ratio in force, which is not necessarily what you set. 0 means the gate is off — which is the default, and is also what a mount reports before any fill has measured the endpoint's first-byte latency. A non-zero value with no --readahead-evidence-ratio set means the latency-derived policy engaged. To tell the two 0 cases apart, read lith_ttfb_measured alongside it; lith_ttfb_median_seconds is the value the policy decided on.

--prefetch-coverage-min, and why concurrent readers of one file never prefetch

Lith opens with FOPEN_KEEP_CACHE, so every descriptor on an inode shares the kernel page cache. When several readers stream the same object, one reader faults a range in and the others' reads of that range are served by the kernel and never reach lith. Each handle therefore sees an offset stream full of holes — all multiples of 128 KiB, the kernel readahead unit — and its trailing coverage is roughly 1/N.

Against the default 0.5, that means establishment dies between 2 and 4 concurrent readers, which is where 1/N crosses the threshold. Measured on 16 readers of one 3.76 GB object: lith_streaming_handles stays at 0, nothing is prefetched, every read becomes a demand GET, and the run takes 776 s against 3.19 s for one reader alone — 243×. Byte amplification is 1.001, the cleanest count in the whole experiment, so no byte or request counter can see it. With O_DIRECT the same 16 readers establish 16/16 and finish in 2.1 s.

On a real workload this is present but is not where the bytes are. A 48-rank GCHP run found all 12 met objects had interleaving sibling handles, and 252 of 262 substantive handles had one — yet handles below 0.5 coverage carried only 8.8–9.0% of read bytes. So lowering this threshold is worth single-digit percentages there, not 243×.

The trade. Lowering it admits those handles — and also admits genuinely scattered walks, which is what the gate exists to stop: a FITS 2-D cutout over-fetched 4.76× through exactly that path (#222). There is no setting that is right for both, which is why this is a flag and not a new default.

Diagnosing it needs lith_prefetch_coverage_held_total rising while lith_prefetch_low_coverage_total stays flat: contiguous progress refused a window, as opposed to a scattered landing forced Random. The held counter ticks at block boundaries, not per read, so its magnitude is roughly bytes read / --block-size when a handle is fully starved.

--readahead-evidence-ratio, in detail

A handle establishes at its first block crossing — one block (8 MiB) of contiguous evidence — and that buys the full NIC-derived window, ~223 blocks (~1.8 GB) on a 50 Gbps node. With ratio k a handle prefetches ~k× what it has read, so a sequential copy still earns the full window (after consuming max-readahead × block-size / k) while a reader that tiles a slab and jumps does not.

The price of stopping early is about 1 + ratio, and that is not a defect

The ratio's cost falls almost entirely on readers that stop before the end of the object, because whatever the window had run ahead is never used. Measured locally on a plain sequential reader that simply stops, against a 256 MiB object with 8 MiB blocks:

reads to	--readahead-evidence-ratio 1	ratio 2 (default in-region)	gate off
17 MiB	2.35×	3.29×	15.06×
34 MiB	2.12×	3.06×	7.53×
68 MiB	2.00×	2.94×	3.76×

So amplification ≈ 1 + ratio, stable across stop points, against 3.8–15× with the gate off. This is the designed trade, not a bug: the lead is what makes a stream fast, and a reader that abandons it pays for it. If you know a workload reads small slices and stops, the lever is the ratio (or -1 to turn the gate off and --max-readahead to bound the window directly) — not a different mechanism.

It is also the yardstick for judging an over-fetch report. A shape fetching ~3× at ratio 2 is at the price of stopping early; only a shape fetching materially more than that has something else wrong with it. Three externally measured rows were grouped as one over-fetch family for weeks before this was priced, and two of them turned out to be at or below this line (a FITS cutout at 2.20× and a HEMCO time-slice read at 1.79×), while the third (a GRIB .idx sweep at 3.02×) was over it for a specific and different reason — see --readahead-subblock-dispatch.

How much it helps depends on how many readers are streaming at once. The window is clamp(prefetch-budget / streaming-handles, 2, --max-readahead). The measurement below was taken when the divisor counted open descriptors instead, so a 48-rank job with ~600 open handles sat on the floor of 2 and the gate had almost nothing to give back — amplification 2.425× → 1.956× at k=8. Since #301 only handles actually being read sequentially count, so that job divides by ~48 rather than ~600; the numbers here have not been re-taken against the new divisor. A single reader gets the full ~223-block window, which is larger than most single objects, so an established sequential reader prefetches the whole file however little of it it wants. On one process reading one variable of a 1.22 GB NetCDF-4 file, over-fetch is 54×, and over-fetch equals 1/coverage to within 2% across a 54× range of coverage. At k=4 that becomes 1217.8 → 59.8 MB (−95.1%) with byte follow-through rising 0.005 → 0.135. The single-handle case — python, xarray, ncks, h5py, one process against a bucket — is both the worse case and the more common one, so an earlier claim here that the flag "does not improve prefetch precision" was a 48-rank measurement stated as a general one.

What it does not touch. The cold-start granularity tax (the ~1.23 GB floor of #256) is bit-identical with the flag on, because the whole-chunk commitment is taken before any window decision exists. There are two distinct waste channels and this addresses exactly one.

The high-latency cost is #229's establishment burst being capped, and it cannot be tuned away. On an object larger than the window (3.78 GB = 450 blocks) at 58.6 ms RTT the effect is a clean, fully separated factor — median 8.51 s → 21.74 s (2.56×) at k=4, zero overlap across n=8 per arm (exact rank-sum p = 1.6×10⁻⁴) — while moving identical bytes in identical request counts (3,776,834,855 B, 465 GETs) in every cell. In-region the same effect is +7.5% (n=8, p ≈ 0.03).

The mechanism is a single dispatch event. advance(cursor+1+window) emits [frontier, target) in one call, so the window value at establishment is the batch size — and #229 exists to make that batch the full window, precisely to avoid "many small GETs, an underfed NIC on the cold read". With the gate off, one read dispatches 223 blocks at once and every later dispatch is 1 block (p99 = 1); that single burst is the whole of the throughput advantage. With the gate on the burst is capped to what consumption has earned, and nothing later recovers it:

k	burst at establishment	throughput	concurrent streams' worth	median wall
off	223 blocks	443.8 MB/s	3.10	8.51 s
40	41	283.8 MB/s	1.98	13.31 s
4	5	173.7 MB/s	1.21	21.74 s
1	2	160.4 MB/s	1.12	23.55 s

One stream is 8 MiB / 58.6 ms = 143 MB/s and a fully serial read is 465 × 58.6 ms = 27.2 s, so k=1 at 1.12 streams is essentially serial. That is also why raising k does not rescue it: concurrency cannot fall below one stream, so a 40× range in how long the cap binds buys only a 3.1× range in the penalty. The gate bounds committed readahead bytes, and on a long pipe committed bytes are what buy concurrency — so the byte saving and the burst are the same quantity, and no ratio keeps both.

Earlier reports of a bimodal 4 s / 20 s split came from a test object smaller than the window, where readahead extent was never the binding constraint at all.

A bytes-and-requests regression check is not sufficient for this flag. Those are identical on both sides of the effect, so such a check passes while wall clock varies by a factor of 2.8.

So: use it in-region, or where you are paying for bytes and can afford the wall clock. It is not a fix for the granularity floor, and on a high-latency or non-AWS endpoint it costs 1.6–2.8× on reads of objects larger than the prefetch window — at every ratio tested, including one permissive enough to clear the cap after 1.1% of the object.

Since #284 this is on by default in-region, and that decision is measured. Its cost falls entirely on one shape — a fast consumer reading most of an object, where the shallower ramp becomes the bottleneck — and that cost is sharply RTT-scaled:

endpoint	slice reader	whole object, fast consumer	whole object, slow consumer
in-region, 2.2 ms	wins	r = 1.05–1.21	—
cross-region, 58.6 ms	r = 0.84, wins both axes	r = 2.35 (zero overlap)	r = 1.00

Five cells, two boxes, warm and cold mounts. So the gate defaults on where it is cheap and off where it is not. The bound is a first-byte latency, not a round trip — v1.4.0 shipped 5 ms, derived from the 2.2 ms RTT, and in-region first-byte latency is 28.2 ms median (p90 42.7), so it never engaged (#340). It is now 50 ms, anchored on both sides: above the whole measured in-region distribution, and below the 58.6 ms cross-region round trip that a first-byte latency cannot undercut. Read lith_ttfb_median_seconds to see which side a mount is on, lith_ttfb_measured to tell "not measured yet" from "measured and fast" (the median reads 0 for both), and the lith_ttfb_seconds histogram for the spread — lith_ttfb_seconds_bucket{le="0.05"} against lith_ttfb_seconds_count answers "will the default engage here" without estimating a quantile.

The ramp has a second job nobody designed it for (#381). A new connection's S3 first-byte time starts at ~100–150 ms and settles to ~30 ms over roughly its first second of traffic — a per-connection, ramp-like cost, not a setup cost (one request per connection does not fix it). A client that opens many connections at once and measures immediately eats it in full: a 32-connection burst reads 22% of first bytes under 50 ms, against 92% after a one-second warm-up.

The evidence gate's ramp spreads connection births across the burst, so a mount with the gate on largely escapes it — measured, with the gate off lith opens 83% of its fills on new connections every rep and comes in up to 49 points colder on the same box at the same instant, with identical bytes fetched. The same measurement also explains the small reused-connection queueing cost from #350: gate-off has zero (0.01 ms every rep) against 0.76–2.26 ms gate-on, so the ramp both protects against the cold cost and causes the queueing. One mechanism, two signs.

The consequence for gate-off mounts — cross-region, or where the region cannot be determined — is that they open their connections at once and pay the cold cost on every fresh mount. It decays within about a second of traffic per connection, so it is a one-second tax on a long read and most of a short one. That trades against the measured 1.96× ramp penalty for a fast whole-object reader at distance, and the net is unmeasured: do not set --readahead-evidence-ratio either way at distance on the strength of one of these alone.

The input is the REGION PAIR, not latency (#349). The gate engages when the bucket is in the same region as the process, and not otherwise. That is one IMDS lookup and no extra S3 call — the client resolves the bucket's region in order to sign at all — so the gate's state is a function of configuration, identical on every read of a mount's life.

It took three releases to get there, and first-byte latency was the wrong input rather than a bound that was wrong by a factor:

release	bound	what was wrong
v1.4.0	5 ms	derived from a 2.2 ms round trip; the quantity is a first-byte latency (~28 ms in-region), so it never engaged
v1.5.0	50 ms	anchored on both measured sides — and the quantity itself is load-sensitive
now	the region pair	—

The median reads ~28 ms idle in-region and ~100 ms during the mount's own prefetch burst, above both the old bound and the 58.6 ms cross-region round trip its far side was anchored on. Worse, it is downstream of the decision: gate off → unbounded window → deeper burst → higher latency → gate stays off. Measured, the same workload came out at 9.1×, 5.1× and 1.05× over-fetch on three identical cells, with the gauge reading on for 50–81% of ticks depending on the run. A load-invariant floor (p10 over 256 fills) was built to escape that and inherits it one burst later, rising 18 → 74 ms within 0.5 s of the gate turning off. Any statistic of lith's own fills has this shape.

What the region pair encodes is a measured split: in-region, forcing the gate on took six concurrent slice readers from 3281 / 1827 / 375 MB to 358.6 MB in 3/3 (r = 1.00) at no wall-clock cost; cross-region it cost r = 1.96 with zero overlap (min forced 15.80 s > max off 12.07 s, n = 4, p = 1/70) on a fast whole-object reader.

When the region cannot be determined the gate is off — off-EC2, IMDS blocked, or a custom --endpoint with no AWS region. Off is correct in every case measured, since off-EC2 is far and non-AWS endpoints cost 1.6–2.8× at every ratio tested. An on-prem MinIO is near and loses the saving; --readahead-evidence-ratio 4 is for that. lith_ttfb_median_seconds, lith_ttfb_floor_seconds and the lith_ttfb_seconds histogram remain as diagnostics — they are good at describing an endpoint and were not good at deciding this.

Cross-region with several concurrent readers: set the ratio yourself

With the gate off, concurrent readers are not served evenly once the memory tier is contended (#391). The only bound left is the tier-pressure gate, and that is a ceiling on the total: it bounds waste and admits first-come, so the first readers to establish take the budget. Measured cross-region (us-west-2 reading us-east-1), 16 concurrent readers with demand 8× the tier:

	spread (slowest ÷ fastest)	last reader
gate off (the cross-region default)	2.70–3.18	17.7–22.4 s
--readahead-evidence-ratio 2, 4 or 16	1.17–1.63	15.2–16.5 s

Same bytes; the job simply finishes sooner because the last reader is no longer starved. When the tier is not contended there is no unfairness to fix — the same 16 readers against a tier that holds their working set came out at 1.27–1.32 gate-off.

So if a cross-region mount runs many readers at once against a tier small relative to what they read, set --readahead-evidence-ratio in the 2–16 range. Not 64: that was ~3× slower for a single reader in 3 of 5 reps. It is not the default because the reason the gate is off cross-region — a measured 1.96× slowdown for one fast whole-object reader — was not refuted by this cell: the box it ran on is latency-bound at ~375 MB/s per stream, which is the configuration least able to show a ramp penalty. Raw data and scripts: bench/x391/.

The cost is fixed, not proportional. Across a 14.5× object-size range the penalty stayed at +0.07–0.18 s; proportional would have made the 3.78 GB case +2.0 s and it measured +0.14 s. The ratio stays bounded (~+24% worst case in the exposed band) because the baseline carries its own fixed cost of roughly 0.5 s per object — which is unexplained, is 72% of a 260 MB read's wall, and is not the GET count: a cold sequential 64 MiB read issues 22 GETs, about 48 ms even fully serialized at 2.2 ms (#233).

What it does not touch. Objects at or below --parts-max take the whole-file parts fetch and never reach the windowed path. A handle the detector has not established never consults the cap, so a scattered or random reader is unaffected — and evidence_held = 0 in a trace can mean the #221 coverage gate got there first rather than that this one did not bind. The two are in series, coverage first.

Reading it: lith_readahead_evidence_ratio reports the ratio in force, which is not necessarily what you set — 0 includes a mount that has not yet measured its endpoint. lith_prefetch_evidence_clamped_total and lith_prefetch_evidence_withheld_blocks_total are per-read as of #322; before that they were folded in at close and read zero for the whole life of a job that holds its handles open.

Archives and published datasets

Mount a CargoShip archive, or a published dataset by name.

flag	default	why
--cargoship	—	Mount/serve a CargoShip 2.1 archive: build the index in-process from this manifest s3:// URL and present the archive's original file tree (reads fetch only the covering zstd frame(s)). Fails closed — a manifest that cannot be resolved is an error, never a fallback to listing.
s3://bucket/dataset@current / @<version>	—	A published-dataset pointer mount (no flag): resolve the atomic CURRENT pointer (or a pinned version) and mount its prebuilt index. See Published datasets.
lith refresh <mountpoint>	—	Re-read CURRENT and hot-swap a FUSE mount to a new version (explicit; no polling). The gateway does not hot-swap — restart it to adopt a new version.

Gateway (lith serve nfs)

flag	default	why
--listen	:2049	Listen address for the NFSv3 + embedded MOUNT service. The gateway never registers with portmap/rpcbind; clients mount with an explicit port=/mountport=.
--client-idle	5m	Drop a client from the registry (and from the clients gauge) after it has been idle — since its last MOUNT — this long. It is no longer the readahead divisor; see --read-idle (#337).
--read-idle	30s	How long a path counts as an active read stream after its last read, which is what the readahead window is divided by (#337). The divisor used to count MOUNTED clients, so a client that sent one MOUNT and read nothing held a share of the prefetch budget for --client-idle. Measured locally with no client reading a byte: 1 mount → 96 blocks, 8 → 61, 48 → 10, 246 → the 2-block floor. Measured externally on a real gateway, 47 such clients against one streaming client: 1267–1300 MB/s with one client, 519–558 with 48 — 2.4× — with byte-identical transfers in every arm (3.777 GB, 458 GETs, prefetch issued = used), so it was purely window depth and nothing was being wasted. A byte or request counter could not see it. The reported defect is fixed for any value of this flag, because a client that never reads creates no per-path read state at all. What this governs is a second case: a path a still-mounted client read a while ago and has not returned to. Some window is required rather than optional — the only other bound on those paths is the LRU cap on the state map (8192), which over a long-lived gateway is a worse divisor than the client count ever was. Too short and a client pausing between reads is shed and re-granted repeatedly; too long and this becomes the bug it replaces. 30s is ~6× the longest read pause measured on a real HPC workload (a 5.0 s ExtData time-slice cadence, #312) and 10× shorter than the default --client-idle. It is a judgement, not a device-derived figure: the time to drain one window at the NIC rate is 0.04–0.4 s on a 15 Gbps box, which would shed a reader on any compute pause at all.

Operational and diagnostics

flag / command	default	why
--log-level	info	Log verbosity: debug/info/warn/error (on mount and serve nfs). debug surfaces the format-plan diagnostics.
--daemon	off	lith mount forks into the background once the mount is ready.
lith doctor [s3://…] --mountpoint	—	Diagnose whether lith will work here — credentials, bucket access/region, LIST, FUSE, and the given --mountpoint (exists, a dir, owned by you, empty). Exits non-zero on any failure. See the doctor.
lith mounts --prune	off	List active mounts; --prune drops stale records whose process is gone.
lith umount --all / --force / --timeout	—	Unmount one, or --all; --force lazy-unmounts a busy mount; --timeout bounds the wait.

The doctor

lith doctor runs the checks above (add a bucket to include bucket/region/LIST/pointer checks, --index-file to validate an index) and prints each as PASS/FAIL/N/A with a one-line fix. Run it first on a new box — it catches the failures that otherwise surface as a cryptic mount error (wrong region, a root-owned mountpoint, a LIST-denied bucket, missing FUSE).

Benchmark (lith bench)

flag	default	why
--against	—	Comparison reader to run alongside (e.g. mountpoint-s3) for an apples-to-apples table.
--pattern	—	Access pattern to exercise (seq, random, projection, …).
--ops / --runs	—	Operations per run / number of runs.
--readers	1	Concurrent readers (multi-reader mode; requires --objects).
--objects	—	Comma-separated keys for multi-reader mode.
--cache-dir	$TMPDIR	Directory for the bench block cache.