Copy or mount?¶

Your dataset is in S3. You have a job to run against it, a box to run it on, a bill to pay, and maybe a deadline. The reflex is to aws s3 cp it to a volume first. Sometimes that is right. This page tells you when.

The one number that decides it¶

Cost to result, in-region (transfer and egress are free):

cost = instance $/hr × wall-clock  +  staging storage  +  S3 requests

Everything below is about which term dominates for your access shape.

Copy pays staging time — the instance sits idle burning $/hr while bytes land on a volume you had to size, attach, and will have to delete — and it stages the whole object whether your job reads all of it or 1%. Once staged, the job can run on a tiny box and re-read for free.
Mount pays nothing up front: the job starts immediately and lith fetches bytes as the app reads them. In return it wants a NIC and RAM proportional to how fast the job reads. It never stages bytes you don't touch.

Staging storage and S3-request charges are rounding errors here (cents of gp3, sub-cent of GETs); the fight is instance-$/hr × wall-clock, so whichever path finishes sooner usually also costs less.

By access shape¶

Selective reads — region queries, Parquet predicate pushdown, a few Zarr chunks. Mount, at any box size. Copy must stage the whole file to answer a question about a slice of it; lith moves only the slice.

Full single-pass streams — read an object front to back once (e.g. flagstat). Mount. It is a tie with a local-NVMe copy when the app is CPU-bound (both wait on the CPU, not the bytes) — but mount still wins by not staging first, and an EBS copy loses outright to its own staging wall.

Batches — many objects, compute per object. Mount. lith overlaps each object's fetch with the previous object's compute; the copy path serializes staging in front of compute, and its compute then reads back from a throughput-capped volume.

A multi-TB object probed randomly at small record size — an mmap'd hash table, a large reference index. Neither. Copy is not an option at this size and the mount is not viable either, so the honest answer is to restructure the access or put the data on local disk.

This is the one row where the mount looks identical to the good cases right up until you measure throughput, which is why it needs saying out loud. Measured on a 1.206 TB RODA kraken2 database, in-region (#362):


`lith index build` over all 1.206 TB	~1 second, producing a 728-byte index
sequential read of the 1.1 TiB `hash.k2d`	146 MB/s — 62–86% of raw `aws s3 cp`
random 4 KiB `pread`	7.1 probes/s — 141 ms each

The index and the namespace are excellent at this size, and that is exactly the trap: nothing about mounting distinguishes a sequential TB-scale read from a random one. A serial fault stream pays one round trip per miss and lith has no lever on it (#232) — measured at 7.9 probes/s through the mount against 6.9/s for raw S3 at depth 1, so the mount adds no overhead; one synchronous fault at a time is simply slow.

What is recoverable belongs to the application: the same offsets at queue depth 64 ran 68× faster with per-request latency flat (145 → 137 ms). Of the 685× gap to local NVMe on that box, ~100× is concurrency and only ~6.9× is genuine storage advantage. An application that can batch its lookups and issue them asynchronously gets most of it back; kraken2 --memory-mapping and the bwa/samtools mmap paths cannot, because a page fault exposes exactly one offset.

Telling this regime apart from a slow mount, in one scrape¶

The reporter above lost most of a day to this, and the reason is worth stating: the mount was working correctly, so there was nothing wrong to find. What they needed was not a fix but a verdict, and every number for it was already inside lith and none of it was exported (#362). Now it is. With --metrics:

# what is one fault costing?            (expect ~1 round trip, and no better)
histogram_quantile(0.5, rate(lith_fill_seconds_bucket{kind="demand"}[1m]))

# what does this endpoint cost at best? (load-invariant by construction)
lith_ttfb_floor_seconds

# is this mount streaming, or faulting? (the ratio IS the diagnosis)
rate(lith_fill_seconds_count{kind="demand"}[1m])
  / ignoring(kind) sum without(kind)(rate(lith_fill_seconds_count[1m]))

Read it like this:

what you see	what it means
`demand` ≈ the floor, and `demand` is a small share of fills	healthy. Prefetch is covering the stream; the few demand fills are its cold start.
`demand` ≈ the floor, and `demand` is ~all of the fills	this regime. Every read is a fault paying one round trip, and the mount is already as fast as one fault at a time can be. Restructure the access or move the data.
`demand` far above the floor	not this regime — something is queueing. The floor is what the endpoint costs when nothing is in the way, so a large gap is lith's own concurrency or the link, not S3.

kind="demand" means a fill no prediction covered, which is why it is the right signal here and not a measure of randomness: a sequential reader whose prefetch is keeping up shows kind="whole". A strided reader — a FITS cutout, a hyperslab — shows kind="prefetch-range", which is a prediction that was bounded to the extents the next read is expected to want rather than the whole block (#222); seeing those bytes move from whole to prefetch-range is what that fix looks like from outside.

There is deliberately no threshold and no log line. lith reports the three numbers and the judgement is yours — two defaults placed from a plausible-looking constant have already had to be withdrawn from this project (#340, #349), and "is my workload in this regime" is a question about the workload, not about lith.

Many-small-object stores — Zarr, sharded datasets read in key order. The one shape where the answer is nuanced. Reading these cold pays a serial S3 round-trip per object. v0.2 shipped sibling readahead (#63) and the first format-aware plan — chunk-grid–aware Zarr prefetch (#70 tier 1, #87) that walks the grid in access order instead of flat key order. On the 1.4 TB chrtout.zarr yearly-mean the cold read is ~35 s, versus ~25 s for an in-place xarray+s3fs reader on the same box, same day (c8gd.4xlarge, us-east-1; bench/results/apps.csv @ 748b33d). lith's prefetch runs deep and leads, but the uncovered chunks at 2-D grid boundaries each still pay a full cold GET on the app's bounded compute pool while s3fs fires the whole selection at once; closing that gap is #70 tier 2 (whole-selection / grid-plane prefetch). If you read one chunked store repeatedly and can afford the RAM, mount and let the warm cache serve reruns (~22 s warm); if you read it once, cold, an async in-place reader is currently a touch faster.

Columnar files (Parquet). For a projection query in-region, lith streams the file and beats a byte-precise native reader (pyarrow) ~4× single-process; on small NICs it ties. This is counterintuitive — the native reader moves fewer bytes (only the projected columns) — but on a fat in-region pipe a whole-file stream is a handful of coalesced GETs at line rate, while byte-precise fetch pays a round-trip per column chunk through a filesystem that sees the reads one at a time. lith ships tier 1 (footer + head prefetch on open) on by default and streams the rest; a byte-precise projection path exists behind --footer-tier2 but is experimental and off (slower on every box measured; #108, #122).

mmap of a large object, walked randomly — a bwa/samtools reference index the tool memory-maps and page-faults through. Copy it local, or use the gateway / EFS — do not mmap it over a plain mount. A random page fault is a synchronous, serial round-trip: the faulting thread blocks on one S3 GET, and the next fault cannot start until it returns. 3,000 random faults over an 892 MB index cost ~14 s even in the best case and there is no readahead that helps — random access is by definition unpredictable. This is why the cookbook's bwa recipe puts the index on EFS. (lith fetches those faults byte-exact, which minimizes bytes but maximizes round-trips, so it is slower here than a coarser whole-block fetch would be — the opposite of the selective-read case, and the reason this shape is on the copy side of the line; #232.)

The crossover, measured¶

Cost to result, copy → compute vs mount with lith, across the EBS-only Graviton4 ladder (no local NVMe — the case that most favors copying). The copy path is given its best tool and tier: s5cmd tuned (~1 GB/s, 3× faster than aws s3 cp) staging to a gp3 volume provisioned to 1000 MB/s, which un-caps both staging and the read-back the compute does (#84). Each cell is copy $ / lith $ (wall copy / wall lith); lower is better. Full data: bench/results/crossover-v0.2.csv, method in #77.

access shape	c8g.large (2 vCPU)	c8g.xlarge (4)	c8g.2xlarge (8)	c8gd.4xlarge (16)	lith margin
selective (14 GB CRAM, region set, ~0.8 GB touched)	$0.0064 / $0.0012	$0.0054 / $0.0021	$0.0065 / $0.0042	$0.0146 / $0.0109	1.3–5× cheaper
stream (3.8 GB CRAM, `flagstat`)	$0.0036 / $0.0014	$0.0042 / $0.0022	$0.0069 / $0.0043	$0.0155 / $0.0103	1.4–2.6× cheaper
batch (8 CRAMs, 60 GB, parallel = vCPU)	$0.044 / —²	$0.023 / $0.0105	$0.023 / $0.0134	$0.052 / $0.040	1.2–2.2× cheaper

lith is still cheaper on every box — the crossover is at or below c8g.large for all three shapes; nothing flips. But given its best tool and tier the copy path is far more competitive than the naive aws s3 cp → baseline-gp3 it replaces: on the high-NIC boxes (2xlarge, 4xl) the margin collapses to ~1.2–1.5× (it was up to 5.6×). What remains of lith's edge is the staging it never does — now only 5–60 s on a fast box — plus selectivity (a region query still moves 0.8 GB, not 14). On the smaller boxes staging is NIC-bound, so the best copier is no faster there and lith wins by more (2xlarge/4xl measured; large/xlarge estimated).

Against mountpoint-s3 (the other in-place reader), lith wins or ties every app shape: selective (region query, tabix) and small-object (Zarr) it wins clearly, and on the two large-file shapes it now matches — stream (S1) is a tie, and large-file batch (W3) is 141 s vs mountpoint's 144 s on a 4xl once the NIC in-flight budget is sized to the detected baseline (#79; an earlier 184 s was pre-#79 and drove the now-closed #85).

² lith's batch cell on the 2-vCPU c8g.large wasn't run (60 GB × 8-way on 2 cores overran the test budget); the copy-best cost there (~$0.044) is shown for scale. The crossover stays ≤ large — copy-best on a 2-vCPU box is dominated by staging 60 GB over its small NIC and then flagstat-ing on 2 cores, both of which favor the mount.

Why warm reuse doesn't change this for CPU-bound jobs

On the 8-vCPU box, re-running the selective and stream jobs against a warm cache saved only 3–7 % (region query 48.2 s → 44.6 s; flagstat 47.9 s → 46.6 s). These jobs are samtools-CPU-bound — decode is the ~45 s floor — so lith's cold read is already ~compute-time, and there is little for a warm cache to shave. lith's win on these shapes is the staging it avoids, not cache hits. Warm reuse matters for repeated selective queries over a working set that fits RAM (see Sizing the node).

Where the margin grows¶

The crossover table above uses a 14 GB object, so staging is a ~10-second afterthought on a fast box and lith's edge looks modest (1.2–1.5×). The margin widens with:

Object (or store) size. Staging scales with bytes; lith's selective read does not. On the 1.4 TB chrtout.zarr store, a one-month query touches 0.13 GB — lith answers it cold in 7.3 s (mountpoint-s3 36 s), while copying the store with the best copier did not finish in 20 minutes (886 of 1434 GB staged). At RODA scale, "stage first" isn't slower — it's infeasible.
Smaller NICs. Staging is NIC-bound on small boxes, so the copy path is no faster there while lith still moves only the bytes touched — the margin is larger on a c8g.large than on a 4xl.
Repeated jobs. A second query over the same working set is warm and free on lith; the copy path re-stages every fresh volume.
Fan-out. N nodes each stage their shard; lith mounts the same index N times and moves no bulk data (see Meet a deadline).
Pack small files once. Many small objects read cold pay a round-trip each; pack them with CargoShip and mount the archive as a tree, and lith streams a few framed chunks instead — 600 small files walk in 0.34 s / 8 GETs vs 2.56 s / 604 GETs native (see CargoShip archives).
Pack and publish once; mount it anywhere by name. cargoship publish writes a versioned dataset with an atomic pointer, and lith mount s3://bucket/dataset@current mounts it — no index file to track, and two machines mounting the same name serve byte-identical namespaces (see Published datasets).
Round-trips, not bytes. In-region, round-trips cost time; bytes don't. A reader that saves bytes by fetching a precise slice can still lose to a whole-file stream if it pays more round-trips to do it (see Parquet, above).

Operations, not just time¶

Time-to-result is one axis; bytes-touched is a second. The third — the one that shows up on the S3 bill and against the rate limit — is the operations a tool issues. These counts are from a CloudTrail S3 data-event ledger, not from any tool's self-report (lith's own counter matches the ledger exactly; mount-s3 and pyarrow don't report one). A 1,985-file tree walk, cold:

tool	GET	HEAD	LIST	total ops	request $
lith-cargoship (packed, 1 chunk)	12	0	0	12	$0.000005
lith-native	1,998	0	0	1,998	$0.0008
mount-s3	2,013	4,018	4,021	10,052	$0.0225
copy (s5cmd)	1,985	0	2	1,987	$0.0008

The gap is two halves. Metadata: lith is 0 HEAD / 0 LIST — the index answers stat and readdir locally; mount-s3 issues a HeadObject per open and a ListObjects per readdir (that is its 5× ops, not data). Data: packing collapses 1,985 object GETs into 12; format plans and coalescing keep the rest sparse (R4 Parquet: lith 684 GETs vs pyarrow's 6,162 byte-precise ones).

Three sentences on why it matters — the same op count reads three ways:

It's small per call. In-region a GET is $0.0004/1,000, so a single walk is cents. It becomes a budget line at scale: a 10M-file walk is ≈ $113 on mount-s3 (dominated by LISTs), $4 on lith-native, ~$0.10 packed — nightly, that is the difference between a rounding error and a line item.
It's on your bill. On a requester-pays bucket every op is billed to the reader; mount-s3's 5 ops/file is 5× the invoice of lith's 1, ~800× the packed path.
It's your ceiling. S3 throttles ~5,500 GET/s per prefix, so op count is a fan-out ceiling: at 5 ops/file that is ~1,100 files/s, at 1 GET/file 5,500/s, and a 64-node array multiplies per-node ops against the shared prefix.

The reframe: the index makes metadata free, plans and CargoShip make data ops sparse — the win is operations you never issue.

The rule¶

Pay for the bytes you touch, not the bytes you own. Copy first only when you will re-read a dataset many times from fast local storage and the staging cost amortizes across those reads — and then copy to local NVMe, never EBS. For query-once, selective, or batch work, mount.

And a third option for a cluster: when many nodes read the same dataset, one node can serve it to the rest over NFS — the shared data is fetched from S3 once, not once per node, and the second job is free. (The gateway reads faster than the FUSE mount — no FUSE hop — so sharing costs no per-read speed; it's a funnel on aggregate cold throughput, so size it by total demand.) Distinct data per node → per-node mounts; shared data → gateway.