Limits

Layer is limited by certain constraints of the underlying components we ship with. We will lift these as demand increases.

• Single-node Aerospike. We enforce this for simplicity and also believe that a single large NVMe drive offers enough storage for almost every dataset.
• ~4,090 Turbopuffer namespaces. We use Aerospike sets for logical separation of data, which are limited by the Aerospike Community Edition AGPL license.
• ~3 TB cache size. Another limitation of the Aerospike license.
• 10,000 distinct values per scan facet field. Pre-computed snapshot scans cap each facet field’s cardinality. If a field exceeds the cap, it is noted in fields_skipped[] rather than fields[], so readers can treat every emitted field as complete. See snapshots.
• 1,000,000 distinct values per values scan. On-demand values scans accumulate their histogram in gateway memory. A job that crosses the cap completes with truncated: true: the cap applies after the full pass, keeping the top values by count — each with an exact count — and dropping the low-count tail. See scans.

No limits

These have no enforced ceiling, but practical limits exist and will show up under load.

• CRD instances (Index, Function, Pipeline, Scaling) — bounded only by the etcd and operator throughput of your Kubernetes cluster.
• Snapshot history per namespace — durable in S3; bounded by spec.snapshot.retention when set, or by object storage cost under retention: never.
• Search history retention — accumulates indefinitely in S3; no automatic expiry.
• Clickstream event volume — accumulates indefinitely in S3; no automatic expiry.
• UDF concurrency per function — KEDA scales replicas to match queue depth, bounded by your cluster’s capacity.
• Pipeline queue depth — pipeline queues, including chunked document queues, store document IDs and chunk ID lists in S3 manifests and keep only segment state and counters in Postgres.
• Document size and attribute count — bounded by Turbopuffer and Aerospike record limits, not by Layer.