Latest-row aggregates
POST /v2/namespaces/{namespace}/aggregate accepts latest_by for an exact
Count/Sum over the winning revisions observed by a complete source read.
This is separate from ranked collapse. It returns aggregates, never revisions.
The same strict request is accepted on /query when latest_by is present.
Generated clients expose aggregateLatest / aggregate_latest with typed
LatestRelationRequest and LatestRelationResponse models.
{
"latest_by": {
"key": ["entity"],
"order_by": [
{"field":"version", "direction":"desc", "nulls":"last"},
{"field":"id", "direction":"desc", "nulls":"last", "canonical_key":"entity"}
]
},
"scan": {"max_rows":100000, "max_bytes":67108864, "consistency":"observed"},
"filters": ["status", "Eq", "active"],
"aggregate_by": {"entities":["Count"], "amount":["Sum","amount"]}
}
The relation selects one complete row per key, then evaluates filters, then
aggregates. An older active revision cannot reappear when the winning row is
inactive. Ordering is lexicographic in the supplied terms. canonical_key
compares whether the physical id equals that row’s named key attribute;
desc prefers the canonical ID. It is allowed only on field: "id".
A descending physical-ID tie breaker is always appended. IDs must be uniformly
strings or unsigned integers; strings compare by UTF-8 bytes. Attribute types
must be consistent across nonnull values. An ordering attribute cannot mix
floating-point values with integers outside the exact binary64 integer range
(−2^53 through 2^53); that whole source domain is rejected before returning
winners. Integer-only ordering preserves the full int64/uint64 range.
Null and missing compare together,
with placement explicit for each term. Logical keys must contain nonempty
strings, numbers or booleans; null, missing and empty string keys fail the
request rather than merging unrelated rows.
Count counts selected matching rows. Sum ignores null/missing values and returns zero for an empty set. Non-numeric values and numeric overflow fail. Integer sums remain integers; fractional sums use compensated binary64 arithmetic. A fractional sum with an integer subtotal outside ±2^53 is rejected. JSON integers remain exact on the wire; consumers must preserve integer precision when decoding large totals. Filters support Eq, NotEq, Gt, Gte, Lt, Lte, In, NotIn, Contains, NotContains, ContainsAny, NotContainsAny, And, Or and Not. Eq null matches missing as well as explicit null; range comparisons never match null. Unknown operators and options fail before source reads. Attribute references are literal flat names, including names containing dots; JSON paths and nested object traversal are not supported.
Cost and consistency
Turbopuffer reads all revisions in strong ID-ordered pages inside Layer.
For a namespace with active sharding, Layer scans independent shards
concurrently using the existing namespace scan thread default and readiness
maximum (Index.spec.scan.threads; see scan concurrency).
Each shard advances its own ordered cursor to exhaustion. A disjoint remainder
partition reads rows whose shard tag is missing, null or outside that topology.
Cached readiness is a partitioning hint, never proof that every row is tagged.
The remainder is read on every request, under the same concurrency and global
budgets; it is not a second full scan. Layer then selects winners once across
all partitions, before applying mutable predicates. Invalid manifest namespace
identity or completion metadata fails the request, as do topology auth, throttle,
transport and server errors. Absent or unsupported metadata permits a complete
unpartitioned read. Counts of zero or above 4,096 use one complete serial cursor
before partition dispatch rather than constructing an unbounded filter.
Unsharded namespaces retain a single complete cursor. No physical shards are
created or resized by this request. If sharding is not ready, the complete
unpartitioned scan includes tagged and untagged data and excludes the namespace
metadata marker. Every partition excludes the marker after decoding, without
sending a string-ID filter to unsigned-integer ID namespaces. Marker response
bytes and pages still count.
For scalar requests without named grouped plans, the HTTP adapter requests only attributes referenced by the logical key, winner ordering, every filter branch and Sum metrics. Physical IDs are retained for ordering and ties. This avoids transferring and decoding unused attributes such as large payloads. All revisions are still read; predicates run in Layer after winner selection. If a partition’s first projected query returns Turbopuffer’s explicit HTTP 400 error for an attribute absent from its schema, Layer retries that partition once with all attributes. This preserves missing-value semantics, including Sum of an absent field and null filters. The failed read adds a source query and its response bytes count against the byte budget; any reported billing is retained. It may add backend cost. Other errors propagate. After any successful page in that partition, schema loss fails the request without restarting the scan. Projection is attempted afresh on each request, so later schema additions remain visible.
Grouped requests read all nonvector attributes, including attributes not
referenced by any plan, to preserve complete prefilter winner binding on
continuations. Existing adapters without explicit projection support retain
their full-source implementation. Layer processes every observed revision,
and gateway memory scales with revision count and returned row size. Each page is a separate source query;
the backend may rescan data on every page. Physical source work, billed bytes
and latency are unmeasured and may grow with namespace size times page count.
Attribute projection does not establish fewer physical rows read, fewer storage
bytes scanned or lower billing. This is not a native latest-row pushdown. No rows are transported to the API
consumer. The scan does not use the cache. Each page
is a separate read: concurrent writes, updates or deletes may change the
observed relation. observed does not mean a point-in-time snapshot,
and can miss inserts behind the cursor or observe a winner that is later
replaced. snapshot is rejected on this backend.
Both scan budgets are required. Exceeding either fails the whole request;
there is no partial total or silent 10,000-row cap. Limits bound accepted
source rows and decoded JSON bytes; up to the configured scan width of source pages can be in flight before
a byte overrun is detected. The row and byte limits apply to the whole request,
not separately to each shard. A failed shard fails the request and cancels
pending sibling reads; caller cancellation also drops those reads. These are gateway transfer budgets, not limits on backend scanned or billable
bytes. Source byte counts measure decoded row-page response bytes. Metadata
and sharding checks can make additional source calls outside these counters.
Billing and end-to-end latency must be measured separately.
Response relation reports the actual source rows, pages, bytes, selected
and matching rows, the declared consistency and the exhausted-source flag.
complete means the scan reached its end under that consistency contract.
It does not certify a snapshot under concurrent writes.
Only explicitly supported adapters may execute this plan. Other query options, ranked queries, joins and native group_by are rejected in this initial slice. Cross-namespace join isolation requires separate validation; native physical-row Aggregate support does not imply latest-row support. There is no persistent analytics store and no reuse of a previous request’s selected rows.
Named grouped plans
Add grouped to evaluate up to 16 named plans on the same complete selected,
filtered relation. Scalar aggregate_by remains required. For example:
{"grouped":{"projects":{"group_by":[{"kind":"field","field":"project"}],"metrics":{"sessions":{"kind":"count"},"spend":{"kind":"sum","field":"amount"}},"page_size":100}}}
Each plan accepts literal flat field dimensions (field, utc_day) and an
optional expansion: timestamps with field and IANA timezone, or models
with an object field mapping model names to numeric metric objects. Timestamp
expansion enables weekday (Monday=0) and hour; model expansion enables
model and model_sum. UTC dates consume integer epoch milliseconds.
Repeated DST hours merge while counting every instance. Empty expansions
produce no instances. Missing scalar dimensions form a typed null group;
empty strings remain distinct. No absent groups are filled with zero.
Row sum under model expansion rejects; per-model amounts are never inferred
from session totals. Timestamp expansion repeats a row sum per instance.
Response grouped[name] contains typed keys, aggregates, next_cursor,
total_groups, and complete for the group page. Outer relation.complete
certifies the observed source scan; it is distinct from page completeness.
Pass a page’s cursor in that named plan’s cursor. Every continuation repeats
the complete scan and grouping. It rejects changed selected content, selection
or filter plan, named plan, page size, namespace, store or authenticated caller.
It does not pin an upstream snapshot. Client scope/boundary fields reject.
All plans share finite ceilings: 10 million expanded instances and 100,000 accumulated groups, divided equally among the submitted plans. Each plan has at most 10,000 output groups per page. Any budget or numeric failure rejects the entire request, including scalar totals and earlier plans. Work scales with revisions plus selected rows times plans plus expanded instances; named plans share one source scan and one latest/filter selection, not grouping work. Integer sums are exact; mixing any integer outside ±2^53 with floats rejects. Unsupported grade/tool/cell predicates and joins require consumer fallback. The grade semi/anti-join kernel has no authorized HTTP surface here.