Embed

Bring your own embedding model without changing the client wire. An autoscaler-served profile can name any Hugging Face checkpoint the configured inference provider supports, including a finetuned checkpoint. Clients still send Turbopuffer-compatible embed schema and Embed query expressions.

For a stock model on Turbopuffer, omit embed.serving or set prefer: native. Layer validates and forwards the native wire. Choose prefer: autoscaler for a BYO checkpoint or Layer extensions such as revision pins, instructions, chunking, and image embeddings. Choose prefer: lattice for the local CPU-only erikkaum/lattice-retrieval model.

The modes are explicit. A provider failure returns an error; Layer does not switch a request to another mode.

Query with Embed

Embed is the query half of schema-attribute embedding:

// source attribute: infer the model from its schema
"rank_by": ["text", "ANN", ["Embed", "chest pain radiating to left arm"]]

// derived vector attribute: name the model explicitly
"rank_by": ["embed_text", "ANN", ["Embed", "chest pain radiating to left arm", {
  "model": "acme/clinical-retrieval-v3"
}]]

response = await client.query_namespace("clinical-notes", {
    "rank_by": ["text", "ANN", ["Embed", "chest pain radiating to left arm"]],
    "top_k": 10,
})
print(response.rows)

response, err := client.QueryNamespace(ctx, "clinical-notes", &hevlayer.QueryRequest{
    RankBy: []any{"text", "ANN", []any{"Embed", "chest pain radiating to left arm"}},
    TopK:   10,
})

const response = await client.queryNamespace("clinical-notes", {
  rank_by: ["text", "ANN", ["Embed", "chest pain radiating to left arm"]],
  top_k: 10,
});

curl -X POST "$LAYER_GATEWAY_URL/v2/namespaces/clinical-notes/query" \
  -H "Authorization: Bearer $LAYER_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "rank_by": ["text", "ANN", ["Embed", "chest pain radiating to left arm"]],
    "top_k": 10
  }'

Native mode forwards Embed unchanged on Turbopuffer. Autoscaler mode resolves it through the configured provider, then sends a normal ANN vector to the store. Lattice mode resolves it in the gateway process. Query vectors are cached for 60 seconds by default; set LAYER_EMBED_CACHE_TTL_MS to change the TTL. A missing provider returns 503 service_unavailable.

Lattice

Lattice is a compact static retriever for text workloads where CPU throughput and deployment size matter more than transformer-level retrieval quality. It is an explicit serving leg and never falls back to native or autoscaler.

Generate a deployment artifact with the upstream Lattice slicer, place its model.safetensors and tokenizer.json together, and set LAYER_LATTICE_MODEL_PATH to the model file before starting the gateway. The supported model id is erikkaum/lattice-retrieval; the requested embed.dims must match the loaded artifact and only text modality is supported.

uv run slicer slice \
  --dim 512 \
  --quant int4_row \
  --output-dir /var/lib/hevlayer/lattice
export LAYER_LATTICE_MODEL_PATH=/var/lib/hevlayer/lattice/model.safetensors

"text": {
  "type": "string",
  "embed": {
    "model": "erikkaum/lattice-retrieval",
    "dims": 512,
    "serving": { "prefer": "lattice" }
  }
}

The recommended operating point is an int4-per-row, 512-dimensional Lattice artifact. Int4 quantizes the model’s lookup-table weights only. Layer writes the resulting normalized vectors as [512]f32; Turbopuffer’s int8 minimum for quantized vector storage is a separate choice and is not used by this path.

End-to-end example

Declare the Lattice profile on a string attribute, write rows, and query with Embed. The gateway embeds both sides in-process — no external inference provider is involved.

Write two rows into a namespace whose text attribute carries the Lattice profile shown above:

curl -X POST "$LAYER_GATEWAY_URL/v2/namespaces/articles/write" \
  -H "Authorization: Bearer $LAYER_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "upsert_rows": [
      {"id": "planet-1", "title": "Planet",
       "text": "Jupiter is the biggest planet in the Solar System."},
      {"id": "photo-1", "title": "Photosynthesis",
       "text": "Plants turn sunlight, water, and carbon dioxide into food."}
    ],
    "schema": {
      "text": {
        "type": "string",
        "embed": {
          "model": "erikkaum/lattice-retrieval",
          "dims": 512,
          "serving": { "prefer": "lattice" }
        }
      }
    }
  }'

Query by meaning rather than exact phrase:

curl -X POST "$LAYER_GATEWAY_URL/v2/namespaces/articles/query" \
  -H "Authorization: Bearer $LAYER_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "rank_by": ["text", "ANN", ["Embed", "largest planet in the solar system"]],
    "top_k": 3,
    "include_attributes": ["title", "text"]
  }'

{
  "rows": [
    { "id": "planet-1", "$dist": 0.137, "title": "Planet",
      "text": "Jupiter is the biggest planet in the Solar System." }
  ],
  "performance": {
    "embedding_tokens": 7,
    "embedding_ms": 1   // in-process lookup — no network hop to a provider
  }
}

A live example of exactly this contract is the Wikipedia × Lattice demo: all 283,997 Simple English Wikipedia articles (1.56M paragraph chunks) embedded through prefer: lattice and searched on Turbopuffer, with the performance echo displayed beside each result. Source at github.com/hev/wiki.

When rank_by names the source attribute, Layer reads the model from the schema. When it names embed_<attr>, the {model} argument is required; omitting it returns 422 with a model name must be provided.

Embed with Auto

An inline Embed lets query routing execute a semantic or fused leg in one request. Without a vector or an embedding profile, the router returns routing.executed: false and leaves embedding to the caller.

BYO model settings

Use a provider-namespaced Hugging Face repo id. The autoscaler path does not apply a gateway allowlist: model load or support errors come from the configured provider.

• embed.revision pins a stock or finetuned checkpoint revision.
• embed.instructions.document and embed.instructions.query add the prefixes required by asymmetric retrieval models. Both affect the query-cache key.
• embed.modality: image embeds writes with a CLIP-family image tower and query text with its text tower.
• embed.chunk splits source text before write-time embedding.

These fields require prefer: autoscaler and are never forwarded upstream. Client-side interoperability is unchanged: applications use the same schema, Embed expression, and response shape for stock and BYO models.

Performance accounting

Write and query responses report provider measurements under performance:

{
  "rows": [ /* ... */ ],
  "performance": {
    "embedding_tokens": 8,
    "embedding_ms": 42
  }
}

Queries omit embedding_tokens on a cache hit. Layer merges autoscaler provider measurements into the same object and exposes echoed work through hevlayer_embed_tokens_total and hevlayer_embed_compute_seconds_total, labeled by namespace, store kind, model, and serving mode.