---
# What is NodeDB (docs/introduction/what-is-nodedb)

---
title: What is NodeDB
description: A distributed hybrid database with eight peer engines for multi-modal AI and agentic workloads.
---

# What is NodeDB

NodeDB is a single Rust binary that provides eight peer engines — Document (schemaless), Document (strict), Key-Value, Columnar, Timeseries, Spatial, Vector, and Array — sharing one storage core, plus Graph and Full-Text Search as cross-engine overlays on any collection. Seven of the eight are selected per-collection via `WITH (engine='<name>')`; Array uses its own `CREATE ARRAY` DDL family. Each engine is built with purpose-specific data structures. All share the same storage, memory, and query planner. Cross-engine queries execute in one process with zero network hops.

## The Problem

Modern applications don't fit in one database. A healthcare app needs patient records (relational), medical imaging embeddings (vector), care team relationships (graph), device telemetry (timeseries), and offline-first sync for field workers (CRDT). The industry answer is a polyglot stack: PostgreSQL for relational, Qdrant for vectors, Neo4j for graphs, ClickHouse for timeseries, Redis for caching. Each system has its own protocol, deployment, and failure modes. Cross-database queries require application-level joins.

Some databases claim to solve this by bolting on capabilities — "graph" as recursive JOINs, "timeseries" without columnar compression or continuous aggregation. The features exist in name but not in performance.

## Eight Engines

**Vector** — HNSW index with SQ8, PQ, and IVF-PQ quantization. SIMD-accelerated distance math. Adaptive bitmap pre-filtering.

**Graph** — CSR adjacency index with 13 native algorithms (PageRank, WCC, Louvain, SSSP, etc.) and a Cypher-subset MATCH pattern engine. GraphRAG fusion with vector search.

**Document** — Two modes per collection. Schemaless: MessagePack blobs with CRDT sync. Strict: Binary Tuples with O(1) field extraction and 3-4x cache density over BSON.

**Columnar / Timeseries / Spatial** — Three peer engines sharing one compressed-column storage core (ALP, FastLanes, FSST, Gorilla, LZ4 codecs; block statistics; predicate pushdown). `columnar` for general analytics; `timeseries` adds append-only ingest, retention, and continuous aggregation; `spatial` adds R*-tree, geohash, and H3 indexing.

**Key-Value** — Hash-indexed O(1) point lookups with typed value fields. Native TTL, secondary indexes, atomic INCR/CAS, and predicate-filtered scans. SQL-queryable and joinable.

**Full-Text Search** — Block-Max WAND optimized BM25 with 16 Snowball stemmers, 27-language stop words, CJK bigram tokenization, posting compression, fuzzy matching, and native hybrid vector fusion.

**Array** — ND sparse multi-dimensional engine with tile-based storage, Z-order indexing, per-tile MBR statistics, and bitemporal cells. Replaces TileDB / Zarr / SciDB / Rasdaman for genomics, single-cell biology, raster cubes, and climate models.

**CRDT** — Loro-backed conflict-free replicated data types. AP on the edge, CP in the cloud. SQL constraint validation at sync time with compensation hints.

## Three Deployment Modes

**Origin (server)** — Full distributed database. Multi-Raft consensus, Thread-per-Core Data Plane with io_uring, PostgreSQL-compatible SQL over pgwire. Horizontal scaling with automatic shard rebalancing.

**Origin (local)** — Same binary, single-node. No cluster overhead. Like running PostgreSQL locally.

**NodeDB-Lite (embedded)** — In-process library for phones, browsers (WASM), and desktops. All eight engines run locally with sub-millisecond reads. CRDT sync to Origin via WebSocket.

## PostgreSQL Compatible

Connect with `psql`, any PostgreSQL driver, or ORM. Six wire protocols are available:

- **pgwire** — PostgreSQL wire protocol (port 6432)
- **HTTP** — REST, SSE, WebSocket (port 6480)
- **NDB** — Native MessagePack protocol (port 6433)
- **RESP** — Redis-compatible KV protocol (optional)
- **ILP** — InfluxDB Line Protocol for timeseries ingest (optional)
- **Sync** — WebSocket sync for NodeDB-Lite clients (port 9090)

## What NodeDB Replaces

The combination of PostgreSQL + pgvector + Redis + Neo4j + ClickHouse + Elasticsearch — unified into one binary with shared storage and zero network hops between engines.


---
# Quick Start (docs/introduction/quickstart)

---
title: Quick Start
description: Start NodeDB and run your first queries across multiple engines in under five minutes.
---

# Quick Start

This guide takes you from zero to running multi-engine queries in five minutes.

## 1. Start NodeDB

```bash
# Linux: prebuilt binary (recommended — best performance)
gh release download --repo NodeDB-Lab/nodedb --pattern 'nodedb-*-linux-x64.tar.gz' \
  && tar -xzf nodedb-*-linux-x64.tar.gz \
  && ./nodedb

# macOS / Windows / WSL2: Docker
docker compose up -d

# Or from source (any Linux with Rust 1.94+)
./target/release/nodedb
```

See [Installation](/docs/introduction/installation) for the full setup, configuration, and systemd unit examples.

## 2. Connect

```bash
psql -h localhost -p 6432
```

## 3. Documents (Schemaless)

```sql
CREATE COLLECTION users;

INSERT INTO users (id, name, email, age)
VALUES ('u1', 'Alice', 'alice@example.com', 30);

INSERT INTO users (id, name, email, role)
VALUES ('u2', 'Bob', 'bob@example.com', 'admin');

-- Object literal syntax also works
INSERT INTO users { name: 'Charlie', email: 'charlie@example.com', age: 25 };

SELECT name, email FROM users WHERE age > 25;
```

## 4. Strict Documents (Schema-Enforced)

```sql
CREATE COLLECTION orders (
    id TEXT PRIMARY KEY,
    customer_id TEXT,
    total FLOAT,
    status TEXT,
    created_at TIMESTAMP
) WITH (engine='document_strict');

INSERT INTO orders (id, customer_id, total, status)
VALUES ('o1', 'u1', 99.99, 'pending');

SELECT * FROM orders WHERE status = 'pending' ORDER BY created_at DESC;
```

## 5. Vector Search

```sql
CREATE COLLECTION articles;
CREATE VECTOR INDEX idx_embed ON articles METRIC cosine DIM 384;

INSERT INTO articles (id, title, embedding)
VALUES ('a1', 'Intro to AI', ARRAY[0.1, 0.2, 0.3, ...]);

SEARCH articles USING VECTOR(embedding, ARRAY[0.1, 0.3, ...], 10);
```

## 6. Graph

```sql
CREATE COLLECTION social;

INSERT INTO social (id, name) VALUES ('alice', 'Alice');
INSERT INTO social (id, name) VALUES ('bob', 'Bob');

GRAPH INSERT EDGE IN 'social' FROM 'alice' TO 'bob' TYPE 'knows' PROPERTIES { since: 2020 };

GRAPH TRAVERSE FROM 'alice' DEPTH 2;

GRAPH ALGO PAGERANK ON social DAMPING 0.85 ITERATIONS 20 TOLERANCE 1e-7;
```

## 7. Key-Value

```sql
CREATE COLLECTION sessions (key TEXT PRIMARY KEY) WITH (engine='kv');

INSERT INTO sessions (key, value) VALUES ('sess_abc', 'token-abc');

SELECT * FROM sessions WHERE key = 'sess_abc';
```

## 8. Columnar (Analytics)

```sql
CREATE COLLECTION web_events (
    ts TIMESTAMP,
    user_id UUID,
    page VARCHAR,
    duration_ms INT
) WITH (engine='columnar');

SELECT page, AVG(duration_ms), COUNT(*)
FROM web_events
WHERE ts > now() - INTERVAL '7 days'
GROUP BY page ORDER BY COUNT(*) DESC;
```

## 9. Timeseries

```sql
CREATE COLLECTION cpu_metrics (
    ts TIMESTAMP TIME_KEY,
    host VARCHAR,
    cpu FLOAT
) WITH (engine='timeseries', partition_by='1h');

SELECT time_bucket('5 minutes', ts) AS bucket, host, AVG(cpu)
FROM cpu_metrics
WHERE ts > now() - INTERVAL '1 hour'
GROUP BY bucket, host;
```

## 10. Triggers

```sql
CREATE TRIGGER notify_on_order AFTER INSERT ON orders FOR EACH ROW $$
BEGIN
    INSERT INTO notifications (id, user_id, message)
    VALUES (NEW.id || '_notif', NEW.customer_id, 'Order received');
END;
$$;
```

## What's Next

- [Architecture](/docs/architecture/overview) — how the three-plane execution model works
- [Data Modeling](/docs/data-modeling/collections) — collections, schemas, and engine selection
- [SQL Reference](/docs/sql/overview) — full query language reference


---
# Installation (docs/introduction/installation)

---
title: Installation
description: Install NodeDB via prebuilt binary, Docker, or from source. Linux kernel 5.1+ required.
---

# Installation

NodeDB requires Linux kernel 5.1+ (for io_uring), regardless of how you install it.

There are three ways to install NodeDB:

1. [Prebuilt binary](#prebuilt-binary-linux) — **recommended on Linux.** Direct kernel access to io_uring, no virtualization overhead, best raw performance.
2. [Docker](/docs/introduction/docker) — **recommended on macOS / Windows / WSL2**, or when you want a one-command setup with zero host configuration.
3. [Build from source](#build-from-source) — for development or custom features.

All three share the same [configuration](#configuration) and connection paths described below.

## Prebuilt binary (Linux)

Each tagged release ships a static `nodedb` tarball on GitHub for `linux-x64` and `linux-arm64`. macOS and Windows users should use [Docker](/docs/introduction/docker) until those targets ship.

```bash
# Resolve the latest tag and your architecture
TAG=$(curl -fsSL https://api.github.com/repos/NodeDB-Lab/nodedb/releases/latest \
        | grep '"tag_name"' | cut -d'"' -f4)
ARCH=$(uname -m | sed 's/aarch64/arm64/; s/x86_64/x64/')

# Download and extract
curl -L -o nodedb.tar.gz \
  "https://github.com/NodeDB-Lab/nodedb/releases/download/${TAG}/nodedb-${TAG#v}-linux-${ARCH}.tar.gz"
tar -xzf nodedb.tar.gz

# Optional: install system-wide
sudo mv nodedb /usr/local/bin/

# Run with all defaults (data goes to ~/.nodedb/data)
nodedb
```

If you have the [GitHub CLI](https://cli.github.com/) installed, this is one command:

```bash
gh release download --repo NodeDB-Lab/nodedb --pattern 'nodedb-*-linux-x64.tar.gz' \
  && tar -xzf nodedb-*-linux-x64.tar.gz
```

Run with a config file or a custom data directory:

```bash
# Point at an explicit data dir
NODEDB_DATA_DIR=/var/lib/nodedb nodedb

# Or load a config file (env vars still override TOML keys)
nodedb --config /etc/nodedb/nodedb.toml
```

For a long-running server, drop a unit file at `/etc/systemd/system/nodedb.service`:

```ini
[Unit]
Description=NodeDB
After=network.target

[Service]
Type=simple
User=nodedb
Group=nodedb
ExecStart=/usr/local/bin/nodedb --config /etc/nodedb/nodedb.toml
Restart=on-failure
LimitNOFILE=1048576

[Install]
WantedBy=multi-user.target
```

Then `sudo systemctl enable --now nodedb`. The user/group must be able to read the config file and write `data_dir`.

For a specific version or to browse changelogs, see the release page: <https://github.com/NodeDB-Lab/nodedb/releases>. The SQL surface is still pre-1.0 and changes between tags, so pin a version in production.

## Docker

See [Docker](/docs/introduction/docker) for the full Compose and `docker run` setup. The right choice on macOS, Windows, or any host where you don't want to manage a binary directly.

## Build from Source

Requires Rust 1.94+ and a Linux host.

```bash
git clone https://github.com/NodeDB-Lab/nodedb.git
cd nodedb

# Release build (all crates)
cargo build --release

# Run tests (use nextest — the cluster integration tests rely on
# the test groups defined in .config/nextest.toml and will hang
# under plain `cargo test`)
cargo install cargo-nextest --locked  # one-time
cargo nextest run --all-features
```

The build produces two binaries:

- `target/release/nodedb` — the database server
- `target/release/ndb` — the terminal client (TUI with syntax highlighting, tab completion, history search)

Start the server:

```bash
./target/release/nodedb

# Or with a config file
./target/release/nodedb --config nodedb.toml
```

Default startup output:

```
2026-01-01T00:00:00Z INFO nodedb: Starting NodeDB
2026-01-01T00:00:00Z INFO nodedb: Listening on 127.0.0.1:6433 (native)
2026-01-01T00:00:00Z INFO nodedb: Listening on 127.0.0.1:6432 (PostgreSQL)
2026-01-01T00:00:00Z INFO nodedb: Listening on 127.0.0.1:6480 (HTTP)
2026-01-01T00:00:00Z INFO nodedb: Data directory: ~/.nodedb/data
```

## Configuration

This section applies to **every** install method — prebuilt binary, Docker, and source builds all read the same TOML schema and respond to the same environment variables. Pick whichever is convenient:

- **TOML file** — pass `--config /path/to/nodedb.toml` on the command line. Best for production / systemd / pre-baked images.
- **Environment variables** — prefix `NODEDB_*`. Best for Docker (`-e`), Compose (`environment:`), and Kubernetes. Env vars **override** values from the TOML file when both are set.

All protocols share one bind address (`host`); only the port differs per protocol.

```toml
# nodedb.toml
[server]
host = "127.0.0.1"
data_dir = "/var/lib/nodedb"
memory_limit = "4GiB"
data_plane_cores = 4
max_connections = 1024
log_format = "text"

[server.ports]
native = 6433
pgwire = 6432
http = 6480
resp = 6381          # Optional: set to enable
ilp = 8086           # Optional: set to enable
```

| Config field       | Environment variable      | Default                                              |
| ------------------ | ------------------------- | ---------------------------------------------------- |
| `host`             | `NODEDB_HOST`             | `127.0.0.1`                                          |
| `ports.native`     | `NODEDB_PORT_NATIVE`      | `6433`                                               |
| `ports.pgwire`     | `NODEDB_PORT_PGWIRE`      | `6432`                                               |
| `ports.http`       | `NODEDB_PORT_HTTP`        | `6480`                                               |
| `ports.resp`       | `NODEDB_PORT_RESP`        | disabled                                             |
| `ports.ilp`        | `NODEDB_PORT_ILP`         | disabled                                             |
| `data_dir`         | `NODEDB_DATA_DIR`         | `~/.nodedb/data` (binary), `/var/lib/nodedb` (Docker) |
| `memory_limit`     | `NODEDB_MEMORY_LIMIT`     | `1GiB`                                               |
| `data_plane_cores` | `NODEDB_DATA_PLANE_CORES` | CPUs - 1                                             |
| `max_connections`  | `NODEDB_MAX_CONNECTIONS`  | `4096`                                               |
| `log_format`       | `NODEDB_LOG_FORMAT`       | `text`                                               |

## Connect

```bash
# With the ndb TUI client (source build)
./target/release/ndb

# With psql
psql -h localhost -p 6432

# Health check via HTTP
curl http://localhost:6480/health
```

## System Requirements

- **OS**: Linux (kernel 5.1+ for io_uring) — required for the binary and source builds; Docker users can run on any host that supports a Linux container with io_uring.
- **Rust**: 1.94+ (source builds only)
- **Memory**: 512 MiB minimum, 4+ GiB recommended
- **Disk**: NVMe recommended for Data Plane I/O


---
# Docker (docs/introduction/docker)

---
title: Docker
description: Run NodeDB with a single docker run command or Docker Compose. Single binary, all ports exposed.
---

# Docker

NodeDB is a single binary. One `docker run` command gets you a running server. Requires Linux kernel 5.1+ (for io_uring).

## Quick Start

```bash
docker run -d --name nodedb \
  -p 6432:6432 \
  -p 6433:6433 \
  -p 6480:6480 \
  -p 9090:9090 \
  -v nodedb-data:/var/lib/nodedb \
  farhansyah/nodedb
```

That's it. Connect with `psql -h localhost -p 6432` or `curl http://localhost:6480/healthz`.

## Ports

| Port   | Protocol                          | Required |
| ------ | --------------------------------- | -------- |
| `6432` | PostgreSQL wire protocol (pgwire) | Yes      |
| `6433` | Native MessagePack protocol (NDB) | Yes      |
| `6480` | HTTP API (REST, SSE, WebSocket)   | Yes      |
| `9090` | WebSocket sync (NodeDB-Lite)      | Yes      |

RESP (Redis) and ILP (InfluxDB) are disabled by default. Enable via environment variables:

```bash
docker run -d --name nodedb \
  -p 6432:6432 \
  -p 6433:6433 \
  -p 6480:6480 \
  -p 9090:9090 \
  -p 6381:6381 \
  -p 8086:8086 \
  -v nodedb-data:/var/lib/nodedb \
  -e NODEDB_PORT_RESP=6381 \
  -e NODEDB_PORT_ILP=8086 \
  farhansyah/nodedb
```

## Docker Compose

```yaml
services:
  nodedb:
    image: farhansyah/nodedb
    ports:
      - "6432:6432"   # pgwire
      - "6433:6433"   # native
      - "6480:6480"   # HTTP
      - "9090:9090"   # sync
    volumes:
      - nodedb-data:/var/lib/nodedb
    environment:
      NODEDB_MEMORY_LIMIT: "4GiB"
      # NODEDB_PORT_RESP: "6381"    # uncomment to enable Redis protocol
      # NODEDB_PORT_ILP: "8086"     # uncomment to enable ILP ingest

volumes:
  nodedb-data:
```

```bash
docker compose up -d
```

## Stop

```bash
# Stop (data preserved)
docker compose down

# Stop and wipe all data
docker compose down -v

# Or with docker run
docker stop nodedb && docker rm nodedb
# Data persists in the nodedb-data volume
```

## Verify

```bash
curl http://localhost:6480/health
```

## Environment Variables

| Variable                  | Default    | Description                  |
| ------------------------- | ---------- | ---------------------------- |
| `NODEDB_MEMORY_LIMIT`     | 75% of RAM | Total memory budget          |
| `NODEDB_DATA_PLANE_CORES` | CPUs - 1   | Number of Data Plane threads |
| `NODEDB_LOG_FORMAT`       | `text`     | `text` or `json`             |
| `NODEDB_HOST`             | `0.0.0.0`  | Bind address                 |
| `NODEDB_PORT_RESP`        | disabled   | Set to enable Redis protocol |
| `NODEDB_PORT_ILP`         | disabled   | Set to enable ILP ingest     |
| `NODEDB_DATA_DIR`         | `/var/lib/nodedb` | Data directory inside container |

## Custom Port Mapping

Remap any port on the host side. The container always listens on the same internal ports:

```bash
docker run -d --name nodedb \
  -p 5432:6432 \
  -p 8080:6480 \
  -v nodedb-data:/var/lib/nodedb \
  farhansyah/nodedb
```


---
# Architecture Overview (docs/architecture/overview)

---
title: Architecture Overview
description: The three-plane execution model — Control Plane, Data Plane, and Event Plane connected by lock-free ring buffers.
---

# Architecture Overview

NodeDB splits work across three planes connected by lock-free ring buffers. Each plane does exactly what it is best at and nothing else. Mixing planes is a correctness bug.

```
┌───────────────────────────────────────────┐
│           Control Plane (Tokio)           │
│  SQL parsing, query planning, connections │
│           Send + Sync, async              │
└─────────────┬──────────────┬──────────────┘
              │ SPSC Bridge  │ Event subscriptions
              │              │
┌─────────────▼──────────┐  ┌▼────────────────────────────────────┐
│  Data Plane (TPC)      │  │  Event Plane (Tokio)                │
│  Physical execution    ├─►│  AFTER trigger dispatch             │
│  Storage I/O, SIMD     │  │  CDC change streams                 │
│  !Send, io_uring       │  │  Cron scheduler                     │
│  Emits WriteEvents     │  │  Durable pub/sub, webhook delivery  │
└────────────────────────┘  └─────────────────────────────────────┘
```

## Plane Boundaries

| Plane         | Does                                                           | Does not do                                     |
| ------------- | -------------------------------------------------------------- | ----------------------------------------------- |
| Control Plane | SQL parsing, query planning, connection handling               | Event processing, trigger execution, storage I/O |
| Data Plane    | Physical I/O, SIMD math, WAL append, BEFORE triggers           | Event delivery, AFTER triggers, cross-shard work |
| Event Plane   | AFTER triggers, CDC, cron, webhooks, durable pub/sub           | Query planning, storage I/O, TPC tasks           |

If code needs to cross a plane boundary, it goes through the SPSC bridge (Control-Data) or the Event Bus (Data-Event).

## Query Entry Paths

**SQL path** — All user-facing interfaces accept SQL. The Control Plane parses via sqlparser, plans via EngineRules, and dispatches a `SqlPlan` through the SPSC bridge to the Data Plane.

```
psql / ndb CLI / HTTP /v1/query
  → SQL parser (sqlparser-rs)
  → EngineRules::plan_*()
  → SqlPlan → SPSC Bridge → Data Plane
```

**Native opcode path** — The Rust SDK and FFI/WASM bindings dispatch typed opcode messages over the NDB protocol. The Control Plane converts them directly to a plan, skipping SQL parsing.

```
nodedb-client / FFI / WASM
  → Native opcode + typed fields
  → build_plan()
  → PhysicalPlan → SPSC Bridge → Data Plane
```

Both paths produce the same plan and execute identically on the Data Plane.

## Cross-Engine Identity

Every row in every engine — document, KV, columnar, timeseries, spatial, vector, array, graph node, FTS posting — carries a stable global `u32` **surrogate** allocated at insert from a WAL-durable, Raft-replicated monotonic counter. Every engine keys its internal indexes on the surrogate, so cross-engine prefilter and join reduce to roaring-bitmap intersections with **zero per-query translation**.

A query like "find product cells whose embedding is near `$q`, that have FTS hits for `'memory leak'`, that live within 5km of point P, in tenant 42" turns into:

```
vector_index.search($q, k)         → roaring bitmap A
fts_index.match("memory leak")     → roaring bitmap B
spatial_index.dwithin(P, 5km)      → roaring bitmap C
metadata_index.tenant_id = 42      → roaring bitmap D

A ∩ B ∩ C ∩ D                      → final candidate surrogate set
```

No `HashMap<surrogate, doc_id>` translations between hops. Adding a new engine does not require new translation paths — it just allocates surrogates from the same counter. This is the mechanism behind every cross-engine query example you'll see in the SQL reference.


---
# Control Plane (docs/architecture/control-plane)

---
title: Control Plane
description: Tokio-based async runtime for SQL parsing, query planning, and connection handling.
---

# Control Plane

The Control Plane runs on the standard Tokio thread pool. All types are `Send + Sync`. It handles three responsibilities:

## Connection Handling

Accepts connections on all wire protocols:

- **pgwire** (port 6432) — PostgreSQL wire protocol
- **HTTP** (port 6480) — REST, SSE, WebSocket
- **NDB** (port 6433) — Native MessagePack protocol
- **RESP** — Redis-compatible (optional)
- **ILP** — InfluxDB Line Protocol (optional)
- **Sync** (port 9090) — WebSocket for NodeDB-Lite clients

Each connection is a Tokio task. Connection state (prepared statements, session variables, transaction context) lives on the connection task.

## Query Planning

SQL text arrives from any protocol and flows through:

1. **sqlparser-rs** — parses SQL text into an AST
2. **nodedb-sql EngineRules** — resolves the target engine and produces a `SqlPlan`
3. **Plan conversion** — `SqlPlan` becomes a `PhysicalPlan` for the Data Plane

`EngineRules` is the single source of truth for what each engine supports. One implementation per engine type (vector, graph, document, columnar, kv, fts, crdt). The compiler enforces exhaustive handling — adding a new engine or operation requires implementing every trait method.

## Dispatch

The Control Plane dispatches `PhysicalPlan` messages to Data Plane cores via the SPSC bridge. Routing is by vShard — each request targets a specific shard, which maps to a specific core.

The Control Plane never touches storage, never runs SIMD math, and never processes events. Those responsibilities belong to the Data Plane and Event Plane respectively.


---
# Data Plane (docs/architecture/data-plane)

---
title: Data Plane
description: Thread-per-Core execution with io_uring for physical query execution and storage I/O.
---

# Data Plane

The Data Plane uses a Thread-per-Core (TPC) architecture. Each CPU core runs an isolated, shared-nothing shard. Types are `!Send` by design — no data crosses core boundaries.

## Execution Model

Each core owns:

- A dedicated event loop (no Tokio — raw TPC)
- io_uring submission and completion queues for NVMe I/O
- A jemalloc arena (no allocator lock contention)
- Lock-free telemetry ring buffers for metrics

There are no locks, no atomics, and no cross-core sharing. The Data Plane achieves predictable latency by eliminating all sources of contention.

## What the Data Plane Does

- Executes `PhysicalPlan` nodes dispatched from the Control Plane
- Reads from NVMe via io_uring
- Runs SIMD-accelerated vector distance math
- Appends to the WAL (O_DIRECT)
- Evaluates BEFORE triggers (synchronous, same transaction)
- Emits `WriteEvent` records to the Event Plane via per-core ring buffers

## What the Data Plane Does Not Do

- Spawn Tokio tasks
- Handle HTTP or pgwire connections
- Process AFTER triggers or CDC events
- Coordinate across shards

## WriteEvent Emission

After each successful WAL commit, the Data Plane emits a `WriteEvent` containing:

- `sequence` — monotonic per-core counter
- `collection` — target collection name
- `op` — Insert, Update, or Delete
- `row_id`, `lsn`, `tenant_id`, `vshard_id`
- `source` — User, Trigger, RaftFollower, or CrdtSync
- `new_value`, `old_value` — for trigger and CDC consumption

Events are fire-and-forget — the Data Plane never blocks waiting for the Event Plane. If the ring buffer overflows, the Event Plane replays from the WAL.

## Page Fault Hazard

A major page fault on an mmap region blocks the faulting TPC thread, stalling the entire shard's reactor. The Data Plane pre-fetches pages asynchronously via `io_uring IORING_OP_READ` or `madvise(MADV_WILLNEED)` before compute touches them.


---
# Event Plane (docs/architecture/event-plane)

---
title: Event Plane
description: Async event processing for triggers, CDC, cron scheduling, and durable pub/sub.
---

# Event Plane

The Event Plane is the third architectural layer. It runs on Tokio (`Send + Sync`) and handles all asynchronous, event-driven work. It never does storage I/O and never spawns TPC tasks.

## Responsibilities

**AFTER trigger dispatch** — Trigger bodies execute asynchronously after the write commits. Retry with exponential backoff. Persistent failures go to a dead-letter queue. `EventSource` tagging prevents trigger cascades — the Event Plane skips non-User events.

**CDC / change streams** — Routes `WriteEvent` records to matching change stream consumers. Per-partition offsets, consumer groups with rebalancing, and exactly-once transactional consumption.

**Cron scheduler** — 1-second evaluation loop. Scheduled SQL is dispatched back through the Control Plane → Data Plane path. Per-collection affinity runs jobs on the collection's shard leader.

**Durable pub/sub** — Named topics with consumer groups, offset tracking, and configurable retention. Consumers resume from last committed offset after disconnect.

**Webhook delivery** — HTTP POST with exponential backoff retry and idempotency headers.

## Event Bus (Data → Event)

The Event Bus consists of one bounded ring buffer per Data Plane core — no cross-core contention.

- Data Plane emits `WriteEvent` records after WAL commit
- Fire-and-forget: Data Plane never blocks on Event Plane
- WAL-backed: if a ring buffer overflows, the Event Plane replays from the WAL LSN watermark

## Backpressure

All Event Bus queues are bounded:

- **> 85% utilization** — throttled, Data Plane continues but Event Plane signals pressure
- **> 95% utilization** — suspended, Event Plane enters WAL Catchup Mode (replays from WAL instead of ring buffer)

All backpressure transitions emit metrics and trace events.

## Side Effects

When a trigger body or scheduled job produces a write, the Event Plane dispatches it back through the normal Control Plane → Data Plane path. The Event Plane handles routing and delivery, not compute.


---
# SPSC Bridge (docs/architecture/spsc-bridge)

---
title: SPSC Bridge
description: Bounded lock-free ring buffers connecting the Control Plane and Data Plane.
---

# SPSC Bridge

The SPSC (Single-Producer, Single-Consumer) bridge is the only communication path between the Control Plane and Data Plane. No `Arc<Mutex<T>>`, no shared state — only bounded, lock-free ring buffers.

## Request Envelope

Control Plane → Data Plane:

| Field              | Type             | Purpose                            |
| ------------------ | ---------------- | ---------------------------------- |
| `request_id`       | u64              | Correlates request with response   |
| `tenant_id`        | u32              | Tenant scoping                     |
| `vshard_id`        | u16              | Routes to the correct core         |
| `plan`             | PhysicalPlan     | The physical execution plan        |
| `deadline_ns`      | u64              | Absolute deadline (monotonic clock)|
| `priority`         | u8               | Scheduling priority                |
| `trace_id`         | u64              | Distributed trace propagation      |
| `consistency`      | ReadConsistency  | Read consistency level             |
| `idempotency_key`  | Option\<u64\>    | Dedup key for non-idempotent writes|
| `event_source`     | EventSource      | User, Trigger, Raft, or CrdtSync   |
| `user_roles`       | Vec\<String\>    | Caller's RBAC roles                |

## Response Envelope

Data Plane → Control Plane:

| Field            | Type     | Purpose                          |
| ---------------- | -------- | -------------------------------- |
| `request_id`     | u64      | Matches the originating request  |
| `status`         | u8       | Success, error, partial          |
| `attempt`        | u32      | Retry attempt counter            |
| `partial`        | bool     | Whether more results follow      |
| `payload`        | Payload  | Response data                    |
| `watermark_lsn`  | u64      | Latest committed LSN             |
| `error_code`     | u16      | Typed error (if status != ok)    |

## Memory Ownership

- Cross-plane payloads use `Arc<[u8]>` or slab IDs with explicit reclaim ACK
- Producer owns buffer until ACK/NACK received
- Zero-copy buffers remain valid until consumer ACK

## Ordering

- Strict FIFO per `(connection_id, vshard_id)` stream
- Best-effort ordering across streams
- Cancellation is cooperative: Control Plane emits `CANCEL(request_id)`, Data Plane stops at the next safe point

## Backpressure

| Queue utilization | Action                         |
| ----------------- | ------------------------------ |
| < 85%             | Normal operation               |
| 85% - 95%         | Reduce read depth              |
| > 95%             | Suspend new reads              |

All transitions are observable via metrics and trace spans.


---
# Storage Tiers (docs/architecture/storage-tiers)

---
title: Storage Tiers
description: Three-tier storage hierarchy — RAM, NVMe, and S3 — with temperature-aware data placement.
---

# Storage Tiers

NodeDB uses tiered storage to match data temperature to the right medium.

## Tiers

| Tier      | Medium | Contents                                        | I/O Method            |
| --------- | ------ | ----------------------------------------------- | --------------------- |
| L0 (hot)  | RAM    | Memtables, active CRDT states, incoming metrics | None (in-memory)      |
| L1 (warm) | NVMe   | HNSW graphs, metadata indexes, segment files    | mmap + madvise        |
| L2 (cold) | S3     | Historical logs, compressed vector layers       | Parquet + HTTP range  |
| WAL       | NVMe   | Write-ahead log                                 | O_DIRECT via io_uring |

## Critical Rules

**WAL uses O_DIRECT.** Bypasses the kernel page cache entirely for deterministic write latency. Group commit batches multiple writes per io_uring submission for NVMe IOPS efficiency.

**L1 indexes use mmap.** Zero-copy deserialization. SIMD reads directly from mapped pages. `madvise(MADV_WILLNEED)` pre-fetches before compute touches the data.

**WAL and L1 never share page cache.** O_DIRECT (WAL) and mmap (L1) use fundamentally different I/O paths. Mixing them would cause cache coherency issues.

## Per-Core Memory

Each Data Plane core is pinned to a dedicated jemalloc arena via `nodedb-mem`. This eliminates allocator lock contention in the TPC architecture. Memory budgets are enforced per engine — no single engine can starve others.

## Compaction

L1 segment files undergo three-phase crash-safe compaction:

1. Write new merged segments to temporary files
2. Atomically swap file references in the catalog
3. Delete old segments after all readers have released them

Compaction preserves monotonic LSN ordering. Delete bitmaps (Roaring) track removed rows without rewriting segments.

## Cold Storage

L2 uses Parquet format with predicate pushdown. A packed single-file format enables HTTP range requests for minimal egress from S3/GCS/Azure.


---
# Write-Ahead Log (docs/architecture/wal)

---
title: Write-Ahead Log
description: O_DIRECT WAL with group commit, CRC32C checksums, and AES-256-GCM encryption.
---

# Write-Ahead Log

The WAL ensures durability. Every write is persisted to the WAL before being acknowledged. On crash recovery, the WAL replays to reconstruct any state not yet flushed to segments.

## Record Format

```
┌──────────┬─────────────────┬────────────┬─────┬───────────┬───────────┬─────────────┬─────────┐
│  magic   │ format_version  │ record_type│ lsn │ tenant_id │ vshard_id │ payload_len │ crc32c  │
│  4 bytes │    2 bytes      │   2 bytes  │ 8B  │   4 bytes │  2 bytes  │   4 bytes   │ 4 bytes │
└──────────┴─────────────────┴────────────┴─────┴───────────┴───────────┴─────────────┴─────────┘
```

## Properties

**O_DIRECT** — Bypasses the kernel page cache. Writes go directly to NVMe via io_uring. This provides deterministic write latency — no interference from page cache eviction or writeback.

**Page size** — 4 KiB or 16 KiB, alignment-compatible with O_DIRECT requirements.

**CRC32C** — Every page has a checksum for silent bit-rot detection.

**Group commit** — Multiple writes batch into a single io_uring submission for NVMe IOPS efficiency. A double-write buffer ensures atomicity.

**Segmented** — The WAL rolls over to a new segment file automatically. Old segments are eligible for cleanup once all records have been flushed to L1 segments.

**Tombstone GC** — Each checkpoint garbage-collects WAL rows for collections that have been hard-deleted (tombstoned), so tombstone records do not accumulate across restarts. On replay, the startup path merges persisted WAL tombstones with tombstones extracted from the WAL itself — a crash mid-purge cannot resurrect a dropped collection.

**Encryption** — Optional AES-256-GCM encryption at the page level. Key management is external.

## Crash Recovery

On startup, NodeDB:

1. Scans WAL segments from the last known checkpoint
2. Validates CRC32C checksums on each page
3. Replays valid records to reconstruct in-memory state
4. Discards any partially written records (torn writes)

The Event Plane uses WAL LSN watermarks to resume event processing from the correct position after a crash.

## Write Path

A write is acknowledged only after:

1. WAL append is persisted (O_DIRECT + fsync)
2. Raft quorum commit (for replicated namespaces)

Single-node mode: linearizable writes for shard leader. Replicated mode: linearizable writes within each Raft group.


---
# Consistency Model (docs/architecture/consistency)

---
title: Consistency Model
description: Read consistency levels, snapshot isolation, and cross-engine query consistency.
---

# Consistency Model

## Write Consistency

A write is acknowledged only after:

- **WAL append** is persisted (O_DIRECT + fsync)
- **Raft quorum commit** (for replicated namespaces)

Single-node: linearizable writes for the shard leader. Replicated: linearizable writes within each Raft group.

## Read Consistency Levels

```sql
-- Strong: leader-read after commit index >= required LSN (default for metadata/constraints)
SET read_consistency = 'strong';

-- Bounded staleness: follower read allowed if lag <= duration
SET read_consistency = 'bounded_staleness_5s';

-- Eventual: local read for CRDT edge sync (monotonic convergence)
SET read_consistency = 'eventual';
```

| Level               | Behavior                                     | Use case                          |
| ------------------- | -------------------------------------------- | --------------------------------- |
| Strong              | Leader read, waits for commit index          | Metadata, constraints, OLTP      |
| Bounded Staleness   | Follower read if lag within configured bound | Read replicas, analytics          |
| Eventual            | Local read, monotonic convergence            | CRDT edge sync, NodeDB-Lite      |

## Snapshot Isolation

Cross-engine queries execute against a shared snapshot watermark. A query that combines vector search, graph traversal, and document field access sees a consistent point-in-time view across all engines.

Default isolation: Snapshot Isolation (SI) per query. Write skew detection is enforced for UNIQUE and FK constraints at commit.

## Cross-Model Queries

RRF fusion and cross-engine joins include source watermarks in response metadata. This lets clients verify the freshness of results that span multiple engines.


---
# Cluster Topology (docs/architecture/cluster-topology)

---
title: Cluster Topology
description: Multi-node deployment with vShards, Raft groups, and QUIC transport.
---

# Cluster Topology

NodeDB scales horizontally by distributing data across multiple nodes using virtual shards (vShards).

## Nodes and Roles

Every node in a NodeDB cluster runs the same binary. Roles are determined by Raft group membership:

- **Leader** — accepts writes and reads for its Raft groups
- **Follower** — replicates from leader, serves stale reads (bounded staleness)
- **Learner** — non-voting replica, used for read scaling or cross-region copies

## vShards

Data is partitioned into virtual shards (vShards). Each vShard is assigned to a Raft group. A single node may host multiple vShards.

Benefits of virtual sharding:

- **Rebalancing** — move a vShard to a different node without reshuffling the entire dataset
- **Granular replication** — each vShard can have a different replication factor
- **Independent failure domains** — a vShard failure affects only the data in that shard

## Transport

Cluster communication uses QUIC (via nexar + quinn). QUIC provides:

- Multiplexed streams over a single connection
- 0-RTT connection resumption
- Built-in TLS 1.3
- Connection migration (IP address changes)

## Request Routing

The Control Plane routes each request to the correct Data Plane core based on vShard assignment. If the local node does not own the target vShard, the request is forwarded to the correct node over QUIC.


---
# Multi-Raft Consensus (docs/architecture/multi-raft)

---
title: Multi-Raft Consensus
description: Independent Raft groups per vShard for leader election, log replication, and snapshots.
---

# Multi-Raft Consensus

NodeDB uses Multi-Raft — each vShard is its own independent Raft group with its own leader, log, and snapshot schedule. This avoids the bottleneck of a single Raft group for the entire cluster.

## Per-vShard Raft

Each Raft group handles:

- **Leader election** — automatic failover when the current leader becomes unreachable
- **Log replication** — WAL entries replicated to followers before acknowledgement
- **Snapshots** — periodic state snapshots to truncate the Raft log

## Write Path (Replicated)

1. Client sends write to the vShard leader
2. Leader appends to local WAL
3. Leader replicates to Raft followers
4. Quorum acknowledges (majority of replicas)
5. Leader commits and responds to client

Writes are linearizable within each Raft group.

## Raft group kinds

NodeDB runs three kinds of Raft groups simultaneously:

| Kind | Purpose | Count |
|---|---|---|
| **Data** | One per vShard — replicates WAL entries for that shard's data | One per vShard |
| **Meta** | Cluster membership, catalog, schema | One per cluster |
| **Sequencer** | Cross-shard transaction ordering (Calvin epoch log) | One per cluster |

Each kind has independent leader election. A sequencer leader failure does not affect data-group leaders, and vice versa.

### Sequencer Raft group

The sequencer group exists solely to produce a globally-ordered log of cross-shard transaction batches (epochs). It has its own dedicated group ID outside the data-group range so it can never accidentally alias a vShard. See [Cross-Shard Transactions](./cross-shard-transactions) for how the sequencer group interacts with the scheduler and executor.

Single-shard writes never touch the sequencer group — they go directly through the relevant data-group's Raft.

## Advantages of Multi-Raft

- **Independent leaders** — different vShards can have leaders on different nodes, distributing write load
- **Parallel commits** — vShards commit independently, no global ordering bottleneck
- **Granular failover** — a node failure only triggers leader election for the vShards it led, not the entire cluster
- **Failure isolation** — sequencer leader election is independent of data and meta group elections


---
# vShards (docs/architecture/vshards)

---
title: vShards
description: Virtual sharding for data distribution, rebalancing, and per-shard replication.
---

# vShards

Virtual shards (vShards) are NodeDB's unit of data distribution. Each collection's data is partitioned across vShards, and each vShard maps to a Raft group and a Data Plane core.

## How vShards Work

- Data is hashed to a vShard based on the document ID (or a configured shard key)
- Each vShard is assigned to a node and a specific Data Plane core on that node
- The assignment is tracked in the cluster catalog and updated during rebalancing

## Rebalancing

When nodes join or leave the cluster, vShards are redistributed:

1. The coordinator selects vShards to move based on load metrics
2. Target node begins replicating the vShard's data
3. Once caught up, ownership transfers atomically in the catalog
4. Old node drops the vShard data

Rebalancing is transparent to clients — requests are forwarded to the new owner during migration. There is no downtime.

## Shard Count

The number of vShards is set at cluster creation and does not change. Choose a count that is several times the expected maximum number of cores across all nodes — this gives the rebalancer enough granularity to distribute evenly.


---
# Replication (docs/architecture/replication)

---
title: Replication
description: Raft-based replication with configurable replication factor per vShard.
---

# Replication

NodeDB replicates data via Raft log replication. Each vShard's Raft group independently replicates its WAL entries to followers.

## Replication Factor

Configurable per collection or per vShard:

- **RF=1** — no replication, single-node durability only
- **RF=3** — default for production, tolerates one node failure
- **RF=5** — high durability, tolerates two node failures

## How Replication Works

1. Leader receives a write and appends to its WAL
2. WAL entry is sent to all followers in the Raft group
3. Each follower appends to its own WAL and acknowledges
4. Leader waits for a quorum (majority) of acknowledgements
5. Leader commits the entry and responds to the client

## Follower Reads

Followers can serve reads with bounded staleness:

```sql
SET read_consistency = 'bounded_staleness_5s';
SELECT * FROM orders WHERE status = 'pending';
```

The follower checks that its replication lag is within the configured bound before serving the read. If the lag exceeds the bound, the read is forwarded to the leader.

## Cross-Region Replication

Learner replicas (non-voting Raft members) can be placed in remote regions for read scaling. They replicate asynchronously and do not participate in quorum decisions, so they do not add write latency.


---
# Cross-Shard Transactions (docs/architecture/cross-shard-transactions)

---
title: Cross-Shard Transactions
description: Calvin sequencer architecture for deterministic, atomically-committed cross-shard writes.
---

# Cross-Shard Transactions

When a write touches rows that hash to more than one vShard, NodeDB routes it through the **Calvin sequencer** — a dedicated Raft-backed coordination layer that guarantees all participating shards commit the transaction in the same order, with no mid-flight aborts.

Single-shard writes bypass this entirely and take the normal per-vShard Raft fast path.

## Why Calvin, not 2PC

Two-phase commit with compensating actions was the earlier design. It was replaced because compensation creates observable intermediate states (a row exists, then disappears), compensation can itself fail, and every read path that crosses an in-flight transaction has to know about pending compensations. The concern spreads outward indefinitely.

Calvin (Thomson et al., SIGMOD 2012) eliminates the problem by validating the full read/write set *before* any shard touches the transaction. Every shard executes against a globally-ordered input log. There is no concept of "shard A committed, shard B failed" — either all shards execute the transaction or none do.

## Architecture

```
Client / Control Plane
  │  declares read/write set
  ▼
SEQUENCER  (dedicated Raft group, Control Plane)
  │  batches transactions into epochs (default 20 ms)
  │  replicates each epoch via Raft — globally ordered
  ▼
SCHEDULER  (per vShard, Control Plane)
  │  acquires locks in deterministic global order
  │  single-threaded per shard
  ▼
EXECUTOR   (Data Plane, existing engine handlers)
  │  executes deterministically using the sequenced batch
  │  no application-level aborts — all constraint checks happen upstream
```

### The sequencer Raft group

The sequencer runs as its own independent Raft group (`SEQUENCER_GROUP_ID`), separate from the per-vShard data groups and the metadata group. This means:

- **Failure isolation** — sequencer leader election doesn't disrupt metadata reads, schema operations, or data writes
- **Independent tuning** — epoch duration (default 20 ms) is tuned separately from data-group commit latency
- **Geographic placement** — the sequencer triad can sit on the lowest-latency nodes without constraining where data groups live

### Epochs

The sequencer leader batches incoming transactions into epoch windows (default 20 ms). At the end of each window it:

1. Runs a pre-validation pass — detects intra-batch write-set conflicts; admits the first txn for a conflicting key, rejects others with `SequencerConflict` so the client retries
2. Proposes the validated batch to the sequencer Raft group
3. Once committed, fans the epoch out to each participating vShard's scheduler

### Determinism

All replicas must produce byte-identical WAL output for the same epoch. The executor is forbidden from using wall-clock time, non-seeded randomness, or non-deterministic map iteration on the cross-shard write path. System-time columns (bitemporal `sys_from`, KV TTL expiry, graph HLC ordinals) are seeded from the epoch timestamp so all replicas stamp the same value.

## Session variable

```sql
-- Default: route multi-vShard writes through the sequencer (atomic)
SET cross_shard_txn = 'strict';

-- Explicit opt-out: each shard commits independently (NOT atomic)
SET cross_shard_txn = 'best_effort_non_atomic';

SHOW cross_shard_txn;
```

`best_effort_non_atomic` is intended for bulk loads where atomicity is not required. It is deliberately named to discourage accidental use.

## EXPLAIN output

`EXPLAIN` reports whether a query routes through the sequencer:

```sql
EXPLAIN INSERT INTO orders VALUES (...);
-- cross-shard: sequenced | vshards: [3, 7] | epoch: <assigned at admission> | position: <assigned at admission>

EXPLAIN INSERT INTO local_cache VALUES (...);
-- single-shard: vshard 3
```

## OLLP — value-dependent predicates

For `UPDATE ... WHERE balance > 10000` where the write set depends on a scan result, NodeDB uses Optimistic Lock Location Prediction:

1. The planner runs the query optimistically once to capture the predicted write set
2. Submits to the sequencer with the predicted set
3. At execution time the executor re-scans and compares — if the set changed due to a concurrent commit, it returns `OllpRetryRequired` and retries transparently

To prevent retry storms, each predicate class (the parsed predicate AST, ignoring bound parameters) has:

- **Adaptive backoff** — starts at 10 ms, doubles up to 5 s
- **Circuit breaker** — opens after >50% retry rate over a 60 s window; half-opens after 30 s; closes after 4 consecutive successes
- **Per-tenant budget** — 1000 retries/min per tenant; excess returns `OllpTenantBudgetExceeded`

Static predicates (`WHERE id IN (...)`, `WHERE id = ?`) compute a deterministic write set at parse time and never enter the OLLP path.

## Admission limits

The sequencer enforces caps to prevent a single transaction from monopolising the cluster:

| Cap | Default | Error on exceed |
|---|---|---|
| `max_plans_bytes_per_txn` | 1 MiB | `TxnTooLarge` |
| `max_participating_vshards_per_txn` | 64 | `FanoutTooWide` |
| `max_txns_per_epoch` | 1024 | queued to next epoch |
| `max_bytes_per_epoch` | 16 MiB | queued to next epoch |
| Per-tenant inbox quota | inbox / 8 | `TenantQuotaExceeded` |

## Failure modes

| Failure | Behaviour |
|---|---|
| Sequencer leader dies | Raft re-elects; in-flight inbox submissions dropped; clients see `Unavailable` and retry |
| Sequencer follower dies | No client-visible impact; quorum remains |
| Scheduler shard crashes | Rebuilt from sequencer log on restart; shard unavailable until rebuild completes |
| Executor panic mid-apply | Locks held; shard restarts and replays the txn from WAL (determinism ensures identical result) |
| Network partition | Partitioned shard stops applying new epochs; serves reads with snapshot semantics; catches up on heal |

## Metrics

| Metric | Type | Description |
|---|---|---|
| `nodedb_sequencer_epochs_total` | counter | Epochs proposed |
| `nodedb_sequencer_epoch_duration_ms` | histogram | Time to drain + propose each epoch |
| `nodedb_sequencer_admitted_txns_total{outcome}` | counter | Per-outcome admission counts (`admitted`, `rejected_conflict`, `rejected_inbox_full`, `rejected_txn_too_large`, `rejected_fanout_too_wide`, `rejected_tenant_quota`, `rejected_not_leader`) |
| `nodedb_sequencer_inbox_depth` | gauge | Pending txns in the inbox |
| `nodedb_calvin_scheduler_lock_wait_ms_total{vshard}` | counter | Cumulative lock-wait time per shard |
| `nodedb_calvin_executor_txn_duration_ms{vshard}` | histogram | Per-shard execution time |
| `nodedb_calvin_ollp_retries_total{predicate_class,outcome}` | counter | OLLP retry outcomes |
| `nodedb_calvin_ollp_circuit_state{predicate_class}` | gauge | 0=closed, 1=half-open, 2=open |
| `nodedb_calvin_ollp_backoff_ms{predicate_class}` | gauge | Current OLLP backoff delay |
| `nodedb_calvin_infra_abort_total{reason}` | counter | Infrastructure-level aborts (disk error, OOM, etc.) |


---
# Collections (docs/data-modeling/collections)

---
title: Collections
description: Collections are NodeDB's top-level data containers. Each collection has a storage engine chosen at creation time.
---

# Collections

A collection is NodeDB's top-level data container — analogous to a table in PostgreSQL or a collection in MongoDB. Each collection has a storage engine chosen at creation time.

## Creating Collections

```sql
-- Schemaless document (default)
CREATE COLLECTION users;

-- Strict document (schema-enforced)
CREATE COLLECTION orders (
    id UUID DEFAULT gen_uuid_v7(),
    customer_id UUID NOT NULL,
    total DECIMAL NOT NULL,
    status STRING DEFAULT 'pending'
) WITH (engine='document_strict');

-- Columnar (analytics)
CREATE COLLECTION events (
    ts TIMESTAMP TIME_KEY,
    user_id UUID,
    event VARCHAR,
    duration_ms INT
) WITH (engine='columnar');

-- Key-Value
CREATE COLLECTION sessions (key TEXT PRIMARY KEY) WITH (engine='kv');
```

## Storage Engines

NodeDB has eight peer engines. Seven of them are picked per collection via `WITH (engine='<name>')` — the default (no `engine=`) is `document_schemaless`. The eighth, Array, uses its own `CREATE ARRAY` DDL family. See [Array Engine](../storage-engines/array) for that path.

| Engine                | Selector                            | Best for                                          |
| --------------------- | ----------------------------------- | ------------------------------------------------- |
| `document_schemaless` | (default)                           | Flexible data, prototyping, agent state           |
| `document_strict`     | `WITH (engine='document_strict')`   | OLTP, transactions, known schemas                 |
| `columnar`            | `WITH (engine='columnar')`          | Analytics, reporting, scan-heavy workloads        |
| `timeseries`          | `WITH (engine='timeseries')`        | Append-only metrics, retention, continuous aggs    |
| `spatial`             | `WITH (engine='spatial')`           | Geo-primary workloads, R*-tree, OGC predicates    |
| `kv`                  | `WITH (engine='kv')`                | Sessions, caches, counters, key-dominant access   |
| `vector`              | `WITH (engine='vector', vector_field='emb')` | Vector-primary collections                |

Column modifiers add capabilities to any engine:

| Modifier         | Effect                                                            |
| ---------------- | ----------------------------------------------------------------- |
| `TIME_KEY`       | Marks the time column (used by timeseries engine, allowed elsewhere) |
| `SPATIAL_INDEX`  | R*-tree index on a `GEOMETRY` column                              |
| `PRIMARY KEY`    | Row identity column (inline only — table-level not supported)     |

## Cross-Engine Indexes

Any collection can have indexes from multiple engines:

```sql
CREATE COLLECTION products;

-- Add a vector index for semantic search
CREATE VECTOR INDEX ON products METRIC cosine DIM 384;

-- Add a full-text index for keyword search
CREATE SEARCH INDEX ON products FIELDS title, description ANALYZER 'english';

-- Add a spatial index for location queries
CREATE SPATIAL INDEX ON products FIELDS location;

-- Add graph edges for relationships
GRAPH INSERT EDGE IN 'products' FROM 'products:p1' TO 'products:p2' TYPE 'similar';
```

## Converting Between Engines

Collections can be converted at any time without data loss:

```sql
CONVERT COLLECTION users TO document_strict;
CONVERT COLLECTION cache TO kv;
CONVERT COLLECTION events TO document_schemaless;
```

NodeDB infers the schema from existing documents when converting to strict mode.


---
# Schemas & Types (docs/data-modeling/schemas-types)

---
title: Schemas & Types
description: Data types, schema enforcement, typeguards, and schema evolution.
---

# Schemas & Types

## Data Types

| Type        | Description                          | Example                |
| ----------- | ------------------------------------ | ---------------------- |
| `STRING`    | UTF-8 text                           | `'hello'`              |
| `TEXT`      | Alias for STRING                     | `'hello'`              |
| `VARCHAR`   | Alias for STRING                     | `'hello'`              |
| `INT`       | 64-bit signed integer                | `42`                   |
| `FLOAT`     | 64-bit IEEE 754                      | `3.14`                 |
| `DECIMAL`   | Arbitrary precision                  | `99.99`                |
| `BOOL`      | Boolean                              | `true`                 |
| `TIMESTAMP` | UTC timestamp (nanosecond precision) | `now()`                |
| `DATETIME`  | Alias for TIMESTAMP                  | `'2026-01-01T00:00Z'`  |
| `UUID`      | 128-bit UUID                         | `gen_uuid_v7()`        |
| `GEOMETRY`  | OGC geometry (WKB internally)        | `ST_Point(-73.9, 40.7)`|
| `ARRAY`     | Ordered list                         | `ARRAY[1, 2, 3]`       |
| `OBJECT`    | Nested document                      | `{ name: 'Alice' }`   |

## Schema Modes

**Schemaless** — No schema required. Fields can vary between documents. Types are inferred on insert. This is the default when you `CREATE COLLECTION x`.

**Strict** — Schema is defined at creation time and enforced on every write. O(1) field extraction via binary tuple format. Created with `(...) WITH (engine='document_strict')`.

## Typeguards (Schemaless Validation)

Typeguards add write-time validation to schemaless collections without changing the storage format. Guarded fields are type-checked; unguarded fields pass freely.

```sql
CREATE TYPEGUARD ON users (
    email STRING REQUIRED CHECK (email LIKE '%@%.%'),
    age INT CHECK (age >= 0 AND age <= 150),
    role STRING DEFAULT 'user',
    updated_at TIMESTAMP VALUE now()
);
```

| Modifier   | Behavior                                        |
| ---------- | ----------------------------------------------- |
| `REQUIRED` | Field must be present and non-null               |
| `DEFAULT`  | Inject a value when the field is absent          |
| `VALUE`    | Always inject/overwrite (computed fields)        |
| `CHECK`    | SQL boolean expression validated at write time   |

```sql
-- Modify guards
ALTER TYPEGUARD ON users ADD score FLOAT CHECK (score >= 0);
ALTER TYPEGUARD ON users DROP age;

-- Audit existing data against guards
VALIDATE TYPEGUARD ON users;

-- Graduate to strict schema
CONVERT COLLECTION users TO document_strict;
```

## Schema Evolution (Strict Mode)

```sql
-- Add a column with a default (zero-downtime, multi-version reads)
ALTER COLLECTION orders ADD COLUMN region STRING DEFAULT 'us-east';

-- Drop a column
ALTER COLLECTION orders DROP COLUMN region;
```

`ALTER ADD COLUMN` uses multi-version reads — existing rows are served with the default value without rewriting storage. New writes include the column natively.


---
# Document Engine (docs/storage-engines/document)

---
title: Document Engine
description: Two modes — schemaless (MessagePack, CRDT sync) and strict (Binary Tuples, O(1) field extraction).
---

# Document Engine

NodeDB supports two document storage modes per collection. Choose based on your workload and convert at any time.

## Schemaless Documents

Flexible JSON-like documents stored as MessagePack. No schema required — fields can vary between documents.

```sql
CREATE COLLECTION users;

INSERT INTO users { name: 'Alice', email: 'alice@example.com', age: 30 };
INSERT INTO users { name: 'Bob', role: 'admin', tags: ['ops', 'dev'] };

-- Standard SQL also works
INSERT INTO users (id, name, email) VALUES ('u3', 'Charlie', 'charlie@example.com');

CREATE INDEX ON users FIELDS email;
SELECT * FROM users WHERE age > 25;
```

**Best for:** Prototyping, AI agent state, user profiles, nested data, CRDT sync.

## Strict Documents

Schema-enforced documents stored as Binary Tuples with O(1) field extraction. The engine jumps directly to the byte offset of any column — 3-4x better cache density than MessagePack or BSON.

```sql
CREATE COLLECTION orders (
    id UUID DEFAULT gen_uuid_v7(),
    customer_id UUID NOT NULL,
    total DECIMAL NOT NULL,
    status STRING DEFAULT 'pending',
    created_at TIMESTAMP DEFAULT now()
) WITH (engine='document_strict');

INSERT INTO orders (customer_id, total, status) VALUES ($cust_id, 149.99, 'shipped');
SELECT * FROM orders WHERE id = '...';

-- Schema evolution (zero-downtime)
ALTER COLLECTION orders ADD COLUMN region STRING DEFAULT 'us-east';
```

**Best for:** OLTP, transactions, known schemas, HTAP with columnar materialized views.

## Comparison

|               | Schemaless                     | Strict                          |
| ------------- | ------------------------------ | ------------------------------- |
| Schema        | Flexible, evolves freely       | Fixed, enforced on write        |
| Field access  | Parse MessagePack              | O(1) byte offset                |
| Cache density | Good                           | 3-4x better                     |
| CRDT sync     | Native                         | Via adapter                     |
| HTAP          | No                             | Yes (materialized views)        |

## Typeguards

Add validation to schemaless collections without changing to strict mode:

```sql
CREATE TYPEGUARD ON users (
    email STRING REQUIRED CHECK (email LIKE '%@%.%'),
    age INT CHECK (age >= 0 AND age <= 150),
    role STRING DEFAULT 'user'
);
```

See [Schemas & Types](/docs/data-modeling/schemas-types) for full typeguard reference.

## Converting

```sql
CONVERT COLLECTION users TO document_strict;      -- schemaless → strict
CONVERT COLLECTION logs TO columnar;     -- document → columnar
CONVERT COLLECTION cache TO kv;          -- document → key-value
```


---
# Key-Value Engine (docs/storage-engines/kv)

---
title: Key-Value Engine
description: Hash-indexed O(1) point lookups with TTL, atomic operations, sorted indexes, and rate gates.
---

# Key-Value Engine

Purpose-built hash-indexed store with O(1) point lookups, native TTL, and secondary indexes. Unlike a standalone KV store, data is SQL-queryable, joinable, and syncable via CRDTs.

## When to Use

- Session state and tokens
- Feature flags and configuration
- Rate limiters and counters
- Caching without an external cache
- Leaderboards and rankings

## SQL Usage

```sql
CREATE COLLECTION sessions (key TEXT PRIMARY KEY) WITH (engine='kv');

-- Insert with TTL; plain INSERT raises unique_violation (23505) on duplicate key.
INSERT INTO sessions { key: 'sess_abc', user_id: 'alice', role: 'admin', ttl: 3600 };

-- Set-or-overwrite (Redis SET semantics)
UPSERT INTO sessions { key: 'sess_abc', user_id: 'alice', role: 'admin', ttl: 3600 };

-- Set-if-absent (Redis SETNX semantics)
INSERT INTO sessions { key: 'sess_abc', user_id: 'alice', role: 'admin', ttl: 3600 }
ON CONFLICT DO NOTHING;

-- Conditional merge: bump counter on conflict, EXCLUDED references the incoming row.
INSERT INTO sessions (key, user_id, role, hits) VALUES ('sess_abc', 'alice', 'admin', 1)
ON CONFLICT (key) DO UPDATE SET
  role = EXCLUDED.role,
  hits = sessions.hits + 1;

-- Get by key
SELECT * FROM sessions WHERE key = 'sess_abc';

-- Analytical queries work too
SELECT role, COUNT(*) FROM sessions GROUP BY role;

-- Join with other collections
SELECT u.name, s.role FROM users u JOIN sessions s ON u.id = s.user_id;
```

## Atomic Operations

```sql
SELECT KV_INCR('counters', 'page_views', 1);
SELECT KV_DECR('credits', 'user-123', 50);
SELECT KV_CAS('state', 'player-1', 'idle', 'in_match');
SELECT KV_GETSET('token', 'sess-1', 'new-token');
SELECT KV_INCR('daily_logins', 'user-1', 1, TTL => 86400);
```

## Sorted Indexes (Leaderboards)

```sql
CREATE SORTED INDEX lb_global ON scores (score DESC, updated_at ASC) KEY player_id;

SELECT RANK(lb_global, 'player-123');
SELECT * FROM TOPK(lb_global, 10);
SELECT SORTED_COUNT(lb_global);

-- Time-windowed leaderboard
CREATE SORTED INDEX lb_daily ON scores (score DESC) KEY player_id WINDOW DAILY ON updated_at;
```

## Rate Gates

```sql
SELECT RATE_CHECK('attack_cooldown', 'player-123', 3, 10);
-- {"allowed": true, "remaining": 2, "resets_in_ms": 7500}

SELECT RATE_REMAINING('api_calls', 'tenant-1', 100, 60);
SELECT RATE_RESET('attack_cooldown', 'player-123');
```

## Redis-Compatible Access (RESP)

Enable by setting `ports.resp = 6381` in config. Any Redis client works:

```bash
redis-cli -p 6381
SET sess_abc '{"user":"alice"}' EX 3600
GET sess_abc
INCR page_views
ZADD leaderboard 1500 player-123
ZRANK leaderboard player-123
```

Supported: GET, SET, DEL, EXISTS, MGET, MSET, EXPIRE, TTL, SCAN, KEYS, HGET, HSET, INCR, DECR, ZADD, ZREM, ZRANK, ZRANGE, SUBSCRIBE, PUBLISH, and more.


---
# Columnar Engine (docs/storage-engines/columnar)

---
title: Columnar Engine
description: Per-column compression with block statistics, predicate pushdown, and 20-40x compression ratios.
---

# Columnar Engine

Typed columns with per-column compression — the same approach as ClickHouse and DuckDB, living alongside your OLTP data.

## When to Use

- Analytical queries (GROUP BY, aggregations, window functions)
- Reporting dashboards
- Data science workloads
- Any scan-heavy workload reading few columns from many rows
- HTAP: pair with strict documents for combined OLTP + OLAP

## Compression Codecs

Each column gets a codec chain tuned for its data type:

| Codec      | Target                        | Approach                            |
| ---------- | ----------------------------- | ----------------------------------- |
| ALP        | Floats                        | Lossless float-to-integer conversion |
| FastLanes  | Integers                      | SIMD bit-packing                    |
| FSST       | Strings                       | Substring dictionary compression    |
| Gorilla    | Metrics                       | XOR-based compression               |
| Pcodec     | Complex numerics              | Adaptive numeric compression        |
| rANS       | Cold-tier data                | Entropy coding                      |
| LZ4        | All                           | Terminal stage compression          |

Multi-stage pipeline achieves 20-40x compression on typical workloads.

## Block Statistics

Data is stored in 1024-row blocks. Each block has min/max/null-count statistics. The query engine skips blocks that can't match the predicate — no decompression needed.

## DDL

```sql
-- Plain columnar
CREATE COLLECTION logs (
    ts TIMESTAMP TIME_KEY,
    host VARCHAR,
    level VARCHAR,
    message VARCHAR
) WITH (engine='columnar');

-- Timeseries (peer engine — append-only, retention, continuous aggregates)
CREATE COLLECTION metrics (
    ts TIMESTAMP TIME_KEY, host VARCHAR, cpu FLOAT
) WITH (engine='timeseries', partition_by='1h');

-- Spatial (peer engine — R-tree + geohash + OGC predicates)
CREATE COLLECTION locations (
    geom GEOMETRY SPATIAL_INDEX, name VARCHAR
) WITH (engine='spatial');
```

## Queries

```sql
-- Point-get by primary key hits the segment PK index (not a full scan).
SELECT * FROM logs WHERE ts = '2026-04-24T10:00:00Z';

SELECT level, COUNT(*) FROM logs
WHERE ts > now() - INTERVAL '1 hour'
GROUP BY level ORDER BY COUNT(*) DESC;

-- ORDER BY is supported on columnar scans
SELECT ts, host, message FROM logs
WHERE level = 'error'
ORDER BY ts DESC
LIMIT 100;

-- Window functions
SELECT host, message,
       ROW_NUMBER() OVER (PARTITION BY host ORDER BY ts DESC) AS rank
FROM logs;
```

Plain `INSERT` on a columnar collection raises `unique_violation` on primary-key conflict; use `UPSERT` or `INSERT ... ON CONFLICT (pk) DO UPDATE SET col = EXCLUDED.col` for overwrite semantics.

## HTAP Bridge

Combine strict (OLTP) with columnar (OLAP):

```sql
CREATE COLLECTION orders (...) WITH (engine='document_strict');

CREATE MATERIALIZED VIEW order_analytics AS
SELECT status, DATE_TRUNC('day', created_at) AS day, COUNT(*), SUM(total)
FROM orders GROUP BY status, day;

-- Point lookups → strict engine. Scans → columnar engine. Automatic routing.
```

## Delete Bitmaps

Deleted rows are tracked with Roaring Bitmaps. Three-phase crash-safe compaction reclaims space: write new segments → swap references → delete old segments.


---
# Timeseries Engine (docs/storage-engines/timeseries)

---
title: Timeseries Engine
description: Peer engine for time-ordered data with retention, continuous aggregation, ILP ingest, and PromQL.
---

# Timeseries Engine

Timeseries is a peer engine sharing storage with `columnar` and `spatial`. It adds retention policies, continuous aggregation, ILP ingest, and dedicated time-series SQL functions. Data is stored in the same compressed-column memtables with a `TIME_KEY` column driving partition-by-time and block-level skip.

Selected via `WITH (engine='timeseries')` on `CREATE COLLECTION`, or the `CREATE TIMESERIES <name>` convenience alias.

## DDL

```sql
CREATE COLLECTION cpu_metrics (
    ts TIMESTAMP TIME_KEY,
    host VARCHAR,
    region VARCHAR,
    cpu_usage FLOAT,
    mem_usage FLOAT
) WITH (engine='timeseries', partition_by='1d', retention='90d');

-- Convenience alias
CREATE TIMESERIES cpu_metrics;
```

## Queries

```sql
-- Time-bucketed aggregation
SELECT time_bucket('5 minutes', ts) AS bucket, host, AVG(cpu_usage) AS avg_cpu
FROM cpu_metrics
WHERE ts > now() - INTERVAL '1 hour'
GROUP BY bucket, host ORDER BY bucket DESC;

-- Approximate aggregation (mergeable across shards)
SELECT approx_count_distinct(host), approx_percentile(cpu_usage, 0.95)
FROM cpu_metrics WHERE ts > now() - INTERVAL '24 hours';
```

## Continuous Aggregation

Incrementally maintained views — no full re-scan on refresh:

```sql
CREATE CONTINUOUS AGGREGATE cpu_hourly ON cpu_metrics
  BUCKET '1 hour'
  AGGREGATE time_bucket('1 hour', ts) AS hour, host, AVG(cpu_usage), ts_percentile(cpu_usage, 0.99)
  GROUP BY hour, host
  WITH (refresh_interval = '1m');

REFRESH CONTINUOUS AGGREGATE cpu_hourly;
```

## Timeseries SQL Functions

| Function               | What it does                         |
| ---------------------- | ------------------------------------ |
| `ts_rate`              | Per-second rate of change            |
| `ts_delta`             | Difference between consecutive values|
| `ts_moving_avg`        | Moving average over a window         |
| `ts_ema`               | Exponential moving average           |
| `ts_interpolate`       | Gap-fill with interpolated values    |
| `ts_percentile`        | Percentile calculation               |
| `ts_zscore`            | Z-score anomaly detection            |
| `ts_bollinger_upper/lower/mid/width` | Bollinger Bands       |
| `ts_moving_percentile` | Rolling percentile                   |
| `ts_correlate`         | Correlation between two series       |
| `ts_lag` / `ts_lead`   | Previous/next value in a series      |

## ILP Ingest

Enable with `ports.ilp = 8086`. Any ILP-compatible client (Telegraf, Vector) pushes metrics directly:

```bash
echo "cpu,host=web-01 usage=72.5 1609459200000000000" | nc localhost 8086
```

Adaptive batching and per-series core routing — self-tuning, no configuration needed.

## Grafana / PromQL

NodeDB works as a native Grafana Prometheus data source at `http://nodedb:6480/v1/obsv/api`. Full PromQL engine (Tier 1+2+3 functions). Also supports Prometheus remote write/read for long-term storage.


---
# Spatial Engine (docs/storage-engines/spatial)

---
title: Spatial Engine
description: Peer engine for geo-primary workloads — R*-tree index, OGC predicates, geohash, H3 hexagonal indexing, and hybrid spatial-vector search.
---

# Spatial Engine

Spatial is a peer engine sharing storage with `columnar` and `timeseries`. Collections selected via `WITH (engine='spatial')` and a `GEOMETRY SPATIAL_INDEX` column get automatic R*-tree indexing over the geometry column.

## DDL

```sql
-- Spatial peer engine (R-tree + geohash + OGC predicates)
CREATE COLLECTION restaurants (
    location GEOMETRY SPATIAL_INDEX,
    name VARCHAR,
    cuisine VARCHAR,
    rating FLOAT
) WITH (engine='spatial');

-- Or add a spatial index to any collection
CREATE COLLECTION restaurants;
CREATE SPATIAL INDEX ON restaurants FIELDS location;
```

## Queries

```sql
-- Find within 1km
SELECT name, ST_Distance(location, ST_Point(-73.990, 40.750)) AS dist
FROM restaurants
WHERE ST_DWithin(location, ST_Point(-73.990, 40.750), 1000)
ORDER BY dist;

-- Geofencing (point-in-polygon)
SELECT name FROM restaurants
WHERE ST_Within(location, ST_GeomFromGeoJSON('{
    "type": "Polygon",
    "coordinates": [[[-74.0, 40.7], [-73.9, 40.7], [-73.9, 40.8], [-74.0, 40.8], [-74.0, 40.7]]]
}'));

-- H3 hexagonal binning
SELECT h3_latlngtocell(40.748, -73.985, 9) AS hex;

-- Spatial join
SELECT r.name, z.zone_name
FROM restaurants r, delivery_zones z
WHERE ST_Contains(z.boundary, r.location);

-- Hybrid spatial-vector: nearby AND semantically similar
SELECT name, vector_distance(embedding, $query_vec) AS similarity
FROM restaurants
WHERE ST_DWithin(location, ST_Point(-73.990, 40.750), 2000) AND embedding <-> $query_vec
LIMIT 10;
```

## Geometry Functions

- **Predicates** — `ST_Contains`, `ST_Within`, `ST_Intersects`, `ST_Disjoint`, `ST_DWithin`, `ST_IsValid`
- **Constructors** — `ST_Point`, `ST_MakePoint`, `ST_GeomFromText`, `ST_GeomFromWKB`, `ST_GeomFromGeoJSON`, `ST_MakeLine`, `ST_MakePolygon`, `ST_MakeEnvelope`
- **Accessors** — `ST_AsText`, `ST_AsGeoJSON`, `ST_X`, `ST_Y`, `ST_GeometryType`, `ST_NPoints`, `ST_SRID`
- **Measures** — `ST_Distance`, `ST_Length`, `ST_Perimeter`, `ST_Area` (geodesic, meters)
- **Operations** — `ST_Buffer`, `ST_Envelope`, `ST_Centroid`, `ST_Union`, `ST_Intersection`

Each works in every geometry position — `INSERT` values, projections, and
predicate arguments — and constructors nest. See
[Spatial Queries](/docs/sql/spatial-queries) for the full reference.

## Format Support

WKB, WKT, and GeoJSON interchange. GeoParquet v1.1.0 and GeoArrow metadata for bulk export.

## Combined with Timeseries

```sql
-- Fleet tracking: spatial + time
CREATE COLLECTION fleet (
    ts TIMESTAMP TIME_KEY,
    vehicle_id VARCHAR,
    position GEOMETRY SPATIAL_INDEX,
    speed FLOAT
) WITH (engine='timeseries', partition_by='1d');
```


---
# Vector Engine (docs/storage-engines/vector)

---
title: Vector Engine
description: HNSW index with quantization, adaptive pre-filtering, and SIMD-accelerated distance math for semantic search.
---

# Vector Engine

The vector engine powers semantic search — nearest-neighbor retrieval over high-dimensional embeddings. It uses a custom HNSW index with multiple quantization levels and hardware-accelerated distance math.

## When to Use

- Semantic search over embeddings (text, images, audio)
- RAG pipelines for AI agents
- Recommendation systems
- Similarity matching and deduplication

## Key Features

- **HNSW index** — Multi-layer proximity graph. Construction at full precision (FP32/FP16) for structural integrity; traversal on quantized payloads for cache residency.
- **Quantization** — SQ8 (~4x memory reduction), PQ (~4-8x), IVF-PQ (~16 bytes/vector for 100M+ datasets).
- **Adaptive pre-filtering** — Roaring Bitmap-based filtering. Automatic strategy selection: pre-filter (selective), post-filter (broad), or brute-force.
- **Distance metrics** — L2, cosine, inner product, Manhattan, Chebyshev, Hamming, Jaccard, Pearson.
- **Cross-engine fusion** — Combine with graph (GraphRAG), full-text (hybrid BM25+vector), or spatial filtering.

## SQL Usage

```sql
-- Create a collection with a vector index
CREATE COLLECTION articles;
CREATE VECTOR INDEX idx_embed ON articles METRIC cosine DIM 384;

-- Insert with embedding
INSERT INTO articles { title: 'Understanding Transformers', embedding: [0.12, -0.34, 0.56, ...] };

-- Nearest neighbor search
SEARCH articles USING VECTOR(embedding, ARRAY[0.1, 0.3, -0.2, ...], 10);

-- Filtered vector search
SELECT title, vector_distance(embedding, ARRAY[0.1, 0.3, ...]) AS score
FROM articles
WHERE category = 'machine-learning'
  AND id IN (SEARCH articles USING VECTOR(embedding, ARRAY[0.1, 0.3, ...], 10));

-- Hybrid BM25 + vector (RRF fusion)
SELECT title, rrf_score(
    vector_distance(embedding, $query_vec),
    bm25_score(body, 'transformer attention')
) AS score
FROM articles
LIMIT 10;
```

## Quantization & Storage Precision

**Storage Precision** (`storage_dtype` — raw HNSW structural storage):

| Type    | Bytes/dim | Best for                           |
| ------- | --------- | ---------------------------------- |
| `F32`   | 4         | Maximum accuracy, small indexes    |
| `F16`   | 2         | Balanced memory vs accuracy        |
| `BF16`  | 2         | Better dynamic range than F16 (preferred for embeddings) |

**Quantization** (`quantization` — index-level compression, independent of storage_dtype):

Codes are traversed for speed, then top candidates are re-ranked against full-precision vectors. Memory is the per-vector code size at 384 dimensions; smaller codes trade recall for footprint and scale.

| `quantization` | Code size (384d)        | Best for                                  |
| -------------- | ----------------------- | ----------------------------------------- |
| `none`         | 1536 B (`dim×4`)        | < 1M vectors, maximum accuracy            |
| `sq8`          | 384 B (`dim×1`)         | 1–10M vectors, near-lossless              |
| `pq`           | `m` B (default `m`=16)  | 10M+ vectors, classic Product Quantization |
| `opq`          | `m` B (rotation + PQ)   | PQ + learned rotation, minor accuracy bump |
| `binary`       | 48 B (`dim/8`)          | Massive scale, sign-bit asymmetric search |
| `rabitq`       | 48 B + scalars          | 1-bit frontier with bounded error         |
| `bbq`          | 62 B (`dim/8` + 14 B)   | Centroid-asymmetric 1-bit + corrective    |
| `ternary`      | ~77 B (`dim/5`)         | BitNet b1.58 trit-packed                   |

## How It Works

Vectors are indexed in the HNSW graph at full precision. During search, quantized copies are traversed for speed, then top candidates are re-ranked against full-precision vectors. When metadata filters are present, the engine builds a Roaring Bitmap of matching IDs and selects the optimal strategy based on selectivity.


---
# Array Engine (docs/storage-engines/array)

---
title: Array Engine
description: ND sparse array engine — coordinate-tuple indexed, tile-based, bitemporal. Replaces TileDB / Zarr / SciDB / Rasdaman.
---

# Array Engine

NodeDB's array engine stores multi-dimensional sparse data with bitemporal support — system time (when the cell was written) and valid time (when the cell represents). Cells are coordinate-tuple indexed, grouped into tiles, compressed per-tile, and queryable via SQL table-valued functions.

This is a peer engine but uses its own DDL family (`CREATE ARRAY`) — not `CREATE COLLECTION ... WITH (engine='array')`.

## When to Use

- Genomics: `(chromosome × position × sample × allele)` — replaces TileDB-VCF
- Single-cell biology: `(gene × cell × condition × replicate)` — replaces TileDB-SOMA
- Earth observation: `(lat × lon × band × time)` raster cubes — replaces Zarr / TileDB-Geo
- Climate models: `(lat × lon × level × time × variable)` — replaces HDF5 + Dask
- Astronomy: `(RA × Dec × wavelength × time)` — replaces custom Zarr stacks
- Sparse ML features: `(user × item × context)` — replaces specialized matrix-factorization systems

## Key Features

- **ND coordinate-tuple keying** — arbitrary number of dimensions; only materialized cells are stored
- **Tile-based compression** — cells grouped into tiles; each tile independently compressed (ALP, FastLanes, Gorilla, LZ4 via `nodedb-codec`)
- **Z-order indexing** — Hilbert/Z-order curve linearization for spatial locality and fast range queries
- **Per-tile ND MBR statistics** — minimum bounding rectangle skip; queries prune entire tiles before decompressing
- **Bitemporal** — both system time (audit trail) and valid time (temporal semantics) tracked per tile
- **Row-major or column-major layout** — `cell_order` chosen at creation
- **Cross-engine surrogate identity** — array cells participate in cross-engine bitmap intersections alongside vector / graph / document / columnar
- **Distributed** — tiles vShard-routed; queries scatter-gather across cores and nodes
- **WAL-durable + Raft-replicated** — same durability guarantees as the rest of NodeDB
- **Tile-level retention** — `audit_retain_ms` enables GDPR / data-minimization compliance

## DDL Syntax

```sql
CREATE ARRAY spatial_grid
  DIMS (
    x INT64 [0..1000],
    y INT64 [0..1000],
    z INT64 [0..1000]
  )
  ATTRS (
    temperature FLOAT64 NOT NULL,
    pressure    FLOAT64,
    humidity    FLOAT64
  )
  TILE_EXTENTS (64, 64, 64)
  CELL_ORDER ZORDER
  TILE_ORDER ROW_MAJOR
  WITH (
    prefix_bits = 8,
    audit_retain_ms = 86400000,
    minimum_audit_retain_ms = 3600000
  );
```

| Parameter         | Required | Default     | Description                                                                                              |
| ----------------- | -------- | ----------- | -------------------------------------------------------------------------------------------------------- |
| `DIMS`            | Yes      | —           | Dimensions. Each has a name, type (`INT64`, `FLOAT64`, `TIMESTAMP_MS`, `STRING`), and optional domain `[lo..hi]`. |
| `ATTRS`           | Yes      | —           | Attributes (cell values). Each has a name, type (`INT64`, `FLOAT64`, `STRING`, `BYTES`), and optional `NOT NULL`. |
| `TILE_EXTENTS`    | Yes      | —           | Tile extent per dimension; all > 0. Determines cell locality and compression block granularity.          |
| `CELL_ORDER`      | No       | `HILBERT`   | Unquoted: `ROW_MAJOR`, `COL_MAJOR`, `HILBERT`, `ZORDER` (Z-order curve). Affects cell layout.           |
| `TILE_ORDER`      | No       | `HILBERT`   | Unquoted: `ROW_MAJOR`, `COL_MAJOR`, `HILBERT`, `ZORDER`. Affects tile layout.                           |
| `prefix_bits`     | No       | `8`         | Range 1–16. Bits used in prefix codec for compression. Higher = finer granularity, lower = better compression. |
| `audit_retain_ms` | No       | `NULL`      | Tiles older than `now - audit_retain_ms` (system time) become eligible for purge. `NULL` = keep all.    |
| `minimum_audit_retain_ms` | No | `NULL`    | Minimum retention even if `audit_retain_ms` is lower. Used to enforce compliance minimums. |

`ALTER ARRAY <name> SET (audit_retain_ms = ...)` updates retention; `DROP ARRAY <name>` is two-phase like `DROP COLLECTION`.

## Insert

```sql
CREATE ARRAY elevation_map
  DIMS (
    lon FLOAT64 [-180..180],
    lat FLOAT64 [-90..90]
  )
  ATTRS (height FLOAT64)
  TILE_EXTENTS (256, 256);

INSERT INTO ARRAY elevation_map (lon, lat, height) VALUES
  (-73.5, 40.7, 10.5),
  (-73.6, 40.8, 12.3),
  (-73.7, 40.6, 8.9);

-- Force the in-memory tiles to durable storage
SELECT ARRAY_FLUSH('elevation_map');
```

## Query Functions

Array queries are expressed as table-valued functions in `FROM`. System time and valid time apply via `AS OF` clauses.

### `ARRAY_SLICE` — multi-dimensional range

```sql
SELECT * FROM ARRAY_SLICE(
  'elevation_map',
  {lon: [-74.0, -73.0), lat: [40.0, 41.0)},
  ['height'],   -- attribute projection (optional)
  1000          -- max cells (optional)
);
```

| Parameter | Required | Type           | Description                                                     |
| --------- | -------- | -------------- | --------------------------------------------------------------- |
| `array`   | Yes      | `STRING`       | Array name                                                      |
| `bounds`  | Yes      | `OBJECT`       | `{ dim: [lo, hi) }`. Omitted dims = full range.                 |
| `attrs`   | No       | `ARRAY[STRING]`| Attributes to project. `NULL` = all attributes.                 |
| `limit`   | No       | `INT64`        | Max cells returned. `NULL` = no limit.                          |

### `ARRAY_PROJECT` — attribute projection

```sql
SELECT * FROM ARRAY_PROJECT('spatial_grid', ['temperature', 'pressure']);
```

### `ARRAY_AGG` — reduce a dimension

Aggregates an attribute over a dimension, reducing dimensionality:

```sql
-- Sum temperature over x; result keeps y and z
SELECT * FROM ARRAY_AGG('spatial_grid', 'temperature', 'SUM', 'x');
```

Reducers: `'SUM'`, `'AVG'`, `'MIN'`, `'MAX'`, `'COUNT'`.

### `ARRAY_ELEMENTWISE` — between two arrays of the same shape

```sql
SELECT * FROM ARRAY_ELEMENTWISE('current_grid', 'baseline_grid', 'SUBTRACT', 'temperature');
```

## Maintenance

```sql
SELECT ARRAY_FLUSH('spatial_grid');    -- force memtable flush
SELECT ARRAY_COMPACT('spatial_grid');  -- merge tile versions, reclaim space
```

`ARRAY_FLUSH` always returns `{result: true}` on success; failure raises. Compaction also runs automatically in the background.

## Bitemporal Queries

Every array cell carries two times:

- **System time** — when the value was written (audit trail, compliance, point-in-time recovery)
- **Valid time** — when the value represents (forecasts, backdated corrections, scientific replays)

```sql
-- Read cells as the array existed in the past
SELECT * FROM ARRAY_SLICE('data', {x: [0, 100), y: [0, 100)}, ['value'])
AS OF SYSTEM TIME 1700000000000;

-- Read cells whose valid-time interval includes a given moment
SELECT * FROM ARRAY_SLICE('forecast', {x: [0, 100), y: [0, 100)}, ['temp'])
AS OF VALID TIME 1700000000000;

-- Both clauses combined
SELECT * FROM ARRAY_SLICE('forecast', {x: [0, 100), y: [0, 100)}, ['temp'])
AS OF SYSTEM TIME 1700000000000 AS OF VALID TIME 1700000001000;
```

System-time–based retention is the path to GDPR and data-minimization compliance: `audit_retain_ms` makes tiles older than the window eligible for irreversible purge during compaction.

## Cross-Engine Queries

Array cells participate in surrogate-identity bitmaps with the rest of the engines, so a single query can prefilter by vector neighborhood and slice an array:

```sql
SELECT *
FROM ARRAY_SLICE('spatial_data', {x: [0, 1000), y: [0, 1000)}, ['attr1', 'attr2'])
WHERE id IN (
  SEARCH vectors USING VECTOR(embedding, $query, 100)
);
```

See [Architecture Overview](../architecture/overview) for the cross-engine identity model.

## Performance

- **Tile-level parallelism** — each tile is read and processed on its own core
- **Compression** — typical 5–20× depending on data homogeneity
- **Range queries** — Z-order layout gives cache-friendly access; ND-MBR per-tile stats prune irrelevant tiles before decompression
- **Sparse-friendly** — only materialized cells are stored; implicit zeros and empty regions cost nothing


---
# Graph Engine (docs/storage-engines/graph)

---
title: Graph Engine
description: CSR adjacency index with 13 algorithms, Cypher-subset MATCH, and GraphRAG fusion.
---

# Graph Engine

The graph engine uses a native CSR (Compressed Sparse Row) adjacency index — not recursive JOINs. At 1 billion edges, CSR uses ~10 GB vs ~60 GB for naive adjacency lists. Sub-millisecond multi-hop traversals, 13 native algorithms, Cypher-subset pattern matching, and GraphRAG fusion.

## When to Use

- Knowledge graphs and entity relationships
- Social networks and recommendation
- Fraud detection (pattern matching)
- Supply chain and dependency analysis
- RAG pipelines with graph context (GraphRAG)

## Graph is an Overlay

Graph edges are an overlay on document collections. Any collection can have graph edges — you don't create a separate "graph collection."

```sql
CREATE COLLECTION people;
INSERT INTO people (id, name) VALUES ('alice', 'Alice');
INSERT INTO people (id, name) VALUES ('bob', 'Bob');

GRAPH INSERT EDGE IN 'people' FROM 'alice' TO 'bob' TYPE 'knows' PROPERTIES { since: 2020, weight: 0.9 };
-- The IN '<collection>' clause is required — edges live on a named collection.
```

## Traversal

```sql
-- BFS traversal
GRAPH TRAVERSE FROM 'alice' DEPTH 3;
GRAPH TRAVERSE FROM 'alice' DEPTH 2 LABEL 'follows' DIRECTION out;

-- Immediate neighbors
GRAPH NEIGHBORS OF 'bob' LABEL 'follows' DIRECTION both;

-- Shortest path
GRAPH PATH FROM 'alice' TO 'charlie' MAX_DEPTH 5 LABEL 'knows';
```

## MATCH Pattern Queries

Cypher-subset pattern matching:

```sql
-- Friend-of-friend
MATCH (a:Person)-[:knows]->(b:Person)-[:knows]->(c:Person)
WHERE a.name = 'Alice'
RETURN b.name, c.name;

-- Variable-length paths
MATCH (u:User)-[:follows*2..3]->(recommended:User)
WHERE u.id = 'you'
RETURN DISTINCT recommended.id LIMIT 10;

-- Anti-join
MATCH (a:User)-[:follows]->(b:User)
WHERE NOT EXISTS { MATCH (b)-[:blocked_by]->(a) }
RETURN a.id, b.id;

-- OPTIONAL MATCH
MATCH (a:Person)-[:knows]->(b:Person)
OPTIONAL MATCH (b)-[:works_at]->(c:Company)
RETURN a.name, b.name, c.name;
```

## 13 Algorithms

```sql
GRAPH ALGO PAGERANK ON 'social' DAMPING 0.85 ITERATIONS 20 TOLERANCE 1e-7;
GRAPH ALGO PAGERANK ON 'social' PERSONALIZATION {"alice": 1.0, "bob": 0.5};
GRAPH ALGO WCC ON 'knowledge_graph';
GRAPH ALGO SSSP ON 'routes' FROM 'city:chicago';
GRAPH ALGO COMMUNITY ON 'products' ITERATIONS 10 RESOLUTION 1.0;
GRAPH ALGO BETWEENNESS ON 'network' SAMPLE 500;
GRAPH ALGO KCORE ON 'collaboration';
GRAPH ALGO TRIANGLES ON 'social' MODE global;
GRAPH ALGO DIAMETER ON 'web';
```

**Personalized PageRank** biases node importance toward seed nodes via the `PERSONALIZATION` clause — a JSON object mapping node IDs to seed weights.

| Algorithm         | Computes                                     |
| ----------------- | -------------------------------------------- |
| PageRank          | Node importance via link structure            |
| WCC               | Weakly connected components                   |
| Label Propagation | Community detection via label spreading        |
| LCC               | Local clustering coefficient                  |
| SSSP              | Single-source shortest path (Dijkstra)        |
| Betweenness       | Bridge node identification                    |
| Closeness         | How close a node is to all others             |
| Harmonic          | Closeness for disconnected graphs             |
| Degree            | Connection count (in/out/both)                |
| Louvain           | Community detection via modularity             |
| Triangles         | Triangle count (per-node or global)           |
| Diameter          | Longest shortest path                         |
| k-Core            | Coreness decomposition                        |

## GraphRAG

Combines vector similarity with graph traversal in one query:

```sql
GRAPH RAG FUSION ON entities
  QUERY $embedding
  VECTOR_FIELD 'embedding'
  VECTOR_TOP_K 50
  EXPANSION_DEPTH 2
  EDGE_LABEL 'related_to'
  FINAL_TOP_K 10
  RRF_K (60.0, 35.0);
```

1. Vector search finds semantically similar seed nodes
2. BFS expands seeds along edges
3. RRF merges vector rank with graph hop distance

## Graph Statistics

The collection name is an optional single-quoted literal:

```sql
SHOW GRAPH STATS;                                       -- tenant-wide aggregate
SHOW GRAPH STATS 'collection_name';
SHOW GRAPH STATS 'collection_name' VERBOSE;
SHOW GRAPH STATS 'collection_name' AS OF SYSTEM TIME <ms>;
```

Returns live edge counters and optional cardinality details:

- Omitting the collection name returns tenant-wide aggregate edge counts
- `VERBOSE` adds a per-edge-type (label) breakdown — one row per `(collection, label, edge_count)`
- `AS OF SYSTEM TIME <ms>` retrieves historical counts (O(1) lookup from WAL snapshots)


---
# Full-Text Search Engine (docs/storage-engines/fts)

---
title: Full-Text Search Engine
description: Block-Max WAND BM25 with 27-language support, CJK tokenization, fuzzy matching, and hybrid vector fusion.
---

# Full-Text Search Engine

Block-Max WAND (BMW) optimized BM25 ranking with 16 Snowball stemmers, 27-language stop words, CJK bigram tokenization, posting compression, fuzzy matching, and native hybrid fusion with vector search.

## When to Use

- Text search across documents, articles, products, logs
- Search-as-you-type with fuzzy matching
- Multilingual content search (including CJK, Arabic, Hindi)
- Hybrid retrieval: keyword matching + semantic similarity

## SQL Usage

```sql
CREATE COLLECTION articles;
CREATE SEARCH INDEX ON articles FIELDS title, body ANALYZER 'english' FUZZY true;

-- Basic search
SELECT title, bm25_score(body, 'distributed database') AS score
FROM articles
WHERE text_match(body, 'distributed database')
ORDER BY score DESC LIMIT 20;

-- Fuzzy search
SELECT title FROM articles WHERE text_match(title, 'databse', { fuzzy: true, distance: 2 });

-- Hybrid BM25 + vector (RRF)
SELECT title, rrf_score(
    vector_distance(embedding, $query_vec),
    bm25_score(body, 'distributed systems')
) AS score
FROM articles
LIMIT 10;

-- Synonyms
CREATE SYNONYM GROUP db_terms AS ('database', 'db', 'datastore');
```

## Analyzers

| Analyzer     | Behavior                                        |
| ------------ | ----------------------------------------------- |
| `standard`   | NFD normalize, lowercase, English stop/stem     |
| `simple`     | Lowercase + whitespace split                    |
| `keyword`    | Entire input as a single token                  |
| `cjk_bigram` | CJK bigram tokenization                        |
| `ngram:2:4`  | Character n-grams (min:max)                     |
| `edge_ngram` | Prefix-anchored n-grams for autocomplete        |

16 language-specific analyzers: `ar`, `da`, `nl`, `en`, `fi`, `fr`, `de`, `hu`, `it`, `no`, `pt`, `ro`, `ru`, `es`, `sv`, `tr`.

CJK text is automatically routed to bigram tokenizer regardless of configured analyzer. Optional dictionary segmentation via feature gates: `lang-ja`, `lang-zh`, `lang-ko`, `lang-th`.

## Internals

- **BMW scoring** — WAND pivot selection + 128-doc block pruning via precomputed upper bounds
- **Posting compression** — Delta-encoded, variable-width bitpacked doc IDs with SIMD unpack (SSE2/NEON)
- **SmallFloat fieldnorms** — 1-byte length quantization (4x space reduction)
- **LSM storage** — In-memory memtable → immutable segments → level-based compaction (8x8 tiering)
- **AND-first with OR fallback** — Tries AND; falls back to OR with coverage penalty if zero results
- **Phrase proximity boost** — Consecutive tokens at consecutive positions get up to 3x score boost


---
# CRDT Engine (docs/storage-engines/crdt)

---
title: CRDT Engine
description: Loro-backed conflict-free replicated data types for offline-first edge sync with cloud consistency.
---

# CRDT Engine

The CRDT engine provides conflict-free replication for offline-first applications. Locally, writes are available immediately (AP). Globally, deltas are committed through Multi-Raft consensus (CP). SQL constraints are enforced at sync time.

## How It Works

1. Device writes locally — available immediately, no network required
2. Write produces a CRDT delta (Loro)
3. When connectivity returns, delta syncs to Origin via WebSocket
4. Origin validates SQL constraints (UNIQUE, FK, CHECK) at Raft commit
5. If constraints pass, delta is committed and replicated
6. If constraints fail, Origin sends a typed `CompensationHint` back to the device

## Conflict Resolution

Declarative conflict policies per collection:

```sql
-- Last-writer-wins (default)
CREATE COLLECTION notes WITH (conflict_policy = 'lww');

-- Rename suffix for UNIQUE conflicts
CREATE COLLECTION profiles WITH (conflict_policy = 'rename_suffix');
```

The `PolicyRegistry` stores per-collection resolution strategies. Available policies: `lww` (last-writer-wins), `rename_suffix` (append suffix on UNIQUE conflict), `cascade_defer` (retry with backoff for FK violations), `custom` (webhook), `escalate_to_dlq` (send to dead-letter queue).

## Constraint Validation

SQL constraints are checked on Origin at sync time, not on the device:

- **UNIQUE** — duplicate key → `CompensationHint::Rename` or `CompensationHint::Merge`
- **FK** — dangling reference → `CompensationHint::CreateParent` or `CompensationHint::Discard`
- **CHECK** — constraint violation → `CompensationHint::Adjust`

The application handles compensation — no silent data loss.

## Dead-Letter Queue

Deltas that persistently fail constraint validation are routed to a dead-letter queue with diagnostic metadata. Operators can inspect, replay, or discard them.

## Related

- [CRDT Sync](/docs/crdt-sync/overview) — Full sync protocol and offline patterns
- [Document Engine](/docs/storage-engines/document) — Schemaless documents with CRDT sync


---
# HNSW Index (docs/indexes/hnsw-index)

---
title: HNSW Index
description: Hierarchical Navigable Small World graph for approximate nearest neighbor search on vector embeddings.
---

# HNSW Index

HNSW (Hierarchical Navigable Small World) is a multi-layer proximity graph for approximate nearest neighbor search. It provides logarithmic search complexity with high recall.

## Creating an HNSW Index

```sql
CREATE VECTOR INDEX idx_embed ON articles METRIC cosine DIM 384;

-- With explicit parameters
CREATE VECTOR INDEX idx_embed ON articles
    METRIC cosine DIM 384
    M 16 EF_CONSTRUCTION 200;
```

## Parameters

| Parameter          | Default | Description                                                   |
| ------------------ | ------- | ------------------------------------------------------------- |
| `METRIC`           | —       | Distance metric: `l2`, `cosine`, `inner_product`, `manhattan`, `chebyshev`, `hamming`, `jaccard`, `pearson` |
| `DIM`              | —       | Vector dimension (must match your embeddings)                 |
| `M`                | 16      | Max connections per node per layer. Higher = more accurate, more memory |
| `EF_CONSTRUCTION`  | 200     | Search width during index build. Higher = slower build, better graph quality |

## How It Works

- **Construction** — Vectors inserted at full precision (FP32/FP16) to maintain structural integrity
- **Search** — Traverses quantized copies for speed, then re-ranks top candidates against full-precision vectors
- **Layers** — Upper layers provide coarse navigation; bottom layer is fully connected

## Quantization

Add quantization to reduce memory:

| Quantization | Memory reduction | Recall impact |
| ------------ | ---------------- | ------------- |
| SQ8          | ~4x              | Minimal       |
| PQ           | ~4-8x            | ~5% loss      |
| IVF-PQ       | ~16 bytes/vector | ~5-15% loss   |

## Search

```sql
-- k-NN search
SEARCH articles USING VECTOR(embedding, ARRAY[0.1, 0.3, ...], 10);

-- With pre-filtering
SELECT * FROM articles
WHERE category = 'ml'
  AND id IN (SEARCH articles USING VECTOR(embedding, $vec, 10));
```


---
# CSR Index (docs/indexes/csr-index)

---
title: CSR Index
description: Compressed Sparse Row adjacency index for cache-resident graph traversal.
---

# CSR Index

CSR (Compressed Sparse Row) is the graph engine's core index format. It stores adjacency data in contiguous arrays for cache-resident traversal.

## Layout

```
CsrIndex (per tenant):
  out_offsets:  Vec<u32>                 [num_nodes + 1]  — offset into target array per node
  out_targets:  DenseArray<u32>          [num_edges]      — destination node IDs (contiguous, mmap-capable)
  out_labels:   DenseArray<u32>          [num_edges]      — edge labels (parallel array)
  out_weights:  Option<DenseArray<f64>>                   — optional, allocated only when weighted

  in_offsets / in_targets / in_labels / in_weights — symmetric for inbound
```

## Tenant Partitioning

The in-memory index is `ShardedCsrIndex` — one `CsrIndex` per tenant. Algorithms and traversals receive a single tenant's partition; there is no lexical tenant prefix on node names. Each `CsrIndex` is assigned a unique partition tag at construction, and public APIs that return dense node indices hand out `LocalNodeId { id, partition_tag }`. Using a node id from one partition with another partition's API panics at the boundary.

## Memory Efficiency

At 1 billion edges, CSR uses ~10 GB vs ~60 GB for naive adjacency lists (6x improvement). Node IDs are interned as `u32`, labels as `u32`.

## Storage

Edges are persisted in a redb B-Tree with forward and reverse indexes, both keyed by `(tenant_id: u32, "src\x00label\x00dst")` tuples. Tenant isolation is structural (first-class key component), not lexical. The CSR index is built at query time for bulk operations. Writes go to a mutable buffer and become visible immediately. Compaction merges the buffer into dense CSR arrays when the buffer exceeds 10% of the dense size.

## Graph Operations

The CSR index supports all traversal and algorithm operations:

- BFS/DFS traversal
- Shortest path (Dijkstra)
- All 13 native graph algorithms
- MATCH pattern matching
- GraphRAG fusion


---
# R*-tree Index (docs/indexes/rtree-index)

---
title: R*-tree Index
description: Spatial index for range queries, nearest neighbor, and OGC predicate evaluation.
---

# R*-tree Index

The R*-tree is the spatial engine's primary index. It supports range queries, nearest neighbor search, and spatial predicate evaluation.

## Creating a Spatial Index

```sql
-- Automatic via SPATIAL_INDEX column modifier (spatial peer engine)
CREATE COLLECTION locations (
    geom GEOMETRY SPATIAL_INDEX,
    name VARCHAR
) WITH (engine='spatial');

-- Or add to any collection
CREATE SPATIAL INDEX ON restaurants FIELDS location;
```

## Operations

- **Range query** — `ST_DWithin`, `ST_Within`, `ST_Intersects`
- **Nearest neighbor** — `ORDER BY ST_Distance(geom, point) LIMIT k`
- **Bulk load** — Optimized for batch inserts
- **Spatial join** — R*-tree probe join between two collections

## Query Execution

When a spatial predicate is present, the R*-tree narrows the candidate set before the columnar sparse index does final refinement. Queries without spatial predicates read directly from the columnar memtable — the R*-tree is not involved.

```sql
-- R*-tree lookup → sparse refinement → result
SELECT name FROM locations WHERE ST_DWithin(geom, ST_Point(-73.98, 40.75), 500);

-- No spatial predicate → direct columnar scan
SELECT name FROM locations WHERE name LIKE 'Park%';
```


---
# B-tree Index (docs/indexes/btree-index)

---
title: B-tree Index
description: redb B-Tree indexes for metadata lookups and secondary indexes on document collections.
---

# B-tree Index

B-tree indexes power metadata lookups and secondary indexes on document collections. Backed by redb's ACID B-Tree storage.

## Creating Indexes

```sql
-- Secondary index on a document collection
CREATE INDEX ON users FIELDS email;

-- Compound index
CREATE INDEX ON orders FIELDS customer_id, status;

-- Unique index
CREATE UNIQUE INDEX ON users FIELDS email;
```

## When Used

- Point lookups on document collections (`WHERE email = 'alice@example.com'`)
- Range scans (`WHERE age > 25 AND age < 40`)
- Sorting (`ORDER BY created_at DESC`)
- Constraint enforcement (UNIQUE)

## Graph Edge Storage

Graph edges are persisted in redb B-Trees with forward and reverse indexes, keyed by `(tenant_id: u32, "src\x00label\x00dst")` tuples. Tenant isolation is a first-class key component, not a lexical prefix; the composite portion enables prefix scans for outbound traversal within a tenant.


---
# Inverted Index (docs/indexes/inverted-index)

---
title: Inverted Index
description: LSM-based inverted index for full-text search with Block-Max WAND scoring.
---

# Inverted Index

The inverted index maps terms to document IDs for full-text search. It uses an LSM architecture with posting compression and Block-Max WAND scoring.

## Creating a Search Index

```sql
CREATE SEARCH INDEX ON articles FIELDS title, body ANALYZER 'english' FUZZY true;
```

## Architecture

**Memtable** — Writes accumulate in-memory (`HashMap<String, Vec<CompactPosting>>`). When the memtable exceeds the threshold (32M posting entries or 100K unique terms), it flushes to an immutable segment.

**Segments** — Compressed on-disk segments with delta-encoded, bitpacked posting lists. Level-based compaction (8 levels, 8 segments per level).

**Query merge** — Searches merge the active memtable with all persisted segments.

## Posting Compression

- Delta encoding for sorted doc IDs
- Variable-width bitpacking (3-byte header: `[count: u16][bit_width: u8]`)
- SIMD-accelerated unpack (SSE2 on x86_64, NEON on AArch64)
- SmallFloat fieldnorms (1 byte per document, 4x space reduction)

## Block-Max WAND

Posting lists are split into 128-document blocks with precomputed `block_max_tf` and `block_min_fieldnorm`. During scoring, blocks that can't beat the current top-k threshold are skipped entirely.


---
# Hash Index (docs/indexes/hash-index)

---
title: Hash Index
description: O(1) hash-based index for key-value point lookups.
---

# Hash Index

The hash index provides O(1) point lookups by user-defined key. It is the primary index for KV collections.

## How It Works

Keys are hashed to locate the value directly — no tree traversal. This provides constant-time reads regardless of collection size.

## Creating a KV Collection

```sql
CREATE COLLECTION sessions (key TEXT PRIMARY KEY) WITH (engine='kv');
```

The `PRIMARY KEY` column is automatically hash-indexed.

## Secondary Indexes

KV collections can also have secondary B-tree indexes on value fields:

```sql
CREATE INDEX ON sessions FIELDS role;
SELECT key, user_id FROM sessions WHERE role = 'admin';
```

## TTL

Native TTL with an index-backed expiry wheel. Keys expire automatically:

```sql
INSERT INTO sessions { key: 'sess_abc', user_id: 'alice', ttl: 3600 };
```


---
# Bitemporal Queries (docs/temporal/bitemporal)

---
title: Bitemporal Queries
description: Track data along system time (when written) and valid time (when valid). Audit trails, backdated corrections, and compliance-grade history across multiple engines.
---

# Bitemporal Queries

Bitemporal databases track data along two independent time dimensions:

- **System time** — when the database recorded the value. Stored *in the key* (structural), derived from the WAL LSN at Raft commit. Used for audit trails and asking "what did the database think was true at moment X?".
- **Valid time** — when the value represents. Stored *in the value*, client/device-assigned. Used for forecasts, backdated corrections, and asking "what was true in the world at moment X?".

Closed-open intervals: `[from, to)`; an open upper bound is `i64::MAX`.

## Supported Engines

| Engine                                                       | System Time | Valid Time | Example use case                  |
| ------------------------------------------------------------ | :---------: | :--------: | --------------------------------- |
| [Graph](../storage-engines/graph) (edges + nodes)            |     Yes     |    Yes     | Entity-relationship timelines     |
| [Document (strict)](../storage-engines/document)             |     Yes     |    Yes     | Versioned profiles, ledgers       |
| [Document (schemaless)](../storage-engines/document)         |     Yes     |    Yes     | Event logs with backdated entries |
| [Columnar](../storage-engines/columnar)                      |     Yes     |    Yes     | Audit tables, corrected metrics   |
| [Timeseries](../storage-engines/timeseries)                  |     Yes     |    Yes     | Forecast corrections, data repair |
| [Array](../storage-engines/array) (tile-level versioning)    |     Yes     |    Yes     | Historical spatial snapshots      |

**Index engines (Vector, Full-Text Search, Spatial R\*-tree, Key-Value)** do not carry temporal columns themselves — they index records that live in data-bearing collections. To query at a point in time, attach the index to a `bitemporal=true` collection; `AS OF` filtering happens at the collection layer, not the index. See [Index engines and temporal composition](#index-engines-and-temporal-composition).

## SQL Syntax

`AS OF` clauses sit in the `FROM` part of a query. Times must be one of:
1. Integer milliseconds since Unix epoch (e.g., `1700000000000`)
2. `NOW()` — current time
3. ISO-8601 string literal (e.g., `'2024-01-15T00:00:00Z'`)

```sql
-- Read the database state as of a past system time
SELECT * FROM collection
AS OF SYSTEM TIME 1700000000000;

-- Read rows whose valid-time interval includes a given moment
SELECT * FROM collection
AS OF VALID TIME 1700000000000;

-- Both: rows that were valid AND in the system at a point
SELECT * FROM collection
AS OF SYSTEM TIME 1700000000000
AS OF VALID TIME 1700000001000;
```

## Examples

### Audit Trail (System Time)

```sql
CREATE COLLECTION user_accounts (
    id UUID DEFAULT gen_uuid_v7(),
    email VARCHAR,
    balance DECIMAL,
    created_at TIMESTAMP DEFAULT now()
) WITH (engine='document_strict', bitemporal=true);

INSERT INTO user_accounts (email, balance) VALUES ('alice@example.com', 100.00);
UPDATE user_accounts SET balance = 150.00 WHERE email = 'alice@example.com';

-- Database state 10 minutes ago
SELECT email, balance FROM user_accounts
AS OF SYSTEM TIME '2026-06-07T11:50:00Z';
-- → alice@example.com, 100.00

-- Current state
SELECT email, balance FROM user_accounts;
-- → alice@example.com, 150.00
```

### Backdated Corrections (Valid Time)

```sql
CREATE COLLECTION sensor_readings (
    ts TIMESTAMP TIME_KEY,
    location VARCHAR,
    temperature FLOAT
) WITH (engine='timeseries', bitemporal=true);

-- Original reading
INSERT INTO sensor_readings (ts, location, temperature, valid_time)
VALUES ('2026-04-01T10:00:00Z', 'warehouse-a', 21.5, '2026-04-01T10:00:00Z');

-- Discover it was wrong; insert a correction valid as-of April 2nd
INSERT INTO sensor_readings (ts, location, temperature, valid_time)
VALUES ('2026-04-01T10:00:00Z', 'warehouse-a', 22.3, '2026-04-02T15:30:00Z');

-- What we knew on April 1st (before correction)
SELECT location, temperature FROM sensor_readings
WHERE ts BETWEEN '2026-04-01' AND '2026-04-02'
AS OF VALID TIME '2026-04-01T10:00:00Z';

-- What we know now (after correction)
SELECT location, temperature FROM sensor_readings
WHERE ts BETWEEN '2026-04-01' AND '2026-04-02'
AS OF VALID TIME '2026-04-02T15:30:00Z';
```

### Bitemporal Array Snapshot

```sql
CREATE ARRAY climate_grid
  DIMS (lon INT64 [-180..180], lat INT64 [-90..90])
  ATTRS (temp_c FLOAT64)
  TILE_EXTENTS (64, 64)
  WITH (audit_retain_ms = 7776000000);  -- 90 days

-- Cells as committed yesterday
SELECT lon, lat, temp_c FROM ARRAY_SLICE(
    'climate_grid',
    {lon: [-10, 10), lat: [0, 20)},
    ['temp_c']
)
AS OF SYSTEM TIME '2026-06-06T00:00:00Z';
```

### Lineage and Compliance

```sql
SELECT status, _ts_system FROM transactions
WHERE id = 'txn-1'
AS OF SYSTEM TIME NULL  -- returns every system-time version, ascending
ORDER BY _ts_system ASC;
```

`AS OF SYSTEM TIME NULL` returns every system-time version of each matching row, ascending, with the `_ts_system` column projected. Supported on the Document (strict and schemaless), Columnar, and Timeseries engines; not supported on the Graph or Array engines, nor through a database clone — those return a typed error rather than collapsing to a single version.

## Index Engines and Temporal Composition

Vector, FTS, KV, and Spatial don't carry temporal columns — by design. They are **index engines** that point at records living in data-bearing engines. Indexes don't have time; the records they reference do.

To query any of these "as of", attach the index to a collection with `bitemporal=true`. The collection holds the payload + temporal columns, the index returns candidate IDs, and `AS OF` filters at the collection layer.

### Bitemporal Vector Search

```sql
CREATE COLLECTION product_embeddings (
    id UUID DEFAULT gen_uuid_v7(),
    product_id UUID,
    description TEXT,
    embedding FLOAT[384],
    updated_at TIMESTAMP DEFAULT now()
) WITH (engine='document_strict', bitemporal=true);

CREATE VECTOR INDEX idx_product_vec ON product_embeddings METRIC cosine DIM 384;

-- Nearest neighbors as of 30 days ago
SELECT p.product_id, p.description,
       vector_distance(p.embedding, $query_vec) AS score
FROM product_embeddings p
AS OF SYSTEM TIME '2026-05-08T00:00:00Z'
WHERE p.id IN (
    SEARCH product_embeddings USING VECTOR(embedding, $query_vec, 20)
)
ORDER BY score
LIMIT 10;
```

The vector index narrows the candidate set; the `AS OF SYSTEM TIME` clause on the collection removes candidates that didn't exist at that system time.

### Bitemporal Full-Text Search

```sql
CREATE COLLECTION articles (
    id UUID DEFAULT gen_uuid_v7(),
    title VARCHAR,
    body TEXT,
    published_at TIMESTAMP
) WITH (engine='document_strict', bitemporal=true);

CREATE SEARCH INDEX ON articles FIELDS title, body ANALYZER 'english';

SELECT id, title FROM articles
AS OF SYSTEM TIME '2026-06-06T00:00:00Z'
WHERE text_match(body, 'distributed consensus raft')
ORDER BY bm25_score(body, 'distributed consensus raft') DESC
LIMIT 20;
```

### Bitemporal Spatial Queries

```sql
CREATE COLLECTION store_locations (
    id UUID DEFAULT gen_uuid_v7(),
    name VARCHAR,
    location GEOMETRY,
    opened_at TIMESTAMP,
    closed_at TIMESTAMP
) WITH (engine='document_strict', bitemporal=true);

CREATE SPATIAL INDEX ON store_locations FIELDS location;

SELECT id, name, ST_Distance(location, ST_Point(-73.990, 40.750)) AS dist_m
FROM store_locations
AS OF SYSTEM TIME 1704067200000
WHERE ST_DWithin(location, ST_Point(-73.990, 40.750), 5000)
ORDER BY dist_m;
```

### Temporal Key-Value: Use `document_strict`

The KV engine targets O(1) point lookups and doesn't carry temporal columns. For versioned config, auditable feature flags, or time-stamped session state, use a `document_strict` collection with a unique index — O(1) lookup is preserved, temporal history comes from the document layer:

```sql
CREATE COLLECTION config_entries (
    id UUID DEFAULT gen_uuid_v7(),
    key VARCHAR,
    value TEXT,
    updated_at TIMESTAMP DEFAULT now()
) WITH (engine='document_strict', bitemporal=true);

CREATE UNIQUE INDEX ON config_entries(key);

INSERT INTO config_entries (key, value) VALUES ('feature_x_enabled', 'true')
ON CONFLICT (key) DO UPDATE SET value = EXCLUDED.value;

SELECT value FROM config_entries WHERE key = 'feature_x_enabled';

-- What was the value 7 days ago?
SELECT value FROM config_entries
AS OF SYSTEM TIME '2026-05-31T00:00:00Z'
WHERE key = 'feature_x_enabled';
```

## GDPR and Data Minimization

`audit_retain_ms` enforces automatic purge of old versions during compaction:

```sql
CREATE COLLECTION user_activity (
    user_id UUID,
    action VARCHAR,
    ts TIMESTAMP TIME_KEY
) WITH (engine='columnar', audit_retain_ms=2592000000);  -- 30 days
```

Tiles or row-versions older than the window are irreversibly purged. Historical queries beyond the retention window return no rows for that range.

## Performance Notes

- **System time queries** read from historical snapshots — performance depends on snapshot availability for the requested time
- **Valid time queries** scan all versions and filter — slower than single-version reads
- **Both clauses** intersect — slower still, but produce precise audit trails

For large collections with heavy correction traffic: archive old versions to L2 (S3) cold storage periodically, reduce `audit_retain_ms` once the compliance window expires, and rely on columnar compression to keep storage overhead low.


---
# SQL Reference Index (docs/sql/index)

---
title: SQL Reference Index
description: Complete index of NodeDB SQL features — standard SQL, engine extensions, DDL, DML, and known gaps.
---

# SQL Reference

NodeDB exposes a PostgreSQL-compatible SQL surface over pgwire, HTTP, and the native protocol. All features below are available in every deployment mode (Origin cloud, Origin local, NodeDB-Lite where noted).

## Core SQL

| Page | What it covers |
| ---- | -------------- |
| [Overview](overview) | Query execution model, PG compatibility basics, isolation level |
| [DDL](ddl) | Collections, indexes, triggers, functions, materialized views, cron, backup |
| [SELECT](select) | Filtering, aggregates, window functions, CTEs, recursive CTEs, set operations |
| [INSERT & UPSERT](insert-upsert) | INSERT, ON CONFLICT, UPSERT, COPY FROM |
| [UPDATE & DELETE](update-delete) | UPDATE, UPDATE … FROM, DELETE, RETURNING, MERGE, transactions |
| [JOINs & Aggregations](joins-aggregations) | Inner/outer/semi/anti joins, LATERAL, GROUP BY ROLLUP/CUBE/GROUPING SETS |
| [Functions](functions) | Scalar, aggregate, date/time, KV atomic, timeseries, vector, FTS, CRDT |

## Engine-Specific SQL

| Page | What it covers |
| ---- | -------------- |
| [Vector Search](vector-search) | ANN, filtered search, quantization, vector-primary collections |
| [Full-Text Search](fulltext-search) | BM25, fuzzy, NOT operator, synonyms, hybrid search |
| [Graph Queries](graph-queries) | Edge ops, traversal, MATCH patterns, algorithms, GraphRAG |
| [Spatial Queries](spatial-queries) | OGC predicates, proximity, geofencing, H3, spatial joins |
| [CRDT Operations](crdt-operations) | crdt_state(), crdt_apply(), conflict policies |
| [Fusion & RRF](fusion-rrf) | Two-source and three-source Reciprocal Rank Fusion, FUSION DSL |

## SQL Extensions

| Page | What it covers |
| ---- | -------------- |
| [JSON Operators](json-operators) | `->`, `->>`  `#>`, `#>>`, `@>`, `<@`, `?`, `?|`, `?&` |
| [COPY](copy) | COPY FROM (import), COPY TO (export), NDJSON / CSV / JSON array |
| [LISTEN / NOTIFY](listen-notify) | Async pgwire notifications, transaction buffering, tenant scoping |
| [Custom Types](custom-types) | CREATE TYPE AS ENUM, CREATE TYPE AS composite, ALTER TYPE ADD VALUE |
| [Window Frames](window-frames) | ROWS, RANGE, GROUPS frame modes with all boundary combinations |
| [Lateral Subqueries](lateral-subqueries) | LATERAL FROM, JOIN LATERAL, planning modes |
| [Recursive CTEs](recursive-cte) | WITH RECURSIVE, value-generating forms, validation rules |
| [Grouping Sets](grouping-sets) | GROUP BY ROLLUP, CUBE, GROUPING SETS, GROUPING() function |

## Diagnostics

| Page | What it covers |
| ---- | -------------- |
| [EXPLAIN](explain) | Query plans, cross-shard dispatch, session variables, introspection |

## Known Gaps

Features not yet available:

- **Transactional DDL** — `CREATE COLLECTION`, `CREATE INDEX`, and other DDL statements auto-commit and cannot be rolled back inside a multi-statement transaction.
- **`EXPLAIN ANALYZE`** — per-operator runtime stats require Data Plane instrumentation not yet wired. `EXPLAIN` (plan-only) works.
- **Custom types as pgwire wire-format codecs** — enum and composite types are stored and queryable, but the pgwire layer sends them as `TEXT` rather than their registered OID binary format.
- **`ALTER TYPE` beyond `ADD VALUE`** — renaming labels, dropping labels, and reordering composite fields are not yet supported.
- **`ALTER TYPE … RENAME VALUE`** — blocked by the same gap above.
- **`DROP TYPE` cascade** — `DROP TYPE` is protected; it fails if the type is in use. Cascade dropping the dependent collections first is required manually.
- **`CHECK` constraints, `FOREIGN KEY`, `REFERENCES`** — rejected at DDL time. See [DDL](ddl) for the canonical replacements.
- **`UPDATE/DELETE … JOIN` syntax** — rewrite as a subquery. `UPDATE … FROM` is supported; `UPDATE … JOIN` is not.
- **Full `ALTER TABLE` / `ALTER COLLECTION`** — `ADD COLUMN` and `DROP COLUMN` work; renaming columns, changing column types, and adding/dropping constraints are not yet supported.
- **`COPY FROM STDIN`** — the streaming STDIN form is handled only by the backup/restore path. File-path `COPY FROM` and `COPY TO` work.
- **`LISTEN` on non-collection channels** — `LISTEN` subscribes to a collection's change stream. Arbitrary string channel names (Postgres style) are not supported.
- **Prepared statements for DDL** — only DML (SELECT, INSERT, UPDATE, DELETE) and some DQL statements support the extended-query (prepared) protocol.


---
# SQL Overview (docs/sql/overview)

---
title: SQL Overview
description: NodeDB uses SQL as its primary query language, extended with engine-specific syntax for vectors, graphs, spatial, and more.
---

# SQL Overview

NodeDB uses SQL as its primary query language. Whether you connect via `ndb`, `psql`, or HTTP — the same SQL works everywhere.

## Query Execution

```
ndb CLI (NDB protocol)   ──┐
psql    (pgwire)         ──┼──► SQL Parser ──► EngineRules ──► PhysicalPlan ──► Data Plane
HTTP    (REST/JSON)      ──┘
```

Three doors, one room. Same parser, same optimizer, same execution engine.

## Standard SQL Support

NodeDB supports standard SQL with high PostgreSQL compatibility:

- SELECT with WHERE, GROUP BY, HAVING, ORDER BY, LIMIT, OFFSET
- JOINs (INNER, LEFT, RIGHT, CROSS, SEMI, ANTI)
- Window functions (ROW_NUMBER, RANK, LAG, LEAD, SUM OVER)
- CTEs (WITH, WITH RECURSIVE)
- Subqueries and derived tables
- Set operations (UNION, INTERSECT, EXCEPT)
- Transactions with savepoints (BEGIN, COMMIT, ROLLBACK, SAVEPOINT)
- Prepared statements

## Engine-Specific Extensions

Beyond standard SQL, NodeDB extends the language for each engine:

- **Vector** — `vector_distance()` with `<->` / `<=>` / `<#>` operators, ANN-aware `ORDER BY ... LIMIT k`
- **Graph** — `GRAPH TRAVERSE`, `GRAPH ALGO`, `MATCH` patterns
- **Full-Text** — `text_match()`, `bm25_score()`, `search_score()`
- **Spatial** — `ST_DWithin()`, `ST_Contains()`, `ST_Distance()`, etc.
- **Timeseries** — `time_bucket()`, `ts_rate()`, continuous aggregates
- **KV** — `KV_INCR()`, `KV_CAS()`, sorted indexes, rate gates
- **CRDT** — `crdt_state()`, `crdt_apply()`

## PostgreSQL Compatibility

These standard Postgres surface elements work natively over pgwire — no rewriting, no extension.

### Identifiers vs. Literals

`"quoted"` is a case-preserved identifier; `'quoted'` is a string literal. `SELECT "userId"` projects the column named `userId` exactly; `SELECT 'userId'` returns the constant string.

### Pattern Matching

```sql
SELECT * FROM users WHERE name LIKE 'Ali%';            -- case-sensitive
SELECT * FROM users WHERE email ILIKE '%@EXAMPLE.COM'; -- case-insensitive
```

### JSON Operators

```sql
doc -> 'addr'              -- field as JSON
doc ->> 'name'             -- field as text
doc #> '{addr,city}'       -- nested path as JSON
doc #>> '{addr,city}'      -- nested path as text
doc @> '{"role":"admin"}'  -- left contains right
doc <@ '{"role":"admin"}'  -- left contained in right
doc ? 'email'              -- top-level key exists
doc ?| ARRAY['phone','email']  -- any key exists
doc ?& ARRAY['name','email']   -- all keys exist
```

See [JSON Operators](json-operators) for full reference and examples.

### Full-Text Search (Postgres syntax)

```sql
SELECT * FROM articles
WHERE to_tsvector('english', body) @@ to_tsquery('english', 'transformer & attention');
```

NodeDB also exposes `text_match()`, `bm25_score()`, and `search_score()` for finer control — see [Full-Text Search](fulltext-search).

### Vector Distance Operators

| Operator | Function                       | Metric                |
| -------- | ------------------------------ | --------------------- |
| `<->`    | `vector_distance`              | L2 (default)          |
| `<=>`    | `vector_cosine_distance`       | Cosine                |
| `<#>`    | `vector_neg_inner_product`     | Negative inner product |

```sql
SELECT id FROM articles ORDER BY embedding <=> $query LIMIT 10;
```

## Isolation Level

Snapshot Isolation (SI). Reads see a consistent snapshot from `BEGIN` time. Write conflicts detected at `COMMIT`.


---
# DDL (docs/sql/ddl)

---
title: DDL
description: CREATE, ALTER, and DROP statements for collections, indexes, triggers, functions, and more.
---

# DDL (Data Definition Language)

## Database Management

```sql
-- Create database with optional quota settings
CREATE DATABASE emp_prod WITH (
    max_memory_bytes = 1073741824,
    max_storage_bytes = 10737418240,
    max_qps = 1000,
    max_connections = 100,
    cache_weight = 1,
    priority_class = 'standard',
    maintenance_cpu_pct = 25
);

-- Drop database
DROP DATABASE emp_prod;          -- errors if non-empty
DROP DATABASE emp_prod CASCADE;  -- cascades drop to all collections
DROP DATABASE emp_prod FORCE;    -- synonym for CASCADE: forces materialization of dependent clones, then drops

-- Rename database
ALTER DATABASE emp_prod RENAME TO emp_prod_old;

-- Set or update quota
ALTER DATABASE emp_prod SET QUOTA (
    max_memory_bytes = 2147483648,
    max_qps = 2000,
    cache_weight = 2
);

-- Enable DML audit
ALTER DATABASE emp_prod SET AUDIT_DML = 'writes';  -- 'none' | 'writes' | 'all'

-- Set idle session timeout
ALTER DATABASE emp_prod SET IDLE_TIMEOUT 1800;  -- seconds; 0 = disabled

-- Clone database
CLONE DATABASE staging FROM emp_prod AS OF SYSTEM TIME 1730000000000;
CLONE DATABASE preview FROM emp_prod AS OF LATEST;

-- Force materialization of a clone (blocks until complete)
ALTER DATABASE staging MATERIALIZE;

-- Mirror database
MIRROR DATABASE emp_eu FROM emp_prod MODE = async;
MIRROR DATABASE emp_eu FROM emp_prod MODE = sync;

-- Promote mirror to writable
ALTER DATABASE emp_eu PROMOTE;

-- Move tenant between databases (offline, requires all sessions drained)
MOVE TENANT acme FROM emp_us TO emp_eu;

-- Set per-tenant quota within a database
ALTER TENANT acme IN DATABASE emp_prod SET QUOTA (
    max_memory_bytes = 536870912,
    max_qps = 500,
    max_connections = 50,
    max_vector_dim = 1536,
    max_graph_depth = 5
);

-- Session management
USE DATABASE emp_prod;  -- or pgwire \c emp_prod
SHOW DATABASES;
SHOW DATABASE LINEAGE FOR staging;
SHOW DATABASE MIRROR STATUS FOR emp_eu;
SHOW DATABASE QUOTA FOR emp_prod;
SHOW DATABASE USAGE FOR emp_prod;
SHOW TENANT QUOTA FOR acme IN DATABASE emp_prod;
SHOW TENANT USAGE FOR acme IN DATABASE emp_prod;
SHOW SESSIONS;
SHOW SESSIONS IN DATABASE emp_prod;
KILL SESSION '550e8400-e29b-41d4-a716-446655440000';
SHOW AUDIT IN DATABASE emp_prod WHERE event_type = 'database_cloned';

-- Role and access
GRANT DATABASE_OWNER ON DATABASE emp_prod TO alice;
GRANT DATABASE_EDITOR ON DATABASE emp_prod TO bob;
GRANT DATABASE_READER ON DATABASE emp_prod TO charlie;
GRANT CREATE COLLECTION ON DATABASE emp_prod TO developers_role;
GRANT SELECT ON DATABASE emp_prod TO analysts_role;
GRANT SELECT ON COLLECTION users TO analysts_role;
GRANT SELECT ON TABLE users TO analysts_role;
ALTER USER alice SET DEFAULT DATABASE emp_prod;

-- API keys with database scope
CREATE API KEY prod_key FOR alice WITH DATABASES (emp_prod, emp_staging);
ALTER API KEY prod_key SET DATABASES (emp_prod);
SHOW API KEYS;

-- Service accounts with database scope
CREATE SERVICE ACCOUNT sa_prod FOR DATABASE emp_prod;
ALTER SERVICE ACCOUNT sa_prod SET DATABASES (emp_prod, emp_staging);
DROP SERVICE ACCOUNT sa_prod;

-- OIDC / SSO providers
CREATE OIDC PROVIDER corp_sso
  ISSUER 'https://idp.corp.com'
  JWKS_URI 'https://idp.corp.com/.well-known/jwks.json'
  AUDIENCE 'nodedb-prod'
  CLAIM MAPPING
    WHEN groups = 'eng' ADD DATABASES [1];

ALTER OIDC PROVIDER corp_sso SET CLAIM MAPPING
    WHEN groups = 'eng' ADD DATABASES [1, 2];

DROP OIDC PROVIDER IF EXISTS corp_sso;
SHOW OIDC PROVIDERS;
```

See the administration guides for detailed walkthroughs: [Database Management](../administration/databases), [Database Clone](../administration/database-clone), [Database Mirror](../administration/database-mirror), [Move Tenant](../administration/move-tenant), [Quotas](../administration/quotas), [Session Management](../administration/session-management), [OIDC / SSO](../administration/oidc-sso).

## Collections

```sql
-- Schemaless document (default)
CREATE COLLECTION users;

-- Strict document
CREATE COLLECTION orders (
    id TEXT PRIMARY KEY, customer_id TEXT, total FLOAT, status TEXT, created_at TIMESTAMP
) WITH (engine='document_strict');

-- Key-Value
CREATE COLLECTION sessions (key TEXT PRIMARY KEY) WITH (engine='kv');

-- Columnar (plain analytics)
CREATE COLLECTION logs (ts TIMESTAMP TIME_KEY, host VARCHAR, level VARCHAR, message VARCHAR)
    WITH (engine='columnar');

-- Timeseries (peer engine, append-only with retention + continuous aggregates)
CREATE COLLECTION metrics (ts TIMESTAMP TIME_KEY, host VARCHAR, cpu FLOAT)
    WITH (engine='timeseries', partition_by='1h', retention='90d');

-- Spatial (peer engine, R-tree + geohash + OGC predicates)
CREATE COLLECTION locations (geom GEOMETRY SPATIAL_INDEX, name VARCHAR)
    WITH (engine='spatial');

-- Convenience alias for timeseries
CREATE TIMESERIES metrics;

DROP COLLECTION users;          -- soft-delete: tombstoned, still restorable
UNDROP COLLECTION users;        -- restore while within the retention window
DROP COLLECTION users PURGE;    -- admin-only: immediate hard-delete, not restorable
SHOW COLLECTIONS;
DESCRIBE users;
```

`DROP COLLECTION` is two-phase. A plain drop marks the collection as tombstoned and enqueues background L2 cleanup after a configurable retention window; within that window, `UNDROP COLLECTION` restores the collection and its data. Admins can bypass the window with `DROP ... PURGE`.

```sql
-- Retention window knobs
ALTER SYSTEM SET collection_gc_retention = '7d';
ALTER TENANT acme SET collection_gc_retention = '24h';

-- Inspect the lifecycle
SELECT name, dropped_at, size_bytes_estimate FROM _system.dropped_collections;
SELECT * FROM _system.l2_cleanup_queue;
```

Dropping a collection also cascades downstream catalog state: dependent materialized views are unregistered, change-stream consumer groups are torn down, and the L2 cleanup worker reclaims segment files.

## Constraint Policy

NodeDB accepts a deliberately small subset of SQL constraints in `CREATE COLLECTION` / `CREATE TABLE`. Anything outside that subset is rejected with SQLSTATE `0A000` (`feature_not_supported`) — the parser tells you the canonical replacement instead of silently stripping the constraint.

**Accepted:** inline `<col> <TYPE> PRIMARY KEY` on a single column. The named column becomes the row identity; without one, NodeDB auto-injects `_rowid INT64 PRIMARY KEY`.

**Rejected at DDL time:**

| Form                                       | Replacement                                                |
| ------------------------------------------ | ---------------------------------------------------------- |
| Table-level `PRIMARY KEY (col)`            | Inline form: `<col> <TYPE> PRIMARY KEY`                    |
| Named `CONSTRAINT pk_x PRIMARY KEY (col)`  | Inline form                                                |
| Inline or table-level `UNIQUE`             | `CREATE UNIQUE INDEX ON tbl(col)`                          |
| Inline or table-level `CHECK (...)`        | Application-side validation (typeguards on schemaless docs) |
| `FOREIGN KEY (...) REFERENCES ...`         | Application-side enforcement                               |
| Column-level `REFERENCES other(col)`       | Application-side enforcement                               |

These are launch-frozen choices. UNIQUE/CHECK/FK/REFERENCES would each pull in a fixed semantic that's expensive to walk back, and the canonical replacements (unique indexes, app-side validation) cover the same use cases without locking the engine into a particular cross-row enforcement strategy.

## Schema Evolution

```sql
ALTER TABLE orders ADD COLUMN priority INT;
ALTER COLLECTION orders ADD COLUMN region STRING DEFAULT 'us-east';
ALTER COLLECTION orders DROP COLUMN region;
```

## Storage Conversion

```sql
CONVERT COLLECTION cache TO kv;
CONVERT COLLECTION users TO document_strict;
CONVERT COLLECTION logs TO document_schemaless;
```

## Indexes

```sql
CREATE INDEX idx_email ON users(email);
CREATE INDEX IF NOT EXISTS idx_email ON users(email);
CREATE UNIQUE INDEX ON users(username);
CREATE VECTOR INDEX ON articles METRIC cosine DIM 384 M 16 EF_CONSTRUCTION 200;
CREATE SEARCH INDEX ON articles FIELDS title, body ANALYZER 'english' FUZZY true;
CREATE SPATIAL INDEX ON locations(geom);
CREATE SORTED INDEX lb ON scores (score DESC) KEY player_id;
DROP INDEX idx_email;
DROP INDEX IF EXISTS idx_email;

-- Rebuild all indexes on a collection (non-blocking concurrent path recommended)
REINDEX CONCURRENTLY articles;

-- Rebuild a specific named index only
REINDEX INDEX idx_email CONCURRENTLY users;

-- Blocking rebuild (holds write lock for the duration — use only in maintenance windows)
REINDEX articles;
```

`CREATE INDEX ... IF NOT EXISTS` and `DROP INDEX ... IF EXISTS` make DDL idempotent — re-running a script does not error if the index already exists (or does not exist).

`REINDEX CONCURRENTLY` builds the new index in a shadow location while reads and writes continue against the live index. Cutover is a single atomic swap. Supported for HNSW vector indexes, FTS LSM indexes, and Graph CSR indexes. Secondary B-tree indexes use the blocking path regardless of `CONCURRENTLY` (rebuild is fast for B-tree).

## Triggers

```sql
-- ASYNC (default): Event Plane, zero write-latency impact
CREATE TRIGGER notify AFTER INSERT ON orders FOR EACH ROW $$
BEGIN INSERT INTO notifications (user_id, message) VALUES (NEW.customer_id, 'Order placed'); END; $$;

-- SYNC: same transaction, ACID
CREATE TRIGGER enforce AFTER UPDATE ON accounts FOR EACH ROW WITH (EXECUTION = SYNC) $$
BEGIN IF NEW.balance < 0 THEN RAISE EXCEPTION 'Negative balance'; END IF; END; $$;

DROP TRIGGER notify ON orders;
SHOW TRIGGERS;
```

The op tag (INSERT vs UPDATE) is derived from storage prior-bytes, not the surface SQL verb. An `UPSERT` or `INSERT ... ON CONFLICT (pk) DO UPDATE` that overwrites an existing row fires `AFTER UPDATE`; the same statement against a non-existent key fires `AFTER INSERT`. `ON CONFLICT DO NOTHING` emits no event on conflict.

## Functions & Procedures

```sql
-- SQL expression (inlined, zero overhead)
CREATE FUNCTION full_name(first VARCHAR, last VARCHAR) RETURNS VARCHAR
LANGUAGE SQL IMMUTABLE AS $$ first || ' ' || last $$;

-- Procedural
CREATE PROCEDURE transfer_funds(from_id UUID, to_id UUID, amount DECIMAL) BEGIN
    UPDATE accounts SET balance = balance - amount WHERE id = from_id;
    UPDATE accounts SET balance = balance + amount WHERE id = to_id;
END;

CALL transfer_funds('acc_a', 'acc_b', 50.00);
```

## Change Streams & Topics

```sql
CREATE CHANGE STREAM order_events ON orders WITH (URL = 'https://hooks.example.com/orders');
CREATE CONSUMER GROUP processors ON order_events;
CREATE TOPIC alerts WITH (RETENTION = '1 hour');
PUBLISH TO alerts 'message';
```

## Materialized Views

```sql
CREATE MATERIALIZED VIEW order_stats AS SELECT status, COUNT(*), SUM(total) FROM orders GROUP BY status;
REFRESH MATERIALIZED VIEW order_stats;

CREATE CONTINUOUS AGGREGATE cpu_hourly ON cpu_metrics AS
    SELECT time_bucket('1 hour', ts) AS hour, host, AVG(cpu) FROM cpu_metrics GROUP BY hour, host
WITH (refresh_interval = '1m');
```

## Cron Scheduler

```sql
CREATE SCHEDULE nightly_cleanup CRON '0 2 * * *' AS BEGIN
    DELETE FROM sessions WHERE expires_at < now();
END;
```

## Backup & Restore

```sql
BACKUP TENANT acme TO '/backups/acme.bak';
RESTORE TENANT acme FROM '/backups/acme.bak';
RESTORE TENANT acme FROM '/backups/acme.bak' DRY RUN;
PURGE TENANT acme CONFIRM;
```


---
# SELECT (docs/sql/select)

---
title: SELECT
description: Query data with filtering, aggregation, window functions, CTEs, and subqueries.
---

# SELECT

```sql
SELECT [DISTINCT] <columns>
FROM <collection>
[WHERE <predicate>]
[GROUP BY <fields>] [HAVING <predicate>]
[ORDER BY <field> [ASC|DESC], ...]
[LIMIT <n>] [OFFSET <m>]
```

## Filtering

```sql
SELECT * FROM users WHERE age > 30 AND status = 'active';
SELECT * FROM users WHERE name LIKE 'Ali%';
SELECT * FROM users WHERE email ILIKE '%@EXAMPLE.COM';
SELECT * FROM orders WHERE total BETWEEN 10 AND 100;
SELECT * FROM users WHERE role IN ('admin', 'editor');
SELECT * FROM users WHERE deleted_at IS NULL;
```

## Aggregates

```sql
SELECT status, COUNT(*), AVG(age), MIN(salary), MAX(salary)
FROM employees WHERE department = 'sales'
GROUP BY status HAVING COUNT(*) > 5;

SELECT COUNT(DISTINCT user_id) FROM orders;
```

## Window Functions

```sql
SELECT id,
    ROW_NUMBER() OVER (ORDER BY created_at) AS rn,
    RANK() OVER (PARTITION BY dept ORDER BY salary DESC) AS rank,
    LAG(salary, 1) OVER (ORDER BY created_at) AS prev_salary,
    SUM(amount) OVER (ORDER BY created_at ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS running_total
FROM employees;
```

`ROWS`, `RANGE`, and `GROUPS` frame modes are all supported. See [Window Frames](window-frames) for the full frame syntax and all boundary combinations.

## CTEs & Subqueries

```sql
WITH active AS (SELECT id FROM users WHERE status = 'active')
SELECT * FROM orders WHERE user_id IN (SELECT id FROM active);

-- Recursive CTE
WITH RECURSIVE subordinates AS (
    SELECT id, name, manager_id FROM employees WHERE id = 'emp_root'
    UNION ALL
    SELECT e.id, e.name, e.manager_id FROM employees e JOIN subordinates s ON e.manager_id = s.id
)
SELECT * FROM subordinates;
```

See [Recursive CTEs](recursive-cte) for the full reference including value-generating forms, cycle prevention, and validation rules.

## Set Operations

```sql
SELECT id FROM collection_a UNION ALL SELECT id FROM collection_b;
SELECT id FROM collection_a INTERSECT SELECT id FROM collection_b;
SELECT id FROM collection_a EXCEPT SELECT id FROM collection_b;
```

## Computed Columns

```sql
SELECT price * qty AS total,
       UPPER(name) AS name_upper,
       CASE WHEN price > 100 THEN 'expensive' ELSE 'cheap' END AS tier
FROM orders;
```


---
# INSERT & UPSERT (docs/sql/insert-upsert)

---
title: INSERT & UPSERT
description: Insert documents, upsert with merge semantics, and bulk import.
---

# INSERT & UPSERT

## INSERT

```sql
-- Single row
INSERT INTO users (id, name, email) VALUES ('u1', 'Alice', 'alice@example.com');

-- Multiple rows
INSERT INTO users (id, name) VALUES ('u1', 'Alice'), ('u2', 'Bob');

-- Object literal syntax (schemaless collections)
INSERT INTO users { name: 'Alice', email: 'alice@example.com', age: 30 };

-- INSERT ... SELECT
INSERT INTO archive SELECT * FROM orders WHERE created_at < '2025-01-01';
```

Plain `INSERT` is strict: a duplicate primary key raises `unique_violation` (SQLSTATE `23505`). Use `ON CONFLICT` or `UPSERT` for "insert if absent" or "insert or overwrite" semantics.

## INSERT ... ON CONFLICT

```sql
-- Skip rows that would collide with an existing PK (no error)
INSERT INTO users (id, name) VALUES ('u1', 'Alice')
ON CONFLICT DO NOTHING;

-- Overwrite selected fields on conflict. EXCLUDED refers to the incoming row;
-- bare column names refer to the existing row.
INSERT INTO users (id, name, login_count) VALUES ('u1', 'Alice', 1)
ON CONFLICT (id) DO UPDATE SET
  name        = EXCLUDED.name,
  login_count = users.login_count + EXCLUDED.login_count;
```

`ON CONFLICT DO UPDATE` fires `AFTER UPDATE` triggers (not `AFTER INSERT`) when a row is overwritten — the trigger op tag is derived from storage prior-bytes, not the surface SQL verb.

## UPSERT

Insert or overwrite if the primary key already exists:

```sql
UPSERT INTO users (id, name, role) VALUES ('u1', 'Alice', 'admin');

-- Object literal syntax
UPSERT INTO users { id: 'u1', name: 'Alice', role: 'admin' };
```

`UPSERT` is equivalent to `INSERT ... ON CONFLICT (<pk>) DO UPDATE SET <all-columns> = EXCLUDED.<col>`. It fires `AFTER UPDATE` on overwrite and `AFTER INSERT` on first write.

## Bulk Import

```sql
-- Auto-detected format (NDJSON, JSON array, or CSV)
COPY users FROM '/path/to/users.ndjson';
COPY users FROM '/path/to/users.csv' WITH (FORMAT csv);
```

See [COPY](copy) for the full syntax, all format options, and `COPY TO` for export.


---
# UPDATE & DELETE (docs/sql/update-delete)

---
title: UPDATE & DELETE
description: Point and bulk mutations, transactions, and limitations.
---

# UPDATE & DELETE

## UPDATE

```sql
UPDATE users SET role = 'admin' WHERE id = 'u1';
UPDATE users SET status = 'inactive' WHERE last_login < '2025-01-01';
```

## DELETE

```sql
DELETE FROM users WHERE id = 'u1';
DELETE FROM orders WHERE status = 'cancelled';
TRUNCATE users;
```

## Transactions

```sql
BEGIN;
INSERT INTO orders (id, total) VALUES ('o1', 99.99);
UPDATE inventory SET stock = stock - 1 WHERE id = 'item1';
COMMIT;

-- Rollback
BEGIN;
DELETE FROM users WHERE id = 'u1';
ROLLBACK;

-- Savepoints
BEGIN;
SAVEPOINT sp1;
INSERT INTO users (id, name) VALUES ('u1', 'Alice');
ROLLBACK TO sp1;
COMMIT;
```

## Atomic Transfers

```sql
-- Fungible (currency, resources)
SELECT TRANSFER('wallets', 'player-A', 'player-B', 'gold', 500);

-- Non-fungible (unique items)
SELECT TRANSFER_ITEM('inventory', 'inventory', 'sword-of-doom', 'player-A', 'player-B');
```

## RETURNING

`UPDATE` and `DELETE` support a `RETURNING` clause to get the affected rows back in the same round-trip.

```sql
-- UPDATE: returns the post-update image
UPDATE users SET role = 'admin' WHERE id = 'u1' RETURNING *;
UPDATE users SET status = 'inactive' WHERE last_login < '2025-01-01' RETURNING id, status;

-- DELETE: returns the pre-delete image
DELETE FROM orders WHERE id = 'o1' RETURNING *;
DELETE FROM orders WHERE status = 'cancelled' RETURNING id, total;
```

`RETURNING *` projects all columns. `RETURNING col1, col2` projects named columns. Arithmetic expressions in `RETURNING` are not supported — bare column names and `*` only.

Works in both simple-query and extended-query (prepared statement) protocols.

## UPDATE … FROM

`UPDATE … FROM` updates rows in the target collection based on matching rows in one or more source collections. This is the standard alternative to `UPDATE … JOIN` syntax.

```sql
-- Apply a price increase from a pricing table
UPDATE products p
SET price = pricing.new_price
FROM pricing
WHERE p.id = pricing.product_id;

-- Update orders using data from a users table
UPDATE orders o
SET region = u.region
FROM users u
WHERE o.user_id = u.id;

-- Use a CTE as the source
WITH discounts AS (
    SELECT user_id, 0.10 AS pct FROM promotions WHERE active = true
)
UPDATE orders o
SET total = o.total * (1 - d.pct)
FROM discounts d
WHERE o.user_id = d.user_id;
```

Constraints:

- Exactly one source table (or CTE) is supported.
- The `WHERE` clause must contain an equi-join predicate between target and source (`target.col = source.col`).
- Inline subqueries in `FROM` are not supported; use a CTE.
- `FROM` source collections cannot themselves contain `JOIN`s.

## MERGE

`MERGE` is a single-statement upsert/delete driven by a source query. It evaluates `WHEN MATCHED` and `WHEN NOT MATCHED` clauses to decide what to do with each row from the source.

```sql
MERGE INTO orders AS target
USING staging_orders AS source ON target.id = source.id
WHEN MATCHED THEN
    UPDATE SET total = source.total, status = source.status
WHEN NOT MATCHED THEN
    INSERT (id, user_id, total, status)
    VALUES (source.id, source.user_id, source.total, source.status);
```

### WHEN NOT MATCHED BY SOURCE

Optionally delete (or update) target rows that have no corresponding source row:

```sql
MERGE INTO inventory AS target
USING new_stock AS source ON target.sku = source.sku
WHEN MATCHED THEN
    UPDATE SET qty = source.qty
WHEN NOT MATCHED BY TARGET THEN
    INSERT (sku, qty) VALUES (source.sku, source.qty)
WHEN NOT MATCHED BY SOURCE THEN
    DELETE;
```

### MERGE clause kinds

| Clause                        | Fires when                                              | Allowed actions          |
| ----------------------------- | ------------------------------------------------------- | ------------------------ |
| `WHEN MATCHED`                | Source row matches a target row                         | `UPDATE SET …`, `DELETE` |
| `WHEN NOT MATCHED [BY TARGET]`| Source row has no corresponding target row              | `INSERT`                 |
| `WHEN NOT MATCHED BY SOURCE`  | Target row has no corresponding source row              | `UPDATE SET …`, `DELETE` |

Each clause may include an optional `AND <predicate>` condition:

```sql
MERGE INTO accounts AS t
USING updates AS s ON t.id = s.id
WHEN MATCHED AND s.balance > 0 THEN
    UPDATE SET balance = s.balance
WHEN MATCHED AND s.balance <= 0 THEN
    DELETE
WHEN NOT MATCHED THEN
    INSERT (id, balance) VALUES (s.id, s.balance);
```

## Limitations

- `UPDATE/DELETE … JOIN` syntax is not supported. Use `UPDATE … FROM` or a subquery: `DELETE FROM orders WHERE user_id IN (SELECT id FROM users WHERE ...)`
- `UPDATE/DELETE` on timeseries collections is not supported (append-only). Use retention policies.


---
# JOINs & Aggregations (docs/sql/joins-aggregations)

---
title: JOINs & Aggregations
description: Inner, outer, semi, and anti joins. Aggregate functions and GROUP BY.
---

# JOINs & Aggregations

## JOINs

```sql
-- Inner join
SELECT u.name, o.total FROM users u JOIN orders o ON u.id = o.user_id;

-- Left join
SELECT u.name, o.total FROM users u LEFT JOIN orders o ON u.id = o.user_id;

-- Cross join
SELECT * FROM sizes CROSS JOIN colors;

-- Semi join (EXISTS)
SELECT * FROM users u WHERE EXISTS (SELECT 1 FROM orders o WHERE o.user_id = u.id);

-- Anti join (NOT EXISTS)
SELECT * FROM users u WHERE NOT EXISTS (SELECT 1 FROM orders o WHERE o.user_id = u.id);
```

JOINs work across collections of different storage types — you can join a KV collection with a strict document collection.

## LATERAL Joins

`LATERAL` allows the right-hand subquery to reference columns from the outer table. See [Lateral Subqueries](lateral-subqueries) for the full reference. Quick example:

```sql
-- Top 3 most recent orders for each user
SELECT u.id, u.name, o.id AS order_id, o.created_at
FROM users u
JOIN LATERAL (
    SELECT id, created_at FROM orders WHERE user_id = u.id
    ORDER BY created_at DESC LIMIT 3
) AS o ON true;

-- LEFT JOIN LATERAL — preserve users with no orders
SELECT u.id, o.last_order_at
FROM users u
LEFT JOIN LATERAL (
    SELECT MAX(created_at) AS last_order_at FROM orders WHERE user_id = u.id
) AS o ON true;
```

## Aggregate Functions

| Function                | Description             |
| ----------------------- | ----------------------- |
| `COUNT(*)`              | Row count               |
| `COUNT(DISTINCT col)`   | Distinct count          |
| `SUM(col)`              | Sum                     |
| `AVG(col)`              | Average                 |
| `MIN(col)` / `MAX(col)` | Minimum / Maximum       |

## Approximate Aggregates

Mergeable across shards, usable in continuous aggregation:

| Function                        | Description        |
| ------------------------------- | ------------------ |
| `approx_count_distinct(col)`    | HyperLogLog        |
| `approx_percentile(col, p)`     | t-digest           |
| `approx_topk(col, k)`          | SpaceSaving        |

## Weighted Random Selection

```sql
SELECT * FROM WEIGHTED_PICK('loot_table', weight => 'drop_rate', count => 1);
SELECT * FROM WEIGHTED_PICK('gacha_pool', weight => 'probability', count => 10, SEED => 'player:pull');
```


---
# Functions Reference (docs/sql/functions)

---
title: Functions Reference
description: Built-in scalar, aggregate, and engine-specific SQL functions.
---

# Functions Reference

## String Functions

`LENGTH`, `SUBSTR`, `UPPER`, `LOWER`, `TRIM`, `LTRIM`, `RTRIM`, `CONCAT` (or `||`), `REPLACE`, `SPLIT`

## Numeric Functions

`ABS`, `CEIL`, `FLOOR`, `ROUND`, `SQRT`, `POWER`, `MOD` (or `%`), `GREATEST`, `LEAST`

## Date/Time Functions

| Function                        | Description                            |
| ------------------------------- | -------------------------------------- |
| `NOW()`                         | Current UTC timestamp                  |
| `CURRENT_TIMESTAMP()`           | Alias for NOW()                        |
| `EXTRACT(field FROM ts)`        | Extract year, month, day, hour, etc.   |
| `DATE_TRUNC(unit, ts)`          | Truncate to unit boundary              |
| `DATE_FORMAT(ts, fmt)`          | Format timestamp as string             |
| `time_bucket(interval, ts)`     | Truncate to interval boundary          |

`time_bucket` accepts the interval as a string in any of these forms: short (`'1s'`, `'5m'`, `'15m'`, `'1h'`, `'6h'`, `'1d'`, `'1w'`), long / Postgres-style (`'30 seconds'`, `'5 minutes'`, `'1 hour'`, `'2 hours 30 minutes'`, optionally with an `INTERVAL` prefix), ISO 8601 (`'PT5M'`, `'P1D'`), or a bare integer (milliseconds).

## Type Functions

`CAST(expr AS type)`, `TRY_CAST(expr AS type)`, `expr::type`

## ID Generation

`gen_uuid_v7()` — time-ordered UUID v7

## KV Atomic Functions

| Function                               | Description                    |
| -------------------------------------- | ------------------------------ |
| `KV_INCR(coll, key, delta [, TTL])` | Atomic increment               |
| `KV_DECR(coll, key, delta)`           | Atomic decrement               |
| `KV_INCR_FLOAT(coll, key, delta)`     | Float increment                |
| `KV_CAS(coll, key, expected, new)`    | Compare-and-swap               |
| `KV_GETSET(coll, key, new)`           | Atomic get-and-set             |

## Leaderboard Functions

| Function                    | Description              |
| --------------------------- | ------------------------ |
| `RANK(index, key)`          | Player's rank            |
| `TOPK(index, k)`           | Top-k entries (TVF)      |
| `RANGE(index, min, max)`    | Score range query (TVF)  |
| `SORTED_COUNT(index)`       | Total entries            |

## Rate Limiting Functions

| Function                                       | Description                |
| ---------------------------------------------- | -------------------------- |
| `RATE_CHECK(gate, key, max_count, window_secs)` | Check and consume          |
| `RATE_REMAINING(gate, key, max_count, window_secs)` | Check remaining budget |
| `RATE_RESET(gate, key)`                         | Reset a cooldown           |

## Transfer Functions

| Function                                               | Description          |
| ------------------------------------------------------ | -------------------- |
| `TRANSFER(coll, source, dest, field, amount)`          | Fungible transfer    |
| `TRANSFER_ITEM(src_coll, dst_coll, item, src, dst)`   | Non-fungible transfer|

## Timeseries Functions

`ts_rate`, `ts_delta`, `ts_moving_avg`, `ts_ema`, `ts_interpolate`, `ts_percentile`, `ts_correlate`, `ts_lag`, `ts_lead`, `ts_rank`, `ts_stddev`, `ts_derivative`, `ts_zscore`, `ts_bollinger_upper/lower/mid/width`, `ts_moving_percentile`

## Vector Functions

`vector_distance(col, vec)`, `rrf_score(vector_distance, bm25_score [, k1, k2])`

## Full-Text Functions

`text_match(col, query)`, `bm25_score(col, query)`, `search_score(col, query)`

## Document Navigation

`doc_get(payload, path)`, `doc_exists(payload, path)`, `doc_array_contains(payload, path, value)`

## CRDT Functions

`crdt_state(coll, id)`, `crdt_apply(coll, id, delta)`


---
# Vector Search Queries (docs/sql/vector-search)

---
title: Vector Search Queries
description: Nearest neighbor search, filtered vector search, and multi-vector queries.
---

# Vector Search Queries

## Nearest Neighbor

```sql
SELECT id, vector_distance(embedding, ARRAY[0.1, 0.3, -0.2, ...]) AS dist
FROM articles
ORDER BY dist LIMIT 10;
```

Returns the 10 nearest neighbors by the metric configured on the index (l2, cosine, inner_product, manhattan, chebyshev, hamming, jaccard, or pearson). The planner detects `ORDER BY vector_distance(...) LIMIT k` and rewrites it into an ANN top-k plan. The `SEARCH()` function is an alias for full-text search; vector search uses `ORDER BY vector_distance(...)` instead.

## Filtered Vector Search

```sql
SELECT title, vector_distance(embedding, ARRAY[0.1, 0.3, ...]) AS score
FROM articles
WHERE category = 'machine-learning'
ORDER BY score LIMIT 10;
```

The engine builds a Roaring Bitmap of matching IDs and selects the optimal strategy: pre-filter (selective filters), post-filter (broad filters), or brute-force (very selective).

## Distance Function

```sql
SELECT id, vector_distance(embedding, $query_vec) AS dist
FROM articles
ORDER BY dist LIMIT 10;
```

### Operator Forms

| Operator | Function                       | Metric                  |
| -------- | ------------------------------ | ----------------------- |
| `<->`    | `vector_distance`              | L2                      |
| `<=>`    | `vector_cosine_distance`       | Cosine                  |
| `<#>`    | `vector_neg_inner_product`     | Negative inner product  |

```sql
SELECT id FROM articles ORDER BY embedding <=> $query_vec LIMIT 10;
```

### ANN Tuning (Named Arguments)

`vector_distance` and its cosine / inner-product peers accept named tuning arguments via `=>`. JSON-string options are not accepted — every option is a typed, closed-set named argument:

```sql
SELECT id, vector_distance(
    embedding, $query_vec,
    quantization => 'rabitq',
    oversample   => 4,
    ef_search    => 128,
    target_recall => 0.95
) AS dist
FROM articles
ORDER BY dist LIMIT 10;
```

| Argument            | Type    | Notes                                                                                 |
| ------------------- | ------- | ------------------------------------------------------------------------------------- |
| `quantization`      | string  | `none`, `sq8`, `pq`, `binary`, `ternary`, `rabitq`, `bbq`, `opq`                      |
| `oversample`        | u8      | Candidates fetched before re-ranking. Default `3`. Final rerank set is `oversample × ef_search`. |
| `query_dim`         | u32     | Coarse-to-fine on first-N dims of Matryoshka embeddings. `None` = full dimensionality. |
| `meta_token_budget` | u8      | MetaEmbed multivec scoring budget for MaxSim / PLAID                                  |
| `ef_search`         | u32     | HNSW / Vamana search-time beam width. Default `64`.                                   |
| `target_recall`     | f32     | Adaptive recall target. The cost-model planner picks oversample / ef_search to hit this. |

Unknown names, duplicate keys, positional 3rd args, or wrong operators (`=` instead of `=>`) all return typed errors that list the canonical names.

### Quantization Choice

| Codec     | Bits/dim | Recall (typ.) | When to pick it                                                |
| --------- | -------- | ------------- | -------------------------------------------------------------- |
| `none`    | 32       | 100%          | Small index (< 1M vectors), latency not critical               |
| `sq8`     | 8        | ~99%          | Balanced default for medium index sizes                        |
| `pq`      | ~2       | ~95%          | Large memory-bound indexes; classic Product Quantization       |
| `opq`     | ~2       | ~96%          | PQ + learned random rotation; minor accuracy bump over `pq`    |
| `rabitq`  | 1        | ~97%          | Frontier 1-bit with `O(1/√D)` error bound (SIGMOD 2024)        |
| `bbq`     | 1        | ~98%          | Centroid-asymmetric 1-bit + 14-byte corrective; oversample-rerank |
| `binary`  | 1        | ~85%          | Hamming-only, no rerank — for ultra-cold tiers                 |
| `ternary` | 1.58     | ~96%          | BitNet-style `{-1, 0, +1}` — cold/hot pack for AVX-512         |

The cost-model planner (`target_recall`) will pick `oversample` and `ef_search` automatically once you set the recall target. Manually set those two only when you need a hard latency ceiling.

## Index Implementations

You don't pick the underlying index directly — it's chosen by the planner from collection metadata + workload signals:

- **HNSW** — in-memory hierarchical graph, the default for moderate-size indexes
- **Vamana / DiskANN** — flat-beam SSD-resident graph for billion-scale on a single node (`Tier 2` of the vector frontier)
- **NaviX adaptive-local filtered traversal** (VLDB 2025) — switches per-hop between standard / directed / blind heuristics based on local selectivity. Replaces classic ACORN-1 filtered ANN.
- **SIEVE workload-driven subindex collections** — the planner builds specialized HNSW subindices for stable predicates (e.g. `tenant_id`) and routes filtered queries to them.
- **MetaEmbed multi-vector + ColBERT MaxSim + PLAID** (ICLR 2026) — learnable Meta Tokens replace per-token explosion; budgeted MaxSim at query time via `meta_token_budget`.
- **Matryoshka adaptive-dim querying** — coarse-to-fine ranking on the first-N dimensions of MRL embeddings via `query_dim`.
- **SPFresh streaming updates** (SOSP 2023) — LIRE topology-aware local rebalancing; no full-rebuild stalls when vectors are added/removed.

## Vector-Primary Collections

By default, vectors are an *index* attached to a column on a normal collection — the document/strict store is the source of truth, and the vector index is a side path. For pure-vector workloads (RAG corpora, recommendation memory, embedding stores) you can flip a collection into **vector-primary** mode, where the vector index becomes the primary access path and the document store is a metadata sidecar:

```sql
CREATE COLLECTION corpus (
    id UUID DEFAULT gen_uuid_v7(),
    embedding FLOAT[384],
    title TEXT,
    tenant_id UUID,
    created_at TIMESTAMP DEFAULT now()
) WITH (
    primary='vector',
    vector_field='embedding',
    dim=384,
    metric='cosine',
    storage_dtype='bf16',
    quantization='rabitq',
    m=32,
    ef_construction=200,
    payload_indexes=['tenant_id', 'created_at']
);
```

| Option            | Notes                                                                |
| ----------------- | -------------------------------------------------------------------- |
| `primary`         | `'document_schemaless'` (default), `'document_strict'`, `'kv'`, `'columnar'`, `'timeseries'`, `'spatial'`, `'vector'` |
| `vector_field`    | Required when `primary='vector'`. Column name of the embedding.       |
| `dim`             | Required. Embedding dimensionality.                                   |
| `metric`          | `'cosine'`, `'l2'`, `'inner_product'`, etc.                          |
| `storage_dtype`   | Raw HNSW storage precision: `'F32'` (4 B/dim, default), `'F16'` (2 B/dim), `'BF16'` (2 B/dim bfloat16, better dynamic range, preferred for embeddings). |
| `quantization`    | Storage-level codec (same vocabulary as the query-time arg above).   |
| `m`, `ef_construction` | HNSW build-time parameters.                                     |
| `payload_indexes` | Per-field equality / range / boolean indexes over the metadata sidecar for filtered ANN. Replaces Pinecone metadata filters. |

In vector-primary mode the planner treats the vector index as the source of truth for IDs; metadata fetches only happen for hit IDs. Cross-engine queries, CRDT sync, and SQL semantics all keep working — `primary='vector'` is purely an *access-path* hint, not a different engine.

Default `primary='document'` is unchanged: the existing `CREATE VECTOR INDEX ON ...` syntax continues to work for vector-as-side-index workloads.

## Hybrid Vector + Text (RRF)

```sql
SELECT title, rrf_score(
    vector_distance(embedding, $query_vec),
    bm25_score(body, 'transformer attention')
) AS score
FROM articles
LIMIT 10;
```

Reciprocal Rank Fusion merges BM25 text results with vector similarity in a single pass.

## Multiple vector columns per collection

A collection can carry several vector indexes, one per embedding column — name
the column in parentheses after the collection. Each index gets its own metric
and parameters; queries pick the column they search:

```sql
CREATE VECTOR INDEX idx_text  ON products (text_embedding)  METRIC cosine DIM 384;
CREATE VECTOR INDEX idx_image  ON products (image_embedding) METRIC cosine DIM 512;

-- Search the text-embedding index:
SELECT id FROM products ORDER BY text_embedding  <=> $text_vec  LIMIT 10;
-- ...or the image-embedding index:
SELECT id FROM products ORDER BY image_embedding <-> $image_vec LIMIT 10;
```

Omitting `(<column>)` targets the collection's default (unnamed) vector field —
fine when there's only one embedding column (as in the `CREATE VECTOR INDEX
idx_embed ON articles ...` examples above).


---
# Full-Text Search Queries (docs/sql/fulltext-search)

---
title: Full-Text Search Queries
description: BM25 text matching, fuzzy search, highlighting, and synonyms in SQL.
---

# Full-Text Search Queries

## Basic Search

```sql
SELECT title, bm25_score(body, 'distributed database rust') AS score
FROM articles
WHERE text_match(body, 'distributed database rust')
ORDER BY score DESC LIMIT 20;
```

## Fuzzy Search

```sql
SELECT title FROM articles WHERE text_match(title, 'databse', { fuzzy: true, distance: 2 });
```

## CJK Search

CJK text is automatically tokenized via character bigrams:

```sql
SELECT title FROM articles WHERE text_match(body, '全文検索');
```

## NOT Operator

Exclude documents that match specific terms using the `NOT` keyword or the `-` prefix (Lucene-style, no space before the term).

Both forms require at least one positive term — a query consisting only of negations is rejected.

```sql
-- Exclude documents that mention 'python'
SELECT title FROM articles
WHERE text_match(body, 'rust NOT python');

-- Multiple exclusions
SELECT title FROM articles
WHERE text_match(body, 'database NOT mysql NOT oracle');

-- Lucene-style dash prefix (equivalent to NOT)
SELECT title FROM articles
WHERE text_match(body, 'database -mysql -oracle');
```

Parenthesised NOT groups are not supported. Use flat negations:

```sql
-- Not supported:  rust NOT (python OR ruby)
-- Supported:      rust NOT python NOT ruby
SELECT title FROM articles
WHERE text_match(body, 'rust NOT python NOT ruby');
```

## Synonym Groups

A synonym group makes the FTS engine treat a set of terms as interchangeable during indexing and querying. When a document contains any term in the group it also matches queries for any other term in the group.

```sql
-- Define synonyms
CREATE SYNONYM GROUP db_terms AS ('database', 'db', 'datastore');
CREATE SYNONYM GROUP ml_terms AS ('machine learning', 'ml', 'artificial intelligence', 'ai');

-- Now a query for 'db' also matches documents that contain 'database' or 'datastore'
SELECT title FROM articles WHERE text_match(body, 'db performance');
```

### Managing synonym groups

```sql
-- List all groups
SHOW SYNONYM GROUPS;

-- Drop a group
DROP SYNONYM GROUP db_terms;
DROP SYNONYM GROUP IF EXISTS ml_terms;
```

Synonym group names are case-insensitive. Terms within a group are stored lowercased. Duplicate terms within the same group are rejected at creation time.

## Hybrid Search (BM25 + Vector)

```sql
SELECT title, rrf_score(
    vector_distance(embedding, $query_vec),
    bm25_score(body, 'distributed systems')
) AS score
FROM articles
LIMIT 10;
```


---
# Graph Queries (docs/sql/graph-queries)

---
title: Graph Queries
description: GRAPH commands, MATCH pattern matching, and algorithm execution in SQL.
---

# Graph Queries

## Edge Operations

```sql
GRAPH INSERT EDGE IN 'edges' FROM 'alice' TO 'bob' TYPE 'knows' PROPERTIES { since: 2020 };
GRAPH DELETE EDGE IN 'edges' FROM 'alice' TO 'bob' TYPE 'knows';
```

The `IN '<collection>'` clause is **required** — edges are overlays on a named document collection. Statements without `IN` fail to parse.

## Traversal

```sql
GRAPH TRAVERSE FROM 'alice' DEPTH 3;
GRAPH TRAVERSE FROM 'alice' DEPTH 2 LABEL 'follows' DIRECTION out;
GRAPH NEIGHBORS OF 'bob' LABEL 'follows' DIRECTION both;
GRAPH PATH FROM 'alice' TO 'charlie' MAX_DEPTH 5 LABEL 'knows';
```

## MATCH Pattern Queries

Cypher-subset pattern matching:

```sql
MATCH (a:Person)-[:knows]->(b:Person)-[:knows]->(c:Person)
WHERE a.name = 'Alice'
RETURN b.name, c.name;

-- Variable-length paths
MATCH (u:User)-[:follows*2..3]->(rec:User) WHERE u.id = 'you' RETURN DISTINCT rec.id LIMIT 10;

-- Anti-join
MATCH (a:User)-[:follows]->(b:User) WHERE NOT EXISTS { MATCH (b)-[:blocked_by]->(a) } RETURN a.id, b.id;

-- OPTIONAL MATCH
MATCH (a:Person)-[:knows]->(b:Person) OPTIONAL MATCH (b)-[:works_at]->(c:Company) RETURN a.name, b.name, c.name;
```

## Algorithms

```sql
GRAPH ALGO PAGERANK ON 'social' DAMPING 0.85 ITERATIONS 20 TOLERANCE 1e-7;
GRAPH ALGO PAGERANK ON 'social' DAMPING 0.85 PERSONALIZATION {"alice": 1.0, "bob": 0.5};
GRAPH ALGO WCC ON 'knowledge_graph';
GRAPH ALGO SSSP ON 'routes' FROM 'city:chicago';
GRAPH ALGO COMMUNITY ON 'products' ITERATIONS 10 RESOLUTION 1.0;
GRAPH ALGO BETWEENNESS ON 'network' SAMPLE 500;
GRAPH ALGO KCORE ON 'collaboration';
GRAPH ALGO TRIANGLES ON 'social' MODE global;
GRAPH ALGO DIAMETER ON 'web';
```

Available: `PAGERANK`, `WCC`, `COMMUNITY` (or `LABEL_PROPAGATION` alias), `LCC`, `SSSP`, `BETWEENNESS`, `CLOSENESS`, `HARMONIC`, `DEGREE`, `LOUVAIN`, `TRIANGLES`, `DIAMETER`, `KCORE`.

**Personalized PageRank** uses the optional `PERSONALIZATION` clause to bias node importance toward specified seed nodes. The JSON object maps node IDs to seed weights (higher weight = more biased teleport probability toward that node). Example: `PERSONALIZATION {"alice": 1.0, "bob": 0.5}` makes PageRank favor paths leading to Alice twice as much as Bob.

## GraphRAG

```sql
GRAPH RAG FUSION ON entities
  QUERY $embedding VECTOR_FIELD 'embedding' VECTOR_TOP_K 50 EXPANSION_DEPTH 2
  EDGE_LABEL 'related_to' FINAL_TOP_K 10 RRF_K (60.0, 35.0);
```


---
# Spatial Queries (docs/sql/spatial-queries)

---
title: Spatial Queries
description: OGC spatial predicates, distance queries, geofencing, and spatial joins.
---

# Spatial Queries

## Proximity Search

```sql
SELECT name, ST_Distance(location, ST_Point(-73.990, 40.750)) AS dist
FROM restaurants
WHERE ST_DWithin(location, ST_Point(-73.990, 40.750), 1000)
ORDER BY dist;
```

## Geofencing (Point-in-Polygon)

```sql
SELECT name FROM restaurants
WHERE ST_Within(location, ST_GeomFromGeoJSON('{
    "type": "Polygon",
    "coordinates": [[[-74.0, 40.7], [-73.9, 40.7], [-73.9, 40.8], [-74.0, 40.8], [-74.0, 40.7]]]
}'));
```

## OGC Predicates

| Function          | Description                                          |
| ----------------- | ---------------------------------------------------- |
| `ST_Contains`     | Geometry A contains geometry B                       |
| `ST_Within`       | Geometry A is within geometry B                      |
| `ST_Intersects`   | Geometries share any space                           |
| `ST_Disjoint`     | Geometries share no space                            |
| `ST_DWithin`      | Within a given geodesic distance in meters           |
| `ST_IsValid`      | Whether a geometry is well-formed                    |

## Constructors

| Function                                            | Result                                                        |
| --------------------------------------------------- | ------------------------------------------------------------- |
| `ST_Point(lng, lat)`                                | Point                                                          |
| `ST_MakePoint(x, y [, z])`                          | Point                                                          |
| `ST_GeomFromText(wkt [, srid])`                     | Geometry from WKT                                              |
| `ST_GeomFromGeoJSON(json)`                          | Geometry from GeoJSON                                          |
| `ST_GeomFromWKB(bytes [, srid])`                    | Geometry from WKB; accepts `X'...'` or its hex string          |
| `ST_MakeLine(point, point, ...)`                    | LineString                                                     |
| `ST_MakePolygon(ring, ...)`                         | Polygon from arrays of `[lng, lat]` pairs                      |
| `ST_MakeEnvelope(min_lng, min_lat, max_lng, max_lat)` | Rectangular Polygon                                          |

## Accessors and Measures

Measures are geodesic — meters, and square meters for area.

| Function                | Result                                                              |
| ----------------------- | ------------------------------------------------------------------- |
| `ST_AsText(geom)`       | WKT rendering                                                        |
| `ST_AsGeoJSON(geom)`    | GeoJSON rendering                                                    |
| `ST_X(geom)` / `ST_Y(geom)` | Ordinate of a Point; NULL for any other geometry                 |
| `ST_GeometryType(geom)` | Type name, e.g. `Point`                                              |
| `ST_NPoints(geom)`      | Total vertex count                                                   |
| `ST_SRID(geom)`         | Always `4326` — geometry is stored as GeoJSON, i.e. WGS 84           |
| `ST_Distance(a, b)`     | Distance between two geometries                                      |
| `ST_Length(geom)`       | Length of linear components                                          |
| `ST_Perimeter(geom)`    | Boundary length of areal components                                  |
| `ST_Area(geom)`         | Area of areal components, less holes                                 |

## Geometry Operations

| Function                             | Result                                        |
| ------------------------------------ | --------------------------------------------- |
| `ST_Buffer(geom, meters [, segments])` | Geometry grown by a distance                |
| `ST_Envelope(geom)`                  | Bounding box                                  |
| `ST_Centroid(geom)`                  | Centroid, weighted by highest dimension       |
| `ST_Union(a, b)`                     | Union of geometries                           |
| `ST_Intersection(a, b)`              | Intersection geometry                         |

## Reading Stored Geometry

Every function above works in every position a geometry expression may appear —
an `INSERT` value, a `SELECT` projection, and a predicate's query-geometry
argument. Constructors nest, so a buffered point is a valid search area, and a
bare string literal is read as WKT or GeoJSON.

```sql
SELECT name, ST_AsText(location), ST_X(location), ST_Y(location)
FROM restaurants;

SELECT name FROM restaurants
WHERE ST_Within(location, ST_Buffer(ST_Point(-73.990, 40.750), 1000));

SELECT name FROM restaurants
WHERE ST_DWithin(location, 'POINT(-73.990 40.750)', 1000);
```

## Spatial Join

```sql
SELECT r.name, z.zone_name FROM restaurants r, delivery_zones z
WHERE ST_Contains(z.boundary, r.location);
```

## H3 Hexagonal Indexing

```sql
SELECT h3_latlngtocell(40.748, -73.985, 9) AS hex;
```

## Hybrid Spatial-Vector

```sql
SELECT name, vector_distance(embedding, $query_vec) AS similarity
FROM restaurants
WHERE ST_DWithin(location, ST_Point(-73.990, 40.750), 2000) AND embedding <-> $query_vec
LIMIT 10;
```


---
# CRDT Operations (docs/sql/crdt-operations)

---
title: CRDT Operations
description: Read CRDT state and apply deltas via SQL.
---

# CRDT Operations

## Read State

```sql
SELECT crdt_state('collab_docs', 'doc123');
```

Returns the current merged state of the CRDT document.

## Apply Delta

```sql
SELECT crdt_apply('collab_docs', 'doc123', '<delta_bytes>');
```

Applies a Loro CRDT delta to the document. On Origin, the delta goes through Raft consensus. On Lite, it is applied locally and queued for sync.

## Conflict Policies

Conflict policies (last-writer-wins, field-merge, custom) are configured per-collection at create time and via the native NodeDB protocol's `ALTER POLICY` command. SQL DDL for inspecting and updating conflict policies — `SHOW CONFLICT POLICY ON <collection>` and `ALTER COLLECTION ... SET ON CONFLICT ...` — is available alongside the rest of the SQL surface. See the protocol reference for the native-protocol form, and [CRDT Sync](/docs/crdt-sync/overview) for full sync protocol details.


---
# Cross-Engine Fusion (RRF) (docs/sql/fusion-rrf)

---
title: Cross-Engine Fusion (RRF)
description: Reciprocal Rank Fusion for combining results from multiple engines in one query.
---

# Cross-Engine Fusion (RRF)

Reciprocal Rank Fusion merges ranked results from two engines into a single result set. No application-level merging needed. Two pairwise fusions are supported: vector + BM25 text via `rrf_score(...)`, and vector + graph via either `GRAPH RAG FUSION ON ...` or `SEARCH ... USING FUSION(...)`.

## How RRF Works

For each result, compute: `score = Σ 1 / (k + rank_i)` where `k` is a constant (default 60) and `rank_i` is the result's rank from each source.

## Vector + Full-Text

```sql
SELECT title, rrf_score(
    vector_distance(embedding, $query_vec),
    bm25_score(body, 'distributed database')
) AS score
FROM articles
LIMIT 10;
```

## GraphRAG (Vector + Graph)

```sql
GRAPH RAG FUSION ON entities
  QUERY ARRAY[0.1, 0.3, -0.2, ...]
  VECTOR_FIELD 'embedding'
  VECTOR_TOP_K 50
  EXPANSION_DEPTH 2
  EDGE_LABEL 'related_to'
  FINAL_TOP_K 10
  RRF_K (60.0, 35.0);
```

Equivalent shorthand:

```sql
SEARCH entities USING FUSION(
  ARRAY[0.1, 0.3, -0.2, ...]
  VECTOR_FIELD 'embedding' VECTOR_TOP_K 50
  DEPTH 2 LABEL 'related_to'
  TOP 10 RRF_K (60.0, 35.0)
);
```

Vector search finds seed nodes, graph BFS expands context, RRF merges both rankings.

## Three-Source Fusion (Vector + Text + Graph)

When you want vector similarity, BM25 text relevance, and graph context all merged in one pass, add the `BM25 '<query>' ON '<field>'` leg to the `SEARCH … USING FUSION` DSL. The `RRF_K` tuple becomes a triple.

```sql
SEARCH entities USING FUSION(
    ARRAY[0.1, 0.3, -0.2, ...]
    VECTOR_FIELD 'embedding' VECTOR_TOP_K 50
    BM25 'transformer attention' ON 'body'
    DEPTH 2 LABEL 'related_to'
    TOP 10 RRF_K (60.0, 35.0, 50.0)
);
```

The three `RRF_K` values correspond to the vector leg, the graph expansion leg, and the BM25 leg respectively. When all three are present the planner routes through the three-source RRF plan variant.

The two-source `GRAPH RAG FUSION ON … BM25 '…' ON '…'` form also accepts the extra text leg inline:

```sql
GRAPH RAG FUSION ON entities
  QUERY $embedding VECTOR_FIELD 'embedding' VECTOR_TOP_K 50
  BM25 'transformer attention' ON 'body'
  EXPANSION_DEPTH 2 EDGE_LABEL 'related_to' FINAL_TOP_K 10
  RRF_K (60.0, 35.0, 50.0);
```

## Cross-Model Queries

All engines share the same snapshot. A query that combines vector similarity, graph traversal, spatial filtering, and document field access sees a consistent point-in-time view.

```sql
-- Spatial filter → vector rank → document fields
SELECT name, vector_distance(embedding, $vec) AS sim
FROM restaurants
WHERE ST_DWithin(location, ST_Point(-73.99, 40.75), 2000)
  AND embedding <-> $vec
LIMIT 10;
```


---
# JSON Operators (docs/sql/json-operators)

---
title: JSON Operators
description: PostgreSQL-compatible JSON path operators for navigating and testing document fields in NodeDB.
---

# JSON Operators

NodeDB supports all nine PostgreSQL JSON path operators on document fields and any expression that produces a JSON value.

## Navigation Operators

### `->` — field as JSON

Returns the value at the given key (or array index) as a JSON value. The result is `NULL` if the key does not exist.

```sql
-- Object field
SELECT payload -> 'address' FROM users;

-- Nested path via chaining
SELECT payload -> 'address' -> 'city' FROM users;

-- Array element by index
SELECT tags -> 0 FROM articles;
```

### `->>` — field as text

Same as `->` but casts the result to `TEXT`. Useful in `WHERE` clauses that compare to string literals.

```sql
SELECT payload ->> 'name' FROM users;

SELECT * FROM users WHERE payload ->> 'role' = 'admin';
```

### `#>` — path as JSON

Navigates a multi-element path supplied as a text array. Equivalent to chaining multiple `->` operators.

```sql
SELECT payload #> '{address, city}' FROM users;

-- Same result as: payload -> 'address' -> 'city'
SELECT payload #> ARRAY['address', 'city'] FROM users;
```

### `#>>` — path as text

Same as `#>` but casts the result to `TEXT`.

```sql
SELECT payload #>> '{address, city}' FROM users;

SELECT * FROM orders WHERE payload #>> '{shipping, country}' = 'DE';
```

## Containment Operators

### `@>` — left contains right

Returns `true` when the left JSON value contains every key/value pair in the right JSON object.

```sql
-- Rows where role is 'admin'
SELECT * FROM users WHERE payload @> '{"role": "admin"}';

-- Rows where the tags array contains the element 'rust'
SELECT * FROM articles WHERE tags @> '["rust"]';
```

### `<@` — left contained in right

The inverse of `@>`. Returns `true` when the left value is fully contained in the right.

```sql
SELECT * FROM users WHERE '{"role": "admin"}' <@ payload;
```

## Existence Operators

### `?` — key exists

Returns `true` if the top-level object contains the given key, or if the array contains the given string element.

```sql
-- Only rows that have an 'email' field
SELECT * FROM users WHERE payload ? 'email';
```

### `?|` — any key exists

Returns `true` if the object contains **any** of the supplied keys.

```sql
SELECT * FROM users WHERE payload ?| ARRAY['phone', 'email'];
```

### `?&` — all keys exist

Returns `true` if the object contains **all** of the supplied keys.

```sql
SELECT * FROM users WHERE payload ?& ARRAY['name', 'email', 'role'];
```

## Operator Summary

| Operator | Left type | Right type  | Returns | Description               |
| -------- | --------- | ----------- | ------- | ------------------------- |
| `->`     | JSON      | TEXT or INT | JSON    | Field / index as JSON     |
| `->>`    | JSON      | TEXT or INT | TEXT    | Field / index as text     |
| `#>`     | JSON      | TEXT[]      | JSON    | Multi-element path as JSON|
| `#>>`    | JSON      | TEXT[]      | TEXT    | Multi-element path as text|
| `@>`     | JSON      | JSON        | BOOL    | Left contains right       |
| `<@`     | JSON      | JSON        | BOOL    | Left contained in right   |
| `?`      | JSON      | TEXT        | BOOL    | Key exists                |
| `?|`     | JSON      | TEXT[]      | BOOL    | Any key exists            |
| `?&`     | JSON      | TEXT[]      | BOOL    | All keys exist            |

## Notes

- On **schemaless document** collections the top-level document payload is accessed directly as a JSON value.
- On **strict document** collections fields are binary-encoded; the JSON operators trigger a decode step. For tight-loop analytics on strict collections, prefer direct column projections over JSON navigation.
- Operator precedence follows PostgreSQL conventions. When mixing with boolean operators, add parentheses: `WHERE (payload ? 'email') AND status = 'active'`.
- Indexed containment queries (`@>`) benefit from a `CREATE SEARCH INDEX` or a dedicated secondary index on the field being tested.


---
# COPY FROM / COPY TO (docs/sql/copy)

---
title: COPY FROM / COPY TO
description: Bulk import and export of collections using NDJSON, JSON array, and CSV formats.
---

# COPY FROM / COPY TO

`COPY` moves data between a NodeDB collection and a file on the server's filesystem. It is the fastest way to load or dump large datasets.

## COPY FROM — import

```sql
COPY <collection> FROM '<path>' [WITH (FORMAT <fmt> [, DELIMITER '<char>'] [, HEADER <bool>])]
```

Reads the file at `<path>` on the server and inserts every record into `<collection>`. The format is either specified explicitly or auto-detected from the file extension.

### Auto-detected formats

| Extension           | Format       |
| ------------------- | ------------ |
| `.ndjson`, `.jsonl` | NDJSON       |
| `.json`             | JSON array   |
| `.csv`              | CSV          |

### Examples

```sql
-- NDJSON (one JSON object per line) — format auto-detected from extension
COPY users FROM '/data/users.ndjson';

-- JSON array (a single top-level array of objects)
COPY users FROM '/data/users.json' WITH (FORMAT json_array);

-- CSV with header row (default when FORMAT csv)
COPY orders FROM '/data/orders.csv' WITH (FORMAT csv);

-- CSV without header row
COPY orders FROM '/data/orders.csv' WITH (FORMAT csv, HEADER false);

-- CSV with non-comma delimiter
COPY products FROM '/data/products.tsv' WITH (FORMAT csv, DELIMITER '\t');
```

### WITH options

| Option      | Values                          | Default           | Description                             |
| ----------- | ------------------------------- | ----------------- | --------------------------------------- |
| `FORMAT`    | `ndjson`, `json_array`, `csv`   | auto from extension | File format                           |
| `DELIMITER` | Any single character            | `,`               | Field separator (CSV only)              |
| `HEADER`    | `true` / `false`                | `true`            | Whether the first CSV row is a header   |

## COPY TO — export

```sql
COPY <collection> TO '<path>' [WITH (FORMAT <fmt> [, DELIMITER '<char>'] [, HEADER <bool>])]
COPY (SELECT ...) TO '<path>' [WITH (...)]
```

Writes every row from `<collection>` (or the result of a SELECT) to a file at `<path>` on the server. The format is auto-detected from the extension or specified explicitly.

### Examples

```sql
-- Export entire collection as NDJSON
COPY users TO '/exports/users.ndjson';

-- Export with explicit format
COPY users TO '/exports/users.json' WITH (FORMAT json_array);

-- Export as CSV with header
COPY orders TO '/exports/orders.csv' WITH (FORMAT csv);

-- Export the result of a query
COPY (SELECT id, name, email FROM users WHERE status = 'active')
    TO '/exports/active_users.csv' WITH (FORMAT csv);

-- Query export as NDJSON
COPY (SELECT * FROM orders WHERE created_at > '2025-01-01')
    TO '/exports/recent_orders.ndjson';
```

## Notes

- Paths are resolved on the **server** filesystem. The client never streams file bytes — this is not `\copy` (psql client-side copy).
- `COPY FROM STDIN` is reserved for the backup/restore path and is not available as a general import mechanism.
- Rows that fail type coercion during import are rejected with a parse error; no partial-row insertion occurs.
- `COPY TO` is non-transactional: the file is written even if the session is later rolled back.
- The `COPY (SELECT ...) TO` form supports all SELECT features including JOINs, CTEs, and WHERE clauses.


---
# LISTEN / NOTIFY / UNLISTEN (docs/sql/listen-notify)

---
title: LISTEN / NOTIFY / UNLISTEN
description: Async pgwire change notifications — subscribe to collection mutations and send custom payloads.
---

# LISTEN / NOTIFY / UNLISTEN

NodeDB implements PostgreSQL-compatible asynchronous notifications over pgwire. A client subscribes to a **collection** and receives a notification message every time a row in that collection is mutated by a committed transaction.

## Subscribing

```sql
LISTEN <collection>;
```

After this command the connection enters a listening state for the named collection. Multiple collections can be listened on the same connection.

```sql
LISTEN orders;
LISTEN inventory;
```

## Unsubscribing

```sql
-- Stop listening to a specific collection
UNLISTEN orders;

-- Stop listening to all collections on this connection
UNLISTEN *;
```

## Sending a manual notification

```sql
NOTIFY <channel>, '<payload>';
```

`NOTIFY` sends an arbitrary text payload to all connections currently listening on the named channel. The channel name does not have to be an existing collection — it is just a string label.

```sql
NOTIFY alerts, 'deployment started';
NOTIFY user_events, '{"user_id": "u1", "event": "login"}';
```

## Receiving notifications

Notifications are delivered as PostgreSQL `NotificationResponse` messages. In psql they appear immediately after the triggering transaction commits:

```
Asynchronous notification "orders" with payload "" from server process with PID 12345.
```

Most PostgreSQL client libraries expose an async notification callback:

```python
# Python (psycopg2)
conn.set_isolation_level(0)  # autocommit required for LISTEN
cur.execute("LISTEN orders")
while True:
    select.select([conn], [], [])
    conn.poll()
    while conn.notifies:
        notify = conn.notifies.pop()
        print(f"channel={notify.channel} payload={notify.payload}")
```

## Transaction semantics

Notifications triggered by collection mutations are **buffered until `COMMIT`** and **dropped on `ROLLBACK`**.

```sql
BEGIN;
INSERT INTO orders (id, total) VALUES ('o1', 99.99);  -- notification queued, not sent yet
COMMIT;                                                -- notification delivered now

BEGIN;
INSERT INTO orders (id, total) VALUES ('o2', 50.00);
ROLLBACK;                                              -- notification dropped, never sent
```

This guarantees that listeners never see notifications for writes that were rolled back.

## Tenant scoping

Notifications are tenant-scoped. A connection authenticated to tenant `acme` only receives notifications for mutations within the `acme` tenant. A connection cannot receive notifications for a different tenant's collections, even if it knows the collection name.

## Notification payload

For collection-mutation notifications the payload is empty by default. The full change data is available via [Change Streams](../real-time/change-streams) if you need the actual before/after row values.

`NOTIFY <channel>, '<payload>'` delivers the payload string exactly as provided.

## Notes

- `LISTEN` subscribes to a **collection name**, not an arbitrary string channel. Listening on a name that does not correspond to an existing collection returns no error but will never fire.
- The connection must remain open to receive notifications — notifications are not queued for reconnection.
- A single connection can listen on multiple collections simultaneously.
- `NOTIFY` without a payload (`NOTIFY channel`) is not supported; the payload argument is required.


---
# Custom Types (docs/sql/custom-types)

---
title: Custom Types
description: CREATE TYPE for enums and composite types, ALTER TYPE ADD VALUE, and DROP TYPE.
---

# Custom Types

NodeDB supports user-defined types in two forms: **enum types** (a fixed, ordered set of string labels) and **composite types** (a named record with typed fields). Both can be used as column types in `CREATE COLLECTION` and `CREATE TABLE` statements.

## Enum Types

```sql
CREATE TYPE <name> AS ENUM ('<label1>', '<label2>', ...)
```

Defines a type whose valid values are the listed labels. Labels are case-sensitive strings. Duplicates are rejected at definition time.

```sql
CREATE TYPE order_status AS ENUM ('pending', 'processing', 'shipped', 'delivered', 'cancelled');

CREATE TYPE priority AS ENUM ('low', 'medium', 'high', 'critical');
```

### Using enum types

```sql
CREATE COLLECTION orders (
    id TEXT PRIMARY KEY,
    status order_status,
    priority priority
) WITH (engine = 'document_strict');

INSERT INTO orders (id, status, priority) VALUES ('o1', 'pending', 'high');

SELECT * FROM orders WHERE status = 'pending';
```

### Adding a new label

```sql
ALTER TYPE order_status ADD VALUE 'on_hold';
```

The new value is appended to the end of the enum. Existing rows are unaffected. `ADD VALUE` is the only supported `ALTER TYPE` form.

## Composite Types

```sql
CREATE TYPE <name> AS (<field1> <type1>, <field2> <type2>, ...)
```

Defines a named record type. Each field has a name and a SQL data type.

```sql
CREATE TYPE address AS (
    street TEXT,
    city   TEXT,
    zip    TEXT,
    country TEXT
);

CREATE TYPE money_amount AS (
    value    FLOAT,
    currency TEXT
);
```

### Using composite types

```sql
CREATE COLLECTION users (
    id      TEXT PRIMARY KEY,
    name    TEXT,
    billing address
) WITH (engine = 'document_strict');

INSERT INTO users (id, name, billing)
VALUES ('u1', 'Alice', ROW('123 Main St', 'Springfield', '12345', 'US'));

SELECT billing.city FROM users WHERE id = 'u1';
```

## Inspecting types

```sql
SHOW TYPES;
```

Lists all user-defined types in the current tenant, their kind (enum or composite), and their labels or fields.

## Dropping types

```sql
DROP TYPE order_status;
DROP TYPE IF EXISTS address;
```

`DROP TYPE` is **protected**: it fails if any collection has a column of that type. Drop or alter the dependent collections first, then drop the type.

## OIDs

Custom types are assigned OIDs starting at 70001. These appear in `pg_type` catalog queries and in the pgwire type-description messages for columns that use the custom type.

<Callout kind="warning" title="pgwire binary codec">
Custom types are currently sent over pgwire in TEXT format regardless of the client's requested format. Binary-format decoding for custom type OIDs is not yet implemented. Most PostgreSQL client libraries fall back gracefully to text parsing.
</Callout>

## Validation rules

- **Enum labels** must be non-empty strings. Duplicate labels within the same type are rejected.
- **Composite fields** must have distinct names within the same type. The field types can be any built-in SQL type; nested composite types are not yet supported.
- **Type names** are lowercased and must be unique within the tenant.


---
# Window Frames (docs/sql/window-frames)

---
title: Window Frames
description: ROWS, RANGE, and GROUPS frame modes for window functions — all boundary combinations with examples.
---

# Window Frames

Window functions operate over an ordered set of rows called the **window frame**. The frame is a subset of the partition defined by the `OVER` clause. NodeDB supports all three frame modes: `ROWS`, `RANGE`, and `GROUPS`.

## Frame syntax

```sql
<window_fn>() OVER (
    [PARTITION BY <col>, ...]
    ORDER BY <col> [ASC|DESC]
    [ROWS | RANGE | GROUPS] BETWEEN <start> AND <end>
)
```

When no frame clause is written, the default is `RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW`.

## Frame modes

### ROWS

`ROWS` counts **physical rows**. Each peer row in the ordering is distinct regardless of its sort value.

```sql
-- Running sum over the previous 2 rows and current row
SELECT id, amount,
    SUM(amount) OVER (
        ORDER BY created_at
        ROWS BETWEEN 2 PRECEDING AND CURRENT ROW
    ) AS rolling_3
FROM transactions;

-- Running total from start of partition to current row
SELECT id, amount,
    SUM(amount) OVER (
        PARTITION BY user_id ORDER BY created_at
        ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
    ) AS cumulative
FROM transactions;
```

### RANGE

`RANGE` groups rows that have **equal values** in the ORDER BY column. All peers (rows with the same sort value) fall inside the same frame boundary.

```sql
-- Sum all rows with the same or earlier date as the current row
SELECT order_date, total,
    SUM(total) OVER (
        ORDER BY order_date
        RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
    ) AS daily_cumulative
FROM daily_orders;

-- Moving sum of rows within a numeric distance of 10 from the current row's value
SELECT score,
    SUM(score) OVER (
        ORDER BY score
        RANGE BETWEEN 10 PRECEDING AND 10 FOLLOWING
    ) AS nearby_sum
FROM scores;
```

<Callout kind="info" title="RANGE with numeric/interval offsets">
`RANGE BETWEEN N PRECEDING AND N FOLLOWING` with a numeric or interval literal requires the ORDER BY column to be a numeric or timestamp type. The planner validates this at planning time.
</Callout>

### GROUPS

`GROUPS` counts **peer groups** (groups of rows with equal ORDER BY values), not individual rows. `1 PRECEDING` means the previous peer group, not the previous row.

```sql
-- Sum the current group and the two preceding groups
SELECT category, day, revenue,
    SUM(revenue) OVER (
        PARTITION BY category ORDER BY day
        GROUPS BETWEEN 2 PRECEDING AND CURRENT ROW
    ) AS group_rolling
FROM daily_revenue;
```

## Frame boundaries

| Boundary             | Meaning                                                              |
| -------------------- | -------------------------------------------------------------------- |
| `UNBOUNDED PRECEDING` | First row/range/group in the partition                              |
| `N PRECEDING`         | N rows/ranges/groups before the current row                         |
| `CURRENT ROW`         | The current row (for ROWS), or the current peer group (for RANGE/GROUPS) |
| `N FOLLOWING`         | N rows/ranges/groups after the current row                          |
| `UNBOUNDED FOLLOWING` | Last row/range/group in the partition                               |

The start boundary must not be greater than the end boundary.

## Common patterns

```sql
-- Sliding window average over 7 days
SELECT ts, value,
    AVG(value) OVER (
        ORDER BY ts
        ROWS BETWEEN 6 PRECEDING AND CURRENT ROW
    ) AS avg_7d
FROM metrics;

-- Percent of partition total
SELECT dept, salary,
    salary / SUM(salary) OVER (PARTITION BY dept) AS pct_of_dept
FROM employees;

-- First and last value in the partition
SELECT id, score,
    FIRST_VALUE(score) OVER (PARTITION BY user_id ORDER BY played_at
        ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) AS first_score,
    LAST_VALUE(score)  OVER (PARTITION BY user_id ORDER BY played_at
        ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) AS last_score
FROM game_sessions;
```

## Named windows

Repeated window definitions can be factored out with a `WINDOW` clause:

```sql
SELECT id, amount,
    SUM(amount)   OVER w AS running_sum,
    AVG(amount)   OVER w AS running_avg,
    COUNT(*)      OVER w AS running_count
FROM transactions
WINDOW w AS (ORDER BY created_at ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW);
```

## Supported window functions

`ROW_NUMBER`, `RANK`, `DENSE_RANK`, `PERCENT_RANK`, `CUME_DIST`, `NTILE`, `LAG`, `LEAD`, `FIRST_VALUE`, `LAST_VALUE`, `NTH_VALUE`, `SUM`, `AVG`, `MIN`, `MAX`, `COUNT`.


---
# Lateral Subqueries (docs/sql/lateral-subqueries)

---
title: Lateral Subqueries
description: LATERAL FROM and JOIN LATERAL — correlated subqueries in the FROM clause with three planner modes.
---

# Lateral Subqueries

A `LATERAL` subquery in the `FROM` clause (or in a `JOIN`) can reference columns from tables listed earlier in the same `FROM` clause. Without `LATERAL`, a subquery cannot see the outer tables.

## Syntax

### LATERAL in FROM (comma join)

```sql
SELECT <cols>
FROM <outer_table> [AS <alias>],
     LATERAL (SELECT ... WHERE ... = <outer_table>.<col>) AS <lateral_alias>
```

### JOIN LATERAL

```sql
SELECT <cols>
FROM <outer_table> [AS <alias>]
[LEFT] JOIN LATERAL (<subquery>) AS <lateral_alias> ON true
```

`ON true` is idiomatic when the correlation condition is already inside the subquery's `WHERE` clause. `LEFT JOIN LATERAL` preserves outer rows for which the subquery returns no rows (it produces `NULL` for the lateral columns).

## Examples

### Top-N per group

Return the three most recent orders for each user:

```sql
SELECT u.id, u.name, o.id AS order_id, o.created_at
FROM users u
JOIN LATERAL (
    SELECT id, created_at
    FROM orders
    WHERE user_id = u.id
    ORDER BY created_at DESC
    LIMIT 3
) AS o ON true;
```

### Expand array column into rows

```sql
SELECT p.id, tag
FROM products p,
     LATERAL (SELECT unnest(p.tags) AS tag) AS t;
```

### LEFT JOIN LATERAL — keep users with no orders

```sql
SELECT u.id, o.last_order_at
FROM users u
LEFT JOIN LATERAL (
    SELECT MAX(created_at) AS last_order_at FROM orders WHERE user_id = u.id
) AS o ON true;
```

## Planning modes

The planner selects one of three execution strategies automatically based on the shape of the correlation in the subquery.

### Hash-join rewrite

When the subquery is equi-correlated (the `WHERE` clause has `inner.col = outer.col` with no non-equi predicates) and has **no `LIMIT`**, the planner removes the correlation and rewrites the query as a regular hash join. This is the most efficient path.

```sql
-- Equi-correlated, no LIMIT → hash-join rewrite
SELECT u.id, o.total
FROM users u,
LATERAL (SELECT total FROM orders WHERE user_id = u.id) AS o;
```

### LateralTopK

When the subquery is equi-correlated **and has a `LIMIT k`**, the planner emits a `LateralTopK` plan. For each outer row it performs a bounded index-seek on the inner collection and returns at most `k` rows. This is the plan used for "top-N per group" queries and is the most common LATERAL shape.

```sql
-- Equi-correlated + LIMIT → LateralTopK
SELECT u.id, o.id
FROM users u
JOIN LATERAL (
    SELECT id FROM orders WHERE user_id = u.id ORDER BY created_at DESC LIMIT 5
) o ON true;
```

### LateralLoop

When the subquery has non-equi correlations (e.g., range predicates, expressions, or function calls involving the outer column), the planner emits a `LateralLoop` plan. For each outer row the full inner subquery is evaluated. To prevent runaway execution, `LateralLoop` is limited to **100 000 outer rows**; queries that would exceed this cap are rejected at planning time with a clear error.

```sql
-- Non-equi correlation → LateralLoop
SELECT e.id, recent.id AS nearby_event
FROM events e,
LATERAL (
    SELECT id FROM events
    WHERE ts BETWEEN e.ts - INTERVAL '5 minutes' AND e.ts + INTERVAL '5 minutes'
      AND id <> e.id
    LIMIT 10
) AS recent;
```

<Callout kind="tip" title="Performance guidance">
Prefer equi-correlated LATERAL with a LIMIT when possible — the `LateralTopK` plan uses an index seek per outer row rather than a full scan. If you need a non-equi correlation over a large outer set, add a `WHERE` clause on the outer table to reduce the outer row count below 100 000.
</Callout>

## Notes

- The `LATERAL` keyword must appear before the opening `(` of the subquery.
- `LATERAL (SELECT ...)` must produce a `SELECT` body — set operations (`UNION`, `INTERSECT`, etc.) at the top level of the lateral body are not supported.
- References to outer columns must be to tables listed **before** the `LATERAL` in the `FROM` clause. Forward references are not allowed.


---
# Recursive CTEs (docs/sql/recursive-cte)

---
title: Recursive CTEs
description: WITH RECURSIVE for hierarchical queries, graph traversal in SQL, and value-generating sequences.
---

# Recursive CTEs

`WITH RECURSIVE` defines a CTE that refers to itself, allowing hierarchical or iterative computations in a single SQL statement.

## Syntax

```sql
WITH RECURSIVE <name> [(<col1>, <col2>, ...)] AS (
    <anchor>           -- non-recursive base query
    UNION [ALL]
    <recursive_arm>    -- references <name>
)
SELECT ... FROM <name>;
```

The anchor query runs once. The recursive arm is then evaluated repeatedly with the previous iteration's output as the input, until it produces no new rows. Results are the union of all iterations.

`UNION` deduplicates across iterations; `UNION ALL` keeps all rows including duplicates and is faster when you know no cycles exist.

## Examples

### Hierarchy traversal

Walk an employee reporting tree from a given root node:

```sql
WITH RECURSIVE subordinates AS (
    -- Anchor: start at the root employee
    SELECT id, name, manager_id, 0 AS depth
    FROM employees
    WHERE id = 'emp_root'

    UNION ALL

    -- Recursive arm: follow manager_id links
    SELECT e.id, e.name, e.manager_id, s.depth + 1
    FROM employees e
    JOIN subordinates s ON e.manager_id = s.id
)
SELECT id, name, depth FROM subordinates ORDER BY depth;
```

### Path finding

Accumulate the path as a text column:

```sql
WITH RECURSIVE path_cte AS (
    SELECT id, name, manager_id, CAST(name AS TEXT) AS path
    FROM employees
    WHERE id = 'emp_root'

    UNION ALL

    SELECT e.id, e.name, e.manager_id, p.path || ' → ' || e.name
    FROM employees e
    JOIN path_cte p ON e.manager_id = p.id
)
SELECT id, path FROM path_cte;
```

### Value-generating form

Generate a sequence of integers without a backing collection:

```sql
WITH RECURSIVE counter(n) AS (
    SELECT 1
    UNION ALL
    SELECT n + 1 FROM counter WHERE n < 10
)
SELECT n FROM counter;
```

Generate a date series:

```sql
WITH RECURSIVE dates(d) AS (
    SELECT CAST('2025-01-01' AS TIMESTAMP)
    UNION ALL
    SELECT d + INTERVAL '1 day' FROM dates WHERE d < '2025-01-31'
)
SELECT d FROM dates;
```

## Validation rules

The planner enforces these rules at planning time:

1. **UNION or UNION ALL only.** The recursive arm must be connected to the anchor with `UNION` or `UNION ALL`. `INTERSECT` and `EXCEPT` are not allowed.
2. **Single self-reference.** The recursive arm must reference the CTE name exactly once. Multiple self-references in the same arm are rejected.
3. **Column-count match.** If you declare an explicit column list (`WITH RECURSIVE name (col1, col2)`) the anchor's SELECT must produce the same number of columns.
4. **Maximum depth.** Recursion stops after **1 000 iterations** by default. Queries that would exceed this depth are terminated and an error is returned.

## Cycle prevention

For graph-like data where cycles are possible, use `UNION` (instead of `UNION ALL`) to deduplicate rows, or track visited IDs in an accumulated array column:

```sql
WITH RECURSIVE traversal(id, visited) AS (
    SELECT id, ARRAY[id] FROM nodes WHERE id = 'start'
    UNION ALL
    SELECT n.id, t.visited || n.id
    FROM nodes n
    JOIN edges e ON e.from_id = t.id AND e.to_id = n.id
    JOIN traversal t ON e.from_id = t.id
    WHERE NOT (n.id = ANY(t.visited))
)
SELECT id FROM traversal;
```

## Notes

- `WITH RECURSIVE` works across all engine types. The anchor and recursive arm can reference collections from different engines.
- The recursive arm is evaluated on the Data Plane; complex expressions in the arm (e.g., aggregates, window functions, subqueries) are supported subject to the same constraints as non-recursive queries.
- For deep hierarchies (depth > several hundred levels), consider using the [Graph engine](graph-queries) TRAVERSE or ALGO commands, which use a native BSP implementation and are not subject to the 1 000-depth cap.


---
# Grouping Sets (docs/sql/grouping-sets)

---
title: Grouping Sets
description: GROUP BY ROLLUP, CUBE, and GROUPING SETS for multi-dimensional aggregation, with the GROUPING() function.
---

# Grouping Sets

`ROLLUP`, `CUBE`, and `GROUPING SETS` let you compute aggregates over multiple grouping combinations in a single query, which is far more efficient than running multiple `GROUP BY` queries and combining the results with `UNION ALL`.

## ROLLUP

`ROLLUP(a, b, c)` produces all "leading prefix" subsets plus the grand total:

- `(a, b, c)`
- `(a, b)`
- `(a)`
- `()`  ← grand total

Useful for subtotals in hierarchical reports (e.g., region → country → city).

```sql
SELECT region, country, SUM(revenue) AS total
FROM sales
GROUP BY ROLLUP (region, country);
```

The rows where `country` is `NULL` are the per-region subtotals; the row where both are `NULL` is the grand total.

## CUBE

`CUBE(a, b)` produces every possible subset combination:

- `(a, b)`
- `(a)`
- `(b)`
- `()`

For N columns, `CUBE` generates 2^N grouping sets.

```sql
SELECT product_line, quarter, SUM(units) AS units_sold
FROM inventory
GROUP BY CUBE (product_line, quarter);
```

## GROUPING SETS

`GROUPING SETS` gives you explicit control over which combinations to compute:

```sql
SELECT region, country, SUM(revenue)
FROM sales
GROUP BY GROUPING SETS ((region, country), (region), ());
```

The three sets above are: country-level detail, region subtotals, and grand total — the same as `ROLLUP(region, country)`.

## Mixing plain GROUP BY with ROLLUP

You can mix a plain column with a `ROLLUP` or `GROUPING SETS` expression:

```sql
SELECT channel, region, country, SUM(revenue)
FROM sales
GROUP BY channel, ROLLUP (region, country);
```

This produces all `ROLLUP` groupings nested inside each `channel` value. Only one `ROLLUP`/`CUBE`/`GROUPING SETS` expression is allowed per `GROUP BY` clause.

## GROUPING() function

When `NULL` appears in a grouping column it is ambiguous: it could be a null in the data, or it could be a null because the column was excluded from the current grouping set. `GROUPING(col)` resolves this:

- Returns `0` if `col` is part of the current grouping set (the `NULL` is a real data null or the value is present).
- Returns `1` if `col` was excluded from the current grouping set (the `NULL` is a rollup placeholder).

```sql
SELECT
    region,
    country,
    SUM(revenue) AS total,
    GROUPING(region)  AS region_is_subtotal,
    GROUPING(country) AS country_is_subtotal
FROM sales
GROUP BY ROLLUP (region, country);
```

For the grand-total row, both `region_is_subtotal` and `country_is_subtotal` are `1`. For a per-region subtotal row, `region_is_subtotal` is `0` and `country_is_subtotal` is `1`.

```sql
-- Filter to only the subtotal rows (not the detail rows)
SELECT region, SUM(revenue)
FROM sales
GROUP BY ROLLUP (region, country)
HAVING GROUPING(country) = 1;
```

## FILTER clause with aggregates

All aggregate functions support a `FILTER (WHERE ...)` clause, which is especially useful inside grouping sets to compute conditional subtotals alongside unconditional ones:

```sql
SELECT region,
    SUM(revenue) AS total,
    SUM(revenue) FILTER (WHERE channel = 'online') AS online_total
FROM sales
GROUP BY ROLLUP (region);
```

## Notes

- `NULL` in a grouping column has its usual meaning (missing data). Use `GROUPING()` to distinguish data nulls from rollup placeholders.
- The result set from `ROLLUP(a, b, c)` has at most N+1 rows per distinct value combination, where N is the number of rollup columns.
- `CUBE` with many columns produces exponentially many grouping sets. For N > 5 columns the result set and memory usage can be large.
- Only one `ROLLUP`/`CUBE`/`GROUPING SETS` expression per `GROUP BY` clause is supported. Mixing two such expressions in the same query is rejected at planning time.


---
# EXPLAIN (docs/sql/explain)

---
title: EXPLAIN
description: View query execution plans and introspect session state.
---

# EXPLAIN

## Query Plan

```sql
EXPLAIN SELECT * FROM users WHERE age > 30;
```

Shows the logical and physical query plan. Useful for understanding how the query planner routes to engines and which indexes are used.

## Cross-shard dispatch

For writes that span multiple vShards, `EXPLAIN` reports whether the query routes through the Calvin sequencer:

```sql
EXPLAIN INSERT INTO orders (id, region, total) VALUES ('o1', 'eu', 99.99);
-- Calvin static dispatch | vshards: [3, 7] | mode: strict | epoch: <assigned by sequencer at admission> | position: <assigned by sequencer>

EXPLAIN INSERT INTO local_cache (id, val) VALUES ('k1', 'v');
-- (single-shard plans emit no preamble — the plan rows that follow describe the local execution)
```

The cross-shard preamble appears only for sequenced multi-vShard writes. The `epoch` and `position` fields show placeholder text at `EXPLAIN` time — they are assigned by the sequencer when the transaction is actually submitted. Other dispatch modes (`Calvin dependent-read dispatch (OLLP)`, `Best-effort multi-shard dispatch [NON-ATOMIC]`) replace the leading label.

## Session Variables

```sql
SET nodedb.consistency = 'eventual';
SHOW nodedb.consistency;
SHOW ALL;
RESET nodedb.consistency;
```

## Change Tracking

```sql
SHOW CHANGES FOR users SINCE '2025-01-01' LIMIT 100;
```

## Introspection Commands

```sql
SHOW COLLECTIONS;
DESCRIBE users;
SHOW INDEXES;
SHOW TRIGGERS;
SHOW FUNCTIONS;
SHOW PROCEDURES;
SHOW CHANGE STREAMS;
SHOW TOPICS;
SHOW SCHEDULES;
SHOW CONTINUOUS AGGREGATES;
SHOW MATERIALIZED VIEWS;
SHOW CONNECTIONS;
SHOW USERS;
SHOW CLUSTER;
SHOW NODES;
SHOW RAFT GROUPS;
SHOW AUDIT LOG LIMIT 100;
```

## Limitations

`EXPLAIN ANALYZE` with actual per-operator runtime stats is not yet supported — it requires instrumentation across the SPSC bridge to collect per-core execution stats from the Data Plane. The cross-shard preamble (epoch, position, participating vshards) does appear for sequenced queries.


---
# LIVE SELECT (docs/real-time/live-select)

---
title: LIVE SELECT
description: Register a query and receive matching changes in real time. No polling.
---

# LIVE SELECT

Register a query and receive matching inserts, updates, and deletes as they happen.

```sql
LIVE SELECT * FROM orders WHERE total > 100.00;
LIVE SELECT id, status FROM orders WHERE status != 'pending';
```

## Protocol Support

- **pgwire** — Delivered as `NotificationResponse` messages. Works in `psql` and JDBC.
- **WebSocket** — JSON frames on `/ws`
- **NDB** — MessagePack frames on native connection

## Cancellation

```sql
CANCEL LIVE SELECT <subscription_id>;
```

The subscription remains active until cancelled or the session ends.

## vs Change Streams

LIVE SELECT is session-scoped and push-based. For durable, multi-consumer event streaming, use [Change Streams](/docs/real-time/change-streams).


---
# Change Streams (CDC) (docs/real-time/change-streams)

---
title: Change Streams (CDC)
description: Durable, cursor-tracked mutation streams with consumer groups and external delivery.
---

# Change Streams (CDC)

Change streams provide durable access to the mutation log. Unlike LIVE SELECT, they survive reconnects, support consumer groups, and deliver to external systems.

```sql
CREATE CHANGE STREAM order_events ON orders;

-- With webhook delivery
CREATE CHANGE STREAM order_events ON orders
WITH (URL = 'https://hooks.example.com/orders');

-- With log compaction (keep only latest per key)
CREATE CHANGE STREAM user_state ON users WITH (COMPACTION = 'key', KEY = 'id');

DROP CHANGE STREAM order_events;
-- Drop atomically tears down the stream's consumer groups and persisted offset
-- rows. Recreating a stream with the same name starts from the head; it does
-- not resume at stale offsets.
SHOW CHANGE STREAMS;
```

## Pull-Based Consumption

```sql
SHOW CHANGES FOR orders SINCE '2024-01-15T00:00:00Z' LIMIT 1000;
```

## External Delivery

**Webhook** — HTTP POST with retry, idempotency headers, and HMAC signing.

**Kafka bridge** — Transactional exactly-once publishing (feature-gated `--features kafka`):

```sql
CREATE CHANGE STREAM order_events ON orders
WITH (DELIVERY = 'kafka', BROKERS = 'localhost:9092', TOPIC = 'orders');
```

**SSE** — `GET /v1/streams/{stream}/events?group={group}` with `Accept: text/event-stream`.

**HTTP long-poll** — `GET /v1/streams/{stream}/poll?group={group}&limit=100`.

The poll response includes gap-detection fields alongside the events:

```json
{
  "events": [...],
  "evicted_since_last_poll": 0,
  "oldest_available_lsn": 19240
}
```

- `evicted_since_last_poll` — number of events dropped from the buffer since the previous poll call. A non-zero value means the consumer fell behind and the stream's ring buffer wrapped; events in the gap are permanently lost for this consumer.
- `oldest_available_lsn` — the lowest LSN still present in the stream buffer. Consumers can compare this against their last-seen LSN to detect gaps without waiting for the next event to arrive.

The `nodedb_cdc_events_dropped_total{tenant,stream}` counter tracks drops per named stream (not globally). Alert on this increasing for a stream whose consumer is active.

## Streaming Materialized Views

```sql
CREATE MATERIALIZED VIEW order_stats STREAMING AS
SELECT time_bucket('5 minutes', event_time) AS bucket, count(*), sum(total)
FROM order_changes WHERE event_type = 'INSERT' GROUP BY bucket;
```

O(1) per event. Supports COUNT, SUM, MIN, MAX, AVG. Watermark-driven finalization.


---
# Consumer Groups (docs/real-time/consumer-groups)

---
title: Consumer Groups
description: Independent offset tracking for multiple consumers on the same stream or topic.
---

# Consumer Groups

Consumer groups track read positions independently, enabling multiple consumers to process the same stream at their own pace.

```sql
-- Create consumer groups
CREATE CONSUMER GROUP analytics ON order_changes;
CREATE CONSUMER GROUP billing ON order_changes;

-- Commit offset for a specific partition
COMMIT OFFSET PARTITION 0 AT 42 ON order_changes CONSUMER GROUP analytics;

-- Batch commit all partitions
COMMIT OFFSETS ON order_changes CONSUMER GROUP analytics;

DROP CONSUMER GROUP analytics ON order_changes;
```

## On Topics

Consumer groups also work with durable topics:

```sql
CREATE CONSUMER GROUP processors ON order_events;
SELECT * FROM TOPIC order_events CONSUMER GROUP processors LIMIT 100;
COMMIT OFFSETS ON order_events CONSUMER GROUP processors;
```

Consumers resume from their last committed offset after disconnect.


---
# Webhooks (docs/real-time/webhooks)

---
title: Webhooks
description: HTTP POST delivery for change stream events with retry and HMAC signing.
---

# Webhooks

Change streams can deliver events to external HTTP endpoints:

```sql
CREATE CHANGE STREAM order_events ON orders
WITH (
    DELIVERY = 'webhook',
    URL = 'https://hooks.example.com/orders',
    RETRY = 3,
    TIMEOUT = '5s'
);
```

## Headers

Each POST includes:

| Header                | Description                    |
| --------------------- | ------------------------------ |
| `X-Idempotency-Key`  | Unique per delivery attempt    |
| `X-Event-Sequence`    | Monotonic sequence number      |
| `X-Partition`          | Source partition               |
| `X-LSN`                | WAL log sequence number        |

## Retry Behavior

Exponential backoff. 4xx client errors (except 429) are not retried. 5xx and timeouts retry up to the configured limit. Persistent failures go to the dead-letter queue.


---
# Cron Scheduler (docs/real-time/cron-scheduler)

---
title: Cron Scheduler
description: Distributed cron scheduler for periodic SQL jobs with leader-aware execution.
---

# Cron Scheduler

The Event Plane includes a distributed cron scheduler. Jobs are evaluated per-second and dispatched through the Control Plane → Data Plane path.

```sql
CREATE SCHEDULE nightly_cleanup
CRON '0 2 * * *'
AS BEGIN
    DELETE FROM sessions WHERE expires_at < now();
    INSERT INTO maintenance_log { task: 'nightly_cleanup', ran_at: now() };
END;

CREATE SCHEDULE refresh_stats
CRON '*/5 * * * *'
AS BEGIN
    REFRESH CONTINUOUS AGGREGATE order_stats;
END;

DROP SCHEDULE nightly_cleanup;
SHOW SCHEDULES;
```

## Cron Syntax

Standard 5-field cron: `minute hour day-of-month month day-of-week`.

| Expression      | Meaning                     |
| --------------- | --------------------------- |
| `0 2 * * *`     | 2:00 AM UTC daily           |
| `*/5 * * * *`   | Every 5 minutes             |
| `0 0 * * 0`     | Midnight Sunday             |
| `0 */6 * * *`   | Every 6 hours               |

## Leader-Aware

In clustered mode, scheduled jobs run on the collection's shard leader. If a leader changes due to failover, the new leader picks up the schedule. Job history and missed execution policies are tracked in redb.

## LISTEN/NOTIFY

PostgreSQL-compatible ephemeral notifications (session-scoped, cluster-wide):

```sql
LISTEN order_events;
NOTIFY order_events, 'order 123 shipped';
```

For durable delivery, use [Change Streams](/docs/real-time/change-streams) or [Durable Topics](/docs/sql/ddl).


---
# pgwire (PostgreSQL Protocol) (docs/connectivity/pgwire)

---
title: pgwire (PostgreSQL Protocol)
description: Standard PostgreSQL wire protocol. Any Postgres-compatible tool works with NodeDB.
---

# pgwire

NodeDB speaks the PostgreSQL wire protocol on port 6432. Any tool that speaks Postgres works.

```bash
psql -h localhost -p 6432
```

## Supported Features

- Simple Query protocol
- Extended Query protocol (prepared statements: Parse, Bind, Describe, Execute)
- `COPY FROM` (bulk import)
- `LISTEN/NOTIFY` (cluster-wide)
- SCRAM-SHA-256 authentication
- TLS
- Session variables
- Server-side cursors (SCROLL, BACKWARD, MOVE, WITH HOLD)
- Temporary tables (session-scoped)

## Introspection & Administration

### SHOW Commands

```sql
SHOW ROLES;
SHOW STATS;                        -- alias: SHOW SERVER STATS
SHOW METRICS;
SHOW MEMORY;
SHOW TENANTS WITH NAME <name>;
```

`SHOW STATS` returns live query latencies, throughput, and resource usage. `SHOW METRICS` exports Prometheus-format metrics. `SHOW MEMORY` shows per-engine memory allocation and jemalloc stats.

### SET Commands

Session-level configuration:

```sql
SET TENANT = '<name>' | <id> | DEFAULT;  -- superuser only — switch session tenant
```

### pg_catalog Compatibility

NodeDB exposes virtual `pg_catalog` tables for Postgres-client compatibility, BI tools, and ORM introspection:

- `pg_class` — Collections and materialized views
- `pg_namespace` — Databases and schemas
- `pg_attribute` — Collection columns and types
- `pg_index` — Index metadata
- `pg_type` — Data types

Tools like DBeaver, Tableau, and SQLAlchemy automatically query these tables. No special setup required.

## Compatible Clients

Any PostgreSQL client library: libpq, JDBC, psycopg2, node-postgres, SQLAlchemy, Prisma, Diesel, tokio-postgres, and more.

## GUI Tools

DBeaver and pgAdmin connect directly. Configure as a PostgreSQL connection to `localhost:6432`.


---
# HTTP API (docs/connectivity/http-api)

---
title: HTTP API
description: REST API for SQL execution, streaming, health checks, and Prometheus metrics.
---

# HTTP API

REST API on port 6480 for web clients and services.

## Endpoints

### Execute SQL

```bash
curl -X POST http://localhost:6480/v1/query \
  -H "Authorization: Bearer ndb_..." \
  -H "Content-Type: application/json" \
  -H "Accept: application/vnd.nodedb.v1+json" \
  -d '{"sql": "SELECT * FROM users LIMIT 10"}'
```

### Stream Results (NDJSON)

```bash
curl -X POST http://localhost:6480/v1/query/stream \
  -d '{"sql": "SELECT * FROM large_table"}'
```

### Health & Readiness

```bash
curl http://localhost:6480/healthz        # k8s readiness — 503 until startup completes
curl http://localhost:6480/health/live    # liveness probe
curl http://localhost:6480/health/ready   # WAL recovered, ready for queries
```

### Prometheus Metrics

```bash
curl http://localhost:6480/metrics
```

70+ system metrics: per-engine, per-core, connection, query, replication, storage. Latency histogram with 13 buckets.

### PromQL

```
http://localhost:6480/v1/obsv/api
```

Full Prometheus query engine. Point Grafana at this URL as a Prometheus data source.

### CDC Streams

```
GET /v1/streams/{stream}/events?group={group}   # SSE
GET /v1/streams/{stream}/poll?group={group}      # Long-poll
```

### WebSocket

`/v1/ws` endpoint for JSON-RPC: SQL execution, LIVE SELECT delivery, session reconnect.

## Versioning

All non-probe routes are under the `/v1/` prefix. JSON responses carry `Content-Type: application/vnd.nodedb.v1+json; charset=utf-8`. Clients may opt into version negotiation by sending `Accept: application/vnd.nodedb.v1+json`; sending only an unsupported `application/vnd.nodedb.vN+json` returns `406 Not Acceptable`. Probe routes (`/healthz`, `/health/*`, `/metrics`) are unversioned and always reachable, including during startup.


---
# Native Protocol (NDB) (docs/connectivity/native-protocol)

---
title: Native Protocol (NDB)
description: Binary MessagePack protocol for the ndb CLI, Rust SDK, and FFI/WASM bindings.
---

# Native Protocol (NDB)

Binary MessagePack protocol on port 6433. Used by the `ndb` CLI, Rust SDK (`nodedb-client`), and FFI/WASM bindings.

## Two Modes

**SQL** — SQL text transported as a MessagePack message. Same parser and planner as pgwire.

**Native opcodes** — Typed messages that skip SQL parsing. Used by SDKs for hot-path operations:

```rust
// Native mode — typed, skip SQL parsing
let user = client.get("users", "u1").await?;
client.put("users", "u1", &doc).await?;
client.vector_search("articles", &query_vec, 10, None).await?;

// SQL mode — flexible, any query
let rows = client.sql("SELECT * FROM users WHERE age > 30").await?;
```

Both modes produce the same `PhysicalPlan` and execute identically.

## Connection

```bash
# ndb CLI
./target/release/ndb
./target/release/ndb --host localhost --port 6433
```

## Handshake

Every native connection performs a versioned handshake before any opcode frame. SDKs do this automatically on first use; you only need this section if you're implementing a wire-level client.

### `HelloFrame` (client → server)

| Field          | Width | Value                                   |
| -------------- | ----- | --------------------------------------- |
| `magic`        | 4 B   | `NDBH`                                  |
| `proto_min`    | u16   | Minimum protocol version client accepts |
| `proto_max`    | u16   | Maximum protocol version client supports |
| `capabilities` | u64   | Bitmask of optional features (send `0` for none; unknown bits ignored by server) |

### `HelloAckFrame` (server → client)

| Field            | Width                | Value                                                 |
| ---------------- | -------------------- | ----------------------------------------------------- |
| `magic`          | 4 B                  | `NDBA`                                                |
| `proto_version`  | u16                  | Negotiated version (`max(proto_min_client, proto_min_server) ≤ v ≤ min(proto_max_client, proto_max_server)`) |
| `capabilities`   | u64                  | Server-side capability bitmask                        |
| `server_version` | length-prefixed UTF-8 | Build identifier (e.g. `"NodeDB/0.3.0"`)             |
| `limits`         | `Limits` struct      | Per-op caps the server enforces (see below)           |

### `HelloErrorFrame`

Returned when no protocol version overlaps. Carries a typed `code` (`VersionMismatch`) and a UTF-8 reason. The connection is closed after the frame is sent.

### Server-enforced `Limits`

The `HelloAckFrame` carries the server's per-op caps. SDKs surface these via `client.limits()`. Sending a request that exceeds a cap returns a typed `LimitExceeded { limit_name, value, max }` error.

| Field                  | Type          | Caps                                            |
| ---------------------- | ------------- | ----------------------------------------------- |
| `max_vector_dim`       | `Option<u32>` | Vector embedding dimensionality                 |
| `max_top_k`            | `Option<u32>` | `top_k` for any retrieval op                    |
| `max_scan_limit`       | `Option<u32>` | Result set size for scans                       |
| `max_batch_size`       | `Option<u32>` | Rows per batch INSERT/UPSERT                    |
| `max_crdt_delta_bytes` | `Option<u32>` | Single CRDT delta payload                       |
| `max_query_text_bytes` | `Option<u32>` | SQL text length                                 |
| `max_graph_depth`      | `Option<u32>` | `MAX_DEPTH` for graph traversal                 |

`None` means uncapped. Defaults are uncapped — operators set caps via configuration.

## Capabilities

`Capabilities` is a typed wrapper around the `u64` bitmask returned in `HelloAckFrame`. SDK consumers query specific features via accessor methods rather than testing raw bits, so feature additions never break clients:

```rust
let caps = client.capabilities();
if caps.has_graphrag_fusion() { ... }
if caps.has_continuous_aggregates() { ... }
```

The `Capabilities::has(bit)` escape hatch is available for forward compatibility, but typed accessors are preferred.

## Server Identity

```rust
client.proto_version()   // u16 — negotiated protocol version
client.server_version()  // String — server build identifier
client.limits()          // &Limits — per-op caps
client.capabilities()    // Capabilities — typed feature flags
```


---
# RESP (Redis Protocol) (docs/connectivity/resp)

---
title: RESP (Redis Protocol)
description: Redis-compatible wire protocol for KV operations. Disabled by default.
---

# RESP (Redis Protocol)

NodeDB speaks RESP2 for KV operations. Existing Redis clients work out of the box.

## Enable

```toml
# nodedb.toml
[server.ports]
resp = 6381
```

Or: `NODEDB_PORT_RESP=6381`

## Usage

```bash
redis-cli -p 6381

SELECT sessions          # switch to a KV collection
SET sess_abc '{"user":"alice"}' EX 3600
GET sess_abc
DEL sess_abc
INCR page_views
ZADD leaderboard 1500 player-123
ZRANK leaderboard player-123
SUBSCRIBE sessions
```

## Supported Commands

`GET`, `SET` (EX/PX/NX/XX), `DEL`, `EXISTS`, `MGET`, `MSET`, `EXPIRE`, `PEXPIRE`, `TTL`, `PTTL`, `PERSIST`, `SCAN`, `KEYS`, `HGET`, `HMGET`, `HSET`, `FLUSHDB`, `DBSIZE`, `SUBSCRIBE`, `PUBLISH`, `PING`, `ECHO`, `SELECT`, `INFO`, `QUIT`, `INCR`, `DECR`, `INCRBY`, `DECRBY`, `INCRBYFLOAT`, `GETSET`, `ZADD`, `ZREM`, `ZRANK`, `ZRANGE`, `ZCARD`, `ZSCORE`.

## Same Data, Two Interfaces

RESP accesses the same KV data as SQL. A key set via `redis-cli` is queryable via `SELECT * FROM sessions WHERE key = 'sess_abc'`.


---
# ILP (InfluxDB Line Protocol) (docs/connectivity/ilp)

---
title: ILP (InfluxDB Line Protocol)
description: High-throughput timeseries ingest via InfluxDB Line Protocol. Disabled by default.
---

# ILP (InfluxDB Line Protocol)

NodeDB accepts metrics via InfluxDB Line Protocol over TCP for high-throughput timeseries ingest.

## Enable

```toml
[server.ports]
ilp = 8086
```

Or: `NODEDB_PORT_ILP=8086`

## Usage

```bash
echo "cpu,host=web-01,region=us-east usage=72.5,mem=84.2 1609459200000000000" | nc localhost 8086
```

## Telegraf Integration

```toml
# telegraf.conf
[[outputs.socket_writer]]
address = "tcp://localhost:8086"
data_format = "influx"
```

Any ILP-compatible client (Telegraf, Vector, InfluxDB client libraries) works.

## Adaptive Batching

NodeDB auto-tunes batch sizes based on ingest rate. Per-series core routing eliminates cross-core contention. No configuration needed.


---
# Sync Protocol (WebSocket) (docs/connectivity/sync-protocol)

---
title: Sync Protocol (WebSocket)
description: WebSocket-based CRDT sync for NodeDB-Lite clients on phones, browsers, and desktops.
---

# Sync Protocol

WebSocket-based sync on port 9090 for NodeDB-Lite clients. CRDT deltas flow between edge devices and Origin for all eight storage engines and their overlays.

## Supported Collections

All collection engines participate in sync:

- **Document** (schemaless + strict)
- **Key-Value**
- **Columnar, Timeseries, Spatial**
- **Vector** and **Array**
- **Graph** and **Full-Text Search** overlays on synced collections

Schema changes (new columns, new collections) are discovered automatically by connected Lite clients after the Origin catalog commit.

## How It Works

1. Lite client connects via WebSocket to Origin
2. Client sends its current LSN watermark
3. Origin sends any missed deltas since that watermark
4. Client sends locally accumulated CRDT deltas
5. Origin validates constraints (UNIQUE, FK, CHECK) via Raft
6. Committed deltas are broadcast to all connected clients with matching shape subscriptions

## Replica Identity

Every CRDT operation is attributed to the **peer id** of the replica that made it. A peer id must belong to exactly one replica of one collection.

Two replicas that share a peer id number their operations from the same counter range. The merge cannot tell those apart from a client re-sending history it has already delivered, so it discards whichever arrives second — the writes are gone, and to the client they looked accepted.

Origin therefore binds each peer id to the first producer that uses it in a collection, and refuses any other producer:

```
DeltaReject: PEER_ID_COLLISION: peer id 1 on collection 'notes' is already
owned by another replica; generate a new peer id and resync
```

To recover, generate a fresh peer id for the replica and resync. Applications should:

- Derive a peer id that is unique per **installation**, not per user or per device model.
- Generate a **new** peer id whenever the local store is reset, reinstalled, or restored from a backup taken by another replica. A replica that keeps its old peer id after wiping its store collides with its own earlier history, which Origin cannot refuse — it owns that peer id — and the writes are reported as duplicates rather than applied.

The same peer id in two different collections is not a collision; each collection is a separate document whose counter ranges never meet.

### Observing discarded operations

Each sync session's close line reports what the merge absorbed:

```
sync: session closed session=... applied=40 rejected=0 deduplicated=10 ops_trimmed=120
```

`ops_trimmed` counts operations that arrived already known to Origin. A healthy resync re-sends a prefix and then advances, so `applied` rises alongside it. A session whose deltas are all deduplicated with nothing applied is the peer-id collision shape.

## Shape Subscriptions

Devices subscribe to a subset of data via shape subscriptions (wire-level sync feature, not SQL):

```
Shape subscription: users WHERE user_id = $me
```

The client only receives data matching this filter. Changes within a device's shape are pushed in real time. Changes outside the shape are not sent.

## Compensation Hints

If a local write violates a constraint on Origin, a typed `CompensationHint` is sent back:

- `UniqueViolation` — duplicate key detected
- `ForeignKeyMissing` — FK target doesn't exist
- `SchemaViolation` — CHECK constraint or type rule failed
- `IntegrityViolation` — other integrity-rule violation
- `PermissionDenied` — insufficient privilege
- `RateLimited` — quota or rate limit exceeded
- `Custom` — application-specific handling, including `peer_id_collision` (see [Replica Identity](#replica-identity))

The application handles the conflict — no silent data loss.


---
# Rust Client (docs/connectivity/rust-client)

---
title: Rust Client
description: The nodedb-client crate connects over the NDB protocol with both SQL and native modes.
---

# Rust Client

The `nodedb-client` crate connects over the NDB protocol (port 6433). It supports both SQL and native modes on the same connection.

## Installation

```toml
[dependencies]
nodedb-client = "0.1"
```

## Usage

```rust
use nodedb_client::NodeDbRemote;

let client = NodeDbRemote::connect("localhost:6433").await?;

// SQL mode
let rows = client.sql("SELECT * FROM users WHERE age > 30").await?;

// Native mode (skip SQL parsing)
let user = client.get("users", "u1").await?;
client.put("users", "u1", &doc).await?;
client.vector_search("articles", &query_vec, 10, None).await?;
```

## The NodeDb Trait

Both `NodeDbRemote` (network client) and `NodeDbLite` (embedded) implement the `NodeDb` trait. Application code can be generic over the trait — same logic works in-process or over the network.

```rust
async fn search<D: NodeDb>(db: &D, query: &[f32]) -> Result<Vec<Row>> {
    db.vector_search("articles", query, 10, None).await
}
```


---
# Python Driver (docs/connectivity/python-driver)

---
title: Python Driver
description: Connect to NodeDB from Python using any PostgreSQL driver.
---

# Python Driver

NodeDB speaks PostgreSQL's wire protocol, so any Python PostgreSQL driver works.

## psycopg2

```python
import psycopg2

conn = psycopg2.connect(host="localhost", port=6432, dbname="nodedb")
cur = conn.cursor()

cur.execute("SELECT * FROM users WHERE age > %s", (30,))
rows = cur.fetchall()

conn.close()
```

## asyncpg

```python
import asyncpg

conn = await asyncpg.connect("postgresql://localhost:6432/nodedb")
rows = await conn.fetch("SELECT * FROM users WHERE age > $1", 30)
await conn.close()
```

## SQLAlchemy

```python
from sqlalchemy import create_engine

engine = create_engine("postgresql://localhost:6432/nodedb")
with engine.connect() as conn:
    result = conn.execute("SELECT * FROM users LIMIT 10")
```

## ORMs

Django, SQLAlchemy ORM, Prisma, and other PostgreSQL-compatible ORMs work with NodeDB's pgwire protocol.


---
# Ecosystem (psql, ORMs, GUI Tools) (docs/connectivity/ecosystem)

---
title: Ecosystem (psql, ORMs, GUI Tools)
description: Compatible tools, ORMs, connection poolers, and GUI clients.
---

# Ecosystem

NodeDB's pgwire compatibility means most PostgreSQL tools work out of the box.

## psql

```bash
psql -h localhost -p 6432
```

Full support for `\d`, `\dt`, prepared statements, COPY, and LISTEN/NOTIFY.

## GUI Tools

**DBeaver** — Connect as PostgreSQL to `localhost:6432`. Schema browser, query editor, and result export work.

**pgAdmin** — Connect as PostgreSQL. Dashboard and query tool work. Some admin features may not apply.

## ORMs

| Framework    | Connection string                              |
| ------------ | ---------------------------------------------- |
| SQLAlchemy   | `postgresql://localhost:6432/nodedb`            |
| Django       | `HOST=localhost PORT=6432 ENGINE=django.db.backends.postgresql` |
| Prisma       | `postgresql://localhost:6432/nodedb`            |
| Diesel       | `postgres://localhost:6432/nodedb`              |
| GORM (Go)    | `host=localhost port=6432 dbname=nodedb`        |

## Connection Poolers

PgBouncer and pgcat work with NodeDB. Configure them to connect to `localhost:6432`. Use transaction-level pooling for best results with NodeDB's session variable support.


---
# Configuration (docs/administration/configuration)

---
title: Configuration
description: Server configuration via nodedb.toml and environment variables.
---

# Configuration

NodeDB is configured via `nodedb.toml`. Environment variables take precedence.

```toml
[server]
host = "127.0.0.1"
data_dir = "/var/lib/nodedb"
memory_limit = "4GiB"
data_plane_cores = 4
max_connections = 1024
log_format = "text"

[server.ports]
native = 6433
pgwire = 6432
http = 6480
resp = 6381          # optional
ilp = 8086           # optional

[server.tls]
cert_path = "/etc/nodedb/tls/server.crt"
key_path = "/etc/nodedb/tls/server.key"
```

## Environment Variables

| Config field       | Env var                   | Default      |
| ------------------ | ------------------------- | ------------ |
| `host`             | `NODEDB_HOST`             | `127.0.0.1`  |
| `ports.pgwire`     | `NODEDB_PORT_PGWIRE`      | `6432`       |
| `ports.native`     | `NODEDB_PORT_NATIVE`      | `6433`       |
| `ports.http`       | `NODEDB_PORT_HTTP`        | `6480`       |
| `ports.resp`       | `NODEDB_PORT_RESP`        | disabled     |
| `ports.ilp`        | `NODEDB_PORT_ILP`         | disabled     |
| `data_dir`         | `NODEDB_DATA_DIR`         | `~/.local/share/nodedb` |
| `memory_limit`     | `NODEDB_MEMORY_LIMIT`     | `1GiB`       |
| `data_plane_cores` | `NODEDB_DATA_PLANE_CORES` | CPUs - 1     |
| `max_connections`  | `NODEDB_MAX_CONNECTIONS`  | `4096`       |
| `log_format`       | `NODEDB_LOG_FORMAT`       | `text`       |

## Audit Configuration

```toml
[audit]
level = "standard"   # minimal, standard, full, forensic
```

## Auth Configuration

```toml
[auth.jwt]
providers = [
    { issuer = "https://your-domain.auth0.com/", audience = "your-api" },
]
```

## TLS per Protocol

```toml
[server.tls]
native = true
pgwire = true
http = true
resp = true
ilp = false
```


---
# Databases (docs/administration/databases)

---
title: Databases
description: The Database primitive — top-level container above tenants. Owns the collection namespace and quota budget.
---

# Databases

A **Database** is a top-level container that owns the collection namespace and quota budget. It is the unit of clone, mirror, and backup. One database hosts multiple tenants; a tenant's data does not span databases.

## What is a Database?

A Database is a deployment unit — typically one per app, environment, or region. It provides:

- **Collection namespace** — all collection names are scoped per database. The same collection name can exist in different databases with no conflict.
- **Quota parent** — the database owns a resource budget (storage, memory, connections). Tenants inherit from the database quota.
- **Durability boundary** — backups, clones, and mirrors operate at the database level.
- **Access control** — database-level roles (`DatabaseOwner`, `DatabaseEditor`, `DatabaseReader`) determine who can manage collections within it.

For scaling across regions or tenants, use one database per environment and separate tenants within it. Do not create one database per customer — use [Multi-Tenancy](multi-tenancy) instead.

## The Default Database

NodeDB reserves `DatabaseId(0)` for the `default` database. This database:

- Always exists and cannot be dropped
- Serves as the fallback if no database is specified
- Is the migration target for legacy single-database deployments
- Has the durable identity `DatabaseId(0)` even if renamed

## Creating a Database

```sql
CREATE DATABASE emp_prod;
CREATE DATABASE staging WITH (quota_storage_bytes = 107374182400);
```

The `quota_storage_bytes` option sets the database-level storage quota (default: unlimited). See [Quotas](#quotas) for all available quota options.

## Dropping a Database

```sql
DROP DATABASE staging;
```

By default, `DROP DATABASE` fails if the database contains collections. Use `CASCADE` to drop with all collections:

```sql
DROP DATABASE staging CASCADE;
```

Use `FORCE` to drop and automatically materialize any dependent clones (see [Cloning Databases](database-clone)) before removal:

```sql
DROP DATABASE staging FORCE;
```

The default database cannot be dropped.

## Renaming and Altering

Rename a database:

```sql
ALTER DATABASE staging RENAME TO emp_staging;
```

The durable identity is `DatabaseId`, not the name. Renaming is a catalog-only operation (fast, non-disruptive).

Update quotas:

```sql
ALTER DATABASE emp_prod SET QUOTA quota_storage_bytes = 214748364800;
```

## Listing Databases

```sql
SHOW DATABASES;
```

Returns:

| Column             | Type      | Description                          |
| ------------------ | --------- | ------------------------------------ |
| `name`             | `text`    | Database name                        |
| `database_id`      | `integer` | Durable unique identifier            |
| `status`           | `text`    | `Active` or `Degraded`               |
| `created_at`       | `timestamp` | Creation timestamp                  |
| `collection_count` | `integer` | Number of collections in database    |
| `tenant_count`     | `integer` | Number of tenants using database     |
| `parent_clone`     | `text`    | Source database name if cloned; null otherwise |
| `quota_bytes`      | `integer` | Storage quota in bytes               |

## Connecting to a Database

### pgwire (PostgreSQL Protocol)

Specify the database in the connection string:

```bash
psql -h localhost -p 6432 -d emp_prod -U alice
```

Or switch mid-session:

```bash
psql> \c emp_prod
```

Switching databases aborts any open transaction, invalidates prepared statements, and re-binds the session.

### HTTP

Use the `X-NodeDB-Database` header (preferred):

```bash
curl -H "X-NodeDB-Database: emp_prod" http://localhost:6480/v1/query
```

Or query parameter fallback:

```bash
curl http://localhost:6480/v1/query?database=emp_prod
```

### Native Client

```rust
use nodedb_client::ConnectionBuilder;

let conn = ConnectionBuilder::new("localhost:6432")
    .database("emp_prod")
    .user("alice")
    .password("secret")
    .connect()
    .await?;
```

## Session Resolution Chain

When a new session connects, the database is resolved in this order:

1. **Explicit** — connection string / startup message `database` parameter
2. **User default** — `DEFAULT DATABASE` set for the user (see [User Defaults](#user-defaults))
3. **Tenant default** — configured default for the tenant (if using [Multi-Tenancy](multi-tenancy))
4. **Fallback** — `default` database

The first match is used. Once set, the session's database is immutable until switched via `\c` / `USE DATABASE`.

## User Defaults

Set a default database for a user:

```sql
ALTER USER alice SET DEFAULT DATABASE emp_prod;
```

When `alice` connects without specifying a database, she is routed to `emp_prod`.

## Database Roles and Grants

Three database-level roles control who can access and manage a database:

- **`DatabaseOwner`** — full control: create/alter/drop collections, grant permissions
- **`DatabaseEditor`** — read/write collections, cannot alter schema or grants
- **`DatabaseReader`** — read-only access

Grant a role to a user:

```sql
GRANT DATABASE_OWNER ON DATABASE emp_prod TO alice;
GRANT DATABASE_READER ON DATABASE emp_prod TO bob;
```

Revoke with:

```sql
REVOKE DATABASE_OWNER ON DATABASE emp_prod FROM alice;
```

See [RBAC](rbac) for the full privilege matrix.

## Cross-Database Queries Are Forbidden

A collection in a different database returns `COLLECTION_NOT_FOUND` — identical to querying a collection that does not exist anywhere. This design prevents accidental cross-database leaks.

To query data across databases, open two separate connections:

```rust
let conn_a = ConnectionBuilder::new("localhost:6432").database("db_a").connect().await?;
let conn_b = ConnectionBuilder::new("localhost:6432").database("db_b").connect().await?;

let rows_a = conn_a.query("SELECT * FROM collections_table").await?;
let rows_b = conn_b.query("SELECT * FROM collections_table").await?;

// Merge in application code
```

## Per-Database Metrics

Prometheus metrics are exposed per database with labels:

```
nodedb_database_collection_count{database="emp_prod"} 42
nodedb_database_storage_bytes{database="emp_prod"} 1099511627776
nodedb_database_memory_bytes{database="emp_prod"} 536870912
nodedb_database_connections_active{database="emp_prod"} 8
nodedb_database_qps{database="emp_prod"} 1500
nodedb_database_errors_total{database="emp_prod",code="UNAUTHORIZED"} 5
```

Use these for dashboards, alerting, and capacity planning. See [Monitoring](monitoring) for integration.

## Common Errors

| Error                            | Cause                               | Solution                           |
| -------------------------------- | ----------------------------------- | --------------------------------- |
| `DATABASE_NOT_FOUND`             | Database does not exist             | Create with `CREATE DATABASE`     |
| `CANNOT_DROP_DEFAULT_DATABASE`   | Attempted to drop `default`         | Use a different database name      |
| `COLLECTION_NOT_FOUND`           | Collection exists in different DB   | Switch to correct database         |
| `ACCESS_DENIED`                  | User lacks privilege on database    | `GRANT DATABASE_READER` or higher |
| `CLONE_DEPENDENCY`               | Cannot drop DB with dependent clone | Use `DROP DATABASE … FORCE`       |


---
# Cloning Databases (docs/administration/database-clone)

---
title: Cloning Databases
description: Copy-on-write CLONE DATABASE for branching, point-in-time staging, and forensic snapshots.
---

# Cloning Databases

A **clone** is a copy-on-write (CoW) database created at a point-in-time snapshot of a source. Reads delegate to the source; writes go to the clone. Returns in milliseconds regardless of source size.

## What is a Clone?

When you clone a database, NodeDB:

1. Records the source database and a point-in-time LSN (Log Sequence Number)
2. Creates a new database with the same collections
3. Copies only the catalog metadata — zero storage copying
4. Routes reads to the source for data not yet written in the clone
5. Copies rows to the clone on first write (copy-on-write)

This means a clone of a 100 GB database returns in milliseconds and initially uses negligible storage.

## Creating a Clone

Clone at the latest commit:

```sql
CLONE DATABASE staging FROM prod;
```

Clone at a specific point in time:

```sql
CLONE DATABASE prod_yesterday FROM prod AS OF SYSTEM TIME 1730000000000;
```

The timestamp is in milliseconds since epoch. NodeDB resolves it to the nearest LSN and captures that snapshot. Reads on `prod_yesterday` see `prod`'s state at that moment.

## Read-Path Delegation

Until a row is written in the clone, reads are served from the source at the chosen LSN:

```
1. Query the clone for a row
2. If the row exists in the clone → return it
3. If the row was explicitly deleted in the clone → return not-found
4. If the row exists only in the source → return it (delegated read)
5. If the row doesn't exist anywhere → return not-found
```

This delegation is transparent — your query is unaware it is reading from the source.

## Write-on-Clone (Copy-Up)

When you first modify a row that exists only in the source:

```sql
UPDATE staging.users SET status = 'active' WHERE id = 100;
```

If row 100 exists only in the source:

1. The row is **copied up** from the source to the clone
2. The UPDATE is applied
3. The copy is durably recorded
4. Subsequent reads see the clone's copy

Delete on a source-only row inserts a **tombstone** — the row is invisible to subsequent reads without being physically removed.

All writes are durable through the WAL and Raft replication.

## Bitemporal Correctness

If the source is [bitemporal](bitemporal), the clone preserves time-travel semantics:

```sql
SELECT * FROM staging.events AS OF SYSTEM TIME 1729000000000 WHERE event_id = 42;
```

- If query time ≤ clone's LSN → read from source at that time
- If query time > clone's LSN → read from clone (clone did not exist before its creation)
- If query time < clone creation → empty result with metadata note

This is why cloning from a point-in-time staging database works: you see the exact historical state at clone time, then your own edits afterward.

## Materializing a Clone

Background materialization gradually copies all rows from source to clone, freeing the clone from source dependency:

```sql
ALTER DATABASE staging MATERIALIZE;
```

This blocks until all rows are copied. Useful when you want to:

- Drop the source database
- Stop relying on source's read-path performance
- Create an independent snapshot for long-term archival

Before materializing completes, the clone remains usable — reads continue delegating if needed.

## Viewing Lineage

See the ancestor chain of a clone:

```sql
SHOW DATABASE LINEAGE FOR staging;
```

Returns:

```
 source_database | created_at_ms | as_of_ms | status
-----------------+---------------+----------+----------
 prod            | 1730000000000 | null     | Shadowed
```

For nested clones (clone of a clone):

```
 source_database | created_at_ms | as_of_ms | status
-----------------+---------------+----------+----------
 prod            | 1730000000000 | null     | Shadowed
 staging         | 1730086400000 | null     | Shadowed
```

## Clone Depth Limits

Clone depth is limited to 8 levels to prevent query-path explosion. Attempting to clone from a clone-8 returns `CLONE_DEPTH_EXCEEDED`.

To exceed the limit, materialize the source clone first:

```sql
ALTER DATABASE staging MATERIALIZE;
CLONE DATABASE stage2 FROM staging;  -- now staging is Materialized, so this succeeds
```

## Restrictions and Errors

| Restriction              | Solution                             |
| ----------------------- | ------------------------------------ |
| Cannot clone a mirror    | Promote the mirror first; then clone |
| Clone depth > 8          | Materialize source; retry            |
| Source dropped with dependents | Use `DROP DATABASE … FORCE` |

## Practical Examples

### Point-in-Time Staging

Clone production at 1 AM daily for staging/QA tests:

```sql
-- 1 AM: Create daily snapshot
CLONE DATABASE qa_daily_$(date +%Y%m%d) FROM prod AS OF SYSTEM TIME (EXTRACT(EPOCH FROM '2026-05-10 01:00:00'::timestamp) * 1000)::bigint;

-- QA team tests against the snapshot
-- After testing: DROP or materialize for archival
```

### Forensic Snapshots

Clone at the time of a suspected incident:

```sql
-- Incident at 1730000000000 (ms)
CLONE DATABASE incident_snapshot FROM prod AS OF SYSTEM TIME 1730000000000;

-- Investigate without modifying production
SELECT * FROM incident_snapshot.events WHERE status = 'failed';
```

### Blue-Green Deployment

Create a staging clone, apply schema changes, validate, then promote:

```sql
CLONE DATABASE blue FROM prod;
ALTER TABLE blue.users ADD COLUMN new_field INT;
-- Run tests on blue
-- Once validated, swap DNS / connection strings
```

## Composition with Other Features

**Quotas:** A clone inherits the source's quota settings. Adjust independently:

```sql
ALTER DATABASE staging SET QUOTA quota_storage_bytes = 53687091200;
```

**Mirroring:** A clone can be mirrored (preview-environment DR). A mirror cannot be cloned — promote first.

**Multi-engine:** Clones work across all engines (vector, graph, columnar, etc.) transparently.

## Performance Notes

- **Clone creation**: O(catalog size), typically < 100 ms even for TB sources
- **Materialization**: Background, respects database maintenance budget. Small rows (< 1 KB) typically materialize at 100K–500K rows/sec
- **Read overhead while Shadowed**: Negligible — clone check is a fast hashtable lookup
- **Write overhead while Shadowed**: Single copy-up latency per source row; subsequent writes unaffected


---
# Mirroring Databases (docs/administration/database-mirror)

---
title: Mirroring Databases
description: MIRROR DATABASE for cross-region read replicas via Raft observer. Promote one-way for DR.
---

# Mirroring Databases

A **mirror** is a continuously-updated read-only copy of a database in a different cluster. It replicates the source's Raft log and applies all writes under its own storage layer. Promotion converts a mirror to a writable, independent database.

## What is a Mirror?

Mirrors are useful for:

- **Read replicas** in a distant region (faster reads for users far from origin)
- **Disaster recovery** — promote if origin is lost (one-way, permanent)
- **Data locality** — copy entire database to compliance-required regions

Unlike [clones](database-clone) (which delegate reads to source), mirrors apply all writes locally. Unlike active-active replication, mirrors are read-only until promoted.

## Creating a Mirror

```sql
MIRROR DATABASE replica FROM prod MODE = async;
```

### Modes

**`async` (default):**
- Mirror trails source; lag is observable
- Recommended for cross-region use (no latency penalty on source)
- Lag threshold for `Degraded` status: 5 seconds (configurable)

**`sync`:**
- Source waits for mirror ACK before committing writes
- Not recommended cross-region (adds source write latency)
- Lag threshold: 100 ms
- Use only for same-cluster high-availability mirrors

## Mirror Lifecycle

A mirror progresses through statuses:

```
Bootstrapping { bytes_done, bytes_total }
    ↓
Following
    ↓ (if lag > threshold)
Degraded { lag_ms }
    ↓ (if disconnected)
Disconnected
    ↓ (on promotion)
Promoted
```

### Bootstrap Phase

When you create a mirror:

1. Source creates a consistent snapshot at LSN = S₀
2. Snapshot streams to mirror cluster over QUIC (cross-cluster transport)
3. Mirror applies snapshot and sets `last_applied = S₀`
4. Mirror requests log entries from S₀ + 1 onward
5. Source streams and mirror applies
6. Status flips to `Following`

Bootstrap time depends on database size. Progress is tracked in `bytes_done` and `bytes_total`.

### Following Phase

Source continuously streams log entries to mirror. Writes apply with bounded lag. Status remains `Following` if lag < threshold.

### Degraded Phase

If mirror lag exceeds the threshold (5 s for async, 100 ms for sync), status becomes `Degraded { lag_ms }`. Writes still apply normally; degraded is informational.

Common causes:

- Network congestion
- Mirror apply bottleneck
- Source burst write rate

Monitor the `nodedb_database_mirror_lag_ms` metric for alert thresholds.

### Disconnected Phase

If the mirror loses connection to source, status becomes `Disconnected`. Reconnection is automatic with exponential backoff. Reads continue serving stale data.

## Read Consistency on a Mirror

Before promotion, mirrors are read-only. Three consistency levels are supported:

**`Strong` (leader read):**
```sql
SELECT * FROM replica.users WHERE id = 100 CONSISTENCY = 'strong';
```
Returns `STALE_READ_NOT_LEADER` with a hint to query the source instead.

**`BoundedStaleness(duration)` (default):**
```sql
SELECT * FROM replica.users WHERE id = 100 CONSISTENCY = 'bounded_staleness(5s)';
```
Served against `last_applied` if lag ≤ duration. Returns error if lag > duration.

**`Eventual`:**
```sql
SELECT * FROM replica.users WHERE id = 100 CONSISTENCY = 'eventual';
```
Served immediately from whatever state is locally available.

## Write Rejection Pre-Promotion

All writes to a non-promoted mirror return `MIRROR_READ_ONLY`:

```sql
INSERT INTO replica.users VALUES (...);
-- Error: MIRROR_READ_ONLY
```

This prevents accidental writes that would be lost on promotion.

## Promotion

Permanently convert a mirror to a writable, independent database:

```sql
ALTER DATABASE replica PROMOTE;
```

### What Happens

1. Mirror stops observing source
2. Mirror becomes a normal Raft group with its own leader election
3. Mirror accepts writes
4. `MirrorStatus` → `Promoted`
5. `mirror_origin` is retained for audit; no functional purpose

### One-Way and Permanent

Promotion is irreversible. There is no `DEMOTE` command. Once promoted:

- The former mirror is a standalone database
- Source is unaffected (not notified)
- To re-establish mirroring: `DROP` the promoted database and create a fresh clone or mirror

This design reflects typical DR scenarios where source is unreachable or lost.

## Mirror Status Inspection

```sql
SHOW DATABASE MIRROR STATUS FOR replica;
```

Returns:

| Column            | Type      | Description                         |
| ----------------- | --------- | ----------------------------------- |
| `mode`            | `text`    | `sync` or `async`                   |
| `status`          | `text`    | `Bootstrapping`, `Following`, `Degraded`, `Disconnected`, `Promoted` |
| `last_applied`    | `integer` | LSN most recently applied           |
| `lag_ms`          | `integer` | Estimated lag in milliseconds       |
| `source_cluster`  | `text`    | Cluster ID of source                |
| `source_database` | `text`    | Database name on source             |
| `bytes_done`      | `integer` | Bootstrap progress (if Bootstrapping) |
| `bytes_total`     | `integer` | Total size (if Bootstrapping)       |

## Lag Observability

Monitor mirror lag with Prometheus:

```
nodedb_database_mirror_lag_ms{database="replica"}
```

Set alerting thresholds based on your RTO/RPO requirements. For example:

- **Warning**: lag > 30 s
- **Critical**: lag > 300 s OR status = `Disconnected` for > 5 min

## Failure Modes and Recovery

| Failure                      | Behavior                                  | Recovery                                    |
| ---------------------------- | ----------------------------------------- | ------------------------------------------- |
| Network partition            | Status → `Disconnected`; reads stale      | Automatic reconnect with backoff            |
| Mirror apply too slow        | Backpressure throttles source send rate   | Automatic; monitor lag metric               |
| Source data loss             | Mirror is a replica, not a backup         | Use [Backup](backup-recovery) for data-loss recovery |
| Mirror data loss             | Drop mirror; re-mirror (re-runs bootstrap) | Recreate with `MIRROR DATABASE` again      |
| Mirror crash (before promote) | Restart resumes from persisted `last_applied` | Automatic; no data loss                    |
| Source unavailable indefinitely | Promote to make mirror independent       | `ALTER DATABASE … PROMOTE`                 |

## Composition with Other Features

**Cloning:** A clone CAN be mirrored (useful for preview-environment DR). A mirror CANNOT be cloned — promote first if you need a clone of the mirror.

**Quotas:** Mirror inherits source's quota. Adjust after bootstrap completes:

```sql
ALTER DATABASE replica SET QUOTA quota_storage_bytes = 107374182400;
```

**Multi-engine:** Mirrors work across all engines (vector, graph, columnar, etc.).

## Practical Examples

### Cross-Region Read Replica

Create async mirror on a distant cluster for local reads:

```sql
-- On us-west cluster (origin)
MIRROR DATABASE eu-replica FROM prod MODE = async;
-- Deployed to eu-west cluster

-- Clients in EU read from eu-replica (low latency)
-- Writes still go to us-west origin
```

Monitor lag to ensure SLA compliance:

```
nodedb_database_mirror_lag_ms{database="eu-replica"} < 5000  # 5 sec
```

### Disaster Recovery Standby

Create a mirror on standby hardware. On origin failure, promote:

```sql
-- Standby cluster
MIRROR DATABASE standby FROM prod MODE = async;

-- Origin lost? Promote standby to take over
ALTER DATABASE standby PROMOTE;

-- Update connection strings to point to former standby
-- (now an independent writable database)
```

### Pre-Deployment Validation

Clone the prod database, mirror the clone to a staging cluster, and test:

```sql
-- On prod cluster
CLONE DATABASE pre_deploy FROM prod;
MIRROR DATABASE staging_val FROM pre_deploy MODE = async;
-- Deploy staging_val to staging cluster

-- Validate schema / data
-- If OK: promote staging to serve traffic
-- If not: drop and retry
```

## Permissions

Both `MIRROR DATABASE` and `ALTER DATABASE … PROMOTE` require superuser privileges. See [RBAC](rbac) for the full permission matrix.

```sql
GRANT SUPERUSER ON DATABASE prod TO admin;
```


---
# Moving Tenants Between Databases (docs/administration/move-tenant)

---
title: Moving Tenants Between Databases
description: Offline MOVE TENANT for relocating a tenant across databases with crash-safe phased recovery.
---

# Moving Tenants Between Databases

**MOVE TENANT** relocates a tenant's data from one database to another. The move is **offline** — the tenant's connections are temporarily paused during migration.

## Why Move a Tenant?

Common reasons:

- **Regional rebalancing** — move customer data to a closer region
- **Database consolidation** — merge customers from one database to another
- **Quota reallocation** — move to a database with higher limits
- **Compliance** — shift data to a jurisdiction-specific database

## Basic Syntax

```sql
MOVE TENANT acme FROM us_east TO us_west;
```

The move is atomic: either the tenant is fully relocated, or fully remains at the source.

## Migration Phases

MOVE TENANT is a five-phase process. Each phase is durable to WAL + Raft:

### Phase 1: Pre-flight

NodeDB validates that all collections the tenant has data in exist in the target database with compatible schemas. No state changes yet.

If validation fails (missing collections or schema mismatch), the move aborts immediately.

### Phase 2: Drain

Active sessions for the tenant on the source are revoked. New writes are rejected. In-flight transactions are allowed to complete with a bounded timeout (default: 30 seconds).

If timeout expires without all in-flight completing, the move aborts. The tenant's sessions are un-revoked and writes resume. Client code should retry the MOVE TENANT after in-flight workload quiets.

### Phase 3: Snapshot

Source creates a consistent backup of the tenant to an in-cluster temporary area. The backup is durable when flushed to storage.

If snapshot fails (disk full, I/O error), the move aborts. The drain is reversed and writes resume on source.

### Phase 4: Cutover

In a single Raft proposal, NodeDB atomically:
1. Drops the tenant from source
2. Restores the tenant into target
3. Updates the tenant↔database catalog mapping

This proposal is atomic at Raft commit — there is no partial state. If the proposal fails, the snapshot artifact remains in temp storage; the tenant is unaffected on source.

### Phase 5: Resume

Writes are accepted on the target database. The temporary snapshot is reaped.

## Crash Recovery

If NodeDB crashes mid-move, startup replays the move journal:

```
Last durable phase: Pre-flight
  → On restart, re-run Pre-flight (idempotent) and continue from Phase 2

Last durable phase: Drain
  → On restart, detect drain is in progress, un-revoke sessions, abort

Last durable phase: Snapshot
  → On restart, complete snapshot or clean up and abort

Last durable phase: Cutover
  → On restart, the Cutover proposal was committed to Raft
    (so the move is already applied on all replicas). Clean up temp snapshot.

Last durable phase: Resume
  → Move completed; temp snapshot already reaped.
```

The journal guarantees no silent partial state.

## Idempotent Retry

If MOVE TENANT succeeds but the client crashes before reading the response, re-issuing the command returns `MOVE_TENANT_ALREADY_AT_TARGET` immediately:

```sql
MOVE TENANT acme FROM us_east TO us_west;
-- Client crashes; connection lost

-- On recovery, re-issue:
MOVE TENANT acme FROM us_east TO us_west;
-- Response: MOVE_TENANT_ALREADY_AT_TARGET
```

## Failure Scenarios and Recovery

| Phase       | Failure                    | Action                                        | Recovery                                         |
| ----------- | -------------------------- | --------------------------------------------- | ------------------------------------------------ |
| Pre-flight  | Schema mismatch            | Abort immediately                             | Fix target schema; retry                         |
| Drain       | Timeout on in-flight       | Abort; un-revoke sessions                     | Wait for workload to quiet; retry                |
| Snapshot    | Disk full / I/O error      | Abort; delete partial snapshot; un-revoke    | Free disk; retry                                 |
| Cutover     | Raft proposal fails        | Abort; snapshot remains; un-revoke            | Investigate Raft quorum; retry                   |
| Crash       | Restart during any phase   | Replay journal; resume or compensate          | Automatic; no manual action needed               |

## Required Role

MOVE TENANT requires superuser privileges. See [RBAC](rbac) for the full permission matrix.

```sql
GRANT SUPERUSER ON DATABASE us_east TO admin;
```

## Supported Mode: Offline

`MOVE TENANT` is an offline operation. The drain phase revokes all sessions for the tenant on the source database before the snapshot/cutover phases run, so there is a (typically brief) window during which clients see connection-revocation errors and must reconnect. This is by design: the offline path keeps the cutover atomic and auditable, and the bounded-drain timeout makes failure modes deterministic.

If a no-downtime cross-database move is required, perform it at the application layer using dual-write to both databases until the cutover, then `MOVE TENANT` an empty (already-mirrored) shadow.

## Practical Examples

### Rebalancing a Region

Move customer `acme` from overloaded `us_east` to `us_west`:

```sql
-- Check target exists and has room
SHOW DATABASES;

-- Check in-flight activity
SELECT COUNT(*) FROM _system.active_sessions WHERE tenant_id = 'acme' AND database = 'us_east';

-- Move during low-traffic window (e.g., 2 AM)
MOVE TENANT acme FROM us_east TO us_west;

-- Verify
SELECT database FROM _system.tenants WHERE name = 'acme';
```

### Consolidating Databases

Move multiple customers from `old_db` to `new_db`:

```sql
-- Move them sequentially to avoid drain timeouts
MOVE TENANT customer_a FROM old_db TO new_db;
MOVE TENANT customer_b FROM old_db TO new_db;
MOVE TENANT customer_c FROM old_db TO new_db;

-- Check old_db is empty
SHOW DATABASE USAGE FOR old_db;
-- Should show 0 tenants

-- Safe to drop
DROP DATABASE old_db;
```

### Compliance-Driven Relocation

Move EU customer data to EU-only database:

```sql
-- Create EU-exclusive database (if not present)
CREATE DATABASE eu_prod WITH (quota_storage_bytes = 107374182400);

-- Move customer
MOVE TENANT eu_customer FROM global_prod TO eu_prod;

-- Verify
SHOW DATABASE LINEAGE FOR eu_prod;  -- shows all tenants now
```

## Monitoring

Monitor move progress via system tables:

```sql
SELECT * FROM _system.move_tenant_journal WHERE tenant = 'acme';
```

Returns: `phase`, `source_database`, `target_database`, `last_durable_lsn`, `status`.

In production, set alerts for:

- Move duration > expected (e.g., > 5 min)
- Drain timeout (repeat attempts indicate persistent in-flight)
- Failures requiring manual recovery


---
# Quotas (docs/administration/quotas)

---
title: Quotas
description: Three-tier resource hierarchy — global, database, tenant. Memory, connections, QPS, WAL priority, cache weight, maintenance CPU.
---

# Quotas

NodeDB enforces resource quotas across four hierarchical levels: global cluster ceiling, per-database budget, per-tenant budget within each database, and engine-internal usage. Quotas protect the cluster from noisy neighbors, burst overages, and fairness violations.

## Three-Tier Hierarchy

Resources flow through a three-level authorization gate before reaching the Data Plane:

```
Global ceiling (cluster-wide)
  ↓
Database budget (per-database, set at CREATE)
  ↓
Tenant budget (per-tenant within database)
  ↓
Engine internal usage (per-engine within tenant)
```

**Enforcement points** (in admission order):

1. **Tenant quota check** — Rate limiting and concurrency per `(database, tenant)` pair
2. **Database quota check** — Rate limiting and concurrency per `database`
3. **Global pressure check** — Cluster-level backpressure if request queues overflow
4. **Memory allocation** — Hierarchical reservation from global → database → tenant → engine

## Database-Level Quotas

Set quotas at database creation or alter after:

```sql
CREATE DATABASE sales WITH QUOTA (
    max_memory_bytes    = 1073741824,  -- 1 GB
    max_storage_bytes   = 10737418240, -- 10 GB
    max_qps             = 1000,
    max_connections     = 100,
    cache_weight        = 2,
    priority_class      = 'critical',
    maintenance_cpu_pct = 25
);

ALTER DATABASE sales SET QUOTA (max_qps = 2000, cache_weight = 3);
```

### Quota Fields

| Field | Meaning | Default | Notes |
|-------|---------|---------|-------|
| `max_memory_bytes` | RAM ceiling | Unlimited (within global) | Per-database L0 + index memory |
| `max_storage_bytes` | Durable storage ceiling | Unlimited | All engines, all shards combined |
| `max_qps` | Queries per second | Unlimited | Hard limit; returns `DATABASE_QUOTA_EXCEEDED` when exceeded |
| `max_connections` | Concurrent connection cap | 1000 | Per-database; new logins rejected when full |
| `cache_weight` | Relative LRU cache share | 1 | 0–100; higher weight = larger cache allocation |
| `priority_class` | WAL fsync priority | `standard` | `critical \| standard \| bulk`; see [Priority Classes](#priority-classes) |
| `maintenance_cpu_pct` | Background work budget | 25 | % of core time for compaction, HNSW maintenance, etc. |

## Tenant-Level Quotas

Tenants inherit quotas from the database, but can be further subdivided:

```sql
ALTER TENANT marketing IN DATABASE sales SET QUOTA (
    max_memory_bytes  = 536870912,  -- 512 MB (half the database)
    max_qps           = 500,         -- half the database's 1000
    max_connections   = 50
);

SHOW TENANT QUOTA FOR marketing IN DATABASE sales;
```

**Sum-of-tenant constraint**: The sum of all tenant quotas within a database cannot exceed the database quota. NodeDB enforces this at write time:

```sql
-- This fails with QUOTA_OVERCOMMIT: tenant limits sum to 2500, but database max_qps is 1000
ALTER TENANT team_a IN DATABASE sales SET QUOTA (max_qps = 600);
ALTER TENANT team_b IN DATABASE sales SET QUOTA (max_qps = 1000);  -- ERROR
```

## Inspecting Quotas and Usage

Check what's set and what's being used:

```sql
SHOW DATABASE QUOTA FOR sales;
SHOW DATABASE USAGE FOR sales;

SHOW TENANT QUOTA FOR marketing IN DATABASE sales;
SHOW TENANT USAGE FOR marketing IN DATABASE sales;
```

**Quota** output columns: `database`, `max_memory_bytes`, `max_storage_bytes`, `max_qps`, `max_connections`, `cache_weight`, `priority_class`, `maintenance_cpu_pct`

**Usage** output columns: `database`, `memory_bytes`, `storage_bytes`, `qps_current`, `qps_p99`, `active_connections`, `maintenance_cpu_seconds`

## Priority Classes and WAL Commitment

Write-ahead log (WAL) fsync is the most expensive operation. NodeDB batches writes into three independent priority groups so critical databases don't wait behind bulk workloads:

| Priority | Behavior | Use Case |
|----------|----------|----------|
| `critical` | Own fsync group, committed first | Production payment system, real-time analytics |
| `standard` | Default batch group (most databases) | User-facing API, transactional |
| `bulk` | Extended timeout, lower fsync rate | Batch ETL, daily reports, backfill |

Set priority at database creation or alter:

```sql
CREATE DATABASE critical_payments WITH QUOTA (priority_class = 'critical');
ALTER DATABASE bulk_processing SET QUOTA (priority_class = 'bulk');
```

A write to a `critical` database blocks until its fsync completes; a write to `bulk` waits longer but doesn't delay `critical` or `standard` commits.

## Weighted-Fair Queue on the SPSC Bridge

Each Data Plane core has a request ring buffer. To prevent one database from saturating an entire core, requests are scheduled via **deficit round-robin (DRR)** weighted by `priority_class`:

- `critical` databases get first pick each scheduling cycle
- `standard` databases get next
- `bulk` databases get the remainder

If one database saturates its share of the core, it throttles only its own writes. Co-resident databases stay responsive.

## Document Cache — Per-Database Allocation

The in-memory document cache is shared across all databases proportional to their `cache_weight`:

```sql
-- This database gets 10x the cache share of others
ALTER DATABASE hot_reads SET QUOTA (cache_weight = 10);
```

When the cache fills, NodeDB evicts entries from the database with the highest current-vs-weight overshoot. A hot database cannot evict a cold database below its proportional fair share.

## Background Task Budget

Maintenance tasks (vector HNSW link cleanup, graph edge sweeps, timeseries segment compaction, array tile compaction, FTS LSM compaction) are CPU-hungry. Each database has a quota on how much core time maintenance can consume per minute:

```sql
ALTER DATABASE large_vector_search SET QUOTA (maintenance_cpu_pct = 50);  -- 50% of core time
```

The scheduler tracks CPU-seconds spent in maintenance per database per minute. Tasks over-cap are deferred to the next window. This prevents one database's compaction from starving interactive queries in another.

## Hierarchical Rate Limiting

Requests are bucketed at four scopes (most-specific first; first to deny wins):

```
user:{user_id} → org:{org_id} → tenant:{tenant_id} → database:{database_id}
```

The database bucket has capacity equal to `max_qps`. A request hitting any bucket's rate limit returns `DATABASE_QUOTA_EXCEEDED` or `TENANT_QUOTA_EXCEEDED` depending on which bucket triggered.

## Error Codes

Quota enforcement produces these errors:

| Error | Trigger |
|-------|---------|
| `TENANT_QUOTA_EXCEEDED` | Tenant rate limit or concurrency exhausted |
| `DATABASE_QUOTA_EXCEEDED` | Database rate limit or concurrency exhausted |
| `SERVER_OVERLOAD` | Global cluster backpressure (queue > 95%) |
| `QUOTA_OVERCOMMIT` | Sum of tenant quotas > database quota (or database > global) |
| `TENANT_VECTOR_DIM_EXCEEDED` | Vector dimension exceeds tenant `max_vector_dim` |
| `TENANT_GRAPH_DEPTH_EXCEEDED` | Graph traversal depth exceeds tenant `max_graph_depth` |

See [error codes reference](../reference/error-codes.md) for full details.

## Metrics and Observability

All quota-related metrics are labeled by database and (where applicable) tenant:

| Metric | Type | Labels |
|--------|------|--------|
| `nodedb_database_qps` | gauge | `database="..."` |
| `nodedb_database_memory_bytes` | gauge | `database="..."` |
| `nodedb_database_storage_bytes` | gauge | `database="..."` |
| `nodedb_database_active_connections` | gauge | `database="..."` |
| `nodedb_database_bridge_queue_depth` | gauge | `database="..."` |
| `nodedb_database_wal_commit_latency_p99` | histogram | `database="..."` |
| `nodedb_database_maintenance_cpu_seconds` | counter | `database="..."` |
| `nodedb_tenant_qps` | gauge | `database="..."`, `tenant="..."` |
| `nodedb_tenant_memory_bytes` | gauge | `database="..."`, `tenant="..."` |
| `nodedb_tenant_storage_bytes` | gauge | `database="..."`, `tenant="..."` |

Scrape your Prometheus instance for these metrics and alert on `_p99` latency growth or queue depth > 85%.


---
# Authentication (docs/administration/authentication)

---
title: Authentication
description: Password auth (SCRAM-SHA-256), API keys, JWKS (JWT), and mTLS.
---

# Authentication

NodeDB supports multiple authentication methods simultaneously.

## Password Auth (SCRAM-SHA-256)

```sql
CREATE USER alice WITH PASSWORD 'strong_password';
CREATE USER IF NOT EXISTS alice WITH PASSWORD 'strong_password';
CREATE USER bob WITH PASSWORD 'secret' ROLE readonly;
DROP USER alice;
DROP USER IF EXISTS alice;
```

`CREATE USER ... IF NOT EXISTS` and `DROP USER ... IF EXISTS` make user DDL idempotent.

```bash
psql -h localhost -p 6432 -U alice
```

## API Keys

API keys enable programmatic access for services and applications:

```sql
CREATE API KEY FOR alice [EXPIRES <seconds>] [WITH SCOPES '<scope>', ...] [WITH DATABASES (<db1>, <db2>)];
LIST API KEYS FOR alice;        -- or SHOW API KEYS FOR alice
REVOKE API KEY <key_id>;
```

- `FOR <user>` is mandatory
- Key is shown once; store it securely
- `EXPIRES` is seconds until revocation (optional)
- `WITH SCOPES` restricts operations (e.g., 'read:collections', 'write:data')
- `WITH DATABASES` restricts collection access (optional; empty = user's default databases)

Example:

```bash
curl -H "Authorization: Bearer <api-key>" http://localhost:6480/v1/query \
     -d '{"sql": "SELECT 1"}'
```

### Service Accounts

Service accounts are privileged accounts designed for application-to-database connections:

```sql
CREATE SERVICE ACCOUNT etl_worker FOR DATABASE analytics_db;
ALTER SERVICE ACCOUNT etl_worker SET DATABASES (analytics_db, staging_db);

CREATE API KEY FOR etl_worker WITH DATABASES (analytics_db);
```

Service accounts are scoped to a single tenant (inherited from the caller) and support per-database access control, inheriting or narrowing their scope when API keys are created on them.

## OIDC / SSO

NodeDB supports OpenID Connect (OIDC) for enterprise Single Sign-On integration. See (oidc-sso) for complete configuration, claim mapping, token refresh, and session lifetime management.

OIDC bearer tokens are supported on the **native protocol and HTTP** entry points only. pgwire connections use SCRAM-SHA-256 exclusively.

## JWKS (JWT)

Multi-provider support (Auth0, Clerk, Supabase, Firebase, Keycloak, Cognito):

```toml
[auth.jwt]
providers = [{ issuer = "https://your-domain.auth0.com/", audience = "your-api" }]
```

JWT claims map to `$auth.*` session variables for RLS:

| Claim     | Variable         | Usage                          |
| --------- | ---------------- | ------------------------------ |
| `sub`     | `$auth.id`       | `WHERE user_id = $auth.id`     |
| `role`    | `$auth.role`     | `WHERE $auth.role = 'admin'`   |
| `org_id`  | `$auth.org_id`   | `WHERE org_id = $auth.org_id`  |
| `scope`   | `$auth.scopes`   | Scope-based access control     |
| `database_id` | `$auth.database_id` | `WHERE db_shard = $auth.database_id` |

Supported algorithms: RS256, ES256.

## mTLS

```toml
[server.tls]
cert = "/path/to/server.crt"
key = "/path/to/server.key"
client_ca = "/path/to/ca.crt"     # enables mTLS
```

## Auth Priority

1. mTLS → 2. JWT Bearer → 3. API key → 4. SCRAM-SHA-256


---
# OIDC Single Sign-On (docs/administration/oidc-sso)

---
title: OIDC Single Sign-On
description: Bearer-token authentication via OIDC providers on the native and HTTP entry points.
---

# OIDC Single Sign-On

Authenticate users via external OpenID Connect providers (Okta, Auth0, Keycloak, etc.) without storing passwords in NodeDB.

## Overview

NodeDB validates JWT bearer tokens against your OIDC provider's public key set (JWKS), maps JWT claims to database access, and creates ephemeral authenticated sessions. No permanent user records are required for SSO logins.

**Wire-protocol scope**: OIDC bearer tokens work on the **native protocol** and **HTTP**. pgwire (PostgreSQL protocol) does not support bearer tokens natively and uses SCRAM-SHA-256 only. If you need pgwire OIDC, configure a pgwire proxy or run separate SCRAM users alongside SSO.

## Registering a Provider

Register your OIDC provider's details:

```sql
CREATE OIDC PROVIDER okta ISSUER 'https://yourorgname.okta.com/' JWKS_URI 'https://yourorgname.okta.com/.well-known/oauth2/default/v1/keys' AUDIENCE 'https://nodedb.example.com'
  CLAIM MAPPING
    WHEN email = null SET DEFAULT_DATABASE = 1
    WHEN groups = 'engineering' SET ADD DATABASES [2] ADD ROLES ['data_analyst'];
```

**Fields:**

**Syntax:**
- `ISSUER '<url>'` — Token issuer URL (e.g., https://accounts.google.com)
- `JWKS_URI '<url>'` — Public key set URL for signature validation
- `AUDIENCE '<aud>'` — (optional) Expected JWT `aud` claim; mismatches are rejected
- `CLAIM MAPPING WHEN ... [SET DEFAULT_DATABASE = <id>] [ADD DATABASES [...]] [ADD ROLES [...]]` — Rules mapping JWT claims to NodeDB identity

NodeDB fetches the JWKS once, caches it in memory, and refreshes it when:
- A token arrives with an unknown key ID (`kid`)
- The cache TTL expires (typically 24 hours)

## Claim Mapping

Claim mapping rules translate JWT claims into NodeDB identity fields and access grants. Each rule in the `CLAIM MAPPING` clause specifies:

- `WHEN <claim> = '<value>'` — JWT claim name and value to match (e.g., `WHEN email = 'alice@example.com'` or `WHEN groups = 'engineering'`)
- `SET DEFAULT_DATABASE = <db_id>` — (optional) set as the user's default database (numeric ID)
- `ADD DATABASES [<id>, ...]` — (optional) grant access to these databases (numeric IDs)
- `ADD ROLES ['<role>', ...]` — (optional) assign these roles to the session (quoted role names)

**Example**: An Okta JWT like:

```json
{
  "sub": "00u1234567890abcdef",
  "email": "alice@acme.com",
  "groups": ["engineering", "admins"],
  "org_id": 1
}
```

Maps to:

```sql
CREATE OIDC PROVIDER okta ISSUER 'https://yourorgname.okta.com/' JWKS_URI 'https://...' AUDIENCE 'api'
  CLAIM MAPPING
    WHEN email = 'alice@acme.com' SET DEFAULT_DATABASE = 2
    WHEN groups = 'engineering' ADD DATABASES [3] ADD ROLES ['data_analyst']
    WHEN groups = 'admins' ADD ROLES ['cluster_admin'];
```

The session inherits all matching rules' effects. If multiple rules grant database access, the union is used.

**Failure modes:**
- Missing claim → no effect (rule skipped)
- Claim value doesn't match the rule's `claim_value` → rule skipped
- Unknown role in `add_roles` → error at provider creation time

## Updating a Provider

Replace the claim mapping rules after creation:

```sql
ALTER OIDC PROVIDER okta SET CLAIM MAPPING
    WHEN email = 'alice@acme.com' SET DEFAULT_DATABASE = 2
    WHEN dept_name = 'sales' ADD DATABASES [5] ADD ROLES ['sales_role'];
```

`ALTER OIDC PROVIDER ... SET CLAIM MAPPING` replaces the full rule set. To change the issuer, JWKS URI, or audience, drop and recreate the provider.

Changes take effect immediately for new logins. Existing sessions tied to the provider remain valid until token expiry.

## Listing Providers

See all configured OIDC providers:

```sql
SHOW OIDC PROVIDERS;
```

**Output includes provider name, issuer, audience, JWKS URI, claim mapping rules, and creation timestamp.**

## Removing a Provider

Drop a provider:

```sql
DROP OIDC PROVIDER IF EXISTS okta;
```

Existing sessions using tokens from that provider are revoked at their next request.

## Validating JWT Tokens

NodeDB validates tokens in this order:

1. **Decode header** → extract `kid` (key ID), `alg` (algorithm), `iss` (issuer)
2. **Look up provider** by issuer
3. **Fetch/cache JWKS** → retrieve public key for `kid`
4. **Validate signature** → verify JWT signature matches the public key
5. **Check audience** → `aud` claim must match the provider's configured audience
6. **Check expiry** → `exp` claim must be in the future
7. **Apply claim mapping** → extract fields, map to identity fields
8. **Build identity** → create ephemeral `AuthenticatedIdentity` (no database user record needed)

**Supported algorithms**: RS256, ES256 (RSA and ECDSA).

## End-to-End Example

### Step 1: Configure the Provider

```sql
CREATE OIDC PROVIDER auth0
  ISSUER 'https://your-tenant.us.auth0.com/'
  JWKS_URI 'https://your-tenant.us.auth0.com/.well-known/jwks.json'
  AUDIENCE 'https://api.example.com'
  CLAIM MAPPING
    WHEN email = null SET DEFAULT_DATABASE = 1
    WHEN https://api.example.com/roles = 'readwrite' ADD ROLES ['readwrite'];
```

### Step 2: Get a Token from Auth0

User logs in via your web app, Auth0 issues a JWT:

```
eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.
eyJzdWIiOiIxMjM0NTY3ODkwIiwiZW1haWwiOiJqb2huQGFjbWUuY29tIiwi
aHR0cHM6Ly9hcGkuZXhhbXBsZS5jb20vcm9sZXMiOlsicmVhZHdyaXRlIl0s
ImF1ZCI6Imh0dHBzOi8vYXBpLmV4YW1wbGUuY29tIiwiaWF0IjoxNTE2MjM5MDIyLCJleHAiOjE3MTYyMzkwMjJ9.
...signature...
```

### Step 3: Authenticate to NodeDB

Send the token in the `Authorization` header:

```bash
curl -H "Authorization: Bearer eyJhbGciOi..." \
     http://localhost:6480/v1/query \
     -d '{"sql": "SELECT 1"}'
```

Or via native protocol (e.g., with `nodedb-client`) on port 6433:

```rust
let auth = AuthMethod::OidcBearer {
    token: "eyJhbGciOi...".to_string(),
    provider: "auth0".to_string(),
};
let session = nodedb_client::connect("localhost:6433", auth).await?;
```

### Step 4: Query as the Authenticated User

NodeDB validates the token and applies claim mapping rules:

```
Token decoded and validated:
  email = "john@acme.com"
  https://api.example.com/roles = "readwrite"
  audience = "https://api.example.com"  ✓
  iss = "https://your-tenant.us.auth0.com/"  ✓
  exp > now()  ✓

Claim mapping applied:
  Rule 1 (email): matches → default_database = production
  Rule 2 (roles = readwrite): matches → add_roles = [readwrite]

Session created:
  username = "john@acme.com" (from email claim)
  default_database = "production"
  roles = [readwrite]

Query executed as authenticated session
```

## Token Lifetime and Session Close

Sessions created via OIDC inherit the token's lifetime:

```
session.token_expiry = jwt.exp
```

When a session's token expires, the connection closes with error `TOKEN_EXPIRED`. The user must re-authenticate with a fresh token.

If both an idle timeout and token expiry are configured on the database, the session closes at whichever comes first:

```
close_at = min(idle_deadline, token_expiry)
```

## Token Refresh

OIDC bearer tokens typically have short lifetimes (minutes to hours). Refresh is handled entirely by the client:

1. Client detects token expiry (or error `TOKEN_EXPIRED` from NodeDB)
2. Client calls Auth0's token refresh endpoint (if available)
3. Client reconnects to NodeDB with the new token
4. New session created with the new token's expiry

NodeDB does not participate in token refresh — it validates whatever token is presented.

## Audit Trail

Every OIDC login is recorded:

```sql
SHOW AUDIT WHERE event_type = 'auth_success' AND database = 'sales';
```

**Audit record includes:**
- Provider name (e.g., `auth0`)
- JWT subject (`sub` claim) for forensics
- Username (from claim mapping)
- Timestamp
- Client IP

Failures are also logged:

```sql
SHOW AUDIT WHERE event_type = 'auth_failure' AND reason LIKE '%oidc%';
```

## Required Role

Creating, altering, or dropping OIDC providers requires:

```
CREATE OIDC PROVIDER — ClusterAdmin or Superuser
ALTER OIDC PROVIDER  — ClusterAdmin or Superuser
DROP OIDC PROVIDER   — ClusterAdmin or Superuser
```

Regular users cannot modify providers.

## Troubleshooting

**Token rejected: invalid signature**
- JWKS cache may be stale; it will auto-refresh on next unknown key ID
- Check that the issuer matches exactly (including trailing `/`)

**Token rejected: audience mismatch**
- JWT `aud` claim doesn't match the configured audience. Check both the token and the provider config.

**Login succeeds but permission denied immediately**
- Claim mapping may have failed (missing required claim). Check the claim-mapping rules and the JWT contents.

**OIDC provider not found**
- Provider was dropped or the `provider` field in the login request is misspelled. List providers: `SHOW OIDC PROVIDERS`

See [authentication](./authentication.rdx) for general auth troubleshooting.


---
# Roles & Permissions (RBAC) (docs/administration/rbac)

---
title: Roles & Permissions (RBAC)
description: Role-based access control with GRANT, REVOKE, and built-in role hierarchy.
---

# Roles & Permissions (RBAC)

## Built-in Roles

| Role           | Permissions                    |
| -------------- | ------------------------------ |
| `readonly`     | SELECT on all collections      |
| `readwrite`    | SELECT, INSERT, UPDATE, DELETE |
| `tenant_admin` | Admin within a tenant          |
| `monitor`      | Read metrics, health, audit; no data access |
| `superuser`    | Unrestricted (cross-tenant)    |
| `cluster_admin` | Cluster-wide admin operations without full superuser capabilities |

Higher roles inherit all permissions of lower roles.

## Database-Scoped Roles

NodeDB supports roles scoped to specific databases:

```sql
GRANT DATABASE_OWNER ON DATABASE analytics TO alice;
GRANT DATABASE_EDITOR ON DATABASE analytics TO bob;
GRANT DATABASE_READER ON DATABASE analytics TO charlie;

ALTER USER alice SET DEFAULT DATABASE analytics;
```

| Role | Permissions | Scope |
| --- | --- | --- |
| `DATABASE_OWNER(db)` | All operations within database, including schema and user management | Specific database |
| `DATABASE_EDITOR(db)` | SELECT, INSERT, UPDATE, DELETE on collections | Specific database |
| `DATABASE_READER(db)` | SELECT only | Specific database |

## ClusterAdmin Role

The `ClusterAdmin` role enables privileged cluster operations without granting full superuser access. ClusterAdmins can perform cluster-wide administration tasks but cannot bypass row-level security or read data from databases they don't own.

```sql
GRANT ROLE cluster_admin TO alice;

-- ClusterAdmin can:
ALTER DATABASE ... RENAME TO;
ALTER DATABASE ... SET QUOTA;
ALTER DATABASE ... SET IDLE_TIMEOUT;
CREATE OIDC PROVIDER;
ALTER OIDC PROVIDER;
```

## Admin DDL Gating Matrix

Certain administrative DDL operations are gated by required roles. Attempting an operation without the required role returns `INSUFFICIENT_PRIVILEGE` and emits a `PermissionDenied` audit entry.

| DDL Operation | Required Role | Notes |
| --- | --- | --- |
| `CREATE DATABASE` | Superuser or ClusterAdmin | Requires cluster-wide privilege |
| `DROP DATABASE` (non-default) | Superuser | Permanent deletion; limited to superuser |
| `DROP DATABASE ... FORCE` | Superuser | Force-drop with cascade safety override |
| `ALTER DATABASE ... RENAME` | Superuser or ClusterAdmin | Cosmetic change; durable identity is `DatabaseId` |
| `ALTER DATABASE ... SET QUOTA` | Superuser or ClusterAdmin | Changes resource limits |
| `ALTER DATABASE ... SET IDLE_TIMEOUT` | Superuser or ClusterAdmin | Session timeout configuration |
| `ALTER DATABASE ... SET AUDIT_DML` | Superuser or ClusterAdmin | Changes audit behavior |
| `ALTER DATABASE ... MATERIALIZE` | Superuser, ClusterAdmin, or DatabaseOwner | Materializes a clone |
| `ALTER DATABASE ... PROMOTE` | Superuser only | One-way mirror promotion; locked to superuser due to operational risk |
| `CLONE DATABASE` | Superuser | Cross-database operation requiring cluster privilege |
| `MIRROR DATABASE` | Superuser | Read-only replica setup |
| `MOVE TENANT` | Superuser | Cross-database tenant relocation |
| `BACKUP DATABASE` | Superuser or DatabaseOwner | Export entire database |
| `RESTORE DATABASE` | Superuser | Disaster recovery operation |
| `KILL SESSION` | Superuser, ClusterAdmin, or session-owner | Terminate a session |
| `CREATE/ALTER/DROP OIDC PROVIDER` | Superuser or ClusterAdmin | SSO configuration |

## Custom Roles

```sql
CREATE ROLE analyst;
CREATE ROLE IF NOT EXISTS analyst;
CREATE ROLE data_engineer;
DROP ROLE analyst;
DROP ROLE IF EXISTS analyst;
```

`CREATE ROLE ... IF NOT EXISTS` and `DROP ROLE ... IF EXISTS` make role DDL idempotent.

## Granting Permissions

```sql
GRANT SELECT ON COLLECTION orders TO analyst;
GRANT SELECT ON TABLE orders TO analyst;
GRANT INSERT, UPDATE ON COLLECTION orders TO data_engineer;
GRANT ALL ON orders TO admin;
GRANT EXECUTE ON FUNCTION full_name TO analyst;
GRANT BACKUP ON TENANT acme TO ops_user;
```

`ON COLLECTION <name>` and `ON TABLE <name>` are explicit object-type keywords (default is collection if neither specified).

## Revoking

```sql
REVOKE INSERT ON orders FROM analyst;
```

## SECURITY DEFINER

Functions and triggers can execute with owner's permissions:

```sql
CREATE FUNCTION admin_count() RETURNS INT SECURITY DEFINER
AS BEGIN RETURN (SELECT COUNT(*) FROM audit_log); END;
```

## Introspection

```sql
SHOW GRANTS FOR analyst;
SHOW PERMISSIONS;
```

See (databases) for database creation, quota, and management. See (quotas) for resource limits. See (audit-logging) for permission denial auditing. See (oidc-sso) for OIDC provider setup and claim mapping.


---
# Row-Level Security (docs/administration/rls)

---
title: Row-Level Security
description: Per-row filtering based on authenticated user context. Works across all eight engines.
---

# Row-Level Security (RLS)

RLS policies filter rows transparently on every query. No application code changes needed.

```sql
CREATE RLS POLICY user_orders ON orders FOR READ USING (customer_id = $auth.id);
CREATE RLS POLICY user_write ON orders FOR WRITE USING (customer_id = $auth.id);
CREATE RLS POLICY admin_bypass ON orders FOR READ USING ($auth.role = 'admin' OR customer_id = $auth.id);
CREATE RLS POLICY org_access ON projects FOR ALL USING (org_id = $auth.org_id);
```

## Policy Types

| Type    | Applies to             |
| ------- | ---------------------- |
| `READ`  | SELECT                 |
| `WRITE` | INSERT, UPDATE, DELETE |
| `ALL`   | Both                   |

## Permissive vs Restrictive

Multiple policies: permissive (default, OR-combined) or restrictive (AND-combined):

```sql
CREATE RLS POLICY org_filter ON docs FOR READ USING (org_id = $auth.org_id) RESTRICTIVE;
CREATE RLS POLICY not_deleted ON docs FOR READ USING (status != 'deleted') RESTRICTIVE;
```

## Session Context Variables

RLS policies can reference the authenticated session context via `$auth.*` variables:

| Variable | Type | Description |
| --- | --- | --- |
| `$auth.id` | string | User ID (from JWT `sub` or user record) |
| `$auth.username` | string | Username |
| `$auth.role` | string | Current role |
| `$auth.tenant_id` | u64 | Tenant ID of the authenticated user |
| `$auth.database_id` | u64 | Database ID of the session |

Example combining tenant and database scoping:

```sql
CREATE RLS POLICY multi_scope ON data FOR READ
  USING (tenant_id = $auth.tenant_id AND database_shard = $auth.database_id);
```

Substitution is fail-closed: if the auth context lacks a required variable (e.g., `$auth.database_id` is null), the query rejects the row rather than allowing wide-open access.

## Cross-Engine Behavior

RLS filters are injected at plan time, before engine dispatch:

- **Vector search** — unauthorized vectors excluded from results
- **Graph traversal** — edges to invisible nodes are skipped
- **FTS** — only matching documents passing RLS are returned
- **KV** — blocked rows return empty

## Managing

```sql
SHOW RLS POLICIES;
DROP RLS POLICY user_orders ON orders;
```


---
# TLS & mTLS (docs/administration/tls)

---
title: TLS & mTLS
description: Transport encryption for all protocols with optional mutual TLS.
---

# TLS & mTLS

## Server TLS

```toml
[server.tls]
cert_path = "/etc/nodedb/tls/server.crt"
key_path = "/etc/nodedb/tls/server.key"
```

Per-protocol TLS toggle:

```toml
[server.tls]
native = true
pgwire = true
http = true
resp = true
ilp = false    # disable TLS for high-throughput ILP ingest
```

## mTLS (Mutual TLS)

```toml
[server.tls]
cert = "/path/to/server.crt"
key = "/path/to/server.key"
client_ca = "/path/to/ca.crt"
crl = "/path/to/revocation.crl"
```

When `client_ca` is set, clients must present a valid certificate signed by the CA.

## Encryption at Rest

- **Data volumes** — AES-256-XTS
- **WAL segments** — AES-256-GCM with per-file data encryption keys
- **Lite devices** — AES-256-GCM + Argon2id key derivation


---
# Audit Logging (docs/administration/audit-logging)

---
title: Audit Logging
description: Hash-chained audit trail with configurable levels and SIEM export.
---

# Audit Logging

Tamper-evident, hash-chained audit log. Every entry includes a SHA-256 hash of the previous entry — if any record is modified, the chain breaks.

```sql
SHOW AUDIT LOG;
SHOW AUDIT LOG LIMIT 50;
```

## Audit Levels

```toml
[audit]
level = "standard"
```

| Level      | Records                                        |
| ---------- | ---------------------------------------------- |
| `minimal`  | Auth events (login, failure, denial)           |
| `standard` | + admin actions, DDL, sessions, config changes |
| `full`     | + every query, RLS denials                     |
| `forensic` | + row-level mutations, CRDT deltas             |

## Key Events

### Authentication & Session Events

AuthSuccess, AuthFailure, PermissionDenied, SessionConnect/Disconnect, SessionRevoked, LockoutTriggered, LoginRateLimited.

### Database Lifecycle Events

DatabaseCreated, DatabaseDropped, DatabaseRenamed, DatabaseQuotaChanged, DatabaseCloned, DatabaseMirrored, DatabasePromoted, DatabaseMaterialized, TenantMoved, DatabaseBackedUp, DatabaseRestored, DatabaseAuditDmlChanged, DatabaseIdleTimeoutChanged.

### Authorization & Audit Events

PrivilegeChange, RlsRejected, AdminAction, TenantCreated/Deleted.

### System Events

SnapshotBegin/End, RestoreBegin/End, CertRotation, KeyRotation, NodeJoined/Left, QueryExec, RowChange, OidcProviderChanged.

All events carry `database_id` when applicable, enabling filtering of audit trails per database.

## DML Audit (Optional Per-Database)

Enable audit logging of all data modifications on a per-database basis:

```sql
ALTER DATABASE production SET AUDIT_DML = 'writes';   -- INSERT, UPDATE, DELETE only
ALTER DATABASE production SET AUDIT_DML = 'all';      -- All queries
ALTER DATABASE production SET AUDIT_DML = 'none';     -- Disabled (default)
```

When enabled, every write produces a `DmlAudit` entry carrying:
- User ID, database, collection, operation type, row ID, LSN
- Statement digest and execution timestamp
- Sourced from the Event Plane (non-blocking to writers)

## Per-Database Audit Filtering

Filter audit entries by database:

```sql
SHOW AUDIT IN DATABASE production;
SHOW AUDIT IN DATABASE production WHERE event_type = 'DmlAudit';
```

## Hash Chain Integrity

Every audit entry includes a SHA-256 hash of the previous entry. The chain extends each entry's hash with `database_id` when scoped, preserving compatibility with pre-database entries whose `database_id` was null.

If any record is modified, the chain breaks and tampering is detected.

## SIEM Export

```sql
CREATE CHANGE STREAM audit_export ON _system.audit
    DELIVERY WEBHOOK 'https://siem.example.com/ingest'
    WITH (format = 'json', hmac_secret = 'your-secret');
```

See (session-management) for session revocation audit events and (oidc-sso) for authentication provider changes.


---
# Session Management (docs/administration/session-management)

---
title: Session Management
description: SHOW SESSIONS, KILL SESSION, idle timeout, lockout, login rate limit.
---

# Session Management

Manage active user connections with monitoring, termination, and automatic timeout controls.

## Listing Sessions

View all active sessions across the cluster:

```sql
SHOW SESSIONS;
SHOW SESSIONS IN DATABASE sales;
SHOW SESSIONS WHERE user = 'alice';
```

**Output columns:**

| Column | Meaning |
|--------|---------|
| `session_id` | Unique session identifier (UUID) |
| `addr` | Client IP and port |
| `user` | Username |
| `database` | Connected database name |
| `tenant` | Tenant ID |
| `started_at` | Connection timestamp (UTC) |
| `last_active_ms` | Milliseconds since last statement |
| `idle_timeout_secs` | Database's idle timeout setting |
| `token_expiry_ms` | JWT/token lifetime (0 if password auth) |
| `bytes_in` | Total bytes received |
| `bytes_out` | Total bytes sent |
| `current_statement_digest` | Statement hash if query running |

## Killing a Session

Force-terminate a session immediately:

```sql
KILL SESSION 'f47ac10b-58cc-4372-a567-0e02b2c3d479';
```

**Authorization**: Requires `Superuser`, `ClusterAdmin`, or ownership of the session's database.

**Behavior**:
- Returns error `SESSION_NOT_FOUND` (SQLSTATE 42704) if the session ID doesn't exist
- For an active query, closes the connection at the next request boundary
- For an idle connection, force-closes immediately
- Emits an audit row with `KillReason::AdminKill` for forensics

## Idle Timeout

Automatically close idle connections after a configurable duration:

```sql
ALTER DATABASE sales SET IDLE_TIMEOUT 1800;  -- 30 minutes
ALTER DATABASE sales SET IDLE_TIMEOUT 0;     -- Disable (default)
```

**Behavior**:
- The Control Plane runs a background timer that checks sessions every 10 seconds
- A session is idle if `time_now - last_activity > idle_timeout`
- Closes with error `SESSION_IDLE_TIMEOUT`
- Emits an audit row with `KillReason::IdleTimeout`
- Only affects sessions where idle timeout is enabled; password-auth sessions track `last_active_ms` on every statement

## Kill Reasons

Every session termination is tagged with a reason for audit and debugging:

| Kill Reason | Trigger |
|-------------|---------|
| `Alive` | Session active (not killed) |
| `UserDropped` | User account deleted (DROP USER) |
| `IdleTimeout` | Exceeded database idle timeout |
| `TokenExpired` | JWT/OIDC token lifetime ended |
| `AdminKill` | KILL SESSION command |

Check the audit log to see which sessions closed and why:

```sql
SHOW AUDIT WHERE event_type = 'session_revoked' AND database = 'sales';
```

## Session Revocation

When a user's permissions change, existing sessions are automatically updated:

### Hard Revocation (Connection Closes)

Triggered by:
- `DROP USER alice`
- User soft-delete (setting `is_active = false`)
- Full role purge (removing all roles)

Action: Connection closes at the next request boundary with error `SESSION_REVOKED`. An audit entry is written before the connection closes.

### Soft Revocation (Identity Rehydrated)

Triggered by:
- `GRANT ROLE admin TO alice`
- `REVOKE ROLE admin FROM alice`
- `ALTER USER alice SET ROLE newrole`

Action: On the next statement, the session re-fetches the user record, rebuilds the `AuthenticatedIdentity`, and swaps it atomically. No reconnect needed; the user sees their new permissions immediately.

## In-Flight Permission Propagation

Grant a role in one connection, and another open connection sees it immediately:

**Connection 1:**
```sql
GRANT ROLE admin TO alice;
```

**Connection 2 (already logged in as alice):**
```sql
-- Runs your statement with updated permissions (no reconnect)
SELECT * FROM admin_view;  -- Now allowed
```

This works because:
- `GRANT/REVOKE` bumps the user's version counter
- Request entry checks the version; if it moved, rebuilds the identity and swaps the cached `Arc`
- No disconnect, no token refresh — just a cheap version check on each request

## Persistent Login Lockout

After repeated failed login attempts, an account is locked:

```sql
-- Account locked after 5 failed attempts, unlocks after 30 minutes
```

**Behavior**:
- Lockout state is persisted in the `_system.lockout_state` table and survives restart
- On login, the server checks if the account is locked before attempting SCRAM/Argon2
- A failed login increments the counter; a successful login resets it
- Locked accounts are automatically unlocked after the lockout duration expires
- Audit log records `LockoutTriggered` with the IP address for forensics

Check which accounts are locked (superuser only):

```sql
SELECT username, failed_count, locked_until_ms, last_failure_ip
FROM _system.lockout_state
WHERE failed_count > 0;
```

## Pre-Auth Login Rate Limiting

Two token buckets limit login attempts *before* password verification (to prevent username enumeration):

| Bucket | Capacity | Window |
|--------|----------|--------|
| `login_ip:{addr}` | 30 attempts/min | 1 minute sliding |
| `login_user:{username}` | 10 attempts/min | 1 minute sliding |

Whichever bucket triggers first causes a login rate-limit denial.

**Security properties**:
- Constant-time error path: regardless of whether the rate limit is exceeded or the password is wrong, the error is always generic `INVALID_CREDENTIALS` with uniform delay
- No information leakage: an attacker cannot determine if a username exists or if they've hit the IP limit
- In-memory only: restarts reset the window

## Session Cap

The cluster has a maximum number of concurrent sessions:

```
cluster.max_active_sessions = 10000 (default)
```

When reached, new logins return `SESSION_CAP_EXCEEDED`. This is a safety valve to prevent resource exhaustion. Increase if your user count grows.

## Audit Denial Events

Permission denials and session closures are logged with full context:

| Audit Event | Trigger |
|-------------|---------|
| `PermissionDenied` | Query denied due to RLS or role check |
| `RlsRejected` | Row-level security policy rejected access |
| `SessionRevoked` | Connection force-closed (admin or revocation) |
| `LockoutTriggered` | Account locked due to failed attempts |
| `LoginRateLimited` | Login denied due to IP or username rate limit |

Query the audit log to analyze denials:

```sql
SHOW AUDIT WHERE event_type = 'permission_denied'
    AND database = 'sales'
    AND created_at > now() - interval '1 hour';
```

See [audit-logging](./audit-logging.rdx) for full details.

## Required Roles

| Operation | Required Role |
|-----------|---------------|
| `KILL SESSION` | `Superuser` or `DatabaseOwner` of that session's database |
| `ALTER DATABASE SET IDLE_TIMEOUT` | `ClusterAdmin` or `Superuser` |
| `SHOW SESSIONS` | Any user (shows their own + other tenants' if privileged) |


---
# Multi-Tenancy (docs/administration/multi-tenancy)

---
title: Multi-Tenancy
description: Tenant isolation, quotas, backup, and GDPR purge.
---

# Multi-Tenancy

Each tenant has fully isolated storage, indexes, and security policies. Cross-tenant data access is impossible by design.

## Scope: Database vs Tenant

**Database** is the deployment unit — a collection-namespace owner, the quota parent, and the unit of clone, mirror, and backup. One Origin instance hosts many databases.

**Tenant** is a customer within a deployment — a row-level scoping label inside collections within a database. One database hosts many tenants.

A tenant's data does not span databases. To move a tenant across databases, use `MOVE TENANT`. Cross-database queries are forbidden — applications that need to access multiple databases must open separate connections.

See (databases) for database creation, quota, and backup operations.

## Creating Tenants

```sql
CREATE TENANT acme;
CREATE USER alice WITH PASSWORD 'secret' ROLE readwrite TENANT 42;
```

### Superuser Session Tenant Switching

A superuser can switch the session tenant at runtime:

```sql
SET TENANT = 'acme' | 1234 | DEFAULT;
```

This changes which tenant's data is visible for subsequent queries in the session. Non-superusers cannot use `SET TENANT`.

## Quotas

```sql
ALTER TENANT acme SET QUOTA max_qps = 5000;
ALTER TENANT acme SET QUOTA max_storage_bytes = 53687091200;
ALTER TENANT acme SET QUOTA max_connections = 50;

SHOW TENANT USAGE FOR acme;
SHOW TENANT QUOTA FOR acme;
EXPORT USAGE FOR TENANT acme PERIOD '2026-03' FORMAT 'json';
```

## Backup & Restore

```sql
BACKUP TENANT acme TO '/backups/acme.bak';
RESTORE TENANT acme FROM '/backups/acme.bak' DRY RUN;
RESTORE TENANT acme FROM '/backups/acme.bak';
```

Backups cover all 7 engines. Encrypted with AES-256-GCM using the tenant WAL key.

## GDPR Purge

```sql
DROP TENANT acme;                -- catalog metadata only
PURGE TENANT acme CONFIRM;      -- ALL data across all engines (permanent)
```

## Isolation Model

| Layer   | Isolation                                     |
| ------- | --------------------------------------------- |
| Storage | Separate key prefixes per tenant              |
| Indexes | Tenant-scoped, no cross-tenant overlap        |
| WAL     | Per-tenant segments with per-tenant encryption |
| Queries | Tenant ID injected at plan time               |
| Audit   | Per-tenant audit entries                      |


---
# Monitoring (docs/administration/monitoring)

---
title: Monitoring
description: Prometheus metrics, OpenTelemetry, health checks, and Grafana integration.
---

# Monitoring

## Prometheus Metrics

```bash
curl http://localhost:6480/metrics
```

70+ system metrics: per-engine, per-core, connection, query, replication, and storage. Latency histogram with 13 buckets.

## PromQL Engine

Full Prometheus query engine at `/obsv/api`. Point Grafana at this URL as a Prometheus data source. Supports all Tier 1+2+3 functions (rate, irate, delta, histogram_quantile, holt_winters, etc.).

## Prometheus Remote Write/Read

Use NodeDB as a long-term Prometheus storage backend:

```yaml
remote_write:
  - url: "http://nodedb:6480/obsv/api/v1/write"
remote_read:
  - url: "http://nodedb:6480/obsv/api/v1/read"
```

## OpenTelemetry

- **OTLP ingest** — Metrics, traces, and logs via HTTP (4318) and gRPC (4317)
- **OTLP export** — Push NodeDB's own traces/metrics to any OTLP collector

Feature-gated: `--features otel`, `--features promql`, `--features monitoring`.

## Health Checks

```bash
curl http://localhost:6480/healthz        # k8s readiness — 503 until startup completes
curl http://localhost:6480/health/live    # liveness probe
curl http://localhost:6480/health/ready   # WAL recovered, ready for queries
```

## Graph Statistics

For per-collection edge counts and graph cardinality, use `SHOW GRAPH STATS`:

```sql
SHOW GRAPH STATS 'collection_name' VERBOSE;
SHOW GRAPH STATS 'collection_name' AS OF SYSTEM TIME <ms>;
```

See [Graph Engine](../storage-engines/graph) for complete `SHOW GRAPH STATS` documentation.

## Key Metrics

| Metric Category | Examples                                       |
| --------------- | ---------------------------------------------- |
| Engine          | Per-engine memory, query count, latency         |
| Core            | Per-core CPU, queue depth, io_uring completions |
| Connection      | Active connections, auth failures               |
| Replication     | Raft log lag, replication latency               |
| Storage         | WAL fsync latency, segment count, compaction debt |
| Database        | Per-database resource usage and performance     |
| Tenant          | Per-tenant query rates and memory usage         |

## Per-Database & Per-Tenant Metrics

Database-scoped and tenant-scoped metrics enable precise monitoring of multi-tenant deployments.

### Per-Database Metrics

All metrics labeled with `database="<name>"`:

| Metric | Type | Description |
| --- | --- | --- |
| `nodedb_database_qps` | counter | Queries per second for the database |
| `nodedb_database_memory_bytes` | gauge | Memory currently used by the database |
| `nodedb_database_storage_bytes` | gauge | Storage footprint of all collections in the database |
| `nodedb_database_connections` | gauge | Active connections to the database |
| `nodedb_database_bridge_queue_depth` | gauge | Pending requests in SPSC bridge for the database |
| `nodedb_database_wal_commit_latency_p99` | histogram | 99th percentile WAL commit latency for writes to the database |
| `nodedb_database_maintenance_cpu_seconds` | counter | Cumulative CPU time spent on background maintenance tasks |
| `nodedb_database_mirror_lag_ms` | gauge | Replication lag for mirror databases (zero if not mirrored) |

### Per-Tenant Metrics

All metrics labeled with `database="<name>"` and `tenant="<name>"`:

| Metric | Type | Description |
| --- | --- | --- |
| `nodedb_tenant_qps` | counter | Queries per second for the tenant |
| `nodedb_tenant_memory_bytes` | gauge | Memory currently used by the tenant |
| `nodedb_tenant_storage_bytes` | gauge | Storage footprint of the tenant's data |

These metrics are emitted from the dispatch layer (qps), memory governor (memory), WAL (storage), connection listener (connections), and background scheduler (maintenance_cpu_seconds), ensuring consistent attribution across request processing.

## Cross-shard transaction metrics

| Metric | Type | Description | 
| ------ | ---- | ----------- |
| `nodedb_sequencer_epochs_total` | counter | Epochs proposed by the sequencer |
| `nodedb_sequencer_epoch_duration_ms` | histogram | Time to drain and propose each epoch |
| `nodedb_sequencer_admitted_txns_total{outcome}` | counter | Admission outcomes: `admitted`, `rejected_conflict`, `rejected_inbox_full`, `rejected_txn_too_large`, `rejected_fanout_too_wide`, `rejected_tenant_quota`, `rejected_not_leader` |
| `nodedb_sequencer_inbox_depth` | gauge | Pending transactions in the sequencer inbox |
| `nodedb_calvin_scheduler_lock_wait_ms_total{vshard}` | counter | Cumulative lock-wait time per shard |
| `nodedb_calvin_executor_txn_duration_ms{vshard}` | histogram | Per-shard execution time for cross-shard txns |
| `nodedb_calvin_ollp_retries_total{predicate_class,outcome}` | counter | OLLP retry outcomes (`succeeded`, `retried`, `exhausted`, `circuit_open`, `tenant_budget_exceeded`) |
| `nodedb_calvin_ollp_circuit_state{predicate_class}` | gauge | 0 = closed, 1 = half-open, 2 = open |
| `nodedb_calvin_ollp_backoff_ms{predicate_class}` | gauge | Current OLLP retry backoff delay |
| `nodedb_calvin_infra_abort_total{reason}` | counter | Infrastructure aborts (disk error, OOM, corruption) |

## Memory backpressure metrics

| Metric | Type | Description |
| ------ | ---- | ----------- |
| `nodedb_backpressure_critical_total{engine}` | counter | Write handlers that entered the Critical-pressure flush path |
| `nodedb_backpressure_emergency_total{engine}` | counter | Write handlers rejected by Emergency-pressure |

## IO priority metrics

| Metric | Type | Description |
| ------ | ---- | ----------- |
| `nodedb_io_queue_depth{priority}` | gauge | Pending tasks per IO priority tier (`background`, `normal`, `high`, `critical`) |
| `nodedb_io_wait_ns{priority}` | histogram | Submission-to-completion latency per tier |

## CDC metrics

| Metric | Type | Description |
| ------ | ---- | ----------- |
| `nodedb_cdc_events_dropped_total{tenant,stream}` | counter | Events dropped from a named stream's buffer due to overflow — per stream, not global |

Alert on this increasing for a stream whose consumer is active — it means the consumer is falling behind.

## Corruption quarantine metrics

See [Corruption Quarantine](./corruption-quarantine) for the full quarantine runbook.

| Metric | Type | Description |
| ------ | ---- | ----------- |
| `nodedb_segments_quarantined_total{engine,collection}` | counter | Cumulative segments quarantined since startup |
| `nodedb_segments_quarantined_active{engine,collection}` | gauge | Segments currently in quarantine |


---
# Corruption Quarantine (docs/administration/corruption-quarantine)

---
title: Corruption Quarantine
description: Automatic detection and isolation of corrupt segments, with operator tooling for inspection and recovery.
---

# Corruption Quarantine

NodeDB automatically detects corrupt segments using CRC32C checksums and isolates them so one bad segment cannot take down an entire collection or shard.

## How it works — two-strike rule

**First CRC failure** on a segment: log a warning and retry the read once.

**Second failure** on the same segment: the segment is quarantined —

1. The file is renamed to `<original-path>.quarantined.<unix_ts_ms>`
2. The segment ID is recorded in the quarantine registry
3. Subsequent reads of that segment return a typed `SegmentQuarantined` error
4. All other segments in the collection continue serving reads normally

On restart, NodeDB scans the data directory for `*.quarantined.*` files and rebuilds the registry automatically — quarantine state survives restarts.

The two-strike rule prevents transient I/O errors (flipped bit on a warm SSD) from quarantining healthy segments, while ensuring persistently corrupt segments are isolated after the first retry.

## Affected engines

Quarantine is wired into reads for:

- **Columnar** — segment scan, retention scan, prior-value read
- **FTS** — redb backend byte retrieval
- **Raft snapshots** — snapshot chunk install
- **Vector** — wrapper present; currently no production read sites (segments held in-memory; quarantine activates if disk-resident vector segments are added in future)

## Inspect quarantined segments

```bash
curl http://localhost:6480/v1/cluster/debug/quarantined-segments
```

Response:

```json
{
  "segments": [
    {
      "segment_id": "col-00042",
      "engine": "columnar",
      "collection": "events",
      "quarantined_at_unix_ms": 1746480000000,
      "strikes": 2,
      "last_error": "FooterCrcMismatch"
    }
  ]
}
```

An empty `segments` array means no segments are currently quarantined.

## Metrics

| Metric | Type | Description |
|---|---|---|
| `nodedb_segments_quarantined_total{engine,collection}` | counter | Cumulative segments quarantined since startup |
| `nodedb_segments_quarantined_active{engine,collection}` | gauge | Segments currently in quarantine |

Alert on `nodedb_segments_quarantined_total` increasing, or `nodedb_segments_quarantined_active > 0`.

## Recovery

A quarantined segment means data in that segment is unreadable. Options:

**Option 1 — Restore from backup.** If you have a recent backup, restore it. The quarantined file is preserved as-is until you delete it manually.

```sql
RESTORE TENANT acme FROM '/backups/acme-latest.bak';
```

**Option 2 — Rebuild the index.** For vector and FTS indexes, the index can be rebuilt from the source data without data loss.

```sql
REINDEX CONCURRENTLY my_collection;
```

**Option 3 — Drop and repopulate.** If the collection can be repopulated from an upstream source, drop and recreate it.

**After recovery:** the `.quarantined.<ts>` files can be deleted manually once you've confirmed data is restored. NodeDB does not auto-delete them.

## Storage location

By default quarantined files stay alongside their originals on local disk. To archive quarantined files to object storage instead, configure `quarantine_storage` — see [Backup & Recovery](./backup-recovery#quarantine-storage).


---
# Backup & Recovery (docs/administration/backup-recovery)

---
title: Backup & Recovery
description: Per-tenant backup, point-in-time restore, and WAL replay.
---

# Backup & Recovery

## Backup

```sql
BACKUP TENANT acme TO '/backups/acme-2026-04.bak';
```

Backups cover all 7 engines: documents, indexes, vectors, graph edges, KV tables, timeseries, and CRDT state. Encrypted with AES-256-GCM using the tenant WAL key.

The backup envelope embeds catalog rows and the source tombstone set alongside segment data, so a restored snapshot reconstructs the catalog deterministically and refuses to resurrect collections tombstoned before the backup was taken. Each `StoredCollection` row carries a `size_bytes_estimate` field surfaced through `_system.dropped_collections` for sizing the L2 cleanup queue before `PURGE`.

## Validate

```sql
RESTORE TENANT acme FROM '/backups/acme-2026-04.bak' DRY RUN;
```

Validates the backup without restoring.

## Restore

```sql
RESTORE TENANT acme FROM '/backups/acme-2026-04.bak';
```

## Crash Recovery

On startup, NodeDB:

1. Scans WAL segments from the last checkpoint
2. Validates CRC32C checksums
3. Replays valid records to reconstruct state
4. Discards partially written records

The Event Plane resumes from its LSN watermark — no events are lost.

## WAL Archiving

WAL segments can be archived for point-in-time recovery. Old segments are eligible for cleanup once all records have been flushed to L1 segments.

## Object storage for snapshots and quarantine

By default, snapshots and quarantined segment files are stored on local disk alongside the data directory. Both can be redirected to S3-compatible object storage.

```toml
[snapshot_storage]
endpoint = "https://s3.amazonaws.com"
bucket   = "my-nodedb-snapshots"
region   = "us-east-1"
prefix   = "cluster-prod"
access_key = "AKIA..."
secret_key = "..."

[quarantine_storage]
endpoint = "https://s3.amazonaws.com"
bucket   = "my-nodedb-quarantine"
region   = "us-east-1"
prefix   = "cluster-prod"
access_key = "AKIA..."
secret_key = "..."
```

Omit `endpoint` (or leave it empty) to use the local filesystem — this is the default. Both sections are independent: you can put snapshots on S3 and leave quarantine files local, or vice versa.

The cold data tier (`[cold_storage]`) is a separate config from these two — it covers Parquet archives and timeseries L2 data, not snapshots or quarantine files.


---
# Cluster Operations (docs/administration/cluster-operations)

---
title: Cluster Operations
description: Cluster setup, node management, shard rebalancing, and rolling upgrades.
---

# Cluster Operations

## Introspection

```sql
SHOW CLUSTER;
SHOW NODES;
SHOW RAFT GROUPS;
```

## Adding Nodes

New nodes join the cluster and receive vShard assignments during rebalancing.

## Shard Rebalancing

When nodes join or leave, vShards redistribute automatically:

1. Coordinator selects vShards based on load metrics
2. Target node replicates the vShard's data
3. Ownership transfers atomically
4. Old node drops the vShard data

Transparent to clients — requests forward during migration.

## Rolling Upgrades

1. Drain a node (move its vShard leadership to other nodes)
2. Upgrade the binary
3. Restart the node
4. Repeat for each node

Raft leader election handles temporary unavailability.

## Failure Recovery

- **Single node failure** — Raft elects new leaders for affected vShards
- **Minority failure** — Cluster remains available with quorum
- **Majority failure** — Cluster becomes read-only until quorum restores

## Cross-shard transaction mode

```sql
-- Require atomic cross-shard writes (default)
SET cross_shard_txn = 'strict';

-- Opt out of atomicity for bulk loads (each shard commits independently)
SET cross_shard_txn = 'best_effort_non_atomic';

SHOW cross_shard_txn;
```

See [Cross-Shard Transactions](../architecture/cross-shard-transactions) for details on the Calvin sequencer and OLLP.

## Debug endpoints

```bash
# List all Raft groups (data, meta, sequencer)
curl http://localhost:6480/v1/cluster/debug/raft/{group_id}

# QUIC transport diagnostics
curl http://localhost:6480/v1/cluster/debug/transport

# Catalog descriptor dump
curl http://localhost:6480/v1/cluster/debug/catalog/descriptors

# Segments currently in quarantine (corrupt and isolated)
curl http://localhost:6480/v1/cluster/debug/quarantined-segments
```

See [Corruption Quarantine](./corruption-quarantine) for the quarantine runbook.


---
# Memory Management (docs/administration/memory-management)

---
title: Memory Management
description: Per-engine memory budgets, jemalloc arenas, and memory governance.
---

# Memory Management

## Memory Limit

```toml
[server]
memory_limit = "4GiB"
```

Or: `NODEDB_MEMORY_LIMIT=4GiB`

## Per-Engine Budgets

Memory is divided among engines. No single engine can starve others. Default allocation:

| Budget               | Default |
| -------------------- | ------- |
| Vector               | 30%     |
| Sparse / Metadata    | 15%     |
| CRDT                 | 10%     |
| Timeseries           | 10%     |
| Query Execution      | 20%     |
| Unallocated headroom | 15%     |

## Per-Core Arenas

Each Data Plane core is pinned to a dedicated jemalloc arena via `nodedb-mem`. This eliminates allocator lock contention in the TPC architecture.

## Backpressure

When memory pressure rises:
- Engine memtables flush to disk earlier
- SPSC bridge reduces read depth at 85% utilization
- New reads suspend at 95% utilization


---
# Performance Tuning (docs/administration/performance-tuning)

---
title: Performance Tuning
description: Data Plane cores, io_uring tuning, engine-specific parameters, and query optimization.
---

# Performance Tuning

## Data Plane Cores

```toml
[server]
data_plane_cores = 4    # default: CPUs - 1
```

More cores = more parallel shards. Each core is an isolated, shared-nothing shard.

## io_uring

The Data Plane uses io_uring for NVMe I/O. Ensure:

- Linux kernel 5.1+ (5.11+ recommended for full io_uring features)
- NVMe storage (io_uring benefits are minimal on spinning disks)
- `ulimit -l unlimited` for locked memory

## Vector Engine

| Parameter         | Default | Tune for              |
| ----------------- | ------- | --------------------- |
| M                 | 16      | Higher = more recall  |
| EF_CONSTRUCTION   | 200     | Higher = better graph |
| Quantization      | none    | SQ8/PQ for large datasets |

## FTS Engine

| Parameter         | Default | Tune for              |
| ----------------- | ------- | --------------------- |
| Memtable threshold | 32M postings | Higher = fewer flushes |
| Compaction levels  | 8x8    | More levels = less write amplification |

## Query Optimization

```sql
EXPLAIN SELECT * FROM users WHERE age > 30;
```

Check that:
- Vector searches use the HNSW index (not brute force)
- Spatial queries use the R*-tree (check for `SpatialOp`)
- Columnar queries show predicate pushdown and block skip
- FTS queries use the inverted index (not full scan)


---
# Production Checklist (docs/administration/production-checklist)

---
title: Production Checklist
description: Pre-production checklist for deploying NodeDB in production.
---

# Production Checklist

## Infrastructure

- [ ] Linux kernel 5.11+ (full io_uring support)
- [ ] NVMe storage for Data Plane I/O
- [ ] Sufficient locked memory (`ulimit -l unlimited`)
- [ ] Memory limit set appropriately (`NODEDB_MEMORY_LIMIT`)
- [ ] Data Plane cores configured (`NODEDB_DATA_PLANE_CORES`)

## Security

- [ ] TLS enabled for all protocols
- [ ] Authentication configured (SCRAM, JWT, API keys, or mTLS)
- [ ] RBAC roles defined and assigned
- [ ] RLS policies for multi-tenant data
- [ ] Audit logging level set (at least `standard`)
- [ ] Default passwords changed

## Replication

- [ ] Replication factor >= 3 for production data
- [ ] Cluster has odd number of nodes (3, 5, 7) for Raft quorum
- [ ] Cross-region learner replicas if needed

## Monitoring

- [ ] Prometheus scraping `/metrics`
- [ ] Grafana dashboards configured
- [ ] Health check endpoint monitored (`/health/ready`)
- [ ] Alerting on key metrics (WAL fsync latency, replication lag, memory pressure)

## Backup

- [ ] Regular tenant backups scheduled
- [ ] Backup validation (DRY RUN) tested
- [ ] Restore procedure documented and tested
- [ ] WAL archiving configured for PITR

## Operations

- [ ] Rolling upgrade procedure documented
- [ ] Shard rebalancing tested
- [ ] Failure recovery tested (single node, minority failure)


---
# CRDT Sync Overview (docs/crdt-sync/overview)

---
title: CRDT Sync Overview
description: AP on the edge, CP in the cloud. Loro-backed offline-first sync with constraint validation.
---

# CRDT Sync Overview

NodeDB bridges **AP** (available, partition-tolerant) on edge devices with **CP** (consistent, partition-tolerant) in the cloud.

## How It Works

1. **Device writes locally** — Available immediately, no network required
2. **CRDT delta produced** — Each write generates a Loro CRDT delta
3. **Connectivity returns** — Deltas sync to Origin via WebSocket (port 9090)
4. **Origin validates** — SQL constraints (UNIQUE, FK, CHECK) checked at Raft commit
5. **Committed** — Delta replicated via Raft, broadcast to connected clients
6. **Conflict** — Failed constraints produce a typed `CompensationHint` back to device

## Shape Subscriptions

Devices subscribe to a subset of data — not the entire database. This is a wire-level sync feature (ShapeSubscribe message), not a SQL statement:

```
Shape subscription for: users WHERE user_id = $me
Shape subscription for: tax_rates WHERE jurisdiction IN ('US-CA', 'US-NY')
```

Changes within a device's shape are pushed in real time via the sync protocol.

## Multiple Devices

Multiple devices converge to the same state regardless of operation order. Loro's operation-based CRDTs guarantee eventual consistency without coordination.

## Supported Engines for Sync

All eight engines now participate in CRDT sync:

- **Data-bearing engines** — Document (schemaless + strict), KV, Columnar, Timeseries, Spatial, Vector, Array
- **Overlay engines** — Graph traversal and Full-Text Search built on synced collections
- **Schema changes** — DDL (new columns, collections) broadcast to connected Lite clients after Origin catalog commit; embedded clients auto-discover schema

All engines work locally on NodeDB-Lite with sub-millisecond reads and no network dependency.


---
# Conflict Policies (docs/crdt-sync/conflict-policies)

---
title: Conflict Policies
description: Declarative per-collection conflict resolution strategies.
---

# Conflict Policies

## Setting Policies

Conflict policies are set per constraint. When a sync writes to a collection violates a constraint, the policy determines how to handle it:

```sql
ALTER COLLECTION notes SET ON CONFLICT LAST_WRITER_WINS FOR UNIQUE;
ALTER COLLECTION profiles SET ON CONFLICT CASCADE_DEFER FOR FOREIGN_KEY;
```

## Available Policies

| Policy           | Behavior                                         |
| ---------------- | ------------------------------------------------ |
| `LAST_WRITER_WINS` | Concurrent writes to the same field: timestamp-latest wins (default) |
| `RENAME_SUFFIX` (alias `RENAME_APPEND_SUFFIX`) | Duplicate key: append suffix to ID and retry insert |
| `CASCADE_DEFER` | Foreign key missing: defer validation until next cascade check |
| `ESCALATE_TO_DLQ` | Violation: move the row to a dead-letter queue for manual review |

## Per-Collection Configuration

Policies are stored per collection and per constraint type. Each collection can have different policies for UNIQUE, FOREIGN_KEY, and CHECK constraints:

```sql
ALTER COLLECTION notes SET ON CONFLICT LAST_WRITER_WINS FOR UNIQUE;
ALTER COLLECTION orders SET ON CONFLICT CASCADE_DEFER FOR FOREIGN_KEY;
SHOW CONFLICT POLICY ON notes;
```

`CUSTOM` policies are configured via the native protocol and are not available via SQL DDL.


---
# Constraint Validation (docs/crdt-sync/constraint-validation)

---
title: Constraint Validation
description: SQL constraints enforced at sync time on Origin with typed compensation hints.
---

# Constraint Validation

SQL constraints are checked on Origin at sync time, not on the device. This keeps local writes fast (no network round-trip) while preserving global consistency.

## Enforced Constraints

- **UNIQUE** — Duplicate key → `CompensationHint::UniqueViolation`
- **FOREIGN KEY** — Dangling reference → `CompensationHint::ForeignKeyMissing`
- **CHECK / schema** — Type or CHECK violation → `CompensationHint::SchemaViolation`
- **Integrity** — Other integrity-rule violation → `CompensationHint::IntegrityViolation`
- **Permission** — Insufficient privilege → `CompensationHint::PermissionDenied`
- **Rate limit** — Quota exceeded → `CompensationHint::RateLimited`
- **Custom** — Application-defined handling → `CompensationHint::Custom`

## Compensation Hints

When a local write violates a constraint on Origin, a typed `CompensationHint` is sent back to the device. The application handles the conflict:

- **UniqueViolation** — The document ID or key already exists. Offer the user to rename or merge.
- **ForeignKeyMissing** — A FK target doesn't exist. Offer to create it or discard the child.
- **SchemaViolation** — A CHECK constraint or type rule failed. Show the violation and let the user fix it.
- **IntegrityViolation** — Another integrity rule was violated. Surface the error for manual resolution.
- **PermissionDenied** — The operation exceeds the user's role. Show the denial and suggest re-authentication.
- **RateLimited** — A quota or rate limit was exceeded. Retry after cooldown.
- **Custom** — Application-specific handling via custom compensation logic.

No silent data loss — the application always decides.

## State Constraints

```sql
ALTER COLLECTION invoices ADD CONSTRAINT invoice_flow
    ON COLUMN status TRANSITIONS (
        'draft'     -> 'submitted',
        'submitted' -> 'approved' BY ROLE 'manager',
        'approved'  -> 'issued'   BY ROLE 'accountant'
    );
```

State transitions are validated at sync time. Invalid transitions (e.g., `draft -> issued`) are rejected with a compensation hint.


---
# Dead-Letter Queue (docs/crdt-sync/dead-letter-queue)

---
title: Dead-Letter Queue
description: Handling persistently failing CRDT deltas with diagnostic metadata.
---

# Dead-Letter Queue

Deltas that persistently fail constraint validation on Origin are routed to a dead-letter queue (DLQ). Each entry includes diagnostic metadata: the original delta, the constraint that failed, and the compensation hint.

## Inspection

Operators can inspect DLQ entries to understand why syncs are failing and take corrective action.

## Resolution Options

- **Replay** — Fix the constraint issue and replay the delta
- **Discard** — Permanently discard the failed delta
- **Manual merge** — Resolve the conflict manually and apply a corrected delta

## Trigger DLQ

AFTER triggers that persistently fail (after retry with exponential backoff) also route to the DLQ. This applies to both CRDT sync triggers and regular async triggers.


---
# Offline Sync Patterns (docs/crdt-sync/offline-sync)

---
title: Offline Sync Patterns
description: Patterns for provisional numbering, state machines, and offline-first workflows.
---

# Offline Sync Patterns

## Provisional Numbering

Official document numbers (invoices, receipts) must be sequential and gap-free. Offline devices can't reserve numbers from Origin.

**Pattern:**

1. Create documents with provisional ULID-based identifiers (`DRAFT-01JXQ3...`)
2. Sync to Origin via CRDT
3. Origin assigns official sequential number via `nextval('invoice_seq')`
4. Official number syncs back to device

```sql
-- Device (offline)
INSERT INTO invoices (invoice_number, customer_id, amount, status)
VALUES ('DRAFT-01JXQ3...', 'cust-005', 750.00, 'draft');

-- Origin (on sync) — trigger assigns official number
-- Invoice becomes INV-26-04-00024
```

## State Machine Transitions

```sql
ALTER COLLECTION invoices ADD CONSTRAINT invoice_flow
    ON COLUMN status TRANSITIONS (
        'draft'     -> 'submitted',
        'submitted' -> 'approved' BY ROLE 'manager',
        'approved'  -> 'issued'   BY ROLE 'accountant',
        'issued'    -> 'voided'   BY ROLE 'controller'
    );
```

Devices can create drafts and submit them. Approval requires `manager` role — only available on Origin.

## Tax Rate Sync

Sync relevant jurisdiction rates to devices via shape subscriptions. The device client subscribes to:

```
Shape: tax_rates WHERE jurisdiction IN ('US-CA', 'US-NY')
```

Lookup at invoice time with temporal query. Reconcile on sync if rates changed.

## NodeDB-Lite Usage

```swift
// iOS
let db = NodeDbLite.open(path: "invoices.db")
db.execute("INSERT INTO invoices ...")
db.sync(url: "wss://origin.example.com/sync", token: authToken)
```

```javascript
// Browser (WASM)
const db = await NodeDbLite.open("invoices");
await db.execute("INSERT INTO invoices ...");
await db.sync("wss://origin.example.com/sync");
```

All eight engines work locally with sub-millisecond reads. CRDT sync is transparent.


---
# Error Codes (docs/reference/error-codes)

---
title: Error Codes
description: Numeric error codes returned by NodeDB with descriptions and resolution hints.
---

# Error Codes

NodeDB uses `NodeDbError` — a struct with numeric `ErrorCode`, human-readable `message`, machine-matchable `ErrorDetails`, and optional chained `cause`.

## Error Categories

| Code Range | Category   | Examples                          |
| ---------- | ---------- | --------------------------------- |
| 1000–1099  | Write path | Constraint violation, deadline    |
| 1100–1199  | Read path  | Collection not found              |
| 1200–1299  | Query      | Parse error, plan error           |
| 2000–2099  | Auth       | Authorization denied              |
| 3000–3099  | Sync       | Delta rejected, sync error        |
| 4000–4099  | Storage    | Segment corrupted, WAL error      |
| 5000–5099  | Config     | Bad request, invalid config       |
| 6000–6099  | Cluster    | No quorum, shard unavailable      |
| 7000–7099  | Memory     | Budget exceeded                   |
| 8000–8099  | Encryption | Key error, decrypt failure        |
| 9000–9099  | Internal   | Unexpected internal error         |

## Common Errors

| Error                  | Code | Resolution                  |
| ---------------------- | ---- | --------------------------- |
| `COLLECTION_NOT_FOUND` | 1100 | Check collection name       |
| `TYPE_MISMATCH`        | 1020 | Check data types in query   |
| `CONSTRAINT_VIOLATION` | 1000 | Duplicate key on plain INSERT maps to SQLSTATE `23505` (`unique_violation`). Use `UPSERT` or `INSERT ... ON CONFLICT DO NOTHING / DO UPDATE` for non-error semantics. |
| `DEADLINE_EXCEEDED`    | 1002 | Query took too long         |
| `INSUFFICIENT_BALANCE` | 1022 | Not enough for TRANSFER     |
| `OVERFLOW`             | 1021 | i64 overflow on INCR        |
| `AUTHORIZATION_DENIED` | 2000 | Check RBAC grants           |
| `SYNC_DELTA_REJECTED`  | 3001 | CRDT sync constraint failed |

## Database Boundary Errors

| Error | Code | Resolution |
| --- | --- | --- |
| `DATABASE_NOT_FOUND` | 1101 | Verify database name; check `SHOW DATABASES` |
| `CANNOT_DROP_DEFAULT_DATABASE` | 1102 | The `default` database is reserved; drop user databases instead |

## Database Quota Errors (7000–7099)

| Error | Code | Resolution |
| --- | --- | --- |
| `SERVER_OVERLOAD` | 7000 | Cluster under heavy pressure; retry with exponential backoff |
| `TENANT_QUOTA_EXCEEDED` | 7001 | Tenant's rate/concurrency/memory limit exhausted; reduce load or raise quota via `ALTER TENANT … IN DATABASE … SET QUOTA` |
| `DATABASE_QUOTA_EXCEEDED` | 7002 | Database's rate/concurrency/memory limit exhausted; reduce load or raise quota via `ALTER DATABASE … SET QUOTA` |
| `QUOTA_OVERCOMMIT` | 5001 | Sum of tenant quotas exceeds database quota, or database quota exceeds global; rebalance quotas |
| `TENANT_VECTOR_DIM_EXCEEDED` | 1023 | Vector embedding dimensionality exceeds tenant's `max_vector_dim`; reduce dimensionality or raise quota |
| `TENANT_GRAPH_DEPTH_EXCEEDED` | 1024 | Graph traversal depth exceeds tenant's `max_graph_depth`; reduce traversal depth or raise quota |
| `SESSION_CAP_EXCEEDED` | 7003 | Cluster session capacity exhausted; close idle sessions or raise `max_active_sessions` |

## Database Clone Errors (5000–5099)

| Error | Code | Resolution |
| --- | --- | --- |
| `CLONE_DEPTH_EXCEEDED` | 5002 | Clone lineage exceeds `MAX_CLONE_DEPTH = 8`; promote source clone to materialize and flatten lineage |
| `CLONE_DEPENDENCY` | 5003 | Source database has dependent clones; `DROP` clones first or use `DROP DATABASE source FORCE` to trigger materialization |
| `CANNOT_CLONE_MIRROR` | 5004 | Mirrors are read-only and create ambiguous bitemporal lineage; `PROMOTE` the mirror first, then clone |

## Database Mirror Errors (1100–1199, 6000–6099)

| Error | Code | Resolution |
| --- | --- | --- |
| `MIRROR_READ_ONLY` | 1103 | Mirror is read-only until promoted; use `ALTER DATABASE <mirror> PROMOTE` to switch to writable mode |
| `STALE_READ_NOT_LEADER` | 1104 | `Strong` consistency requested on a mirror (which is stale); use `BoundedStaleness` or `Eventual` consistency, or connect to the source database for strong reads |
| `MIRROR_DEGRADED` | 6001 | Mirror lag exceeds threshold (default: 5 s async, 100 ms sync); check source–mirror network and availability |
| `MIRROR_DISCONNECTED` | 6002 | Mirror disconnected from source; reconnecting with exponential backoff |

## Move Tenant Errors

| Error | Code | Resolution |
| --- | --- | --- |
| `MOVE_TENANT_ALREADY_AT_TARGET` | 5005 | Tenant already moved to target database; re-issuing is idempotent (safe to retry) |

## Session Lifecycle Errors (2000–2099)

| Error | Code | Resolution |
| --- | --- | --- |
| `SESSION_REVOKED` | 2001 | Session terminated due to identity change (DROP USER, role revocation, or soft-delete via `is_active=false`); reconnect to authenticate |
| `SESSION_IDLE_TIMEOUT` | 2002 | Session closed due to inactivity exceeding `idle_session_timeout_secs`; reconnect and keep activity |
| `SESSION_NOT_FOUND` | 1105 | `KILL SESSION` targeted a non-existent session ID; verify `SHOW SESSIONS` for active session IDs |

## Error Construction

Errors are created via constructors: `NodeDbError::storage("detail")`, `NodeDbError::collection_not_found("users")`, etc.


---
# SQLSTATE Mapping (docs/reference/sqlstate-mapping)

---
title: SQLSTATE Mapping
description: PostgreSQL-compatible SQLSTATE codes returned over pgwire.
---

# SQLSTATE Mapping

NodeDB maps internal error codes to PostgreSQL SQLSTATE codes for pgwire compatibility.

| NodeDB Error          | SQLSTATE | PostgreSQL Meaning           |
| --------------------- | -------- | ---------------------------- |
| Syntax error          | 42601    | Syntax error                 |
| Collection not found  | 42P01    | Undefined table              |
| Column not found      | 42703    | Undefined column             |
| Type mismatch         | 42846    | Cannot coerce                |
| Unique violation      | 23505    | Unique violation             |
| FK violation          | 23503    | Foreign key violation        |
| Check violation       | 23514    | Check violation              |
| Not null violation    | 23502    | Not null violation           |
| Permission denied     | 42501    | Insufficient privilege       |
| Numeric overflow      | 22003    | Numeric value out of range   |
| Division by zero      | 22012    | Division by zero             |
| Rate limit exceeded   | 54001    | Too many resources           |
| Tenant quota exceeded | 54001    | Too many resources           |
| Database quota exceeded | 54001  | Too many resources           |
| Server overload       | 54001    | Too many resources           |
| Quota overcommit      | 54000    | Program limit exceeded       |
| Database not found    | 42P01    | Undefined table              |
| Mirror read only      | 25006    | Read-only SQL transaction    |
| Stale read on mirror  | 0A000    | Feature not supported        |
| Clone depth exceeded  | 54000    | Program limit exceeded       |
| Session not found     | 42704    | Undefined object             |
| Session idle timeout  | 08006    | Connection failure           |
| Session revoked       | 08006    | Connection failure           |
| Internal error        | XX000    | Internal error               |


---
# Type OIDs (docs/reference/type-oids)

---
title: Type OIDs
description: PostgreSQL type OID mappings for pgwire protocol compatibility.
---

# Type OIDs

NodeDB maps its types to PostgreSQL OIDs for pgwire compatibility.

| NodeDB Type  | PostgreSQL Type | OID  |
| ------------ | --------------- | ---- |
| STRING       | text            | 25   |
| INT          | int8            | 20   |
| FLOAT        | float8          | 701  |
| BOOL         | bool            | 16   |
| TIMESTAMP    | timestamp       | 1114 |
| TIMESTAMPTZ  | timestamptz     | 1184 |
| INTERVAL     | interval        | 1186 |
| UUID         | uuid            | 2950 |
| DECIMAL      | numeric         | 1700 |
| ARRAY        | jsonb           | 3802 |
| OBJECT       | jsonb           | 3802 |
| GEOMETRY     | bytea           | 17   |
| VECTOR       | float4[]        | 1021 |

`TIMESTAMP` (OID 1114) is timezone-naive. `TIMESTAMPTZ` (OID 1184) is UTC-normalised. Use `TIMESTAMPTZ` for wall-clock event times; `TIMESTAMP` for values that are inherently local (e.g. a schedule expressed in the user's timezone before conversion).

Complex types (ARRAY, OBJECT) are serialized as JSONB over pgwire. GEOMETRY is serialized as WKB (bytea). VECTOR is serialized as a float4 array.


---
# Wire Protocol Specs (docs/reference/wire-protocol-specs)

---
title: Wire Protocol Specs
description: Protocol specifications for NDB (native), pgwire extensions, and HTTP API.
---

# Wire Protocol Specs

## NDB (Native Protocol)

Binary MessagePack over TCP (port 6433). Two message types:

- **Sql** — SQL text as MessagePack string. Parsed by sqlparser.
- **Native opcodes** — Typed messages (Get, Put, VectorSearch, etc.) that skip SQL parsing.

Both produce the same `PhysicalPlan`.

## pgwire Extensions

NodeDB extends the PostgreSQL wire protocol with:

- `LIVE SELECT` — async notification delivery via `NotificationResponse`
- `GRAPH` commands — parsed as SQL, executed via EngineRules
- `MATCH` patterns — Cypher-subset parsed inline
- `SEARCH ... USING VECTOR` — vector search syntax
- Session variables: `SET nodedb.consistency = 'eventual'`

## HTTP API

REST endpoints on port 6480:

All data routes are under `/v1/`. Probe routes (`/healthz`, `/health/*`, `/metrics`) are unversioned and always reachable.

| Method | Path                          | Description                              |
| ------ | ----------------------------- | ---------------------------------------- |
| POST   | `/v1/query`                   | Execute SQL                              |
| POST   | `/v1/query/stream`            | Stream NDJSON results                    |
| GET    | `/v1/status`                  | Node status                              |
| GET    | `/v1/cluster/status`          | Cluster status                           |
| POST   | `/v1/auth/exchange-key`       | Exchange API key for session token       |
| POST   | `/v1/auth/session`            | Create session                           |
| DELETE | `/v1/auth/session`            | Delete session                           |
| POST   | `/v1/collections/{name}/crdt/apply` | Apply CRDT delta                   |
| GET    | `/v1/streams/{id}/events`     | SSE named-stream events                  |
| GET    | `/v1/streams/{id}/poll`       | Long-poll named-stream                   |
| GET    | `/v1/cdc/{collection}`        | SSE CDC stream                           |
| GET    | `/v1/cdc/{collection}/poll`   | Long-poll CDC                            |
| GET    | `/v1/obsv/api/v1/...`         | PromQL endpoints (Prom-spec suffix)      |
| WS     | `/v1/ws`                      | WebSocket JSON-RPC                       |
| GET    | `/healthz`                    | k8s readiness (503 until startup ready)  |
| GET    | `/health/live`                | Liveness probe                           |
| GET    | `/health/ready`               | Readiness probe                          |
| POST   | `/health/drain`               | Cooperative drain                        |
| GET    | `/metrics`                    | Prometheus metrics                       |

JSON responses carry `Content-Type: application/vnd.nodedb.v1+json; charset=utf-8`. Clients sending only `Accept: application/vnd.nodedb.vN+json` for an unsupported `N` get `406 Not Acceptable`.


---
# WAL & Segment Formats (docs/reference/wal-segment-formats)

---
title: WAL & Segment Formats
description: On-disk format specifications for WAL records and segment files.
---

# WAL & Segment Formats

## WAL Record Format

```
┌─────────┬────────────────┬─────────────┬──────┬───────────┬───────────┬─────────────┬──────────┬─────────┐
│  magic  │ format_version │ record_type │  lsn │ tenant_id │ vshard_id │ payload_len │ reserved │ crc32c  │
│  4 bytes│    2 bytes     │   4 bytes   │  8B  │   8 bytes │  4 bytes  │   4 bytes   │ 16 bytes │ 4 bytes │
└─────────┴────────────────┴─────────────┴──────┴───────────┴───────────┴─────────────┴──────────┴─────────┘
```

Byte offsets: magic 0..4, format_version 4..6, record_type 6..10, lsn 10..18, tenant_id 18..26, vshard_id 26..30, payload_len 30..34, reserved 34..50, crc32c 50..54. Total header: **54 bytes**.

Bit 14 of `record_type` signals AES-256-GCM encryption of the payload. The logical record type is `record_type & !(1 << 14)`.

- Page size: 4 KiB or 16 KiB (O_DIRECT alignment)
- CRC32C per page for bit-rot detection
- Segmented with auto-rollover
- AES-256-GCM encryption (optional, per-record)

## Segment File Footer

All persistent segment files embed a footer:

```
┌─────────────────┬────────────┬──────────┬─────────┬─────────┐
│ format_version  │ created_by │ checksum │ min_lsn │ max_lsn │
└─────────────────┴────────────┴──────────┴─────────┴─────────┘
```

- Compaction preserves monotonic LSN ordering
- Footer enables crash-safe validation on startup

## Columnar Segment Format (NDBS)

- Magic bytes: `NDBS`
- Versioned footer with CRC32C
- Per-column codec metadata
- 1024-row blocks with min/max/null-count statistics
- Delete bitmaps (Roaring)


---
# Engine Configuration (docs/reference/engine-config)

---
title: Engine Configuration
description: Per-engine configuration options for memory budgets, index parameters, and compression.
---

# Engine Configuration

## Vector Engine

| Parameter         | Default | Description                             |
| ----------------- | ------- | --------------------------------------- |
| M                 | 16      | Max HNSW connections per node per layer |
| EF_CONSTRUCTION   | 200     | Search width during index build         |
| Memory budget     | 30%     | Percentage of total memory              |

## Graph Engine

Graph, KV, and FTS engines do not have dedicated budget config fields. They draw from the sparse budget.

| Parameter  | Default | Description            |
| ---------- | ------- | ---------------------- |
| Compaction | 10%     | Buffer merge threshold |

## FTS Engine

| Parameter          | Default     | Description                |
| ------------------ | ----------- | -------------------------- |
| Memtable threshold | 32M entries | Flush to segment threshold |
| Compaction levels  | 8x8         | Level-based compaction     |

## Columnar / Timeseries / Spatial Engines

The `timeseries` peer engine has its own memory budget knob `timeseries_budget_fraction` (default 10%); `columnar` and `spatial` draw from the columnar engine pool.

| Parameter    | Default | Description                |
| ------------ | ------- | -------------------------- |
| Block size   | 1024    | Rows per block             |
| partition_by | none    | Time partition interval    |
| retention    | none    | Data retention period      |

## KV Engine

KV draws from the sparse budget. There is no dedicated `kv_budget_fraction` config field.

## Memory Budget Fields

These are the actual config fields controlling memory allocation:

| Config field                | Default | Controls                          |
| --------------------------- | ------- | --------------------------------- |
| `vector_budget_fraction`    | 0.30    | HNSW index and vector data        |
| `sparse_budget_fraction`    | 0.15    | Graph, KV, FTS, and metadata      |
| `crdt_budget_fraction`      | 0.10    | CRDT state and sync buffers       |
| `timeseries_budget_fraction`| 0.10    | Timeseries peer engine            |
| `query_budget_fraction`     | 0.20    | Query execution working memory    |


---
# Limits & Defaults (docs/reference/limits-defaults)

---
title: Limits & Defaults
description: System limits, default values, and capacity constraints.
---

# Limits & Defaults

## Per-Op Server Limits

Server-enforced caps announced to every native client in the `HelloAckFrame.limits` struct. Defaults are uncapped (`None`); operators set caps via configuration. Exceeding a cap returns a typed `LimitExceeded { limit_name, value, max }` error rather than truncating silently.

| Limit                  | Applies to                                        |
| ---------------------- | ------------------------------------------------- |
| `max_vector_dim`       | Vector embedding dimensionality                   |
| `max_top_k`            | `top_k` for vector / FTS / hybrid retrieval       |
| `max_scan_limit`       | Result set size on scans                          |
| `max_batch_size`       | Rows per batch INSERT/UPSERT                      |
| `max_crdt_delta_bytes` | Single CRDT delta payload                         |
| `max_query_text_bytes` | SQL text length                                   |
| `max_graph_depth`      | `MAX_DEPTH` for graph traversal                   |

SDKs surface these via `client.limits()`. See [Native Protocol — Handshake](../connectivity/native-protocol#handshake).

## Connection Limits

| Setting          | Default | Max        |
| ---------------- | ------- | ---------- |
| Max connections   | 1024    | Configurable |
| Data Plane cores  | CPUs-1  | CPU count  |

## Collection Limits

| Limit                | Value          |
| -------------------- | -------------- |
| Collection name      | 255 chars      |
| Field name           | 255 chars      |
| Document size        | 16 MiB         |
| Vector dimensions    | 65536          |
| Edge properties size | 16 MiB         |

## Query Limits

| Limit                   | Value       |
| ----------------------- | ----------- |
| SQL text length          | 1 MiB       |
| Result set (default)     | No limit    |
| MATCH MAX_DEPTH default  | 10          |
| BFS MAX_VISITED default  | 10000       |

## Default Ports

| Protocol | Port |
| -------- | ---- |
| pgwire   | 6432 |
| NDB      | 6433 |
| HTTP     | 6480 |
| Sync     | 9090 |
| RESP     | off  |
| ILP      | off  |

## Default Memory

| Setting      | Default      |
| ------------ | ------------ |
| memory_limit | 1 GiB        |
| Docker       | 75% of RAM   |

## Database Boundary

| Setting | Value | Notes |
| --- | --- | --- |
| `DatabaseId` reserved range | 0–1023 | System databases; user IDs start at 1024 |
| `default` database | `DatabaseId(0)` | Cannot be dropped; always exists |

## Clone & Mirror

| Setting | Value | Notes |
| --- | --- | --- |
| `MAX_CLONE_DEPTH` | 8 | Maximum clone lineage depth; `CLONE_DEPTH_EXCEEDED` if exceeded |
| Mirror lag threshold (async) | 5 s (default) | Configurable per mirror; `MIRROR_DEGRADED` when exceeded |
| Mirror lag threshold (sync) | 100 ms (default) | Configurable per mirror; `MIRROR_DEGRADED` when exceeded |

## Role & Auth

| Setting | Value | Notes |
| --- | --- | --- |
| `MAX_ROLE_INHERITANCE_DEPTH` | 8 | Maximum depth of role inheritance chain |
| Login rate limit (`login_ip`) | 30 per minute | Per IP address; returns `INVALID_CREDENTIALS` after capacity exhausted |
| Login rate limit (`login_user`) | 10 per minute | Per username; returns `INVALID_CREDENTIALS` after capacity exhausted |

## Per-Database Quota Defaults

| Setting | Default | Notes |
| --- | --- | --- |
| `cache_weight` | 1 | Relative weight for doc cache eviction |
| `priority_class` | `standard` | `critical` (own fsync group, committed first), `standard` (batched normally), `bulk` (extended timeout) |
| `maintenance_cpu_pct` | 25 | Percentage of core time available for background compaction |
| `idle_session_timeout_secs` | 0 | Session idle timeout; 0 = disabled |
| `audit_dml` | `none` | `none` (disabled), `writes` (DML inserts/updates/deletes), `all` (includes selects) |


---
# Feature Flags (docs/reference/feature-flags)

---
title: Feature Flags
description: Compile-time feature gates for optional capabilities.
---

# Feature Flags

NodeDB uses Cargo feature flags for optional capabilities.

| Feature      | What it enables                                        |
| ------------ | ------------------------------------------------------ |
| `promql`     | PromQL query engine at `/obsv/api`                     |
| `otel`       | OpenTelemetry OTLP ingest and export                   |
| `grafana`    | Grafana data source endpoints                          |
| `monitoring` | Meta-feature: enables `promql` + `otel` + `grafana`    |
| `kafka`      | Kafka bridge for change stream delivery                |

The `nodedb-fts` crate defines additional language features (`lang-ja`, `lang-zh`, `lang-ko`, `lang-th`) for dictionary-based CJK/Thai segmentation. These are crate-level features, not workspace-level.

## Building with Features

```bash
cargo build --release --features promql,otel,monitoring
cargo build --release --all-features
```

## Runtime Configuration

Feature-gated capabilities are configurable at runtime via `nodedb.toml` and environment variables once compiled in.