Cluster Operations

Introspection

SHOW CLUSTER;
SHOW NODES;
SHOW RAFT GROUPS;

Adding Nodes

New nodes join the cluster and receive vShard assignments during rebalancing.

Shard Rebalancing

When nodes join or leave, vShards redistribute automatically:

1. Coordinator selects vShards based on load metrics
2. Target node replicates the vShard's data
3. Ownership transfers atomically
4. Old node drops the vShard data

Transparent to clients — requests forward during migration.

Rolling Upgrades

1. Drain a node (move its vShard leadership to other nodes)
2. Upgrade the binary
3. Restart the node
4. Repeat for each node

Raft leader election handles temporary unavailability.

Failure Recovery

• Single node failure — Raft elects new leaders for affected vShards
• Minority failure — Cluster remains available with quorum
• Majority failure — Cluster becomes read-only until quorum restores

Cross-shard transaction mode

-- Require atomic cross-shard writes (default)
SET cross_shard_txn = 'strict';

-- Opt out of atomicity for bulk loads (each shard commits independently)
SET cross_shard_txn = 'best_effort_non_atomic';

SHOW cross_shard_txn;

See Cross-Shard Transactions for details on the Calvin sequencer and OLLP.

Debug endpoints

# List all Raft groups (data, meta, sequencer)
curl http://localhost:6480/v1/cluster/debug/raft/{group_id}

# QUIC transport diagnostics
curl http://localhost:6480/v1/cluster/debug/transport

# Catalog descriptor dump
curl http://localhost:6480/v1/cluster/debug/catalog/descriptors

# Segments currently in quarantine (corrupt and isolated)
curl http://localhost:6480/v1/cluster/debug/quarantined-segments

See Corruption Quarantine for the quarantine runbook.