Clustering

ObjectStore v0.0.8 includes a local cluster prototype. Storage nodes hold replicas while PostgreSQL tracks placement and object metadata. The supplied Compose setup runs these services on one Docker host; it is not a validated multi-server production deployment.

How replication works

The gateway splits an object into segments and attempts to place each segment on three different host IDs. It acknowledges a write only after at least two host IDs accept every segment. The replication target is fixed at three; adding a fourth node spreads segments across more nodes but does not create a fourth copy of each segment. PostgreSQL holds the map of objects to replicas, so node copies alone are not a metadata backup.

The bundled Compose cluster puts every node on one physical machine. Its CLUSTER_TEST_NODE_DOMAINS=true setting counts containers as separate fault domains for testing. Those copies do not protect against losing that machine. For a disposable two-machine test with real host separation, follow the two-machine durability drill.

Start the local service and check replicas

1. Check out the v0.0.8 source. Copy .env.cluster.example to a private file outside the repository and set S3_ACCESS_KEY, S3_SECRET_KEY, CLUSTER_TOKEN, CLUSTER_REPAIR_TOKEN, and POSTGRES_PASSWORD.

2. Start the three nodes, gateway, and PostgreSQL from the repository root:

   docker compose --env-file /path/to/cluster.env -f compose.cluster.yaml up -d --build
   curl http://127.0.0.1:9001/ready

3. Run the cluster test suite in this disposable setup. To inspect or restore missing copies later, run one repair pass:

   docker compose --env-file /path/to/cluster.env -f compose.cluster.yaml run --rm repair

   Check replicas_restored, segments_under_replicated, and segments_unrecoverable in its output. A repair pass needs at least one readable copy of a segment; it cannot reconstruct a segment whose every copy is lost.

Add storage capacity

The included node-d is a local expansion example. It needs its own persistent volume. Back up PostgreSQL and the existing node volumes before changing the cluster.

1. Start the new node:

   docker compose --env-file /path/to/cluster.env -f compose.cluster.yaml \
     --profile expansion up -d node-d

2. Register it using the host ID configured for node-d in compose.cluster.yaml:

   docker compose --env-file /path/to/cluster.env -f compose.cluster.yaml \
     run --rm --no-deps --entrypoint /usr/local/bin/objectstore gateway \
     cluster-join http://node-d:9100 5f1447b5-3f9e-457b-8ee7-e26f0c475b5f

3. Add CLUSTER_NODES=http://node-a:9100,http://node-b:9100,http://node-c:9100,http://node-d:9100 to the private environment file. If you also want a higher logical capacity limit, raise MAX_TOTAL_BYTES there. Allow for up to three physical copies of each segment when sizing disks.

4. Recreate the gateway with the new configuration, then run repair with the same node list:

   docker compose --env-file /path/to/cluster.env -f compose.cluster.yaml up -d --no-deps gateway
   docker compose --env-file /path/to/cluster.env -f compose.cluster.yaml run --rm repair
   curl http://127.0.0.1:9001/ready

New writes can use the added node. Repair also moves verified existing segments toward the new placement and reports segments_rebalanced. Keep every node referenced by existing replica metadata in CLUSTER_NODES; do not remove an old node just because the new one is online.

Expand capacity and restore replicas together

If a node is unavailable, keep its URL configured, add the replacement or extra node using the steps above, and run repair. The repair pass copies each readable segment to preferred available nodes and updates its recorded placement after verifying the copy. Check the under-replicated and unrecoverable counts before relying on the cluster. Increasing MAX_TOTAL_BYTES changes the logical limit; it does not increase the number of copies per segment.

For separate physical servers, assign a different CLUSTER_HOST_ID to each host and use private networking and node TLS. The supplied Compose file and expansion commands above are local tests, not a ready multi-server deployment. The operations page covers the remaining limits, backups, and recovery.