TECHNICAL GUIDE · STORAGE ARCHITECTURE

Architecting high-performance S3 object storage: NVMe pools, Ceph and erasure coding

Modern unstructured workloads — from massive AI training sets and vector databases to immutable ransomware backups — demand throughputs that traditional SAN appliances cannot deliver. This technical guide explores how to build resilient, wire-speed S3 storage clusters that scale linearly without proprietary controller bottlenecks.

Updated october 2026
7 min read
Altovar Technical Editorial Team

1. Why traditional SAN architectures struggle with object workloads

Unstructured data grows faster than most storage budgets. AI training datasets, media archives, log retention and vector databases generate many concurrent reads and writes of large and small objects. Traditional dual-controller SAN (storage area network) appliances were designed around block I/O and a small number of controllers. Under heavy concurrent load the controller pair, its cache and its host ports become the bottleneck, and adding capacity does not add throughput in a linear way.

Object storage, accessed through an S3-compatible API, changes the model. Data is spread across many storage nodes, and each node serves part of the requests. Throughput grows with the number of nodes and drives rather than with the power of a single head unit. That is why software-defined designs, such as Ceph clusters on NVMe drives, are common for this class of workload. The trade-off is operational: you own more moving parts, so you need solid monitoring and a team that understands distributed systems.

2. Hardware building blocks: NVMe, networking and failure domains

NVMe drives connect over PCIe and remove much of the latency and queueing of older SATA and SAS paths. In object clusters they are usually deployed as one storage daemon per drive or per small group of drives. Capacity and drive count per node should be balanced: a few very large nodes make recovery slow and concentrate risk in a single machine, while many small nodes increase the cost of cabling, power and management.

The network matters as much as the disks. Plan bandwidth per node and an oversubscription ratio that fits the expected traffic, separate client traffic from replication and recovery traffic where possible, and adopt RDMA (remote direct memory access, also used as RoCE over Ethernet) only when the team can operate and test it. Define failure domains such as servers, racks and power feeds, so that data placement rules reflect the real physical risks of the room rather than the layout of the spreadsheet.

3. Replication versus erasure coding

Replication keeps full copies of each object. Erasure coding splits each object into data chunks and parity chunks, so the same durability can be reached with less raw space. The table compares common choices. Overhead is the raw space needed for each unit of usable space. A profile with low overhead saves hardware, but it also changes how much work the cluster does when something fails.

Replication is simple and recovers quickly, but it multiplies raw capacity. The 4+2 profile stores six chunks for four chunks of data, which means 1.5 times the usable size, and it survives the loss of any two chunks. The price is CPU for encoding and decoding, and more complex recovery when a node fails, because rebuilding a chunk requires reading several others. Durability figures in marketing material depend on assumptions about independent failures, recovery times and placement. Ask how any headline number was calculated and which failure domains it assumes.

SchemeRaw space per usable unitChunks that can be lostTypical use
3x replication3.0x2 copiesHot data, small objects, fast recovery
Erasure coding 4+21.5x2 chunksBalanced capacity and durability
Erasure coding 8+31.375x3 chunksLarge objects and capacity-oriented pools

4. Data placement and recovery planning

The placement rule decides whether a single rack or power failure can take out an object. A profile with k data chunks and m parity chunks needs at least k+m independent failure domains to place every chunk separately. A cluster with fewer domains than that loses much of the protection the profile suggests, even if the arithmetic looks correct on paper, because two chunks of the same object may share a rack or a power feed.

Recovery time matters as much as the profile. After a drive failure the cluster rebuilds data, which consumes network and disk bandwidth on the surviving nodes. Plan spare capacity so the cluster can heal before another failure occurs, and test rebuilds under realistic load. Slow or disruptive rebuilds are a common surprise during the first real incident, and they are much easier to plan for than to explain afterwards.

5. Operational concerns: monitoring, upgrades and tenancy

A large object cluster needs monitoring of drive health (SMART data), latency percentiles, recovery progress and capacity trends. Alerts should warn while there is still time to add capacity, because a nearly full cluster behaves badly during rebuilds and the margin for error shrinks quickly. Dashboards that show only averages hide the slow requests that users actually notice. Percentile latency and recovery progress show the problem before users complain.

Upgrades are rolling operations that should be rehearsed in a test environment first. Shared clusters need quotas, access policies per bucket and a clear separation of credentials between applications. Object lock, which stops deletion or overwrite until a retention period ends, helps protect backups against ransomware, but it must match retention rules the business has agreed on in advance, because a retention period that is too long can also block legitimate cleanup.

6. How to start: a sizing exercise

Start from the workload, not from the hardware catalogue. Measure object sizes, request rates, the ratio of reads to writes and retention periods. Those numbers drive every later decision, from the redundancy profile to the number of nodes, and they are the only fair basis for comparing offers from different vendors. Without them, every comparison turns into a discussion about marketing claims.

If you do not have the team to run this in-house, ask potential partners to walk you through a design for your own workload, including rebuild behaviour and the procedure for replacing a failed node. Their answers will show more about operational maturity than any datasheet, especially if they can explain what happens to latency during a rebuild. A partner who cannot answer that question has probably not operated a cluster through a real failure.

  1. Profile the workload: object size distribution, request mix, peak throughput and expected growth.
  2. Define durability and recovery targets with the business owners of the data.
  3. Choose the redundancy profile and check it against the number of failure domains you can build.
  4. Size nodes, drives and network from the target throughput, keeping headroom for rebuilds.
  5. Run a proof of concept with representative data and a deliberate failure test.
  6. Document the procurement, support model and upgrade process before going into production.

Frequently Asked Questions

What erasure coding profile is best for enterprise S3 storage?

There is no universal profile. Replication suits hot data and fast recovery. Erasure coding profiles such as 4+2 or 8+3 reduce raw capacity overhead, but they need enough failure domains and enough headroom for rebuilds. Choose the profile from your durability target, the number of racks or sites you can use and the performance you need during recovery.

Is object storage always faster than a SAN?

No. Object storage scales throughput by adding nodes, which suits large parallel reads and writes. For low-latency transactional workloads that depend on block semantics, a SAN or local NVMe may still be the better fit. Measure your own access pattern before deciding.

Do I need RDMA for good performance?

Not always. RDMA can reduce CPU overhead and latency on the storage network, but it adds configuration complexity and needs a network set up correctly, including lossless behaviour where the design requires it. Start with a well-tuned standard network, measure, and adopt RDMA only if the bottleneck justifies it and your team can operate it.

How do I protect backups in object storage against ransomware?

Use buckets with object lock and a retention period that matches your policy, keep separate credentials for backup writers and administrators, and store a copy in a separate failure domain or location. Test restores regularly. Object lock stops deletion within the retention period, but it does not replace access control, monitoring and a documented recovery procedure.

How much free capacity should a cluster keep?

Keep enough free space to absorb the rebuild after the failure you plan for, typically one drive or one node. The exact amount depends on the redundancy profile, cluster size and rebuild speed. Monitor fill levels and treat a steady upward trend as an early warning, not a late emergency.

About this content

General informational guide. Regulations and technical requirements change over time: always check the official sources and assess your specific case with a qualified technical or legal advisor.

GET IN TOUCH

Talk to our team.

We'll show how Altovar integrates into your infrastructure. No commitment, no sales pitch. Just a technical conversation.

BOOK A DEMO →