SAN, HCI or NVMe: Choosing Private Cloud Storage Layer

Most private cloud specifications open with a host count and a CPU model, and that is exactly backwards. The storage architecture you pick determines how many hosts you need, how many NICs you burn, how far the platform scales, and what you replace first at refresh. Get storage virtualization right and the rest of the design falls into place. Get it wrong and every later decision, from switch port counts to application availability, inherits the mistake.

This post compares three tiers on the trade-offs a UK buyer actually feels in operations, not on benchmark numbers: block SAN, with its predictable shared storage and separate fabric; HCI, which buys simplicity by spending CPU and RAM on data services; and direct-attached NVMe, which wins on latency and cost per usable terabyte but pushes availability up into the application. We will work through why the storage layer comes first, what virtualization means in this context, the fabric tax, refresh economics, and a decision framework you can take to your next spec meeting. If you want to talk it through with someone who is on hand to answer questions, that conversation is worth having before the spec is signed.

Why the Storage Layer Comes Before Hosts, CPUs and Switches

As set out above, that order is backwards.

Data services are not free compute. Replication, erasure coding and deduplication all consume CPU cores and memory that would otherwise run workloads; erasure coding’s parity calculation in particular is CPU-intensive. The same VM estate therefore needs more or larger hosts depending on the tier.

Networking follows the same logic. A block SAN demands a dedicated fabric with HBAs, switches and zoning, covered in our guide to how a storage area network works. HCI wants fast east-west bandwidth for replication and rebuild traffic. Direct-attached NVMe needs very little east-west traffic but pushes replication into the application or hypervisor.

Refresh economics differ too. Arrays and fabrics do not age at the same rate as compute, so storage-first thinking creates a second budget line, a second support contract and a second skills requirement.

Block SAN, HCI and direct-attached NVMe are all legitimate answers; the right choice depends on your skills, availability model and scaling pattern.

Storage Virtualization, Network Virtualization, and What Virtualization Means Here

One idea underpins everything that follows. Virtualization abstracts physical compute, storage and network resources behind a software layer, so workloads stop being tied to one box. That is why a storage choice ripples outward: once capacity is abstracted, host, adapter and switch design change with it.

Storage virtualization pools physical media, whether local NVMe, array shelves or HCI node drives, into a logical layer that presents LUNs, volumes or datastores. Thin provisioning, snapshots, clones and replication ride on that layer, and each consumes CPU, RAM or bandwidth somewhere.

Network virtualization does the same for paths. Overlays, virtual switches and software-defined networking split or share segments logically; VXLAN, for instance, exists to provide overlay networks in virtualized data centres accommodating multiple tenants (RFC 7348). Storage traffic still travels that wire.

The two are entangled. Choosing iSCSI, Fibre Channel or NVMe over Fabrics is a topology decision as much as a protocol one, and east-west replication is a switch port decision. A storage-only conversation does not exist.

Plainly, virtualization moves complexity out of cables and into software. Someone still has to run, patch and troubleshoot it, and that software’s appetite for cores, memory and bandwidth is the first question in choosing server hardware for hypervisor and private cloud.

Block SAN: Predictable Shared Storage and the Fabric Tax

A storage area network (SAN) keeps a deliberate share of that complexity in hardware. Dedicated controllers and shelves present LUNs to many hosts over Fibre Channel, iSCSI or NVMe over Fabrics, with multipathing failing over when a path drops.

Where it wins: predictable shared capacity across large host counts, mature snapshot and replication features, array failure domains you can place in a different rack or room from the hosts, and easier workload mobility because capacity is not trapped in a chassis.

The fabric tax: switches, HBAs, optics, cabling and zoning, plus storage administrator skills, controller firmware cycles and a support contract that runs on the array’s clock rather than the servers’. Hosts can be smaller and carry no meaningful local capacity, but each needs dual-port connectivity, and fabric design becomes a storage project, not a server project.

Fair framing for UK buyers: SAN remains the right answer where shared capacity, mature data protection and multi-host consistency outweigh the cost of a second fabric and a specialist skill set. It is a commitment, not a default. If you are weighing it up, we are happy to talk through host count, port counts and headroom before you commit.

HCI: Simplicity You Pay For in CPU and RAM

HCI removes the fabric but not the cost. Node-local drives are pooled by distributed software into a shared datastore, with redundancy handled by replicating or coding data across nodes, not inside an array.

Where it wins: one support surface, expansion by adding a node, and a practical path for teams without a dedicated storage specialist.

The cost that never appears on the quote is CPU and RAM. Data services consume cores and memory you would otherwise sell to workloads, so measure cost per usable terabyte and cost per running VM after that overhead.

Treat drive-cost claims carefully. Vendor-originated comparisons have put server-class drives at several times less than the same drives in a dedicated array, a range from a vendor blog cited by Storage Switzerland, typical not measured.

HCI dictates more, smaller nodes carrying local drives, an east-west network sized at design time for rebuild and replication traffic, and expansion one node at a time. Peer-reviewed analysis of HCI in enterprise data centres lists node expansion, data localisation and network bandwidth among the scalability issues.

One disclosure: Scale Computing, an HCI vendor, is a supplier to this publisher, so weigh favourable HCI commentary, including our own HCI solutions. If that simplicity is the draw, why choose Scale HCI is worth answering against your own workload profile, and against what the overhead costs you per running VM.

Direct-Attached NVMe: Latency and Cost Per Usable Terabyte, Availability Pushed Up the Stack

Direct-attached NVMe removes the array and the fabric from the data path. Flash sits in the host, so every read and write takes the shortest route to the media.

Where It Wins

Low latency, high IOPS per pound, the most capacity per rack unit, no HBA budget and no fabric to design, zone or license.

Direct-Attached NVMe: Latency and Cost Per Usable Terabyte, Availability Pushed Up the Stack

The Catch

Availability moves up the stack. Software RAID, synchronous replication, clustered applications or erasure coding become the protection mechanism, not the array’s job.

Operational Reality

Failure domains, rebuild times, backup windows and recovery tests become host or software tasks. Your team must run and prove that model. Erasure coding incurs high repair costs because rebuilding one failed coded block reads more data than the block held. Anyone who has only relied on an array should test it before committing.

What It Dictates

Fewer, larger hosts per workload class is an option, no dual-port HBA requirement, sharper dependence on application-level design and operational maturity, plus a different NIC mix for management, backup and any synchronous replication. If you want to talk through where availability should sit, our blog covers more.

Side by Side: What Each Tier Does to Host Count, NICs and Headroom

Taken together, each tier changes host count, ports and headroom differently:

An illustrative example: 40 VMs, 20 TB usable, three-way replication, so about 60 TB raw before metadata, spare and filesystem overhead. Host vCPU, RAM and port counts depend on hardware, so treat this as shape, not sizing. SAN: fewer, smaller hosts, each dual-port fabric connected. HCI: more nodes, east-west ports for rebuild and replication. DAS NVMe: fewer, larger hosts, no HBA, extra NICs for replication and backup. Illustrative; substitute your own. No primary latency or cost-per-usable-terabyte measurements underpin this.

Count ports and skills before drives; they bind first. See our guide to buying Intel or AMD servers for a UK business in 2026. Energy, floor space and staff availability matter; no UK price, residency or colocation figures are quoted, and this analysis is general.

Refresh Economics: What Gets Replaced, When, and by Whom

Ports and skills tell you what a design costs on day one; refresh economics tell you what it costs every three to five years.

In a SAN design, compute and storage refresh on separate clocks: the array and its support contract run to their own schedule while servers cycle independently. HCI couples them at the node level, and direct-attached NVMe couples them hardest, because the drives leave with the chassis.

Expansion headroom deserves the same honesty. Controller headroom is usually cheaper than a node, until you reach an array ceiling you cannot scale out of, at which point a forklift upgrade replaces node-by-node growth.

Then price the recurring side: fabric skills and array support in one column, a single HCI support surface in another, host-level software operations in the third.

Second-life options stretch a two-clock budget. This publisher supplies refurbished servers available on site, and retrofit hardware support can keep the compute tier productive while storage is replaced.

The question that exposes a weak design: what happens at year three when you need 30 percent more capacity but not 30 percent more compute?

A Decision Framework to Take to Your Next Spec Meeting

Those refresh-clock questions become a short set of decisions you can take into the room. Work through these five in order, because each answer narrows the next.

Where should availability live? In the array, the cluster, or the application? Block SAN places it in the array, HCI spreads it across nodes, and direct-attached NVMe pushes it into software. That single answer usually eliminates one or two tiers outright.
Can you keep the skills? Fabric zoning and HBA multipathing are specialist work; if you buy them in, budget for the contract, not just the first project. HCI removes the fabric but adds distributed storage software your team still patches, monitors and troubleshoots.
How many hosts in three years? That figure, not today’s, sets switch ports and the east-west bandwidth that replication and rebuild traffic will consume.
Do compute and storage grow together? Capacity outpacing compute favours a shared array; parallel growth suits node-by-node expansion.
What is the cost per usable terabyte? Work it out after replication or erasure coding and after the CPU and RAM reserved for data services, rather than from the raw drive line on the quote.

Those answers belong on page one of the spec.

Conclusion: Storage First, Everything Else Follows

The order of operations is fixed: storage tier first, then hosts, NICs and headroom.

Match the tier to where you want availability to live and to the skills you can genuinely keep in house or under contract, not to whichever option had the best story on a slide. If your team has never run application-layer replication, direct-attached NVMe is a commitment, not a shortcut.

The Scale Computing disclosure made in the HCI section applies throughout.

If the trade-offs still feel finely balanced, we keep refurbished servers available on site and offer guidance to organisations evaluating enterprise server strategy or running Supermicro BBS infrastructure. The team is on hand to talk through host count, NIC and storage questions before you commit.

SAN, HCI or NVMe? Choose Private Cloud Storage First