What is Sharding?
Also known as: Horizontal Partitioning
Sharding splits a logical dataset across multiple database instances using a shard key, enabling horizontal scaling and fault isolation beyond a single server's limits.
Sharding is a database architecture pattern that horizontally partitions data across multiple independent database servers, called shards. Each shard holds a subset of the data, determined by a shard key a value in each record that maps to a specific shard. This approach allows a database to handle larger datasets and higher throughput than a single server can support.
Sharding works by having a router or proxy layer that directs queries to the appropriate shard based on the shard key. For example, a user ID might be used as a shard key, with users 1-1000 on shard A, 1001-2000 on shard B, and so on. This distribution can be done via range-based or hash-based partitioning. Consistent hashing is often used to minimize data movement when shards are added or removed. The application must be designed to be shard-aware, as cross-shard queries can be expensive and transactions spanning shards require distributed coordination.
In the wider database stack, sharding sits above replication and within distributed database systems. It is a common strategy for NoSQL databases like MongoDB and Cassandra, but also used in some relational databases with middleware. Sharding is distinct from vertical partitioning (splitting by column) and replication (copying data). It provides scalability but adds complexity in maintenance, backup, and operations.
Key facts
- Uses a shard key to determine data distribution across servers.
- Supports horizontal scaling by adding more database nodes.
- Commonly paired with replication for high availability.
- Requires careful shard key selection to avoid hotspots.
- Cross-shard operations and joins are expensive or unsupported.
How it works in practice
Related terms
References
More in Databases
ACID
ACID (Atomicity, Consistency, Isolation, Durability) is a set of properties guaranteeing reliable database transaction processing, ensuring data integrity despite concurrent access or failures.
B-Tree Index
A B-Tree index is a self-balancing tree data structure that maintains sorted data for efficient insertion, deletion, and range queries in database systems.
BASE
BASE is a design philosophy for distributed databases that prioritizes availability and partition tolerance over immediate consistency, making it a looser alternative to ACID in NoSQL systems.
CAP Theorem
CAP theorem states that a distributed data system cannot simultaneously provide consistency, availability, and partition tolerance; it can only guarantee two of the three.
Connection Pool
A managed cache of database connections that applications reuse rather than opening and closing a connection for each query, reducing latency and server load.
Failover
Failover is the process of automatically or manually promoting a replica database to primary when the active node fails, ensuring continued availability.
Foreign Key
A column or set of columns in a database table whose values must match a primary key in another table, ensuring referential integrity between the two tables.
Hash Index
A data structure that maps keys to storage locations using a hash function, providing constant-time equality lookups but no ordered or range scans.
Materialized View
A database object that stores the precomputed result of a query as a table, refreshed periodically or on demand to improve read performance and reduce computational overhead.
NoSQL
NoSQL is a family of non-relational database systems designed for flexible schemas, horizontal scaling, and high-throughput data access that traditional SQL databases cannot easily provide.