High message throughput creates a storage scaling challenge. At maximum capacity, yearly data growth can reach ~84 TB—far beyond what traditional "store everything" archive nodes can handle.
Conservative Maximum
Daily messages: 10 million ~230 million
Average message size: ~1 KB ~1 KB
Daily data growth: ~10 GB ~230 GB
Yearly data growth: ~3.6 TB ~84 TB
Traditional archive nodes store everything. That works for Bitcoin (~500 GB total) but not for a high-throughput message chain.
The Sharded Solution
Obsidian solves this with sharded archives: instead of every archive storing all history, nodes store specific epoch ranges:
Each shard group has multiple nodes for redundancy.
How It Works
Epoch Ranges
An epoch is a consensus time period (~6.4 minutes, 32 slots). Shard groups are responsible for ranges:
Shard Group
Epoch Range
Approximate Time Period
Group 1
0 - 1,000
Genesis → Day 4.4
Group 2
1,001 - 2,000
Day 4.4 → Day 8.9
Group 3
2,001 - 3,000
Day 8.9 → Day 13.3
...
...
...
Group N
(N-1)×1000+1 - N×1000
Rolling window
Node Assignment
When you register as a sharded archive:
Choose your shard group (or get assigned)
Download that epoch range from existing archives
Serve queries for your assigned range
Earn rewards proportional to your coverage
Query Routing
When someone queries historical data:
Example: Query for epoch 1,500 → Router maps to Group 2 → Node D, E, or F responds
The network maintains a registry of which nodes serve which ranges.
Benefits
For Node Operators
Benefit
Description
Lower hardware
Store 1/N of total history
Predictable scope
Fixed epoch range, known size
Easier entry
Join without downloading TB of data
Flexible commitment
Run multiple shards as capacity grows
For the Network
Benefit
Description
More operators
Lower barrier = more participation
Better distribution
Geographic and operator diversity
Redundancy per shard
Multiple nodes per range
Scalable
Add shards as history grows
Storage Requirements
Full Archive
Storage depends heavily on actual usage:
Sharded Archive (1,000 epochs)
A sharded archive operator can run on a modest VPS instead of dedicated hardware.
Reward Distribution
Reward distribution depends on the incentive mechanism deployed on the network.
Conceptually, rewards should be proportional to:
The epoch ranges a node serves (coverage)
Reliability/uptime and correct responses
Full archives would earn more per node, while sharded archives would earn proportionally with much lower costs.
Running a Sharded Archive
Step 1: Choose Your Shard
Newer shards typically have fewer nodes = higher rewards per operator.
Step 2: Sync Your Range
Step 3: Register On-Chain
Registration mechanics (if any) are implementation-defined.
Step 4: Serve Queries
Your node automatically responds to queries for its epoch range.
Step 5: Claim Rewards
Shard Lifecycle
As the chain grows, new shards are created:
Event
Time
Shard Created
Genesis
Day 0
Shard 1 (epochs 0-1000)
Range filled
~Day 4
Shard 2 (epochs 1001-2000)
Range filled
~Day 9
Shard 3 (epochs 2001-3000)
...
...
continues forever
Early shards become "historical", frozen data, stable requirements. Latest shard is "active" and still growing until range fills.
Future: Shard Migration
As older shards become less queried, operators can:
Stay: Continue serving historical data (lower query load, steady rewards)
Migrate: Move to newer, more active shards
Expand: Add additional shard coverage
The market determines where operators focus based on query demand and reward rates.
Conservative (10M msgs/day):
Year 1: ~3.6 TB
Year 3: ~11 TB
Year 5: ~18 TB
Maximum throughput (~230M msgs/day):
Year 1: ~84 TB
Year 3: ~252 TB
Year 5: ~420 TB
Fixed: ~30-50 GB per shard
Doesn't grow (historical data is frozen)
Available shard groups:
Group 1 (epochs 0-1000): 8 nodes, well-covered
Group 2 (epochs 1001-2000): 5 nodes, needs more
Group 3 (epochs 2001-3000): 3 nodes, high reward opportunity
...
# Sync only your shard's epoch range
obsidian-archive --shard-group 3 --epochs 2001-3000