Sharded Archive Nodes

The Scaling Challenge

Obsidian is designed for massive message volume:

Projected daily messages: 9+ million
Average message size: ~1 KB
Daily data growth: ~9 GB
Yearly data growth: ~3.3 TB
5-year projection: ~16 TB

Traditional archive nodes store everything. That works for Bitcoin (~500 GB total) but not for a high-throughput message chain.

The Sharded Solution

Instead of every archive storing all data, Obsidian divides history into epoch ranges and assigns them to shard groups:

┌─────────────────────────────────────────────────────────────┐
│                    Complete Chain History                    │
├─────────────┬─────────────┬─────────────┬─────────────┬─────┤
│  Epochs     │  Epochs     │  Epochs     │  Epochs     │     │
│  0-1000     │  1001-2000  │  2001-3000  │  3001-4000  │ ... │
├─────────────┼─────────────┼─────────────┼─────────────┼─────┤
│  Shard      │  Shard      │  Shard      │  Shard      │     │
│  Group 1    │  Group 2    │  Group 3    │  Group 4    │ ... │
└─────────────┴─────────────┴─────────────┴─────────────┴─────┘
        │             │             │             │
        ▼             ▼             ▼             ▼
   ┌────────┐    ┌────────┐    ┌────────┐    ┌────────┐
   │Node A  │    │Node D  │    │Node G  │    │Node J  │
   │Node B  │    │Node E  │    │Node H  │    │Node K  │
   │Node C  │    │Node F  │    │Node I  │    │Node L  │
   └────────┘    └────────┘    └────────┘    └────────┘

   Each shard group has multiple nodes for redundancy

How It Works

Epoch Ranges

An epoch is a consensus time period (~6.4 minutes, 32 slots). Shard groups are responsible for ranges:

Shard Group	Epoch Range	Approximate Time Period
Group 1	0 - 1,000	Genesis → Day 4.4
Group 2	1,001 - 2,000	Day 4.4 → Day 8.9
Group 3	2,001 - 3,000	Day 8.9 → Day 13.3
...	...	...
Group N	(N-1)×1000+1 - N×1000	Rolling window

Node Assignment

When you register as a sharded archive:

1. Choose your shard group (or get assigned)

2. Download that epoch range from existing archives

3. Serve queries for your assigned range

4. Earn rewards proportional to your coverage

Query Routing

When someone queries historical data:

Query: "Get messages from epoch 1,500"
         │
         ▼
┌─────────────────────────┐
│   Archive Router        │
│   epoch 1500 → Group 2  │
└─────────────────────────┘
         │
         ▼
┌─────────────────────────┐
│   Shard Group 2 Nodes   │
│   Node D, E, or F       │
└─────────────────────────┘
         │
         ▼
      Response

The network maintains a registry of which nodes serve which ranges.

Benefits

For Node Operators

Benefit	Description
Lower hardware	Store 1/N of total history
Predictable scope	Fixed epoch range, known size
Easier entry	Join without downloading TB of data
Flexible commitment	Run multiple shards as capacity grows

For the Network

Benefit	Description
More operators	Lower barrier = more participation
Better distribution	Geographic and operator diversity
Redundancy per shard	Multiple nodes per range
Scalable	Add shards as history grows

Storage Requirements

Full Archive

Year 1: ~3.3 TB
Year 3: ~10 TB
Year 5: ~16 TB
Growing forever...

Sharded Archive (1,000 epochs)

Fixed: ~30-50 GB per shard
Doesn't grow (historical data is frozen)

A sharded archive operator can run on a modest VPS instead of dedicated hardware.

Reward Distribution

Rewards flow based on coverage:

Archive Pool: 700 OBS (daily example)

Distribution:
├── Full Archive Nodes: Get share of ALL epochs they serve
└── Sharded Nodes: Get share of THEIR epoch range

Example:
  - 10 full archive nodes (100% coverage each)
  - 50 sharded nodes (10% coverage each, different shards)

  Effective total coverage: 10×100% + 50×10% = 1500%

  Per full node: 700 × (100/1500) = 46.67 OBS
  Per sharded node: 700 × (10/1500) = 4.67 OBS

Full archives earn more per node, but sharded archives earn proportionally with much lower costs.

Running a Sharded Archive

Step 1: Choose Your Shard

Available shard groups:
  Group 1 (epochs 0-1000): 8 nodes, well-covered
  Group 2 (epochs 1001-2000): 5 nodes, needs more
  Group 3 (epochs 2001-3000): 3 nodes, high reward opportunity
  ...

Newer shards typically have fewer nodes = higher rewards per operator.

Step 2: Sync Your Range

# Sync only your shard's epoch range
obsidian-archive --shard-group 3 --epochs 2001-3000

Step 3: Register On-Chain

await archivePool.registerSharded(
  nodeId,
  shardGroup: 3,
  epochStart: 2001,
  epochEnd: 3000,
  { value: stakeAmount }
);

Step 4: Serve Queries

Your node automatically responds to queries for its epoch range.

Step 5: Claim Rewards

await archivePool.claim();

Shard Lifecycle

As the chain grows, new shards are created:

Genesis
  └── Shard 1 (epochs 0-1000) created

After ~4 days
  └── Shard 2 (epochs 1001-2000) created

After ~9 days
  └── Shard 3 (epochs 2001-3000) created

... continues forever

Early shards become "historical" — frozen data, stable requirements. Latest shard is "active" — still growing until range fills.

Future: Shard Migration

As older shards become less queried, operators can:

• Stay: Continue serving historical data (lower query load, steady rewards)

• Migrate: Move to newer, more active shards

• Expand: Add additional shard coverage

The market determines where operators focus based on query demand and reward rates.

---

Next: Running an Archive Node — Technical setup guide