PriceStorageManager

Central price data storage and retrieval layer for PriceCollector. Receives real-time price updates, persists them in 10-second binary shard buckets, and serves queries with interpolation semantics tied to a finalization model.

High-Level Design

Data Flow

HyperliquidApiManager            Consumers
  (allMids event, ~1/s)          (PriceAlert, UserPortfolio, AccountValueTracker)
         │                            ▲
         │                            │
┌──────────────────────────────────────────────┐
│        │     PriceStorageManager    │        │
│        ▼                            │        │
│      Buffer                       Query      │
│  ┌────────────┐             ┌──────────────┐ │
│  │ inMemory   │             │              │ │
│  │ PriceData  │────────────▶│ getPriceData │ │
│  │ (live buf) │             │              │ │
│  └────────────┘             └──────────────┘ │
│        │                            ▲        │
│      Flush (every 10s)              │        │
│        ▼                            │        │
│  ┌────────────┐                     │        │
│  │ SQLite     │ ────────────────────┘        │
│  │ price_data │ ◀───────────────────┐        │
│  └────────────┘                     │        │
└──────────────────────────────────────────────┘
         │                            │
         ▼                            │       
   CandleAggregationManager     BackfillManager
   onBucketWritten()            persistSingleTimestamp()

Core Concepts

1. Two-Tier Storage

In-memory tier serves two purposes:

• inMemoryPriceData — Live write buffer. Incoming prices land here first. Complete 10-second buckets are flushed to SQLite; the current (incomplete) bucket stays in memory.
• inMemoryPriceDataCache — Read cache populated on SQLite queries. Capped at 30,000 entries; entries older than 6 hours are evicted on overflow.

SQLite tier stores durable 10-second buckets in binary shard blob format in the price_data table. Each blob encodes all assets' price points for that bucket window.

2. 10-Second Bucketing

Prices arrive every ~1 second but are grouped into 10-second buckets (Math.floor(ts / 10) * 10) for storage efficiency. A bucket is only flushed to SQLite once complete — meaning the clock has moved past it. This ensures each SQLite row represents a full 10-second window.

Flush trigger: Every handleAllMids() call invokes flushPriceData(), which checks for complete buckets and writes them. After writing, it notifies CandleAggregationManager.onBucketWritten() so candle aggregation can process the new data.

3. Binary Shard Blob Format

All SQLite blobs use a compact binary format (magic: 0x50524345 = "PRCE"):

┌──────────────────────────┐
│ Header (12 bytes)        │
│  magic, assetCount,      │
│  dataSectionLength       │
├──────────────────────────┤
│ Asset Index              │
│  per asset: id, count,   │
│  offset, length          │
├──────────────────────────┤
│ Data Section             │
│  per asset: [ts, price]  │
│  records (uint32+int64)  │
└──────────────────────────┘

This format allows efficient per-asset random access without deserializing the entire blob. The same format is used for candle tables (price_candles_1m, price_candles_1h) but with OHLC records instead of single price values.

4. Finalization Model

Not all timestamps are queryable. A timestamp is finalized when BackfillManager has confirmed there are no gaps at or before it:

Condition	Finalized?
timestamp < BackfillManager.minTimestampToCheckForGaps	Yes
timestamp >= minTimestampToCheckForGaps	No
Development env: timestamp < now - 10s	Yes

Why this matters: Consumers call getPriceData([timestamp]). For finalized timestamps with missing data, the manager interpolates from nearby data. For non-finalized timestamps, it returns empty PriceData — a signal that backfill is still in progress and the caller should wait or retry.

This prevents serving stale or incomplete data during DO startup or network interruptions.

5. Interpolation Strategy

When a finalized timestamp has no exact match, interpolation proceeds in two phases:

Phase 1: Bounded bidirectional search (±1s to ±9s)
  Target: T=105
  Search: T=104, T=106, T=103, T=107, ... T=96, T=114
  → Returns nearest match within 10 seconds

Phase 2: Unbounded forward-fill (if Phase 1 fails)
  Backward: SELECT ... WHERE timestamp_bucket <= T ORDER BY DESC LIMIT 1
  Forward:  SELECT ... WHERE timestamp_bucket > T ORDER BY ASC LIMIT 1
  → Handles fresh DOs that only have newer data

This ensures every finalized timestamp returns price data, even during brief collection gaps, while respecting the temporal ordering guarantee.

6. Merge-on-Write

When flushing to a bucket that already has data (e.g., BackfillManager wrote historical data, then live collection catches up), the manager merges:

1. Decode existing blob from SQLite
2. Merge new data in with a sorted merge algorithm
3. New data wins on timestamp collision
4. Re-encode and replace the blob

Both merge and deduplication operations assert sorted input invariants and throw on violations.

Interactions

┌─────────────────────┐           polls for             ┌──────────────────────┐
│ HyperliquidApi      │ ─────── allMids event ────────> │ PriceStorageManager  │
│ Manager             │                                 │                      │
└─────────────────────┘                                 │                      │
                                                        │                      │
┌─────────────────────┐        onBucketWritten()        │   notifies after     │
│ CandleAggregation   │ <────────────────────────────── │   each flush         │
│ Manager             │                                 │                      │
└─────────────────────┘                                 │                      │
                                                        │                      │
┌─────────────────────┐        persistSingleTimestamp() │                      │
│ BackfillManager     │ ─────── writes backfill ──────> │                      │
│                     │ <────── reads cursor, cache ─── │                      │
│                     │ ─────── finalization gate ────> │                      │
└─────────────────────┘                                 │                      │
                                                        │                      │
┌─────────────────────┐        getPriceData() RPC       │ aggregates data from │
│ PriceCollector DO   │ <────────────────────────────── │ buffer and storage   │
│ (exposes as RPC)    │                                 └──────────────────────┘
└─────────────────────┘
        │
        │ RPC calls
        ▼
  PriceAlert, UserPortfolio, AccountValueTracker

BackfillManager has the tightest coupling — it reads lastUpdateTimestamp to know where live data starts, writes backfilled data via persistSingleTimestamp(), reads via queryPriceData() to detect gaps, and controls finalization via minTimestampToCheckForGaps. It also writes directly to inMemoryPriceData and inMemoryPriceDataCache.

CandleAggregationManager is notified per bucket flush and reads from the price_data table to build 1-minute and 1-hour candle aggregates, stored in price_candles_1m and price_candles_1h.

HyperliquidApiManager is the upstream data source. PriceStorageManager subscribes to allMids events during initialize().

Special Handling

Price Transformation

Raw Hyperliquid prices arrive as decimal strings (e.g., "42156.5"). transformAllMidsToTokenPrices() converts them to scaled integers using per-symbol decimal configuration:

"42156.5" × 10^decimals → integer (e.g., 4215650 at 2 decimals)

Uses dynamic symbols from SymbolRegistry when available, falling back to static HyperliquidSymbolInfo. Validates that scaled values don't exceed Number.MAX_SAFE_INTEGER.

Cache Warming on Initialize

On startup, initialize() queries SQLite for the last 6 hours of bucket data and populates inMemoryPriceDataCache. If no recent data exists, it searches the entire DB for the most recent bucket. This ensures the DO is immediately queryable after eviction/restart.

Testing Support

Price schedule overrides allow tests to inject deterministic prices at specific timestamps without a live exchange connection. Supports exact-match and nearest-match (within configurable window) lookups. These overrides are applied as a final step in getPriceData() and getPriceDataRange(), overriding any real data.

Storage Schema

Table	Key	Value	Written By
price_data	timestamp_bucket (10s-aligned)	Binary shard blob (price points)	PriceStorageManager, BackfillManager
price_candles_1m	timestamp (minute-aligned)	Binary shard blob (OHLC)	CandleAggregationManager
price_candles_1h	timestamp (hour-aligned)	Binary shard blob (OHLC)	CandleAggregationManager

See Also

• BackfillManager — Gap detection and recovery, controls finalization cursor
• CandleAggregationManager — Builds candles from bucket data
• HyperliquidApiManager — Upstream price data source