ExternalDataSourceManager

Last-resort fallback for missing price data via external oracle networks when cross-location backfill fails.

High-Level Design

Position in the Backfill Pipeline

ExternalDataSourceManager sits at the end of a three-tier fallback chain:

Price gap detected
  ↓
Tier 1: In-memory cache (PriceStorageManager)
  ↓ miss
Tier 2: Cross-location RPC (BackfillManager → other PriceCollectors)
  ↓ miss
Tier 3: External oracle (BackfillManager → ExternalDataSourceManager → Pyth Network)

BackfillManager owns the orchestration. ExternalDataSourceManager only knows how to fetch from external sources -- it has no awareness of tiers 1 or 2.

Data Flow

BackfillManager.backfillGapsBatch()
  │
  │  for each remaining timestamp:
  │
  ├─ fetchExternalPriceData(timestamp)
  │    │
  │    ├─ Age gate: skip if < 10s old (Pyth finalization delay)
  │    │
  │    ├─ Pyth Network HTTP fetch (5s timeout)
  │    │    → returns float prices per symbol
  │    │
  │    ├─ Scale floats → integers using per-asset decimals
  │    │    e.g. BTC $50,000.00 → 50000 (0 decimals)
  │    │         ETH $3,000.50  → 30005 (1 decimal)
  │    │
  │    ├─ Validate: ≥13 symbols (50% of 26 tracked assets)
  │    │
  │    └─ Return PriceData map or undefined
  │
  ├─ On success: BackfillManager persists to SQLite
  └─ On failure: BackfillManager increments attempt counter

Key Concepts

Cascading source fallback -- Sources implement the HistoricalPriceSource interface and are tried in registration order. First success wins. Currently only Pyth Network, but the interface allows adding more sources without changing the core flow.

Finalization age gate -- Pyth Network needs ~10 seconds to finalize prices. The manager exposes MIN_EXTERNAL_SOURCE_AGE_SECONDS = 10 so BackfillManager can distinguish "too recent to fetch" from "genuinely failed" -- the former doesn't count as a retry attempt, preserving the 5-attempt budget.

Price scaling -- External sources return floats, but the system uses scaled integers. Each symbol has a configured decimal count (via HyperliquidSymbolInfo), and scalePrice() converts: Math.round(floatPrice * 10^decimals).

Per-timestamp fetching -- Unlike batched cross-location RPC, external fetches happen one timestamp at a time. This gives fine-grained timeout/error handling per request and avoids one bad timestamp blocking others.

Stateless design -- No SQLite tables, no persistent state. All gap tracking and data persistence lives in BackfillManager. ExternalDataSourceManager is a pure fetch-and-transform layer.

Error Handling

Scenario	Behavior	Counted as attempt?
Timestamp < 10s old	Skip immediately	No
5s request timeout	Abort via AbortController, return undefined	Yes
< 13 symbols returned	Discard as invalid	Yes
Invalid/non-positive price	Skipped during conversion (reduces symbol count)	Depends on final count
Network/parse error	Caught, logged, return undefined	Yes
Testing environment	Return undefined immediately	N/A

After 5 failed attempts per timestamp, BackfillManager marks the gap as permanently failed.

See Also

• BackfillManager -- Orchestrates the backfill pipeline
• PriceStorageManager -- Stores the fetched external data