AceTrader Docs

Appearance

CrawlerConfigManager ​

Persists and manages crawler configuration settings in SQLite with in-memory caching.

Purpose ​

CrawlerConfigManager owns the configuration that controls how the SymbolRegistry crawler operates: how many top symbols to consider (topN), how frequently to crawl (crawlIntervalMs), and whether crawling is enabled at all.

High-Level Design ​

Write-Through Cache Pattern ​

The manager uses a dual-layer storage model — an in-memory CrawlerConfig object backed by a SQLite key-value table (crawler_config). This avoids SQLite reads on every alarm cycle while guaranteeing persistence across DO eviction.

                    ┌─────────────────────────┐
  updateCrawlerConfig()  ──▶  Validate ranges  ──▶  Update in-memory config
                    │                                      │
                    │                              saveConfigValue()
                    │                                      │
                    │                                      ▼
                    │                              INSERT OR REPLACE
                    │                              into crawler_config
                    └─────────────────────────┘

  initialize()  ──▶  loadConfig()  ──▶  SELECT * FROM crawler_config
                                              │
                                     rows empty? → keep DEFAULT_CRAWLER_CONFIG
                                     rows found? → parse into this.config

On initialization, SQLite rows are loaded into memory. If the table is empty (first boot), the in-memory defaults are used without any explicit null checks — the load loop simply doesn't execute.

How Config Drives the Alarm Cycle ​

CrawlerConfigManager doesn't schedule alarms or run the crawler itself. It provides config values that SymbolRegistry's alarm handler consumes:

SymbolRegistry.alarm()
    │
    ├─ config = crawlerConfigManager.getConfig()
    │
    ├─ if config.enabled:
    │      crawlingManager.crawlAndSync(config.topN)
    │
    └─ setAlarm(Date.now() + (config.enabled ? config.crawlIntervalMs : 15min))
  • enabled = true: Crawler runs every crawlIntervalMs, fetching topN symbols from CMC/CoinGecko. Reconciliation also runs.
  • enabled = false: Crawler is skipped, but the alarm still fires every 15 minutes for reconciliation (which always runs regardless of this flag).
  • Mid-cycle config changes: Take effect on the next alarm, not the current one.

Validation Constraints ​

FieldMinMaxRationale
topN10200Too few misses symbols; too many wastes API quota
crawlIntervalMs5 min24 hoursPrevents API rate limits and stale data
enabled——Boolean, no validation needed

Non-Atomic Partial Updates ​

Updates accept Partial<CrawlerConfig>. Fields are validated and persisted one at a time. If field B fails validation after field A succeeded, field A is already saved. This is intentional — each field is independent and the partial update behavior is useful for admin operations.

Edge Cases & Error Handling ​

  • Missing SQLite rows on first boot: falls back to DEFAULT_CRAWLER_CONFIG defaults.
  • Returns a defensive copy of the config object to prevent external mutation.
  • SQLite stores all values as TEXT; type coercion (parseInt, === "true") happens on read.

See Also ​