anomaly-atlas
Honesty doctrine. Every candidate anomaly is an artifact until proven otherwise; in-sample results are never findings; past statistical regularity does not imply future returns. This is research on statistical properties of market data — not investment advice, not a trading system.

Research / research/data_source_profile.md

Data source profile

reviewed

Data source profile · created Tue Aug 11 2026 20:00:00 GMT-0400 (heure avancée de l’Est) · updated Tue Aug 11 2026 20:00:00 GMT-0400 (heure avancée de l’Est) · Simon-Pierre Boucher

Data source profile — hfmarketdata.io#

Phase 0.5 output (Experiment A). Everything below was established empirically against the live API on 2026-08-12, through the single client (src/anomaly_atlas/data/hf_client.py); every probe is cached and recorded in data_manifest/index.jsonl. Raw evidence: results/expA_data_reality/20260812T054515Z/results.json.

1. API shape#

OpenAPI-documented FastAPI service, open (no auth), JSON/CSV:

Endpoint Purpose
GET /v1/status full dataset inventory (assets × timeframes × adjustments × ticker counts)
GET /v1/{asset}/tickers ticker lists (search, limit)
GET /v1/bars/{asset}/{ticker} OHLCV bars (timeframe, adjustment, start, end, order, limit, format)
GET /v1/bars/{asset} same, multi-ticker (tickers=A,B,C)
GET /v1/snapshot/{asset} last bar ≤ instant at for each ticker (cross-section)
GET /v1/options/{quarters,tickers,chain,expirations,history} daily options chains

Bar row schema: {ticker, datetime, open, high, low, close, volume} — index bars have no volume field. Errors: 404 unknown ticker/asset, 400 invalid timeframe. end is an exclusive instant (a bare date means midnight — start=D&end=D returns nothing; use end = D+1 day).

2. Coverage and history depth (verified per class)#

Asset Tickers (1min) Earliest 1min Latest seen Adjustments
stock 7 670 2000-01-04 09:30 (AAPL) 2026-08-07 19:59 UNADJUSTED, adj_split, adj_splitdiv
etf 5 161 2000-01-03 09:31 (SPY) 2026-08-07 19:59 UNADJUSTED, adj_split, adj_splitdiv
futures (continuous) 131 2008-01-02 06:00 (ES) 2026-08-06 23:59 contin_UNadj, contin_adj_absolute, contin_adj_ratio
futures_contracts ~14.5k archive + 2.5k live — — archive / update
index 123 2008-01-02 09:30 (SPX; 1day back to 2000-11) 2026-08-07 16:50 none
fx 79 2010-01-03 17:00 (EURUSD) 2026-08-07 16:59 none
crypto 74 2013-04-01 (BTC) 2026-08-09 23:59 none
options ~6 000 underlyings/quarter 2010_q1 2026_q3_partial (67 quarters) n/a

Timeframes: 1min, 5min, 30min, 1hour, 1day — 1-minute is genuinely served (verified), it is the finest granularity. Note: UNADJUSTED intraday exists only at 1min and 1day for stock/etf; 5min/30min/1hour exist only adjusted.

Data lag: this is a periodically refreshed archive, not a live feed. On Wednesday 2026-08-12, equities/fx/index ended 2026-08-07 (previous Friday), futures 2026-08-06, crypto 2026-08-09. Design experiments accordingly — no same-day data.

3. Timestamp semantics (critical for lead-lag work)#

  • Timestamps are US/Eastern wall-clock strings with no timezone marker (YYYY-MM-DD HH:MM:SS). Evidence: FX week runs Sunday 17:00 → Friday 16:59 (the classic ET convention); equity sessions run 04:00–19:59.
  • Bars are bar-start labeled (first RTH bar 09:30, last extended bar 19:59).
  • Session windows on the probe day (Thu 2026-08-06): stock/etf 04:00–19:59 (pre + RTH + post), futures ≈ 24 h, fx 24 h (ET week), crypto 24/7, index SPX 09:30–16:20 (settlement prints after the close).
  • These are vendor-consolidated last-trade bars at minute resolution. Assume vendor time ≈ exchange time, but cross-asset closes are non-synchronous by construction (16:00 auction vs 16:20 index prints vs 24 h sessions) — a built-in source of spurious lead-lag (see artifact taxonomy).

4. Missing data: bars exist only where trades occurred#

  • No zero-volume placeholder bars anywhere (0 found on probe day).
  • Liquid names are near-complete in RTH (AAPL & SPY: 390/390 RTH minutes) but sparse pre/post (AAPL: 904 of 960 extended minutes).
  • Illiquid names are sparse even in RTH: AIZN printed 38 bars in the whole day. A last-observation-carried-forward join makes such series look autocorrelated and cross-predictable — this is the stale-price artifact, to be neutralized explicitly (Experiment B).

5. Daily-bar semantics (do NOT mix carelessly with 1min)#

For AAPL on 2026-08-06 (adj_splitdiv):

  • Daily bar = RTH-only OHLC with the official auction close: daily close 312.41 vs 312.49 for the last 1min RTH bar — the closing auction is not in the 1min series.
  • Daily volume includes consolidated/auction volume absent from 1min bars: daily 46.14 M vs 34.67 M (sum of all extended-hours 1min) vs 25.69 M (RTH 1min only). Any volume-based signal must pick one convention and stick to it.

6. Corporate actions / adjustments (verified on AAPL 4:1, 2020-08-31)#

Series 2020-08-28 close 2020-08-31 close
UNADJUSTED 499.23 129.04
adj_split 124.8075 (= 499.23/4) 129.04
adj_splitdiv 121.06 125.1654
  • Split arithmetic is exact.
  • adj_splitdiv re-bases the entire history to the dataset build date — even the 2020-08-31 close differs from its traded price. Adjusted series are therefore not point-in-time stable: they change whenever a new dividend occurs. For any experiment sensitive to this, use UNADJUSTED + explicit adjustment, or freeze the cache (which our client does by design).

7. Options#

Daily granularity: one row per contract per trade_date, 17 columns — bid/ask, last_price, bid_iv/ask_iv, delta/gamma/vega/theta/rho, open_interest, volume, strike, expiry, call_put. 67 quarters (2010_q1 → 2026_q3_partial). SPY had 35 listed expirations on 2026-06-15. Old quarters may carry null Greeks (documented server-side NaN handling). This is the only bid/ask information anywhere in the source — options spreads may inform equity cost models (Experiment G), with care.

8. Limits, performance, credits#

  • Hard cap: 50 000 rows per JSON response (requested 1 000 000, got 50 000). The client paginates on the last datetime; verified on a full year of SPY 1min: 218 006 rows, 0 duplicates, strictly ascending.
  • No rate-limit headers, no 429 observed at gentle sequential rates (~5–7 req/s). Latency: 0.08–0.22 s small requests, ~1.7 s per 50 000-row page. There is no credit system — but the client throttles anyway (min 0.15 s between requests) and caches everything; the cache is the reproducibility anchor.
  • format=csv exists (not exercised in Experiment A; JSON + local parquet cache is our path).

9. Consequences for the research design#

  1. Universe: ample for Q1–Q3 — thousands of stock/etf tickers × 26 years × 1min. Holdout years are affordable.
  2. Artifact taxonomy seeds (→ artifact_taxonomy.md): stale/missing minutes; non-synchronous session ends across assets; auction close absent from 1min bars; daily-vs-intraday volume mismatch; rolling adjustment anchor; continuous-futures splicing method (3 variants exposed — good: the choice is testable).
  3. No bid/ask on bars: spreads must be estimated (Roll model et al.) — the bounce null (Experiment B) is mandatory, not optional.
  4. Minute data is the floor: intraday lead-lag finer than 1 minute is untestable here; hypotheses must respect that (Epps effect at 1min scale).
  5. Data ends ~the previous Friday: "recent regime" claims are bounded.