Pipeline

How a number on this site is produced, from state vector to column.

The chain

flowchart TD
    SV["OSN state vectors<br/>S3, ~10 GB per period"]
    CL["Cleaned tracks<br/>osn_tracks_clean"]
    SEG["A8 segmentation<br/>group on icao24, break on a<br/>real callsign change"]
    ASG["Assignment table<br/>(icao24, event_time, track_id)"]
    EXT["track_extents<br/>trk_start, trk_end per track_id<br/><b>from the UNFILTERED table</b>"]
    NM[("Network Manager<br/>~1,181 aerodromes<br/>AOBT_3, ARVT_3, TAXI_TIME_3")]
    AP[("APDF<br/>94 aerodromes<br/>MVT_TIME + BLOCK_TIME")]
    GT["Ground truth<br/>t_off, t_land, aobt, aibt, t_source"]
    OJ["overlap_join<br/>icao24 + interval containment"]
    OFF["Per-flight offsets<br/>off_s, land_s, match_class<br/>+ gt_adep, gt_ades"]
    DET["Detection<br/>left join ground truth to matched"]
    AGG["Per-aerodrome aggregation<br/>percentiles, capture, index"]
    SITE["This site<br/>rankings + one page per aerodrome"]

    SV --> CL --> SEG --> ASG
    ASG --> EXT
    NM --> GT
    AP --> GT
    ASG --> OJ
    GT --> OJ
    OJ --> OFF
    EXT --> OFF
    GT --> DET
    OJ --> DET
    OFF --> AGG
    DET --> AGG
    AGG --> SITE

    classDef cluster fill:#1f4e5f,stroke:#2b6b80,color:#ffffff
    classDef local fill:#3d3d5c,stroke:#5a5a85,color:#ffffff
    classDef source fill:#5a4632,stroke:#7d6446,color:#ffffff
    class SV,CL,SEG,ASG,EXT,OJ,OFF,DET cluster
    class AGG,SITE local
    class NM,AP source

Dark blue runs on the cluster; purple runs anywhere, including CI; brown is reference data from PRISME.

Two edges that carry the whole result

Both are silent failure modes, producing complete, plausible numbers instead of errors, so the diagram is drawn to make them visible.

track_extents comes from the assignment table, not overlap_join

overlap_join clips every sample to [t_off, t_land], so extents taken after it land only inside the interval: an early-starting or late-ending track would be invisible, and a merged track would score near-zero error.

Detection needs ground truth too

overlap_join is an inner join, so a flight OpenSky never saw produces no rows — invisible to every metric computed from matched, including every V1 track study metric. Recovering it needs a left join from the full ground-truth table.

Stages

	period	state vectors	ground-truth flights	arm	produced (UTC)	git
0	2026	153,407,259	93,150	recommended (A8)	2026-08-28 13:09	8077c4a
1	2025	142,943,290	92,338	recommended (A8)	2026-08-28 13:17	8077c4a
2	2024	137,655,927	90,167	recommended (A8)	2026-08-28 13:37	8077c4a

Row counts come from data/_manifest.json, so this page can’t claim a count the run didn’t produce; a stage with no entry renders as unverified.

What runs where

Stage	Module	Runs on	Needs credentials
Ground-truth extract	`scripts/fetch_reference.R`	work laptop	PRISME / ROracle
Mirror reference to S3	`opdi/benchmarks/mirror_reference.py`	cluster	S3
Segmentation + offsets	`scripts/run_offsets.py`	OSN Kubernetes	S3
Aggregation	`scripts/aggregate.py`	anywhere	no
Page generation	`scripts/gen_pages.py`	anywhere	no
Render	`quarto render site`	anywhere, incl. GitHub Actions	no

Only the first three stages need access to anything; from there, the site rebuilds from committed data alone, letting CI republish on every push.