Observed climate data¶
CHART shows observed months (what heat did to health in a past month) from stored data only. Nothing is downloaded or scored when someone opens a place, changes the month, the pregnancy window or the outcome. This page describes how the data gets there, how long it takes and how much space it needs.
Architecture¶
Copernicus CDS (ERA5 hourly 2 m temperature)
│ one request per country and calendar year
▼
ERA5 cache (raw NetCDF, on the climate volume) ◄── public mirror (S3 era5-cache/)
│ derive every configured place: cell-centre or cell-overlap weights
▼
PostgreSQL district_climate (monthly) + district_climate_day (daily)
│ India: score every live-model area, month and pregnancy window (R)
▼
PostgreSQL prediction_result
│ read only
▼
API /heat-outlook/{place}/observed and /observed/map
│
▼
Dashboard: one observed view for India and Kenya
- Places come from the published model configuration
(
MODEL_CACHE_DIR/outlooks/index.json), the same list the dashboard's pickers offer: for India, Madhya Pradesh, its 10 divisions and 52 districts (63 places); for Kenya, the country, 47 counties and 289 constituencies (337 places). A place outside the configuration is never downloaded. - Kenya's counties are scored when read, by the calibrated DLNM, from the stored monthly and daily climate. India's divisions and state are scored once, when their months are loaded, for every validated pregnancy window; the dashboard offers the windows Settings leaves on. Districts and constituencies hold climate but have no fitted observed model, so their observed estimate shows as unavailable.
- New months arrive by the six-hourly top-up sensor, which pulls and scores the newest published ERA5 months (see Data pipeline).
Loading history¶
Run once per country, then again with more years whenever needed; a re-run only downloads the years it adds.
make climate-backfill COUNTRY=IN YEARS=2 # local
make climate-verify COUNTRY=IN YEARS=2 # lists any place-month missing
On the sandbox, run it in a one-off container so a deployment cannot interrupt
it (docker compose run containers are not replaced by up -d):
cd ~/chart-deploy/aws
docker compose --env-file .env.prod run --rm --no-deps -T chart-dagster-code \
python -m chart.climate.history backfill --country IN --years 20
docker compose --env-file .env.prod run --rm --no-deps -T chart-dagster-code \
python -m chart.climate.history score --country IN --years 20
score re-scores stored months only (for example after a model release or a
change of validated windows). New installations default to 2 years for one
country (India) to try CHART; the sandbox keeps 20 years for both.
Measured on the sandbox¶
Measured on 30 October 2026 on the sandbox (EC2, 2 vCPU, 7.6 GB RAM), India and Kenya running side by side. Copernicus queue time dominates and varies with load: one two-year round took between 50 seconds and 28 minutes.
| History | Months | Wall time added (per country, both in parallel) | ERA5 cache | Database | Daily climate rows |
|---|---|---|---|---|---|
| none | – | – | 38 MB | 83 MB | 60 k |
| 2 years | 33 | 28 min (India, including scoring) | 168 MB | 224 MB | 0.46 M |
| 5 years | 69 | 45–60 min more | 316 MB | 367 MB | 0.90 M |
| 10 years | 129 | 95 min more | 560 MB | 606 MB | 1.63 M |
| 20 years | 249 | 3 h 15 min – 3 h 25 min more | 1,046 MB | 1,091 MB | 3.09 M |
| 45 years (from Jan 1981, 3 Nov 2026) | 549 | 23.4 h (India), 17.8 h (Kenya) more | +1.2 GB | 2.93 GB | 5.8 M |
- From nothing to 20 years for both countries took about 6 hours of wall
time with the two countries running side by side (12:56 to 19:18 UTC,
including two rounds lost to a deployment restart).
verifythen reported all 63 India places and all 337 Kenya places holding all 249 months (January 2006 to September 2026). - Raw ERA5 per year: about 20 MB for Madhya Pradesh and 29 MB for Kenya.
- Database: about 290 bytes per daily row including indexes; 400 places produce about 146,000 daily rows per year.
- India scoring: 21 area-outcomes (10 divisions and the state for LBW across their validated windows, 10 divisions for under-five), about 0.3 s per stored result; five years took 7.5 minutes.
- While a backfill runs, both CPUs are busy deriving places from each year's grid. The site stayed responsive (HTTP 200 in under 0.1 s), but SSH sessions occasionally timed out; run long loads outside working hours.
Kenya calibration for every county¶
Kenya's county estimates shift stored ERA5 onto the model's temperature scale
with per-county, per-month offsets (calibration.json in the release).
Release v2_2026_10_06 shipped 44 of 47 counties: Mombasa and Vihiga then held
no ERA5 cell centre, and Narok's handover rows did not join. Now that every
county holds ERA5 for 2011–2020, core/scripts/kenya_calibration_extend.py
adds the missing ones from the release's published 2011–2020 typical months
and the stored ERA5, after reproducing the published offsets (43 of 44
exactly; Nakuru's handover rows differ from its published outlook, and its
published offsets are kept). Added entries are marked derived.
The same script calibrates all 289 constituencies (constituencies in
calibration.json), each from its own published 2011–2020 typical months, its
own stored ERA5 and its own 1991–2020 under-five reference. Kenya's models are
one national curve, and the modellers' constituency projections apply it to
each constituency's temperatures; the observed months now do the same, so a
constituency shows its own observed estimate instead of "no model".
Constituency offsets differ from their county's by 0.8 °C at the median (up to
3.5 °C in hilly counties), which is why each is calibrated separately. Stored ERA5
for 2011–2020 matches the release's ERA5 file exactly for all 45 counties
that file covers.
Places without a fitted model¶
Models are fitted for some levels only: in India the state and its 10
divisions, in Kenya the national curve. A place below them (a Madhya Pradesh
district) is scored with its nearest fitted ancestor's curve (its
division's) on the place's own ERA5 temperatures, stored as the place's
result and labelled "No model was fitted for X; scored with the model fitted
for Y". The model registry resolves this (get_active_model_mapping, with
inherited_from naming the ancestor), so predictions, the backfill's
score step and the observed view all treat the district alike. The planning
screens' place lists are unchanged. This mirrors how the modellers' district
and constituency projections apply a wider curve to local temperatures.
ERA5-Land map layer¶
--product era5-land (make climate-backfill PRODUCT=era5-land) pulls
Copernicus ERA5-Land, the same reanalysis re-run over land at 0.1° (about
9 km), for the same configured places and stores monthly mean daily maximum,
monthly maximum and heatwave days under land_* variable names, without
daily rows. Explore shows them as the Temperature and Heatwave days layers
(GET /heat-outlook/{place}/climate).
They are deliberately not a model input. The India LBW model was fitted on
monthly ERA5 temperatures and Kenya's on the modellers' own 0.1 °C product,
reached from ERA5 through calibrated offsets; scoring ERA5-Land would shift
every estimate without validation. No scorer, the VRA or the top-up reads
land_* variables, and the backfill never scores this product. Moving the
health estimates to ERA5-Land needs the modelling team to refit or
recalibrate, then a measured comparison.
Sharing the downloads¶
A maintainer with bucket write access can publish the retained grids so other installations skip the Copernicus queue entirely (first published 6 November 2026: 1,205 files, 6.56 GB; see data loading for storage, capacity and runbooks):
The downloader reads ERA5_MIRROR_URL (default
https://chart-predictive-models.s3.eu-north-1.amazonaws.com/era5-cache) before
Copernicus. Mirrored files keep their cache keys and request sidecars, so
provenance is unchanged. ERA5 is Copernicus data under the Copernicus licence,
which permits redistribution with attribution.