Skip to content

Observed climate data

CHART shows observed months (what heat did to health in a past month) from stored data only. Nothing is downloaded or scored when someone opens a place, changes the month, the pregnancy window or the outcome. This page describes how the data gets there, how long it takes and how much space it needs.

Architecture

Copernicus CDS (ERA5 hourly 2 m temperature)
      │  one request per country and calendar year
      ▼
ERA5 cache (raw NetCDF, on the climate volume)  ◄──  public mirror (S3 era5-cache/)
      │  derive every configured place: cell-centre or cell-overlap weights
      ▼
PostgreSQL  district_climate (monthly)  +  district_climate_day (daily)
      │  India: score every live-model area, month and pregnancy window (R)
      ▼
PostgreSQL  prediction_result
      │  read only
      ▼
API  /heat-outlook/{place}/observed  and  /observed/map
      │
      ▼
Dashboard: one observed view for India and Kenya
  • Places come from the published model configuration (MODEL_CACHE_DIR/outlooks/index.json), the same list the dashboard's pickers offer: for India, Madhya Pradesh, its 10 divisions and 52 districts (63 places); for Kenya, the country, 47 counties and 289 constituencies (337 places). A place outside the configuration is never downloaded.
  • Kenya's counties are scored when read, by the calibrated DLNM, from the stored monthly and daily climate. India's divisions and state are scored once, when their months are loaded, for every validated pregnancy window; the dashboard offers the windows Settings leaves on. Districts and constituencies hold climate but have no fitted observed model, so their observed estimate shows as unavailable.
  • New months arrive by the six-hourly top-up sensor, which pulls and scores the newest published ERA5 months (see Data pipeline).

Loading history

Run once per country, then again with more years whenever needed; a re-run only downloads the years it adds.

make climate-backfill COUNTRY=IN YEARS=2     # local
make climate-verify   COUNTRY=IN YEARS=2     # lists any place-month missing

On the sandbox, run it in a one-off container so a deployment cannot interrupt it (docker compose run containers are not replaced by up -d):

cd ~/chart-deploy/aws
docker compose --env-file .env.prod run --rm --no-deps -T chart-dagster-code \
  python -m chart.climate.history backfill --country IN --years 20
docker compose --env-file .env.prod run --rm --no-deps -T chart-dagster-code \
  python -m chart.climate.history score --country IN --years 20

score re-scores stored months only (for example after a model release or a change of validated windows). New installations default to 2 years for one country (India) to try CHART; the sandbox keeps 20 years for both.

Measured on the sandbox

Measured on 30 October 2026 on the sandbox (EC2, 2 vCPU, 7.6 GB RAM), India and Kenya running side by side. Copernicus queue time dominates and varies with load: one two-year round took between 50 seconds and 28 minutes.

History Months Wall time added (per country, both in parallel) ERA5 cache Database Daily climate rows
none – – 38 MB 83 MB 60 k
2 years 33 28 min (India, including scoring) 168 MB 224 MB 0.46 M
5 years 69 45–60 min more 316 MB 367 MB 0.90 M
10 years 129 95 min more 560 MB 606 MB 1.63 M
20 years 249 3 h 15 min – 3 h 25 min more 1,046 MB 1,091 MB 3.09 M
45 years (from Jan 1981, 3 Nov 2026) 549 23.4 h (India), 17.8 h (Kenya) more +1.2 GB 2.93 GB 5.8 M
  • From nothing to 20 years for both countries took about 6 hours of wall time with the two countries running side by side (12:56 to 19:18 UTC, including two rounds lost to a deployment restart). verify then reported all 63 India places and all 337 Kenya places holding all 249 months (January 2006 to September 2026).
  • Raw ERA5 per year: about 20 MB for Madhya Pradesh and 29 MB for Kenya.
  • Database: about 290 bytes per daily row including indexes; 400 places produce about 146,000 daily rows per year.
  • India scoring: 21 area-outcomes (10 divisions and the state for LBW across their validated windows, 10 divisions for under-five), about 0.3 s per stored result; five years took 7.5 minutes.
  • While a backfill runs, both CPUs are busy deriving places from each year's grid. The site stayed responsive (HTTP 200 in under 0.1 s), but SSH sessions occasionally timed out; run long loads outside working hours.

Kenya calibration for every county

Kenya's county estimates shift stored ERA5 onto the model's temperature scale with per-county, per-month offsets (calibration.json in the release). Release v2_2026_10_06 shipped 44 of 47 counties: Mombasa and Vihiga then held no ERA5 cell centre, and Narok's handover rows did not join. Now that every county holds ERA5 for 2011–2020, core/scripts/kenya_calibration_extend.py adds the missing ones from the release's published 2011–2020 typical months and the stored ERA5, after reproducing the published offsets (43 of 44 exactly; Nakuru's handover rows differ from its published outlook, and its published offsets are kept). Added entries are marked derived.

The same script calibrates all 289 constituencies (constituencies in calibration.json), each from its own published 2011–2020 typical months, its own stored ERA5 and its own 1991–2020 under-five reference. Kenya's models are one national curve, and the modellers' constituency projections apply it to each constituency's temperatures; the observed months now do the same, so a constituency shows its own observed estimate instead of "no model". Constituency offsets differ from their county's by 0.8 °C at the median (up to 3.5 °C in hilly counties), which is why each is calibrated separately. Stored ERA5 for 2011–2020 matches the release's ERA5 file exactly for all 45 counties that file covers.

Places without a fitted model

Models are fitted for some levels only: in India the state and its 10 divisions, in Kenya the national curve. A place below them (a Madhya Pradesh district) is scored with its nearest fitted ancestor's curve (its division's) on the place's own ERA5 temperatures, stored as the place's result and labelled "No model was fitted for X; scored with the model fitted for Y". The model registry resolves this (get_active_model_mapping, with inherited_from naming the ancestor), so predictions, the backfill's score step and the observed view all treat the district alike. The planning screens' place lists are unchanged. This mirrors how the modellers' district and constituency projections apply a wider curve to local temperatures.

ERA5-Land map layer

--product era5-land (make climate-backfill PRODUCT=era5-land) pulls Copernicus ERA5-Land, the same reanalysis re-run over land at 0.1° (about 9 km), for the same configured places and stores monthly mean daily maximum, monthly maximum and heatwave days under land_* variable names, without daily rows. Explore shows them as the Temperature and Heatwave days layers (GET /heat-outlook/{place}/climate).

They are deliberately not a model input. The India LBW model was fitted on monthly ERA5 temperatures and Kenya's on the modellers' own 0.1 °C product, reached from ERA5 through calibrated offsets; scoring ERA5-Land would shift every estimate without validation. No scorer, the VRA or the top-up reads land_* variables, and the backfill never scores this product. Moving the health estimates to ERA5-Land needs the modelling team to refit or recalibrate, then a measured comparison.

Sharing the downloads

A maintainer with bucket write access can publish the retained grids so other installations skip the Copernicus queue entirely (first published 6 November 2026: 1,205 files, 6.56 GB; see data loading for storage, capacity and runbooks):

make climate-mirror-publish ERA5_CACHE_DIR=<copy of /opt/chart/data/climate/era5-cache>

The downloader reads ERA5_MIRROR_URL (default https://chart-predictive-models.s3.eu-north-1.amazonaws.com/era5-cache) before Copernicus. Mirrored files keep their cache keys and request sidecars, so provenance is unchanged. ERA5 is Copernicus data under the Copernicus licence, which permits redistribution with attribution.