01Can pilot reports teach a forecast where it's wrong?
Bumpy's forecast is at its best over the US, where the US weather agency NOAA has tuned it against turbulence that airliners actually measure. Leave that airspace and we're estimating turbulence from wind shear on a coarse grid.
So we've started logging every pilot report we can get, about two thousand a day, next to what we forecast for the same spot. Pilots describe the air they actually flew through, which makes each report a small test of the forecast.
Two weeks in, this post covers what we've collected, what's wrong with it, and how we plan to train and test a calibrated model on it. We'll start where today's forecast is weakest.
02The weather models underneath
Every number in Bumpy starts as a forecast from a weather model. Turbulence has several causes, and each forecast catches some of them far better than others.
We use two forecasts:
- Over North America: GTG. NOAA's Graphical Turbulence Guidance (GTG) combines several turbulence measures from a detailed weather model and is tuned against turbulence measured by airliners. It gives separate clear-air and mountain-wave forecasts, and we keep the larger of the two.
- Everywhere else: a GFS estimate. We estimate turbulence ourselves from the winds of the Global Forecast System (GFS), NOAA's global weather model, using the Ellrod-Knapp wind-shear index. It's a fair guide to jet-stream turbulence on a coarse grid, and not much more.
Pick a cause below to see which inputs capture it.
Storms
Thunderstorms are the biggest gap. Neither forecast has a storm component. Storms reach Bumpy only indirectly: through wind shear around large systems, and through reports from pilots who have already flown through the bumps.
We plan four inputs that would add a direct storm signal: storm warnings (SIGMETs, the significant-weather advisories issued to pilots), radar echo tops, lightning, and instability (CAPE, convective available potential energy). They would feed the model and also a separate storm layer in the app, because flights usually route around storm cells rather than through them.
Pilots avoid storms, sensibly, so reports near them are scarce. Storms will need their own evaluation.
03Data: pairing forecasts with what pilots felt
Pilot reports (PIREPs) give a position, an altitude and how rough the ride was. Pilots report smooth air too, and we need those reports as much as the rough ones.
Each hour we match every new report against each forecast run we still hold. We record what that run predicted at the report's exact place and time, plus the strongest forecast nearby. Several runs meet each report, so one report yields several examples, one per run, at different lead times.
Select a run to see the example it produces.
04The dataset so far
Logging began on 1 November 2026. These figures cover the first two weeks, up to 14 November.
Where and when reports come from
The reports follow the traffic, and mostly US traffic at that. Our main source is US pilot reports. The rest of the world shows up mainly through AIREPs, the automated position reports that airliners send on oceanic routes.
Why the data is sparse
Thirty thousand reports is less than it sounds, because of where they are. They cluster on busy US routes in daytime. Oceans, nights and most of Asia, Africa and South America barely appear, which is the wrong way round for us: the forecast we most want to fix covers exactly those places.
The rough end is thin too. Pilots report bumps more readily than smooth air, so smooth air is under-represented. Even so, only 86 reports in two weeks were severe or extreme, and those often came several at a time from one patch of sky.
Some pairs are incomplete. 4,164 have no forecast because the report is below the grid's lowest level. The 2,170 AIREPs carry a flight number where the aircraft type would be; every other report logs the type.
Finally, the leads are short and there is only one global source. Each report is compared only with runs made within about two hours of it. The global product from WAFS (the World Area Forecast System) was unavailable throughout, so every pair outside the GTG area uses the GFS estimate.
How today's forecast scores
For each report we take the newest run made before it, which gives 28,501 fair pairs. The forecast ranks turbulence well but reads low: most moderate-or-worse reports get a forecast below our moderate line.
05Problem formulation
Our forecasts use eddy dissipation rate (EDR), a standard measure of how turbulent the air is, independent of the aircraft flying through it. Pilots use words: smooth, light, moderate, severe and the steps between.
To put both on one scale, we map each word to an approximate EDR for a medium-size airliner. The first model then answers a yes-or-no question: given the forecast and its context, how likely is a pilot to report turbulence at or above a threshold?
Switch the threshold to see which reports count as "yes".
06From the air to the cabin: aircraft size
EDR describes the air, not the ride. The same patch of turbulence feels very different in a four-seat Cessna and in an A380, because heavier aircraft respond less to the same gusts.
That makes every report depend on who filed it. A "moderate" from a regional jet and a "moderate" from a widebody aren't the same observation. Today we ignore this: our forecasts and thresholds assume a medium-size airliner.
The plan is five size classes, following the way air traffic control already groups aircraft by size and wake. The class comes from the aircraft type logged with each report, which the 2,170 AIREPs lack.
We considered a separate model for each class and decided against it. Pilot reports are already sparse, and split five ways the smallest and largest classes would have almost no turbulence reports to learn from. So aircraft class is an input to one shared model instead.
07Features
Each training example starts from the forecast at the report's position, then adds the context that changes how far that forecast can be trusted. We record some of these today. Others need the forecast pipeline to log more of what it already computes.
| Group | Features | Why it matters |
|---|---|---|
| Forecast | Point EDR, strongest EDR nearby, forecast source | The signal itself, plus tolerance for position error and a flag for which product produced it |
| Where | Flight level, latitude, terrain height | Jet-stream turbulence peaks near cruise levels; mountain waves follow terrain |
| When | Lead time, month | Error grows with lead time; the jet stream strengthens in winter |
| Who | Aircraft size class | Converts air turbulence into what the cabin feels |
| Atmosphere (planned) | Wind speed, vertical shear, deformation, Ellrod index, Richardson number | Lets a model learn from the ingredients of turbulence, not only from our final number |
| Convection (planned) | Distance to the nearest active storm cell or convective SIGMET, echo top relative to flight level, lightning density, CAPE | Adds the storm signal that today's forecast inputs mostly lack |
08Modelling approach
Expected gain depends more on the region than on the method. Inside the GTG area, NOAA already calibrates the forecast against turbulence measured on board airliners, so we expect little improvement there. Outside it the forecast is our uncalibrated GFS estimate, and that's where calibration should help most.
The method comes in two steps. A calibration layer comes first, because it needs far fewer rough-air reports than anything more flexible. A small tree model over the features above may follow once there's enough data to support it.
09Calibrating forecast probabilities
Our first model is a calibration layer. It learns a mapping from forecast EDR to the probability of moderate-or-worse turbulence, using isotonic or logistic regression, with one mapping per forecast source and altitude band.
Calibration doesn't move where the forecast puts turbulence, only what its numbers mean. It runs when the forecast grid is built, so nothing downstream has to change. Drag the slider to apply it.
10Where the model runs
The model runs once an hour, when we build the forecast grid. Its inputs are global weather grids, and its output goes into the same tiles the app already downloads.
Training is separate and slower: a weekly fit on the paired reports and forecasts collected so far.
Two rough numbers size the job. Both are back-of-the-envelope estimates, not measurements.
One global run is a 0.25° grid, 22 levels and 25 forecast times, which comes to roughly 570 million cells, and the model scores every one. That sounds heavy and isn't. A calibration table is one lookup per cell and costs almost nothing.
A small tree model over a full run, and each weekly retraining, should take minutes of CPU on the same server.
11Evaluation and rollout
Pilot reports are strongly correlated in space and time: a single weather system can generate dozens of them. A random split would put reports from the same system in both the training and the test set, and flatter the model. We split by time instead, holding out whole weeks.
Metrics and acceptance criteria:
- Detection. Probability of detection (POD), the share of moderate-or-worse reports the forecast flagged, and false-alarm ratio (FAR), the share of its flags where the report turned out milder. Both are taken at the operating threshold.
- Probabilities. Brier score (the mean squared error of the predicted probabilities), area under the ROC curve (AUC, how reliably rough reports are ranked above smooth ones) and the reliability diagram above.
- By region. Every metric is reported separately for the GTG area and the GFS region, and by altitude band.
- Acceptance. A better Brier score than the baseline on held-out weeks in the GFS region, with no regression inside the GTG area.
Slicing the results
A single average hides the cases that matter most, so we break every metric down by forecast source, altitude band, lead time, aircraft class and season.
Moderate-or-worse reports are rare, and some slices will be small. We report how many turbulence cases each slice has. Confidence intervals come from a block bootstrap that resamples whole days, since reports from the same day are related.
Shadow mode before launch
A model that passes offline tests doesn't go straight to users. It first runs in shadow mode inside the hourly job, scoring every run alongside the current forecast. Both are logged against the pilot reports that come in afterwards.
The forecast switches over only when the new model beats the current one on later weeks it has never seen. If it later does worse, we switch the previous model back in for the next hourly run.
12Challenge: forecast lead time
A report can only be compared with the forecast runs we still have when it arrives. We keep the three most recent hourly runs, so in practice every pair falls within about two hours either side of the report.
The app shows forecasts up to about 35 hours ahead, and errors grow with lead time. A model fitted only on short leads would be over-confident further out. Covering longer leads means storing more runs, at roughly 600 MB each.
13Challenge: pilot reports as labels
The dataset section showed how lopsided the reports are: nearly all over North America, skewed toward bumps, thin at the severe end. That's a sampling problem. There's a second problem underneath it, which is that each individual report is noisy.
Label noise
Even a perfectly representative set of reports would be noisy:
- Position. Often approximate, sometimes given relative to a beacon tens of kilometres away.
- Time. Can lag the encounter by several minutes.
- Intensity. A judgment call in a busy cockpit, and the line between light and moderate is fuzzy.
We handle this three ways. The nearby-maximum feature absorbs small position and timing errors. Borderline categories are tested both ways rather than forced to one side. And we prefer simple models until there's enough data for more.
Automated EDR measurements from equipped airliners are a natural extension: they are denser, cover smooth air as well as turbulence, and don't depend on how a pilot describes the ride.
Further out, the phone itself could become a sensor. Its accelerometer measures what a passenger actually feels, in the exact aircraft we're forecasting for. It would have to be strictly opt-in and summarised on the device. We haven't built it, but it's the most direct label for the question Bumpy answers.
14Failure modes we're watching for
- Learning the US and applying it everywhere. Most training data comes from US airspace, where the forecast is GTG. A model can look good overall while learning little about the GFS region it's meant to fix. Per-source metrics and acceptance criteria guard against this.
- Convective turbulence. Calibration can make the numbers more honest, but it can't add skill the inputs don't contain. Today's inputs carry little storm signal and the GFS estimate carries none. Fixing that needs the storm inputs described under The weather models underneath.
- Seasonal drift. The jet stream moves and strengthens with the seasons. A model fitted in autumn may be miscalibrated by spring, which is one reason to retrain weekly and to evaluate by season.
- Upstream changes. When NOAA updates a forecast model, the relationship between its output and real turbulence can shift overnight. Shadow-mode logging makes these shifts visible quickly.
- Over-confidence at long lead times. Until the dataset covers longer leads, the model is evaluated only on short ones, and the app should treat long-range answers with more caution.
None of these argues against building the calibration layer. They're why it goes through shadow mode first and gets judged one region at a time.