01Is two weeks of pilot reports enough to learn from?
Since 1 November we've logged every pilot turbulence report we can get against the forecast we had for that place, height and time. Before training anything on that log, we looked at what's actually in it.
The short version. There is enough data to check and correct today's forecast over North America, but not enough to train a model of our own.
- Few rough reports. 192 are moderate-to-severe or worse.
- Little outside North America. About 700 reports.
- Short lead times. Every forecast was under two hours old. The app looks days ahead.
02About one report a minute
We rebuild the forecast grid every hour. Each build is a run, and we keep the newest three.
Every new report is logged once against each of those runs, so one report becomes three rows.
Logging began on 1 November at about 06:00 UTC. Daily volume swings between about 1,500 and 3,100 reports, depending on the weather and on weekday traffic. 14 November is a partial day.
Reports logged per day
03Where the reports come from
97.6% of reports are over North America. That's the source, not a filter of ours. The data comes from aviationweather.gov, and most of it is US PIREPs (pilot reports): a pilot radios in how the ride feels.
The rest of the world shows up only through AIREPs, automated position reports from an airline's aircraft that sometimes carry a turbulence note. 97% of them come from one airline, United, over the Pacific and Atlantic.
The dense lines on the map are the busy US routes.
| Region (rough boxes) | Reports | Share |
|---|
04What pilots said
Pilot reports are words, not measurements. The pilot picks a category for how the ride felt in their own aircraft.
To compare words with a forecast, we turn each category into a rough EDR for a medium jet, from 0.15 for light to 0.60 for extreme. EDR is eddy dissipation rate, the standard measure of turbulence strength, and higher is bumpier. On the forecast side we call 0.10 light, 0.22 moderate and 0.34 severe.
Smooth reports are logged too, and they make up 27% of the total. A model needs them as much as the bumps. Without them it only ever sees places where something happened.
Reports by what the pilot said
05Altitude: about half are at cruise
A flight level (FL) is altitude in hundreds of feet, so FL350 is about 35,000 ft.
About half of all reports sit between FL300 and FL400, where airliners cruise. The low levels are busy too, with climbs, descents and small aircraft, and that's where smooth reports are far more common. Switch to “Share smooth” to see it.
Reports by flight level
06Aircraft: mostly 737s and A320s
PIREPs record the aircraft's type code, such as B738 for a 737-800. AIREPs record the flight number instead (UAL58, say), so we don't know what they were flying.
We sort the type codes into five size classes, using the same rules the forecast uses to score flights. Anything the rules don't recognise counts as medium.
- Super light: pistons and light turboprops, such as the Cessna 172, Pilatus PC-12 and King Air.
- Light: business jets, such as Citations, Challengers and Gulfstreams.
- Medium: narrowbodies, regional jets and airline turboprops, such as the 737, A320, E175 and CRJ-900.
- Heavy: widebodies, including the 747.
- Super: the A380.
Reports by aircraft size, split by what was reported
07Time of day
Reports follow US daytime traffic. “Local solar time” shifts each report's UTC time by its longitude, so the chart shows the time of day where the aircraft was.
The overnight hours are thin, so a model will be least sure about overnight flights.
Reports per hour
08Lead time: two hours either way, no more
Lead time is how long before the report the forecast was made. We compare each report with the three runs on disk when it's logged, and those range from about two hours older than the report to two hours newer.
A negative lead means the run came after the pilot reported, so it isn't a fair test.
The app shows forecasts for flights hours or days away. The log can't say yet how good a 6-hour or 24-hour forecast is, and that is the number a nervous flyer cares about.
Rows by lead time, in minutes
09How the forecast compares with what pilots felt
Over North America the forecast comes from GTG (Graphical Turbulence Guidance), the turbulence forecast from NOAA, the US weather agency. Everywhere else it's a rougher estimate that we work out from GFS (the Global Forecast System), NOAA's global weather model.
For each report we take the newest run made before it. That gives 28,501 fair pairs, with a median lead of 29 minutes.
The chart shows how forecast EDR is spread for each kind of report. Drag the slider to move the cut-off and see what it would have caught.
It ranks well
Pick a random moderate-or-worse report and a random smooth one. Inside the GTG area, the bumpy one has the higher forecast 79% of the time. That's an AUC (area under the ROC curve) of 0.79. Outside, it's 77%.
So the forecast does know where the air is rougher.
It reads low
At our 0.22 moderate cut-off, only 9% of moderate-or-worse reports inside the GTG area get a moderate forecast. The median forecast at those reports is 0.137.
This is the clearest finding so far, and a recalibration could fix it now. It holds for every size class: the median is 0.137 for mediums and 0.150 for heavies.
One possible cause, not yet tested, is that averaging fine GTG cells into the coarser grid flattens the peaks.
| Reported | Area | Pairs | 10th pct | Median | 90th pct | Nearby max | Forecast moderate+ |
|---|
10What one record looks like
Each record pairs one report with one forecast run. This one is real: a pilot report of moderate turbulence. The raw field is the text the pilot or airline system sent.
11What this sample can and can't support
What it can do now
The sample can recalibrate how forecast EDR maps to smooth, light, moderate and severe, over North America and at short lead. There are about 6,600 moderate-or-worse and 7,400 smooth fair pairs. That's plenty for a simple correction, and it would be the first real test of what the app's words mean.
What it can't do yet
- Longer leads. It can't judge forecasts more than two hours ahead, which is most of what the app shows. The log has to record longer leads first.
- Severe turbulence. 86 severe-or-extreme reports, often several from one patch of sky, are too few to learn from.
- The rest of the world. There are about 700 reports outside North America, nearly all from one airline.
- The seasons. This is two weeks of November. The jet stream, summer storms and winter mountain waves each need their own months.
A trained model starts to make sense after about a year of logging that includes 6–24-hour leads. At today's rate that's about 790,000 reports, about 5,000 of them moderate-to-severe or worse.
12Gaps in the data
- No global turbulence forecast. WAFS (the World Area Forecast System), the global turbulence product, has never loaded. NOAA's public server refused or dropped our requests, and the product has since left that server. So outside the GTG area, every forecast in this log is the GFS estimate.
- Lead time stops at two hours. Checking a day-ahead forecast means sampling each run against reports that arrive hours and days later. A full run is about 570 MB, so the plan is to keep a thinned copy of older runs.
- AIREPs have flight numbers, not aircraft types. That affects 2,170 reports. The plan is to look up the type from the flight number when the report is logged.