The Weather Report
X-Plane can be configured to model real-world weather conditions. It does this by downloading weather report data from a server, operated by laminar research . That server, in turn, periodically fetches three total files from two different sources at NOAA: one for METAR-format weather condition data, and another source for two grib2-format files describing, separately, winds and turbulence. Internally, we tend to refer to this system, overall, as Live Weather, or, in the context of anything server-related, just Weather.
Around March 22nd, it broke down.
What Went Wrong
As is often the case: several things.
X-Plane began reporting that it couldn’t fetch all the data it needed for real-world weather. This was a little surprising since our weather server is supposed to cache the latest data it received, even if it’s outdated, to smooth over any connectivity or availability issues with NOAA’s servers. Further investigation revealed that only one of the three files it needed was both missing from its local cache, and unavailable from NOAA.
The root cause of that data being absent from cache hasn't been tracked down, for the reason that that part of the system has been replaced (see below). There may be a bug in the caching logic. The server may have suffered a silent crash-and-restart some time after NOAA stopped serving the file. We're not sure.
Investigation into the causes of the missing data was sidetracked for a little while because, right around the same time, NOAA happened to suffer a widely-reported, significant disruption to their networks and server operations. Naturally, we initially assumed this was the cause of the problem, and our efforts were mainly aimed at mitigating the harm of a temporary outage, not dealing with a longer-term problem.
In fact, what had happened was a longer-term problem, and the outage was, at most, only part of the cause of the disruption we were seeing...