Friday, September 27, 2013

A couple excellent resources

I haven't posted anything further since the initial analysis of the rain amounts and model forecasts, but here are two great resources that have been put together.

One is a summary from CIRES in Boulder, that includes information about many aspects of the event:

And one is a new site set up by the Colorado Climate Center that will be a repository for many of the data and analyses that will be conducted about the floods:

Tuesday, September 17, 2013

How well was the 11-16 September Colorado rainstorm predicted?

Anytime a weather event that causes major impacts occurs, meteorologists try to understand how well the event was predicted, what might have gone wrong in the forecasts, and how that information can help to improve future forecasts.  Here's a rough start at this, which will primarily focus on the numerical weather prediction models that are used by human forecasters to issue forecasts and warnings to the public.  Assessing the quality and importance of human forecasts is equally important, but arguably much more difficult, because one must consider the importance of trust, relationships between the forecasters and users of the forecasts, how well the forecasts are disseminated and communicated, and so forth.  This is likely to also be the focus of much research going forward.

A quick warning that this discussion will be just a little bit technical and would benefit from at least some basic familiarity with numerical weather prediction models.  A great resource to find out what the models are and how they are similar and different is the COMET MetEd operational models matrix:  And another opening factor to point out is the overall difficulty in predicting heavy rainfall: precipitation forecasting has been called "the poorest performance area of forecast systems worldwide" (Fritsch and Carbone 2004, Bulletin of the American Meteorological Society).  So keeping this in mind, let's begin...

Medium-range forecasts

There were plenty of indications of a rainy week in Colorado (and much of the western US), several days in advance.  One illustration of this is the 7-day precipitation forecast issued by the NOAA Weather Prediction Center on Monday morning, September 9, for the period through the following Monday, the 16th.  (Note that this is a forecast made by human forecasters, using the numerical models as well as their knowledge of meteorology, etc.)
Fig. 1: WPC 7-day precipitation forecast, issued 6 am Monday 9 September, for the period through 6 am Monday 16 September.
The WPC forecast shows that over 2" of precipitation was predicted to occur over nearly the entire state of Colorado in the next week, with an area of more than 4" over parts of northern Colorado -- exactly the areas that would receive the heavy rainfall later in the week!  The rainfall amounts in this forecast clearly turned out to be too low.  But at this timeframe (a few days before the really heavy rains would start), and considering the mission of the WPC to provide forecasts over the entire country rather than to provide specifics about local areas, this appears to be a very good forecast.  And 4" of rain in a week in Colorado is still something that happens only every few years in most places, so even taking the forecast rain amounts literally would suggest something fairly unusual was on its way.

When forecasters consider the medium range (beyond, say, 2 or 3 days into the future), they use ensemble forecasts -- a series of forecasts that use slightly different initial conditions, or slightly different model configurations.  Ensembles can provide some information about the range of possible weather outcomes, and can increase or reduce confidence depending on how much agreement there is among the members of the ensemble.  The medium-range ensemble that is generally considered the best in the world is that from the European Centre for Medium Range Weather Forecasts (ECMWF).  However, the ECMWF charges a *lot* of money to access their forecasts in real-time, so most US forecasters couldn't have actually seen the EC ensemble prior to this event.  Researchers can access it after the fact, though, and we'll use the forecast from Monday Sept. 9 here as a sort of benchmark for the medium-range forecasts. 
Fig. 2: ECMWF ensemble precipitation forecasts initialized at 1200 UTC (6 am) Monday 9 September, for the 7-day period from then through 1200 UTC Monday 16 September.  The left panel shows the probability of exceeding 50 mm (about 2 inches) of rain, the middle panel 100 mm (4 inches), and the right panel 150 mm (6 inches).  Purple contours show the probabilities at 4%, 10%, and every 10% above that.  Probabilities above 50% are colored.  The thick dashed black line indicates the ensemble mean, and the green contour indicates the observed precipitation at that threshold.  ECMWF forecasts were obtained from the TIGGE portal at
The three panels of this figure indicate that there was a very high probability of 2" of rain over Colorado during the week to come, a decent chance of 4" of rain in northern Colorado and New Mexico, but low probabilities of more than 6" of rain.  (One of the 51 members in the EC ensemble at this lead time predicted a major rain event in northern CO.)

Let's also consider the US version of a medium range ensemble, known as the Global Ensemble Forecast System (GEFS).
Fig. 3: As in Fig. 2, except for the US Global Ensemble Forecast System.
It paints a similar picture: very high probability of 2" of rain, modest probability of 4", and little to no probability of 6".  These are all very consistent with the WPC forecast shown above (and in fact, the WPC likely used the ECMWF ensemble guidance, as well as the GEFS, quite heavily in making their forecasts, as WPC is one place in the US that does have access to the ECMWF ensemble.)  So it appears that at the medium range, the models and human forecasters were pointing to the chance for widespread heavy rainfall in Colorado, with 2-4" of rain likely in a lot of places, though it was too early to be very confident in extremely large amounts. 

1-2 days ahead

Typically, as you get closer in time to the weather system you're trying to predict, you can narrow things down, being more specific about (in the case of rainfall), where, when, and how much. This is true of both the numerical models and the human forecasters.  Here, we'll focus only on the models going forward, and perhaps will revisit the human forecasts later on.

In my opinion, the key forecasts in this event were those issued on Wednesday Sept. 11 -- good forecasts at this lead time could provide adequate time for preparations in advance of the predicted heavy rains on 12th and 13th.  However, rather than narrowing in on a specific outcome for this event, the models showed a fair amount of disagreement.  Some were pointing to extremely heavy rainfall, whereas others had more modest amounts.  Let's look at the forecasts from a bunch of different models from 0000 UTC 11 September (6 pm Tuesday evening).
Fig. 4: 12--36-h precipitation forecasts from models initialized at 0000 UTC 11 September (6 pm MDT 10 September) and valid for the period from 1200 UTC (6 am) 11 Sept. to 1200 UTC (6 am) 12 Sept.  Models are (top row, left to right) GFS, NAM, short-range ensemble (SREF) bias-corrected mean, and ECMWF; (second row, left to right) CSU 12-km WRF members 1-4; (third row, left to right) CSU 12-km WRF member 5, CSU 12-km WRF mean, CSU 4-km WRF, and NAM 4-km nest).  Bottom left corner shows the NOAA Stage IV precipitation analysis, used here as the "truth".  Note: you may want to right-click and "view image" to see the full-size versions of these figures.
Fig. 5: As in Fig. 4, except for the 12--60 hour forecasts covering the period from 1200 UTC 11 Sept (Weds morning) to 1200 UTC 13 Sept (Friday morning).  Note that the color scale is different.
A few of the models indicated heavy rain over north-central Colorado, but certainly not all of them.  Of the main operational models (top row), the GFS was pointing toward the heaviest amounts, but it also had a large, questionable "bulls-eye" out in western Kansas that may have led forecasters to doubt its predictions elsewhere.  The SREF mean (and individual SREF members, not shown) also had an indication of 4+" in the mountains of northern Colorado.  The NAM and ECMWF, however, had their heaviest rains in southern Wyoming and comparatively little rain in Colorado.

Here at CSU, we run several WRF model configurations at 12-km horizontal grid spacing over the western US.  These runs aren't done on a supercomputer, but on modest desktop machines!  However, they rely on the NCEP models like the NAM and GFS for their initial conditions. (More details on these runs can be found here: Anyway, it turns out that a couple of these runs---particularly members 2 and 3---were predicting a lot of rain for northern Colorado.  Member 2 had a maximum amount for this 48-hour period of over 10", which ended up being very close to what was observed.  Interestingly, member 2 is initialized from the GFS, and member 3 from the NAM -- they didn't even share the same initial conditions.  In hindsight, a "blend" of members 2 and 3 would've been an outstanding precipitation forecast, though the other members of the CSU ensemble did not perform nearly as well.  Still, these runs seemed to have a stronger signal of the extreme rains to come than several of the other models did.  I'd like to think that there's something special about the configuration of our runs, but we don't have a good sense yet for whether they got the forecast right for the right reasons or were somehow "lucky".  We plan to look more closely into this.

Often in the modeling world, it's expected that increasing the resolution of the model (i.e., decreasing the grid spacing) will give better results.  And often this is indeed the case -- the more realistically you can represent the atmosphere in the model, the more realistic your forecast should be!  But sometimes it doesn't turn out that way, and at least from this preliminary look that appears to be the case here.  (There are past research papers on cases where higher resolution actually hurt the forecast.)

The two runs that might currently be considered "high resolution" on Fig. 4 above are the 4-km CSU WRF (which is run on a larger computer cluster; info at and the 4-km NAM nest (the 3rd and 4th entries in the 3rd row.)  The CSU run did produce some extreme rainfall amounts -- over 6" -- but this was out over the eastern Plains.  The atmospheric processes that produce heavy rainfall on the Plains are often distinct from those that affect the mountains, so even though this forecast showed large accumulations over the Plains, a forecaster likely wouldn't translate that to mean a potential for extreme rainfall in the mountains.  The NAM 4-km nest precipitation forecast looks broadly similar to its coarser-resolution parent model.  These comparisons aren't exactly apples-to-apples, but at least this preliminary picture shows that a couple of the coarser-resolution runs performed better than their higher-resolution counterparts.  More research will be needed to see if this is a robust result by configuring the model in exactly the same way, except for the grid spacing.

Thus, the model forecasts from 0000 UTC 11 September painted an uncertain picture regarding the potential for extreme rainfall in Colorado -- it appeared possible (and thus warranted close attention), but perhaps not all that likely.  Considering that extreme rainfall is rare in September in this area, it's not unreasonable that a forecaster might discount the "high-end" members of an ensemble in this case; that's essentially what I was thinking on Wednesday as well.

We don't run our CSU ensemble at 1200 UTC (again, because these runs are being done on computers that people need for other things during the day!), but we can look at the other models initialized at that time, and also valid at 6 am Thursday.
Fig. 6: As in Fig. 5, except for models initialized at 1200 UTC (6 am) Sept. 11 for the 0--48-hour forecast.  CSU forecasts are not run for the 1200 UTC cycle, thus they are blank here.
If anything, these runs were showing less potential for extreme rainfall in northern Colorado -- even though the atmospheric ingredients were still in place for heavy rainfall (a subject we'll look more closely at in a future post), these model runs certainly wouldn't have instilled great confidence that a historic event was on its way.

So then, the heavy rains fell on Wednesday night and Thursday morning, initiating the devastating floods in many of the rivers and streams along the Front Range.  However, there was still rain falling, and the chance of more heavy rain on Thursday and Friday that could make the flooding even worse.  For these rains on Thursday into Friday morning, the numerical models appear to have performed quite a bit better:
Fig. 7: As in Fig. 4, except for models initialized at 0000 UTC Sept. 12 for the 12--36-hr forecasts for the period ending at 1200 UTC (6 am) Friday Sept 13. 
Although there are differences in the amounts and the specific locations, there was general agreement that heavy rain would fall both in the mountains and on the plains from Thursday morning through Friday morning.  Again, member 2 of the CSU WRF ensemble performed admirably in this event.  A closer look into what led to the differences in forecast skill between the two days is warranted.

Short range

One numerical prediction system that has revolutionized short-term (i.e., from 0-15 hours) forecasting is the High Resolution Rapid Refresh (HRRR), developed at NOAA's labs in Boulder.  It has 3-km horizontal grid spacing, covers most of the US, and runs every hour out to a 15-hour forecast.  However, this also means there are lots of forecasts to look at and try to interpret. A couple of the HRRR forecasts illustrated the short-term potential for very heavy rain in Boulder County---the 2300 UTC run showed two swaths with more than 3" of rain over the upcoming 12-h period---but the runs immediately after that moved the heavy precipitation out of Boulder County. 
Fig. 8: Animation showing 7 HRRR precipitation forecasts valid at 1200 UTC 12 September.  The first was initialized at 2100 UTC 11 September, and the last at 0300 UTC 12 September.  Note that the precipitation accumulation period is not the same for all of the runs, but all cover the primary period of heavy rainfall in the night of 11 Sept. and early morning of 12 Sept.  Obtained from the HRRR website at:



In this preliminary analysis, one might conclude that the numerical model forecasts of the September extreme rains in Colorado were something of a mixed bag.  There were strong medium-range signals for widespread rainfall in Colorado during the week of 9 September, though there was less indication of the potential for extreme amounts.  At shorter lead times, there remained substantial uncertainty about how intense the rainfall really would be, and even whether it would be focused on the mountains or the plains (or if it would occur in Wyoming rather than Colorado).   The "high-end" rainfall possibilities were generally out of character with the climatology of precipitation in September in Colorado, but so were the atmospheric conditions -- these conditions, and what clues they might have offered in advance that this might happen -- will be the subject of the next entry.

Rainfall amounts from the Colorado rain and flood event of 11-16 September 2013

The state of Colorado has experienced a major disaster in the past week -- prolonged heavy rains fell on a large portion of northern Colorado and resulted in deadly and destructive floods.  The flooding impacted both the steep canyons along the Front Range of the Rockies, including the heavily populated and traveled Boulder Canyon and Big Thompson canyon, as well as the plains, with the South Platte River flowing above its previous record stage for more than four full days.  The photos and videos from the flooded areas are hard to believe, and large numbers of people have needed to be evacuated from mountain areas that had all of their roads washed out.  There's a link on the right that provides some suggestions of organizations you can donate to to support the flood relief effort.

Because most of my scientific research focuses on the understanding and prediction of storms that produce heavy rainfall, and because this event has hit so close to home (both literally and figuratively), I wanted to try to summarize some of what we know about this event so far.

This entry will focus on the rainfall -- where it happened and how it fits into historical context.  The next one will focus on how well it was predicted.  Future entries will cover other aspects of the meteorology, flooding, and impacts of this event.

The rainfall totals, especially from Wednesday night, September 11, to Friday afternoon, September 13, were remarkable.  Boulder County appears to have seen the largest amounts, with numbers that are difficult to comprehend.  The previous wettest September in Boulder had 5.5" of rain -- this storm produced 9.08" one day.  The storm total at Boulder was 17.15" from Monday the 9th through Monday the 16th.  And this was not an isolated event: nearly every automated rain gauge in the foothills and mountains of Boulder county reported more than 10" of rain, with many reporting similar totals to the official station in the city of Boulder.  Some of these numbers come from the excellent summary by Bob Henson of UCAR, which can be read here:

As a preliminary way to put these rainfall totals into historical context, we compared the NOAA Stage IV precipitation analysis, which incorporates radar estimates of rainfall and rain gauge observations, to NOAA's "Atlas 14", which uses statistical methods to estimate the probability of a given amount of rain to occur in a given amount of time at different locations around the country.  These values have recently been updated for Colorado and much of the rest of the country.   For example, the Atlas 14 will tell you, for a point in Colorado, the amount of rain that is associated with having a 0.1% chance of happening in any given year.  Some refer to this as a "1000-year rainstorm," but that terminology is misleading, since it implies that if an event occurs once, it won't happen again for 1000 years.  Instead, what it really is is a very low-probability event, but next year will have that same low probability, and thus it could happen again if the atmosphere brings the ingredients for heavy rainfall together in a similar way.  It's also important to separate the rarity of the rainfall from the rarity of the flooding that resulted.  Based on the above analysis, we can say pretty confidently that this was a "1000-year rainstorm" (or, more correctly, a 0.1% annual probability rainstorm) over a large portion of northern Colorado.  But it's much more difficult to place the flooding in this same perspective yet.  This depends on which river or creek you're interested in, and hydrologists will be doing research to sort out exactly where this flood fits in historical context.

This first map is for the 12-hour period from 6 pm Wednesday (the 11th) to 6 am Thursday (the 12th).  This is the time when the heaviest rain fell in the Boulder area -- some gauges reported over 6" of rain in this 12-hour period.  When comparing to the expected recurrence intervals, we see that the Boulder area, as well as much of Boulder county, experienced 12-hour rainfall amounts that exceeded the "500-year" recurrence interval, with a few areas exceeding the "1000-year" value.  (Remember that these really represent a 1/500 and 1/1000 chance of happening in a given year.) 
NOAA Stage IV precipitation analysis (right) and points where it exceeded various recurrence intervals in the 12 hours ending Thursday morning, September 12, at 6 am. Red X's are where a "1000-year rainfall" occurred, which really means an event that has a 1 in 1000 chance of happening in a given year.
The second map is similar, but for the two-day period from Wednesday morning (the 11th) to Friday morning (the 13th).  After the incredible rainfall rates on Wednesday night and Thursday morning, rain continued to fall through much of the day on Thursday and Friday.  For this 2-day period, nearly all of Boulder County, and a large chunk of southern Larimer County (near Estes Park), had rainfall accumulations exceeding the 1000-year amount!  Not to mention smaller areas near Colorado Springs, the eastern Denver metro area (especially near Aurora up through Brighton), and some locations on the eastern Plains.

NOAA Stage IV precipitation analysis (right) and points where it exceeded various recurrence intervals in the 48 hours from Weds morning (11 Sept.) to Friday morning (13 Sept.). Red X's are where a "1000-year rainfall" occurred, which really means an event that has a 1 in 1000 chance of happening in a given year.
**UPDATE: The NOAA Office of Hydrologic Development has conducted a very similar analysis and has some nice maps and graphs in this PDF:

Now, extreme rainfall is occurring almost all the time *somewhere* in the world, and we've found that, depending on the duration of rainfall you look at, if you consider the United States there are roughly 50 rain events each year that exceed the "100-year", or "1% annual chance" recurrence threshold.  And there is also an important history of extreme rainfall in northern Colorado: the 1976 Big Thompson Canyon flood resulted from a localized, nearly stationary cluster of thunderstorms that developed in the worst possible place: near the top of a very steep canyon.  Similarly, the July 1997 Fort Collins flash flood was caused by a localized, slow-moving group of thunderstorms.

But this event was different in a few important ways:
  • it happened in September, which is not typically a rainy month in northern Colorado.  There have been times with heavy rains in September -- State Climatologist Nolan Doesken recalled a case with over 3.5" of rain in one day in 1938 at Fort Collins -- but the totals in this event were several times that.
  • it covered such a large area.  As noted above, most of what we know about truly extreme rainfall amounts in Colorado come from relatively isolated thunderstorms in late July and early August.  This is what happened in 1976, 1997, and other notable Colorado flash floods.  This event, instead of covering a single canyon or watershed, had extraordinary rains over entire large counties.
  • it lasted a long time.  At most locations, the heaviest rains fell on Wednesday night and Thursday morning, but the entire week was rainy, which exacerbated the flooding problems.
This all added up to a historic week for rainfall in Colorado, which ultimately led to some of the most, if not the most, devastating floods in the state's history.

Now, a couple comments about estimating return frequencies of extreme rain events, and the relationship between the rainfall and the flooding.  You may ask, "how do we know what amount of rain represents a 1000-year event when we only have a little over 100 years of records?"  And you'd be right that we can't know these with perfect certainty with such a short data record.  However, estimating the probabilities of rare events is a very active and robust area of research in statistics.  I don't claim to understand all of them, but there are several methods which can give both a "best estimate" and the error bars around that estimate...and those error bars can be fairly large for the low-probability events.

Finally, the results shown above are somewhat preliminary.  The Stage IV precipitation analysis can be problematic in mountainous areas because it uses radar data, and radar beams are blocked in regions of high terrain.  However, from the rain gauge reports that are available, it appears that if anything, it underestimated the actual amount of rain that fell, so the real story might be even worse.  This is where my colleagues here at CSU are asking for your help!  Even though there are now far more rain gauges around Colorado than in the past---thanks in large part to the CoCoRaHS project that started here at CSU after the 1997 Fort Collins flash flood---there are still many areas that don't have observations.  If you measured rainfall during the event and it wasn't reported to CoCoRaHS, they still want that information!  See here for more details: