San Onofre: MHI document release by NRC and what it really means

On Friday, late in the afternoon (but not late enough to bypass all of the media cycle) the NRC released documents that it had requested from Mitsubishi Heavy Industries, or MHI, regarding the San Onofre steam generator situation.  This sort of document has never been released to the public before, and there is certain to be wide confusion over what it really says.  I'm going to try to make the important points as clear as possible.  However, the material is very technical, and since some very prominent people (Senator Barbara Boxer and Representative Ed Markey, to name two specifically) have decided that they are able to review such reports and come to solid technical conclusions, this post will be fairly involved so that the error of their conclusions becomes clear.

Click here to see the MHI report and supplement.

Root Cause - In-Plane FEI or Fluid Elastic Instability

The root cause of this problem is essentially not a design flaw per se, nor even a miscalculation.  It is the basic, fundamental belief that in-plane fluid elastic instability  in a vertical U-tube steam generator is not possible if out-of-plane instability has been guarded against.  This principle is described several times in the linked MHI material.  For example:

RCA (Root Cause Analysis) pg. 16:  "The practice in the nuclear industry at the time the SONGS RSGs were designed was to provide measures to preclude out-of-plane FEI in the U-bend region, which was based on the understanding set forth above.  Reflecting this industry practice, the Japan Society of Mechanical Engineers' "Guideline for Fluid-elastic Vibration Evaluation of U-Bend Tubes in Steam Generators" states that in-plane FEI does not need to be considered if out-of-plane FEI is controlled."  (APR note:  SONGS = San Onofre Nuclear Generating Station; RSGs = Replacement Steam Generators.)

We can find a repeat of this information on page 61 of the supplement and references elsewhere.

What this means is that the designers of steam generators have essentially used a Maginot Line principle in preventing out-of-plane fluid elastic instability... which was supposed to inherently then preclude any chance of in-plane fluid elastic instability, which leads to serious tube-to-tube wear.  Singularly focused effort to prevent out-of-plane FEI has led to the situation in which in-plane FEI did actually occur through a set of very complicated circumstances which, when investigated, eventually implicates practices of manufacturing tolerances, fitting, calculation and even design variance between the two units' steam generators.  (On that last point, manufacturing lessons learned in making the steam generators for Unit 2 were then applied to Unit 3's steam generators, leading to better manufacturing tolerances and more precision but actually less of the contact forces in support structure that would have prevented in-plane FEI induced tube vibration from leading to excessive wear.)

Mitsubishi delivers its assessment of the whole root cause of the failures at San Onofre on page 22 of the RCA, and I fully and totally agree with what they've said.  Here is the text of that section:

"Summary-  Thus, the organizational and programmatic Root Cause for the In-plane FEI as set forth in this RCA is the insufficient programmatic requirement to assure effective AVB contact force to control in-plane FEI under high localized thermal-hydraulic conditions (steam quality {void fraction}, flow velocity and hydrodynamic pressure).  The underlying reason for this insufficiency is that the MHI SONGS RSG design did not consider the phenomenon of in-plane FEI because contemporary knowledge and industry U-tube SG operation experience did not indicate a need to consider in-plane FEI."  (APR note:  AVB = Anti-Vibration Bar.)

What does this mean?  Well, for starters, here is a really important thing to understand:  This phenomenon - that is, in-plane Fluid Elastic Instability - has never actually been seen in any operating steam generator before.  Never.  It's a phenomenon which is known about, as you can see, but which has always been prevented in every other steam generator (with, as the report says, perhaps the exception of those at one single other plant) by just what you've read above --- which essentially says "if you prevent A you can't get to B."  It turns out that you can, if enough other small things either go wrong or are miscalculated in the process.  As usual in the nuclear industry today, when something goes wrong, it takes a gigantic number of small items all put together to make it go wrong. 

In the case we are looking at with SONGS' steam generators, it turns out that a variety of factors has led to the supports and parts that are intended to align the U-tubes inside the steam generators not being able to provide the amount of proper restraining force in the proper directions. 

Now, many have heard about the statement (which came out a while back) that MHI used incorrect assumptions in its computer codes when calculating various parameters for these steam generators.  This error appears to have led to an underestimation of the void fraction -- or how dry the steam gets -- and this directly leads to fluid elastic instability.  So the steam generators can make dryer, higher velocity steam in the zone of the U-tubes than calculated originally.  Still, this would be far less of a problem if the support structures (AVB's or Anti Vibration Bars, primarily) were sufficient.

I am pointing out the fundamental assumption that in-plane is prevented if out-of-plane is prevented as the basic cause because that assumption leads to omitting any serious consideration, investigation, experimentation into the "what if" occurrence of in-plane FEI.  Had MHI (or any other builder, really) been forced to conduct such an investigation every time, this whole scenario may have been prevented.  Industry standards have always said "not required," and finally after all these years, and after Mitsubishi having made something like 90 or more steam generators since 1970, in-plane FEI finally did appear. 


What about the other kinds of vibration and failures?

We can see pretty clearly that variation in manufacturing the steam generators for the two units led to far higher failure rates at Unit 3 than Unit 2.  The Unit 2 steam generators were made first, Unit 3 later; tolerances were tightened up on Unit 3's parts, meaning just that variances in all parts of a given type were smaller.  What this actually appears to have done is reduce some of the contact between U-tubes and supports in Unit 3, allowing more range to vibrate.  If one reads the report, all the way through the conclusions and remedies, it's clear that Mitsubishi understands how to fix the problem in future manufacturing.

The Big Question:  Did Anyone Know Before?

Quite highly publicized have been allegations by Senator Barbara Boxer and Representative Ed Markey -- two persons very well known as anti-nuclear -- that Southern California Edison and/or Mitsubishi knew the steam generators were defective before they were installed.  Articles have been printed to this effect; of course, that all stems from this letter that Boxer and Markey sent to NRC Chairman Macfarlane.  (Of course, Macfarlane responded to that the next day with this letter.)

I seriously question the ability of two elected officials to be able to properly analyze any such technical report; thus, either Senator Boxer and Rep. Markey have attempted this themselves and failed quite badly, or else they've had it done for them.  However, it's quite clear from reading the report that the portion that they have quoted in their letter (and press releases, and articles to the press for wide publication) is taken deliberately out of context, and then deliberately (and slanderously) misinterpreted. 

On page 48 of the RCA (the first document) we see the assessment by MHI that while the design of replacement SG for SONGS required close scrutiny (because these were essentially custom units not built to an existing MHI prior design) there was believed to be very sufficient margin in all aspects for safety.  Two quotes on this page stand out:

"The Team concluded that the SONGS design was significantly more conservative than previous designs in addressing U-bend tube vibration and wear."

"..MHI did compare the SONGS RSG design with previous steam generator designs, and in particular did a detailed evaluation of different options of the AVB design taking into account other large steam generator designs."

So we see two things here:  First, MHI considered its design conservative and safe.  Second, MHI did in fact compare other designs in building the SONGS steam generators -- and all that previous experience (as I showed above) says "prevent out-of-plane and you're safe on in-plane."  MHI at the time of delivery of the units thought it had IMPROVED the Unit 3 steam generators, in fact.

Let's look at another quote, this time from page 56 of the supplement to support that:

"Based on experience gained in the fabrication of the Unit 2 RSGs, additional precision was incorporated into the fabrication of the Unit 3 RSGs to more efficiently implement the effective zero gap, uniformity and parallelism of the U-bend assembly."

To reiterate, we now see that MHI felt it had conservative (safe) design margins, and actually had BETTER steam generators for Unit 3.  This led to the following sentiment at MHI after the shipment of all four new steam generators to San Onofre (also from pg. 56 of the supplement):

"In October 2005 the AVB Design Team agreed that the RSG design was optimized for the SONGS application.  At the time of shipment of the SONGS RSGs it was believed that they had greater margin against U-bend tube vibration and wear than other similar SGs."

Now we can see clearly that MHI felt that it had developed replacement steam generators for San Onofre which were world-class in their design prevention of tube vibration, based on all the years of experience MHI has had in fabricating steam generators for PWR-type nuclear power plants.

So what are Boxer and Markey calling out as "proof MHI and SCE knew the design was bad?"  Well, there are portions of the report in which MHI attempts to detail some of the design process for these steam generators.  As some may know, there is a requirement covering nuclear plant replacement parts (known as 10CFR50.59) that requires the parts be what we might call "like for like" or else exceedingly close replacements.  If they're not, license amendments are required.

In fact, SCE got TWO -- that's right, folks -- TWO license amendments for these steam generators.  You can see them both -- click here and also click here.  None further was deemed necessary; the NRC signed off on the steam generator design as being sufficient to satisfy 10CFR50.59.  Southern California Edison fully believed the steam generators were sufficiently well designed to preclude any such problems.  Click here for a large press release on just this topic by SCE.

During the design of any such large piece of equipment a variety of considerations are made - essentially a "we could do this, or do that" kind of process to finalize everything --- materials, thickness of those materials, size and number of components and so on.  Decisions have to be made about all of these things together as a harmonious unit in order to both get a workable (safe, efficient) design and also to satisfy regulatory requirements.  Some very excellent designs might not work -- they might not fit, or they might be too heavy.  They also might vary enough from the original design, or accepted designs, that a major further revision and many further license amendments would be required.  Of course, you could come up with a simple and inexpensive design (a low end option) that isn't good enough and is too small .. just to point out a range of ideas.  What appears to be the case (to this writer) is that MHI dialed in a design that it felt was superior in its conservative approaches to other steam generators, fit the size-weight-power-durability envelope dictated by the plant as it was originally built and operational life requirements and did not have so many design alterations that more license amendments were required.  This is an exceedingly simple way to explain a very highly complicated and technically involved situation, but it works well as a very basic description.

As we see from the links above, not only did Southern California Edison agree, but so did the NRC.

The Mitsubishi RSGs were delivered with a warranty.  Payments on the warranty by Mitsubishi to the tune of $45 million have already been made, and this has been publicly reported after SCE reported the payment to the state regulator. 

NOTHING in what you've just seen hints that anyone at MHI, SCE or NRC "knew they were bad."  None of it.  This is at best a rookie interpretation of a technical report by someone who has not done this before.  At worst, it's slanderous.

Mitsubishi Heavy Industries has a decades-long history of constructing nuclear power plants.  It first worked as a partner in a license agreement with Westinghouse, and later went on its own.  According to sales information here, Mitsubishi has constructed 24 PWR nuclear plants in Japan, and of these 20 were designed, manufactured and constructed by MHI as the prime contractor (the earliest 4 were Westinghouse nuclear plants built outside Japan and imported.)  The only nuclear plants allowed to start up post-Fukushima and thus now operating (Ohi 3 and 4) in Japan are Mitsubishi PWR plants.  Also, Mitsubishi built the first steam generator ever made in Japan (for Mihama Unit 2 in 1970) and since then has built over 90 steam generators either for brand new nuclear plants or for replacement of older types. 

Perhaps more importantly, MHI is right now still in the process of attempting to get its highly modern US-APWR nuclear plant licensed for operation here in the United States, for operation at North Anna and Comanche Peak (prospective builds.)   The false allegations by Boxer / Markey could damage Mitsubishi's reputation, affect these projects, and negatively impact its business. This is deeply disturbing - that unqualified persons in public office have requested public release of a type of report never previously released, and then deliberately misinterpreted the findings in order to achieve a political end (the shutdown of San Onofre) -- also possibly damaging the business of a private corporation.

One would hope I'm wrong about Boxer and Markey's agenda.  I'd love to know who interpreted this MHI report for them.  I'd like to believe they've been misled.  I'm willing to read a retraction from them on their allegations that parties involved deliberately supplied known bad equipment.

Is there hope?

Sure there is.  Some people are telling the politicians to shut up and get out of the process.  One piece in the U-T San Diego was titled "Let Engineers, Not Politicians, Decide San Onofre's Fate."   Another such piece, also recent, was titled "San Onofre Grinding Toward Restart - Despite Carping."  I hope people are coming to realize that the prospect of shutting down nuclear plants and then also stopping import to California of fossil-generated power means hard questions about where to get the power that California needs.  They're for sure also realizing, as we see from the two linked local articles, that there is a whole lot of political grandstanding going on.

With any luck, the engineers will figure out the solutions, the politicians will get out of the details and step back into their elected roles, and the plant will be allowed to restart Unit 2 with a new 70% power cap for testing to see if the FEI is avoided, as is now believed, completely at that level of power.  If that test occurs and proves out to be valid and sustainable, it's conceivable that the plants could get back on the grid and assist with system stabilization and improve reliability -- to say nothing of beginning to recover monies spent by providing electricity and revenue therefrom.

--------------------------------------------------------

BACKGROUND / SUPPORTING INFORMATION

I have been covering this developing situation for a long time.  Here is some supporting information in case you've just found this blog.  First, a definition...

"Redacted" -- Some of the MHI report has been removed because it contained proprietary information -- information that could compromise MHI's business.  Those portions removed are hinted at in the report by empty brackets.  It concerns manufacturing details and design features that MHI must have patented, and doesn't want others to use.  After reading the report,  I'm not worried that  any redacted material might have changed my conclusions.

Steam Generators - San Onofre / Links

Click here to see a good APR primer on Steam Generators. 

Click here to see a more detailed post on Steam Generator design - with guest Meredith Angwin.

A recent SCE Press Release includes details on SCE's analysis of the tube wear (see links.)

Details of the restart plan for Unit 2 for testing are given in this press release.

Recent polling shows majority believes nuclear is required and San Onofre is safe.

SCE Information Package on San Onofre -- Facts vs. Myths.  Worth a look.

SCE directly addresses Boxer / Markey letter.

NEI Chief Nuclear Officer letter which ran in LA Times:  Click here

I will be keeping a close eye on this situation -- as I have from the beginning -- and providing updates on this site as they're warranted, in addition to the normal press releases from involved parties.

11:20 AM Eastern 3/10/2013
ATOMIC POWER REVIEW