Editor’s note: One of the nice things about writing a blog is that you get to set the rules yourself. In this case I elect to post an article that has nothing to do with sailing. Instead it is an article I wrote about my youth; the high point of my career when I was only 23 years old. Although the subject is sometimes technical you don’t need any technical knowledge to read it and enjoy it.
Nostalgia: The First Full Scope Simulator, Dresden II
By Dick Mills
Simulators IX, Volume 24, No 4, Society For Computer Simulation, 1992
Abstract
The nuclear power plant training simulator industry was born Monday June 17, 1968 in Morris Illinois, with the inauguration of the Dresden 2 simulator. Many in the industry are familiar with "The Little Blue Schoolhouse" where this simulator is housed. It was built by a consortium of three General Electric divisions starting in 1966. The goal was to provide training which AEC would accept in lieu of the requirement that each operator participate in 5 startups or shutdowns every year.
With no good precedents, no real time models, no standards, and with the hottest new box from the GE Process Computer Department, and technology from the Saturn V rocket, the simulator was quite a challenge. In the end, it was delivered ahead of schedule, under budget, approved by AEC, and the event was reported by Walter Cronkite.
This paper is a light hearted memoir of the project, project members, and the times. It should be enjoyable to those familiar with the industry and big software projects.
Introduction
This paper is a highly personal recollection of the project by a, then, very young and innocent member of the project team. It is intended to be amusing. The names of actual people have been changed to protect the innocent.
In the mid 1960's, the nuclear industry was just making the transition from building pilot plants to production reactors. A contemporary problem was operator training. The AEC required each operator to participate in at least 5 startup or shutdown operations per year in order to remain current. As a result, to train and maintain proficiency of 5 shift crews, it was apparent that new reactors would have to spend a lot of their time starting and stopping just for training purposes.
General Electric's customer engineer in the Chicago office lobbied with his G.E., the customer, and with the AEC to build a simulator to alleviate this problem. He was successful, and the initial unofficial agreement was that if the simulator could be built, and was "sufficiently realistic", it could be used to replace up to 3 of the 5 required startups. The reference plant was to be Dresden II, in Morris Illinois. The plant would be commissioned after the simulator. We didn't call it "The Dresden Simulator", but rather "The Nuclear Power Plant Simulator" because there were no others.
Since there was no precedent, regarding how much realism would be sufficient, nor any precedent of doing a power plant model in real time, the simulator was a real challenge. Most engineers considered it flatly impossible. A major vendor of flight simulators was invited to make a proposal, but GE chose instead to build the simulator as an internal project.
Actually, there was a preceding simulator; the simulator for the nuclear ship NS Savannah. I know little about this simulator, except the handed-down stories that it was done in the early 1960's and that it made such a negative impression on “The Admiral” that he was strongly opposed to the idea of simulators, and negative on computers in general, throughout his career. At the time of the Dresden project, news of the NS Savannah case was either unknown or suppressed, and little is still known to the author. It is therefore conveniently disregarded in the race for being first.
The project was scheduled for completion in 24 months at a cost of $2 million. Actually, it was done 2 months ahead of schedule, $0.25 million under budget, and was proclaimed "good enough" by the AEC. It was a success by all counts.
The Cast of Players
The project was done by teams from three corners of the USA. GE Apollo Support Department, (ASD), in Daytona Beach Florida, was the prime contractor. This department had just completed the project for launch control computers for the Saturn V rocket as part of the Apollo program. They assigned about 15 people to the simulator project.
I worked for GE Electric Utility Engineering Operation (EUEO) in Schenectady New York. EUEO was an in-house consulting group within G.E. One senior engineer (Doug), one programmer, and one 22 year old trainee (me) were assigned to the project. Doug was amazing. He seemed to know by instinct all about the equipment, the process physics, and modeling. I learned a lot working with Doug.
GE Nuclear Energy Department (NED) in San Jose California was the customer. They wanted to reserve their best people for actual power plants. However, the simulator was so fascinating, that it attracted the attention of some of San Jose's best, despite the low dollar value of the contract.
The Project Plan
The split of work was easy to understand. We in Schenectady would do "the dynamic model". This included the models of point kinetics, reactor and auxiliaries, turbine, control, and electrical systems. San Jose supplied project management, data package services, and the 3D neutronics model. ASD would do the hardware and all remaining software. We called the ASD part the logic model because much of it involved logic and component interfaces with switches and lamps on the panels.
The project began in the fall of 1966 and was planned to last for two years. Planning and data collection activities filled most of 1966.
In 1967, the three groups worked mostly at home in isolation. In Schenectady, we did our modeling in FORTRAN using a GE simulation program called FACE. We would get one turn-around per day running test cases. At 0800 we would get the results of the previous day's job. We would examine it, make changes, add new stuff, and prepare cards for the next run. By noon, we would usually have the job submitted to be fetched the following morning. Afternoons we worked on other projects.
In November, we shipped FORTRAN listings to ASD. San Jose did the same with the 3D neutronics model. ASD hand translated these listings to assembler. One-time hand translation is not such a bad job. ASD did it, they checked it, and made very few errors. They also interfaced the global variables to the control panels. This part of the job was done with zero defects. It worked flawlessly from the first day.
The final phase of the project, starting in January 1968, was the system integration and test in Daytona Beach. Doug stayed home in Schenectady, and I was appointed to support our part of the project on site. As I'll soon describe, those became some of the most memorable months of my life.
Soul of a New Simulator
Since I was so young and so green, I thought it normal for big projects like this one to be ahead of schedule and under budget. Years later, I read Tracy Kidder's book, Soul of a New Machine. I was immediately reminded of the Dresden project, and understood much better how and why it was so successful. The parallels were especially strong during the last months.
The final 6 months of the project were carried out at ASD in Daytona Beach Florida; not at all a bad place. GE made an arrangement with the King's Inn, a beach front motel. We had a block of beach front rooms for the first 6 months of 1968. We stayed through the motorcycle races, through the Daytona 500 stock car race, and through the college student spring break invasion. At one point I had a half dozen nubile coeds willing to pay me $100 each to share my room. I am married so, needless to say, I declined, but it sure did boost my self esteem.
The work schedule was incredible, although it didn't seem so at the time. We met at 7:30 for breakfast, and would be in the control room by 8:30. An hour for lunch, and 2 hours for dinner, would be our breaks. We would quit around midnight. Then, we would often go out to the bars and for drinks before retiring for 4 to 5 hours of sleep. This is the schedule we used Monday through Saturday. Sundays we didn't start work until noon. One Sunday a school of whales became beached down the way. It was sad.
Most of us were young, but Harold was in his 40s. He complained about how hard it was to keep up with us kids, but we didn't understand him. Harold's job was to observe and to report to the project manager, John, what was going on. I was only 23, and Hertz refused to rent me a car. Harold graciously volunteered to be my chauffeur for the duration. We became great buddies.
Bill was in his fifties. He is an extraordinary man. Bill had been a GE employee for 20 or 30 years, and had lived in each of 14 foreign countries for a year or more. He was a genuine connoisseur, not a phony. Bill was sent out by San Jose. He didn't seem to have any regular duties, but Bill's true function soon became apparent. Once or twice a week, Bill would grab several of us and take us out for a dinner. These dinners were so spectacular and lavish that they made lifetime memories. After these dinners we were not only relieved of stress, but also rested and eager to work. We often went back to work after the dinners rather than going directly to bed.
Bill had a purple Jaguar and a blond wife 30 years younger than he. Soon after the simulator project was finished, Bill and his wife disappeared, abandoning their apartment, its contents, and the Jaguar. Harold claimed to have gotten a letter from them a year later. He said they established a trading post along the upper Amazon in the Emerald Forest. Nobody ever heard from them again.
John was the project manager. He was really mysterious. He stayed in San Jose. I met him there in 1966, and didn't see him again until 1968 at the inauguration in Morris. In the meantime, nobody saw or heard from John except Harold, but Harold never told us what John was saying. Again, the similarities to Tracy Kidder's book are remarkable.
Swelled Head
Personally, my reputation, and lifetime career direction were established on the test floor at ASD. Most of the ASD people were stars from the Apollo program. They were old, (i.e. 25 to 30), while I was only 23 years old. They also had salaries about double mine. I was awed by them. My job was to stand around and wait for troubles to occur with the dynamic model so that I could fix them or call Schenectady. While waiting, I couldn't help observing and learning.
One day, everything hit the fan. Someone discovered a major bug so massive in extent that it would have threatened the whole project schedule. There were about thirty people in the room and they were all shocked and depressed. "Oh no! What will we do now? We're hosed!" Well, the night before I had been reading the computer manual to put me to sleep and I got an idea from what I had read. I suggested a simple fix that would take less than 30 seconds to make. There was long silent pause; then someone exclaimed, "That's brilliant! Let's do it." Everybody turned and stared at me as if I were a zoo animal. I guess what they were staring at was the sight of my head inflating. To this day, I have not entirely recovered from Daytona Beach hubris.
My Mamma Done Told Me
We were troubled with blinking lights. In the turbine EHC part of the panels were a lot of red and green lights showing valve positions. Every once in a while, during steady-state operations, it seemed as if there were a wave of light which would pass across the panel. We all saw it, but it happened so infrequently and so fast that nobody was sure exactly what they saw. We certainly didn't know what was wrong.
This continued for some weeks. Finally one evening, one of ASD's best programmers exclaimed, "Aha! I've got it." He explained that one of the female engineers on the' project, who wasn't there at the moment, had violated one of the cardinal rules of real time programming. She had an array of lights, in in her program she initially set the array to zero, then calculated the ones which should be on. The problem came when the asynchronous panel input/output program, would happen to sample just in the middle the critical block of code.
The programmers all shook their heads in disgust for such an unforgivable error. One of them said, "My mamma done told me; Don't ever use GLOBAL COMMON for scratch." I never forgot that lesson and in all these years I never committed that particular error.
The Super Swing
In April, we began to notice an irritating instability. It was an oscillation which touched all the major variables in the reactor, turbine, feed water and recirc systems. It never lasted more than 30 seconds, and then it would disappear. Inception didn't seem to be correlated with anything we could find. The circumstances made it particularly difficult to find the cause of the oscillation. I tried and others tried too to find the cause without success. Nevertheless, we didn't worry about it because it didn't disrupt the work very much.
About a month before shipment, somebody was getting nervous. San Jose sent out a delegation of their big guns. Nobody said so, but we all knew that their purpose of being there was to fix the problem which we weren't capable of fixing ourselves. It had reached the point where it was jeopardizing the whole project.
During the morning the experts there, the oscillation came and went twice. The experts were looking nervous because they didn't have any more idea of what to do than we did. Furthermore, they didn’t understand our models and they didn’t know how to program this computer. They faced egg on their faces.
Suddenly, I was seized by an impulse. Without explanation, I rushed to the computer and changed the neutron point kinetics time constant from 0.1 to 0.2 seconds. It only took me 5 seconds to make the change. Somehow, via an intuitive guess, I knew that this would fix the swing and it did. I swear that the idea came to me just at that moment. It was not something I was holding back. My good guess also saved the experts from San Jose a lot of trouble and embarrassment. They didn't have to find the bug the hard way, they escaped reading code in binary, and nor did they have to face the prospect that they couldn't fix it. They were certainly grateful. My head swelled another three sizes.
Days In The Sun
The ASD and San Jose guys with the high salaries, got paid overtime for 92 hour weeks including time-and-a-half, double-time, and triple-time. Most pocketed an extra year's pay for these few months. One guy bought a new Corvette for cash with the excess money. It was my misfortune that Schenectady didn't pay overtime at all. At the end of the project, I tried to tell my boss, Doctor K, how hard I worked, and how much pay I missed out on. His offer was, "Take a couple of days off."
Elves, Trolls, And Gnomes
We had no compiler, no assembler, no keyboard, or printer that we could use with the simulator computer. There was an IBM Selectric typewriter but it couldn't be used for programming. Our method of working was to use bound copies of assembler listings, with changes marked in pencil. We made changes in code and data directly, in binary, using the buttons on the computer console. If the change worked, we would snapshot the whole core onto the drum, and pencil in the change in the listing. Once a week or so, the palace elves (I never found out who) would take our listings away to some GE office in Orlando. They would update card decks from the penciled notes and run them through a cross assembler on a big mainframe. We would have fresh listings in the morning. I can't remember a single case where the elves committed even the smallest error of omission or commission. Their nocturnal activities were, to the best of my memory, always flawless.
Behind the main panels was a rack about the size of a refrigerator. It was the panel I/O hardware. Even by today's standards, it was remarkably compact. Especially considering that we had no integrated circuits. It was also very reliable. It worked almost always from the very first day. Once in a while though, we would get some anomalous results. When that happened, a truly mysterious ritual would be acted out. Someone would call "the man". A guy would appear with a box that looked like a Geiger counter. He also had a wand, about 1 meter long, attached to the box. He waved the wand around the cabinet, stuck it into the openings, and presto the problems would be gone for a week or two. I never found out what was real, what was put-on, or what was pure troll magic.
San Jose hired this guy, Gregor, to be the salesman for the simulator time. One day in March 1968, he walked onto the floor. We were targeting an August project completion. Gregor said that he had sold out all the simulator time for the next two years. Great! Then he said that he had forgotten Dresden II, the reference plant. To make amends, he sold them two months at the front end. He hoped that we would be able to accelerate the delivery to be ready by June instead! For a while, Gregor was the project gnome.
Free Memory
The computer we used for the project was a General Electric GEPAC 4020. In 1967, it was brand new and the hottest machine on the block. Its competitors in those days were the PDP/8 and the Link GP/4. The IBM 1800 had not been released. We got 4020 serial number 2. Exxon, in Aruba, was more important than we. They got serial number 1.
The 4020 had no floating point, 24K words of core memory (24 bits per word). We had 130K words of drum, about half the capacity of today's floppy disk. There was no operating system. We started with a "naked" machine. The machine language, called PAL, was simple and easy to learn. As a point of calibration, the computing power we had to work with on the simulator was comparable to the old Apple II computers, or perhaps 2% of the power of a 1992 PC.
[about 0.01% of a 2006 PC]
One day, somebody noticed in the literature that core memory was priced in 8K banks, but that spare parts were listed in 16K banks only. We had paid for 24K, the documents said 24K, but only 16K or 32K was possible. We checked and sure enough, we had been using 32K for several months without noticing.
Despite the fact that we had no operating system, and no FORTRAN, we did have global COMMON. At least we had a contiguous area of memory where we stored all the variables. A copy of this on drum was a snapshot or IC.
We had two cycles for model programs, one Hertz and ten Hertz. For many years to come, our competitors accused the Dresden simulator as being "not real time", even though they used the same frequencies, one and ten Hertz. I never understood the basis of their criticism.
All debugging was done using the switches on the computer console. One could stop the computer, then examine and modify programs and data in binary using the lights and buttons. To make a snapshot, one loaded the A register with the snapshot number and executed a JSR to a program which wrote data to drum. Program changes were all done by "patches". It sounds wild thinking back, but most of us memorized several hundred of the most important program and data addresses in octal. Nine years later I won a bet by correctly recalling the octal address of reactor pressure.
The AEC Inspector
The immediate goal of the simulator project was to win approval for substitution of simulator training for up to 3 of the 5 required startups. The Atomic Energy Commission (AEC) had unofficially indicated that they would approve the idea if the simulator was "good enough". To prove the goodness, we had to get someone from the AEC to come to Daytona Beach to look at it.
That turned out to be a problem because, at the time, Federal employees were allowed $25 per day for room and meals. In Daytona Beach, motel rooms cost much more than $25 per night, not to mention meals, so government employees couldn't afford to come. After some weeks of impasse, Bill made "arrangements", and the AEC inspector arrived. He played with the simulator one morning. In the afternoon we let him pull the rods until he had nuclear criticality.
Pulling critical in real time with real simulated instruments was so much fun it was almost irresistible. We had to repeat it for every visiting engineer from San Jose, and of course we did it for the AEC man. Each of us, including the AEC man, felt as if he were Enrico Fermi himself under the bleachers in Chicago in 1942. In the evening, Bill took him out for dinner. The simulator was pronounced "good enough" to be used for training in lieu of a certain amount of real plant experience. Ignominious as it may seem, this pronouncement was, and still is, the foundation of our simulator industry.
Inauguration Day
The day of the inauguration, June 17, 1968 was a lot of fun. GE invited press and TV reporters from all over the country. They had a tent set up for a big buffet and open bar in the afternoon. The reporters must have known about the party because about 150 of them showed up. The following Sunday, the New York Times gave it about half of a page, and the 3 TV networks each gave it about 5 minutes on the news. Walter Cronkite was the one who told the world "and that's the way it was” that evening to the background sound of an alarm horn from our simulator.
We jammed more than 100 VIPs and reporters into the very small control room for the grand finale. My job was simple. We had one remaining bug which caused an alarm horn to honk all the time. My job in the demonstration was to hold my finger on the alarm silence button to keep it from honking.
The climax came when we did the station blackout malfunction. The room went black as the overhead lights went out. The horns sounded, (I had released the button). Hundreds of green lights suddenly became red illuminating the faces in the room the color of fresh blood. A woman screamed. Everyone else gasped. This was the birth moment of our industry.
After a few seconds, the lights came back on when the simulator had faithfully started the diesel and began restoring loads. When the recirc pumps started, the simulator did just as we had programmed, and beautifully dimmed the room lights simulating low voltage. We were especially proud of this, because controlling fluorescent lights with a computer was a neat trick. The spice was provided by a Commonwealth Edison VP in the back of the room. His angry voice protested, "We have adequate house power. Our lights don't dim!"
The ultimate irony came as I looked over my shoulder. Outside, peeking in the window, were four Dresden operators who were supposedly being trained that morning. It was their training which we were celebrating. There was no room for them to get into the building, so they had boosted themselves up to the window for a peek. This picture of me was taken on that day.
Later in the day, the press retired to the tent for beer and bourbon. I drove back to O`Hare airport and heard the bulletin on the car radio about the assassination of Bobby Kennedy. I should have stayed for the bourbon. The picture below shows me on that day.
The Instructor's Panel
The instructor's control panel was a marvel of human factors engineering. It had a rectangular matrix of buttons. Most buttons had a maintained position, and a local (i.e. not computer controlled) back light to show status.
Each thing the instructor could do had its own dedicated button. Yes, that's right, an individual button for every IC and one for every malfunction. The matrix of buttons served at the same time as a menu, a control device, a monitor of current status, and a limit to how many functions we could simulate. At the bottom of the consoles were a couple of thumb switches used to set the values of external parameters.
Nobody ever asked for training on how to operate the instructor's panel. In Morris, we put it right in the middle of the control room, just a few feet behind the operators. We didn't know about pedagogy, and nobody complained.
The Green Book
While I was doing all these things in Daytona Beach, Doug continued working on documentation back in Schenectady. It was finished about the same time the simulator was delivered.
The result was an enormous book, about 4 inches thick, full of equations, block diagrams, and curves from transient tests. It was obvious to everybody that this was high calorie technology. In combination with its thickness and timeliness, it impressed the hell out of everybody. It was bound in a green leather cover, and it soon became known as "The Green Book." It was marked as strictly proprietary General Electric information.
Nearly a decade later, I had occasion to have contact with a competitor who had delivered some BWR simulators. I was shocked to hear their engineers refer casually to "The Green Book". They were not supposed to have seen it or even heard of it!
My only consolation to having been the victim of industrial espionage was my knowledge that "The Green Book" documented the state of the models eight months before delivery. The changes made and bugs fixed in Daytona Beach during integration were never reported to Schenectady, nor incorporated in "The Green Book". I dreamt, hopefully, of my competitors wasting millions trying to fix bugs that had been fixed before.
Conclusion
A significant point regarding the simulator project never occurred to me until writing this paper. Whose decision was it to authorize so much paid overtime while the project was continuously ahead of schedule? Furthermore, how did the project come in under budget after paying these extra costs for overtime? Maybe the mysterious project manager John, was more Machiavellian than I ever suspected.
In any event, we who earn a living building or owning nuclear power plant training simulators all owe a debt to this pioneering project.
Posts
