Moon Lander: How We Developed the Apollo Lunar Module (34 page)

“Whaddaya mean,
your
warehouse?” he shouted in his thick New York accent. “It’s a
Grumm’s
warehouse, and my boys are comin’ in!”

Slamming the phone on the hook he stormed over to me and unleashed an outraged tale of how he had located an ideal area in a clean, lightly used warehouse that was already under the administrative control of the corporate Quality Control Department, but the proprietor, one of his own Inspection peers and colleagues, had the temerity to think he could deny use of the needed space to Dinny. I encouraged him to use my name wherever it might help, and after a few more calls to the Quality Control Department hierarchy Dinny secured a clean new home for LM Receiving Inspection.

Fighting Our Way Through

Parallel, round-the-clock testing of several spacecraft pushed all of us to our limits of endurance and capability. NASA rightly insisted upon rigorous discipline of all assembly and test activities. Nobody could touch a lunar module or its associated GSE without prior approved documentation by work orders, drawings, EOs, test preparation sheets, or test plans, and each step had to be witnessed and validated (“stamped”) by a Quality Control inspector. Piles of paperwork were generated, executed, witnessed, and filed on every shift, resulting in the oft-repeated comment, “We don’t need rockets to fly to the Moon—we’ll just pile up all the paper and walk there.”

S/CAT management was vulnerable to sniping because Wright and I, and our spacecraft team managers and supervisors, did not always hear about problems as they occurred on the assembly floor or in the three ACE rooms. In most cases this was as it should be, since the people who could directly resolve the problem always knew about it. But there were instances where problems that required a higher-level decision or approval festered for hours
without being resolved. I became convinced that something needed to be done. We emphasized that if anything was holding S/CAT people up that could not be resolved promptly by those involved, it was their duty to work the problem up their chain of command. This worked fairly well for manufacturing, thanks to the encouragement and support of the departmental leadership. The test directors, however, were often too busy with immediate activities and unable to leave their consoles while a test was underway to divert much attention to seeking help. Together we came up with a simple system which alerted management to their problems with minimum effort on the test director’s part. We installed a four-position switch at each test director’s console, labeled “Off—Running—On Hold—Stopped” and corresponding to green, yellow, and red lights in my and the STMs’ offices. Whenever a test was in progress in each ACE room, the lights were lit and we could ask if they needed help to clear a yellow or red condition.

I often answered these silent alarms, usually calling the test director first but sometimes just walking up to the ACE room to see what was happening. Inside the large, dimly lit room, with green lettering flickering on the CRTs at the consoles on one wall, a frustrated group of test engineers and inspectors would be huddled together, looking at diagrams and reading out the steps in the OCP.

A system-by-system buildup of more than seventy operational checkout procedures led up to the integrated all-systems OCP 61018, the most complete and complex of ACE controlled tests. The OCPs began simply, with circuit continuity checks and on/off commands, and progressed to very sophisticated tests that used the powerful ACE computers to simulate the flight mechanics and external interactions of a real mission. One OCP, for example, exercised the rapid-fire sequence of commands that resulted when the LM commander pushed the abort-stage panic button during powered descent. The LM guidance computer (LGC) issued commands that shut down the descent engine, separated the ascent and descent stages, and fired the ascent engine. It made the rendezvous radar find the command module and lock on, while steering the LM toward it. The LGC noted the change in LM’s weight and mass properties to ascent stage only, and changed reaction control system commands and trajectory calculations accordingly. On the assembly floor the fluids tanks were empty and explosive device igniters were not installed, so no firings took place, but signals generated by the LM flight hardware were sent to the firing locations in the propulsion, explosive devices and RCS systems and verified. It was an end-to-end electrical simulation of the flight mission event.

S/CAT’s ability to hold schedules and do things right the first time was improving, but it was agonizingly slow, always consisting of “two steps forward and one (or more) back.” Wright and I benefited from the steadying influence of Joe Gavin and Ralph H. “Doc” Tripp, who helped us analyze what our fundamental problems were and how to cure them, and protected us from the
continuous fusillade of “help” from Titterton and others. Doc Tripp, long the manager of Grumman’s Instrumentation Department in Flight Test, was brought in early in 1967 as LM program director under Vice President Joe Gavin. He was probably the best manager I ever worked for at Grumman. A firm believer in participative management with tight controls, he began by meeting individually with each person who reported to him and having that subordinate prepare a one page job description and statement of specific goals and objectives. Thereafter he gave each person his head, encouraging his subordinates to take risks and solve problems within their jurisdiction, praising us when we succeeded, defending us when we failed, and welding all of us into a confident LM program team. People- and management-oriented, he made a good counterpart to the more technical and individualistic Gavin. Both of them possessed not only a wealth of good judgment and common sense but also calm temperaments, which allowed them to sail through the recurring crises with equanimity. Tripp had a habit of using his small penknife to whittle designs and figures from wooden coffee stirrers during NASA’s long meetings, yet despite repeated attempts, the NASA people never caught him off guard on the topic under discussion. He also refused to adopt NASA’s example that the top managers must know every technical system and problem in great detail; Tripp would call upon his experts if pressed on such points.

All too often we in S/CAT would goof or blunder all by ourselves, and no amount of capable LM program leadership could save us from ridicule and upper management or NASA intervention. One of the worst foul-ups occurred on LTA-8, that devilishly complex thermal test article. At one point in the testing we were required to drain and flush the water-glycol coolant that circulated throughout both LM stages, cooling electrical equipment and components and the astronauts’ suits and oxygen supply. The test team mounted a fifty-five-gallon drum on the highest level of the workstand, connected by clear plastic tubing to various points of the LM’s cooling circuit. Unthinkingly they used an old drum with a rusted bottom that dripped water-glycol solution all over the spacecraft, requiring several days of painstaking swabbing and cleaning with an acid neutralizing agent to set right. At Titterton’s meeting I hung my head in apology and endured a bitter, sarcastic tongue-lashing.

The cockpit instruments in the LM had hermetically sealed cases to protect them from humidity and dirt. After we had installed the instruments in three cockpits, a clever NASA inspector at Bethpage devised a new technique that was simple but effective in disclosing dirt particles trapped within the sealed case. He simply held the instrument on the workbench, glass face down, and shook and tapped it gently. Then, retaining the glass-down orientation, he lifted it up over his head and looked for dirt particles on the inside surface of the glass. If any at all were visible, the instrument had to be returned to its supplier to have the case opened, cleaned, and resealed.

This improved inspection technique had never been thought of before, either
by the instrument manufacturers or ourselves, and almost none of the instruments that had been delivered could pass it. For the manned flight spacecraft the new test was mandatory, so for LM-3, the cockpit of which was almost complete, we had to remove and replace all the instruments. We faulted ourselves for not devising this obvious quality check long beforehand.

My Chickens Come Home to Roost

My assignment to run S/CAT was a unique learning experience and an appropriate form of retribution. On the assembly floor I came face to face with the troublesome design features I had approved, and in some cases demanded, when I was project engineer, which caused untold hours of toil for the manufacturing technicians who had to make them work in the real world.

Foremost among these were the extremely thin 26-gauge kapton insulated wire and miniature electrical connectors used throughout the LM. Adopted as a weight reduction measure during SWIP in 1966, these fragile wires and tiny connectors were an endless source of problems with wire breakage and improperly mated connectors. Breakage was common in the wires, which were used wherever signal voltages were applied with essentially no current, making them more abundant than the larger-gauge wires.

On the assembly floor I often watched sympathetically as a frustrated technician demonstrated the difficulty of mating and demating a miniature connector containing dozens of fine wires, doing it by feel with gloved hands in a cramped and all-but-inaccessible space. One smiling tech with small but powerful fingers was in great demand because he could handle the most difficult locations.

We ran a special vehicle-level vibration test on LM-1 using electrically driven vibration generators. This test gave us confidence that these wires would not break when the LM was being shaken in flight during launch or from its own rocket engines. Still the wire breakage problem due to installation and handling was a constant drag on assembly and test operations, although it gradually lessened as the technicians improved their techniques for gentle handling of wire bundles. For LM-4 and subsequent vehicles we switched to a special high-strength copper alloy in the wires, which alleviated the breakage problems.

The fire-retarding and moisture-sealing requirements of potting and covering with Beta cloth booties added to the difficulty of handling electrical connectors of all sizes. Much of the potting and bootie installation could be done on the workbenches before the wire harness assemblies were installed into the vehicle, but there were some areas where the wire bundle was threaded through structure and a connector had to be placed on one end inside the spacecraft. The potting material took several hours to cure under heat lamps, after which portable X-ray equipment was used to verify that the wires inside
were all properly routed to the connector pins or sockets. A klaxon horn brayed a warning on the floor before the X-ray was turned on—all personnel had to leave the immediate area. Some nights as I lay in bed that klaxon was still reverberating in my brain. If wires had to be replaced in a connector for any reason, they were physically cut out of the potting, the new wires, pins, or sockets were slipped into place, and the connector was repotted and X-rayed again, a cycle that took at least one whole shift. With many thousands of wires and hundreds of connectors in the LM, this was not an uncommon occurrence.

Even the basic aluminum alloy structure of the LM imposed exacting demands upon the manufacturing process. In our relentless quest for weight reduction, we engineers had made widespread use of the high-strength alloy 7075 and of chemical milling, which enhanced susceptibility to stress corrosion. Controlling stress corrosion required carefully fitting each part upon assembly to avoid clamp-up stresses when the fasteners were tightened. This involved educating our mechanical technicians and inspectors to the causes of stress corrosion and the most common fit-up problems of LM parts, and then enforcing rigorous compliance with the approved fit-up procedures. Even so we had numerous cases in which cracks were discovered in thin aluminum flanges that had retained excessive clamp-up stresses when installed. The astronauts were especially vocal in urging us to stamp out this problem, as they no doubt pictured their return vehicle crumbling beneath them on the Moon, victim of its own locked-in stresses.

Delivering LM-1

By early June 1967, despite all these problems, we were approaching the delivery of LM-1 to KSC. As the first flight LM with many special provisions for unmanned flight, the birth pangs of LM-1 were long and painful. The LM mission programmer (LMP), a combination of computer, electrical relays, and switching racks, was the programmable robotic brain that would perform the mission automatically under command of the MIT-designed Apollo computer (primary mode) or the LMP itself (backup mode). This complex, special-purpose unit required extensive qualification testing and design modifications before we began to have confidence in it. Even more troublesome was the development flight instrumentation, a secondary instrumentation system with pressure, temperature, stress, and vibration sensors and its own separate telemetry and recorders. DFI was intended to provide additional engineering data on the early LM flights that could be used to correct and refine the design prior to the first lunar landing; it was only installed on LM-1 to LM-3. Since it was not essential to mission operations, DFI was not qualified to the same high parts selection and reliability standards as the Instrumentation system used for mission critical measurements, but was more akin to experimental
aircraft test instrumentation. In mid-February 1967 I noted that 56 out of 320 total DFI measurements were down, with little improvement over recent weeks despite extensive trouble shooting, transducer (sensor) replacement, and wire repair.
¹
Many of the DFI measurements were considered mandatory for LM-1’s mission of verifying LM systems performance in space.

To cure the DFI problems I got Gene Goltz, head of the Instrumentation Department, to lead a “tiger team” of instrumentation specialists from both the LM program and Flight Test in a complete item-by-item review of the DFI system. They considered component and system design, test results, supplier quality performance, and environmental requirements of the system. Goltz’s team recommended strengthening some DFI requirements, particularly the use of vibration testing as a quality acceptance test screening of transducers, modems, and other critical components. They recommended that we drop some suppliers whose components were not performing reliably. Although it took some time, the tiger team’s output resulted in acceptable levels of DFI reliability.