Moon Lander: How We Developed the Apollo Lunar Module (27 page)

By mid-February 1968 we had thoroughly inspected six LMs (numbers 3 through 8) and over fourteen hundred accessible components. No major cracks were found. We also switched from 7075-T6 aluminum alloy to the more stress corrosion resistant—T73 temper, effective on LM-4 and subsequent vehicles. This was accomplished by retrofitting the structural tubes on these vehicles. By changing out the tubes that had been the major source of trouble and showing by inspection that fit up parts with thin tabs were not cracked, I felt we had the problem under control, and NASA agreed. By the end of the month, Mueller told NASA Administrator Webb that he was no longer worried about stress corrosion.
⁷

Inspections continued throughout the LM program, and occasionally a cracked part was found and replaced. LM never suffered a structural failure of any kind, so the effect of stress corrosion on the flight missions was nil. But stress corrosion was a nagging problem that never entirely went away, as there was always a small chance that the next inspection would turn up a newly cracked part.

Battery Problems

The LM batteries were constructed with silver and zinc electrodes and liquid potassium hydroxide electrolyte and were not designed to be repeatedly charged and discharged. A limited number of recharges was allowed during ground tests, but in flight the batteries were launched fully charged and discharged until depleted. The EPS had four batteries in the descent stage, producing 3 ampere-hours per pound of battery weight, and two in the ascent stage, which yielded 2.5 ampere-hours per pound. The batteries had alternating plates of silver and zinc connected to positive and negative terminal bus bars by metallic jumpers and separated from each other by paper insulation. The plates were mounted in a row inside a vented plastic case that was filled with electrolyte. A straightforward design with seemingly not much that could go wrong. But looks can be deceiving. Before the LM program ended, in the course of resolving a long list of development problems, Grumman was forced to learn more about these batteries than even the manufacturer knew.

My confidence in the batteries waned when I visited the Eagle Picher factory near Joplin, Missouri, in 1966 as part of a tour of several LM subcontractors. Eagle Picher was primarily a manufacturer of paints and industrial chemicals based on paint pigments. They got into the battery business because
some of the electrode materials, like lead and zinc, were used in their paints, and they saw a way of diversifying their product line with new applications of materials that they understood. Knowing this background did not prepare me for the sight of their factory complex—it was a sprawling industrial wasteland with many acres of low sheds and two-story brick buildings, dominated by dozens of tall smokestacks spewing great clouds of gray and white smoke. White dust covered everything—the roofs and walls of the buildings, the ground, the cars in the parking lots, everything.

The LM battery assembly area was located in one of the low corrugated metal sheds. Inside the windows were open to the swirling particle-laden atmosphere and there were layers of white dust on the windowsills and floors. At rows of worktables, strapping farm boys were doing delicate battery assembly, folding sheets of insulation over the plates, mounting the plates on the bus bars and connecting the jumpers, and installing the plate assemblies into the battery case. One muscular fellow caught my eye particularly; he was wearing a soiled undershirt and had a cigarette dangling from his lips. As I watched incredulously, the ash on his cigarette grew longer as he peered over the open case into which he had just placed a plate assembly, until it finally broke off and fell inside.

That was the last straw for me. I motioned for Eagle Picher’s LM battery program manager to follow me into a glass partitioned cubicle off the assembly floor and exploded. “Don’t you people have any concept of quality?” I shouted. “For God’s sake, look at this place! Clean up the building, close the windows, install air conditioning and filters, prohibit smoking, make the workers wear clean clothes and smocks, maybe even replace the farm hands with nimble-fingered women.”

After that visit I insisted that Grumman hold a monthly review with Eagle Picher, alternating between Joplin and Bethpage. If I could not attend the Joplin meeting, which was frequently the case, Carbee filled in for me if he could. I also prevailed upon Joe Kingfield, LM Quality Control manager, to station a resident Grumman quality inspector at Eagle Picher. It seemed like cruel and unusual punishment, to assign a Grumman man to such a place, but I considered it necessary. Our inspector filed a weekly report on Eagle Picher’s activities and progress in “cleaning up their act.”

All this was in the nature of a preemptive strike, since as yet no battery problems had emerged. Our concern was soon justified as erratic battery performance began to appear in tests, traced to assembly errors and contamination. As battery development progressed, many other problems surfaced. Case cracking and jumper failures during vibration tests were recurring problems, and required several modifications to design details before the batteries could pass the vibration portion of qualification testing. Underperformance was a problem that required redesign to maximize the active area of the plates and to determine the optimum electrolyte concentration. Venting of
hydrogen from the case during battery operation was provided by a small plastic relief valve in the sealed cover. These valves sometimes stuck shut during cold-temperature testing, causing hydrogen gas pressure to build up and the cover to pop free of the case. In space this would cause complete battery failure, as all the electrolyte would leak out. The durability of the paper separator also gave concern. There were some short circuits between plates when pinholes developed in the separators after several ground test recharges. More durable separator material and further limitation of recharges solved this problem.

This list of miscellaneous problems continued through 1967 into 1968, giving me and Grumman a background level of concern. The batteries passed their qualification tests, although several months behind schedule, and the problems that remained did not seriously threaten the Apollo program schedule. I remained wary of the batteries until the completion of the program, preconditioned by my original view of Eagle Picher as a dirty gray scene of desolation from Dante’s
Inferno
.

Tank Failures

Of the thousands of possible failures in the Apollo Mission and the LM, one of the most terrifying was rupture or explosion of a tank. Dozens of tanks on LM held the vital consumables of space exploration: rocket propellants, helium pressurant, oxygen, and water. Most of these tanks contained high levels of pressure energy and could explode in event of failure. Since all aerospace contractors used the same group of subcontractors to make their tanks and the same materials and similar processes and procedures, a tank failure anywhere in the Apollo program sent a shock wave of concern to everyone.

In mid-1965 there were two series of tank failures, both on the CSM program but directly affecting LM. CSM reaction control system fuel and oxidizer tanks each failed at their manufacturer, Bell Aerosystems, which also made very similar tanks for the LM RCS. Then a failure occurred at Beech Aircraft, which was making hydrogen and oxygen tanks for the CSM fuel cell assembly. I sent John Strakosch from LM Structural Design and Frank Drum from Materials to both companies to meet the engineers leading the failure investigations and establish an information channel to Grumman on the findings. They were also able to suggest lines of inquiry to the investigation teams.

Strakosch and Drum returned very concerned because the origin of the failures was unknown and they applied directly to LM. A list of possible causes had been prepared and was being vigorously explored by NASA, North American, Bell, and Beech. Both tanks were of highly stressed titanium and the failures seemed to originate at or near the welded circumferential seam.

Thanks to careful sleuthing, these cases were solved within a few months. The RCS tank failures were caused by the propellant manufacturer improving
his process to increase the purity of the nitrogen tetroxide oxidizer that he produced. The revised production process reduced the amount of trace contaminants, including nitrous oxide, which had played a beneficial, if unsuspected, role in protecting the titanium from attack by the nitrogen tetroxide. By specifying a minimum allowable amount of nitrous oxide in the product, the problem was resolved. To assure control over the formulation of such commercial products, NASA began buying propellants for the whole program under a government specification.

The cause of the Beech problem was quite different. It occurred because a weld rod of lower strength titanium alloy had been inadvertently used to weld the oxygen tank. This was determined by metallurgical examination and analysis. A torrent of procedures and regulations followed, aimed at tightening control and accountability for weld rod and certifying that the proper alloy has been used in each weld.

In October 1966 one of the large service module propellant tanks ruptured while under pressure test at NAA in Downey. After intensive investigation the cause was determined to be incompatibility between titanium and the methanol (methyl alcohol) that was used as the pressure test liquid.
⁸
This oversight stimulated NASA to conduct a comprehensive survey of all the fluids to which the Apollo tanks were exposed in their lifetimes and to perform laboratory tests to establish whether the tank material and the fluid were compatible. It seemed fairly basic, but until then it had not been done. This case reminded me of an experience I had years earlier while developing a small liquid rocket at Lockheed. My test rocket failed because I had not known that the nitric acid oxidizer would attack the nickel in the high strength stainless steel alloy we were using.

Almost a year later another tank failure occurred, this time an LM descent propulsion supercritical helium tank. The inner pressure vessel ruptured while being pressure tested at the manufacturer, Airesearch. The failure originated at the weld, so at first we thought it was another case of a mislabeled weld rod. Henry Graf, supercritical helium system manager at Airesearch, impounded the remainder of the spool of weld rod used in the failed tank, and metallurgical examination showed it to be the proper alloy. Moreover, microscopic inspection of the failure showed some tiny intergranular cracks, typical of stress corrosion. Yet as far as anyone knew the tank had never been exposed to any fluids not approved for compatibility with titanium.

Henry Graf became obsessed with finding the cause of this failure. He led his engineering and quality staff through a minute examination of every step in the manufacturing process, starting with the receipt of the titanium forgings and the quality pedigree that accompanied them. At each step of the process, they looked at what had been done on the failed tank, and asked whether anything in this step was different from their process on previous tanks. Graf’s careful detective work paid off, discovering a cause so trivial that
a less observant investigator would surely have overlooked it. Graf noticed one minor difference in the process for this tank and those that had preceded it: instead of using new cloth pads to wipe the tank surfaces prior to welding, washed, reused cloths were employed. Examination of the washed cloths showed traces of detergent, and test samples that were wiped with them failed under combined stress and humidity testing. The trace detergent attacked titanium! There could be no more gripping example of the extreme sensitivity of highly stressed tank material and welds to contamination.

In September 1968 a descent propellant tank destined for LM-9 (Apollo 15) was found to have a cracked weld after proof test at the manufacturer, Airite Division of Sargent Industries. The test had been conducted with the tank full of distilled water at room temperature. Airite was able to verify that the proper weld rod had been used and that all fabrication processes and procedures were complied with. However, some impurities were found in the water used for the test. Frank Drum spent time with Airite in California and compared Airite’s welding procedures to those of our other titanium tank suppliers, Aerojet and Airesearch. He considered Airite’s procedures to be less conservative than the others in some key process steps, such as tack welding. Aerojet always did tack welding by machine inside a vacuum welding chamber, while Airite did the tack welds by hand in the factory environment before putting the tank into the chamber to complete the full continuous weld by machine.

At our insistence Airite upgraded their procedures to include the best practices of our other suppliers, and consulted with them on the “tricks of the trade” involved in making them work well. With guidance from NASA and Grumman, they also redesigned the butt weld joint to a “J-groove” design, based upon improved strength of this design in coupon samples. Airite further tightened their already strict manufacturing quality controls and improved cleanliness and housekeeping in their shops.

As part of the SWIP weight reduction effort, Grumman had prepared a lightweight descent propellant tank design and put it out for competitive bids. Airite won the competition, replacing the original descent tank supplier, Allison Division of General Motors. Allison tanks were in LMs up to the LM-5 (Apollo 11) and Airite tanks were installed in LM-6 and LM-7 when the failure occurred. LM-8’s tanks had been delivered but not installed. At Grant Hedrick’s suggestion, we began a cryoproof test program for the Airite descent tanks, to see if we could verify that sufficient weld safety margin existed beyond the proof pressure level to let us clear the LM-6 and LM-7 tanks without removing them. Cryoproofing consisted of testing the tank to proof pressure (one and a half times normal operating pressure) while filled with liquid nitrogen at minus three hundred degrees Fahrenheit. Welds were more brittle at this low temperature, so this was a more severe test of the fracture toughness of the weld.