Lufthansa Papa Whiskey

Right after I left Loring AFB in 1982 one of our tankers had an ice build up in the elevator balance bay that caused the elevator to work backwards. That is, pushing forward caused the nose to pitch up, pulling aft caused the nose to pitch down. Of course this was confusing but the crew figured it out and landed. This particular Airbus incident could have been more confusing since the stick inputs on the left are not obvious to the right. (The sticks aren't mechanically linked.) But they figured it out. Matt Thurber does a great job of relating what happened, so here it is in his words

Special Safety Report - The Saga of Papa Whiskey

Matt Thurber, Editor

By all accounts, the Lufthansa Airbus A320 known as Papa Whiskey remains flying today due to the incredible skill and teamwork of the two pilots at the controls on March 21, 2001. Papa Whiskey carried 121 passengers and crew on that fateful day and was scheduled to fly from Frankfurt to Paris.

All was normal prior to takeoff; there was no indication that there was a profound problem with the Airbus's fly-by-wire flight controls. As Papa Whiskey accelerated on the runway and lifted off, the left wing dropped slightly. The captain, who was flying this leg of the trip, reacted normally, instinctively nudging the sidestick gripped in his left hand just a tiny amount to the right, sending an electrical command to the elevator-aileron computer (ELAC), which in turn signaled the hydraulic actuators at the ailerons to move-down in the case of the left wing and up for the right wing aileron--to lift the left wing back to level.

An utterly routine matter, under normal circumstances, but unfortunately Papa Whiskey did not react in the normal fashion. The left wing dropped more, coming to within a few feet of the ground. The captain blurted, "I can't do anything more!"(according to the German accident report).

Lufthansa has been at the forefront of training its flight crews to work as a team and not in a hierarchical manner. The first officer immediately pushed the Takeover Push Button to take control and raised the left wing with a twist of his right wrist, in the correct direction. The two pilots put Papa Whiskey on autopilot and climbed to 12,000 feet and tested both sets of flight controls. The first officer's sidestick worked normally. The captain's sidestick, incredibly, caused the Airbus to bank left when the stick was pushed right and vice versa; it was working backwards.

The pilots elected to return to Frankfurt and they reported the incident. Lufthansa immediately consulted with the German regulatory authorities and Airbus. A lengthy investigation ensued. How could Papa Whiskey's aileron controls be reversed?

The official investigation by the German BFU, similar to the U.S. National Transportation Safety Board, was not released until two years later, in April, 2003. But it didn't take people at Lufthansa and at its maintenance company Lufthansa Technik, which maintains Papa Whiskey, long to find out what had happened and figure out a way to prevent such mistakes from occurring again.

The story of how Papa Whiskey's flight controls were reversed has been told and by now it's not exactly news, but this is the rest of the story; not just how the problem occurred but how a proud, diligent, careful maintenance company like Lufthansa Technik dealt with this near-tragedy and took it firmly to heart, making sure that what happened to Papa Whiskey wasn't brushed off as an anomaly but served as a catalyst for improvement and growth for everyone in the company, from mechanics on the hangar floor to top management.

By skill, teamwork, and perhaps a helping of luck, Papa Whiskey didn't crash that day. Nor did Lufthansa Technik and the mechanics and managers involved get punished. A mistake was made, but the results reveal a fascinating tale of a company that focuses on learning, not punishing, growing, not stagnating, and by sharing this publicly, helping advance aviation safety.

Aviation Maintenance editor Matt Thurber asked Lufthansa Technik officials if the company would be interested in sharing details of the Papa Whiskey incident so that the industry at-large can benefit. The result was a fascinating interview with Dr. Burkhard Andrich, senior vice president aircraft maintenance and Robert Nyenhuis, head of Airbus A320 maintenance. The story that follows is derived from that interview and the report by the German BFU (Federal Bureau of Aircraft Accidents Investigation).

The day started normally, with an on-time departure at 11 a.m. by a Lufthansa Airbus A320-200, D-AIPW, loaded with 115 passengers and six crew. The weather was benign, with scattered clouds at 3,200 feet, good visibility, and a slight crosswind from the east for the takeoff on Frankfurt's Runway 18. The 41-year-old pilot-in-command had logged 3,300 hours in the A320 and the second-in-command, age 27, 1,500 hours. Both were on duty about six and a half hours before taking off in Papa Whiskey.

During preparations for the flight, the pilots performed the normal required flight control check and noted no discrepancies in the flight control system. This was Papa Whiskey's first flight after maintenance.

On March 17, one of Papa Whiskey's two elevator aileron computers (ELACs) failed and was replaced. On March 18, ELAC 2 announced another failure but the pilots were able to reset the system and proceeded on a flight to Moscow. In Moscow that evening, ELAC 2 signaled a failure again. This time, the two ELACs were swapped, and the failure remained on ELAC 2, so the crew pulled the applicable circuit breakers to isolate the failed ELAC and flew back to Frankfurt in accordance with Lufthansa's minimum equipment list procedure.

On the night of March 18 and the early morning of March 19, Lufthansa Technik technicians troubleshooting the ELAC problem found a defect in the X-TALK-BUS (cross-talk bus, which allows the ELACs to communicate) between ELAC 1 and 2. The ELACs are connected to the Airbus flight control system with four large plug segments labeled AA, AB, AD, AE, each of which has 105 wires attached with connection pins. One of the connection pins (Pin 6K) in segment AE was bent, thus the error.

The technicians were unable to repair the connection pin because they could not reinsert the pin's safety spring, which had come out. The decision was made to replace plug segment AE, but there wasn't one in stock. There was an entire plug segment assembly in stock, but the AE segment of the new assembly would not match the remaining three segments, so they decided to replace all four segments, which meant reconnecting 420 connection pins, while working in the avionics bay. The German BFU report calls the bay "a most confined space," and a photo in the report shows the ELACs surrounded by numerous tightly bundled wires.

"There was no time pressure for the people involved," said Dr. Burkhard Andrich, Lufthansa Technik senior vice president aircraft maintenance, during the interview with Aviation Maintenance. A spare A320 was available, and Papa Whiskey was removed from the flight schedule.

On March 19, technicians with the required qualifications began the job and finished rewiring segments AA, AD, and AE. The AB segment was done by the following shift. The work was accomplished by the one-to-one method, where each wire was disconnected from the old plug then immediately installed into the new plug, in an attempt to avoid potential mis-wiring of such a critical component. The A320 has fly-by-wire flight controls and their correct operation is completely dependent on miles and miles of carefully routed and attached wiring.

Papa Whiskey had an unusual design feature in its ELAC wiring. Two pairs of wires--they are all red and blue on the ELAC--are attached in what could appear to be an illogical and odd manner. While the rest of the wire pairs are attached in a red/blue configuration, these two pairs, assigned to pins 3C/3D and 15J/15K, are sequenced blue/red. Airbus engineers were aware of this anomaly and had designed it out of the A330 and A340 models and some later A320s. "An interchange of colours was accepted for a certain transition period," the BFU report stated. Unknown to the Lufthansa Technik technicians, Papa Whiskey was one of those transitional A320s.

Further confusion developed when the technicians tried to interpret the service bulletin status of Papa Whiskey. They ended up referring to an incorrect wiring diagram due to difficulty interpreting the airplane's service bulletin compliance. Because they didn't realize that one service bulletin hadn't been accomplished, the wiring diagram showing a logical sequence of wiring colors (all red/blue) should not have been applicable to Papa Whiskey, yet this was the diagram that the technicians ended up using.

The stage was now set for an error to occur.

The technicians completed the final pin connections on segment AB. Instead of installing pins 3C/3D and 15J/15K in the blue/red sequence, the technicians made the connection in the red/blue sequence specified in the incorrect wiring diagram to which they referred, which also appeared perfectly logical as all the other wires were connected in this sequence.

After working on flight controls, a functional check is required and this was done. An error message on ELAC 1 was noted and the technicians found a faulty bridge on plug segment AA and fixed it. This error, the BFU noted, "did not relate to the original complaint." One can't help wondering, however, if repairing this error might have led the technicians to feel that they had worked all of the bugs out of the ELAC system, predisposing them not to expect further problems and thus not seeking to find any more discrepancies.

The technicians performed another functional check, but only using the right-hand side-stick and in accordance with the aircraft maintenance manual. The manual instructed the technicians to "Push the FLT CTL ELAC 1 (2) pushbutton switch. - Move the side-stick around in its two axis [spelling is correct] from stop to stop." The manual did not suggest that anyone observe the movement of the ailerons and elevators to ensure that they moved in the correct direction. Did the mechanics perhaps assume that if the controls moved in an incorrect direction, a fault message would illuminate? And is this a potential expectation trap that might be prevalent on modern electronically operated aircraft?

"All the functional checks are in the cockpit," said Robert Nyenhuis, Lufthansa Technik's head of Airbus A320 maintenance. The maintenance manual did not specify double-checking the movement of the control surfaces.

The BFU report on the incident flight is unclear on the sequence of events just prior to takeoff, regarding the flight crew's normal flight control check. While the report does note that the crew performed the flight control check, it also said that the crew checked only for full aileron deflection, but not for the correct direction of deflection. The report doesn't say who performed the flight control check with which sidestick, the pilot-in-command in the left seat or the first officer in the right seat. It seems that a flight control check with the right sidestick would have resulted in correct aileron deflection, hence there was a 50-50 chance that this last check would not have resulted in discovery of the problem. In any case, the pilots would also have to have noted the direction of movement of the ailerons, and it's not clear that they were looking at this or simply relying on cockpit indications.

The BFU did issue a safety recommendation to Lufthansa and the German regulatory authority (LBA) suggesting that "operators of fly-by-wire aeroplanes...amend their checklists accordingly."

During the takeoff, it quickly became apparent that something was wrong. As the Airbus rotated, the pilot-in- command, flying from the left seat, saw that the airplane was banking to the left. "He tried to correct the bank angle through an opposite input on the left sidestick," the BFU report said, "but it grew increasingly larger."

The Loral flight data recorder later showed that the left wing dropped nearly 22 degrees. By this time, the Airbus had accelerated to 170 knots. Noticing that the left wing continued to drop, the first officer "instinctively made an input to the right on his sidestick, which prevented the bank angle from increasing even further, but did not lead to an improvement of the situation." Finally, after the pilot stated that he couldn't do anything to control the banking left wing, the first officer pushed his sidestick's TAKE OVER PUSH BUTTON and then he was able to control the Airbus normally.

The pilots climbed to 12,000 feet on autopilot then cautiously tested the flight controls. Operation with the right sidestick was normal, but when they switched control to the left sidestick, lateral control was reversed, with the wing moving opposite of the way the sidestick was moved instead of the same way. The flight returned to Frankfurt, under control of the right-hand sidestick. Lufthansa notified the German LBA immediately and Papa Whiskey reentered Lufthansa Technik's hangars for some serious scrutiny.

It should be noted that Lufthansa and Lufthansa Technik are separate organizations. As aircraft operator, Lufthansa bears responsibility for overseeing maintenance on its aircraft. The BFU reported that this function was weak.

The technicians and managers at Lufthansa Technik quickly began testing Papa Whiskey to learn what had happened. There was no question that the investigation was not seeking to point the blame at a specific person who could then be punished. It is more important, Nyenhuis said, to figure out "what to improve in the process and not to punish people."

The technicians and managers discovered that two wire pairs had been installed incorrectly in the ELAC plug segment AB. The Airbus A320 maintenance manual flight control check, which had been accomplished, did not require verification of the movement of the control surfaces, as is the case on older airplanes, according to Nyenhuis.

They also found that the technicians did not perform a continuity check on all the wires that were re-installed, as specified in the maintenance manual standard practices. It turns out that Lufthansa Technik had its own standard practice, Nyenhuis explained, which permits a deviation from a maintenance manual standard practice. The technicians on Papa Whiskey consulted with Lufthansa Technik's maintenance support department, which agreed that the continuity check was not necessary because they would have to perform a functional check anyway, and this check should uncover any remaining faults. The BFU report also noted this discrepancy between the maintenance manual requirements and the Lufthansa Technik technicians' deviation from those requirements and said it was a factor in the incident.

Immediately after the initial internal investigation, Nyenhuis and Andrich called meetings with all the Lufthansa Technik technicians, beginning at Frankfurt and eventually spreading to the entire organization worldwide. "We discussed what happened," he recalled, "the root cause."

During these meetings, managers focused on a fundamental issue: the first objective during maintenance is to make sure the airplane is safe. Secondary to safety--always--is being on time. The case of Papa Whiskey was dissected, and everyone at the meetings had to sign a document acknowledging that they were familiar with the details.

From feedback received during the initial meetings, Nyenhuis and Andrich launched a change project to re- engineer the entire maintenance organization, from documentation to training to leadership.

Beginning with leadership, the big change was to make sure planning personnel budgeted enough time for team leaders to lead at least 50 percent of the time. This ensures that during critical maintenance operations, leaders are not tied up with functions that are better delegated to someone else and that they can be physically present to observe the critical tasks. "They must be there," Andrich said. "If you go into the flight controls or a chip detector, you must be really aware of what you are doing. You should focus on safety, not efficiency." Team leaders, he added, must make sure their team members understand this and are fully aware of when they are performing critical operations.

To bring this point home, all 350 team leaders and supervisors went through two to three days of off-site training, focusing on the hierarchy of safety and efficiency. The priority actually goes: first, safety and quality procedures; second, work ethics, values, and leadership; third, efficiency.

From feedback generated during this round of training, it was decided that the double-check inspector on duty has the authority a request a test flight if he or she deems it necessary. And the test flight will be a real test, not a revenue flight with passengers.

Another ongoing focus is improving the workplace culture and values. "Communication between hierarchies is very important," said Andrich. A younger workforce helps with that because, he added, younger people are used to a more bottom-up environment, where the traditional lines between internal silos are blurred.

This kind of training is now ongoing, with one day a year for technicians, two days for supervisors and double- check inspectors. "To have these communications costs a lot of money," Nyenhuis admitted. But the training is worthwhile. "Even with the economic crisis last year, we didn't cancel training," he said.

"We tried to change in a systems approach," Nyenhuis added. "Not action-reaction, but change procedures, minds, and the safety culture."

Now, he explained, the double-check inspectors get involved early in any maintenance operation, especially those that are considered critical. And these inspectors are not operating under the efficiency mandate but are considered 100 percent efficient by definition. Whatever they need to do must be done. "The double-checker is there as a mentor," Nyenhuis continued. "We don't need an inspector who is loud and punishes. They are the most experienced people we have. The look over, they ask questions, and they escalate if necessary. They took an active role in improving our procedures."

Another new effort was adapting some of Lufthansa's obviously successful human factors pilot training to the maintenance operation. One immediate change was improving the problem-reporting system and using some ideas from the pilots' anonymous reporting system. More important, however, was convincing the personnel that taking advantage of any of the various methods of reporting problems would not result in punishment. "It's important to say that we trust each other," Nyenhuis said.

Something that once belonged solely to the quality inspectors is now part of everyone's daily work. The company calls this "daily spot checks," and now not only are these done by quality inspectors but they also must be done by the maintenance teams themselves. Management compares the spot checks by the maintenance teams to those from the quality department to calibrate what is actually happening. "They should match," Nyenhuis explained. "It tells us if we're right or wrong."

In order to improve some regular important processes, Lufthansa Technik implemented an integrated quality management system that includes most processes on an employee-accessible intranet, which is available at any of the company's maintenance bases. The mechanics helped developed the processes, which helped them buy into the program because it wasn't just a case of the quality department forcing its desires on the workforce.

To improve communications between the people who do the work and management, the company created quality circles composed of a variety of personnel. The quality circles meet every two weeks, and inspectors from the German regulatory authority--the LBA--are invited as well.

As far as keeping the LBA inspectors up to speed, Lufthansa Technik delivered an action plan to the LBA in July, just four months after the Papa Whiskey incident. Much of the action plan included the improvements mentioned above.

A key to making the improvements really happen is that management must be strongly supportive. "It's really a question of doing it, not just talking about it," said Andrich. "How do you implement it? It means sitting down with front-line people, not just writing checks."

For Andrich, the Papa Whiskey near-disaster and the learning that resulted from the incident has generated permanent beneficial results. Referring to the classic Swiss cheese model (Schweizer Kosemodell in German), which shows how an accident or incident results from the lining up of holes in multiple slices of "cheese," Andrich said, "we're making sure that no seven slices of cheese line up. We really learned a lesson here. If we would face the same thing now, the safety net is much better."

Note

I am told that Swiss Cheese in German is actually Schweizer Käse Modell.

Thurber, Matt, Special Safety Report - The Saga of Papa Whiskey, Aviation Today, April 1, 2005.