I used to think the only argument worth having when it comes to checklist philosophy was the "challenge-do-verify" (CDV) versus the "do-verify" (DV) debate. Many pilots think it is perfectly fine to do everything from memory and then use the checklist to make sure nothing was forgotten. So we'll show how that debate is more settled than many think.

— James Albright

image

Updated:

2018-04-05

image

Boeing Model 299 Crash,
30 Oct 1935, USAF Photo.

The crash of Gulfstream GIV N121JM at Hanscom Field in 2014 shed light on an entirely different checklist problem. Some pilots willfully skip doing checklists entirely. They just "do" (D). So that brings up an even more fundamental question: Are Checklists Required?.

Once we've covered that, we can get into the Challenge-Do-Verify question.

If you are skeptical about CDV and prefer a "flow" (DV) for all checklists, here is a GIV Case Study about how using a flow is not only less accurate but actually takes more time. And skipping using a CDV checklist in this aircraft would have saved at least 7 lives on May 31, 2014.

1 — A history lesson

2 — Are checklists required?

3 — Challenge-do-verify or do-verify?

4 — Gulfstream IV after starting engines case study

image

1

A history lesson

The Model 299 made its first flight on July 28, 1935, flown by Boeing’s chief test pilot Leslie R. Tower. Dubbed the “Flying Fortress” by Seattle reporter Richard L. Williams, the Model 299 was the prototype for 12,730 B-17s which followed. Then on Oct. 30, 1935, with the suddenness that characterizes experimental test flights, Boeing’s great gamble seemed to fail when the beautiful silver Model 299 crashed on takeoff from Wright Field. The tragic event seemed certain to lead to the cancellation of the program and an immediate change in Army Air Corps planning.

At about 9:30 a.m. on that October day, the Model 299 was manned by a very experienced crew, including Maj. Ployer P. Hill, Wright Field’s Flying Branch chief, and his copilot, 1st Lt. Donald L. Putt. Also on board were John B. Cutting, a flight-test observer; Mark H. Koogler, also from the Flying Branch; and Tower.

Observers described the initial run of the Model 299’s takeoff as normal, even though it broke ground at about 74 mph in a “tail low” attitude. As its speed increased, the bomber’s nose went up much higher than normal. Two men, 1st Lt. Robert K. Giovannoli and 1st Lt. Leonard F. Harman, sensed it was in trouble and ran forward as the airplane reached an altitude of about 300 feet.

The Model 299 stalled, turned 180 degrees, and fell back onto a field. It landed on its left wing, cushioning the impact, which probably saved the lives of several crew. Lying flat on the field, the bomber burst into flames. Amazingly, four crew members were able to crawl from the blazing wreckage.

Source: Boyne

Hill Air Force Base, Utah, was named after Major Hill, who died following this crash.

A board of officers convened at Wright Field to investigate the crash. The presiding officer was Lt. Col. Frank D. Lackland, for whom Lackland AFB, Tex., was later named. The board determined the accident was “not caused by”: structural failure; malfunction of flight controls, engines, or propellers; the automatic pilot; or any faulty structural or aerodynamic design. Instead, it ascribed the direct cause to the elevator control being locked.

The tail section of the aircraft was virtually all that survived the fire, but it contained the cause of the accident: an internal control lock that controlled both the elevator and rudder.

The board stated that—due to the size of the airplane and the inherent design of the control system—it was improbable that any pilot, taking off under the same conditions, would discover the locked controls until it was too late to prevent a crash. Ordinarily, pilots make checks of their movement as a precaution, but apparently this did not occur.

To avoid another accident, Air Corps personnel developed checklists the crew would follow for takeoff, flight, before landing, and after landing. The idea was so simple, and so effective, that the checklist was to become the future norm for aircraft operations. The basic concept had already been around for decades, and was in scattered use in aviation worldwide, but it took the Model 299 crash to institutionalize its use.

The creation of the checklist was delayed by an unrealistic reliance on the memory of pilots. This dated all the way back to 1903, with the Wright brothers’ intimate knowledge of airplanes.

By 1918, with vastly expanded pilot training in the United States, a special handbook was created by Curtiss for the JN-4 “Jenny.” One section was titled “Hints on Flying” and provided a list of 18 items, each with considerable detail. Five items were devoted to actions to take before takeoff, nine covered in-flight procedures and safety precautions, two advised on landings, and two discussed ways to avoid stalls and spins.

As was the case with all of these checklist predecessors, pilots were expected to know the manual by rote. Sometimes this led to hubris, which led to accidents.

The nearest relative to a post-1935 checklist is found in “Hints on the Bristol Fighter,” dated March 30, 1918, and written by the officer commanding No. 39 Squadron. The section headings are similar to modern checklist, including specific sections designed to ensure pilots see that the pressure is holding, the ignition is fully advanced, the temperature is at least 65 degrees and not over 85 degrees, the oil pressure is OK, the blinds are open, and the tail lever is forward.

Source: Boyne

2

Are checklists required?

image

N121JM Wreckage, aerial photograph,
from NTSB Accident Docket, figure 6.

On May 31, 2014, two highly experienced Gulfstream pilots attempted to takeoff after starting engines with their gust lock engaged and after skipping the required flight control check. They apparently skipped the five required checklists between engine start and takeoff, two of which would have alerted them to the fact they were about to kill all seven on board.

More about this: Gulfstream GIV N121JM. They were operating under 14 CFR 91.

(a) The pilot in command of an airplane shall ensure that the following flying equipment and aeronautical charts and data, in current and appropriate form, are accessible for each flight at the pilot station of the airplane:

(1) A flashlight having at least two size “D” cells, or the equivalent, that is in good working order.

(2) A cockpit checklist containing the procedures required by paragraph (b) of this section.

(3) Pertinent aeronautical charts.

(4) For IFR, VFR over-the-top, or night operations, each pertinent navigational en route, terminal area, and approach and letdown chart.

(5) In the case of multiengine airplanes, one-engine inoperative climb performance data.

(b) Each cockpit checklist must contain the following procedures and shall be used by the flight crewmembers when operating the airplane:

(1) Before starting engines.

(2) Before takeoff.

(3) Cruise.

(4) Before landing.

(5) After landing.

(6) Stopping engines.

(7) Emergencies.

(c) Each emergency cockpit checklist procedure required by paragraph (b)(7) of this section must contain the following procedures, as appropriate:

(1) Emergency operation of fuel, hydraulic, electrical, and mechanical systems. (2) Emergency operation of instruments and controls.

(3) Engine inoperative procedures.

(4) Any other procedures necessary for safety.

(d) The equipment, charts, and data prescribed in this section shall be used by the pilot in command and other members of the flight crew, when pertinent.

Source: 14 CFR 91, §91.503

Allow me to reiterate: "Each cockpit checklist" . . . "shall be used" . . . "when operating the airplane." Are checklists required? Yes. And you have to use them. And if you need more evidence:

(a) Aircraft operations for the purpose of air navigation. No person may operate an aircraft in a careless or reckless manner so as to endanger the life or property of another.

(b) Aircraft operations other than for the purpose of air navigation. No person may operate an aircraft, other than for the purpose of air navigation, on any part of the surface of an airport used by aircraft for air commerce (including areas used by those aircraft for receiving or discharging persons or cargo), in a careless or reckless manner so as to endanger the life or property of another.

Source: 14 CFR 91, §91.13

So we must use checklists. But how?


3

Challenge-do-verify or do-verify?

The following applies to commercial operators, including 14 CFR 91K. But it should be considered a "best practice" among all non-commercial operators.

3-3401 A. Definition. A checklist is a formal list used to identify, schedule, compare, or verify a group of elements or actions. A checklist is used as a visual or oral aid that enables the user to overcome the limitations of short-term human memory.

3-3402 B. Criticality of Checklist Items. Checklist items can be ranked in criticality according to the potential effect of the crewmember failing to perform the action. Critical items are those items which, if not correctly performed, have a direct, adverse effect on safety. Noncritical items are “housekeeping” items or systems management items, which for operating practices must be routinely accomplished during a specific phase of flight, but if omitted would have a minimal effect on safety.

3-3403 METHODS OF CHECKLIST DESIGN. Operators may choose from at least two accepted methods of checklist design: the “challenge-do-verify” (CDV) method and the “do-verify” (DV) method. Available evidence suggests that safety is enhanced when the operator adopts and applies a consistent checklist design policy. POIs should use the following informative guidance when reviewing the design of an operator’s aircraft checklists.

A. “Challenge-Do-Verify.” The CDV method consists of a crewmember making a challenge before an action is initiated, taking the action, and then verifying that the action item has been accomplished. The CDV method is most effective when one crewmember issues the challenge and the second crewmember takes the action and responds to the first crewmember, verifying that the action was taken. This method requires that the checklist be accomplished methodically, one item at a time, in an unvarying sequence. The primary advantage of the CDV method is the deliberate and systematic manner in which each action item must be accomplished. The CDV method keeps all crewmembers involved (in the loop), provides for concurrence from a second crewmember before an action is taken, and provides positive confirmation that the action was accomplished. The disadvantages of the CDV method are that it is rigid and inflexible and that crewmembers cannot accomplish different tasks at the same time.

B. “Do Verify.” The DV method (or “clean-up” method) consists of the checklist being accomplished in a variable sequence without a preliminary challenge. After all of the action items on the checklist have been completed, the checklist is then read again while each item is verified. The DV method allows the flightcrew to use flow patterns from memory to accomplish a series of actions quickly and efficiently. Each individual crewmember can work independently, which helps balance the workload between crewmembers. The DV method has a higher inherent risk of an item on the checklist being missed than does the CDV method.

C. Selection of Design Method. Both the CDV and the DV methods of checklist design are currently being successfully used for normal checklists. Traditionally, operators have preferred the DV method for normal checklists and the CDV method for non-normal and emergency checklists. Operators have, however, successfully used the CDV method for all checklists. POIs may approve either method for normal checklists. In most circumstances, non-normal and emergency checklists are more effective when the CDV method is used. The correct accomplishment of the actions and procedures incorporated in the non-normal and emergency checklist categories is critical and warrants a methodical approach. Since these checklists are seldom used, however, crewmembers are usually not as familiar with the procedures incorporated into these checklists as they are with the procedures in normal checklists. In addition, many non-normal and emergency checklists do not lend themselves to developing flow patterns that crewmembers can readily recall. The CDV method also enforces crew coordination, cross-checking, and verification, all of which aid the crewmember in overcoming the adverse effects of stress. POIs should not approve or accept the DV method for non-normal or emergency procedures unless the operator can provide substantial evidence that the method is effective for this application.

3-3404 B. 7) In the taxi and pretakeoff phases, aircraft configuration (such as flaps, trim, and speedbrakes) and flight guidance items (such as heading, flight director, altitude select panel settings, and airspeed bugs) have proven to be critical. All flightcrew members should confirm these items, and at least two crewmembers should respond to applicable checklist items.

8) On approach, flight guidance checklist items have proven to be critical items. At least two crewmembers should confirm and respond to these items. A response should be required from each pilot when the same setting is required on two separate devices (such as computers, flight instruments, or altimeters).

9) All checklist items that are critical in the before-landing phase vary with the type of airplane involved. In the operation of small airplanes, the landing gear has proven to be a critical checklist item, and both pilots should confirm and respond to this item. Although the landing gear and flaps are critical items for large, transport category airplanes, the multiple warning devices and systems that are associated with these systems make the need for a response and confirmation by both pilots less critical.

10) All checklists, except the after-takeoff and after-landing checklists, should be accomplished by one crewmember reading the checklist items and a second crewmember confirming and responding to each item. POIs shall ensure that critical items on the before-takeoff and before-landing checklists are confirmed and responded to by at least two crewmembers.

Source: FAA Order 8900.1, Volume 3, Chapter 32

It certainly makes sense that the after takeoff and after landing checklists are exceptions, you are very busy and your eyes need to be outside. But every other checklist needs to be accomplished using the challenge-do-verify method. That should end all arguments from the "I can flow any checklist" crowd. It certainly illustrates that the "silent checklist" cannot be used.

13) Checklists should not be depended on to initiate changes in aircraft configuration. Operators should key aircraft configuration changes to specific operational events. For example, the operator may direct the landing gear to be extended at glideslope intercept. For any adjustment of thrust or configuration, a command from the PF and an acknowledgement from the crewmember taking the action are required.

Source: FAA Order 8900.1, Volume 3, Chapter 32


4

Gulfstream IV after starting engines case study

image

GIV Engine Start and After Engine Start checklists, with highlighted items, from GIV Cockpit Card, extract.

According to FAA Order 8900.1, Volume 3, Chapter 32, §3-3404, ¶B.10), "All checklists, except the after-takeoff and after-landing checklists, should be accomplished by one crewmember reading the checklist items and a second crewmember confirming and responding to each item."

There are many GIV pilots who firmly believe accomplishing the "After Engine Start" checklist using a flow (DV) is faster and more likely to ensure every item is accomplished. I know from first hand experience this is not true. Allow me to demonstrate . . .

image

GIV After Engine Start Flow

Years ago, as the newest pilot in a GIV flight department, I was compelled to operate as the other pilots did. At that point I had more Gulfstream time (GIII and GV) than the other pilots combined, but I had zero time in the GIV. So I tried it their way.

The PF would race through the After Engine Start checklist in a blur, it was as if there was a prize for the quickest performance. Then the PM would pull up the paper checklist and read. As quickly as he could read the PF would answer, usually without looking. The PM was too busy to look. After a few times I noticed the PF was missing a thing or two, now and then. A few times I missed them too, I was too busy reading. We often found ourselves ready to takeoff without all the boost pumps on, a aux pump unarmed, and so fourth. In other words, we as a pilot team missed critical items.

Then I got them to agree to let me use challenge-do-response while they continued to use do-response. Then one day it hit me, let's see how long each method takes.

Method Average Time (min:sec) Mistakes
Do-Verify ("Flow") 3:14 Occasional
Challenge-Do-Verify 2:55 Rare

We adopted Challenge-Do-Verify as flight department policy and we stopped showing up at the runway with something not set correctly. So let us summarize:

  1. Challenge-Do-Verify is recommended (but not required) for commercial operators and is a best practice for everyone else when it comes to all checklists except the after takeoff and after landing checklists. See: Checklist Flows for the source of this statement.
  2. Challenge-Do-Verify is faster.
  3. Challenge-Do-Verify is more accurate.

Your honor, I rest my case.

References

(Source material)

14 CFR 25, Title 14: Aeronautics and Space, Airworthiness Standards: Transport Category Airplanes, Federal Aviation Administration, Department of Transportation

14 CFR 91, Title 14: Aeronautics and Space, General Operating and Flight Rules, Federal Aviation Administration, Department of Transportation

14 CFR 135, Title 14: Aeronautics and Space, Operating Requirements: Commuter and On Demand Operations and Rules Governing Persons on Board Such Aircraft, Federal Aviation Administration, Department of Transportation

Boyne, Walter J. “The Checklist” Air Force Magazine, August 2013, pp 52 – 56

FAA Order 8900.1 Vol 3 Ch 32, Manuals, Procedures, and Checklists for 14 CFR Parts 91K, 121, 125, and 135.