Build a Test Strategy

What Is a Test Strategy?

A test strategy is a planning document that provides the overall direction for the software testing needs of a project. Developing a test strategy is about setting direction and resolving high-level testing questions. The value of the test strategy isn’t in the wording, the writing, or the format of the strategy; the value is in planning an approach for testing.

Test strategies can range from informal to very formal. In its simplest form, the test strategy is exactly that—a strategy. It’s a roadmap for what testing will be done, and details the way in which the testing will be accomplished:

·                     How? Answers to this question might identify the types of testing needed, such as manual or automated testing.

·                     Where? These answers detail the actual test environment, including specific server names, and may include a diagram of all the physical or logical components.

·                     Who? The test strategy must specify the testing resources and other resources needed to accomplish the testing.

·                     When? A good test strategy outlines the time of the first internal build for testing and likely includes a rough schedule for the remainder of the project.

These high-level questions need answers early in a project. The test strategy also might address related testing topics such as purchasing test tools or the defect-reporting process.

Gathering the Information to Build a Test Strategy

One of the most difficult aspects of writing a test strategy is in the timing. The strategy is needed early, but often all the information needed to write a good strategy isn’t available yet. But if you wait until you know everything about a project, you’ll miss out on planning. The answer is to start planning early, but be flexible. This is why writing a draft of the test strategy is important: The reader can tell that you recognize the fact that elements of the project will shift, and with those changes your strategy will change. You’re committing to what you’ve written, but you cannot plan for what you don’t know. Write your test strategy with the best information you can gather at the time, and expect change.

As you learn more about both the project and the product, you can draft the more detailed work of test plans, test cases, or test charters. If high-level changes take place, it’s appropriate and important to update the test strategy and to communicate those changes to the project team. If directional information changes, capture the changes. If the changes are low-level details, those changes can be reflected in status reporting or perhaps other documentation or communication throughout the project. If something strategic changes, however, it’s time to update the test strategy.

As constant informal conversations move projects along, the point isn’t having documentation that’s written perfectly. Documentation doesn’t save projects. Constant steering correction is the key—shifting as needed to accommodate the project and its testing needs as the work continues to unfold. (See principle 4 of the seven basic principles of the context-driven school.)

Building a test strategy takes legwork, likely involving multiple conversations with various team members and stakeholders. Strategic thinking is iterative, which is why working on a strategy early is important—to start digging and asking questions. Learning other people’s testing expectations is a worthwhile activity, and in some situations a political necessity. Talking through testing with other people on the team can help to determine what’s realistic and feasible. I often collect ideas from people both individually and in groups, because the information often differs in those two settings. Finding and closing such "expectation gaps" is best accomplished during development of the test strategy.

Questions Resolved by a Test Strategy

At the beginning of a software project, there are usually more questions and unknowns than answers. Sometimes the only definitive knowledge you have at the start of a project is the glimmer of an ideal product and a rigid completion date. Everything between the ideal product and the deadline can be viewed as a detail, right?

A long list of questions comes to mind at this stage, and working through the strategy helps to answer such questions. The test strategy should address at least the following major areas:

·                     Scope. What do you need to test? It can be just as important to communicate what won’t be tested as what will be tested. Sometimes scope can help to answer the question, "To what extent do we test?"

·                     Resources. Who is on the test team? Who is available to test? This information defines the testing resources. Sometimes the resources are unknown at the planning stage; perhaps resources are yet to be hired or will be outsourced. In these cases, it’s important to identify where resources are needed.

·                     Skill set. What testing skills are needed for each component (or functional area) to be tested? Instead of planning testing based on who you have on staff, consider what skills the project needs, and determine from that information whether you need to add different skill sets and resources to your team.

·                     Test environment. What environment will you use to test? Sometimes a test environment already exists, so the question seems extraneous. But sometimes this is one of the most difficult questions to resolve because the environment doesn’t exist, or the current environment isn’t adequate for the project.

·                     Test data. What data will you use to test? Do you need a sampling of production data? (Can you even get your hands on a sampling of production data?) Sometimes a snapshot of production data is the normal fare for the test environment. Sometimes getting test data into the environment is one of the largest challenges for the project.

·                     Volume. Do you need a volume of data? Planning for volume data often gets pushed aside until later in the project. But if this volume data is overlooked, resources and testing might be planned without appropriately considering volume and variety of test data.

·                     Timeframes. When do you expect to get the first build to the test team? Timeline questions can be complicated to resolve early in the planning. High-level first-pass estimates are sometimes the best you can do. The advantage of early estimations isn’t focusing on specific dates, but rather planning ideas about how and what prerequisites might be needed to accomplish testing.

·                     Defect process. What’s the process for reporting defects? Do you have a defect-reporting tool? Who can report a defect? You may have a well-established process but need to change it for this specific project. You may not have a process at all. If you need to build a new process or tool, I recommend referencing the process or tool in your test strategy document, but writing a separate document to address the defect process details.

·                     Risk. What areas hold the most risk? When you peel back a product and investigate its core components, each component evolves into its own intriguing story, in which the number and types of tests begin to unfold in your mind. And with those test ideas, associated risks and impact come to light, which feeds back into the strategic planning.

Addressing these issues resolves some of the more strategic decisions that need to be made from a testing perspective. Your test strategy may need to address more questions or include other sections. Each time I write a test strategy, it seems there are always specific considerations to address for that product, and I have yet to have develop a boilerplate that can be reused without adaptation (which goes to show that each project has its own nuances).

Optional Components of the Test Strategy Document

Addressing the major areas I’ve just discussed leads to more questions that a test strategy might consider:

·                     Software testing approach. A conversational tone can be useful here—how do you see the testing being conducted?

·                     Feature overview. Outline the product being built. At this point, I often identify the type of testing that will be conducted, and list the features and the testing in a table format.

·                     Test planning activities. If other test planning activities will be part of the schedule or project, identify those activities at this point, so that these tasks aren’t missed during estimate building. Some examples include formal risk analysis, trace matrixes, and tool purchases and implementations.

Still more topics might be added to the strategy: project estimates, test status reporting, release criteria (including a plan for a release "go/no go" meeting, and details on who will make the release decision, how, and when). In this laundry list of topics, you can see the range of informal to formal strategies, and how brief or full-blown and formal both the planning and the document might need to be. It’s important to calibrate the level of planning and formality to match the needs of the particular project and company.

Evolving the Test Strategy

As the project continues, return to your test strategy to see whether testing is progressing as planned. If the project or the testing direction has shifted, update the test strategy. If strategic decisions have changed or the testing scope has increased/decreased, communicate those changes to the team and to the project stakeholders. In formal environments where a final report might be needed, use the report to identify changes made during the project that were strategic in nature and why the changes took place.

Summary

For me, strategic thinking and planning is one of the most interesting aspects of project work. It’s important to begin early, have conversations with a variety of people on the team, host brainstorming meetings, and talk about risk (whether informally or through a formal risk analysis). I communicate the strategy both through conversations and through the test strategy document. When a test strategy represents full-bodied thinking and is well articulated, that test strategy has meaning and value. Throughout the project, as I learn more about both the project and the product, I evolve my test strategy, making sure that my testing represents truly strategic thinking.