In this article, we’ll see how to build a validation plan ensuring your product is matching all your customer’s requests.
Before going further, let’s check some definitions of concepts we mention in this article:
Meaning Unit Under Test, this is how we call the product being tested.
What is it used for?
The more complex products you create, the more difficult it is to check they fit all the customer’s specifications. We saw it in a previous post about the traceability matrices, the dependency connections between the customer’s need and your product’s features can be numerous! And when you deliver your product (or your service), you have to be sure that all those features are correctly working.
Besides, sometimes a new feature request happens and challenges all the work previously done: “Now the car must carry 7 people, not just 5”. “OK so the car must be longer, the engine stronger…”. Those evolutions shall not alter what was correctly working earlier. To validate this point we have to perform “regression tests”.
This is why creating a list of test checking that each and every feature works correctly with the other is crucial. Such a list is what we call a validation plan.
What does it look like?
Depending on the complexity of you project, the way you describe your tests can takes different shapes: from a very light one to the most bullet-proof. We have identified 3 ways:
1. In the same document than your product’s specifications
Your customer has described his need in a requirements document. You reply to it in a specifications document where you explain all the features of you future product. Each of those features is followed by a corresponding test with a status checkbox telling if it is OK or not.
Like this you reduce the number of documents by gathering everything in one.
But if the project grows and the product requires more complex features, you may want to test several of them at once if you can. This format doesn’t allow this kind of test management: you cannot “factorize” the tests and therefore you may have redundancies. Which is not very efficient.
2. List the tests in a separate document
Your customer has described his needs in a requirements document. You reply to it in a specifications document. And you describe the tests on those specifications in a third document. Each test can have specific fields to precise details on the results (measured values, expected results, status, comments…).
The benefit is that you can now factorize the tests: in one bigger test you can check the validity of several specifications at once and reduce the time spent on this activity.
The drawback is now you have to keep track of the links between the specifications and their tests. But it is easy to do so with traceability matrices.
Now, a new hidden problematic comes out. It only occurs if you have to perform your tests several times. Let’s take a concrete example. Imagine you are a reusable anti-COVID masks manufacturer (not the disposable ones). So you deliver lots of masks, all the same, and you have to test each and everyone of them before to put them on the market. For each mask you have to duplicate the test document to perform the campaign, and to fill it with the mask tests results but the test descriptions are exactly the same. That makes a lot of redundancy between the 2 documents. Let’s see how to optimize this…
3. Cut the test document in 2: the procedures and the results
Here is the best way to do, from our point of view:
- The tests procedures are detailed in a first document: the validation plan.
- All the results for the same product are gathered in its own document: the validation results.
OK the obvious drawback is the multiplication of the documents. From “one doc to rule them all” we go to at least 3 (specifications, validation plan and validation results) and more if you deliver several identical products. And off-course you have to manage a traceability matrix for each document relation (red link).
it is the best way for us because of the following reasons:
- Each document deals with only one aspect of the project: specifications design the solution, validation plan builds the tests, validation results report the status on the product.
- Each document is therefore shorter and so easier to maintain up to date.
- Consequently, each document can evolve independently from the other one. And if there is an impact, the traceability matrices will tell you exactly where.
In the following sections, we will explain the organizations of the validation plan and results in two separated document. But even if they are merged it works the same.
The validation plan (the procedures)
In the validation plan you will define all the procedures for the required tests to validate that your product correctly matches its specifications.
The validation plan will contain enough tests if and only if all of the specifications are covered by one test at least. Although it doesn’t mean there should be one test for each specification. And this is one major benefit of the validation plan, you can do:
- one test for one specification
- one test for several specifications at once
- several tests to check only one complex specification
That’s why traceability matrices are very useful! By tracking all those relations they ensure at least one test covers each specification. If it is not the case, you know you need more tests and what to do them on.
How to structure a test procedure?
A test procedure is divided in two parts:
- The test identification
- The test steps
The test identification
This identification gathers several metadata about the test itself. Here are the most essential ones.
A unique reference
This reference allows to identify instantly the test without any doubt. It will be useful in the traceability matrices. It can be followed by a revision number to track the evolutions on the test: each modification of the text increases its revision number.
It is required to have a title telling in a few words the purpose of the test. It is essential to quickly know what the test is about, and refer it easily in a conversation with coworkers for example.
Ex.: “Check the front light”
You explain here the purpose of the test. What is it doing, why and maybe how the test is supposed to end. Sometimes the person writing the test (the author) is not the same person to perform it (the operator). The description will give the operator some details about the context and what he should expect.
Ex.: “The purpose of this test is to check the front light of the car are correctly working and manageable from the pilot seat.”
This is a very important item! Setting initial conditions allows to sequence your validation plan.
Some tests can require to be done within specific conditions. They may be possible only under certain circumstances.
For example, a test might be so long, it would be easier to cut it in half. Therefore the second part can only be performed if the first part is OK. So the success of part one is the “initial condition” for part two.
Another example, several tests can have their first steps in common. You may want to gather those redundant steps in a dedicated test to do it just once and cut this part from the other tests. You will gain some precious time and those tests require to have this initial condition (the first steps) to be previously done.
Ex.: This test requires the followings tests to be passing:
- NPT_ProjectVehicule_VP_0230.1 : Check the car start-up
- NPT_ProjectVehicule_VP_0254.1 : Check the car’s wiring
With this item, you list all the specifications this test validates.
So the compilation of all the tests traceabilities tells you exactly which specifications are tested and which are not. So you can calculate the coverage of your validation plan, you know if it is complet or needs some more tests.
The other possible items
Here are some items you can add to the test identifications:
- The kind of test: tests can be categorized depending on the way to perform them. There are inspection test where you just have to visually check the presence of an element. There are the certification tests where you need to have a certificate from the supplier officially telling you the device is working. The demonstration tests that require to manipulate the device to obtain the expected behavior…
- The gravity: some tests may be more important than other. If a low gravity test is not ok, may be it’s not a big deal and the device is still good enough to be delivered. By giving gravity indication on tests, you can sort them to prioritize the most important ones.
The test steps
This is here we go deep into the test procedure, step by step. The procedure is basically a table where each line is a test step. I strongly recommend to explicitly sequence the procedure step by step. The best procedure is the one that can be performed by a total stranger to the project as everything is completely detailed.
This exhaustiveness principle brings several benefits:
- Having a crystal clear procedure allows the test operator to focus on the unit under test and how it reacts. But if he has to question the procedure or to guess if he understands it well enough, he will also question the results he observes. He won’t know if he’s doing right or if he misses something if the procedure allows to doubt.
- The procedure can be performed by a stranger and this stranger won’t be biased. Such an operator is better at finding bugs or fault (See Fix your product issue with a problem report). Besides if the procedure can be performed by a stranger, and if you need more people to do the tests because of a deadline, anyone can help you!
Let’s see the different columns we suggest you to put in your test procedure:
This number allows precisely identifying the step in a comment, a potential exchange or even in another step to do a loop in the procedure : “If it is NOK, go back to step #6“
The step description
A step can be define in 2 ways:
- it is either an action to perform on the UUT without expecting any specific behavior
- or it is an action implying a specific behavior from the UUT we expect to happen. This is this result we want to validate!
Here you need to be the most obvious possible about the action to perform. Do not let any room for doubt, do not hesitate to push an open door. The operator should not have any choice, even if it means to write a long description.
The expected result
This item is not systematic. If the action doesn’t expect a specific one, do not tell one. On the other hand, if there is one and if it will decide the status of the test, therefore you need to be very specific too:
- If what is expected is a behavior, then be very detailed about it without any possible doubt. We don’t want the operator to miss it
- If a measure is expected: a number of items, a voltage, a length… Then tell the expected measure’s value, unit and tolerance.
Example: you expect a board to be 2 meters long more or less 1 centimeter (or 0,01 m).
If the measured length is 2,005 m then the test is OK.
Some other attributes
Here are some more attributes to add to the procedure table:
- Traceability : you can tell for each step the specification it validates. You may just fill it for the steps with an expected result and not for the others. It allows to rapidly see which part of the procedure is critical but it can be redundant with the traceability given in the test identification.
- How to manage NOK steps: when a step is OK, we implicitly go for the next one. But what if the step is not OK (NOK)? Should we leave the UUT in this state? You may have to properly shut it down. So you could describe in a new column what to do if the step is NOK. You can write a “shut down procedure” in the validation plan, and you can refer to this procedure in the step in the case where the step is NOK.
Et Voilà for the structuring of the test procedures. Don’t forget to add the traceability matrices at the end of the validation plan to keep track of the coverage.
That’s it for today. There were lots of new concepts in this long article. Next time we will see how to organize the report of the results on a specific unit under test.
📝 Contribute to our Study of documentation on business projects.