After part one and the validation plan, now let’s see how to report the results for each product you test.
Before going further, let’s check some definitions of concepts we mention in this article:
Meaning Unit Under Test, this is how we call the product being tested.
The validation results
It’s like the little brother of the test procedures. This report document contains all the results of all the tests for one sample of the product to be delivered. And this time, for each procedure there is only one test report. This will ease the traceability matrix between the validation plan and the validation results as their relation is 1:1.
In the previous article, we saw that the benefits of separating the procedures from the results in 2 different documents is to be able to perform the validation plan on several identical products. So the validation results are belonging to one and only one copy of the product. So this is the first thing to do: uniquely identify the copy of the product to be tested.
Identifying the UUT
UUT or Unit Under Test, is the generic name given to the product that is tested. So now I’ll refer to it as the UUT.
In the very beginning of the result document, we will dedicate a paragraph to identify the UUT. The purpose is to associate the result document to only one copy of the product and be sure not to mix it with another copy. Though the serial number is one piece of data to identify a product, here are a list of information useful to complete the identification:
- Model number or name. Ex.: “iPhone”
- Revision number. Ex.: “12”
- Serial number. Ex.: “98765431”
- Date of manufacture. Ex.: “October 2020”
With all those element : “iPhone 12, S/N: 98765431, October 2020”, we are very sure there is not another copy of the product with the same attributes.
In the eventuality that this precise product is malfunctioning and the customer returns it, it will be easy the find the corresponding validation results to check if all the tests had been correctly done.
To have a better overview, we can define a global validation status telling you at once if the product is OK or not. This status is the compilation of all the test results statuses. This global status can obviously be filled out once all the validation plan tests have been performed at least once. And then, for more details you can go for the procedure result you want.
How to structure the results for each procedures?
Now that the UUT is clearly identified, let’s see the reports. Just like the procedures in the validation plan, they are divided in 2 sections:
- The test results identification
- The results step by step
The test results identification
Remember, to reduce the size of the report, we decided not to put the procedures in it, so to understand the results we recall the reference of the test procedure they are about.
For comfort, we repeat the title of the procedure. It easier when you browse the results to look for an understandable title than a cryptic reference.
(So now you think: what’s the purpose of the reference then? Well, when you need to refer to the test in another document, you will prefer a short reference to a long explicit title.)
We tell the date and time of the moment the results has been obtained / the test sequence has been performed. This would be very useful in case of a future issue with this product. We will be able to cross those data with some other data, like log files or automatic reports from other tools, if we need to investigate.
Identification of the test operator
When several people have to perform the tests or when the operator is different from the developer, it is useful to identify them.
In case of a failed test, the person in charge of fixing the problem may need some more information and would question the operator (See Fix your product issue with a problem report)
The test status
This is the most important piece of information. It is the compilation of all the step statuses. It can tell in a glimpse if the UUT has passed the procedure or not. If not, we can go look for the test comments or for the specific failed step.
Commonly, for a test to be fully “OK”, all the steps must be successfully passed.
On the other side, for the test to fail, it only requires one step to be not OK!
But a failed test doesn’t necessarily imply the UUT is deficient. Maybe the procedure isn’t mature enough and needs to be updated.
It allows the operator to add some precision elements about the course of the procedure. He can write down observations and details about what happened when the test failed for example.
Those attributes we do not repeat from the procedure
As the relation between the report and the procedure are 1:1, there is no need to repeat the test description, the traceability or the initial conditions. They would only make the report heavier and its update more difficult (that’s why we cut it in half).
The results step by step
This is the same number we have on the procedure to link the result to the corresponding step.
This column allows to enter a measure eventually required by the step. If the step doesn’t ask for it, leave a blank case.
You can set a status for each step telling if it has been passed successfully or not:
- “Done” / “OK”: for a step that doesn’t expect any specific result. Therefore you simply check it was done, so a checkbox could be enough.
- “OK”: the step required a specific result that perfectly happened.
- “NOK” (for Not OK) / “Failed” : the step required a specific result that didn’t happen as expected.
Having a comment column to add details to a specific test in case of a failure can be really helpful to investigate the problem.
Some other possible attributes
When a step fails, it might be because of an issue with the UUT and a problem report should be initiated. Or it may be because of the test procedure that must be updated, in this case a reading sheet shall gather the remarks about the procedure. Therefore the Action field is used to give the reference of such file that will document the fixing action. So you can track all the strategy and all the modifications upon the procedure.
This concludes our proposition for a better structure of the product validation plan. Now I’d like to draw your attention on one last thing to reach a greater quality level.
The importance for the author, the maker and the test operator to be 3 different people
This measure can look expensive at first, but if you have to perform tests at great scale, it can significantly reduce the bias and increase the quality.
First of all, the author is the collaborator writing the tests procedures. This is the person who has designed them.
The maker is the person who has built the product or realized the service. He has shaped what was designed.
And finally the test operator is the one performing the tests (designed by the author) onto the product (built by the maker).
In a perfect world (and such a world is very expensive, we all agree), all three of them are different people. But in practice, because of the budget or the planning, all of them might be merged into the same person. This person knows perfectly its product. So when she tests it, she can unconsciously but slightly adapt the procedure to make it pass. Or worse, she will be willing to make it pass. But this is precisely the point of the test: finding the failures to fix them before delivery.
In the other way, a different person as the test operator will have no idea on how a product works. So he will methodically follow the procedure and be very good at finding those failures (from the product or from the procedure).
That is why it is very efficient to separate those 3 steps in the development of a product : design, realization and tests. It will reduce the risks of human bias or conflict of interest. But they need to work very closely to avoid inertia. A project management is always a clever compromise between quality, cost and delays.
(In the end, it is one more reason for identifying the person performing the test in the report!)
As we saw it in this article, building and performing a test plan is a real project within a project. But it is essential to ensure a high quality product.
It would be easy to believe that the multiple documents, detailed here, make the project management heavier. But at the end of the day, they don’t. Yes they require a little more time at the beginning of the project but they induce more flexibility and therefore a better resilience. Ok there are more documents but they are shorter and easier to maintain.
If the customer’s requirements evolve, you’ll only have surgical modifications to make which will be pointed out by the traceability matrices.
What about you?
How do you build your tests campaign?
Do you use a specific software?
What “tasks force” do you deploy to fix the issues?
Do you happen to reuse some tests from a product to another? From a project to another?
What are the best practices you discovered from your own experience?
📝 Contribute to our Study of documentation on business projects.