Proof of testing: why a screenshot is not enough

Full post

The report is delivered, the customer gets stuck on one scenario and the same question lands on the table: “Was this really tried?” If there is nothing but a screenshot, the answer falls back on memory. The tester describes what they did, the customer is not convinced; both are right, because neither has proof.

This article is about how that argument disappears. The answer is not more screenshots; it is a record of what was run, in which environment and when, kept in a form that cannot be edited afterwards.

What a screenshot proves, and what it does not

A screenshot shows one moment: that screen looked like that at that instant. It does not say on which build, which device or after which steps it was taken. It can be cropped, edited or lifted from another session. It is useful when a bug is found; it does not show that the twenty scenarios with no bug were actually tried.

The run record: keeping three things at once

The fix is to take the record out of human hands. An agent running on the tester’s machine takes the environment fingerprint, writes the start and end stamp of every scenario and records a digest (hash) of every attached file. The tester adds screenshots, logs and reproduction steps on top; both go out in one package.

The critical point is immutability. If neither the tester nor the platform can edit the record afterwards, a rejected report is argued over the record, not over who remembers what. The dispute process looks at the same record; a tester found right gets paid.

Independence: the finder and the fixer must not be the same party

The team that writes the product cannot test it impartially; that is not a flaw, it is nature. The developer knows how the flow is supposed to work and follows that path without noticing. The user does not know and goes a different way. So independent testing is not just “let someone from outside look”: when the party that found the finding also verifies the fix, and is not the party that made the fix, verification is genuinely independent.

Why the quota is part of the proof

Invisible competition is the twin of invisible work. If the number of testers a job takes is written up front, the tester commits time knowing how many they are competing with; when the quota fills, the job leaves the list. If the fee is fixed up front and does not change with the number of bugs found, a job with zero bugs is still paid: what is paid for is not the bug but the fact that the scenarios were run.

Summary

From the glossary: Evaluation set (eval) · Human-approved decision (human-in-the-loop)

← All posts