Green Tests, Reversed Behavior
A fully passing test suite is comforting. But tests answer only the questions they ask. If code and tests use the same dictionary, constant, or mistaken assumption, the system can be perfectly self-consistent—and wrong about the outside world.
This is particularly dangerous in protocol engineering. A message may have the right length, pass its integrity check, and be accepted by a parser. Yet the value believed to mean “start” may actually mean “stop.”
Three different correctness claims
A protocol test can make at least three distinct claims:
- Syntactic correctness:* does the message have the expected shape and length?
- Semantic correctness:* do its fields carry the meaning defined by the protocol?
- Operational effect:* did the target interpret the message and produce the expected outcome?
Success at the first layer does not automatically establish the second or third. A valid integrity value may merely transport the wrong meaning without corruption.
How a mirrored error appears
Imagine an application table saying A = start and B = stop. The test derives its expected output from that same table. If the real protocol defines the reverse, the test still passes because implementation and test share one error.
The problem grows when the same protocol fact is copied into a client, packet builder, dissector, and detection rule. Four copies become four potential sources of truth. Correcting one does not correct the rest.
We encountered this pattern during authorized protocol engineering work: one component produced a message, another assigned meaning to it, and a test confirmed their shared assumption. The underlying problem was not one coding mistake. It was the architecture of evidence*.
The external oracle that breaks the loop
An oracle is an independent authority for the expected result. For protocol semantics, confidence is stronger when three forms of evidence converge:
- The normative standard or authoritative vendor documentation,
- Independent dissection of real bytes from a capture with known provenance,
- The observed result on a target in an authorized laboratory.
Agreement raises confidence. Disagreement is more useful than a generic failed test: it identifies the assumption that needs investigation.
Five practices for stronger tests
Do not derive expectations from the implementation.* Test vectors should not simply reuse production constants.
State field meaning explicitly.* Replace “the value should be 2” with “value 2 represents behavior X in the cited specification.”
Use golden captures.* Compare output with an independently obtained, provenance-recorded capture.
Add counterexamples.* Test not only acceptance of the intended value, but rejection—or a distinct effect—for its semantic opposite.
Manage protocol truth once.* Generating client, builder, dissector, and detection representations from one schema reduces the drift surface. The schema itself must still be checked against independent evidence.
Reading green correctly
“All tests pass” is not a conclusion; it is an unfinished sentence. Complete it: which behavior was validated, against which independent authority, in which environment?*
A green test is valuable. It is evidence only for what it actually measures.