A statistical test asks whether a dataset shows a particular kind of departure from a stated model. It does not read the design, inspect secret handling or prove suitability for every use. This guide explains how to interpret a randomness-test report responsibly and how to keep its conclusions within the evidence collected. It is intended for technical review and reproducible laboratory work.
Separate the test question from the security claim
NIST SP 800-22 explicitly warns that statistical testing does not replace cryptanalysis or certify every application. That limitation should appear near a report’s main conclusion, not only in a footnote. A test may examine a pattern in observed output while a security claim concerns what an informed adversary could predict. These are different questions. Begin by writing the claim in plain language, then list which parts the selected tests actually address. Any remaining design or implementation questions need their own evidence.
A comparison table for report interpretation
| Item | Evidence or distinction | Review question |
|---|---|---|
| Output statistical test | Looks for specified deviations in a sample | Which data and test assumptions were used? |
| Entropy-source assessment | Examines the source and its uncertainty model | Were observations collected at the appropriate boundary? |
| Implementation review | Examines how the design is realised | Does the evaluated code match the documented mechanism? |
| Operational health check | Monitors defined failure conditions | How does the system react when a check fails? |
The NIST presentation on random-number standards distinguishes statistical screening from the broader validation model. These activities can support one another, but a result from one row should not be relabelled as evidence that every other row has passed.
Record the dataset before looking at the result
Document how much data was collected, when it was collected and whether any records were omitted. Identify the collection point, generator build and operating mode. Retain a digest of the input file and the exact transformation used to prepare it for a tool. A conversion that changes bit order or drops incomplete records can affect interpretation. If a test requires a particular input length, check that requirement before execution. Do not choose a favourable subset after seeing failures and present it as if it were the original sampling plan.
Interpret p-values without turning them into security scores
A p-value belongs to a defined statistical procedure and null model. It is not the probability that a generator is secure, nor a percentage rating of product quality. A very small value can motivate investigation of the relevant pattern, but the explanation may involve data preparation or a mismatch between assumptions and the sample. A value that does not cross the chosen threshold leaves many other properties unexamined. Record the decision rule before interpreting the run and explain the rule in the report so readers can understand the conclusion rather than merely admire a dashboard.
Think about repeated tests and planned investigation
A large test campaign contains many opportunities for an unusual result. Review the full pattern of outcomes rather than treating every isolated failure as definitive proof of a defect or every pass as a clean bill of health. Define an investigation process that preserves the original result, checks tool configuration and examines the relevant component. A rerun can be useful evidence, but it should not erase the earlier run. If the team changes the implementation, distinguish the new build from a repeat test of the same build. Otherwise a result history can conceal the very fault it was meant to explain.
Keep source evaluation and output screening distinct
SP 800-90B addresses entropy sources rather than merely the appearance of a final output file. For a practical review, ask whether the reported dataset came from the source, a conditioning stage or the final generator. Explain what that collection point permits the reviewer to infer. A polished final stream does not expose every assumption about its upstream source. Conversely, an unusual raw physical signal may be intentionally processed within a documented design. The correct interpretation depends on the boundary and model, not on a preference for a neat-looking histogram.
Statistical-report checklist
- State the question, sample plan and collection boundary.
- Record tool version, input digest and preprocessing.
- Define decision rules before reviewing outcomes.
- Preserve failed runs and explain follow-up work.
- Limit the conclusion to properties actually examined.
The final report should allow another analyst to repeat the calculation and understand why the result matters. Include the unresolved questions and the evidence needed to answer them. An honest report can be useful even when it ends with further investigation required; a sweeping security claim unsupported by the test design cannot.
Frequently asked questions
Does a pass mean the generator is unpredictable?
No. Passing selected statistical tests does not establish every security property. Design, state handling, source assumptions and the intended application require additional review.
Should an isolated failure always be deleted and rerun?
No. Preserve it and investigate the sampling, tool setup and implementation. A later run is additional evidence rather than a replacement history.
Is a p-value the chance that the product is safe?
No. It has meaning within a specified statistical test and model. It is not a product-security probability or quality score.
Sources and verification
- SP 800-22 Revision 1a: statistical randomness tests — NIST — 2026-10-23
- NIST standards on random numbers, 2024 presentation — NIST — 2026-10-23
- SP 800-90B: entropy sources — NIST — 2026-10-23