mirror of
https://github.com/RGBCube/serenity
synced 2025-10-31 05:42:43 +00:00
119 lines
4.6 KiB
Markdown
119 lines
4.6 KiB
Markdown
# LibTest/Randomized
|
|
|
|
Classes in this folder implement a randomized ("property based") testing library
|
|
for SerenityOS.
|
|
|
|
## Example
|
|
|
|
```cpp
|
|
RANDOMIZED_TEST_CASE(addition_commutative)
|
|
{
|
|
GEN(int_a, Gen::unsigned_int());
|
|
GEN(int_b, Gen::unsigned_int());
|
|
CSSPixels a(int_a);
|
|
CSSPixels b(int_b);
|
|
EXPECT_EQ(a + b, b + a);
|
|
}
|
|
```
|
|
|
|
The runner will then run the test case many times, with random data in each of
|
|
the `GEN`'d variables. If it finds any set of values where the expectation fails,
|
|
it shrinks it to a minimal failing example and presents it to the user:
|
|
|
|
```
|
|
Running test 'addition_commutative'.
|
|
int_a = 0
|
|
int_b = 1
|
|
FAIL: /.../SomeTest.cpp:26: EXPECT_EQ(a + b, b + a) failed with lhs=0 and rhs=1
|
|
Failed test 'addition_commutative' in 0ms
|
|
```
|
|
|
|
(Note that the runner most likely started out with much larger values like
|
|
1587197 and 753361. The 0 and 1 are the shrunk, minimal values.)
|
|
|
|
In the above case, the test runner tells us that `CSSPixels(0) + CSSPixels(1)`
|
|
wasn't equal to `CSSPixels(1) + CSSPixels(0)`.
|
|
|
|
> [!NOTE]
|
|
> I made this failing example up. Don't worry, `CSSPixels` is fine :^)
|
|
|
|
## Property based testing?
|
|
|
|
Property based testing (PBT) involves generating random test inputs and checking
|
|
that they satisfy certain properties. This allows testing a large number of
|
|
possible cases automatically, and is exceptionally good at finding edge cases.
|
|
|
|
When a failure is found, it's automatically shrunk to a minimal reproducing
|
|
case. (This is good, because random values contain a lot of unimportant details
|
|
that only hamper understanding of the root cause. See http://sscce.org)
|
|
|
|
This particular implementation belongs to the "internal shrinking" family of PBT
|
|
libraries (Hypothesis, minithesis, elm-test, ...). This is important for the DX
|
|
as it gives the most resilient shrinkers without any user effort, compared to
|
|
the "integrated shrinking" lazy-tree based implementations (Hedgehog etc.) or
|
|
the manual shrinking ones (original QuickCheck etc.). (More information in this
|
|
talk: https://www.youtube.com/watch?v=WE5bmt0zBxg)
|
|
|
|
## Implementation
|
|
|
|
On a very high level, the PBT runner remembers the random bits (`RandomRun`)
|
|
used when generating the random value, and when it finds a value that fails the
|
|
user's test, instead of trying to shrink the generated value it tries to shrink
|
|
these random bits and then generate a new value from them.
|
|
|
|
This makes shrinking work automatically on any data you might want to generate.
|
|
|
|
Generators are implemented in such a way that a smaller/shorter `RandomRun` will
|
|
result in a simpler generated value.
|
|
|
|
If the new RandomRun can be successfully turned into a value and if the test
|
|
still fails on that new simpler value, we keep the RandomRun as our new best and
|
|
try again. Once we can't improve things anymore, we turn on error reporting, run
|
|
the test one last time with the final RandomRun, and report the failing value to
|
|
the user.
|
|
|
|
## Code organization
|
|
|
|
- `TestResult.h`
|
|
- Defines an enum class TestResult.
|
|
This expands the typical "passed / failed": we also need to care about
|
|
a generator rejecting a RandomRun (eg. when the user calls the ASSUME(...)
|
|
macro with a predicate that can't be satisfied).
|
|
|
|
- `Generator.h`
|
|
- Contains generators: fns of shape T(), eg. `u32 Gen::unsigned_int(u32 max)`
|
|
- These implicitly depend on a RandomnessSource held by the singleton
|
|
TestSuite.
|
|
- These can be called directly, but the top-level use by the user should always
|
|
happen via the GEN(...) macro which makes sure the generated value gets
|
|
logged to the user in case of a failure.
|
|
- Example:
|
|
`Gen::vector(1, 4, []() { return Gen::unsigned_int(5); })`
|
|
generates vectors of length between 1 and 4, of unsigned ints in range 0..5.
|
|
Eg. `{2,5,3}`, `{0}`, `{1,5,5,2}`.
|
|
|
|
- `RandomnessSource.h`
|
|
- A source of random bits.
|
|
- There are two variants of `RandomnessSource`:
|
|
- Live: gives `AK/Random` u32 values and remembers them into a `RandomRun`
|
|
- Recorded: gives (replays) u32 values from a static `RandomRun`
|
|
|
|
- `RandomRun.h`
|
|
- A finite sequence of random bits (in practice, `u32`s).
|
|
- Example: `{2,5,0,11,8,0,0,1}`
|
|
|
|
- `ShrinkCommand.h`
|
|
- A high-level recipe for how to try and minimize a given `RandomRun`.
|
|
- For example, "zero this contiguous chunk of it" or "minimize the number on
|
|
this index using binary search".
|
|
- These later get interpreted by the PBT runner on a specific `RandomRun`.
|
|
|
|
- `Chunk.h`
|
|
- A description of a contiguous `RandomRun` slice.
|
|
- Example: `Chunk{size = 4, index = 2}`: [_,_,X,X,X,X,...]
|
|
|
|
- `Shrink.h`
|
|
- Algorithms for interpreting `ShrinkCommand`s and the main shrinking loop
|
|
|
|
- `TestCase.h`
|
|
- The `TestCase::randomized(...)` function contains the main testing loop
|