RGBCube/serenity
1
Fork 0
mirror of https://github.com/RGBCube/serenity synced 2025-05-31 09:58:11 +00:00
Code Issues Activity Actions

AK: Fix off-by-one error in round-to-even logic of FixedPoint

Browse source 
Because of the off-by-one error, the second bit of the fraction was
getting ignored in differentiating between fractions equal to 0.5 or
greater than 0.5. This resulted in numbers like 2.75 being considered
as having fraction equal to 0.5 and getting rounded incorrectly (to 2).
This commit is contained in:
ronak69 2023-08-12 15:50:52 +00:00 • committed by Andrew Kaster
parent 6ef2c34eb4
commit 4c6ea4a963
2 changed files with 5 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

3
Tests/AK/TestFixedPoint.cpp
View file

@ -58,6 +58,9 @@ TEST_CASE(rounding)
EXPECT_EQ(Type(-1.5).ceil(), Type(-1));
EXPECT_EQ(Type(-1.25).trunc(), Type(-1));
EXPECT_EQ(Type(2.75).rint(), Type(3));
EXPECT_EQ(Type(-1.25).rint(), Type(-1));
EXPECT_EQ(Type(0.5).lrint(), 0);
EXPECT_EQ(Type(0.5).lfloor(), 0);
EXPECT_EQ(Type(0.5).lceil(), 1);