1
Fork 0
mirror of https://github.com/RGBCube/serenity synced 2025-07-27 03:07:36 +00:00

AK: Do not VERIFY on invalid code point bytes in UTF8View

The previous behavior was to always VERIFY that the UTF-8 bytes were
valid when iterating over the code points of an UTF8View. This change
makes it so we instead output the 0xFFFD 'REPLACEMENT CHARACTER'
code point when encountering invalid bytes, and keep iterating the
view after skipping one byte.

Leaving the decision to the consumer would break symmetry with the
UTF32View API, which would in turn require heavy refactoring and/or
code duplication in generic code such as the one found in
Gfx::Painter and the Shell.

To make it easier for the consumers to detect the original bytes, we
provide a new method on the iterator that returns a Span over the
data that has been decoded. This method is immediately used in the
TextNode::compute_text_for_rendering method, which previously did
this in a ad-hoc waay.

This also add tests for the new behavior in TestUtf8.cpp, as well
as reinforcements to the existing tests to check if the underlying
bytes match up with their expected values.
This commit is contained in:
DexesTTP 2021-05-30 18:52:24 +02:00 committed by Ali Mohammad Pur
parent 32ee195d62
commit e01f1c949f
6 changed files with 135 additions and 20 deletions

View file

@ -33,7 +33,13 @@ public:
return m_ptr - other.m_ptr;
}
size_t code_point_length_in_bytes() const;
// Note : These methods return the information about the underlying UTF-8 bytes.
// If the UTF-8 string encoding is not valid at the iterator's position, then the underlying bytes might be different from the
// decoded character's re-encoded bytes (which will be an `0xFFFD REPLACEMENT CHARACTER` with an UTF-8 length of three bytes).
// If your code relies on the decoded character being equivalent to the re-encoded character, use the `UTF8View::validate()`
// method on the view prior to using its iterator.
size_t underlying_code_point_length_in_bytes() const;
ReadonlyBytes underlying_code_point_bytes() const;
bool done() const { return m_length == 0; }
private: