Decodes the character that starts at s, where source_length bytes are available. Returns the code
point, the number of bytes it took and how it went. A call with bytes available always takes at least one.
character_utf8_to_utf32 :: (
s: *u8,
source_length: s64,
$strict := false
) -> (u32, s64, Unicode_Result)Input that ends early gives UNI_REPLACEMENT_CHAR, all source_length bytes consumed and
SOURCE_EXHAUSTED; no bytes at all gives 0 consumed.
A broken byte structure is an error in both modes: a continuation byte (0x80 to 0xBF) where a character
should start, or a byte that is not a continuation byte inside a character. The result is
UNI_REPLACEMENT_CHAR with SOURCE_ILLEGAL, and the bytes consumed are the lead byte and the continuation
bytes that were fine, so decoding can resume at the offending byte (a stray continuation byte takes just itself).
A well-formed sequence with no character behind it (an overlong encoding, a surrogate from 0xD800 to
0xDFFF, or a value above UNI_MAX_UTF16) gives UNI_REPLACEMENT_CHAR and takes the whole sequence. With
strict the result is SOURCE_ILLEGAL; without it, which is the default, the sequence is replaced quietly with
CONVERSION_OK, so the result is always a Unicode scalar value.
Parameters
- s: *u8
- source_length: s64
- $strict := false
Returns
- u32
- s64
code, size, status := character_utf8_to_utf32("\xe2\x82\xac".data, 3);
assert(status == .CONVERSION_OK && code == 0x20ac && size == 3);
bad := "\xc0\x80";
code, size, status = character_utf8_to_utf32(bad.data, bad.count, true);
assert(status == .SOURCE_ILLEGAL && code == UNI_REPLACEMENT_CHAR && size == 2);
stray := "\x80";
code, size, status = character_utf8_to_utf32(stray.data, stray.count);
assert(status == .SOURCE_ILLEGAL && code == UNI_REPLACEMENT_CHAR && size == 1);
code, size, status = character_utf8_to_utf32(bad.data, bad.count);
assert(status == .CONVERSION_OK && code == UNI_REPLACEMENT_CHAR && size == 2);