Unicode

Decodes the character that starts at s, where source_length bytes are available. Returns the code point, the number of bytes it took and how it went. A call with bytes available always takes at least one.

code, size, status := character_utf8_to_utf32("\xe2\x82\xac".data, 3);
assert(status == .CONVERSION_OK && code == 0x20ac && size == 3);

bad := "\xc0\x80";
code, size, status = character_utf8_to_utf32(bad.data, bad.count, true);
assert(status == .SOURCE_ILLEGAL && code == UNI_REPLACEMENT_CHAR && size == 2);

stray := "\x80";
code, size, status = character_utf8_to_utf32(stray.data, stray.count);
assert(status == .SOURCE_ILLEGAL && code == UNI_REPLACEMENT_CHAR && size == 1);

code, size, status = character_utf8_to_utf32(bad.data, bad.count);
assert(status == .CONVERSION_OK && code == UNI_REPLACEMENT_CHAR && size == 2);

Writes the UTF-8 form of ch into the buffer preallocated_string points to, and sets its count to the number of bytes used (1 to 4). The buffer must have room for them, or an assert fails; four bytes always do.

buffer: [4] u8;
encoded := string.{ 4, buffer.data };
character_utf32_to_utf8(0x20ac, *encoded);
assert(encoded == "\xe2\x82\xac");

A pointer to the byte after the character that starts at s, going by its first byte only. Nothing is checked: a truncated or malformed character at the end of a buffer gives a pointer past the end.

Decodes the first character of s and removes it from the start of s. Returns its code point and how decoding went, as character_utf8_to_utf32 does without strict.

text := "a\xc3\xa9";
code, status := utf8_next_character(*text);
assert(status == .CONVERSION_OK && code == #char "a");
code, status = utf8_next_character(*text);
assert(status == .CONVERSION_OK && code == 0xe9 && text.count == 0);

Loops over the code points of a string: for: utf8_iter text { ... }. The body sees it, the code point, and it_index, the number of characters before it (not the byte offset). for <: goes from the last character to the first, and it_index is -1 there. Iterating by pointer is not allowed.

text := "a\xc3\xa9\xe2\x82\xac";
codes: [..] u32;
defer array_free(codes);
for: utf8_iter text array_add(*codes, it);
assert(codes.count == 3 && codes[0] == #char "a" && codes[1] == 0xe9 && codes[2] == 0x20ac);

How decoding one character went.

U+FFFD, the code point that stands in for a character that could not be decoded or encoded.

The highest code point of the Basic Multilingual Plane, U+FFFF.

The highest code point Unicode defines, U+10FFFF. Strict decoding and encoding reject anything above it.

The highest value the six byte form of UTF-8 can carry, 0x7FFFFFFF. No decoder here returns a value above UNI_MAX_UTF16; the longer forms decode as UNI_REPLACEMENT_CHAR.

How many continuation bytes follow a first byte, for each byte value. It is 0 for ASCII and for continuation bytes themselves, 1 to 3 for the lead bytes of valid sequences and 4 or 5 for the bytes of the old five and six byte forms (0xF8 and up).

assert(trailingBytesForUTF8[#char "a"] == 0);
assert(trailingBytesForUTF8[0xc3] == 1);
assert(trailingBytesForUTF8[0xe2] == 2);
assert(trailingBytesForUTF8[0xf0] == 3);

The bits of the first byte of a sequence that hold the code point, indexed by the number of continuation bytes that follow (0 to 5).

For a sequence of n + 1 bytes, the amount that classic decoders subtract after adding up the bytes shifted by six bits each. A table for hand-written decoders; the procedures here do not need it.

The top bits of the first byte of a sequence, by sequence length. A table for hand-written encoders; the procedures here do not need it.