Milo

Appearance

std/unicode ​

UTF-8 decoding, code-point traversal, case mapping, display width, and UTF-16 conversion. ASCII byte classification lives in std/string and uses the ascii prefix.

milo
from "std/unicode" import { codepoints, displayWidth, isAlphaStr, isNumeric }

Text queries ​

  • codepointCount(s) counts decoded Unicode code points.
  • codepoints(s) returns the decoded i32 code points.
  • nextCodepointBoundary(s, at) returns a byte offset.
  • displayWidth(s) returns terminal columns.
  • truncateToWidth(s, maxCols) returns a byte offset that fits the column limit.
  • truncateToCodepoints(s, n), truncateToBytes(s, maxBytes) and truncateToUtf16(s, maxUnits) return the byte offset of the longest prefix within that budget, always on a character boundary: s.substr(0, truncateToBytes(s, 255)).
  • utf16Length(s) is JavaScript's .length, for limits a browser also enforces.
  • isValidUtf8(s) checks bytes off the wire; repairUtf8(s) replaces each ill-formed subsequence with one U+FFFD exactly as a browser's TextDecoder does, and returns a valid string untouched.
  • isAlphaStr(s) and isNumeric(s) classify ASCII-only strings.

Code points ​

decodeCodepoint returns a CodePoint containing the decoded value and byte width. encodeCodepoint appends one code point to a UTF-8 string. Width and emoji helpers take an i32 code point, while the surrogate helpers explicitly convert between code points and UTF-16 units.

milo
let points = codepoints("Milo")
let columns = displayWidth("Milo")

Case mapping ​

unicodeLower(s) and unicodeUpper(s) map every code point by the Unicode simple case mapping, so unicodeUpper("привет") is "ПРИВЕТ". The built-in s.toLower() and s.toUpper() change ASCII letters only, and that is deliberate: protocol tokens (header names, URL schemes, hex digits) must compare ASCII-insensitively, and a Unicode mapping would let the Kelvin sign K (U+212A) lowercase to an ASCII k.

milo
from "std/unicode" import { unicodeLower, unicodeUpper }

fn main() {
    print(unicodeUpper("àéî straße"))   // ÀÉÎ STRAßE
    print(unicodeLower("ΣΑΣ"))          // σασ
}

The mapping is one code point to one, generated from UnicodeData.txt by scripts/gen-case-map.ts. Mappings that change length or depend on context are SpecialCasing.txt's, and are not applied: ß has no one-code-point uppercase and stays ß, and a word-final Σ lowercases to σ, not ς.

API reference ​

CodePoint ​

milo
pub struct CodePoint

One decoded codepoint plus the number of bytes it occupied, so a caller can advance without re-inspecting the lead byte.

Fields: value: i32, size: i64.

Functions ​

charWidth ​

milo
pub fn charWidth(cp: i32): i64

Terminal cells occupied by a single codepoint: 0, 1, or 2.

Control characters report 0 because they advance no column when printed — callers that need to reject them should filter before measuring, not rely on a negative sentinel.

codepointCount ​

milo
pub fn codepointCount(s: &string): i64

Number of Unicode codepoints in a string (not bytes).

codepoints ​

milo
pub fn codepoints(s: &string): Vec<i32>

Decode UTF-8 string into Unicode codepoints. Allocates; prefer decodeCodepoint in a scan loop.

decodeCodepoint ​

milo
pub fn decodeCodepoint(s: &string, at: i64): CodePoint

Decode the UTF-8 codepoint starting at byte offset at, without allocating. This is the primitive to scan text with: a lexer keeps its own byte cursor and advances by .size, so no Vec<i32> is materialized just to walk a string.

Malformed input never reads out of bounds and never returns a partial codepoint: a bad lead byte, a truncated tail, or a non-continuation byte all yield U+FFFD with size 1, so a scan advances and terminates on any input. Overlong forms and surrogate-range values are rejected the same way, which is what keeps decode/encode round-trips honest.

displayWidth ​

milo
pub fn displayWidth(s: &string): i64

Columns a string occupies in a terminal.

Measured over grapheme clusters, not codepoints, so the multi-codepoint sequences that real text is full of collapse to the one glyph a terminal actually paints: ZWJ emoji sequences (family, professions) count once, a flag's two regional indicators count once, skin tones and variation selectors fold into the emoji they modify.

encodeCodepoint ​

milo
pub fn encodeCodepoint(out: &mut string, cp: i32): void

Append cp to out as UTF-8. Invalid codepoints encode as U+FFFD rather than emitting bytes that would not decode back.

fromSurrogatePair ​

milo
pub fn fromSurrogatePair(high: i32, low: i32): i32

Combine a surrogate pair back into a single codepoint.

highSurrogate ​

milo
pub fn highSurrogate(cp: i32): i32

isAlphaStr ​

milo
pub fn isAlphaStr(s: &string): bool

Is every byte an ASCII letter? ASCII only, despite living here — it walks BYTES calling asciiIsAlpha, so any multibyte letter fails on its first continuation byte: isAlphaStr("café"), isAlphaStr("привет") and isAlphaStr("αβγ") are all false. Use it as an ASCII-identifier test, not as a Unicode letter test.

The code-point layer in this module IS correct (codepointCount("café") is 4); it is the classification that is ASCII, because a Unicode general-category table is a generated artefact nobody has built here yet (docs/backlog.md).

isCombining ​

milo
pub fn isCombining(cp: i32): bool

Marks that attach to a preceding base character and advance no column.

isNumeric ​

milo
pub fn isNumeric(s: &string): bool

Check if an entire string is numeric (all digits).

isRegionalIndicator ​

milo
pub fn isRegionalIndicator(cp: i32): bool

Regional indicator symbols — two in a row form one flag glyph.

isSkinToneModifier ​

milo
pub fn isSkinToneModifier(cp: i32): bool

Emoji skin-tone modifiers, absorbed by the emoji they follow.

isSupplementary ​

milo
pub fn isSupplementary(cp: i32): bool

UTF-16 conversion. A codepoint above the BMP is a single codepoint but TWO UTF-16 code units, encoded as a surrogate pair — so a byte- or codepoint-oriented count is not a UTF-16 index. Needed by anything crossing a UTF-16 boundary (Windows wide-char APIs, JVM strings).

isValidUtf8 ​

milo
pub fn isValidUtf8(s: &string): bool

Whether s is well-formed UTF-8: no stray continuation byte, truncated sequence, overlong form, surrogate, or value past U+10FFFF. Bytes off the wire (a request body, a file, a socket) carry no such guarantee; a string built from literals does.

isWide ​

milo
pub fn isWide(cp: i32): bool

East Asian Wide/Fullwidth plus the emoji blocks that render double-width.

isZeroWidthFormat ​

milo
pub fn isZeroWidthFormat(cp: i32): bool

Format/invisible characters: joiners, bidi controls, variation selectors, BOM.

lowerCodepoint ​

milo
pub fn lowerCodepoint(cp: i32): i32

The simple lowercase mapping of code point cp (UnicodeData.txt field 13), or cp itself when it has none. One code point to one: the full mappings in SpecialCasing.txt (context-dependent final sigma, language rules) are out of scope.

lowSurrogate ​

milo
pub fn lowSurrogate(cp: i32): i32

nextCodepointBoundary ​

milo
pub fn nextCodepointBoundary(s: &string, at: i64): i64

Byte offset of the next codepoint boundary at or after at.

repairUtf8 ​

milo
pub fn repairUtf8(s: &string): string

s with every maximal ill-formed subsequence replaced by one U+FFFD, the substitution the WHATWG Encoding standard (and so TextDecoder and every browser) makes. Returns s unchanged, without copying, when it is already valid.

This differs from decodeCodepoint's one-U+FFFD-per-byte on a truncated sequence: E2 82 (a 3-byte form cut short) is ONE replacement here, so the server and a browser agree on the repaired text and its length.

truncateToBytes ​

milo
pub fn truncateToBytes(s: &string, maxBytes: i64): i64

Byte offset just past the longest prefix of s that is at most maxBytes bytes and ends on a code point boundary, for a byte budget (a column width in a database, a header limit) that must not leave half a character behind.

truncateToCodepoints ​

milo
pub fn truncateToCodepoints(s: &string, n: i64): i64

Byte offset just past the first n code points of s (s.len when it has fewer). s.substr(0, truncateToCodepoints(s, n)) is an excerpt that never splits a character.

truncateToUtf16 ​

milo
pub fn truncateToUtf16(s: &string, maxUnits: i64): i64

Byte offset just past the longest prefix of s that is at most maxUnits UTF-16 code units. A character outside the BMP (two units) that would straddle the limit is excluded whole, never cut into half a surrogate pair.

truncateToWidth ​

milo
pub fn truncateToWidth(s: &string, maxCols: i64): i64

Byte offset just past the longest prefix of s that fits in maxCols columns. Never splits a codepoint, and never leaves half of a double-width glyph — a wide character that would straddle the limit is excluded entirely.

unicodeLower ​

milo
pub fn unicodeLower(s: &string): string

s with every code point lowercased by lowerCodepoint: "ÀÉÎ Straße ΣΑΣ" becomes "àéî straße σασ". Unicode-aware, unlike the ASCII-only s.toLower(), which is the right one for protocol tokens (header names, schemes, hex): there a Unicode mapping would let the Kelvin sign U+212A match "k". The byte length can change ("ȿ" is 2 bytes, its uppercase "Ȿ" 3). Malformed UTF-8 bytes are copied unchanged.

unicodeUpper ​

milo
pub fn unicodeUpper(s: &string): string

s with every code point uppercased by upperCodepoint: "àéî straße" becomes "ÀÉÎ STRAßE" (ß has no one-code-point uppercase). See unicodeLower for when to use this rather than the ASCII-only s.toUpper().

upperCodepoint ​

milo
pub fn upperCodepoint(cp: i32): i32

The simple uppercase mapping of code point cp (UnicodeData.txt field 12), or cp itself when it has none. ß stays ß: its uppercase "SS" is a full mapping, two code points, which a per-code-point map cannot produce.

utf16Length ​

milo
pub fn utf16Length(s: &string): i64

Length of s in UTF-16 code units: what JavaScript's .length reports, so a limit a browser enforces can be enforced the same way on the server. A malformed byte counts as one unit (it decodes as U+FFFD).

utf16UnitCount ​

milo
pub fn utf16UnitCount(cp: i32): i64