Milo

Appearance

std/fetch ​

The HTTP client and the TLS client socket. fetch resolves the host, speaks TLS for an https URL and follows redirects. It is split out of std/net so a plain-TCP program never links OpenSSL: importing std/fetch is what pulls it in.

milo
from "std/fetch" import { fetch, fetchWith, fetchWithTimeout, FetchOptions, FetchTimeouts, FetchResponse }

Example ​

milo
let resp = fetch("https://example.com/")!
if resp.ok() {
    print(resp.text())
}
match resp.header("Content-Type") {
    Some(ct) => print(ct),
    None => {},
}

let opts = FetchOptions { method: "POST", headers: "Content-Type: text/plain", body: "hi" }
let posted = fetchWith("https://example.com/echo", opts)!
print(posted.status)

Timeouts ​

fetch and fetchWith wait as long as the server takes, so one upstream that accepts the connection and then goes quiet hangs the caller. fetchWithTimeout bounds the connect (with its TLS handshake) and the whole request, redirects included, and answers NetError.Timeout(stage) instead. 0 leaves either limit off. It works the same on a green task and on a plain main. A DNS lookup counts toward the connect limit.

Response size ​

A fetch reads at most FetchOptions.maxBodyBytes response bytes (64 MiB by default; 0 for no limit) and answers NetError.TooLarge past that, never a body cut short. fetchAt takes its own maxBytes and truncates instead.

milo
let limits = FetchTimeouts { connectMs: 5000, totalMs: 20000 }
match fetchWithTimeout("https://example.com/feed.json", FetchOptions { method: "GET", headers: "", body: "" }, limits) {
    Ok(resp) => print(resp.status),
    Err(e) => print(e.message()),   // "timeout: response" when the server stalled
}

Fetching a URL someone else chose ​

A server that fetches user-supplied URLs (link previews, webhooks) has to refuse private addresses, and checking the address then calling fetch resolves the name twice: a DNS answer that changes in between sends the request past the check. fetchAt connects to an address you resolved and vetted yourself, keeps the URL's host for the Host header and the TLS certificate check, follows no redirect, and reads at most maxBytes. Vet and resolve each redirect hop the same way.

milo
let ip = resolve(parseHost(url))?
if !isPublic(ip) { return Err("refused") }   // your own range check
let resp = fetchAt(ip, url, FetchOptions { method: "GET", headers: "", body: "" },
                   FetchTimeouts { connectMs: 5000, totalMs: 10000 }, 262144)?

Connection reuse ​

Requests keep their connection alive (HTTP/1.1) and park it in a per-thread pool when the response is done, so the next request to the same scheme, host and port skips the TCP connect, the DNS lookup and, for https, the TLS handshake.

  • Key. Scheme, host and port; fetchAt adds its pinned address, so a vetted address is never swapped for a connection to another one.
  • Checkout. A request takes a connection out of the pool and has it to itself until the response ends, so concurrent green tasks fetching from one host each get their own. At most 6 idle connections per host are kept (32 in all, oldest dropped first); past that, a request opens a new connection rather than waiting for one, and closes it after.
  • Framing. A response is read by its Content-Length or chunked framing, never to EOF. One with neither, one carrying Connection: close, an HTTP/1.0 one, one cut short by maxBytes, or one followed by stray bytes, closes its connection instead of pooling it.
  • Idle. A connection idle for more than 30 s is not reused. Before reuse a one-byte nonblocking read checks it: EOF or any data means the server closed it (or broke framing), and it is dropped.
  • Stale retry. If a reused connection fails before any byte of the response arrives, the server closed it as the request went out. A GET, HEAD, PUT, DELETE or OPTIONS is sent again once on a fresh connection; any other method returns the error, since the server may have acted on it.
  • Opting out. A Connection: close header in FetchOptions.headers sends a one-shot request: a fresh connection, closed after the response.
  • Threads. The pool is per OS thread. A Promise.blocking worker keeps none: its thread ends after one call and nothing would close what it left behind.

Hosts are resolved with resolveAll, and each address is tried in turn (v6 and v4, in the system's preferred order), so http://localhost:8080/ reaches a server listening on only ::1 or only 127.0.0.1. A connect limit from fetchWithTimeout covers all the attempts together.

API reference ​

FetchOptions ​

milo
pub struct FetchOptions

HTTP request configuration: method, headers, and body.

headers is a CRLF-joined block ("Accept: text/plain\r\nX-Id: 7") with no trailing CRLF; leave it empty for none.

Fields:

  • method: string
  • headers: string
  • body: string: A Connection: close header here opts this request out of the connection pool (see the pool notes in std/fetch); without one, the connection is reused.
  • maxBodyBytes: i64 = 67108864: Most response body bytes a fetch reads before giving up with Err(NetError.TooLarge); 0 for no limit. Without a cap a server that never stops sending grows the caller's memory until it is killed. Chunked framing counts toward the limit. (fetchAt takes its own maxBytes, which truncates instead.)

FetchResponse ​

milo
pub struct FetchResponse

HTTP response with status code, headers, and body.

Not Response: Milo's module namespace is flat and merged, so that name collided with std/http's Response enum and no program could import both — i.e. could not serve HTTP and fetch over HTTPS at once. The client type moved because handlers spell std/http's Response.Text(...) constantly while this one is almost always used through inference.

Fields: status: i32, headers: string, body: string.

FetchResponse.header ​

milo
fn FetchResponse.header(self: &FetchResponse, name: &string): Option<string>

Value of response header name (case-insensitive), or None if the response did not carry it. A header sent with an empty value is Some("").

FetchResponse.json ​

milo
fn FetchResponse.json(self: &FetchResponse): Json

Parse the response body as JSON.

FetchResponse.ok ​

milo
fn FetchResponse.ok(self: &FetchResponse): bool

Return true if the status code is 2xx (success).

FetchResponse.text ​

milo
fn FetchResponse.text(self: &FetchResponse): string

Return the response body as a string.

FetchTimeouts ​

milo
pub struct FetchTimeouts

Limits on one fetch, in milliseconds; 0 means no limit. connectMs bounds each TCP connect together with its TLS handshake; totalMs bounds the whole call from the first connect to the last body byte, redirects included. A DNS lookup is bounded by the sooner of the two; the resolver thread itself cannot be cancelled and is left to finish on its own.

Separate from FetchOptions because it predates field defaults, when a new field broke every FetchOptions literal. Folding it in now (connectMs: i64 = 0 on FetchOptions) would still remove this type and fetchWithTimeout's third argument from every caller.

Fields: connectMs: i64, totalMs: i64.

FormField ​

milo
pub struct FormField

One key/value pair of a form body.

Fields: name: string, value: string.

TlsStream ​

milo
pub struct TlsStream

TLS/SSL connection handle. Frees SSL context and closes fd on drop.

TlsStream.connect ​

milo
fn TlsStream.connect(ip: u32, port: u16, hostname: &string): Result<TlsStream, NetError>

Open a TLS connection to an IPv4 address. hostname is used for SNI and certificate verification.

TlsStream.connectBy ​

milo
fn TlsStream.connectBy(ip: u32, port: u16, hostname: &string, connectBy: i64, overallBy: i64): Result<TlsStream, NetError>

connect() that gives up at connectBy (the TCP connect and the handshake) and at overallBy, both epoch ms with 0 for none. The stream it returns is nonblocking: use sendBy and recvBy on it.

TlsStream.connectWithCA ​

milo
fn TlsStream.connectWithCA(ip: u32, port: u16, hostname: &string, caFile: &string): Result<TlsStream, NetError>

connect() with one extra CA file trusted on top of the system store. The point is to keep verification ON while trusting a private CA: what users otherwise reach for is disabling verification, which throws the whole guarantee away. An empty caFile is exactly connect().

TlsStream.fromFd ​

milo
fn TlsStream.fromFd(fd: i32, hostname: &string, caFile: &string): Result<TlsStream, NetError>

fromTcp over a bare descriptor. OWNERSHIP: fd is taken over by the returned TlsStream, and closed on every error path here. @unsafe because the caller must vouch it owns fd outright and neither closes nor uses it again.

TlsStream.fromFdBy ​

milo
fn TlsStream.fromFdBy(fd: i32, hostname: &string, caFile: &string, by: i64): Result<TlsStream, NetError>

fromFd with the handshake bounded by by (epoch ms; 0 for none); -1 is fromFd's own unbounded, green-parking handshake.

TlsStream.fromTcp ​

milo
fn TlsStream.fromTcp(conn: TcpStream, hostname: &string, caFile: &string): Result<TlsStream, NetError>

Client handshake over an ALREADY-connected socket. This is the entry point a protocol with in-band TLS negotiation needs — Postgres' SSLRequest, SMTP STARTTLS: the socket must carry plaintext first, so TLS setup cannot own the connect() too. The stream is consumed: the returned TlsStream owns its fd, and every error path closes it, so nothing is left behind to use after the handshake.

TlsStream.incoming ​

milo
fn TlsStream.incoming(self: TlsStream): Channel<string>

Stream decrypted inbound bytes as an iterable channel, pumped on a green task — the uniform async-read API, TLS variant. Uses the SSL-aware read (parks on WANT_READ) rather than the raw-fd fdChannel. for chunk in tls.incoming(); the channel closes at EOF / on SSL error. Consumes the stream, so send any request first. The detached pump reads through the SSL state, so the pump has to own it: when incoming() only borrowed, dropping the stream ran SSL_read on a freed SSL. The connection closes when the pump ends (EOF, an SSL error, or the first chunk that arrives after every receiver is gone).

TlsStream.recv ​

milo
fn TlsStream.recv(self: &TlsStream): Result<string, NetError>

Read everything until the peer closes, as one string (blocks to EOF). Prefer incoming() for streaming/incremental consumption.

TlsStream.recvBy ​

milo
fn TlsStream.recvBy(self: &TlsStream, by: i64): Result<string, NetError>

recv() that gives up at by (epoch ms; 0 for none): Err(Timeout) rather than the part of the body that arrived in time.

TlsStream.recvOnce ​

milo
fn TlsStream.recvOnce(self: &TlsStream): string

A single SSL_read of whatever has arrived, unlike recv() which loops to EOF. A server must answer a request while the client still holds the connection open, so recv() would deadlock there; this returns after one record.

TlsStream.recvUpTo ​

milo
fn TlsStream.recvUpTo(self: &TlsStream, by: i64, maxBytes: i64): Result<string, NetError>

recvBy that stops once maxBytes bytes have arrived (0 reads to EOF), for a caller that must not buffer whatever a hostile server chooses to send. The result may run up to one TLS record past the cap.

TlsStream.send ​

milo
fn TlsStream.send(self: &TlsStream, data: &string): Result<i64, NetError>

Send data over the TLS connection. Returns the number of bytes sent.

TlsStream.sendBy ​

milo
fn TlsStream.sendBy(self: &TlsStream, data: &string, by: i64): Result<i64, NetError>

send() that gives up at by (epoch ms; 0 for none).

Functions ​

buildRequest ​

milo
pub fn buildRequest(host: &string, path: &string, opts: &FetchOptions): string

Serialize a request. HTTP does not require Accept or User-Agent, but real servers reject requests without them: Overpass 406s an Accept-less client, and several public APIs 403 an anonymous one. Both are only supplied when the caller has not set their own, so an explicit header always wins. Asks the server to close the connection after its response (Connection: close), so the response ends where the stream does.

fetch ​

milo
pub fn fetch(url: &string): Result<FetchResponse, NetError>

HTTP GET. Resolves the host, uses TLS for an https URL, and follows 301/302/307/308 redirects.

fetchAt ​

milo
pub fn fetchAt(ip: u32, url: &string, opts: FetchOptions, t: FetchTimeouts, maxBytes: i64): Result<FetchResponse, NetError>

One request for url, sent to the IPv4 address ip the caller already resolved (std/net resolve), with no redirect followed: a 3xx comes back as is, Location and all. The Host header and the TLS certificate name still come from url.

This is the primitive for fetching a URL someone else chose. Checking the host's address and then calling fetch resolves the name twice, and a DNS answer that changes in between (rebinding) sends the request somewhere the check never saw; here the address checked is the address connected to. Each redirect hop is the caller's to vet and resolve the same way.

At most maxBytes bytes of the response are read, headers included (0 for no limit), and the response is parsed from what arrived, so a cut-off body comes back short rather than as an error. t bounds the request as in fetchWithTimeout.

fetchDelete ​

milo
pub fn fetchDelete(url: &string): Result<FetchResponse, NetError>

HTTP DELETE.

fetchForm ​

milo
pub fn fetchForm(url: &string, body: &string): Result<FetchResponse, NetError>

POST an application/x-www-form-urlencoded body — build it with formEncode, or hand-roll it with urlEncode. Several APIs that nominally accept a raw body (Overpass, most OAuth token endpoints) only accept this content type.

fetchPatch ​

milo
pub fn fetchPatch(url: &string, body: &string): Result<FetchResponse, NetError>

HTTP PATCH of body with Content-Type: application/json.

fetchPost ​

milo
pub fn fetchPost(url: &string, body: &string): Result<FetchResponse, NetError>

HTTP POST of body with Content-Type: application/json.

fetchPut ​

milo
pub fn fetchPut(url: &string, body: &string): Result<FetchResponse, NetError>

HTTP PUT of body with Content-Type: application/json.

fetchWith ​

milo
pub fn fetchWith(url: &string, opts: FetchOptions): Result<FetchResponse, NetError>

An HTTP request with full control over method, headers and body. Redirects are followed as for fetch; a 301 or 302 turns the retry into a GET with no body.

fetchWithTimeout ​

milo
pub fn fetchWithTimeout(url: &string, opts: FetchOptions, t: FetchTimeouts): Result<FetchResponse, NetError>

fetchWith that gives up: Err(NetError.Timeout(stage)) once t.connectMs passes on a connect, or t.totalMs on the whole request, instead of waiting on a server that stopped answering. Nothing is returned from a response cut off by the deadline.

findHeader ​

milo
pub fn findHeader(headers: &string, name: &string): Option<string>

Value of name in a CRLF-joined header block, or None if it is not there. A header sent with no value (Accept:) is present: Some("").

formEncode ​

milo
pub fn formEncode(fields: &Vec<FormField>): string

Build an application/x-www-form-urlencoded body. Both halves of every pair are percent-encoded, so a value containing '&' or '=' can't split the body into extra fields.

hostHeader ​

milo
pub fn hostHeader(url: &string): string

The authority for the Host header: the host, plus :port whenever the port is not the scheme's default. RFC 7230 5.4 requires it, and leaving it off is not cosmetic — a server on a non-default port that authenticates the request (AWS SigV4 signs the Host header verbatim) rejects a request whose Host says one thing and whose connection says another.

isHttps ​

milo
pub fn isHttps(url: &string): bool

parseHost ​

milo
pub fn parseHost(url: &string): string

parsePath ​

milo
pub fn parsePath(url: &string): string

parsePort ​

milo
pub fn parsePort(url: &string): u16

parseResponse ​

milo
pub fn parseResponse(raw: string): FetchResponse

resolveLocation ​

milo
pub fn resolveLocation(base: &string, loc: &string): string

The URL a redirect's Location points at, resolved against the request URL. A server may answer with a path alone (weather.gov sends Location: /points/38.3396,-122.7011), and fetching that bare string parsed an empty host, so the redirect failed as an unreachable upstream.

urlEncode ​

milo
pub fn urlEncode(s: &string): string

Percent-encode one component of a URL query or an application/x-www-form-urlencoded body. Everything outside RFC 3986's unreserved set becomes %XX, space included: %20 decodes back to a space in both a query string and a form body, whereas + only does in the latter.