class TOML::Lexer

Overview

Streaming lexer for TOML v1.0 documents.

Trivia is emitted, not skipped. Whitespace, newlines and comments are returned as tokens because the AST keeps them attached to nodes for byte-identical round-trips.
Atoms are coarse-grained. Anything that is not a delimiter, string, or trivia is returned as a single BareKeyOrAtom token whose raw is the verbatim source slice. The parser decides whether it is a bare key, integer, float, datetime, boolean, inf, or nan based on context. Two consequences:
- The dot character is always emitted as Dot. A float like 1.5 lexes as Atom("1") Dot Atom("5"); the parser reassembles it. This keeps the lexer context-free at the cost of a tiny bit of parser work.
- The colon : is part of an atom (so a time value like 07:32:00 is a single Atom).
Brackets are always single-character. [ and ] are emitted one at a time even when adjacent. The parser disambiguates [[products]] (array-of-tables header) from [[1, 2]] (nested array) by position: a line that starts with two consecutive LBracket tokens is an AoT header, anywhere else they open two arrays.
Line endings. \n and \r\n are both valid newlines. A bare \r not followed by \n is not a TOML line ending and triggers a ParseError if it appears outside a string.

toml/lexer.cr

def self.new(source : String) #

[View source]

def eof? : Bool #

Returns true if the lexer has consumed all input.

[View source]

def next_token : Token #

Returns the next token. Once the end of input is reached, returns an EOF token at the current position; subsequent calls keep returning EOF.

[View source]