enum TOML::TokenKind

Overview

Token kinds emitted by the lexer.

The lexer is intentionally fine-grained for structural elements (so the parser does not need to look at characters), and coarse-grained for value atoms (Number, BareKeyOrAtom) — those are disambiguated by the parser based on context (a 1979 after = in a key/value pair could be an integer, a float, a date, or part of a datetime; only the parser can decide).

Trivia tokens (Whitespace, Newline, Comment) are emitted rather than skipped, because the AST keeps them attached to their nodes for byte-identical round-trips.

Defined in:

toml/token.cr

Enum Members

Whitespace = 0

Trivia.

Newline = 1

Comment = 2

Equal = 3

Structural punctuation.

Dot = 4

Comma = 5

LBracket = 6

RBracket = 7

LBrace = 8

RBrace = 9

BasicString = 10

Strings (delimiters included in the raw text).

MultilineBasicString = 11

LiteralString = 12

MultilineLiteralString = 13

BareKeyOrAtom = 14

An atom that can be a bare key (a-z A-Z 0-9 _ -) OR a value atom (integer, float, boolean, datetime, inf, nan, true, false). The parser disambiguates by context. We keep the raw text byte-for-byte so the serializer can re-emit it unchanged.

EOF = 15

End of input.

Instance Method Summary

• #bare_key_or_atom?

  Returns true if this enum value equals BareKeyOrAtom

• #basic_string?

  Returns true if this enum value equals BasicString

• #comma?

  Returns true if this enum value equals Comma

• #comment?

  Returns true if this enum value equals Comment

• #dot?

  Returns true if this enum value equals Dot

• #eof?

  Returns true if this enum value equals EOF

• #equal?

  Returns true if this enum value equals Equal

• #l_brace?

  Returns true if this enum value equals LBrace

• #l_bracket?

  Returns true if this enum value equals LBracket

• #literal_string?

  Returns true if this enum value equals LiteralString

• #multiline_basic_string?

  Returns true if this enum value equals MultilineBasicString

• #multiline_literal_string?

  Returns true if this enum value equals MultilineLiteralString

• #newline?

  Returns true if this enum value equals Newline

• #r_brace?

  Returns true if this enum value equals RBrace

• #r_bracket?

  Returns true if this enum value equals RBracket

• #whitespace?

  Returns true if this enum value equals Whitespace

Instance Method Detail