lesson 02 — beginner

Token types and values

Understand the full TokenType enum, how token values are stored, and what memory ownership means in practice.

~10 min · lexer.h

A Token in first is a simple struct with two fields:

typedef struct {
  char *value;      // heap-allocated string
  TokenType type;   // one of the TT_* enum values
} Token;

The value is always a string, even for numbers. The lexer does no type conversion — that is the interpreter's responsibility.

Every token type

The full TokenType enum is defined in include/lexer.h. Each variant has a specific character pattern it matches:

TT_WHITESPACE — spaces and tabs (newlines are their own token, see below)
TT_NUMBER — digits, optionally followed by . and more digits, e.g. "42" or "3.14"
TT_STRING — content inside double quotes, e.g. "hello"
TT_IDENTIFIER — a name starting with a letter or _
TT_KEYWORD_TRUE / TT_KEYWORD_FALSE — the true / false literals
TT_ADDITION / TT_SUBTRACTION / TT_MULTIPLICATION / TT_DIVISION — + - * /
TT_MODULO — %
TT_EXPONENT — **
TT_AND / TT_OR — && / ||
TT_LEFT_BRACE / TT_RIGHT_BRACE — { / }, block delimiters
TT_LEFT_PAREN / TT_RIGHT_PAREN — ( / )
TT_EQUAL — =
TT_EQUAL_EQUAL / TT_NOT_EQUAL — == / !=
TT_LESSTHAN / TT_GREATERTHAN / TT_LESSTHAN_EQUAL / TT_GREATERTHAN_EQUAL — < > <= >=
TT_KEYWORD_LET / TT_KEYWORD_WHEN — the let / when keywords
TT_ARROW — ->, links a when arm's predicate to its consequent
TT_COMMA — separates let bindings and when arms
TT_NEWLINE — a statement terminator, emitted for \n (and \r\n, collapsed to one token)
TT_EOI — end of input sentinel, returned on EOF

exercise

Write a small C program that creates a lexer for the string "(10 + 3) * 2 >= 20" and prints every non-whitespace token's type and value until TT_EOI. Feed the string via a FILE* using fmemopen(). Notice how >= comes back as a single TT_GREATERTHAN_EQUAL token rather than two.

A note on TT_EOI

When fgetc() returns EOF, lex_next_token() creates a token of type TT_EOI. The parser and any other consumer should treat this as the termination signal and stop calling lex_next_token(). Calling it again after EOI is undefined behaviour — the FILE* is exhausted.