How the lexer works

A lexer (also called a scanner or tokenizer) is the first stage of any interpreter. Its job is simple: take a stream of characters and produce a stream of tokens — meaningful units like numbers, operators, and keywords.

In first, the lexer is implemented in lexer.c. The key design decision is that it operates on a FILE* and calls fgetc() one character at a time. It never loads the whole file.

The main function: lex_next_token

Every call to lex_next_token() returns exactly one token. It reads the first character and uses a switch for the small set of single-character tokens that need no lookahead (%, ,, {, }, (, ), EOF), falling through to a default case that's an if/else chain for everything that might need to peek at a second character:

int32_t lex_next_token(Lexer *self, Token *token) {
  int32_t c = fgetc(self->file);
  switch (c) {
  case ',':
    *token = create_ctoken(c, TT_COMMA); break;
  default:
    if (isdigit(c)) {
      ungetc(c, self->file);        // put it back
      char *str = get_while(self, isdigit);
      *token = create_stoken(str, TT_NUMBER);
      free(str);
    }
    // ... plus a branch per compound operator, string, identifier, etc.
  }
  return 0;
}

Notice ungetc(): when we peek ahead and find a digit, we push the character back into the stream before calling get_while(). Every helper function always reads from a consistent starting position.

get_while and get_until

Two helpers handle multi-character tokens:

• get_while(self, predicate) — reads characters as long as predicate(c) is true. Used for numbers and whitespace.
• get_until(self, predicate) — reads characters until predicate(c) is true. Used for string literals.

Memory ownership

Every token's value field is a heap-allocated char * created by either char_to_string() or strdup(). The caller owns this memory and must call free(token.value) after use.