Designing a streaming lexer that never loads the file

The first decision I made about the first lexer was that it would never load the source file into memory. Every token would be produced by reading from a FILE* directly, one character at a time.

This is a slightly unusual choice for a toy interpreter. Most examples slurp the entire source first. So why stream?

Why stream instead of buffer?

Two reasons. First, I wanted the lexer to work on large files and interactive stdin sessions without modification. A buffer-first design needs a complete input before starting. A streaming lexer can tokenize a pipe one character at a time, which is the right architecture for a REPL.

Second, it is more interesting to build. A streaming tokenizer forces you to think carefully about state. You cannot go back three characters — you can only push back exactly one character with ungetc(), and only immediately after a read.

The one-character pushback rule

The C standard guarantees that ungetc() can push back exactly one character. This constraint shapes everything. When lex_next_token() reads a digit, it needs to hand off to get_while(self, isdigit) — but that function expects to start from the first digit. So we push the character back:

if (isdigit(c)) {
  ungetc(c, self->file);
  char *str = get_while(self, isdigit);
  *token = create_stoken(str, TT_NUMBER);
  free(str);
}

get_while() reads one character past the end of the match, then pushes that extra character back. The FILE* cursor is always positioned at the start of the next token when any helper returns.

get_while vs get_until

Numbers and whitespace use get_while: read as long as the predicate holds, stop and push back the non-matching character. String literals use get_until: read until the predicate holds, and include the terminal character in the result. For strings, the closing quote is included in the raw buffer, then trimmed in-place — one allocation, no extra copy.

What one-character pushback costs you

When I first wrote this, first had no multi-character operators — every symbol was a single character, so the pushback limit was theoretical. It stopped being theoretical once ==, !=, <=, >=, &&, ||, and ** landed. The pattern turned out exactly as predicted: read the first character, fgetc() to peek at the second, and either consume both or ungetc() the peek and fall back to the one-character token.

else if (c == '=') {
  int32_t n = fgetc(self->file);
  if (n == '=') {
    *token = create_stoken("==", TT_EQUAL_EQUAL);
  } else {
    ungetc(n, self->file);
    *token = create_ctoken(c, TT_EQUAL);
  }
}

Every two-character operator in the lexer follows this exact shape — one peek, one conditional pushback. It never needed more than the one guaranteed ungetc() slot, because no first operator is more than two characters and the second character always disambiguates fully. The one thing this design still can't do gracefully: & or ! on their own, with no following partner, currently fall through to a hard exit(1) rather than a recoverable parse error — there's no standalone bitwise-AND or logical-NOT token to fall back to.