tokenizer.rs in Rust

A zero-copy tokenizer: every token is a slice into the original source.

#[derive(Debug, PartialEq)]
enum Token<'a> {
    Word(&'a str),
    Number(&'a str),
    Symbol(char),
}

struct Lexer<'a> {
    source: &'a str,
    pos: usize,
}

impl<'a> Lexer<'a> {
    fn new(source: &'a str) -> Self {
        Lexer { source, pos: 0 }
    }

    fn rest(&self) -> &'a str {
        &self.source[self.pos..]
    }

    fn take_while<F: Fn(char) -> bool>(&mut self, test: F) -> &'a str {
        let start = self.pos;
        for c in self.rest().chars() {
            if !test(c) {
                break;
            }
            self.pos += c.len_utf8();
        }
        &self.source[start..self.pos]
    }
}

impl<'a> Iterator for Lexer<'a> {
    type Item = Token<'a>;

    fn next(&mut self) -> Option<Token<'a>> {
        self.take_while(char::is_whitespace);
        let c = self.rest().chars().next()?;

        if c.is_ascii_digit() {
            return Some(Token::Number(
                self.take_while(|c| c.is_ascii_digit()),
            ));
        }
        if c.is_alphabetic() || c == '_' {
            return Some(Token::Word(
                self.take_while(|c| c.is_alphanumeric() || c == '_'),
            ));
        }
        self.pos += c.len_utf8();
        Some(Token::Symbol(c))
    }
}

fn main() {
    let source = "let total = count * 42;";
    let tokens: Vec<Token> = Lexer::new(source).collect();

    for token in &tokens {
        println!("{:?}", token);
    }

    let words = tokens
        .iter()
        .filter(|t| matches!(t, Token::Word(_)))
        .count();
    println!("{words} words, {} tokens", tokens.len());
    assert_eq!(tokens[0], Token::Word("let"));
}

How it works

1. The Token enum borrows from the input, so nothing is copied.
2. The lexer walks bytes and slices at the boundaries it finds.
3. The lifetime on the struct ties every token to that source.

Keywords and builtins used here

• F
• Fn
• Item
• Iterator
• Lexer
• Option
• Self
• Some
• Token
• Vec
• bool
• break
• char
• enum
• fn
• for
• if
• impl
• in
• let
• main
• mut
• new
• next
• rest
• return
• self
• str
• struct
• take_while
• type
• usize

The run, in numbers

Lines

70

Characters to type

1272

Tokens

462

Three-star pace

115 tpm

At the three-star pace of 115 tokens a minute, this run takes about 241 seconds.

Type this snippet

Step 1 of 1 in Encore, step 11 of 11 in Lifetimes & interior mutability.

← Previous