tokenizer
also: lexer, lexical analyzer, scanner
A software component that breaks input text into discrete units called tokens, typically used in parsing, lexical analysis, and command-line argument processing.
A tokenizer is a program or function that reads a stream of input characters and segments them into meaningful units called tokens. Tokens are the smallest meaningful elements—like words, operators, or symbols—that a parser can understand and process.
In Linux, tokenizers are commonly used in shell scripting and command-line processing. For example, when you type ls -la /home/user, the shell's tokenizer breaks this into three tokens: ls, -la, and /home/user. Each token is then processed separately by the shell.
Tokenizers also appear in compilers, text processors, and configuration file parsers. A tokenizer typically ignores whitespace and comments, recognizing patterns like quoted strings, numbers, and identifiers to produce a sequence of tokens that downstream components can analyze.