This document provides a deep dive into the OwnLang language processing pipeline from source code to execution. It covers the complete transformation chain that converts textual source code into executable programs, including lexical analysis, parsing, AST generation, static analysis, and optimization stages.
For information about the runtime value system and function execution, see Function System and Object Model. For details about the module system architecture, see Module System.
The OwnLang language core implements a multi-stage processing pipeline that transforms source code through several distinct phases before execution.
Sources: ownlang-parser/src/test/java/com/annimon/ownlang/parser/ProgramsTest.java37-48 ownlang-parser/src/main/java/com/annimon/ownlang/parser/ast/IncludeStatement.java44-59
The processing pipeline consists of several key stages, each implemented as a Stage<Input, Output> interface:
| Stage | Input Type | Output Type | Primary Responsibility |
|---|---|---|---|
SourceLoaderStage | InputSource | String | Load source code from files/resources |
LexerStage | String | List<Token> | Tokenize source into lexemes |
ParserStage | List<Token> | Node | Generate Abstract Syntax Tree |
LinterStage | Node | Node | Perform static analysis and validation |
OptimizationStage | Node | Node | Apply AST transformations and optimizations |
ExecutionStage | Node | Value | Execute the program |
Sources: ownlang-core/src/main/java/com/annimon/ownlang/util/input/SourceLoaderStage.java11-29 ownlang-parser/src/main/java/com/annimon/ownlang/stages/ParserStage.java11-29
The language core supports multiple input sources through the InputSource interface, enabling loading from files, resources, and other sources.
The InputSourceDetector handles path resolution and determines whether to load from filesystem or classpath resources based on the resource: prefix.
Sources: ownlang-core/src/main/java/com/annimon/ownlang/util/input/InputSource.java5-16 ownlang-core/src/main/java/com/annimon/ownlang/util/input/InputSourceDetector.java6-39
The SourceLoaderStage performs initial source processing and metadata extraction:
StagesData for downstream stagesSources: ownlang-core/src/main/java/com/annimon/ownlang/util/input/SourceLoaderStage.java16-29
The Lexer class implements a character-by-character scanner that converts source text into a sequence of tokens. The lexer recognizes several token categories:
The lexer maintains static mappings for operators and keywords to enable efficient recognition:
TokenType values (e.g., "==" → TokenType.EQEQ)"def" → TokenType.DEF)Sources: ownlang-parser/src/main/java/com/annimon/ownlang/parser/Lexer.java21-111 ownlang-parser/src/main/java/com/annimon/ownlang/parser/Lexer.java157-245
The lexer maintains precise position information for each token, tracking row and column numbers. This enables accurate error reporting and source location mapping:
Sources: ownlang-parser/src/main/java/com/annimon/ownlang/parser/Lexer.java405-447 ownlang-parser/src/test/java/com/annimon/ownlang/parser/LexerPositionsTest.java12-63
The Parser class implements a recursive descent parser that converts the token stream into an Abstract Syntax Tree. The parser uses operator precedence and left-to-right associativity rules.
The parser implements operator precedence through the method call hierarchy, with higher precedence operators parsed deeper in the call stack:
=, +=, -=, etc.?: conditional operator||&&|^&==, !=<, <=, >, >=<<, >>, >>>+, -, ::*, /, %, **+, -, !, ~, ++, --Sources: ownlang-parser/src/main/java/com/annimon/ownlang/parser/Parser.java508-797
The parser implements error recovery to continue parsing after encountering syntax errors:
The recovery mechanism advances through tokens until it finds a position where parsing can continue successfully.
Sources: ownlang-parser/src/main/java/com/annimon/ownlang/parser/Parser.java105-118 ownlang-parser/src/main/java/com/annimon/ownlang/parser/Parser.java87-102
The LinterStage provides configurable static analysis with multiple validation modes:
Each validator implements the Visitor pattern to traverse the AST and check for specific issues:
Sources: ownlang-parser/src/main/java/com/annimon/ownlang/parser/linters/LinterStage.java15-65 ownlang-parser/src/main/java/com/annimon/ownlang/parser/linters/IncludeSourceValidator.java11-34
The OptimizationVisitor applies transformation passes to optimize the AST before execution:
The optimization visitor implements a bottom-up traversal strategy, processing child nodes before their parents. This ensures that optimizations cascade upward through the expression tree.
Key optimization techniques include:
Sources: ownlang-parser/src/main/java/com/annimon/ownlang/parser/optimization/OptimizationVisitor.java10-479 ownlang-parser/src/main/java/com/annimon/ownlang/parser/optimization/OptimizationVisitor.java40-46
The IncludeStatement enables runtime file inclusion with support for both filesystem and classpath resources:
The include system creates a complete processing pipeline for each included file, enabling recursive includes and maintaining proper scope isolation.
Sources: ownlang-parser/src/main/java/com/annimon/ownlang/parser/ast/IncludeStatement.java25-60 ownlang-core/src/main/java/com/annimon/ownlang/util/input/InputSourceDetector.java24-32
Refresh this wiki