# Creating DSL With Antlr4 and Scala

Domain specific languages, when done right, helps a lot in improving developer productivity. First thing which you need while creating a DSL is a parser which can takes a piece of text and transforms it in structured format(like Abstract Syntax Tree) so that your program can understand and do something useful with it. DSL tends to stay for years so while choosing a tool for creating parser for you DSL you need to make sure that its easy to maintain and evolve the language. For parsing simple DSL, you can just use regular expression or scala’s in-built parser-combinators, but for even slightly complex DSL, both of these becomes performance and mantainenance nightmares.

In this post we will see how to use antlr to create a basic grammar and use it in Scala. Full code and grammar for this post is avaliable at https://github.com/saumitras/antlr4-scala/

## ANTLR4

ANTLR can generate lexers, parsers, tree parsers, and combined lexer-parsers. Parsers can automatically generate abstract syntax trees which can be further processed with tree parsers. ANTLR provides a single consistent notation for specifying lexers, parsers, and tree parsers. This is in contrast with other parser/lexer generators and adds greatly to the tool’s ease of use. It supports:

1. Tree construction
2. Tree walking
3. Error recovery
4. Error handling
5. Translation

Antlr supports a large number of target languages, so same grammar can be used for both backend parsing or frontend validations. Following langauges are supported:

## How ANTLR works

On a high level, here’s what you do to parse something with ANTLR

1. Create lexer rules
2. Create parser rules which uses lexer ouput
3. Use lexer and parser to generate source code for a target language
4. Use generated sources to convert some raw input into structured form (AST)
5. Do something with this structured data

We will understand it with an example. Lets say we want to create a DSL for allowing arithmetic operation. A valid input(expression) will be

As humans, if we want to evaluate this expression, here’s what we will do:

• Split this expression into different components.
• For example in above example, each character belongs to one of these group
1. Operands (3, 4, 5)
2. Operation (+ - * /)
3. Whitespaces
• This part is called lexical anaysis where you convert raw text(stream of characters) into tokens
• Create relationship between tokens
• To evaluate it efficiently we can create a tree like structure to define relationship between different expression like this:
• This is called AST (Abstract syntax tree) and this gets by applying rules you define in your grammar on input text. Once you have the AST, to evaluate the expression, we need to traverse or ‘walk’ it in a depth first manner. We start at the root ‘+’ and go as deep into the tree as we can along each child, then evaluate the operations as we come back out of the tree.

We will now setup the tools and try creating a simple grammar.

## Setup

#### IDE

ANTLR provides a GUI based IDE for developing grammar. You can download it from http://www.antlr3.org/works/. It combines an excellent grammar-aware editor with an interpreter for rapid prototyping and a language-agnostic debugger for isolating grammar errors.

#### IntelliJ plugin

Intellij provides a plugin too for ANTLR. Refer this link for adding the plugin https://plugins.jetbrains.com/plugin/7358-antlr-v4-grammar-plugin

#### Command line setup

You can directly create, test and debug grammar from command line too. Here’s steps on how

2. Create an alias for command to generate sources
• alias antlr4='java -jar /home/sam/softwares/antlr/antlr-4.6-complete.jar'
3. Create an alias to test your grammar for some input
• alias grun='java -cp ".:/home/sam/softwares/antlr/antlr-4.6-complete.jar" org.antlr.v4.gui.TestRig'

Add this in ~/.bashrc to be able to directly call antlr4 and grun command from anywhere.

## Creating grammar

A grammar will consist of 2 parts

• Lexer
• Parser

Both of these can be defined in same file, but for maintainence sake its better to define it in separate files. Lets create lexer and parser for a DSL which will allow basic arithmetic operations on 2 numbers. Some valid inputs will be:

Lets first define lexer definitions in a file named ArithmeticLexer.g4 to extract tokens from input:

• First definition for WS is telling to skip all the space, tabs and newline chars
• Second definition for NUMBER is telling to extract all numbers as NUMBER token
• ADD/SUB/MUL/DIV defintion is assigning a named token to respective mathematical operator

Now lets write some parser rules in file named ArithmeticParser.g4 which will processes tokens generated by lexer and create a AST for a valid input

• expr is the base rule and will accept any 2 numbers with one of valid operation.
• operation rule is telling tokens are valid operations

## Generating sources for a target language

Now that we have our grammar, we can generate lexer and parser source in any of the supported language. Run following from command line:

By default it will generate sources in java. You can change that by passing language argument. For example, following command generate sources in javascript:

Instead of generating sources individually for lexer and parsr, you can do in same command too:

After you run above, you will see following java source generated in same directory:

You can also provide a package name for generated sources, like below

Antlr4 provide 2 ways to walk the AST - Listener and Vistor. Antlr doesn’t generate sources for visitor by default. Since we will be using visitor pattern while using it in scala to avoid mutability, so lets generate visitor source too. It can be done by providing visitor flag, like below:

Now you will see source for visitor too

Next compile the sources

Now you are ready to test any input against you DSL.

## Testing the DSL

To do that, run following command

What above command is saying that execute org.antlr.v4.gui.TestRig on Arithmetic grammar and test for rule named expr. -tokens flag will allow us to see the tokens generated by lexer.

Enter any valid input like 10 + 3. Press enter. Press Ctrl+D. You will see input like below:

Since it didn’t show any error, it means your input in valid. Each line is showing token value, token name and its start and end offset.

In case of invalid input, antlr will tell you what was it was expecting, like below

In case of valid input, in addition to tokens, you can also see the AST by passing -gui flag, like below

## Using generated sources in code

We will now see how to extend generated interfaces and use it from within code. As I mentioned above, antlr4 provides 2 ways to walk the AST - Visitor and Listener. We will first see how to use the listener pattern. Although listener method is commonly used by java devs, but scala folks will not like it because it can only return unit, hence you need to use intermediate variables leading to side-effects. Refer to this post for a comparision between two patterns.

First thing you need to do is add antlr dependency:

Next import all the generated source in your project and create a parse method which accept an input expression

• In line 4, we converted input text to a char stream, because lexer operates at char level
• in line 5, we get a lexer object which uses ArithmeticLexer generated using definitions from ‘ArithmeticLexer.g4’ and pass input stream to it
• in line 6, we got all the token obtained by applying lexer rules on input text
• in line 7, we created a parser by applying rules which we defined in ArithmeticParser.g4

Next thing which we need to do is implement some methods in the BaseListener interface. Lets see what content of generated ArithmeticParserBaseListener.

For every rule which we defined in ArithmeticParser.g4, it created a enter and exit method. Since we had 2 rules, expr and operation, so it created 4 methods. As name implies, these will get triggered every time walker enters and exit a matched rule. For now lets focus on entry method of our starting rule expr. This problem can be solved by using visitor instead of listener as discussed in this post.

Notice that every rule has a context which has all the meta information as well as matched input info. Also note that all methods return void which means you need to use mutable variables to store computational values if they needs to be shared among different rules or even by main caller.

So now we create our own class by extending ArithmeticParserBaseListener and implement enterExpr rule.

• In line 4, exprText will have tokenized text for this rule.
• In line 7, expr rule’s context knows about NUMBER and operation. Since NUMBER occurs twice in the rule, hence ctx.NUMBER() will be a list containing 2 numbers.
• In line 11, we get value of operation from expr rule’s context
• We calculate the value and since there is no way to return it to caller from enterExpr method, hence we just print it. We could have stored it in some mutable variable in case caller needed it.

Now that we have implemented this, we need to use it in the parse method which we defined earlier, like below

Lets test it now on some input expressions:

You will see following output:

I hope this post gave you an idea on how to get started with creating your own DSL.