# How to generate a parser for a custom format in Go > Build a Go parser for a nested config format with pigeon, participle, and the ANTLR Go target, and catch a PEG rule that doubles its work per level of nesting. Source: https://voxgig.com/howto/generate-a-parser-in-go - Audience: platform-team - Level: intermediate - Languages: go - Verified: 2026-10-21 - Published: 2026-10-20 ## Short answer Write the grammar as a PEG file, let pigeon generate one Go source file from a `//go:generate` directive, and commit the output, which needs only the standard library. participle skips generation by reading the grammar from struct tags, and the ANTLR4 Go target needs Java and a runtime module. pigeon caches no results by default, so restructure a backtracking rule or pass `Memoize(true)`, and count the work with its `Statistics` option. --- ## You will need Go 1.24 or later, and Java only to regenerate the ANTLR parser. Verified 2026-10-21 against Go 1.24.7, pigeon 1.3.0, participle 2.1.4, the ANTLR 4.13.2 tool with its Go runtime 4.13.1, and OpenJDK 21.0.12.1. The sample parses one service config three ways, compares the results, and counts the work two pigeon grammars do on nested input. Reporting several errors from one parse is covered in [recover from syntax errors and report several in one parse](/howto/report-multiple-syntax-errors-in-one-parse). ## Approaches compared | Approach | When it fits | What it costs you | When to pick something else | | --- | --- | --- | --- | | [ANTLR4 Go target](https://github.com/antlr/antlr4/blob/dev/doc/go-target.md) | The same grammar must also produce a parser in Java, Python, TypeScript, or another of its targets | A Java runtime and the tool jar to regenerate, a runtime module, and generated code that `go vet` rejects | Only Go reads the format, so the Java step buys nothing | | [participle](https://pkg.go.dev/github.com/alecthomas/participle/v2) | A small format, owned by Go code, that changes every few weeks | A grammar scattered across struct tags, readable as a whole only through the EBNF it prints | Reviewers outside the team must read the grammar as a specification | | [pigeon](https://pkg.go.dev/github.com/mna/pigeon) | You want a grammar file with Go actions and a parser that needs only the standard library | No result cache unless you ask for one, so a single backtracking rule can make parse time grow exponentially | The grammar cannot be restructured and the input comes from strangers | pigeon and ANTLR both keep the grammar in a file a reviewer can read. pigeon's output imports only the standard library, while the ANTLR output needs its runtime module, plus a Java toolchain whenever the grammar changes. participle has no generation step, so changing the format means changing a struct, and the grammar exists only as tags until you print it. ## Write the grammar as a PEG file pigeon reads a PEG grammar whose actions are blocks of Go code and writes one Go source file, as its [package documentation](https://pkg.go.dev/github.com/mna/pigeon) describes. The actions build the shared model types, so the parser returns the value the program wants. ```text title="pegconfig/config.peg" Config <- _ es:(Entry _)* EOF { return entries(es), nil } Entry <- Block / Attr // A block may end with a semicolon. Writing that as `(_ ';')?` keeps the // parse linear; see ../pegnaive for the version that does not. Block <- name:Ident _ label:Label? _ '{' _ es:(Entry _)* '}' (_ ';')? { b := &model.Block{Name: name.(string), Entries: entries(es)} if label != nil { b.Label = label.(string) } return b, nil } Attr <- key:Ident _ '=' _ value:Value { return &model.Attr{Key: key.(string), Value: value}, nil } ``` A `//go:generate` directive records the exact command beside the grammar. The [go command](https://pkg.go.dev/cmd/go#hdr-Generate_Go_files_by_processing_source) never runs it during a build, so the generated file is committed and regenerated by hand.[^1] ```go title="pegconfig/generate.go" // The generated parser is committed. Regenerate it after editing config.peg. //go:generate go run github.com/mna/pigeon@v1.3.0 -o config.go config.peg ``` ```bash go generate ./pegconfig && wc -l pegconfig/config.go ``` ```text output 1869 pegconfig/config.go ``` Those 1869 lines import only the standard library and the model package, so pigeon adds nothing to `go.mod`. The `(_ ';')?` at the end of `Block` makes the trailing semicolon optional inside one rule, which matters more than it looks. ## Put the grammar in struct tags participle builds its parser from the types it fills. Each field's tag is a fragment of the grammar, and `MustBuild` assembles them when the program starts. Use the `parser:"..."` form of the tag, because the bare form is not a valid Go struct tag and `go vet` rejects it, a clash the [tag syntax notes](https://pkg.go.dev/github.com/alecthomas/participle/v2#readme-tag-syntax) warn about. Give an integer field a type of its own, as `decimal` does here, because participle fills an `int64` with `strconv.ParseInt` in base 0, which reads `010` as octal 8. ```go title="partconfig/parser.go" type config struct { Entries []*entry `parser:"@@*"` } type entry struct { Block *block `parser:" @@"` Attr *attr `parser:"| @@"` } type block struct { Name string `parser:"@Ident"` Label *string `parser:"@String?"` Entries []*entry `parser:"'{' @@* '}' ';'?"` } type attr struct { Key string `parser:"@Ident '='"` Value value `parser:"@@"` } type value struct { String *string `parser:" @String"` Int *decimal `parser:"| @Int"` Bool *string `parser:"| @('true' | 'false')"` } ``` ```bash go run ./cmd/ebnf ``` ```text output Config = Entry* . Entry = Block | Attr . Block = ? "{" Entry* "}" ";"? . Attr = "=" Value . Value = | | ("true" | "false") . ``` The EBNF that participle prints from the tags is the nearest this approach comes to a grammar file, and it reads like the PEG rules in the first step. Nothing is generated, so a change to the format is a change to a struct. ## Generate the parser with ANTLR ANTLR generates a lexer and a parser from a `.g4` grammar for any of its [ten target languages](https://github.com/antlr/antlr4/blob/dev/doc/targets.md), Go among them with `-Dlanguage=Go`. The Go output depends on the `github.com/antlr4-go/antlr/v4` runtime module. The tool itself is a Java program, and the Go target's guide suggests committing its jar to your repository. ```text title="antlrconfig/Config.g4" grammar Config; config : entry* EOF ; entry : block | attr ; block : name STRING? '{' entry* '}' ';'? ; attr : name '=' value ; // true and false lex as BOOL, so a name takes BOOL as well as IDENT. name : IDENT | BOOL ; value : STRING | INT | BOOL ; BOOL : 'true' | 'false' ; IDENT : [a-zA-Z_] [a-zA-Z0-9_]* ; STRING : '"' ~["]* '"' ; INT : '-'? [0-9]+ ; COMMENT : '#' ~[\n]* -> skip ; WS : [ \t\r\n]+ -> skip ; ``` The directive in `antlrconfig/build.go` runs `java -jar ../tools/antlr-4.13.2-complete.jar -Dlanguage=Go -package antlrconfig -no-listener Config.g4`, which ran by hand to produce the committed files. ```bash wc -l antlrconfig/config_lexer.go antlrconfig/config_parser.go && go vet ./antlrconfig ``` ```text output 143 antlrconfig/config_lexer.go 927 antlrconfig/config_parser.go 1070 total # example.com/configparse/antlrconfig antlrconfig/config_parser.go:270:2: unreachable code antlrconfig/config_parser.go:398:2: unreachable code antlrconfig/config_parser.go:612:2: unreachable code antlrconfig/config_parser.go:733:2: unreachable code antlrconfig/config_parser.go:827:2: unreachable code antlrconfig/config_parser.go:926:2: unreachable code ``` The tool wrote two files with 1070 lines, and `go vet` objects to six of them, one per rule. Each generated rule ends with `goto errorExit` after its `return`, a trick its own comment says prevents an unused-label error, and vet reports the line as unreachable. Keep the generated package out of your vet run, or accept the noise. ## Compare the three on one file `main.go` parses `service.conf` with all three, compares the results through one canonical printout, and then hands each parser two broken files. ```bash go run . ``` ```text output pigeon built the config below participle same antlr same service "api" { port = 8080 debug = false tls { cert = "/etc/api/cert.pem" key = "/etc/api/key.pem" } route "health" { path = "/healthz" timeout_ms = 250 } } "service {\n port 8080\n}\n" pigeon bad.conf:2:8 (17): no match found, expected: "#", "=", "\"", "{" or [ \t\r\n] participle bad.conf:2:3: unexpected token "port" (expected "}" ";"?) antlr 2:8: no viable alternative at input 'port8080' "service {\n port = 8080\n" pigeon bad.conf:3:1 (24): no match found, expected: "#", "}", [ \t\r\n] or [a-zA-Z_] participle bad.conf:3:1: unexpected token "" (expected "}" ";"?) antlr 3:1: extraneous input '' expecting {'}', BOOL, IDENT} ``` All three build the same config. Their errors differ in where they point: pigeon and ANTLR name the token after `port`, where the `=` belongs, while participle names `port`, the start of the entry it could not finish. ANTLR counts columns from zero, so `build.go` adds one before reporting. It also hands over a string token as written, quotes and backslashes included, so `build.go` reads it with `strconv.Unquote`, as pigeon's action does. The module file shows what each approach adds at run time. ```text title="go.mod" require ( github.com/alecthomas/participle/v2 v2.1.4 github.com/antlr4-go/antlr/v4 v4.13.1 ) require golang.org/x/exp v0.0.0-20240506185415-9bf2ced13842 // indirect ``` ## Check it worked `parse_test.go` holds the three parsers to one result on `service.conf` and on 19 inputs where their lexers could part ways, such as escapes, leading zeros, and `true` as a key. It also checks that each refuses both broken files, and pins the counts measured in the next section. ```bash go test -v ./... | grep -E '^(=== RUN|PASS$|FAIL$)' ``` ```text output === RUN TestAllThreeBuildTheSameConfig === RUN TestAllThreeAgreeOnTrickyInput === RUN TestEveryParserRefusesABrokenFile === RUN TestTheNaiveGrammarDoublesItsWorkPerLevel === RUN TestMemoizeAndTheRestructuredGrammarStayLinear === RUN TestMaxExpressionsStopsARunawayParse PASS ``` Six tests pass. `TestTheNaiveGrammarDoublesItsWorkPerLevel` fails if a pigeon release starts to cache results by default, which is a change worth hearing about. ## When it goes wrong The work doubles with every level of nesting. `pegnaive/config.peg` spells the optional semicolon as two alternatives, the way a format's prose description tends to list them. ```text title="pegnaive/config.peg" // A block may end with a semicolon, spelled here as two alternatives, the // way the format's description lists them. When the first fails at the end, // the second parses the whole block again, every nested block included. Block <- name:Ident _ label:Label? _ '{' _ es:(Entry _)* '}' _ ';' { return block(name, label, es), nil } / name:Ident _ label:Label? _ '{' _ es:(Entry _)* '}' { return block(name, label, es), nil } ``` pigeon caches no results unless asked, and its `Statistics` option counts the expressions a parse evaluates. A count, unlike a timing, comes out the same on every machine. ```bash go run ./cmd/memo ``` ```text output nesting naive naive+Memoize restructured 1 540 239 252 2 1248 352 365 4 5496 578 591 8 90456 1030 1043 12 1462104 1486 1499 16 23457624 1946 1959 naive, 16 levels, MaxExpressions(100000): nested.conf:19:1 (158): rule Block: max number of expressions parsed ``` Restructure the rule, as `pegconfig` does, and the count stays linear with no cache. Pass `Memoize(true)` when you cannot restructure, and benchmark it on ordinary files too: the generated documentation warns that it costs memory and time on typical input.[^2] For input from someone else, `MaxExpressions` turns a runaway parse into an error that names the rule. A broken file parses without an error. The default ANTLR error listener prints to stderr, recovers, and returns a tree, with no error value for the caller to check. ```bash go run ./cmd/partial 2>&1 ``` ```text output line 2:7 no viable alternative at input 'port8080' returned a tree with 1 entry: service{port8080} ``` Remove the default listener and collect the errors, as `build.go` does, before you trust any tree. ## When not to do this Do not write a grammar for a format that HCL, TOML, or YAML already covers. The config here is a near relative of HCL, whose [Go package](https://pkg.go.dev/github.com/hashicorp/hcl/v2) parses blocks, labels, and attributes with no grammar for you to maintain. Do not choose ANTLR for a format only Go reads. Its tooling and its other targets pay off when another language needs the same parser; without one, you carry Java, a jar, and a runtime module for nothing. Do not let a pigeon parser read untrusted input without `MaxExpressions`, even once its grammar is restructured. A rule that backtracks is a slow request waiting for the right file. Restructuring stops the backtracking but not the recursion: 200,000 levels of `a{`, a 600 KB file, end the restructured parser with a fatal stack overflow, which `recover` cannot catch. Do not hand the format to participle if people outside the team must review its grammar, since what they would review is struct tags. ## Related how-tos - [Remove left recursion from a PEG grammar](/howto/remove-left-recursion-from-a-peg-grammar) - [Recover from syntax errors and report several in one parse](/howto/report-multiple-syntax-errors-in-one-parse) - [Write a recursive descent parser by hand in TypeScript](/howto/recursive-descent-parser-in-typescript) - [Parse one config format identically in TypeScript and Go](/howto/parse-one-config-format-in-typescript-and-go) - [Write a Pratt parser for operator precedence in filters](/howto/pratt-parser-for-operator-precedence) ## Last verified Verified 2026-10-21 against Go 1.24.7, pigeon 1.3.0, participle 2.1.4, and the ANTLR Go runtime 4.13.1. Every output block is what the command preceding it printed. The ANTLR 4.13.2 tool ran by hand on OpenJDK 21.0.12.1 to generate `antlrconfig`, from a jar checked against the SHA-1 that Maven Central publishes; the checked commands use the committed output. No timing was measured. The counts are expression evaluations, so they repeat on any machine. The fatal stack overflow was run by hand, since it needs over a gigabyte of memory; the tests check only that `MaxExpressions` stops the same nesting. [^1]: `go generate` arrived with Go 1.4, and Rob Pike's [post introducing it](https://go.dev/blog/generate), dated December 22, 2014, warms up by generating a parser from a Yacc grammar. The [Go 1.8 release notes](https://go.dev/doc/go1.8#tool_yacc) later removed `go tool yacc`, which the Go compiler had stopped using in Go 1.7, and moved it to `golang.org/x/tools/cmd/goyacc`. The post's example now begins by installing it from there. [^2]: The cache that `Memoize` switches on is packrat parsing, which Bryan Ford presented at ICFP in Pittsburgh in October 2002, in [a paper whose title promises linear time](https://bford.info/pub/lang/packrat-icfp02/). He described it as a technique for implementing parsers in a lazy functional programming language, and the example parsers published beside the paper are written in Haskell. pigeon offers the same guarantee to Go behind one option.