3. Lexical structure
Contents of ATD 0.1
An .atd file is UTF-8. A byte-order mark at its start is not part of any token and is kept; offsets count it. Lines end with LF or CRLF; a lone CR is whitespace. A position is reported as a 1-based line and a 1-based column counted in Unicode code points, as DISL §8.6 and FBL §2.6 require, and, where a range is given in bytes ($defs/Span, a fix's edits), as UTF-8 byte offsets into the file including the byte-order mark.
//starts a comment that runs to the end of the line./*starts a comment that runs to the first*/; comments do not nest.- A comment that starts with
///is a documentation comment. The documentation comments on the lines directly above a construct, with nothing but whitespace between them and it, are that construct's documentation: their text after///and one space, joined with line ends. A construct'sdocmember adds to it (section 7.3).
Comments and whitespace separate tokens and are otherwise ignored by the grammar; the formatter keeps each with the construct section 16.2 assigns it.
Every token is of one class, which the language servers' semantic tokens and the generated highlighting grammars (highlighting/) use (FR-052):
| Class | Terminals | Examples |
|---|---|---|
| keyword | every string terminal that spells a name (Appendix B) | element, label, create |
| name | NAME |
Topic, siblingName |
| key | KEY |
stroke:, color.surface: |
| assignment | ASSIGN |
count = |
| string | STRING |
"Mind map", r"^\s*#", """…""" |
| bytes | BYTES |
b"…" |
| number | INT, UINT, DOUBLE, VERSION |
24, 3u, 1.5, 0.1 |
| length | LENGTH |
12px, 50%, 1.5em |
| attribute reference | ATTR_REF |
@text, @address.city |
| token reference | TOKEN_REF |
$color.surface |
| parameter reference | PARAM_REF |
#radius |
| origin | ORIGIN |
freeplane/mindmap |
| capability range | CAPABILITY_RANGE |
@^1 |
| word | DASHED_WORD |
y-down, after-last, unreadable-requirement |
| operator and punctuation | every other string terminal | &&, ->, {, : |
| comment | COMMENT, BLOCK_COMMENT |
// …, /* … */ |
The terminals are defined in the grammar's modules (section 4.3); this section states what each means.
- NAME
[A-Za-z_][A-Za-z0-9_]*: DISL's SimpleId, so that every name is a CEL identifier. A dotted name (layout.twoSidedTree) is names joined by.tokens. - KEY: a name, or names joined by
., each of which may hold-, followed by spaces or tabs and:, as one token:stroke:,x-acme:,color.surface:. A setting's key is the token without its colon and the spaces before it. Taking the key and its colon as one token is what lets a setting stand among a construct's members, whatever the key spells (section 6.4). - ASSIGN: a name followed by spaces or tabs and
=, as one token: the start of an assignment to a local variable (section 12.2). - STRING: CEL's string literal, in all its forms: quoted with
"or', with CEL's escapes; tripled ("""…""",'''…''') across lines; or raw, prefixed withrorR, with no escapes. A string's value is CEL's. - BYTES: CEL's bytes literal, in CEL expressions only.
- INT, UINT, DOUBLE: CEL's numeric literals. A value position also admits a number with a leading
-(section 6.1). - LENGTH: a number in any of CEL's forms followed, with no space, by a unit of DISL §2.6:
px,mm,in,emor%. - ATTR_REF
@path, TOKEN_REF$name, PARAM_REF#name: references to an attribute, a theme token and a parameter (section 6.1). - ORIGIN
<vendor>/<type>: a tool type's origin, offered only where an origin is expected (section 5.1). - CAPABILITY_RANGE:
@followed by a semver range (@^1,@~1.2,@>=1.0), directly after a capability's name (section 13.2). - DASHED_WORD: a bare word with hyphens (
y-down,remove-when-empty): a literal (section 6.1), and, where a binding declares or names a binding, a rule, a block, a section or a file rule, a name (section 11.3). - VERSION
<major>.<minor>: the language version in the header (section 5.1).
The terminals of the grammar, and of every grammar a binding carries (section 11.9), are written in FBL §2.5's common subset of regular expressions, without what a token has no use for and what engines read differently:
- Admitted: literals, character classes
[…]and[^…], the escapes\d \D \s \S \w \W \t \n \r \\and escaped punctuation, groups(…)and non-capturing groups(?:…), alternation|, and the greedy quantifiers* + ? {n} {n,} {n,m}. - Not admitted: anchors
^and$; named groups(?<name>…); lazy and possessive quantifiers; backreferences, lookaround, atomic groups, inline flags and Unicode property classes; and.outside a character class, since engines differ on which line ends.matches (write the class:[^\n]). \d,\wand\sand their negations mean their ASCII classes:[0-9],[A-Za-z0-9_]and[ \t\n\r\f\v].- An expression matches the text the leftmost-first semantics of RE2 gives at the position, which the other engines agree with for this subset.
atd-grammar.py --lint refuses every terminal outside the subset.
The lexer is contextual, and its rule is the rule of Lark's contextual lexer (Lark 1.3.1), so that a host that reads a grammar with Lark or with a port of it reads the same tokens as every other implementation (Peter's ruling of 2026-11-01). At each position, the candidates are the terminals the parser's current state can shift, with the ignored ones (whitespace and comments); among them:
- A keyword is read by the pattern that spells it. A string terminal is a keyword of a pattern terminal among the candidates when the two have the same priority and the pattern's match of the string's text, at its start, is the whole text. The keyword is not tried on its own by rule 2, unless it has the flag
iand the pattern does not. - The first that matches wins. The candidates rule 1 leaves are tried in this order, and the first whose expression matches at the position gives the token, with the text that match takes: the higher priority first (a terminal that states none is of priority 0); then the greater width, the length of the longest text the terminal can match, a terminal that matches text of any length before every one that does not; then the longer expression, a string terminal's text or the text of the regular expression a pattern terminal compiles to; then the name, in code-point order.
- A pattern's token spelled as its keyword is the keyword. When the token is a pattern terminal's and its whole text is the text of one of that pattern's keywords, exactly or, for a keyword with
i, in any case of its ASCII letters, the token is that keyword's, the first such keyword in the order of rule 2.
This is not the longest match (informative): a terminal of a higher priority wins with a shorter match, and of two patterns of one priority the one rule 2 tries first wins even where the other would match more. A keyword that must yield to a longer word is written as a string at the priority of the pattern that reads the word, with i where the format reads keywords in any case (section 4.1): beside WORD, _GROUP: "group"i reads grouped as one WORD and Group as the keyword where the state can shift it, while a pattern at a priority above the word's, _GROUP.2: /[Gg][Rr][Oo][Uu][Pp]/, takes the first five letters of grouped and leaves ed. The grammars of definitions/diagrams/structurizr.lark and wardley-map.lark write their keywords so.
Keywords are soft (FR-008). A keyword is a string terminal that spells a name. No terminal of ATD 0.1's own grammar states a priority, so every keyword is a keyword of NAME by rule 1: a word is read as a NAME, and is the keyword by rule 3 only where it is spelled as one and the state can shift the keyword; elsewhere the same word is a NAME. Where the grammar expects a name, it expects the rule name, which admits NAME and every keyword (Appendix B), so a keyword can always stand as a name. A setting's key and an assignment's target are tokens of their own (KEY, ASSIGN), so any word, keywords included, can be a key or a local variable. The names of a binding and of its rules, blocks, sections and file rules are FBL's (FBL §2.2), which admit - and .: where a binding declares or names one, the grammar expects the rule rule_name, a name or a DASHED_WORD, or several joined by . (unreadable-requirement, by-id.v2). atd-grammar.py --probe puts every keyword in the place of every name of the examples and checks that the parser reads it as that name.
CEL's reserved words (as, break, const, continue, else, false, for, function, if, import, in, let, loop, namespace, null, package, return, true, var, void, while) stay reserved inside CEL, as CEL requires: the rule word, which CEL's identifiers use, lists every keyword but those, and the checker refuses one of them used as an identifier in an expression (ATD4003). Outside CEL they are names like any other keyword; true, false and null are not keywords of ATD at all, but bare words, which section 6.6 reads as JSON's literals.
A later version that adds a keyword does not change how an existing file is read: a file is read with the grammar of the version it declares (section 18).