4. Format families and their lossless readings
Contents of FBL 0.8
For each family, this section defines a lossless reading: a tree of nodes over the body's bytes in which every byte belongs to exactly one node or to the trivia (whitespace, comments, line endings) that one node owns. Rules (section 5) select nodes; splices (section 6) replace the spans of nodes. A host MAY parse a body however it likes, but the spans it uses MUST be the ones defined here.
An entry is a node that a rule can match: a mapping member or sequence item (yaml, json), an element (xml), a statement (lines, blocks, bullets), a list item (bullets). Every entry has:
- its own span: the bytes of the entry itself, from its first to its last significant byte;
- its line span, when the entry starts its line and ends its line (nothing but trivia before it on its first line and after it on its last): from the first byte of its first line to the end of the line ending of its last line, extended upwards over its leading comments, the comment lines directly above it at the same indentation with no blank line in between. A trailing comment on the entry's last line is part of its line span.
A value is a node that a slot binds: a scalar, a flow collection, an attribute value, the text of an element, a named group. Its span excludes quotes and delimiters for the purpose of reading, and includes them for the purpose of replace-value (section 6.1).
Blank lines belong to no entry: they are trivia of the enclosing container and are never removed or added by a splice unless this document says so.
Everything no rule binds is unbound: unknown keys, attributes, elements and statements, comments, directives. Unbound content is read past, kept byte for byte, and moves only when a splice removes an entry that contains it (an unknown key inside a removed entry goes with the entry).
Entries have document order, the order of their first bytes. Reading order is document order (DISL relies on it for positionIn); for a folder subject, files are read in the ordinal order of their relative paths, and document order within each.
An entry is presented to CEL (section 2.4) as follows. A yaml or json mapping is a map from key to value; a sequence is a list; a scalar is a string, int, double, bool or null by the YAML 1.2 core schema (JSON by its own types). An xml element is a map from attribute name to string, plus the key text holding its text content. A lines, blocks or bullets statement, and a bullets item, is presented as its groups.
A key that is present with no value is present in the map with the value null, so has(entry.end) is true for end: and false where the key is absent.
Every scalar of a yaml or json body has, beside the typed value of section 4.1.4, a written text: what its author wrote.
| Scalar | Written text |
|---|---|
| yaml plain scalar | Its characters, without leading and trailing spaces and without a comment; a plain scalar over several lines is folded as YAML folds it. |
| yaml quoted scalar (single or double) | Its content, without the quotes, with its escapes resolved. |
yaml block scalar (|, >) |
Its content as YAML defines it for its indicators. |
yaml key with no value (end:) |
The empty text. |
| json string | Its content, with its escapes resolved. |
json number, true, false, null |
Its literal characters. |
A tag does not change the written text. A scalar reached through an alias or a merge key has the written text of the scalar it stands for. The members of a flow collection have a written text each, by the same rules. The xml, lines, blocks and bullets families read text and nothing else, so there the value a slot reads is already what was written.
So yes, "yes" and 'yes' have the written text yes; 007 has 007; 1.0 has 1.0; 0x10 has 0x10. The typed value of each is what the YAML 1.2 core schema gives: a bool, a string, a string, the int 7, the double 1.0, the int 16.
Tree rules (yaml, json) and xml rules select entries with a selector: segments separated by /.
| Segment | Matches |
|---|---|
| a key or name | the member with that key (yaml, json) or the child element with that name (xml) |
* |
any member of a mapping or any item of a sequence (yaml, json), any child element (xml) |
** |
any number of levels, including none |
{name} |
any member of a mapping, whose key is bound to path.name |
name[@A='v'] |
xml: a child element with that name whose attribute A has the value v |
A selector that starts with / is absolute, from the document's root (the root mapping or sequence; the document element's parent in xml, so /map is the root element map). A selector without it is relative to the entry the rule is written for (in insert.container, child), or to the root elsewhere. / alone selects the root. A key containing /, *, { or } cannot be selected in FBL 0.1.
The body is a YAML 1.2 stream. Only the first document is bound; any further document is unbound. The lossless reading follows the YAML 1.2 syntax, with these definitions:
- A block mapping entry is a key and its value; its own span runs from the first byte of the key to the last byte of the value (for a block collection, its last entry's last byte). A block sequence entry is
-and its value; its own span runs from the-. - The indentation of an entry is the column of its first byte: the key, or the
-. For a mapping that is a sequence item (- id: a), the mapping's entries are indented at the column of their keys (idat 4 in- id: a). - A scalar value's span is the scalar as written: with its quotes for quoted scalars; for a plain scalar, without trailing spaces or a comment; for a block scalar (
|,>with indicators), from the indicator to the end of its last content line, without that line's ending. - A flow collection (
[…],{…}) is one value. Its members are readable (CEL sees them), but its only writable span is the whole collection, written byreplace-valuein the style the attribute binding asks for (flowwrites a one-line flow sequence). - Anchors, aliases, tags and merge keys are read (an alias reads as the anchored value; a tag is ignored for typing when it is a core-schema tag and kept otherwise;
<<merges as YAML 1.1 defines it, for reading only). A slot reached through an alias or a merge is read-only. - A YAML syntax error makes the body unreadable (section 7.5). A document whose structure is well-formed YAML but not what a rule expects (a sequence item that is not a mapping, a scalar where a mapping is needed) yields unreadable entries (section 7.4), never an unreadable body.
The body is a JSON text (RFC 8259). Comments are not allowed: a body with a comment is unreadable. The nodes are objects, arrays, members, items and values, with these spans:
- A member's own span runs from the opening quote of its name to the last byte of its value; an item's from the first to the last byte of its value.
- A member or item starts its line when only whitespace precedes it on its line. The separator of an entry is the comma between it and the next entry of the same container.
- A string value's span includes its quotes.
A duplicate member name is read as a finding on the second and later occurrences (std.duplicateId is DISL's for ids; a duplicate key is fbl.duplicate-key), and only the first is bound.
The body is an XML 1.0 document. The lossless reading keeps the prolog, comments, processing instructions and a document type declaration as unbound content; entity declarations are not processed, and a reference to an entity other than the five predefined ones and character references makes the containing element an unreadable entry. Nodes are elements, attributes and text:
- An element's own span runs from
<of its start tag to>of its end tag, or of its tag when it is self-closed (<x/>). - An attribute's value span is the text between its quotes; its own span runs from its name to its closing quote, and attribute order is significant (kept, and used for new attributes, section 6.3).
- An element's text is its character data with references decoded; a
textslot's span is the character data between the start tag and the first child element or the end tag. - Carriage returns are part of the content, not normalised: a reader that uses an XML parser MUST make it keep CR, since XML 1.0 normalises line endings and the body's bytes are the model.
- Escaping when writing: in text,
&,<and>are escaped as&,<and>; in attribute values also"as", LF as
and CR as
. Other characters are written literally in UTF-8. - A
content: "html-paragraphs"text slot reads the element's HTML body as plain text, one line per<p>element, whitespace inside a paragraph collapsed to single spaces; writing it emits one<p>line</p>per line with the escaping above.
The body is a sequence of lines. Each line is exactly one of:
- blank: only whitespace;
- a comment: it matches the binding's
comment; it is owned by the statement directly below it, if no blank line intervenes (a leading comment, section 4.1.1); - a statement: any other line. A statement matched by a rule's
lineis an entry of that rule; the first rule in the binding's order whoselinematches (and whosewhenholds) wins. A statement no rule matches is unbound (and reported whenunmatchedisreport).
A statement's own span is the line without its ending; its line span adds its leading comments and its line ending. Its values are its named groups. A group used with word is split into words: runs of non-whitespace, where a run starting with " extends to the next " and includes both quotes. A word slot's value is the first word of the group that matches the slot's word expression; if that expression has a named group value, the value is that group, otherwise the whole word (without quotes when the word is quoted). A flag word slot's value is true when such a word is present. (FBL 0.7) A word slot of an attribute with many holds every word of the group that matches its word expression, in order, each an item: the word's group value, or the whole word.
The body is lines as in the lines family, with nesting: a statement whose last non-whitespace character, outside double-quoted strings, is { opens a block, which extends to the line whose only non-whitespace content is the matching }. A statement's entry then spans from its line to the end of the closing line. Blocks nest; braces inside double-quoted strings do not count. A body whose braces do not balance is unreadable.
blocks (BlockRule) name the statements that open a block and produce no element (model {, views {), so that rules can state where they apply:
| Property | Type | Description |
|---|---|---|
name |
Name | Required. |
line |
Regex | Required. The statement. |
within |
Name[] | The block rules or element rules whose block the statement must be directly inside; ^ is the top level. Absent: anywhere. |
caseInsensitive |
bool | Match ignoring the case of ASCII letters. |
view |
group name | The block defines a view of the body, named by this group's value. A registration's view header selects it (section 9.3). |
An element rule with opens: true may open a block (it may also appear on one line without one); within on element and relation rules works as for block rules. A statement is matched only by rules whose within admits its enclosing block.
The body is lines, as in the lines family, holding list items that nest by their indentation: the part of a Markdown file that a tool reads as a tree. FBL reads the list items under the headings a binding names, and nothing else of Markdown (section 1.5).
Columns. A line's indentation is the spaces and tabs it starts with, counted in columns from 0, a tab counting four columns wherever it stands (not up to the next multiple of four).
Line kinds. Each line is one of the following, decided in document order. An item is open from its item line until a line closes it.
- The lines after the one that opens a fence, up to and including the one that closes it, are fence lines; the opening line itself is decided by the rules below. A fence opens on a line whose text after its indentation starts with three or more backticks or three or more tildes, and closes on the next line whose text after its indentation is a run of at least as many of the same character and nothing else but whitespace, or at the end of the body. Nothing inside a fence is any other kind of line.
- A blank line holds only whitespace. It closes nothing.
- Any other line first closes every open item whose marker stands at the line's indentation or deeper.
- An item line is indentation, a marker
-,*or+, then whitespace or the end of the line, and is not a thematic break (below). Its item is nested in the deepest item still open, if any, and is then open itself. The item's text is what follows the marker and the whitespace after it, to the end of the line, trailing whitespace included; its content column is the column where the text begins, or the marker's column plus two when the text is empty. - While an item is still open, any other line is a continuation line of the deepest open item. A fence that opens on a continuation line belongs to that item: its lines are continuation lines of it, whatever their indentation.
- With no item open, a line is a heading when it has at most three columns of indentation, then one to six
#, then whitespace or the end of the line. Its level is the number of#, and its text the rest of the line without leading and trailing whitespace and without a closing run of#that whitespace precedes. A line underlined with=or-is not a heading in this family. - With no item open, a line is a thematic break when it holds three or more of one of
-,*and_, and nothing else but spaces and tabs. It is never an item line, open or not. - With no item open, a line that opens a fence makes the fence unbound, with every line in it.
- Any other line is a statement, as in the
linesfamily.
Sections. A binding's sections name headings, as blocks name the statements that open a block:
| Property | Type | Description |
|---|---|---|
name |
Name | Required. Referenced by a rule's within and by insert.container. |
heading |
Regex | Required. Matched against a heading's text. |
caseInsensitive |
bool | Match ignoring the case of ASCII letters. |
A section is the lines after the first heading whose text matches heading, up to the next heading with as many # or fewer, or to the end of the body. A heading with more # inside it does not end it. A later heading that matches too is an ordinary heading. An item stands in a section when its item line is one of the section's lines.
Entries and spans. An item and a statement are entries. An item's own span runs from its marker to the end of its last continuation line or of its last nested item, whichever comes later, without that line's ending; its line span runs from the first byte of its item line to the end of that last line's ending. Blank lines inside an item's own span are part of it; those after it belong to no entry (section 4.1.1). A statement's spans are those of section 4.6. This family has no comments.
Rules. A rule with within matches items of the sections it names: its line is matched against the item's text. A rule without within matches statements, as in the lines family. Of the rules that may match an entry, the first in the binding's order whose line matches and whose when holds takes it (section 5.1). The named groups of line are the entry's values, split into words as section 4.6 says.
Continuation. An item's continuation is its continuation lines from the first to the last before its first nested item, the blank lines between them included; the continuation lines after a nested item are part of the item's span and are unbound. The slot {continuation: true} reads the continuation as text: its lines joined by LF, each with the whitespace before the item's content column taken off (a line indented less loses all of its indentation, and a tab the content column cuts into is replaced by the spaces that remain of it), a blank line read as an empty line. An item without continuation lines reads as the empty text.
Unbound. Headings, thematic breaks, blank lines and unbound fences are unbound. An item that stands in no section a rule's within names is unbound. An item that no rule matches, and an unreadable item (section 7.4), which includes an item whose rule's type the tool type's specification maps as unreadable (DESL section 5.2), are not looked into: they are unbound with everything in their span, and no rule matches an item nested in them. A statement that no rule matches is unbound, and reported only when unmatched is report.
The body is never unreadable for its family: Markdown has no malformed list (section 7.5).