3. Lexical Structure

Source encoding#

Source files are UTF-8. A leading BOM is ignored and takes no column.

A byte sequence that is not valid UTF-8 is an error, reported once per file, at the first malformed run, with the number of runs. The lexer reads each run as one U+FFFD, one column wide, and continues (§4 Strings).

Shebang#

A #! line is allowed on the first line only, and the lexer skips it. Anywhere else, #! is a lexical error. With #!/usr/bin/env -S vader run, a script runs directly on Unix.

#!/usr/bin/env -S vader run
module "main"
main :: fn() -> i32 { return 0 }

Comments#

// line comment, terminates at the next newline

/* block comment */

/*
   /* nested block comments are allowed */
   the outer comment continues here
*/

Block comments nest: a block comment ends at the */ that matches its /*.

VaderDoc#

A doc comment starts with ///. Consecutive /// lines form one doc block, attached to the declaration that follows. /** ... */ is an ordinary block comment.

After the description and a blank line, a doc block may use these tags, one per line:

/// Parse `s` as a base-10 `i32`. No prefix, no underscores, no leading
/// `+` sign; a leading `-` is accepted.
///
/// @param s  the digits-only input.
/// @return   the parsed integer.
/// @error    `ParseError` on empty input, malformed digit, or overflow.
/// @example
///   n :: parse_int("42")   // 42
export parse_int :: fn(s: string) -> i32 | ParseError

Every exported function of the standard library has a doc block in this form.

Identifiers#

[a-zA-Z_][a-zA-Z0-9_]*, case-sensitive. Identifiers are ASCII only in the MVP.

Reserved keywords#

fn struct trait implements enum
if else match is for in return yield await defer break continue
import as
export
true false null
self

Also reserved:

• const, reserved: it cannot be used as an identifier.
• module, a keyword only at the start of a file's module declaration.
• void, which names the type of expressions that produce no value (§6 Declaration). It cannot name a function, struct, parameter, local, type parameter, pattern binding or namespace import.
• Self, a type name rather than a keyword. Inside a trait or implements body it denotes the implementing type. Using it elsewhere is an error. The receiver parameter self is a keyword.

Literals#

Integer literals#

• Decimal: 42, 1_000_000
• Hexadecimal: 0xFF, 0xff, 0xFF_FF (case-insensitive after 0x)
• Binary: 0b1010, 0b1010_1010
• Octal: 0o755, 0o7_5_5

An integer literal is free: it takes the integer type its context requires, i32 by default (§4 Default integer). There is no type suffix (42i32 and 42_i32 are errors); give the type through the slot, an annotation or a cast.

Underscores separate digits, but never two in a row (1__000) and never at the start or end of the digits (0x_FF, 42_).

Float literals#

• Forms: 3.14, 0.5, 00.5 (leading zeros allowed), 1.0e-10, 1.5E+3.
• The point needs digits on both sides: 5. and .5 are errors; write 5.0 and 0.5.
• A float literal is free like an integer literal, f64 by default (§4 Default integer). There is no type suffix (3.14f32 is an error); annotate the slot to get an f32.
• Underscores follow the integer rules.

Boolean / null#

true, false and null are keywords.

Char literals#

'a'        // ASCII codepoint
'é'        // any Unicode codepoint
'\n'       // escape sequence
'\u{1F600}'  // explicit codepoint

• Escapes: \n, \t, \r, \\, \', \0, and \u{H...} with 1 to 6 hex digits. The value must be a Unicode scalar value: a surrogate (D800 to DFFF) or a value above 10FFFF is an error, in a char literal as in a string literal.
• A char literal holds exactly one code point: '' and 'ab' are lexical errors.
• A char literal is free. In an integer context, range types included, it takes that type and holds its code point, which must fit: with b: u8, b == '/' and d: 48..=57: '5' are valid, b == '💩' is an error. Elsewhere it is a char.
• Arithmetic on char literals works on their code points and gives an integer: 'a' - 'A' is 32.
• There is no b'X' byte literal: 'X' does the job.
• A char variable keeps the type char and never converts implicitly. u8(c) converts it, truncating code points above 0xFF.

String literals#

Four forms:

"plain"                        // simple string, supports interpolation
r"raw"                         // raw string, no interpolation, no escape processing
"""
multi-line
"""                            // triple-quoted, supports interpolation, may span newlines
r"""raw multi-line"""          // raw and triple-quoted: may contain `"`, no interpolation, no escapes

• Escapes in "..." and """...""": \n, \t, \r, \\, \", \$ (a dollar sign), \s (a space), \0, \u{H...}.
• In a raw string, a backslash is an ordinary character (\n stays two characters) and ${...} is plain text.
• A """ or r""" whose opening delimiter ends its line is a text block. That line break is not part of the text, and neither is the indentation that only lays the text out in the source:
  • every line loses the indentation all its lines share. The least indented line holding more than spaces and tabs sets it, and so does the closing """ when it stands alone on its line, so moving it left keeps that much indentation;
  • every line loses its trailing spaces and tabs, and a line of spaces and tabs only ends up empty;
  • a space and a tab count as one character each. An escape is text, never indentation: \s keeps a trailing space, \t a leading tab;
  • a line break inside ${...} is no line of the text, and an interpolated value is never cut;
  • every line break in the text, CRLF or a lone CR, reads as \n.
• A """ with text after its opening delimiter, on the same line, keeps every character.

body :: fn(name: string) -> string = """
    fn ${name}
      return
    end
    """                        // "fn f\n  return\nend\n" for name = "f"

String interpolation#

${expr} inside a non-raw string inserts the value of any expression. Interpolations nest: "outer ${"inner ${x}"}".

A single identifier needs no braces: "Hello, $name" is "Hello, ${name}". Anything longer (a field access, a call) needs ${...}.

Operators#

Arithmetic   : + - * / %
Wrapping     : +% -% *%  (integers only: the result wraps around the operand's width)
Bitwise      : & | ^ ~ << >>  (a shift's count keeps its low 5 bits, 6 for a 64-bit type: `1 << 33` on an `i32` is 2, `u8(1) << 9` is 0)
Comparison   : == != < <= > >=
Identity     : === !==   (same object; references only, §4 *Identity*)
Membership   : in   (sugar for `right.contains(left)`)
              !in   (sugar for `!right.contains(left)`)
Type test    : is Type   (in `match` arms and in `if`/`else` conditions; narrows the scrutinee)
              !is Type   (same as `!(x is Type)`)
Logical      : && || !
Assignment   : =  (also compound : += -= *= /= %= +%= -%= *%=, each `lhs op= rhs` meaning `lhs = lhs op rhs`, `lhs` evaluated once)
Declaration  : x :: value   x := value   x: T: value   x: T = value   (§5 *Binding forms*)
Range        : 0..<10 (exclusive)  0..=10 (inclusive)
Cast         : Type(expr) (numeric ↔ numeric, char ↔ integer)
Field access : .name
Index access : [expr]

Characters compare by code point. Strings compare by bytes, which for UTF-8 is also code-point order.

Membership: in#

x in c calls c.contains(x) from the Contains<T> trait (§4 Operator overloading), and x !in c negates it.

With a type on the right, in tests the values of that type:

• x in Hour checks an integer against the bounds of an integer type (§4 Range types);
• x in Weekday checks against the values of an enum (§4 Enums);
• x in char checks against the Unicode scalar values (§4 Casts).

The left operand must have the element type that contains expects. With r: Range<i32>, n in r is an error for n: i64 and for an i32 | null, as r.contains(n) would be. A free literal on the left takes the element type: 4_000_000_000 in r is valid for r: Range<i64>.

A literal range or array on the right takes the type of the left operand (its backing type, for a distinct type). A union on the left gives it no type.

n in 5_000_000_000..=6_000_000_000   // n: i64, the range is a Range<i64>
b in [' ', '\t']                     // b: u8, the array is a u8[]
b in -1..=3                          // b: u8, error: -1 is not a u8
s in "a"..="m"                       // error: Range<string> does not implement Contains

Assignment#

A variable's type is fixed by its declaration. Every assignment (x = v, arr[i] = v, m[k] = v, obj.f = v, and the compound forms) must give a value assignable to the declared type, Into conversions included, as for the initializer.

The check uses the declared type even where the variable is narrowed. Inside if x is A { ... }, x = b is allowed when b fits the declaration of x, and it ends the narrowing.

Operator precedence#

From the tightest (level 1) to the loosest:

Non-associative operators do not chain: a == b == c is a parse error; write (a == b) == c.

A cast Type(expr) is a call, at level 1.

A - written directly against a number literal, where an operand is expected, belongs to the literal. A postfix then applies to the negative value: -5.to_string() is "-5" and -5.abs() is 5. With anything in between, - is the prefix operator and negates the whole call: -x.abs(), -(5).abs(), - 5.abs(). After an operand, - is a subtraction: a -5 is a - 5.

Type tests and narrowing#

x is T tests the run-time type of a value. In the branch where the test holds, x is narrowed to T.

With a union on the right, is tests membership: x is A | B, or x is AB for an alias AB :: A | B, holds when the value is an A or a B, and narrows x to that union. Given U :: A | B | C, a U holding a B satisfies v is AB and can then be passed where an AB is expected.

With a trait on the right, x is Equals holds when the value's type implements the trait. That covers the primitives' impls and an enum's built-in Equals and Hash, and a generic impl counts only where its bounds hold. The branch where the test fails no longer holds those types: after if v is Equals { return } on v: i32 | Point, v is a Point (when Point has no Equals). A value held as a trait or as any is tested by its struct's impls alone.

A test that can never hold is an error: v is D with v: A | B.

An is test binds no name. It narrows the name or field chain it tests, in place. To test a value without a name, bind it first:

r :: f()
if r is T {
    use(r)                 // `r` is a `T` here
}

x is T as y is a parse error.

Conditions with &&, || and !#

• &&: in if x is T && rest, x is narrowed in rest and in the then-branch, so if x is Circle && x.radius > 0 type-checks. The else-branch is not narrowed: a false a && b does not say which operand was false.
• ||: the else-branch narrows every binding an operand tested. After if x is A || x is B, the else-branch sees x without A and B. The then-branch narrows only when every operand tests the same binding, to the union of the tested types: there x is A | B. if x is A || flag narrows nothing in the then-branch, and x in the else-branch.
• Right operand: it sees the outcome of the left one. The right of && sees the left as true, the right of || sees it as false, so y == null || y.a > 0 type-checks.
• ! swaps the two branches. The rules compose at any depth: if !(y is A || y is B) sees y as neither A nor B.

A guard that leaves the block narrows the rest of the block like an else-branch:

if x is A || x is B {
    return
}
// `x` is neither an `A` nor a `B` here

A loop condition narrows the loop body. In for cursor != null { ... cursor = cursor.next }, cursor is non-null until the assignment. The code after the loop is not narrowed, since a break can leave while the condition still holds.

Guards#

A guard is a function of the same module whose body is a single condition on its parameters: = cond, returning bool, not generic. Calling a guard in a condition narrows the arguments as cond narrows the parameters, in both branches.

Weekday :: Int in 0..=6
is_weekend :: fn(d: Weekday) -> bool = d >= 5
is_walk :: fn(a: Activity) -> bool = a is Walk
has_name :: fn(n: string | null) -> bool = n != null

if is_weekend(d) {
    // `d` is a `5..=6` here, and a `0..=4` in the else-branch
}

• Only an argument that is a binding or a field chain is narrowed.
• A guard can be declared after its call, called as d.is_weekend(), and built on another guard.
• In another module, a guard is an ordinary predicate and narrows nothing; @returns makes it narrow everywhere.
• A function with a block body is not a guard. A condition on a field of a parameter (b.label != null) does not narrow the argument.

Declared guards: @returns#

@returns(cond) declares that a free function returns the value of cond. Its calls narrow their arguments as a guard's do, in every module.

The compiler checks the body against cond at each value it returns. Where the function returns true, cond must hold; where it returns false, cond must fail:

@returns(d in Weekend)
is_weekend :: fn(d: Weekday) -> bool = d >= 4   // error: true for d = 4

A block body is checked at each return and at its trailing value, with the narrowings in force there. An if value is checked branch by branch.

The following are errors:

• a function that is generic, does not return bool, or has no body;
• a body that assigns a parameter, directly or from a lambda;
• a condition that says nothing about the parameters;
• more than one condition, or @returns on anything but a free function.

Declared postconditions: @ensures#

@ensures(cond) declares that cond holds once the function returns normally. After a call written as a statement, or as the initializer of a declaration (r :: f(n)), the arguments are narrowed as cond narrows the parameters, for the rest of the block, in every module.

@ensures(h in Hour)
require_hour :: fn(h: i64) -> void {
    if h !in Hour {
        panic("not an hour: ${h}")
    }
}

require_hour(n)
label(n)                   // `n` is an `Hour` here

• Only an argument that is a binding no lambda assigns is narrowed.
• Nothing is checked at run time. The compiler checks the body at each return and at its end: cond must hold there. Stopping the program where cond fails satisfies it, as require_hour does, and so does returning a call to a function that ensures the same condition.
• The function may return any type, but its body must have finished when a call returns: a generator, a function that uses await or a function without a body is an error.
• The other rules of @returns apply.

How long a narrowing lasts#

A narrowing lasts as long as nothing can have changed what it tested. An assignment to the narrowed name or field ends it, unless the new value still fits (Assignment, above). It also ends:

• after a call, for every field narrowing (a.s, self.items) and for each local that some lambda assigns, since a call can write any object it reaches. Writing through a user IndexSet (m[k] = v), await and yield count as calls. A conversion T(x) does not.
• inside a loop, for what an earlier iteration can have changed: field narrowings, and locals the body assigns. The iterable of for x in e is evaluated once, before the loop, and keeps the narrowing.
• after a field assignment through any path (b.s = v, xs[i].s = v), for every narrowing through a field of that name: b may be the same object as a.
• after a call in the right operand of && (or of ||, on its false side), for what the left operand narrowed: in the rest of the condition and in the branch.
• in code that runs later than written (a lambda body, a defer body), which keeps only the narrowings of bindings that are never reassigned. Under for c != null { ... c = c.next }, defer println(c.value) is an error.

The narrowing of a local that nothing else assigns survives all of these. To keep a narrowing across a call, bind the field to a local first:

s :: a.s
if s is Circle {
    reset(a)
    use(s.r)               // `s` still holds the value that was tested
}

Statement separators#

Statements in a block are separated by newlines (Newline-significant, below). ; is not a separator.

A newline right after { is ignored, so a block can start on the next line.

Trailing commas#

Trailing commas are allowed in every comma-separated list: function arguments, function parameters, struct literal fields, struct definition fields, array literals, match arms, generic argument lists, and import modifier lists (import "p" { x as y, }).

Newline-significant#

A newline ends a statement, except in these five cases, where it is ignored:

1. Inside an open ( or [, and inside a ${...} interpolation. Inside a { } block, newlines still separate statements, even when the block is inside brackets (the body of a lambda passed as an argument).
2. After an operator that still needs its operand: +, -, *, /, %, <, <=, >, >=, ==, !=, ===, !==, &&, ||, &, |, ^, <<, >>, +%, -%, *%, ??, ..<, ..=, and the spread .... A newline after ! is not ignored.
3. After a comma.
4. After a token that needs a right-hand side: =, :, ->, ::, :=, and the compound assignments += -= *= /= %= +%= -%= *%=.
5. Before an operator with no prefix form. When the next line starts with +, *, /, %, +%, -%, *%, ^, ??, ==, !=, ===, !==, <, <=, <<, >, >=, >>, &&, ||, & or |, it continues the previous expression, in a { } block too. Comment lines in between are skipped. A compound assignment (+=, *=, ...) does not continue the previous line, nor do -, ~, ! and ., which can start an expression.

There is no backslash continuation: a \ at the end of a line is a lexical error. To break a long expression, use parentheses or break before or after a binary operator.

// OK: break inside parens
total :: (a +
          b +
          c)

// OK: break after operator
total :: a +
         b +
         c

// OK: break before a leading infix-only operator
if a == 1
    && b == 2
    && c == 3 {
    // ...
}
label :: if verbose {
    "value "
        + "${n}"
} else {
    "${n}"
}
name :: first_name(user)
    ?? return "anonymous"

// OK: break after comma
list :: [
    1,
    2,
    3,
]

// error: backslash is not a continuation
x :: a + \
     b           // lexer error: stray '\'