Lil is part of the technology that powers Decker, a multimedia creative tool inspired by HyperCard. Decker uses Lil for adding custom behavior to decks and the widgets within. Lil is designed to be learned in layers, but it is a richly multi-paradigm language which incorporates ideas from imperative, functional, declarative, and vector-oriented languages.
on mode a do # line comment r:() each x in a r[x]:1+r[x] end first extract key orderby value desc from r end mode[1,2,2,3,4,2,1]
Lil's implementation needs to be fairly small, as an interpreter is shipped with every standalone Decker document along with the rest of the runtime. At the same time, Lil should be prepared for the everyday needs of Decker users, with primitives to handle common use-cases and a minimum of boilerplate.
The language therefore tries to thread the needle between a design which is simple and a design which is ergonomically pleasant. There are a small number of datatypes, features and primitives which compose in many useful ways. The query syntax generalizes to manipulation of dictionaries, lists, and strings, and replaces many individual primitive operations in other languages with a single algorithmic framework. Simple things are easy, and complex things are possible.
There are 7 types of value in Lil: numbers, strings, lists, dictionaries, tables, functions, and interfaces.
42 37.5 -29999.\n) are preceded with a backslash escape: "apple" "foo\nbar". The binary fuse and format operators can both be used for concatenating strings.(), and the unary operator list constructs a single-element list from any value. Lists can be joined with the , operator.dict, or a list can be promoted to a dictionary by assigning at non-numeric keys. Dictionaries can be unioned together with the , operator. The keys operator gets the keys of a dictionary, and the range operator gets the values of a dictionary.table or with the insert statement. The binary operators , (append), join (natural join), and cross (cartesian join) offer a selection of tabular joins.on, and may be freely passed around or stored in variables. first of a function gives its name, and keys of a function gives a list of its named arguments.Lil has a soft, spongy, dynamic type system in which values do their best to convert to a more relevant type as the need arises.
When indexing into values, we will uniformly refer to elements:
The number zero (0) and the empty string (""), list (()), or dictionary are all falsey, and any other value (including any table, function, or interface) is truthy.
Lil scripts are a sequence of expressions, including literal values, references or assignments to variables, conditionals (if), each and while loops, and function declarations (on).
Variable and function names may contain any alphanumeric characters (as well as ? and _), but must not start with a digit. Variable assignments use the symbol :, read as "gets" or "becomes":
a: 23
longer_name: "a string"If a value has never been explicitly placed in a variable, it contains the number 0. You can access or assign to elements of a list or dictionary by subscripting with brackets:
(11,22,33)[1] "String"[2] l[4]:2 d["fruit"]:"apple"
For convenience, indexing may also be performed with "dot notation", where a name after the dot is treated as a string index. These expressions are equivalent:
d["fruit"]:"cherry" d.fruit:"cherry"
If you index a list with integers within range of its count it will act like a list. If you index a list like a dictionary- with keys that are not integers within range of its count- it will become a dictionary.
b: 5 # b contains the number 5 b[0]: 5 # b becomes a length-1 list containing the number 5 c # c contains the number 0 c.fruit: "yes" # c now contains a dictionary with the key "fruit" and the value "yes"
When the leftmost part of an assignment expression is an expression- not just a variable name- the value of that expression is amended and a new value with appropriate changes is returned:
(11,22,33)[1]:44 # produces the list (11,44,33) "Cat"[1]:"ive" # produces the string "Civet" ().baz:99 # amend the empty list at index "baz" (promoting it to a dictionary) with the value 99, # producing the single-element dictionary {"baz":99} (11,22).baz:33 # extend (11,22) into a dictionary, and bind "baz", giving {0:11,1:22,"baz":33}
Note the important distinctions between these similar-looking cases:
foo:11,22,33 foo[1]:44 # get the value in "foo", amend it at index 1, and store the amended list in "foo" again. (foo)[1]:44 # the parenthesized portion is a subexpression which gets the value in "foo". # amend it at index 1, and return the amended list, leaving "foo" unchanged. (foo[1]):44 # this is not syntactically valid! # assignment to a subexpression must have a dotted or bracketed index.
If you index with a dot followed immediately by another dot or open-bracket it means "each element". Use this for drilling into nested lists or dictionaries:
t:("AB","CD","EFG") t[1] # "CD" # the 1st element of t t.[1] # ("B","D","F") # the 1st element of each element of t d.a.key:"apple" d.b.key:"pear" d # {"a":{"key":"apple"},"b":{"key":"pear"}} d.a # {"key":"apple"} d.a.key # "apple" d..key # {"a":"apple","b":"pear"}
Lil has a number of unary and binary operators- see appendix 1 and 2 for details. Many, like + and *, should be familiar from mathematics or other programming langugages. The most important thing to remember in Lil is that expressions have uniform operator precedence: expressions are always carried out right-to-left, unless explicitly parenthesized:
3*2+5 # 21 3*(2+5) # 21 (3*2)+5 # 11
You can make decisions with if. The keyword if is followed by a conditional expression, one or more statements, and finally the keyword end. The statements inside the if ... end will only be executed if the conditional expression has a truthy value. You can also optionally include an elseif keyword (followed by a conditional expression) or just an else keyword to divide the body of the if into multiple cases.
if 1>2 "narp" # this doesn't happen end if 5 "yarp" # 5 is truthy, so this does happen! end if 5 "also yarp" # this happens... else "also narp" # but this does not! end if 1>2 "narp" # this doesn't happen elseif 1<2 "yarp" # but this does, else "narp" # so this doesn't end
You can iterate over the elements of a value with an each loop. The keyword each is followed by zero or more variable names, the keyword in, an expression giving the source to iterate over, one or more statements forming the "body" of the loop, and finally the keyword end. The source is always treated as a dictionary, and for each iteration of the loop the variable names provided are assigned to the value, key, and index of the current pairing in that dictionary. For a list, key and index will naturally be identical. Here are a few examples of each loops:
each x in 3,5,7 x end each val key in d val,key end r:0 each x in range 10 r:r+x end
You can also repeat a body of code an indefinite number of times with a while loop. The keyword while is followed by a conditional expression, one or more statements forming the "body" of the loop, and finally the keyword end. The conditional expression is evaluated before each iteration of the loop, and the loop stops if this ever results in a falsey value. Here's an example:
a:5 while a>3 print[a] # prints 5, then 4, then stops. a:a-1 end
To declare a function, use the keyword on followed by a name, zero or more argument names, the keyword do, one or more statements comprising the "body" of the function, and finally the keyword end. To call a function, use its name and a set of bracketed expressions corresponding to the arguments it takes. Extra arguments are ignored, and missing arguments are bound as 0:
on pair x y do x,y end pair[3 5] # returns a list of 3 and 5 pair[3] # returns a list of 3 and 0 pair[3 5 7] # returns a list of 3 and 5 pair[3,5] # careful: ',' joins items into lists- it is not an argument separator! pair 3 5 # wrong: this is three separate statements, not a function call!
If you want to define a function that takes a variable number of arguments- a so-called variadic or ambivalent function- declare the function with a single named argument prefaced with .... When the function is called, this argument will contain a list of every argument provided to the function:
on several ...x do 1-x end several[11 22 33] # (-10,-21,-32)
As we've seen above, thinking about Lil as an everyday garden-variety imperative language is perfectly sufficient for writing scripts. Some characteristics, though, make it well-suited to the functional style of programming, in which we aim to minimize mutation, compose our program from functions which do not have "side-effects", and make use of so-called "higher-order" functions- functions which take functions as arguments.
Values in Lil are truly values. They have copy-on-write semantics: an assignment to part of a value creates an entirely new value, leaving any other references to the original value unchanged:
a:1,2,3 b:a b[1]:5 show[a] # 1,2,3 show[b] # 1,5,3
Statements in Lil are always expressions. That is, they always return a value and can be composed within larger expressions. Assignments evaluate to the value being assigned. if returns the value of the last statement in its taken half. (A missing else just means the falsey half evaluates to 0.) while likewise evaluates to the last statement on the last iteration of its body:
x:y:z # assign both x and y the value of z a: if x>5 99 else 33 end # assign a to either 99 or 33 c: while b<100 b:b*2 end # assign c to 128
The each loop collects together the results of each iteration of the loop and returns them. If the input was a dictionary, the output will be a dictionary with the same keys. Otherwise, the output will be a list. Some other languages call this operation map:
each x in 3,5,7 # (300,500,700) x*100 end t.foo:"one" t.bar:"three" each v in t # {"foo":3,"bar":5} count v end
You can pass functions to other functions by name:
on apply func do func["two"] end on twice x do x,x end apply[twice] # ("two","two")
Since on...end is an expression, you can also directly substitute it into the call, just as with "anonymous" or "lambda" functions in other languages.
apply[on thrice x do x,x,x end]
One advantage of functions always having a bound name is that there's no special combinator plumbing required for anonymous recursion. If the name never matters, consider using the name _ to signal your intent.
Lil uses lexical scope: variables will resolve to the closest nested binding available, and the local variables of a caller to a function will not be visible or modified by the callee (unless the callee's definition is nested in the caller):
global:333 on quux x do v:99 x[77] print[v] # 99 end on zami x do v:23 print[global,v,x] # 333,23,77 end quux[zami]
Furthermore, functions close over variables in their lexical scope, allowing for encapsulated "objects" with their own mutable state:
on counter x do on inc do x:x+1 end end a:counter[100] b:counter[200] print[a[]] # 101 print[a[]] # 102 print[b[]] # 201 print[a[]] # 103 print[x ] # 0
Whenever an assignment is carried out, if the variable name in question has been assigned to in any surrounding lexical scope, the assignment will update the closest definition. If the name has never been assigned to before, a new local variable will be created at the current scope. The on, each, and query (select, update, extract) statements always create new local variables for their arguments, loop variables, or columns, and can thus shadow (take precedence over) outer local variables of the same name. When in doubt, use unique names; it's much less confusing!
It is also possible to explicitly define a local variable by using the keyword local, a name, a colon (:), and an expression giving the local's initial value:
duplicate:"Alpha" on func do local duplicate:"Beta" show[duplicate] # "Beta" end func[] show[duplicate] # "Alpha"
Finally, if you're mentally wed to the idea of expressing algorithms recursively, you may be pleased to discover that Lil supports tail-call elimination. If a recursive function calls itself (or another function) as the final operation in the function, it will not consume extra stack space:
on addrec x y do if x>0 1+addrec[x-1 y] # addition happens after call returns. not tail-recursive! else y end end on addtail x y do if x>0 addtail[x-1 y+1] # tail-recursive call. else y end end addtail[80000 5] # this won't blow the stack.
So far, we haven't looked much at tables. While one can operate on tables by indexing them or iterate over their contents with an each loop, the real power of tables comes from Lil's query syntax, which resembles a simplified version of SQL. Unlike some of their SQL counterparts, select, extract, update, and insert are all pure operations which strictly return a new value as a result.
A query consists of an operation, a sequence of column expressions (possibly with names), any number of where, by, and/or orderby clauses, the keyword from, and then finally an expression giving the source table. If the source is non-tabular, it will be converted into its table equivalent automatically. The operations are:
select: query a table and obtain a table of results.extract: query a table and obtain a list or dictionary of results.update: produce an amended version of a table, preserving its overall structure and order.If a query has multiple clauses, they're carried out right-to-left, just like Lil expressions. You can think of each clause as taking a table and a column expression, evaluating the expression within the context of the table (binding each table column as a variable), and using the resulting column to transform the input table into a different table. The clauses are:
where x: filter the table to only the rows of x which are truthy.orderby x dir: sort the rows of the table such that they would put the values of x in order; dir can either be asc (ascending) or desc (descending). Lists are sorted lexicographically, numbers are sorted by value, and anything else is sorted lexicographically by its string equivalent.by x: group the rows of the table by the dictinct values of x, producing 1 or more "subtables". Once the source table has been grouped, all subsequent clauses and result column expressions are applied to the subtables individually. If more than one by clause appears in a query, the subtables will be "ungrouped" into a single table before re-grouping.When column expressions are evaluated, several "magic" columns and variables are automatically defined:
index: integer list; the original row number in the input table for each row in the current table.gindex: integer list; the row number for each row in the current group (subtable); this is "reset" after each clause is carried out.group: integer list; the group number for each row in the current group (subtable); this is 0 for all rows in an ungrouped query.column: dictionary; all of the columns in the current table or subtable, keyed by name.When "ungrouping" or collecting the final result columns, the table is rectangularized: each group will have as many result rows as the column of maximum count. Grouping may change the order of rows- see update if you want to preserve it.
Given a simple table:
people.name:"Alice","Sam","Thomas","Sara","Walter" people.age:25,28,40,34,43 people.job:"Developer","Sales","Developer","Developer","Accounting" people:table people
Let's perform some selections:
select from people # select all columns, like "select *" in SQL # +----------+-----+--------------+ # | name | age | job | # +----------+-----+--------------+ # | "Alice" | 25 | "Developer" | # | "Sam" | 28 | "Sales" | # | "Thomas" | 40 | "Developer" | # | "Sara" | 34 | "Developer" | # | "Walter" | 43 | "Accounting" | # +----------+-----+--------------+ select name from people # select a specific column # +----------+ # | name | # +----------+ # | "Alice" | # | "Sam" | # | "Thomas" | # | "Sara" | # | "Walter" | # +----------+ select firstName:name dogYears:7*age from people # compute and rename columns # +-----------+----------+ # | firstName | dogYears | # +-----------+----------+ # | "Alice" | 175 | # | "Sam" | 196 | # | "Thomas" | 280 | # | "Sara" | 238 | # | "Walter" | 301 | # +-----------+----------+ select name where name like "S*" from people # where takes a boolean (0/1) column # +--------+ # | name | # +--------+ # | "Sam" | # | "Sara" | # +--------+ select name index orderby name asc from people # orderby takes a column of values and "asc" or "desc" # +----------+-------+ # | name | index | # +----------+-------+ # | "Alice" | 0 | # | "Sam" | 1 | # | "Sara" | 3 | # | "Thomas" | 2 | # | "Walter" | 4 | # +----------+-------+ select name job by job orderby name asc from people # sort names within each group # +----------+--------------+ # | name | job | # +----------+--------------+ # | "Alice" | "Developer" | # | "Sara" | "Developer" | # | "Thomas" | "Developer" | # | "Sam" | "Sales" | # | "Walter" | "Accounting" | # +----------+--------------+ select name job orderby (job join name) asc from people # sort by multiple columns # +----------+--------------+ # | name | job | # +----------+--------------+ # | "Walter" | "Accounting" | # | "Alice" | "Developer" | # | "Sara" | "Developer" | # | "Thomas" | "Developer" | # | "Sam" | "Sales" | # +----------+--------------+
All result columns will be repeated to match the count of the longest result. Thus, if all the new columns yield a non-listy result, the group is effectively collapsed into a summary row:
select employed:(count name) job by job from people # +----------+--------------+ # | employed | job | # +----------+--------------+ # | 3 | "Developer" | # | 3 | "Developer" | # | 3 | "Developer" | # | 1 | "Sales" | # | 1 | "Accounting" | # +----------+--------------+ select employed:(count name) job:(first job) by job from people # +----------+--------------+ # | employed | job | # +----------+--------------+ # | 3 | "Developer" | # | 1 | "Sales" | # | 1 | "Accounting" | # +----------+--------------+
When computing columns, you're working with lists of elements, and taking advantage of the fact that primitives like < and + automatically "spread" to lists. When performing comparisons, be sure to use = rather than ~! If you want to call your own functions- say, to average within a grouped column- write them to accept a list like so:
on avg x do (sum x) / count x end select job:(first job) avg_age:avg[age] by job from people # +--------------+---------+ # | job | avg_age | # +--------------+---------+ # | "Developer" | 33 | # | "Sales" | 28 | # | "Accounting" | 43 | # +--------------+---------+
The same applies to any functions called in a where, by, or orderby expression. The predefined aggregation functions sum, prod, raze, min, and max may come in handy!
The update statement has the same syntax and clauses as select, but behaves differently: results are merged with the original table, and the original order of rows is preserved. orderby can be used with update, but will only impact the inputs to column expressions, not the order of the result rows.
update job:"Engineer" where job="Developer" from people # +----------+-----+--------------+ # | name | age | job | # +----------+-----+--------------+ # | "Alice" | 35 | "Engineer" | # | "Sam" | 28 | "Sales" | # | "Thomas" | 50 | "Engineer" | # | "Sara" | 44 | "Engineer" | # | "Walter" | 43 | "Accounting" | # +----------+-----+--------------+
As with select you can compute new named columns- values will be filled in with 0 for rows masked off by a where clause.
update manager:random[name] where job="Developer" from people # +----------+-----+--------------+---------+ # | name | age | job | manager | # +----------+-----+--------------+---------+ # | "Alice" | 25 | "Developer" | "Sara" | # | "Sam" | 28 | "Sales" | 0 | # | "Thomas" | 40 | "Developer" | "Sara" | # | "Sara" | 34 | "Developer" | "Sara" | # | "Walter" | 43 | "Accounting" | 0 | # +----------+-----+--------------+---------+
When mixing queries with other types of code, it may be very useful to execute a query and get back simple lists, strings, or numbers. The extract statement is another variation on select which unpacks its result directly into a value, instead of a result column in a table:
jobs:extract first job by job from people # ("Developer","Sales","Accounting")
extract gives you access to a variety of useful operations for tables, dictionaries, or even lists:
extract value orderby value asc from jobs # sort a list # ("Accounting","Developer","Sales") extract index orderby value asc from jobs # grade a list # (2,0,1) extract value orderby index desc from jobs # reverse a list # ("Accounting","Sales","Developer") extract list index by value from "ABBAAC" # group a list # ((0,3,4),(1,2),(5)) extract list value by floor index/3 from "ABCDEFGHI" # partition a list (see 'window') # (("A","B","C"),("D","E","F"),("G","H","I")) extract first value by value from "ABBAAC" # distinct items in a list # ("A","B","C")
If names are specified, all results are collected into a dictionary:
extract a:first age b:last age orderby age asc from people # {"a":(25),"b":(43)}
If no columns are specified, the result will be the first column of the table:
extract orderby value asc from "BEDAC" # ("A","B","C","D","E")
The insert statement adds new rows to a table. It is followed by a sequence of one or more column names terminated by the with keyword, and then a series of expressions supplying the values of each row. The insert statement concludes with end, when creating a new table from scratch, or into followed by an expression evaluating to a "source" table when appending to an existing table.
insert name job age with "John" "Writer" 32 end # +--------+----------+-----+ # | name | job | age | # +--------+----------+-----+ # | "John" | "Writer" | 32 | # +--------+----------+-----+ insert name job age with "John" "Writer" 32 into people # +----------+-----+--------------+ # | name | age | job | # +----------+-----+--------------+ # | "Alice" | 25 | "Developer" | # | "Sam" | 28 | "Sales" | # | "Thomas" | 40 | "Developer" | # | "Sara" | 34 | "Developer" | # | "Walter" | 43 | "Accounting" | # | "John" | 32 | "Writer" | # +----------+-----+--------------+
As a special case, if the value to insert into is a number, treat it as an empty table; insert ... into 0 is equivalent to insert ... end.
Lil also offers two basic joining operations: join (natural join), and cross (cartesian/cross join):
jobs:insert job salary with "Sales" 85000 "Developer" 75000 "Accounting" 60000 "Facilities" 50000 end # +--------------+--------+ # | job | salary | # +--------------+--------+ # | "Sales" | 85000 | # | "Developer" | 75000 | # | "Accounting" | 60000 | # | "Facilities" | 50000 | # +--------------+--------+ people join jobs # +----------+-----+--------------+--------+ # | name | age | job | salary | # +----------+-----+--------------+--------+ # | "Alice" | 25 | "Developer" | 75000 | # | "Sam" | 28 | "Sales" | 85000 | # | "Thomas" | 40 | "Developer" | 75000 | # | "Sara" | 34 | "Developer" | 75000 | # | "Walter" | 43 | "Accounting" | 60000 | # +----------+-----+--------------+--------+ guests:insert name with "Alice" "Joan" "Oscar" "Thomas" end # +----------+ # | name | # +----------+ # | "Alice" | # | "Joan" | # | "Oscar" | # | "Thomas" | # +----------+ select a:name b:name_ where name < name_ from guests cross guests # +---------+----------+ # | a | b | # +---------+----------+ # | "Alice" | "Joan" | # | "Alice" | "Oscar" | # | "Joan" | "Oscar" | # | "Alice" | "Thomas" | # | "Joan" | "Thomas" | # | "Oscar" | "Thomas" | # +---------+----------+
For a row-wise join- concatenating tables with the same columns- you can use ,. You can likewise take or drop rows or columns from a table. If the left argument to take is a list of numbers, it picks out those rows very much like a generalization of select:
"name" take people # +----------+ # | name | # +----------+ # | "Alice" | # | "Sam" | # | "Thomas" | # | "Sara" | # | "Walter" | # +----------+ ("age","job") take people # +-----+--------------+ # | age | job | # +-----+--------------+ # | 25 | "Developer" | # | 28 | "Sales" | # | 40 | "Developer" | # | 34 | "Developer" | # | 43 | "Accounting" | # +-----+--------------+ 3 drop people # +----------+-----+--------------+ # | name | age | job | # +----------+-----+--------------+ # | "Sara" | 34 | "Developer" | # | "Walter" | 43 | "Accounting" | # +----------+-----+--------------+ (0,2,3) take people # +----------+-----+-------------+ # | name | age | job | # +----------+-----+-------------+ # | "Alice" | 25 | "Developer" | # | "Thomas" | 40 | "Developer" | # | "Sara" | 34 | "Developer" | # +----------+-----+-------------+
The flip of a table transposes its data, promoting the first column (or, if it exists, a column named key) from the original table to column keys for the result:
expenses:insert kind jan feb with "tax" 11 55 "gas" 22 66 "power" 33 77 "food" 44 88 end # +---------+-----+-----+ # | kind | jan | feb | # +---------+-----+-----+ # | "tax" | 11 | 55 | # | "gas" | 22 | 66 | # | "power" | 33 | 77 | # | "food" | 44 | 88 | # +---------+-----+-----+ flip expenses # +-------+-----+-----+-------+------+ # | key | tax | gas | power | food | # +-------+-----+-----+-------+------+ # | "jan" | 11 | 22 | 33 | 44 | # | "feb" | 55 | 66 | 77 | 88 | # +-------+-----+-----+-------+------+
The key column makes the flip of a table a reversible operation. If you don't need the original keys, you can discard them with drop:
"key" drop flip expenses # +-----+-----+-------+------+ # | tax | gas | power | food | # +-----+-----+-------+------+ # | 11 | 22 | 33 | 44 | # | 55 | 66 | 77 | 88 | # +-----+-----+-------+------+
In some situations you may wish to construct or query tables containing columns which have names that are reserved keywords (like count or range) or are not valid Lil identifiers (like a name with spaces). To define such a column, enclose the name in double-quotes:
denormal: select "with \"escapes":index "count":value from "ABC" # +---------------+-------+ # | with "escapes | count | # +---------------+-------+ # | 0 | "A" | # | 1 | "B" | # | 2 | "C" | # +---------------+-------+ insert "pet name" "pet species" with "Galena" "Chicken" "Pippi" "Chicken" "Chester" "Toad" end # +-----------+-------------+ # | pet name | pet species | # +-----------+-------------+ # | "Galena" | "Chicken" | # | "Pippi" | "Chicken" | # | "Chester" | "Toad" | # +-----------+-------------+
To reference these columns in a query, every column expression has the variable column bound to the entire table (or subtable, when grouping) within column expressions. As usual, you can access specific columns by indexing this table with a string:
select where column["with \"escapes"]>0 from denormal # +---------------+-------+ # | with "escapes | count | # +---------------+-------+ # | 1 | "B" | # | 2 | "C" | # +---------------+-------+ first extract column from denormal # +---------------+-------+-------+--------+-------+ # | with "escapes | count | index | gindex | group | # +---------------+-------+-------+--------+-------+ # | 0 | "A" | 0 | 0 | 0 | # | 1 | "B" | 1 | 1 | 0 | # | 2 | "C" | 2 | 2 | 0 | # +---------------+-------+-------+--------+-------+
Extracting column may occasionally be useful for debugging complex queries!
In general, use select for narrowing down or summarizing tables, update for making changes to a table while preserving or extending its existing structure, extract for pulling data out of a table for use elsewhere, and insert to make new tables or append to existing ones.
It may also be helpful to think in terms of how the shape of output tables relates to input tables for various table operations:
| Query Statement | Output Rows | Output Columns |
|---|---|---|
select | <= input | any number |
update | = input | >= input |
insert | >= input | = input |
extract | <= input | n/a |
join | <= x * y | < x + y |
cross | = x * y | = x + y |
flip | x columns | x rows |
The parse and format primitives are used for breaking strings apart into Lil values, and formatting Lil values into a customizable string representation, respectively. Both take as their left argument a format string with a concise notation for controlling these processes. In many cases, these format strings are symmetrical: the parse of the format under the same Format string will be an identity operation, and vice versa:
f: "0x%04h" # a four-digit zero-padded lowercase hexadecimal number with an '0x' prefix f format 123 # "0x007b" f parse "0x007b" # 123
Format strings consist of a sequence of patterns and literals. Patterns always begin with a % character, may contain several optional flags, and end in an alphabetic character. Pattern flags configure the details of how each pattern behaves, and may appear in the structure %[name]*-0N.DX, where N and D may be 1 or more digits 0-9, and X is a pattern type. All flags are optional. If present, their meanings are as follows:
name may contain any number of non-] characters. If any pattern in a format string has an explicit name, any other patterns without an explicit name are assigned a default name corresponding to their index, starting from 0.* this pattern will be validated, but will not produce output values. When formatting, instead of consuming an input value, these patterns will use the appropriate null value.- specifies left-justification for N, as decribed below. The default is right-justification. The %r/%o patterns use - to invert the set of valid characters.0 specifies padding with 0 for N, as described below. The default is padding with spaces.N indicates that the value will be padded to at least N characters, using the justification and padding character selected with - and 0..D for %f/%c, the value should be formatted with exactly D decimal places. The %r/%o patterns use D to indicate a number of valid characters. For any other type, the formatted string will be truncated at D characters, respecting justification.Pattern types are as follows:
| Type | Null | Parsed | Formatted |
|---|---|---|---|
% | n/a | literal % character. * is implied. | % |
n | n/a | number of chars that have been read. | nothing. |
m | n/a | matched? value is 1 iff the format has matched so far. | nothing. |
z | n/a | value is 1 iff the format matches and read the whole input. | nothing. |
s | "" | string. read N chars or until next literal. | any string, up to D chars. |
u | "" | uppercase string. just like s, but converts to uppercase. | any string, up to D chars, converted to uppercase. |
l | "" | lowercase string. just like s, but converts to lowercase. | any string, up to D chars, converted to lowercase. |
r | "" | repeat. 0 or more (or N) characters within a valid set. | any string, exactly N chars if specified. |
o | "" | optional. 0 or 1 (or N) characters within a valid set. | any string, exactly N chars if specified. |
a | () | ASCII. reads like s; value is list of ASCII ordinals. | list of ASCII ordinals converted to a string. |
b | 0 | reads like s; value is 1 iff first char is in tTyYx1. | any value to true or false based on truthiness. |
f | 0.0 | Lil float. allows any number of decimals. | show D decimal places or however many are needed. |
c | 0.0 | currency. parses values like -$1.23 to Lil floats. | show D or 2 decimal places, like -$1.23. |
C | 0.0 | plain currency. parses values like -1.23 to Lil floats. | show D or 2 decimal places, like -1.23. |
i | 0 | signed integer. | signed integer. |
h | 0 | hexadecimal integer. parses lower- or uppercase. | format int as hexadecimal in lowercase. |
H | 0 | hexadecimal integer. parses lower- or uppercase. | format int as hexadecimal in uppercase. |
j | 0 | a JSON value. | any value to a JSON string. |
J | 0 | a superset of JSON that supports any Lil value. | any value to a string. |
q | "" | quoted. a Lil string literal, like "foo\nbar". | any string to a Lil string literal. |
v | "" | variable. a Lil variable name, like ice_9. | any string. |
e | 0 | read ISO-8601 date-time into a unix epoch int. | format unix epoch int as ISO-8601 date-time. |
p | () | read ISO-8601 date-time as a dictionary of time parts. | format dict as ISO-8601. |
When parsing, each pattern will be matched against input in sequence, consuming some number of input characters and producing output values, and literals will be expected; if at any point a pattern or literal fails to match against input, parsing will cease, and any subsequent patterns in the format string will yield appropriate "null" values. Thus, a given format string will always yield a fixed number of results, no matter the input. In the case where there is exactly one value-yielding pattern in the format string, parse output will simply be that value instead of a list.
"%f %s %i" parse "12 apples" # (12,"apples",0) "%f %ss" parse "12 apples" # (12,"apple") ("amount","noun") dict "%f %ss" parse "12 apples" # {"amount":12,"noun":"apple"} "[%s]" parse "[something]" # "something"
The right argument to parse may be list of strings; in this case the output will be a list of rows, where each row contains the values from parsing one input string. The flip of this would give columns instead. Here's an example of parsing a table of fixed-width records:
form: "%6s%6c%2i" data: "apple $1.00 1\ncherry $0.3515\nbanana $0.75 2" r: form parse "\n" split data #(("apple ",1,1),("cherry",0.35,15),("banana",0.75,2)) table r #+----------+------+----+ #| c0 | c1 | c2 | #+----------+------+----+ #| "apple " | 1 | 1 | #| "cherry" | 0.35 | 15 | #| "banana" | 0.75 | 2 | #+----------+------+----+ flip r #(("apple ","cherry","banana"),(1,0.35,0.75),(1,15,2)) t: table ("name","price","amt") dict flip r #+----------+-------+-----+ #| name | price | amt | #+----------+-------+-----+ #| "apple " | 1 | 1 | #| "cherry" | 0.35 | 15 | #| "banana" | 0.75 | 2 | #+----------+-------+-----+ ("\n",form) format t # round-trip #"apple $1.00 1\ncherry $0.3515\nbanana $0.75 2"
When formatting, literals will be included in the output string and each pattern will control conversion of one value from the input list, with a few exceptions as explained below. Missing arguments will be interpreted as appropriate "null" values.
"%i,%a,%i" format 1,(list 65,66,67) # "1,ABC,0"
The left argument to format may be a list of strings; in this case it is a series of alternating delimiters and format strings, with the last item always being a format string. Each format string is "pushed" one layer deeper into the right argument of format, and (if present), the delimiter will be used to fuse these intermediate strings, allowing a simple statement to recursively format a complex structure:
() format 11,22 # (11,22) # (identity) "%03i" format 11,22 # "011" # format the first item and discard unused(!) (list "%03i") format 11,22 # ("011","022") # format each (":","%03i") format 11,22 # "011:022" # fuse (format each) ("<%s>",":","%03i") format (list 11,22),(list 33) # ("<011:022>","<033>") # format each (fuse (format each)) ("@","<%s>",":","%03i") format (list 11,22),(list 33) # "<011:022>@<033>" # fuse (format each (fuse (format each)))
Recursive formats will "explode" tables into lists of row-lists:
t: insert alpha beta with "one" 11 "two" 22 end #+-------+------+ #| alpha | beta | #+-------+------+ #| "one" | 11 | #| "two" | 22 | #+-------+------+ (list "%u - %i") format t #("ONE - 11","TWO - 22")
If N is specified, the %s,%a and %b patterns will read up to N characters of input when parsing. Otherwise, the next character in the format string will be interpreted as a delimiter, and input characters will be read until that delimiter is encountered. (Note that if %s/%a/%b is immediately followed by a pattern, the delimiter will be %!) If %s appears at the end of the format string, it will simply read the remainder of the input string.
The %p pattern operates on a dictionary with numeric fields for year, month, day, hour, minute, second:
"%p" parse "2021-02-03T04:05:58Z" # {"year":2021,"month":2,"day":3,"hour":4,"minute":5,"second":58} "%p" format ().year:1984 # "1984-01-00T00:00:00Z"
The %n pattern can be used for progressive parsing. It also offers a way of finding the first index of a given character in a string:
data: "one,two,three" "%s,%n" parse data # ("one",4) "%s,%n" parse 4 drop data # ("two",4) "%*sA%n" parse "BBCABA" # the index of the first 'A' is 4
The %m and %z patterns can be used to disambiguate between failing to match and successfully parsing a null-equivalent value. You can also do some kinds of pattern matching:
"%i%m" parse "23" # (23,1) # successful parse "%i%m" parse "0" # (0,1) # successful parse of 0 "%i%m" parse "orange" # (0,0) # mismatch, defaulted to 0 "exe%m" parse "foo.exe","execute" # (0,1) # prefix match "exe%z" parse "execute","exe" # (0,1) # full match
The %r pattern is followed by one or more "valid" characters, the count given by D (or 1 by default). This pattern matches and collects input if and only if the input characters are within this set of valid characters. The - flag inverts this behavior, such that the pattern matches and collects only characters which are not in the valid set. If N is specified, exactly N characters must be matched; otherwise %r will accept zero or more valid characters. The %o pattern is exactly like %r, but if N is unspecified it will accept zero or one valid characters:
"%.2r01" parse "01110201" # "01110" # grab any leading number of binary digits "%*.2r01%z" parse "010","012" # (1,0) # string consists only of binary digits? "%r-%i" parse "----45" # ("----",45) # grab any prefix of minus signs "%o-" parse "A","-A","--A" # ("","-","-") # grab at most one minus sign "%*o-%i" parse "-45" # 45 # discard any sign, and read an unsigned int "#%-r\n\n%s" parse "# comment\nA" # (" comment","A") # read a Lil-style line comment until a newline
The %q and %v patterns are useful for manipulating Lil source code, parsing/matching and formatting Lil string literals and Lil variable names, respectively:
"%v[%q]%m" parse "func[\"foo\"]" # ("func","foo",1) "%q" format "a string" # "\"a string\""
The %j pattern can be used to format or parse data as JSON. When formatting JSON, Lil dictionary keys will be cast to strings, tables will be cast to their rows (as by rows), and anything other than a number, string, list, or dictionary will become a JSON null:
"%j" format (11,22) dict (33,44) # "{\"11\":33,\"22\":44}" "%j" format list 11,22 # "[11,22]" "%j" format table 11,22 # "[{\"value\":11},{\"value\":22}]" "%j" format on x do 2+x end # "null"
When parsing JSON, the value true will become the number 1, and false or null will become the number 0. This JSON parser is highly tolerant and will among other things accept non-string JSON values as dictionary keys, single-quoted strings, missing , and : delimiters, and some missing trailing delimiters. Postel's Law, baby!
"%j" parse "[true,false,null,1]" # (1,0,0,1) "%j" parse "{11:22,33:44" # {11:22,33:44} "%j" parse "{'foo':22}" # {"foo":22}
The %J pattern handles a superset of JSON that can represent all of Lil's basic datatypes and certain Interface types losslessly. Beyond the behavior of %j, %J supports dictionaries with keys of any Lil type, represents tables in <> (otherwise like a string-keyed dictionary of columns), and recognizes "Data Block" encodings for Images, Sounds, and Arrays, as described in The Decker File Format:
"%J" format ((list 11,22),(list 33,44)) dict "one","two" # "{[11,22]:\"one\",[33,44]:\"two\"}" "%J" format insert a b with 11 22 33 44 end # "<\"a\":[11,33],\"b\":[22,44]>" "%J" format image[3,4] # "%%IMG0AAMABAAAAAA="
If any patterns specify names, the result of parse will be a dictionary, and format will likewise expect a dictionary as its right argument:
"%[one]i %[two]i" parse "34 56" # {"one":34,"two":56} "%[one]i %[two]i" format ("one","two") dict 34,56 # "34 56"
The like operator offers another way of performing string matching, based on so-called glob patterns, which are a simplified subset of regular expressions. A Lil glob pattern is a string which may contain the special characters ., *, # or backticks. In a Lil glob pattern, a dot (.) matches any single character, an octothorpe (#) matches any single digit (0-9), a star (*) matches zero or more of any character, a backtick "escapes" a subsequent special character, treating it like a normal character, and any other character matches exactly itself. (You may find that SQL engines you're familiar with offer similar features with their own set of special characters.)
For example, we can use . and # to describe a "mask" of characters to accept within a broader pattern:
"Apple" like "A..le" # 1 "(555)-867-5309" like "(###)-###-####" # 1
We can use backtick escapes when we need to match against a special character:
"2*3" like "#`*#" # 1
And we can use * to perform prefix, infix, or postfix matches for fuzzy searching:
"The Best Orange" like "The*" # prefix "The Best Orange" like "*Best*" # infix "The Best Orange" like "*Orange" # suffix
If the left argument to like is a list of strings, it will produce a list of results indicating whether each string matched the pattern. This allows it to be used conveniently in queries:
t:("widget","plastic dingus","whatsit","extruded plastic dingus","dingus") t like "*dingus" # (0,1,0,1,1) select where value like "*dingus" from t # +---------------------------+ # | value | # +---------------------------+ # | "plastic dingus" | # | "extruded plastic dingus" | # | "dingus" | # +---------------------------+
If the right argument to like is a list of patterns, the result will match if any of the provided patterns match:
t:("apple pie","key lime pie","banana cream pie","apple computer") t like ("apple*","banana*") # (1,0,1,1) select where value like ("apple*","banana*") from t # +--------------------+ # | value | # +--------------------+ # | "apple pie" | # | "banana cream pie" | # | "apple computer" | # +--------------------+
Lil has a number of features influenced by "Vector-oriented" languages like APL, J, K, and Q. The essence of vector-oriented languages is thinking about manipulating entire data structures at once, rather than manipulating their elements serially.
The most obvious vector-oriented feature in Lil is conforming, in which a number of primitive operators like + and - can be applied either to single numbers or entire lists. This functionality is essential to how Lil manipulates columns within queries.
Let's start with a unary operator. Applied to a list, - "penetrates" to each list element:
-(5) # -5 -(10,-35) # (-10,35)
With a binary operator, a non-list item will "spread" and be paired with each element in a list. Given two lists, corresponding element from each list are "paired up":
100+10 # 110 100+(10,20) # (110,120) (100,200)+10 # (110,210) (100,200)+(10,20) # (110,220)
The same pattern is carried out recursively; you can operate on arbitrarily high-dimensional data this way. Sometimes you may still need an each or flip to get your operands to line up the way you want:
2*(list 1,2,3),(list 3,4) # ((2,4,6),(6,8)) 2 cross 3 # ((0,0),(1,0),(0,1),(1,1),(0,2),(1,2)) each x in 2 cross 3 (27,19)+x end # ((27,19),(28,19),(27,20),(28,20),(27,21),(28,21)) flip(27,19)+flip 2 cross 3 # ((27,19),(28,19),(27,20),(28,20),(27,21),(28,21))
Conforming is why Lil has two different equality operators: = (equals) conforms, and ~ (match) does not. Consider these cases:
22=11,22,33 # (0,1,0) 22~11,22,33 # 0 (11,22,33)=11,22,33 # (1,1,1) (11,22,33)~11,22,33 # 1 ()=11,22 # () ()~11,22 # 0
In many situations, = and ~ are equivalent. Prefer ~ when you don't need conforming behavior; it signals your intent more clearly to a reader, since it is easy to tell without context that the result will be a single number.
If two lists do not have the same length, Lil will truncate or repeat the right argument (y) to correspond to the length of the left argument (x), as if by (count x) take y:
(11,22,33,44)+(100,200) # (111,222,133,244) (11,22,33,44)+(100,200,300,400,500) # (111,222,333,444) (100,200)+(11,22,33,44) # (111,222)
This behavior can be useful for finding relative relationships in a list, comparing it to a "shifted" copy of itself:
v:1,2,2,5,3,6,7,7 (1 drop v)=v # (0,1,0,0,0,0,1) # same as previous? (1 drop v)>v # (1,0,1,0,1,1,0) # strictly increasing? (1 drop v)-v # (1,0,3,-2,3,1,0) # relative change
If you're familiar with Q or K, you might recognize this pattern as similar to the applications of the eachprior adverb.
Another application is applying a "mask" pattern to an entire list:
(11,22,33,44,55)*(0,1) # (0,22,0,44,0) # mask off odd items (11,22,33,44,55)*(1,0) # (11,0,33,0,55) # mask off even items
Yet another useful consequence of this behavior is that enlisting the right argument when conforming two lists can be used to compute a matrix giving the cartesian product: every combination of the elements of the left and right lists. Compare this with cross, which produces a flat list of pairings:
(11,22,33) + 100,200 # (111,222,133) (11,22,33) + list 100,200 # ((111,211),(122,222),(133,233)) (100,200) cross 11,22,33 # ((100,11),(200,11),(100,22),(200,22),(100,33),(200,33)) sum @ (100,200) cross 11,22,33 # (111,211,122,222,133,233)
You can also conform dictionaries. Lil takes the union of keys in the dictionaries and applies the operator between dictionary elements (or 0 if the entry is missing). As when taking the union of dictionaries with ,, conforming "prefers" the order of keys in the left argument:
x:("White","Brown","Speckled") dict 10,34,27 y:("Brown","White","Blue" ) dict 9,13,35 x+y # {"White":23,"Brown":43,"Speckled":27,"Blue":35} y+x # {"Brown":43,"White":23,"Blue":35,"Speckled":27}
If you use a conforming operator between a dictionary and non-dictionary value, the non-dictionary value will be spread to every element of the dictionary and then conformed recursively as usual. Thus, if either argument is a dictionary, the result will be a dictionary:
d: ("Alpha","Beta") dict (list 5,7),(list 3) d+100 # {"Alpha":(105,107),"Beta":103} (10,20)*d # {"Alpha":(50,140),"Beta":(30,60)}
Combining conforming operators with reducing operators like sum and raze offers many elegant and direct solutions to problems. Compare each of these approaches to counting how many times a value needle can be found in a list haystack:
needle: "apple" haystack: ("frog","apple","chicken","toadstool","apple","rice","fish") c:0 each x in haystack if x~needle c:c+1 end end c # imperative c:0 each x in haystack c:c+x~needle end c # imperative, without a conditional count extract value where value=needle from haystack # query count needle take haystack # functional (filter) sum needle=haystack # vector-oriented (spread-conforming =)
Transforming iterative code into parallel, vector-oriented algorithms makes Lil much faster and more efficient, and may result in simpler and clearer code. Consider the following two approaches for replacing values in an array that are less than 5 with the number "99":
each v in x # iterative loop if v<5 99 else v end end m:x<5 # compute a "mask" of 0 or 1 (99*m) + x*!m # multiply and add to combine masked and unmasked values
The @ operator is another powerful tool. Given a data structure on the left and a list of indices on the right it picks the element at each index. This operation can be used to replicate, filter, or permute the elements of the source:
"ABC" @ 0,0,1,2,1,2,0 # ("A","A","B","C","B","C","A") ("AB" dict 11,22) @ "BAAB" # (22,11,11,22)
If the left argument is a function, it is applied to each element of the right argument, like a more concise each loop:
on triple x do x,x,x end each x in 11,22,33 triple[x] end # ((11,11,11),(22,22,22),(33,33,33)) triple @ 11,22,33 # ((11,11,11),(22,22,22),(33,33,33))
This also works if the "left argument" is a primitive unary operator, "pushing" the operator onto each element of a list or dictionary:
first "Cherry","Olive","Orange","Lime" # "Cherry" first @ "Cherry","Olive","Orange","Lime" # ("C","O","O","L") count ("Alpha","Beta") dict (list 11,22,33),(list 44,55) # 2 count @ ("Alpha","Beta") dict (list 11,22,33),(list 44,55) # {"Alpha":3,"Beta":2}
Lil contains a number of language features and builtins specifically intended for interaction with the Decker environment.
The send statement is followed by the name of a function, and then a set of arguments in brackets, just as if the function were being called. Instead of calling the named function directly, send finds the next closest binding (or the most recently shadowed binding, if you like) for the name and calls that function. This is particularly useful for event handlers which need to "bubble" events up to a more general handler, like a card's link definition deferring to the deck's link definition.
The send statement can also be handy to override built-in functions like go[] with your own code, so that you have a chance to perform some work before, after, or instead of their default behavior. Defining these overrides in the deck's script will make them apply for events triggered from any card or widget:
on go x do # provide our own novel 'special name': if x~"Random" x:random[deck.cards] end # use 'send' to call the original definition: send go[x] end
An interface appears similar to a dictionary, but indexing or assignment through an interface may have side-effects, and the values in fields may change over time:
sys # <system> "%e" format sys.now # "2021-11-12T02:00:59Z" "%e" format sys.now # "2021-11-12T02:01:07Z" time:sys # <system> time.now # 1636682495
Interfaces cannot be defined from Lil programs- they are furnished by a host application like Decker. Consult Decker's manual for a description of the interfaces you can use in your scripts. It is also not possible to enumerate the keys of an interface; they may have infinitely many keys, populated on the fly. Thus, the in operator will always return 0 when an interface is its right argument, and the keys operator will always return ().
Interfaces can be compared with ~ and = using reference equality and concatenated into lists using , like any other datatype:
sys~sys # 1 sys=(sys,sys,123) # (1,1,0)
All interfaces will accept the "type" index, which behaves like typeof. This is handy if you ever have a list, dictionary, or table containing interfaces:
sys.type # "system" (sys,sys,sys)..type # ("system","system","system")
Accessing an invalid index will return 0, and attempting to write to an invalid (or read-only) index will return the expression to the right, like any other indexed assignment:
sys.bogus # 0 sys.bogus:123 # 123
The unary arithmetic primitives - (negation), ! (logical not), floor, cos, sin, tan, exp (the exponential function), ln (natural log), and sqrt (square root) conform, and generalize to dictionaries, lists, and numbers.
The unary aggregation primitives sum, prod, raze, min, and max take a list and collapse it into a single result as if by combining every element of the list with the binary primitives +, *, ,, & and |, respectively.
The raze of a table x will convert it into a dictionary as if by x[(keys x)[0]] dict x[(keys x)[1]].
typeof gives the name of the type of the argument; one of the strings { "number", "string", "list", "dict", "table", "function" }, or, in the case of an interface, the name of that interface type.
count gives the number of elements in a value. The count of a number is always 1.
first and last pick the first or last elements of a value. The first of a function is the function's name.
range of a number produces a list of the integers [0,x). Applied to anything else, it will produce a list of the values of its dictionary equivalent.
keys produces a list of the keys of a value's dictionary equivalent. The keys of a function are a list of its argument names.
list of anything produces a list of length 1 containing that value.
flip will transpose the elements of a list of lists- swap the x and y axes. For example, flip (list 1,2,3),(list 4,5,6) is ((1,4),(2,5),(3,6)). The result will always be rectangular: sublists that are too short will be padded with 0, and any non-list values will be spread to take up an entire column in the result. Applied to a table x, flip will produce a new table using a column named key (or the first column) as new column keys, and all other columns will become rows. The original keys will become a column named key in the resulting table. This operation can be useful for "pivoting" categorical data.
rows converts a table into a list of dictionaries, each representing a row of the table.
cols converts a table into a dictionary of uniform-length lists, each representing a column of the table.
table constructs a table. Given a list-of-dictionaries (as produced by rows) or a dictionary, it will reconstitute a table, taking the union of any dictionary keys as columns and extending any short columns as with take. A list of lists will be interpreted as a list of rows, padded to rectangularity with 0 if needed, using default column names {c0, c1... cn}. A list containing non-dictionary and non-list values will be converted into a table with a single value column.
mag computes the magnitude of a vector, or the euclidean distance between a point and the origin. If its argument is anything except a number or list of numbers it will descend recursively through the argument before computing a magnitude, similar to how other unary primitives conform. For example, mag ((list 9,0),(list 3,4),(list 0,7)) is (9,5,7). The magnitude of a single number is its absolute value.
heading computes the angle in radians from the origin toward a point. This is similar to the atan2(y,x) quandrant-disambiguating arctangent provided in many math libraries. Like mag, heading will descend recursively through any non list-of-number arguments, so it can be applied to points, lists of points, and so on.
unit computes a unit vector (with mag of 1) pointing toward an angle in radians. This operation conforms. If the angle is between 0 and 2 pi and the original mag was 1, unit is the inverse of heading.
The binary arithmetic and comparison primitives +, -, *, /, % (modulus), ^ (exponentiation/power), <, >, =, & (minimum), and | (maximum) conform, and generalize to dictionaries, lists, and numbers.
The modulus operator % takes its arguments in the opposite order of common notation- the divisor is the left argument. Thus, 5 % 3,4,5,6,7 is (3,4,0,1,2). In common usage, this order will require fewer parentheses given Lil's right-to-left precedence rule.
The <, > and = comparison operators always produce the number 1 or 0 as a result. If both operands are a number, they are compared numerically. If either is an interface, they are equal only if both are identical. Otherwise, the arguments are treated as strings and compared lexicographically.
The & and | operator calculate the minimum or maximum of their arguments. As a consequence, for the numbers 0 and 1 they are equivalent to logical "AND" and logical "OR"- thus their notation. Applied to strings, they compare values lexicographically in the same fashion as < and >.
x ~ y is the match operator. It produces the number 1 if x and y are identical values. Unlike =, match does not convert arguments or automatically "spread" to list elements; x and y must have identical types to begin with. This is particularly important if you want to e.g. check whether an item is the empty list: ()=1,2 yields (), but ()~1,2 yields 0.
x , y is the concatenation operator. It is used for joining items together into lists. If x is a dictionary, y will be converted to a dictionary, and the operator will take the union of their key-value mappings, preferring any bindings in y over x when both are present. If applied to two tables, their rows will be concatenated, with any missing columns supplied as 0. Note that using , on two strings will result in a list of two strings, whereas "" fuse x,y will concatenate the strings together.
x @ y is spread-indexing: x is indexed with each element of y. For example, (11,22,33) @ 0,1,0,1,0 is (11,22,11,22,11). With an appropriate index y, this operator can be used to reorder, duplicate, or filter elements of a list x. The expression x @ y is essentially equivalent to each v in y x[v] end, and so it can also be used as shorthand for any each loop that would otherwise simply be applying a function to each element of its source. This also works if x is a unary primitive: count @ "one","two","three","four" is (3,3,5,4).
x split y breaks a string y apart on any instance of the string x, resulting in a list of strings.
x fuse y conversely combines the strings in the list y with the string x, resulting in a string. Combining split and fuse can allow one to replace instances of one substring with another.
x like y returns a truthy result if a string x matches a glob pattern y. If y is a list of patterns, the result will be truthy if any of the patterns match. If x is a list, the pattern or patterns will be applied to each string of x and return a list of 0 or 1 values.
x dict y constructs a dictionary from a list of keys x and a list of values y.
x take y and x drop y are very general operators for filtering and reshaping data. Their behaviors depend on the type of the left and right arguments, as summarized in the table below. When x is a number, take will repeat elements from y if it isn't long enough. For non-numeric x values, take can be thought of as set intersection, while drop can be thought of as set difference.
x | y | take | drop |
|---|---|---|---|
| number | string | first x characters from y | remove first x characters from y |
| number | list | first x elements from y | remove first x elements from y |
| number | dict | first x key/value pairs from y | remove first x key/value pairs from y |
| number | table | first x rows from y | remove first x rows from y |
| negative number | string | last x characters from y | remove last x characters from y |
| negative number | list | last x elements from y | remove last x elements from y |
| negative number | dict | last x key/value pairs from y | remove last x key/value pairs from y |
| negative number | table | last x rows from y | remove last x rows from y |
| non-number/list | string | only characters in x of y | remove instances of characters in x from y |
| non-number/list | list | only elements in x of y | remove instances of elements in x from y |
| non-number/list | dict | only keys in x of y | remove keys in x from y |
| string | table | only column x of y | remove column x from y |
| string list | table | only columns in x of y | remove columns in x from y |
| number list | table | only rows x of y, in order | remove rows in x from y |
Note that if you represent sets as dictionaries, the take, drop and , operators can provide set intersection, disjunction, and union, respectively:
x:"AB" dict 0 # {"A":0,"B":0} y:"BC" dict 0 # {"B":0,"C":0} (keys x) take y # {"B":0} (intersection) (keys x) drop y # {"C":0} (disjunction) x,y # {"A":0,"B":0,"C":0} (union)
x limit y returns up to x items from y. Equivalent to if x<count y x take y else y end.
x window y divides a list or string y into slices of length mag x. If x is a negative number, the slices will overlap. For example, 3 window "ABCDEF" is ("ABC","DEF"), while -3 window "ABCDEF" is ("ABC","BCD","CDE","DEF"). If there aren't enough elements in y, a positive x may result in a final slice with fewer than x elements.
x in y returns the number 1 or 0 depending upon whether y appears in x. If y is a string, look for the string x anywhere in y. If y is a list, look for x as an element of y. If y is a dictionary, x must be a key of that dictionary. If y is a table, it must likewise be a key (column name) of that table. In all other cases, in returns 0. If x is a list, consider each element of x and return a list of 1 or 0.
x unless y returns x unless y is not the number 0; otherwise it returns y. This "null-coalescing operator" is handy for providing default values when indexing into dictionaries or lists. For example, () unless foo[x] will evaluate to the empty list if foo contains no value with the key x.
x join y performs a natural join upon the tables x and y. The intersection between column names in x and y are used as the matching key. For any combination of rows in x and y where the matching key is identical, the result will contain a row containing the union of the values in that row. The resulting table's column names will be the union of the names in x and y. Natural joins neatly capture foreign-key relationships between tables, provided that column names are consistent. If join is applied to non-table arguments, it produces a list by pairing adjacent elements from x and y. If either argument is a number, it is considered range of that argument, and otherwise it is interpreted as a list. For example, "ABC" join 3 gives (("A",0),("B",1),("C",2)).
x cross y performs a cartesian product (or cross join) of the tables x and y. Every row from x is concatenated with every row in y, producing (count x)*(count y) resulting rows. The resulting table will contain all the columns of both x and y, with any collisions in y gaining an underscore (_) suffix to disambiguate. A cross product filtered by a query can be used to perform inner joins. If cross is applied to non-table arguments, it produces a list of all pairings of elements from x and y. If either argument is a number, it is considered range of that argument, and otherwise it is interpreted as a list. Thus, 2 cross "ABC" gives ((0,"A"),(1,"A"),(0,"B"),(1,"B"),(0,"C"),(1,"C")).
x parse y breaks apart a string or list of strings y based on a format string x. x format y constructs a formatted string from a value or list of values y based on a format string y. The parse and format primitives are generally (but not strictly) inverses, given the same format string.
The following is a slightly hand-waved EBNF description of Lil's syntax. The production ALPHA is taken as any alphabetic character (upper- or lowercase), and the production NON_ESC is any ASCII character between 32 and 126 (except backslash). All whitespace is interchangeable in Lil, any amount of whitespace may appear between tokens, and line comments begin with #.
MONAD := '-'|'!'|'floor'|'cos'|'sin'|'tan'|'exp'|'ln'|'sqrt'|'count'|'first'|'last'|'sum'|'min'|'max'|
'raze'|'prod'|'range'|'keys'|'list'|'rows'|'cols'|'table'|'typeof'|'flip'|'mag'|'unit'|'heading'
DYAD := '+'|'-'|'*'|'/'|'%'|'^'|'<'|'>'|'='|'&'|'|'|','|'~'|'@'|'split'|'fuse'|'dict'|'take'|'drop'|
'in'|'join'|'cross'|'parse'|'format'|'unless'|'limit'|'like'|'window'
DIGIT := '0'|'1'|'2'|'3'|'4'|'5'|'6'|'7'|'8'|'9'
NUMBER := '-'? DIGIT+ '.'? | DIGIT* '.' DIGIT+
STRING := '"' (NON_ESC|'\\'|'\"'|'\n')* '"'
LITERAL := NUMBER | STRING | '(' ')'
NAME := (ALPHA|'_'|'?') (ALPHA|'_'|'?'|DIGIT)*
ITER := (( each' NAME* 'in' ) | 'while') EXPR* 'end'
ON := 'on' ( '...' NAME | NAME+ ) 'do' EXPR* 'end'
IF := 'if' EXPR* ('elseif' EXPR*)* ( 'else' EXPR* )? 'end'
CLAUSE := ('where' EXPR) | ('by' EXPR) | ('orderby' EXPR ('asc'|'desc'))
QUERY := ('select'|'extract'|'update')((NAME|STRING ':')?EXPR)* CLAUSE* 'from' EXPR
INSERT := 'insert' (NAME|STRING)* 'with' EXPR* (('into' EXPR) | 'end')
SEND := 'send' NAME '[' EXPR* ']'
INDEX := ('.' NAME? | '[' EXPR* ']')*
ACCESS := NAME INDEX ( ':' EXPR )?
LOCAL := 'local' NAME ':' EXPR
TERM := LITERAL | ITER | ON | IF | QUERY | INSERT | SEND | '(' EXPR ')' | MONAD EXPR | ACCESS | LOCAL
EXPR := TERM ( INDEX ( ':' EXPR )? | DYAD EXPR )?
PROGRAM := EXPR*