Over the last six months, Decker has seen continuous development. Let’s take a look at what changed between version 1.0 and 1.20, and examine the motivation behind some of those changes. They loosely fall into four categories:
I added several major features to make editing decks faster and more straightforward.
Possibly the most important enhancement was making the value of Field and Grid widgets editable directly from their properties panel. Previously, it was necessary to modify these while in “Interact” mode, which in turn required fields to be unlocked. Having the ability to paste table data directly into a Grid widget in JSON or CSV form is tremendously more discoverable than the Query dialog, and makes it possible to create new tables from scratch without the user needing to understand insert or anything else about the Lil query language.
Via the “Action…” dialog, Button widgets can be made to take a user to another card with an optional transition animation. This dialog now offers a thumbnail preview of the transition, which makes choosing the correct transition much easier. This thumbnail works for custom transitions, too:
On a complex card, many users reported having difficulty remembering widget names while flipping between card and script editing. Swapping between editing the script of multiple widgets was also a bit clumsy, as it was necessary to back out of the editor for one script, select another widget, and then re-enter the script editor every time. Enter: X-Ray Specs! In any script editor, an overlay of the current card’s widgets is a keystroke away (Control-R). This overlay identifies widgets that have scripts defined, and with Control+Click the user can switch directly to the script of any other widget or the card script.
The most frequently demanded feature for drawing was a snap-to-grid setting. Decker now has a configurable grid overlay, and most drawing tools can optionally snap to grid intersections for rapid and accurate isometric drawing. Furthermore, selections can be “bumped” in grid steps by using Shift+Cursors. All of this functionality is also available in FatBits mode.
Decker should be usable on as wide a range of devices as possible, with consistent behavior. To make things concrete, I chose three exceedingly humble and inexpensive devices to target:
The Chromebook represented a worst-case performance scenario for Web-Decker. Profiling and selective inlining of Decker’s drawing primitives brought performance from snail-in-molasses territory to slightly choppy but tolerable. Rewriting the main rendering loop to drop frames (while retaining a 60 frame-per-second logical update rate) made it even better on slow machines, especially during transition animations. Many of the performance enhancements developed for the JavaScript fork lead to corresponding improvements on the C fork.
I find the OLPC irresistably charming. Whenever feasible, I take steps to make my new projects build and run on it, which poses some challenges both for build infrastructure (I’m stuck on an elderly version of GCC, libc, and make) and library availability (Native-Decker uses SDL2, for example, whereas the Fedora 18 repositories clinging to life today only offer SDL 1.2, forcing me to build it from source), with no updates on the horizon. To be honest, though, there’s something comforting about targeting a machine that is frozen in time.
The first serious performance issue I encountered while trying to get Native-Decker running on The Charisma Machine was also the most boneheaded. Decker’s “Edit” menu changes to reflect the contents of the system clipboard, offering certain options based on whether the clipboard contains normal text or a datablock-encoded1 payload. The initial release of Decker naively called SDL_GetClipboardText() multiple times per frame, which profiling revealed to be consuming upwards of 30% of all CPU time on some machines. Yikes!
The other largest issue came down to SDL_RenderCopy() on the OLPC. Upscaling the framebuffer using SDL_RenderCopy() to fill the display proved to be excruciatingly slow, even with integral scaling factors. After experimenting with a wide variety of tweaks to renderer hints and texture formats, I found the most effective solution was to hand-write my own upscaling functions.
The tablet primarily posed issues for input. While keyboard covers, bluetooth keyboards, and even wired USB adapters can be used to furnish text input on multitouch devices, it is not a safe assumption that a user will have any of these accessories. Instead, I wrote my own soft-keyboard which engages and reflows the Decker user interface whenever it is contextually relevant. Performance is acceptable in Web-Decker, but a proper Android port at some point in the future would be even better.
Lilt is a companion application to Decker which provides access to the Lil scripting language (as well as the entire Decker document model) from the command line. With some minor tweaking and polyfilling, I managed to get Lilt building against Cosmopolitan Libc as an “Actually-Portable Executable”. APE binaries are polyglots, a single binary which can run on Windows, MacOS, and several flavors of Linux and BSD. Lil is now easier than ever to try out, and an excellent option for general-purpose cross-platform automation.
As I observed Lil programs in the wild and wrote more of my own, I identified and corrected several key weaknesses and design flaws of the language.
In an effort to keep Lil’s grammar as simple as possible, I originally implemented conditional statements as purely if ... end and if ... else ... end constructs. This is technically sufficient for writing arbitrary programs, and clever tricks like dictionaries of functions worked for some forms of complex dispatch. Nonetheless, some types of logic still resulted in exceedingly unpleasant amounts of nesting, especially when translating code from a mainstream language like Python or C. Beginners shouldn’t need to twist themselves into pretzels to avoid stacking pyramids of nested conditionals. Introducing optional elseif clauses solves the problem at the root.
A somewhat more subtle hazard arises in Lil’s variable declarations- or rather, in the fact that Lil had no such declarations at all. With the exception of formal arguments to functions, loop variables, and “magic variables” within query control structures, a Lil variable is implicitly declared at its first lexical usage. If all variable names are unique, this is no problem, but refactoring code can lead to surprises. Consider these functions:
on helper x do
r:x*3
end
on primary do
r:100
helper[200]
r
end
Both functions declare a local variable named r, but each function scope is the first lexical instance of an assignment, so they’re separate variables. So far, so good. Suppose you then innocently decided to refactor this code by tucking the helper function inside its caller:
on primary do
r:100
on helper x do
r:x*3
end
helper[200]
r
end
Uh-oh! That wasn’t a refactoring at all- both rs are now referring to the same variable, and the helper is modifying the definition in its closure. To address this kind of ambiguity, I introduced the optional local keyword, which explicitly declares the beginning of a new variable’s lifetime, shadowing higher definitions rather than referencing them:
on primary do
r:100
on helper x do
local r:x*3
end
helper[200]
r
end
Advanced programmers, lunatics, pedants: use local to control variable lifetime, making the structure of your programs clearer to a reader. Beginners: just use unique variable names! Lil has something for everyone.
Lil reserves a fair number of juicy names as keywords, which can occasionally pose problems when you need to represent a database schema which uses one of those names. The Lil query forms have been extended to resolve such collisions in two ways. A definition of a column can use a string literal to specify any name, including names which contain whitespace or would otherwise not be a valid Lil identifier:
t:insert count:11,22,33 into 0
^
'count' is a keyword, and cannot be used for a column name.
t:insert "count":11,22,33 into 0
+-------+
| count |
+-------+
| 11 |
| 22 |
| 33 |
+-------+
A reference to a column can take advantage of the magic query-scoped variable column (which, despite the singular name, is a dictionary of all active columns) to refer to any column by name:
select where column.count>20 from t
+-------+
| count |
+-------+
| 22 |
| 33 |
+-------+
select where column["count"]>20 from t
+-------+
| count |
+-------+
| 22 |
| 33 |
+-------+
Another situation that comes up in real-world code is the need to perform a multi-column sort. Originally, the orderby clause of queries could only handle comparing atomic types like strings or numbers, so a multi-column sort would imply hideous string-munging. Instead, Lil now takes a page from the K playbook and lexicographically compares nested lists in an orderby clause. We can take advantage of the “zipping” behavior of the join operator when applied to a pair of lists (or in this case, a pair of columns) to create such a key column:
t:insert a:1,0,1,1,1,0 b:"beta","gamma","alpha","delta","omega","epsilon" into 0
+---+-----------+
| a | b |
+---+-----------+
| 1 | "beta" |
| 0 | "gamma" |
| 1 | "alpha" |
| 1 | "delta" |
| 1 | "omega" |
| 0 | "epsilon" |
+---+-----------+
t.a join t.b
((1,"beta"),(0,"gamma"),(1,"alpha"),(1,"delta"),(1,"omega"),(0,"epsilon"))
select orderby a join b asc from t
+---+-----------+
| a | b |
+---+-----------+
| 0 | "epsilon" |
| 0 | "gamma" |
| 1 | "alpha" |
| 1 | "beta" |
| 1 | "delta" |
| 1 | "omega" |
+---+-----------+
Lil’s ancestral language, K, offers several ways of producing “tacit definitions” (functional compositions without named arguments), which interact neatly with the K concept of an “Adverb”: a syntactic notation for higher-order functions that capture abstract iteration. The most iconic example is the adverb “over” (/), which takes a verb or composition to the left and applies it sequentially between every value of a list on the right:
+/1 2 3
6
(1+2)+3
6
Parsing K-style adverbs poses a moderate amount of grammatical complexity, and they are far less useful without tacit composition or concise syntax for anonymous functions. Lil compromises by providing primitives representing a handful of particularly useful verb-adverb compositions in K, leaving the remaining cases as “Stinking Loops”. For example:
| K | Lil |
|---|---|
+/x |
sum x |
,/x |
raze x |
x,'y |
x join y |
,/x,\:/:y |
x cross y |
x,/:y |
each v in y x,v end |
The Lil @ operator is the closest it comes to adverbs. It can extract multiple items from a source data structure by index, thereby replicating, filtering, or re-ordering the the elements of the original structure:
"ABC" @ 0,0,1,2,1,2,0
("A","A","B","C","B","C","A")
("AB" dict 11,22) @ "BAAB"
(22,11,11,22)
Since Lil unifies indexing and function application (much like K), this also works if the source is a unary function:
(on triple x do x,x,x end) @ 11,22,33
((11,11,11),(22,22,22),(33,33,33))
triple @ 11,22
((11,11,11),(22,22,22))
A recent generalization allows this to work for primitive unary operators to the left as well, slowly gnawing away at situations where a loop is required:
first "Cherry","Olive","Orange","Lime"
"Cherry"
first @ "Cherry","Olive","Orange","Lime"
("C","O","O","L")
I also added features which extend what Decker is capable of creating.
One user-suggested feature was an option for making Canvas widgets “draggable”. In the simplest cases this could be used to build board games or paper-doll toys without necessarily writing any code. There is an interactive tutorial deck available describing more sophisticated uses.
To truly be considered a multimedia environment, Decker needs flexibile facilities for working with sound as well as visuals. The loop event provides a way to sequence or cycle audio clips, whether this is to provide background ambience for cards or to allow users to build drumkits and synthesizers. The details of the loop event are described in their own interactive tutorial deck as well.
Decker provides several ways to set up animated elements on a card, supporting both synchronous scripts (which block most forms of user input while executing) and asynchronous event-driven scripts. Initially, asynchronous animation required a user to set up a centralized event-pump on a card, like so:
on view do
do_something[] # update the frame
go[card] # re-trigger the view[] event
end
To make this more flexible, I later introduced an animated attribute that can be set on any widget. Widgets with this attribute are automatically fed view events so long as they are visible. Animated Canvas widgets are the most obvious application, but the same mechanism allows users to make any widget “poll” the deck and automatically update themselves, instead of waiting for a change event. Consider an animated field widget that computes its value by consulting another widget:
on view do
me.text: price.text*(1+taxes.text)
end
Decker and Lilt originally shipped with functions for to exporting a sequence of images as an animated GIF. This feature was extended to include decoding the frames of a GIF animation, opening up interesting possibilities for importing video clips into Decker. In the process of developing this feature I learned that GIF frame disposal methods are quite a bit stranger semantically than my initial understanding of the GIF89a specification.
Decker’s initial release included a very simple mechanism for defining a “Parent” card with layout and styling which could be inherited by any number of “Child” cards that varied only in their form data. In practice, card inheritance was too limited and rigid for most applications, so I went back to the drawing board and developed a much more flexible and powerful mechanism to replace it: Contraptions. Contraptions are a new widget type that is composed of other widgets:
From the inside, defining a contraption is like making a card. From the outside, instances of a contraption definition act like widgets. It is possible to create useful contraptions without writing any code: for example, creating slide templates for a presentation or decorative border pieces to be used throughout a deck. Anywhere visual or semantic elements are repeated in a Deck, there’s a good chance Contraptions can help.
You can transfer contraptions between decks by simply copying and pasting them using the system clipboard. If an advanced user shares a useful contraption, a less sophisticated user can still take advantage of it in their own decks or even modify it to suit their purposes without needing to understand all the details of its implementation. The Contraption Bazaar thread on the Decker community forum includes a number of examples that you can try for yourself!
Decker today is more usable, more powerful, and far more portable than it was at release, and Lil has continued to mature into a more expressive and delightful scripting language. When Decker was in its infancy, it seemed like every time I tackled a new project I immediately discovered new bugs and design flaws. While I won’t pretend it’s perfect today, I have much more confidence in it. I am enjoying the transition from primarily being a builder of a tool to being one of its users, and watching a small but vibrant community begin to form around it.
If Decker strikes your fancy, perhaps you’ll consider joining us for Decker Fantasy Camp 2023, a month-long online gathering all about creating things with Decker! Your creations and feedback can inspire others and help guide future advancements to the platform.
If you have questions, comments, or anything you’d like to share, feel free to contact me directly or participate in the Decker community forum.
Decker uses a custom text encoding for representing non–textual data in the clipboard, described in detail in the Decker File Format. ↩︎