A year has passed since my last summary of Decker’s progress. Let’s have a look at what changed between version 1.20 and 1.43. As before, changes are grouped into four subjective categories:
Decker’s user interface has not changed radically, but it has accumulated many useful and important features that save time, improve results, and avoid confusion.
While Decker’s user interface is primarily black-and-white, it has always internally been an 8-bit paletted “color” application, with the lowest 32 color IDs reserved for monochrome patterns, followed by a 16-color palette based on early MacOS. Early versions of Decker treated colors as a sort of “easter egg” available via scripting APIs, but users clamored for more direct access. There is now a “Color Mode” for pattern selection, which modifies the toolbar and modal pattern picker. An additional flourish is “underpaint” mode (toggled via the menu or by pressing u), which treats black pixels as opaque, making it easy to apply color to dithered graphics:
With Color more prominent, I also slightly modified the semantics of the patterns interface such that customizations to the color palette are now serialized as part of the deck, rather than only changes to the 1-bit pattern palette.
While editing, it is often the case that a user finds themselves looking at multiple cards which are visually very similar. Copying and pasting cards in particular can be remarkably disorienting. To aid in navigation, the top-right corner of the menu bar now displays the current card name while using any editing tools, allowing users to disambiguate their location at a glance. The same feature also works for identifying contraption prototypes.
To facilitate “upgrading” modules or contraption prototypes over the lifespan of complex projects, both resources have gained a .version property which is displayed in the Font/DA Mover, and importing either resource will now clearly indicate whether the item you have selected is a more recent revision than resources you already have. The “Contraption Prototypes” dialog has also been reworked to offer “Delete” and “Clone” (for starting a new Prototype based on an existing reference):
Originally, the toolbars of native-decker were available only in fullscreen mode. Many users requested that they also be available in windowed mode. The design of the toolbars has been modified slightly so that they neatly match the vertical dimensions of Decker’s default deck size:
Making tidy user interfaces frequently involves lining up the bounding boxes of widgets. Decker now offers alignment guides when repositioning objects on a card or prototype. This feature combines nicely with snap-to-grid and “nudging” items a single pixel or a grid step with the cursor keys:
Since its initial release, Decker has been able to import photos and dither them to a 1-bit palette. For some applications (including sprites intended for resizing), line-art composed of solid regions of drawing patterns are more desirable. To facilitate this without introducing an elaborate layering system, Decker now offers a “tracing” mode that makes the entire window semi-transparent. Other applications can be positioned underneath Decker to provide drawing references. Pressing the y key while using drawing tools is a quick shortcut for toggling tracing mode:
Most of the recent improvements to the portability and accessibility of the Decker ecosystem revolve around Lil.
The Lil Playground offers a browser-based environment for tinkering with Lil scripts that has syntax highlighting via a Lil CodeMirror plugin as well as simple pastebin functionality. This environment is also supplemented with a new fast-paced tutorial, Learn Lil in 10 Minutes.
Decker’s documentation was previously built using multimarkdown. For greater flexibility and the ability to provide Lil syntax highlighting, Decker’s documentation is now processed by lildoc, which is, naturally, implemented in Lil. As is often the case when trailblazing new real-world applications, writing lildoc exposed a number of string-handling performance issues, and design oversights in writexml[]. Having corrected these will benefit many other projects in the future. The same syntax highlighting functionality is available as a standalone tool: cohostify.
I also wrote an alternative Lil implementation, Lila, in the AWK scripting language. AWK is extremely portable by virtue of being a standard component of the POSIX specification, and Lila in turn can run on virtually any POSIX environment. While less efficient than the C or JS reference implementations, Lila can be surprisingly performant, and may offer intriguing potential for “bootstrapping” other parts of the Decker ecosystem in the future.
I’m also pleased to observe that a number of volunteers have begun making lilt and Decker available as packages for BSD and Linux distros. I am extremely thankful for these efforts to make Decker easier to install and use on more platforms! If anyone is interested in creating a package for their favorite distribution, please feel free to reach out if you run into any problems.
The best way I know to evaluate the design of a programming language is to use it to build a wide range of interesting programs, discover idioms, and identify problems that are awkward to solve. Most of the additions I’ve made to Lil over the past year are a direct response to problems I encountered as a user of the language.
The formatting pattern language used by the parse and format operators has gained several generalizations. Named fields (enclosed in [] after a %) cause these operators to emit or consume a dictionary. This makes extracting fields from a complex chunk of formatted text much less error-prone and aids in formatting data into templates which may repeat fields:
"The %[fruit]s is clearly %[state]s." format ("state","fruit") dict ("ripe","orange") # "The orange is clearly ripe."
"(%[x]i,%[y]i)" parse "(11,22)" # {"x":11,"y":22}
Since Lil does not have conventional literals for lists or dictionaries, it can be awkward to describe complex nested structures like lookup tables. Using JSON strings and parsing with %j is a convenient alternative. To reduce the need for escaped double-quotes (which tend to harm legibility), the %j format pattern has been made more permissive when parsing, permitting the use of single-quotes instead of double-quotes around string literals:
"%j" parse "{'one':11,'two':22,'three':33}" # {"one":11,"two":22,"three":33}
Finally, two format patterns were introduced to aid in tokenizing and emitting well-formed Lil code, to permit certain kinds of metaprogramming in concert with eval[]: %v (“variable”, which recognizes Lil identifiers) and %q (“quoted”, which recognizes Lil string literals). These are used in Decker as part of the mechanism which decomposes button scripts created by the “Action…” wizard for editing, as well as in lildoc, producing the syntax highlighting you see here and elsewhere in Decker’s documentation.
Lil is mostly a very simple and straightforward language, with the “table” datatype and Lil’s SQL-like query language accounting for much of its conceptual complexity budget. Based on my experience using these features, I decided to perform several breaking overhauls of their design.
Before you can perform queries, you need tables. The original insert syntax was column-wise, just like select and update, which was severely inconvenient when declaring large tables, defeating the purpose of having the form in the first place. The revised design is row-wise. By enclosing data in with ... end delimiters, given a known set of columns, it is possible to describe tables in an extremely clean manner with a minimal number of tokens. Having excellent first-class table literals elevates the value of the datatype and all their adjacent features:
fruits:insert name sales with "Cherry" 22 "Lime" 5 "Durian" 97 "Banana" 12 "Apple" 1 end
I also overhauled Lil’s query syntax itself. Originally, where, by, and orderby clauses were required to appear in a rigid order, as they do in SQL and QSQL. I realized that generalizing queries to permit these clauses to appear in any order (and even repeat) makes them much more flexible, simpler in implementation, and easier to understand: they can now be viewed as a right-to-left pipeline of operators which consume a column and a table (or grouped subtables), and yield a table (or grouped subtables). This type of generalization is particularly useful for performing convoluted sorts, as illustrated in Perl Weekly Challenge problems 233.2 and 268.2 below:
# sort x in increasing order based on the frequency of the values. # if multiple values have the same frequency, sort them in decreasing order: on f x do extract v orderby c asc orderby v desc from select v:value c:count value by value from x end
# sort x ascending, with each pair descending: on f x do extract value orderby value desc by floor gindex/2 orderby value asc from x end
I made several more additions that help tables interoperate with other datatypes. The raze primitive was extended to collapse tables into dictionaries by taking the first column as keys and the second column as values; this allows insert statements or the output of any query to serve as a convenient new way to form dictionaries:
raze insert name weight with "Pippi" 4.8 "Galena" 4.2 end
# {"Pippi":4.8,"Galena":4.2}
Amending assignments, which were previously allowed for strings, lists, and dictionaries, now also apply to tables. In some situations this is much more concise than an equivalent update, and can also work in situations where the target column varies.
t:insert fruit price amount with "apple" 1.00 1 "cherry" 0.35 15 "banana" 0.75 2 "durian" 2.99 5 "elderberry" 0.92 1 end
t.fruit[2]:"golfball" # replace cell t.price:t.price*2 # replace entire column
Before, functions were entirely opaque values. In several places (including the transition[] and brush[] built-in functions) I found it was convenient to take advantage of the fact that Lil functions are always named and use their names as identity elements for dictionaries. To allow user-defined code to do the same thing, the first operator applied to a function now retrieves its name. The keys operator applied to a function retrieves its argument list, which in turn allows Lil code to perform dispatch based on the valence of a function:
on plus a b do a+b end
first plus # "plus" keys plus # ("a","b")
Lil lacks an equivalent to K’s “Adverbs”, and compensates by providing several “reducer” primitives to handle the most common verb-adverb compositions. While it is useful much less often than sum, min, max or raze, I decided to introduce prod (multiplicative product) to round out the collection of reducers.
The @ primitive’s semantics were simplified slightly. Previously, applying a function f with @ was equivalent to using an each loop that provided the value, key, and index of each element of the right argument to f. In practice, this generality was rarely if ever useful, and it actively interfered with using this shorthand notation on functions that take second or third optional arguments like random[] or rtext.make[]. The revised semantics of @ are less suprising: it simply provides each value to f:
f @ x # the @ expression each v k i in x f[v k i] end # old semantics each v in x f[v ] end # new semantics
Most of Lil’s arithmetic operators conform over lists, providing efficient implicit iteration. After a great deal of consideration I arrived at a simple rule for generalizing conforming to work with dictionaries as well, in a manner that is consistent with the existing behavior of the language: When two dictionaries conform, the result will be a dictionary with the union of their keys, and any missing elements of each dictionary are treated as if they were 0. When dictionaries are conformed with non-dictionary values, the value will be “spread” to every element of the dictionary:
x:("White","Brown","Speckled")dict 10,34,27 y:("White","Speckled","Brown")dict 5, 3, 8 z:("Brown","White","Blue" )dict 9,13,35
x+y # {"White":15,"Brown":42,"Speckled":30} x+z # {"White":23,"Brown":43,"Speckled":27,"Blue":35} x+100 # {"White":110,"Brown":134,"Speckled":127}
Finally, I introduced the new like operator, which performs “glob pattern” matching similarly to SQL’s LIKE or K’s _sm. Having like readily at hand makes it easier to translate a variety of SQL examples with string columns into equivalent Lil. I also quickly found the operator is very useful within Decker for identifying widgets that have semantically-relevant name prefixes or suffixes:
extract where value..name like "box*" from card.widgets
Decker’s scripting APIs have gained a wide variety of generalizations and additions.
The image interface has been considerably expanded, in an effort to minimize the number of operations that must be performed pixel-by-pixel with Lil, which tends to be very slow. The .scale[], .translate[], and .rotate[] functions modify images in-place, with .rotate[] using a quite interesting rotate-by-shearing algorithm which is area-preserving and reversible. The .merge[] routine has been generalized to permit merging together images by a variety of arithmetic operations; you can see this put to good use in The Life of Lil. Finally, .hist calculates a histogram of the pixels in an image; this offers very efficient ways to extract an image’s palette or identify blank areas at the edges of images to trim them.
A simple but impactful addition is the .toggle[] function available on all widget interfaces, which provides a more concise and flexible way of setting their .show attribute, inspired in part by jQuery’s .toggle():
# toggle solid <-> none: x.show:if x.show~"none" "solid" else "none" end # the old way x.toggle[] # the new way
# toggle transparent <-> none: x.show:if x.show~"none" "transparent" else "none" end # the old way x.toggle["transparent"] # the new way
# set visibility to transparent or none based on 'p': x.show:if p "transparent" else "none" end # the old way x.toggle["transparent" p] # the new way
The eval[] built-in function normally executes fragments of Lil code in total isolation, with no access to the surrounding scope or built-in functions; only primitive operators. The second argument can provide a dictionary binding variable names to values within the evaluated code, “opting-in” to any state or functions you wish to make available to the code fragment. Sometimes, however, it’s much more convenient to simply trust that code fragments are well-behaved and grant them access to the entire local environment. If the new third argument to eval[] is truthy, code will execute within the caller’s scope, and even be able to write to external local variables, more like the spicy, dangerous eval() of JavaScript:
a:23 b:45
eval["a:b+c" ("c" dict 1000) 1]
show[a] # 1045
The new panic[] built-in function is a useful debugging tool: it will halt any executing script dead in its tracks and open the Listener, binding any provided argument as _ (the name for the return value of the previous Listener expression). When the opposite is required- minimally invasive logging- the app.show[] and app.print[] functions permit programs to log information to the Decker application’s POSIX stdout:
![It's Pippi, time to `panic[]`!](unionstate2-figures/panic.png)
panic[]!As a consequence of needing to maintain parity between Native-Decker and Web-Decker, Decker is fairly tightly sandboxed. Some users found this limiting, so I struck a careful compromise: the Danger Zone is an interface which contains most of the features from lilt that are normally absent from Decker: filesystem traversal with path[] and dir[], raw file IO with read[] and write[], access to environment variables, and invoking external commands with shell[]. Enabling the Danger Zone requires rebuilding Decker from source, which intentionally limits access to the same kinds of enthusiastic tinkerers who would have uses for it in the first place. Having this standardized API dramatically expands the ways Decker can talk to the outside world, should you choose to opt-in:
danger interface to fetch CSV data over the web via curlTo summarize other miscellaneous API improvements:
writexml[] built-in function now treats Array interfaces as “transparent”; their text contents will not be escaped or otherwise translated into entities. Wrapping strings in this manner makes it possible to embed non-standard entities or pre-formatted fragments, which is particularly useful when generating HTML. To enable similar HTML-oriented applications, outputting whitespace and indentation is now opt-in rather than mandatory.deck and card interfaces now expose .copy[] and .paste[] functions for bulk manipulation of cards and widgets. Previously, lilt offered a limited version of this functionality, but it has been generalized and made available to Decker as well. In practice, being able to easily duplicate widgets and cards on the fly is enormously useful. For some practical examples, see Decker Sokoban.pointer interface now exposes .down and .up flags to readily identify the “rising and falling edge” of click events. Taking advantage of these makes simulating clickable buttons and other UI components within synchronous scripts much more reliable, by avoiding the possibility that short clicks start and finish before a script has a chance to observe changes to .held. This is used to great effect in the Dialogizer module.canvas interface now exposes .segment[] and .textsize[], for drawing 9-slice images (as in Contraption backgrounds) and measuring the dimensions of rich text. Both are examples of surfacing functionality Decker already needed internally, making them cheap additions proportional to their usefulness.rtext interface now exposes .replace[] and .split[], efficient and highly flexible implementations of common string manipulation operations that would be slow, tedious, and error-prone to re-implement in Lil wherever needed.bits interface is a new addition, providing conforming implementations of bitwise XOR, AND, and OR operations. These are very handy when manipulating binary data as well as implementing emulators, assemblers, or low-level virtual machines.app interface is another new addition, providing ways for locked decks to behave more like normal applications, including modifying fullscreen mode, saving the deck, and closing the deck under scripted control..volatile, which makes their state- such as a field’s text contents or a canvas' image- “purgeable” and not saved along with the deck. This can help make saved decks smaller, aid the use of revision control tools by minimizing spurious diffs, and in some cases make it easier to design games and other tools that “reset” after a session.I have also added some entirely new features that open up new possibilities for building user interfaces in Decker:
Button widgets now have a .shortcut attribute, which can be set to any single alphanumeric character or a space. Pressing the corresponding key on a keyboard has the same effect as clicking the button. This provides a nice compromise between allowing users to easily create user interfaces which can be navigated quickly via the keyboard (especially games) and ensuring that UIs remain accessible on touch-based devices that don’t have a physical keyboard:
Decker comes with a variety of shapes and sizes of brush that can be used manually with drawing tools or programmatically on a Canvas widget. The new brush[] built-in function makes it possible to define new brushes, which can either be a simple static shape or a “functional brush” that executes a Lil function as it is drawn, making velocity-sensitive or randomized brushes possible. Just like transitions, custom brushes can be packaged as Lil modules that can be installed and used without requiring users to do any programming or additional configuration.
Previously, Grid widgets allowed users to select a row. With the new “Select by Cell” option, users can indicate a single cell, and navigate vertically or horizontally via the keyboard, making grids a little more spreadsheet-like. The .row, .col, .cell and .cellvalue attributes expose information about the selection, and the new changecell[] event makes it possible to intercept cell edits, applying custom formatting rules or producing side-effects like logging undo history:
Finally, I have written several Lil modules to extend Decker’s built-in functionality and make it easier to build complex programs. Of particular note are Dialogizer, which provides customizable modal dialog boxes, and Puppeteer, which orchestrates animated character “puppets”. Working in concert, these modules make Decker suitable for writing visual novels and other kinds of interactive experiences:
I have already seen these modules used to great effect in a number of creative projects!
None of this year’s changes are as far-reaching as the introduction of Contraptions last year, but they add up to remarkable and impactful improvements in flexibility and usability, as evidenced in the increasingly ambitious projects tackled by the user community.
If you’re looking for an excuse to tinker with Decker, or at least see what other users have been up to, you might be pleased to hear that this July will be another Decker Fantasy-Camp- why not check it out? If you’re reading this from the future and have already missed that boat, I intend to continue holding “game jams” for Decker twice a year in December and July.
If you have questions, comments, or anything you’d like to share, feel free to contact me directly or participate in the Decker community forum.