% language=us \usemodule[present-boring,abbreviations-logos] \startdocument [title={LUAMETATEX}, banner={where do we stand}, location={context\enspace {\bf 2020}\enspace meeting}] \starttitle[title=When it started] \startitemize \startitem About three years ago the idea came up to go this route. \stopitem \startitem At the 2018 meeting it was first mentioned and those present were okay with it. \stopitem \startitem Early 2019 the first beta release took place. \stopitem \startitem At the 2019 meeting the first more official version was presented. \stopitem \startitem Around the 2020 meeting we have more or less arrived at what I had in mind. \stopitem \startitem At the 2021 meeting I expect the code to be stable and repositories to be set up. \stopitem \startitem At the 2022 meeting we can make the official transition from \MKIV\ to \LMTX. \stopitem \startitem Some new options are only enabled in my local \type {cont-exp.tex} file. \stopitem \startitem Knowing that Wolfgang keeps an eye on all those changes makes me more daring. \stopitem \startitem We aim to get less (but more efficient) macro code that on the average looks better. \stopitem \stopitemize \stoptitle \starttitle[title=Why it started] \startitemize \startitem There was an increasing pressure for a stable \LUATEX. \stopitem \startitem There should be no more changes to the interfaces, no more extensions. \stopitem \startitem One can run into interesting comments on the web (as usual), like \startitemize[packed] \startitem The \LUATEX\ program has \quote {many bugs}. \stopitem \startitem The \LUATEX\ manual is bad. \stopitem \startitem The \LUATEX\ program is too slow to be useful. \stopitem \startitem The \LUATEX\ program will never end up in distributions. \stopitem \startitem The \LUATEX\ project is funded and developed in a commercial setting. \stopitem \stopitemize \stopitem \startitem I won't comment on how I read these (demotivating) comments because \unknown \stopitem \startitem \unknown\ it anyway often says more about the writer (attitudes) than about \LUATEX. \stopitem \startitem I also looks like (non \CONTEXT) users are charmed by \LUATEX, and the more they code, the more we need to freeze. \stopitem \startitem So, hopefully, the \LUAMETATEX\ development does not interfere badly with developments outside the \CONTEXT\ community. \stopitem \stopitemize \stoptitle \starttitle[title=The development] The summary on the next pages is partial. More can be found in articles and documents that come with the distribution. \startitemize \startitem \LUATEX\ started out as \CWEB\ code \unknown\ that eventually became just \CCODE\ \unknown\ which in \LUAMETATEX\ has been detached from the (complex) infrastructure. \stopitem \startitem The basic idea is to only keep the core of \TEX, but for instance font loading, file handling and the backend are gone. \stopitem \startitem As a consequence the code has been reorganized (shuffled around). \stopitem \startitem I experimented a lot without bothering about usage elsewhere and I like the result so far. \stopitem \startitem The \CONTEXT\ distribution will at some point ship with the source. \stopitem \stopitemize \starttitle[title=File handling] \startitemize \startitem All file handling goes via \LUA, also read and write related primitives. \stopitem \startitem The same is true for terminal (console) handling. \stopitem \startitem Part of that (the writing) was actually kind of extension code in \TEX\ and partly a system dependency. \stopitem \startitem The \ETEX\ pseudo file \type {\scantokens} primitive uses the same mechanism as \LUA\ does. \stopitem \stopitemize \stoptitle \starttitle[title=The macro machinery] \startitemize \startitem There are extensions to the way macro arguments are handled (less clumsy macros). \stopitem \startitem There are extra if tests (makes for nicer macros). \stopitem \startitem Else branches in conditions can be collapsed using \type {\orelse} and \type {\orunless} which gives cleaner low level code. \stopitem \startitem Tracing gives more detail about node properties and also shows attributes. \stopitem \startitem Some new data carriers have been added that can be played with from \LUA . \stopitem \startitem Macros can efficiently be frozen (new) and protected (redone) and the concepts \quote {long} and \type {outer} are gone. \footnote {In \CONTEXT\ macros were always \type {\long} and never \type {\outer}. Most commands were unexpandable (also in \MKII, pre \ETEX). So, users won't notice this.} \stopitem \startitem Saving and restoring is somewhat more efficient (partly a side effect of wider memory). \stopitem \stopitemize \stoptitle \starttitle[title=Language] \startitemize \startitem Language control settings now use less parameters but bit sets instead. \stopitem \startitem Only basic parameters are stored in the format file now. \stopitem \startitem There are all kind of small improvements. \stopitem \stopitemize \stoptitle \starttitle[title=Typesetting] \startitemize \startitem Attributes (the lists and states) are implemented more efficiently. \stopitem \startitem The paragraph state is stored with the paragraph. \stopitem \startitem Paragraphs can be normalized and options are now set with bit sets. \stopitem \startitem Boxes carry orientation related information (offsets, rotation, etc). \stopitem \startitem Some nodes carry more information. \stopitem \startitem Directions are mostly gone (it's up to the backend). \stopitem \startitem Migrated content is optionally kept with boxes. \stopitem \stopitemize \stoptitle \starttitle[title=Math] \startitemize \startitem Some math concepts have been extended (like prescripts and some more control over styles). \stopitem \startitem There are plenty of new control details. \stopitem \startitem The math parameter settings obey grouping in a math list. \stopitem \startitem We can have math in discretionaries in text and more advanced discretionaries in math as well. \stopitem \stopitemize \stoptitle \starttitle[title=Fonts] \startitemize \startitem Font specification information no longer uses the string pool (which saves a lot). \stopitem \startitem Of course we still have the basic font handler. \stopitem \startitem We only store what is needed for traditional \TEX\ font handling. \stopitem \startitem Virtual fonts are even more virtual (also a backend thing) so we can have more features. \stopitem \stopitemize \stoptitle \starttitle[title=The code] \startitemize \startitem Artifacts from \PASCAL\ and \CWEB\ have been removed. \stopitem \startitem Languages, fonts, marks etc are no longer \quote {register} based. \stopitem \startitem The token interface is more abstract and no longer presents strange numbers. \stopitem \startitem Some internals have been reconstructed because of cleaner \LUA\ interfacing. \stopitem \startitem A side effect of this is better abstraction of the equivalent ranges. \stopitem \startitem The code has been made more abstract (and looks easier in e.g. Visual Studio). \stopitem \startitem The compile farm is used to check if compilation works out of the box. \stopitem \startitem Compilation is fast and easy, otherwise this project was not possible. \stopitem \startitem Readability of the code is constantly improved (the usual: has to look okay in my editor). \stopitem \startitem The code has been made mostly independent of specific operating system needs. \stopitem \startitem Wide characters are dealt with in Windows interfaces. \stopitem \stopitemize \stoptitle \starttitle[title=Libraries] \startitemize \startitem We really want to stay lean and mean: the engine is also a \LUA\ engine. \stopitem \startitem All code is included, a few libraries are used, but these are small, old and stable. \stopitem \startitem In addition some helper libraries are made (including pplib by Pawel). \stopitem \startitem What we ship is what you get: \CONTEXT\ will not depend on more than that. \stopitem \startitem If something is updated (at all) the differences are checked first. \stopitem \stopitemize \stoptitle \starttitle[title=The \LUA\ engine] \startitemize \startitem We use the latest (even alpha) \LUA\ (5.4) because \LUAMETATEX\ is a good test. \stopitem \startitem There is no support for \LUAJIT\ and the \FFI\ interface is gone. \stopitem \startitem There is a limited set of libraries that we support but no code is (and will be) included. \stopitem \startitem There are less callbacks (because we only have a frontend). \stopitem \startitem There are more token scanners and some options have been added. \stopitem \stopitemize \stoptitle \starttitle[title=Efficiency] \startitemize \startitem We benefit some more from the wider memory words (some constructs could go). \stopitem \startitem The format file is smaller and not longer compressed. \stopitem \startitem Memory management is now mostly dynamic and usage is much lower. \stopitem \startitem There are more statistics (also as side effect of memory management). \stopitem \startitem Dumping the format has been made a bit more robust and is faster. \stopitem \startitem The core engine performs a bit better (machines don't get that much faster). \stopitem \startitem We want to be prepared for future architectures. \stopitem \startitem We manage to keep the binary way below 3 MB. \stopitem \startitem The lot runs quite well on e.g.\ a Raspberry Pi 4. \stopitem \stopitemize \starttitle[title=Upgraded \METAPOST] \startitemize \startitem All (eight bit) font stuff has been stripped from the \METAPOST\ library. \stopitem \startitem The library no longer has a \POSTSCRIPT\ backend. \stopitem \startitem The library provides scanners that make extensions possible. \stopitem \startitem All file \IO\ goes via \LUA. \stopitem \startitem There are a few additions like pre|/|postscripts for clip and bounding boxes. \stopitem \stopitemize \stoptitle \starttitle[title=Praise for the users] \startitemize \startitem Much has been done and I probably forget to mention a lot. \stopitem \startitem The number of bugs is relative small compared to what gets changed and added. \stopitem \startitem The test suite gets ran very often, also to check if performance is okay. \stopitem \startitem I could only do this because the \CONTEXT\ users are so tolerant. \stopitem \startitem Some seem to constantly check for updates so they help with fast testing. \stopitem \startitem The \CONTEXT\ code base gets stepwise adapted (split files) which again forces users to test. \stopitem \startitem It takes a lot of time because we take small steps in order not to mess up. \stopitem \startitem I would not do it without the positive attribute of the \CONTEXT\ users. \stopitem \startitem It's all about motivation and I thank the \CONTEXT\ users for providing this friendly and non|-|competitive bubble! \stopitem \stopitemize \starttitle[title=Todo] \startitemize \startitem Maybe add some more sanity checks in order to catch errors intruded by callbacks. Maybe add some more tracing too. \stopitem \startitem Explore variants, like having registers in dedicated eqtb tables so that we can allocate them dynamically (mostly for the fun of doing it). \stopitem \startitem Add some more documentation (read: addition cq.\ remarks about where the original documentation no longer applies, but we have years for doing that). \stopitem \startitem Update the manual (which is done occasionally in batch based on print|-|outs; there is no real need to hurry because we still experiment). \stopitem \startitem Apply some of the new stuff in \LMTX. Take up some challenges. \stopitem \startitem Wrap up new functionality (once it's stable) in articles and other documents. \stopitem \stopitemize \stoptitle \starttitle[title=And \LUATEX ?] \startitemize \startitem Of course \LUATEX\ will be maintained! After all, \MKIV\ needs it and it serves as reference for the front|-|end rendering and back|-|end generation when we're messing with \LUAMETATEX. \stopitem \startitem It is used by \LATEX\ and there are now also plain inspired packages. Because there are spin|-|offs (\LATEX\ has settled on a version with built|-|in font processing) we cannot change much. \stopitem \startitem And \LUATEX\ being nicely integrated into \TEXLIVE\ is another argument for not touching it too much. \stopitem \startitem I have no clue of \LUATEX\ usage but that fact alone already makes an argument for being even more careful. It's bad advertisement for \TEX\ when users who use the low level interfaces get confronted with conceptual changes. \stopitem \startitem So in the end not much will be back ported to \LUATEX: at some point the code base became too different and it's the price paid for the stability demand. That way we cannot introduce new bugs either. It also doesn't pay of. \stopitem \startitem But, a few non|-|intrusive things might actually trickle into it in due time, also out of self interest: it might help to share code between \MKIV\ and \LMTX. \stopitem \stopitemize \stoptitle \stopdocument