1
2
3\environment luametatexstyle
4
5\startdocument[title=Engines]
6
7\startsection[title={Introduction}]
8
9There are good reasons why we started the \LUATEX\ and later \LUAMETATEX\
10projects. Here I will go into some of them. It is just short wrap up of how it
11started, how other engines influenced the process and how we see usage. There are
12plenty of documents out there that go into more detail. The main objective of
13this section is to put documentation into perspective.
14
15\stopsection
16
17\startsection[title={How it started}]
18
19When we started with \CONTEXT, hardware was rather limited compared to what we
20have today. A personal computer had some 640kB memory, possibly bumped to 1MB
21with help from a memory extender. This put some restrictions to how macro
22packages could be defined, also because that memory had to be shared with the
23baseline operating system. However, over time memory and runtime became less of
24an issue and the \TEX\ engine could be configured to use whatever was available.
25Extending the program other than increasing the available memory became more
26feasible.
27
28As with any program, there is always something to wish for which is why the
29\ETEX\ variant came into view. Before those extensions could be used, \PDFTEX\
30showed up. That variant simplified the \quote {\TEX\ plus separate backend
31driver} model to a onestep process. Eventually \ETEX\ was merged into \PDFTEX,
32and that became the de facto standard engine. There was never a follow up on
33\ETEX, and more drastic deviations like \OMEGA\ were never ready for production.
34At some point \XETEX\ came around but that was mostly a font specific extension.
35We were kind of stuck with a wishlist that never would be fulfilled but we
36occasionally pondered a follow up. We drafted an extended \ETEX\ proposal, played
37with some features related to \PDF, improved a few things but that was it.
38
39Having some experiences with \LUA\ as extension language in \SCITE, I wondered
40what something like that would bring to \TEX\ and after discussing this with
41Hartmut he made variant of \PDFTEX\ that has some basic interfaces: we could
42access properties of registers and print something to \TEX\ as if it came from
43file. As is common with some variant, a new name was coined and \LUATEX\ came
44into existence. Were talking 2005.
45
46Because Idris wanted to typeset high quality scholar manuscripts mixing Arabic
47and Latin we discussed how to do that in \CONTEXT\ and his experiences with
48\OMEGA\ were such that alternatives had to be considered: the Oriental \TEX\
49project was started and \LUATEX\ was the starting point. Taco merged some parts
50of \ALEPH\ (a somewhat stable variant of \OMEGA) into the code base and stepwise
51some primitives were added. It was overall a rather large and serious project
52that took a lot of our time. It was not commercially driven, mostly for \CONTEXT\
53users and therefore also a lot of fun to do. As often with such projects, early
54adapters keep things going.
55
56It took a while before \LUATEX\ was stable in the sense that nothing more was
57added. Because the engine was developed alongside what is called \CONTEXT\ \MKIV,
58we could easily adapt both to each other. Even better: users could use both in
59production. However, in order for other macro packages to use \LUATEX\ (per
60request) it had to be frozen, and that happened around 2015, some 10 years after
61we started. However, we were not done yet and in order not to violate this stability
62principle the follow up was called \LUAMETATEX. Because it was a more drastic
63extension project, and also a somewhat drastic separation of the code base from
64the complex \LUATEX\ one, the related \CONTEXT\ code was also separated, this time
65tagged \MKXL, or \LMTX\ when we talk about the combination. The project started
66around 2019 and soon again entered a state of combined development and use in
67production and most users switched to this variant.
68
69There are more complete wrap ups of these developments and we systematically
70reported on them in various documents that are available in the distribution
71andor published in user group journals.
72
73\stopsection
74
75\startsection[title={The engines}]
76
77Of course all starts with original \TEX. We want to be compatible so we keep that
78functionality. However, for practical reasons \LUAMETATEX\ omits two core
79components. Font loading is not present in the frontend and there is no backend.
80Both are supposed to be provided via \LUA\ plugins. This makes sense because in
81the meantime font technologies have changed and keep changing and backend also
82are a moving target. In \CONTEXT\ we already did all that in \LUA, so there was
83no need to keep that font and \PDF\ generation code around in the engine. There
84are a few more deviations, like dropping some system specific features (terminal
85related) and in former times practical features like outer and long macros that
86no longer made sense and complicated integrating new features unnecessarily.
87
88As mentioned in the introduction \PDFTEX\ is the basis for \LUATEX\ and \LUATEX\ is
89where we started with \LUAMETATEX. If we compare \PDFTEX\ with traditional \TEX\
90the main additions are:
91
92\startitemize[packed]
93\startitem
94 There is an integrated \PDF\ backend that also supports for instance
95 hyperlinks and various annotations.
96\stopitem
97\startitem
98 Expansion of glyphs (aka hz) has been added to the engine and integrated in
99 the par builder. The same is true for character protrusion (in the margin).
100\stopitem
101\startitem
102 There is, to some extend, support for intercharacter kerning.
103\stopitem
104\startitem
105 There are some handy helpers, for instance for calculating hashes,
106 randomization, etc.
107\stopitem
108\startitem
109 There is an extension to injection between lines (adjust).
110\stopitem
111\startitem
112 We have few more conditionals (like testing for a csname and absolute values).
113\stopitem
114\startitem
115 A few helpers like \type {\quitvmode} (that we liked to have in \CONTEXT)
116 were added.
117\stopitem
118\stopitemize
119
120Because \PDFTEX\ was actively developed and maintained over many years,
121extensions showed up stepwise, also depending on usage and needs. That is also
122why the \ETEX\ extensions were included:
123
124\startitemize[packed]
125\startitem
126 More that 256 registers, including marks.
127\stopitem
128\startitem
129 Access to discarded material in the vertical splitting code.
130\stopitem
131\startitem
132 Protection against expansion of macros (the \type {\protected} prefix).
133\stopitem
134\startitem
135 A simple right to left typesetting mechanism.
136\stopitem
137\startitem
138 Access to some states, a limited set of last nodes, etc.
139\stopitem
140\startitem
141 There are some additional tracing features.
142\stopitem
143\startitem
144 One can reprocess tokens and produce detokenized lists.
145\stopitem
146\stopitemize
147
148In \LUATEX\ we also looked at what \OMEGA\ could bring:
149
150\startitemize[packed]
151\startitem
152 More that 256 registers.
153\stopitem
154\startitem
155 Multidirectional typesetting.
156\stopitem
157\startitem
158 Local boxes (in lines).
159\stopitem
160\startitem
161 Input processing.
162\stopitem
163\stopitemize
164
165If we combine these lists, we see font expansion and protrusion coming back in
166\LUAMETATEX. However, already in \LUATEX\ expansion and protrusion were dealt
167with a bit differently and even more so in \LUAMETATEX, while protection in
168\LUAMETATEX\ is implemented differently. We also kept injection of vertical
169material but in \LUAMETATEX\ that done quite differently. Most if \ETEX\ is there
170but not right to left typesetting and the register approach. Of course we kept
171the additional conditionals but implemented them a bit different.
172
173In \LUATEX\ we took the \OMEGA\ enlarged register approach and directional
174typesetting although that has been stripped down and redone to right to left
175only. Local boxes are there but redone in \LUAMETATEX. There was no need for
176input processing because we have \LUA. In the end there is little that we kept
177from the other engines which also means that one cannot take the manuals that
178come with these engines and simply assume that it is there.
179
180We should of course mention \METAPOST. That graphical subsystem was integrated in
181\LUATEX\ and on the one hand stripped down (less backend) and extended (remove
182bottlenecks and add some functionality) in \LUAMETATEX. With respect to \LUA\
183we moved to more recent versions and dropped support for just in time compilation.
184
185There is of course a lot in \LUATEX\ that can be found also in \LUAMETATEX\ but
186the later one goes way beyond its predecessor. It actually provides what we always
187wanted (as \CONTEXT\ developers) but never showed up. And this brings us to a next
188topic.
189
190\stopsection
191
192\startsection[title=Reflections]
193
194The previous section, derived from the \LUATEX\ manual, might suggest that
195\LUATEX\ and therefore \LUAMETATEX\ provides most of what \PDFTEX, \ETEX, \XETEX\
196and \OMEGA\ provide but here I must disappoint the reader. So in addition to or
197variation on the above here are some reflections.
198
199We were quite involved in the early days of \PDFTEX\ development, so some
200features of that program we kept in \LUATEX, like expansion, protrusion, basic
201\PDF\ features like annotations, destinations, outlines and literals,
202transformations, image inclusion, plus a few handy extensions like \type
203{\vadjust pre}, \type {\insertht} and \type {\quitvmode} that were introduced for
204\CONTEXT, as well as positioning that when brought into \PDFTEX\ made that we no
205longer needed the indirect method using specials that we used (first with a post
206processing script filtering specials and providing positional information, later
207that became a the \type {dvipos} program). Experimental features (at that time
208introduced for \CONTEXT) like snapping lines could make sense but were easier to
209handle in \LUA\ so even those were dropped. We never used the features that were
210introduced for other macro packages, like color stacks and (un)escaping because
211we already did that otherwise. We also didnt want to burden the evolving
212\LUATEX\ engine with the other kerning features because they lacked control
213anyway.
214
215As we started in 2005 (with a first release in 2006) the \PDFTEX\ of that time is
216of course not the same as of today. Im not sure what the last version is that
217\CONTEXT\ \MKII\ is targeting at because the version numbering changed a bit (at
218some point versions like \type {14e} became \type {140.XX}, so it might be
219around\type {140.17} or so). It might even be that recent versions break \MKII\
220without us noticing. For \LUATEX\ we basically only took what we needed for
221\CONTEXT\ at that time and assumed that \LUA\ could fill in the gaps. Because
222\CONTEXT\ didnt really use much of the \LUATEX\ backend in the end what we kept
223from \PDFTEX\ in \LUAMETATEX\ was a follow up on expansion and protrusion, for
224which that engine set the standard. If it wasnt for \PDFTEX\ the \TEX\ community
225would not be where it was now.
226
227For as much as they make sense \ETEX\ extensions are mostly there but that
228project was basically stopped after the first major release. In fact because
229\PDFTEX\ has these extensions, we never implicitly had to include \ETEX. When we
230started with \LUATEX\ we actually kept in mind the ideas we had at that time
231because before we started with \LUATEX\ we already had plans for extensions
232(flagged \type {eetex}) but those never came to fruit, just as we had some ideas
233about extending \DVI\ which were superseded by the arrival of \PDF. The token
234prepend and append primitives actually were examples of that. The \ETEX\ project
235demonstrated that extending \TEX\ was an option.
236
237Then there is \XETEX. It is supported by \MKII\ but only the font mechanism was
238extended to handle \OPENTYPE\ fonts. For a while \LUATEX\ has some compatible
239math character definition primitives but none of its features. We can assume that
240its internals are quite different when it comes to fonts from \LUATEX\ because
241\LUATEX\ basically provides support for traditional fonts and delegates
242everything \OPENTYPE\ to \LUA. In \LUATEX\ we used font loader code from
243\FONTFORGE\ and some backend code from \DVIPDFMX, although \CONTEXT\ eventually
244did all loading in \LUA .But anyway this program demonstrated that a \UNICODE\
245engine supporting publicly available (fancy) fonts was possible by adapting \TEX\
246so \XETEX\ introduced \TEX ies to the at that point still evolving world of
247\OPENTYPE.
248
249In \OMEGA\ there were input translation mechanisms that some \CONTEXT\ users used
250but that I never looked into myself. These are of course not present in \LUATEX\
251because we can use \LUA\ for input processing. In the end a bit of the
252directionality is all that we kept, most noticeably the initial \type {parnode}
253and \type {dirnode} but we made them first class nodes instead of whatsits and in
254\LUAMETATEX\ the first one serves different purposes. It got us started with
255directions.
256
257Over the last decades other engines showed up, most noticeably those for e.g.
258Japanese but I never looked into these. Im not sure if \LUATEX\ can do the same
259with help from \LUA, but \LUAMETATEX\ has some more features so maybe it can.
260These engines serve a specific (language and script audience) but if Japanese
261\CONTEXT\ user need something more than we provide they can ask.
262
263In the end the extensions in \LUAMETATEX\ come from our own demands combined with
264trying to be complete but of course I might have missed something. It also means
265that flaws in the design are just mine (or ours in the case of \LUATEX). Of
266course quite some are common sense additions, often based what we need and what
267makes macro programming easier, but if users depend on some feature present in
268the other engines that cannot be handled by \LUA, they might ask for something
269similar but then we need details and examples and not some reference to a manual
270or macros that we are unaware of, have never seen, never used or havent catched
271up with.
272
273Of course \LUAMETATEX\ has plenty of what \LUATEX\ has and the core of \LUATEX\
274is pretty much original \TEX. However, in \LUAMETATEX\ nearly all mechanism have
275been extended, optimized, and in the process made a bit more \CCODEish. The
276original documentation describes what happens, the principles behind \TEX, so to
277say. More details will be added but can also found in numerous documents in the
278\CONTEXT\ distribution, articles and presentations. In the end we owe most to Don
279Knuth, who gave is the (very original) original that we can build upon.
280
281\stopsection
282
283\startsection[title=Usage]
284
285Why is it that there has been little fundamental development around \TEX\
286engines? One of the reasons is that macro packages have to be stable. New
287features can be added but if they are only available in one engine (and there are
288a few more around now, like \XETEX\ and the \CJK\ specific ones) a macro package
289has to provide ways around them when they are not available. Risking some
290criticism I dare to say that in order to use \LUATEX\ to its full potential,
291macro package has to be set up such that this is possible and \CONTEXT\ does just
292that. When we talk backends its relatively easy, and when we talk fonts its
293doable. But if you are not willing to adapt the core of your code dramatically
294(and conceptually) all you get from \LUATEX\ is a builtin scripting language and
295some occasional messing around with node lists. In \CONTEXT\ we could transition
296rather well because the user interfaces permitted to do so without users
297noticing. Of course there were changes, for instance because encodings matter
298less, which is also true for e.g.\ \XETEX, and font technologies changed. But for
299macro packages other than \CONTEXT\ just the availability of \LUA\ might be
300enough reasons to use that engine. That also means that documentation of the more
301intricate features is less important: one can just learn by example and \CONTEXT\
302is that example.
303
304With \LUAMETATEX\ we go further because here one really has to make some
305fundamental choices. Again this could be done within the existing user
306interfaces, but here we are not only talking of fundamental improvements, like
307rendering math or breaking paragraphs into lines, but also of more flexible
308handling of alignments, inserts, adjusts, marks, par and page building, etc.\
309Basically all mechanism got extended and opened up. In order to profit from this
310you have to be able to throw away existing solution and use these extensions to
311come up with better ones. If one can put sentiments aside, this also takes quite
312some time.
313
314A very important aspect (at least for me) is that I want the macro code to look
315nice and in that respect stick to the \TEX\ syntax as much as possible. That
316means that we have more programming related primitives, enhanced macro argument
317parsing, more (flexible) conditionals, additional registers, extra expansion
318related features and so on. Instead of some intermediate layer (like the helpers
319in \CONTEXT) we can stick closer to the language itself. Of course this is not
320something that most users will notice. What users might notice, is that on the
321average \CONTEXT\ with \LUAMETATEX\ performs better than with \LUATEX\ or even
322\LUAMETATEX. Even with more performance critical components delegated to \LUA\
323(like the backend \PDF\ generation) we gain and often can compete performance
324wise well with the faster eight bit \PDFTEX\ engine.
325
326The fact that one has to make (and cannot make) drastic choices has a consequence
327for documentation. Most of what is new and interesting is discussed in articles
328and low level manuals. However, it is often discussed in the perspective of
329\CONTEXT. Although we do discuss and show generic solutions it makes little sense
330to go into details there simply because in the end only \CONTEXT\ will use them
331as intended. Its just a waste of time to implement variants that are more
332generic because they will never be used elsewhere, especially in situations where
333the solutions are considered \quote {standard} and will not change. In \CONTEXT\
334we always followed the principle that if we can do better, we will do better, and
335interfaces are such that this can be done.
336
337Of course that brings up the question \quotation {How do you know that these are
338the best solutions} and the answer is that we dont. However, were not talking
339of quick and dirty solutions. For instance it took years to enhance math support:
340experiments, discussion, reconsideration, documenting, writing articles, looking
341at usage, fonts, etc. A wider discussion would not have brought better solutions,
342if at all. If that were the case, there would already have been successors. The
343same is true for most extensions: there was little need for them outside the
344\CONTEXT\ community. So in the end thats what those interested should look at:
345how is \LUAMETATEX\ used in \CONTEXT. It is the combined development together with
346acceptance by users that makes this possible,
347
348\stopsection
349
350\stopdocument
351
352 |