Source Browser ?

1% language=us runpath=texruns:manuals/evenmore
2
3\startcomponent evenmore-whattex
4
5\environment evenmore-style
6
7\startchapter[title={Is \LUAMETATEX\ still \TEX ?}]
8
9\startsection[title={Introduction}]
10
11Is \LUAMETATEX\ really a \TEX\ (compatible) engine? The answer to that depends on
12how you define \TEX. If you think of the program with the same name, the answer
13is definitely \quotation {no}, simply because a program that is not exactly
14behaving like \quotation {\TEX\ The Program} cannot be called \TEX. This is why
15derived programs have \type {tex} in their name but also some addition that
16indicates that it isn't the original: \type {e}, \type {pdf}, \type {lua}. Don't
17confuse that with macro package names that have \type {tex} in their name. If you
18find such binaries that they are likely some stub to an engine (binary) that
19preloads a format file (a memory dump) with the same name.
20
21When you mean \quotation {\TEX\ The Macro Language} the answer is a bit more
22nuanced especially when the results are pretty close to identical. In the next
23sections I will discuss this in more detail from the perspective of how \CONTEXT\
24evolved and what engines it has used.
25
26\stopsection
27
28\startsection[title={Multiple engines}]
29
30When we started with \CONTEXT\ there was not that much choice in engines.
31Basically one just used original \TEX, but although we used the version that came
32with the book, pretty soon we switched to em\TEX, a version that gave more
33memory; later a real huge version showed up. The fonts used were bitmaps and the
34viewer was a \DVI\ bitmap viewer. However, when our new printer could not be set
35up properly we decided to move on to \POSTSCRIPT\ fonts. That also meant using a
36different backend driver (\DVIPSONE). And then of course we also started using a
37previewer that could handle outline fonts. Once you start along that route
38graphics come into play, color shows up and hyperlinks become an option. A couple
39of years later the \PDF\ document rendering format was introduced. This paragraph
40already mentions a lot of different programs and adaptations, but we're still
41talking good old \TEX\ here and \CONTEXT\ was set up in such a way that it
42adapted itself to whatever ecosystem made sense. When looking at \TEX\ one has to
43consider the front as well as the backend, and both have related primitives and
44features. Extensions to the frontend have been driven by the demands of macro
45packages (beyond the original ideas) and those of the backend relate to what the
46evolving rendering demands impose.
47
48A couple of decades ago the \ETEX\ project started. It's objective was to extend
49stable \TEX\ with a couple of more primitives and features: it is a superset and
50therefore still \TEX, but as it really is an extension the name was extended too
51(with the bit unusual character $\varepsilon$). At that point the main reason for
52\CONTEXT\ was convenience because the new features were already kind of present
53in the code base (think of emulated behavior). Again the macro package adapted
54itself at runtime.
55
56Then \PDFTEX\ came around which had some impact. It introduced the concept of a
57built|-|in backend that avoided additional programs. The \ETEX\ extensions were
58merged into this program so that basically meant that it replaced its
59predecessors. For a user \PDFTEX\ was just \TEX. For some reason the narrative
60became that \CONTEXT\ depended on \PDFTEX, probably because it was always quick
61in using its features, a side effect of being close to the development.
62
63The \CONTEXT\ package was an early adopter of \METAPOST\ and that graphic
64subsystem, although still external, was integrated in such a way that users could
65think of it being embedded. This was made possible by the fact that right from
66the start \CONTEXT\ came with an infrastructure that handled processing including
67subruns as needed for \METAPOST. This is why, years later, adding a \METAPOST\
68library to \LUATEX\ was a logical step. As \CONTEXT\ came with a lot of scripts
69(for all kind of tasks related to typesetting and managing a \TEX\ ecosystem)
70adding a scripting language (like \LUA) was not that strange either.
71
72In parallel to \PDFTEX\ the experimental \OMEGA\ program was on its way and
73although at some point a stable \ALEPH\ variant was there, it never was robust
74enough for production. Its main contribution (that survived) was the introduction
75if directional typesetting. There were \CONTEXT\ users using it but for very
76specific applications. It's also why the bidirectional model of \OMEGA\ inspired
77\LUATEX\ more than the simpler model that \ETEX\ used.
78
79\stopsection
80
81\startsection[title={The merge}]
82
83We now move forward to \LUATEX\ and more precisely \LUAMETATEX\ because that is
84for \CONTEXT\ the engine of choice now. To what extend is it \TEX\ or not? The
85naive answer is \quotation {no} because some primitives are not present and|/|or
86are implemented using \LUA. However, these primitives fall into categories. Some
87relate to the backend and in \LUAMETATEX\ the backend is not built|-|in and as a
88consequence a macro package has to provide the primitives as part of its
89implementation of a backend. This is no big deal because the backend related
90primitives in \TEX\ The Program are actually examples of extensions and
91implemented as such. Handling them happens in kind of isolated code. Take \type
92{\special}: it is basically a no|-|op when the \DVI\ driver doesn't interpret
93what is passed to the \DVI\ file. \footnote {\CONTEXT\ \MKII\ has a bunch of
94backend drivers, \TEX\ code, that targets specific postprocessors and they hook
95into primitives like \type {\special} or the additional \type {\pdf...} ones in
96\PDFTEX.} \footnote {We need to keep in mind that by the time \PDFTEX\ and later
97\LUATEX\ were developed memory constraints were lifted so these engines didn't
98have to work around the limitations that for instance \ETEX\ and \OMEGA\ had to
99cope with.}
100
101A more drastic change is the lack of font loaders and that no fonts can be stored
102in the format. Again this relates to the simple fact that todays fonts are more
103demanding so we need to extend the machinery and as we do that via \LUA\
104extensions we can as well do all that way. Less drastic, but it could have side
105effects, is that the machinery has to be able to deal with \OPENTYPE\ math. And
106of course all is \UNICODE\ aware so additional primitives cope with that. But in
107principle the old stuff should still work. Hyphenation is also expanded: patterns
108are loaded runtime and the hyphenation, ligature building and kerning stages are
109split, which actually it a good thing.
110
111The \LUAMETATEX\ code base is a follow up on \LUATEX, that combines good old
112\TEX\ (but adapted with respect to fonts, languages and math as mentioned), parts
113of \ETEX\ (so it provides more primitives), bits of \PDFTEX\ (like protrusion and
114expansion, although adapted), and rudiments of \OMEGA\ (\ALEPH). And of course
115there's a lot of new stuff too, primitives as well as ways to plug in \LUA\ code
116plus some helpers at the \LUA\ end.
117
118As an example of progression, by now the \ETEX\ extensions that we kept are
119integrated more naturally in existing subsystems. A nice detail is that there are
120no longer any version numbers that relate to \ETEX; for a while they were kept
121but suddenly I realized that it makes no sense to waste (four) command codes on
122something that is of not much use: there has never been a real \ETEX\ follow up
123after its stable release so testing for a version makes no sense. No backend
124means no \PDFTEX\ version info too and \OMEGA\ version numbers serve no purpose
125either. If a macro package needs to know what functionality is there, testing for
126the \LUATEX\ version number, revision and maybe functionality level makes enough
127sense. By the way, one reason for a clean up related to \ETEX\ was that where
128\ETEX\ uses change files to replace or extend good old \TEX\ code, \LUATEX\ has
129one integrated code base.
130
131\stopsection
132
133\startsection[title={The verdict}]
134
135So in the end the answer is that \LUAMETATEX\ is mostly \TEX\ but that due to
136developments like for instance \UNICODE, \OPENTYPE\ fonts and math, as well as
137the wish to use images, color, runtime graphics, directionality, features beyond
138what the engine has built, etc.\ in the end it hopefully meets the demands to
139today. In its core the same code is still there although extensions and hooks got
140mixed in more naturally. When in documents (or talks) I speak of \TEX\ I
141basically refer to a concept (materialized in the set of core primitives and
142related functionality) but once extensions come into play I try to talk of
143\LUATEX\ or \LUAMETATEX. This happens kind of automatic because I know what got
144added but I can imagine that users who entered the game later don't always see
145what was added (and when).
146
147\stopsection
148
149\stopchapter
150
151\stopcomponent
152
153% Written on the day I heard Neal Peart had passed away so mixed with watching
154% Rush dvd's which made me realize that times flies. They day before I was at a
155% Jan Akkerman show ... timeless quality too.
156