1
2
3
4
5
6
7
8
9
10\environment evenmorestyle
11
12\startcomponent evenmoreoffloading
13
14\startchapter[title={Offloading}]
15
16The \LUAMETATEX\ source code started from \LUATEX\ with the idea to make a lean
17and mean variant. In the process the whole code base has been overhauled. Because
18we develop and maintain \CONTEXT\ in parallel all kind of experiments can be
19conducted. \footnote {To my best knowledge macro packages like \LATEX\ have
20decided to use the (mostly stable) \LUATEX\ in combination with a built in font
21renderer. The wish for stability from developers of other macro packages was one
22of the reasons for starting the \LUAMETATEX\ project: \LUATEX\ could no longer
23evolve without folks complaining. So, \LUAMETATEX, at least in the short run, is
24no reasonable option for other macro packages, unless policies have changed.}
25
26Already early in the process the decision was made to remove the backend code. I
27already had a working \LUATEX\ \MKIV\ setup with an independent backend so that
28was an easy step. Later I removed that code from \MKIV\ because a dualistic model
29made for a messy \CONTEXT\ code base, and for \LUATEX\ I like to stick to the
30reference implementation. Of course one can wonder if we still have \TEX\ when
31there is no backend but you need to keep in mind that \TEX\ always needs some
32kind of backend program. If \DVI\ was still a used format (with \CONTEXT) I could
33write a \DVI\ backend without much effort, but \PDF\ is kind of the standard now.
34The relevant primitives can easily be defined in \LUA.
35
36What more defines \TEX ? Of course the macro language, but in addition to that we
37have the handling of fonts, hyphenation and building various lists, those that
38eventually make it into paragraphs and pages. The \ETEX, \PDFTEX\ and \OMEGA\
39engines have added bits and pieces and of course \LUATEX\ added its share.
40
41In \LUAMETATEX\ the macro part has been extended. Very few things were dropped,
42most noticeably \type {\long} and \type {\outer} are no longer effective but
43that made for a better \type {\protected} and gave room for \type {\frozen}.
44
45The handling of fonts is mostly delegated to callbacks but we do have the
46original transformation mechanism available. However, loading fonts is now up to
47\LUA. If that is done right, there is no difference with \TEX. One can argue that
48when a missing piece in the binary is complemented by a \LUA\ solution that gets
49plugged in we just are like \TEX. After all, nowhere is said that the engine has
50to be written in one language and in the \CWEB\ setup of \TEXLIVE\ we already mix
51languages anyway.
52
53The language part is also upgraded and because handling hyphenation has been
54extended were, as with fonts, going beyond what traditional \TEX\ offers. The
55code for hyphenating is slightly different because we permit runtime loading and
56extension, compound and weighted patterns, etc.
57
58Another deviation is the handling of input and output. Although currently the log
59file still happens at the engine end, all the reading and writing from files is
60delegated to \LUA. This means that primitives like \type {\openin} and \type
61{\write} have to be implemented in \LUA, which is not that hard. It makes a lot
62of sense because everything already was driven by callbacks so delegating more to
63\LUA\ in the end gave a simpler input handler. For the user (or macro package) it
64makes no difference.
65
66So, assuming that the primitives not present in the engine are provided by \LUA\
67driven counterparts, we can speak of \TEX. So how about \ETEX ? We kept the macro
68language extensions, but dropped some others, like the direction related code.
69Also the querying of internal codes has been adapted to \LUATEXs internals.
70Expressions have been extended a bit. The \type {\scantokens} primitive now uses
71the same machinery as the \LUA\TEX\ pipe, which made for less code and adds to
72consistency. From the \PDFTEX\ engine we only kept a few things, like protrusion
73and expansion. From \OMEGA\ only the concept of localpar and part of the
74directional model was kept, but because its the backend that deals with
75directions there is not much there. We also dropped some \LUATEX\ features, for
76instance first class image handling only stays as concept (a special kind of
77rule) but again it is the backend that really needs to deal with it.
78
79So, in the end we end up, as intended, with a simpler code base indeed. Of course
80there is also stuff that is not in a traditional \TEX\ or \LUATEX. In addition to
81\LUA\ some libraries are present, but we avoid dependencies on large third party
82bodies of code. The continuous updating in \LUATEX\ told me that this dependency
83is a bad thing of we want the program to compile in decades from now (as
84libraries come and go and often also politics are involved). There is a small
85canonical set of what we provide and although one can use extra libraries it
86takes some effort. The internals of \LUAMETATEX\ are hidden.
87
88Just for the record. Because I want to keep as working engine and adapt \CONTEXT\
89in parallel, the process is rather time consuming. Every optimization, removal of
90unused code, addition of a feature, etc.\ takes multiple runs of the test suite,
91checking with \CONTEXT, generating binaries, updating the distribution, and so
92on. When we dont use something in \CONTEXT\ it goes on the todo stack, which
93means that testing is delayed to when I wrap up in documentation, which only
94happens when I think its stable. For instance: when \type {\openin} and friends
95were delegated to \LUA, the \type {\ifeof} primitive was kept around with a
96temporary callback, so that \type {tikz} kept working. We dont use those read
97channels in \CONTEXT\ so that was a compromise. A while later the if test was
98also delegated and the temporary callback removed. There is no way I could do
99this kind of stepwise development were it now within the \CONTEXT\ community
100where users are willing to accept this.
101
102Another example of something that took some time is checking all memory
103allocation code, adding safeguards, make it more dynamic, making sure we have
104more statistics. It needs some experimenting and the \CONTEXT\ tracking code had
105to be extended. The main motivation for this is that there are some users out
106there (most noticeably Massimiliano who is involved in typesetting some scholar
107encyclopedia work) who run really huge jobs and we can run out of memory or even
108crash then. \footnote {Im not that worried about an occasional bug, because the
109number is small compared to what got added, changed and improved, but of course
110there are always folks who ignore that fact and stress bug(let)s. But, as said,
111the \CONTEXT\ users are a patient lot.} Tracing shows that although \TEX\
112allocates its share of memory, in these thousand plus page documents in small
113print the regular few dozen megabytes can grow to hundreds, most noticeably taken
114up by fonts. Tracing also gives some insight in how fast token and node memory
115grows. Of course in this case \LUA\ takes way more memory, something between 1.5
116and 2GB. Again this is due to the large amount of font instances and also is a
117side effect of massive \XML\ processing (keep in mind that the whole tree is in
118memory) and the fact that there are plenty of optimizations wrt typesetting
119implemented and multiple large registers are added. Its this kind of
120(regression) tests that help in stepwise improving \LUAMETATEX. \footnote {In the
121process one sometimes find ways to safe memory. For instance, marks used
122preallocated arrays that took 1280 KB memory while of course in practice no one
123needs that many mark registers. By making that more dynamic we saved a lot.}
124
125\stopchapter
126
127\stopcomponent
128 |
