onandon-110.tex /size: 7677 b    last modification: 2023-12-21 09:43
1% language=us
2
3% After watching \quotation {What Makes This Song Great 30: Alanis Morisette} by
4% Rick Beato, it's tempting to poder \quotation {What makes \TEX\ great}.
5
6\startcomponent onandon-110
7
8\environment onandon-environment
9
10\startchapter[title={Getting there, version 1.10}]
11
12When we decided to turn experiments with a \LUA\ extensions to \PDFTEX\ into
13developing \LUATEX\ as alternative engine we had, in addition to opening up some
14of \TEX's internals, some extensions in mind. Around version 1.00 most was
15already achieved and with version 1.10 we're pretty close to where we want to be.
16The question is, when are we ready? In order to answer that I will look at four
17aspects:
18
19\startitemize[packed]
20\startitem objectives \stopitem
21\startitem functionality \stopitem
22\startitem performance \stopitem
23\startitem stability \stopitem
24\stopitemize
25
26The main {\em objective} was to open up \TEX\ in a way that permit extensions
27without the need to patch the engine. Although it might suit us, we don't want to
28change too much the internals, first of all because \TEX\ is \TEX, the documented
29program with a large legacy. \footnote {This is reflected in the keywords that
30exposed mechanisms use: they reflect internal variable names and constants and as
31a consequence there is inconsistency there.} Discussions about how to extend
32\TEX\ are not easy and seldom lead to an agreement so better is to provide a way
33to do what you like without bothering other users and|/|or interfering with macro
34packages. I think that this objective is met quite well now. Other objectives,
35like embedding basic graphic capabilities using \METAPOST\ have already been met
36long ago. There is more control over the backend and modern fonts can be dealt
37with.
38
39The {\em functionality} in terms of primitives has been extended but within
40reasonable bounds: we only added things that make coding a bit more natural but
41we realize that this is very subjective. So, here again we can say that we met
42our goals. A lot can be achieved via \LUA\ code and users and developers need to
43get accustomed to that if they want to move on with \LUATEX. We will not
44introduce features that get added to or are part of other engines.
45
46We wanted to keeping {\em performance} acceptable. The core \TEX\ engine is
47already pretty fast and it's often the implementation of macros (in macro
48packages) that creates a performance hit. Going \UTF\ has a price as do modern
49fonts. At the time of this writing processing the 270 page \LUATEX\ manual takes
50about 12 seconds (one run), which boils down to over 27 pages per second.
51
52\starttabulate[||c|c|]
53\NC            \BC runtime \BC overhead \NC \NR
54\BC \LUATEX    \NC $12.0$  \NC $+0.6$   \NC \NR
55\BC \LUAJITTEX \NC $ 9.7$  \NC $+0.5$   \NC \NR
56\stoptabulate
57
58Is this fast or slow? One can do tests with specific improvements (using new
59primitives) but in practice it's very hard to improve performance significantly.
60This is because a test with millions of calls that show a .05 second improvement
61disappears when one only has a few thousand calls. Many small improvements can
62add up, but less that one thinks, especially when macros are already quite
63optimal. Also this runtime includes time normally used for running additional
64programs (e.g.\ for getting bibliographies right).
65
66It must be said that performance is not completely under our control. For
67instance, we have patched the \LUAJIT\ hash function because it favours \URL's
68and therefore favours hashing the middle of the string which is bad for our use
69as we are more interested in the (often unique) start of strings. We also
70compress the format which speeds up loading but not on the native windows 64~bit
71binary. At the time this writing the extra overhead is 2~seconds due to some
72suboptimal gzip handling; the cross compiled 64~bit mingw binaries that I use
73don't suffer from this. When I was testing the 32~bit binaries on the machine of
74a colleague, I was surprised to measure the following differences on a complex
75document with hundreds of \XML\ files, many images and a lot of manipulations.
76
77\starttabulate[||c|c|]
78\NC            \BC 1.08 with \LUA\ 5.2 \BC 1.09 with \LUA\ 5.3 \NC \NR
79\BC \LUATEX    \NC $21.5$              \NC $15.2$              \NC \NR
80\BC \LUAJITTEX \NC $10.7$              \NC $10.3$              \NC \NR
81\stoptabulate
82
83Now, these are just rough numbers but they demonstrate that the gap between
84\LUATEX\ and \LUAJITTEX\ is becoming less which is good because at this moment it
85looks like \LUAJIT\ will not catch up with \LUA\ 5.3 so at some point we might
86drop it. It will be interesting to see what \LUA\ 5.4 will bring as it offers an
87\ alternative garbage collector. And imagine that the regular \LUA\ virtual
88machine gets more optimized.
89
90You also have to take into account that having a browser open in the background
91of a \TEX\ run has way more impact than a few tenths of a second in \LUATEX\
92performance. The same is true for memory usage: why bother about \LUATEX\ taking
93tens of megabytes for fonts while a few tabs in a browser can bump memory
94consumption to gigabytes of memory usage. Also, using a large \TEX\ tree (say the
95whole of \TEXLIVE) can have a bit of a performance hit! Or what about inefficient
96callbacks, using inefficient \LUA\ code of badly designed solutions? What we
97could gain here we loose there, so I think we can safely say that the current
98implementation of \LUATEX\ is as good as you can (and will) get. Why should we
99introduce obscure optimizations where on workstations \TEX\ is just one of the
100many processes? Why should we bother too much to speed up on servers that have
101start|-|up or job management overhead or are connected to relatively slow remote
102file system? Why squeeze out a few more milliseconds when badly written macros or
103styles can have an way more impact on performance? So, for now we're satisfied
104with performance. Just for the record, the ratio between \CONTEXT\ \MKII\
105running other engines and \LUATEX\ with \MKIV\ for the next snippet of code:
106
107\starttyping
108\dorecurse{250}{\input tufte\par}
109\stoptyping
110
111is 2.8 seconds for \XETEX, 1.5 seconds for \LUATEX, 1.2 seconds for \LUAJITTEX,
112and 0.9 seconds for \PDFTEX. Of course this is not really a practical test but it
113demonstrates the baseline performance on just text. The 64 bit version of \PDFTEX\
114is actually quite a bit slower on my machine. Anyway, \LUATEX\ (1.09) with \MKIV\
115is doing okey here.
116
117That brings us to {\em stability}. In order to achieve that we will not introduce
118many more extensions. That way users get accustomed to what is there (read: there
119is no need to search for what else is possible). Also, it makes that existing
120functionality can become bug free because no new features can interfere. So, at
121some point we have to decide that this is it. If we can do what we want now,
122there are no strong arguments for more. in that perspective version 1.10 can be
123considered very close to what we want to achieve.
124
125Of course development will continue. For instance, the \PDF\ inclusion code will
126be replaced by more lightweight and independent code. Names of functions and
127symbolic constants might be normalized (as mentioned, currently they are often
128related to or derived from internals). More documentation will be added. We will
129quite probably keep up with \LUA\ versions. Also the \FFI\ interface will become
130more stable. And for sure bugs will be fixed. We might add a few more options to
131control behaviour of for instance of math rendering. Some tricky internals (like
132alignments) might get better attribute support if possible. But currently we
133think that most fundamental issues have been dealt with.
134
135\stopchapter
136
137\stopcomponent
138