Source Browser ?

1% language=us
2
3\startcomponent onandon-53
4
5% copy-edited by Alan Braslau
6
7\environment onandon-environment
8
9\startchapter[title={From \LUA\ 5.2 to 5.3}]
10
11When we started with \LUATEX\ we used \LUA\ 5.1 and then moved seamlessly to 5.2
12when that became available. We didn't run into issues with this language version
13change because there were no fundamental differences that could not be easily
14dealt with. However, when \LUA\ 5.3 was announced in 2015 we were not sure if we
15should make the move. The main reason was that we'd chosen \LUA\ because of its
16clean design part of which meant that we had only one number type: double. In 5.3
17on the other hand, deep down a number can be either an integer or a floating
18point quantity.
19
20Internally \TEX\ is mostly (up to) 32-bit integers so when we go from \LUA\ to
21\TEX\ we are forced to round numbers. Nonetheless, or perhaps because of this,
22one can expect some benefits in using integers in \LUA. Performance|-|wise we
23didn't expect much, and memory consumption would be the same too. So the main
24question then was: can we get the same output and not run into trouble due to
25possible differences in serializing numbers? After all \TEX\ is about stability.
26The serialization aspect is for instance important when we compare quantities
27and|/|or use numbers in hashes, so one must be careful.
28
29Apart from this change in the number model (which comes with a few extra
30helpers), another interesting extension in 5.3 was that bit|-|wise operations are
31now part of the language. However, the lpeg library is still not part of stock
32\LUA. There is also added some minimal \UTF8 support, but less than we provide in
33\LUATEX\ already. So, considering these changes, we were not in a big hurry to
34update. Also, it made sense to wait until this important number|-|related change
35became stable.
36
37But, a few years later, we still had it on our agenda to test the new version of
38\LUA, and after the \CONTEXT\ 2017 meeting we decided to give it a try; here are
39some observations. A quick test involved just dropping in the new \LUA\ code and
40seeing if with this we could still compile a \CONTEXT\ format. Indeed that was no
41big deal but the test run failed because at some point a (for instance) \type {1}
42became a \type {1.0}. It turned out that serializing has some side effects, and
43with some ad hoc prints for tracing (in the \LUATEX\ source) I could figure out
44what was going on. How numbers are seen can (to some extent) be deduced from the
45\type {string.format} function, which is in \LUA\ a combination of parsing,
46splitting and concatenation combined with piping to the \CCODE\ \type {sprintf}
47function: \footnote {Actually, at some point I decided to write my own formatter
48on top of \type {format} and I ended up with splitting as well. It's only now
49that I realize why this is working out so well (in terms of performance): simple
50format (single items) are passed more or less directly to \type {sprintf} and as
51\LUA\ itself is fast, due to some caching, the overhead is small compared to the
52built|-|in splitter method. An advantage is that the \CONTEXT\ formatter has many
53more options and is also extensible.}
54
55\starttyping
56local a =  2   * (1/2) print(string.format("%s",  a),math.type(x))
57local b =  2   * (1/2) print(string.format("%d",  b),math.type(x))
58local c =  2           print(string.format("%d",  c),math.type(x))
59local d = -2           print(string.format("%d",  d),math.type(x))
60local e =  2   * (1/2) print(string.format("%i",  e),math.type(x))
61local f =  2.1         print(string.format("%.0f",f),math.type(x))
62local g =  2.0         print(string.format("%.0f",g),math.type(x))
63local h =  2.1         print(string.format("%G",  h),math.type(x))
64local i =  2.0         print(string.format("%G",  i),math.type(x))
65local j =  2           print(string.format("%.0f",j),math.type(x))
66local k = -2           print(string.format("%.0f",k),math.type(x))
67\stoptyping
68
69This gives the following results:
70
71\starttabulate[|cBT|c|T|c|cT|]
72\BC a \NC  2   * (1/2)\NC   s \NC 1.0 \NC float   \NC \NR
73\BC b \NC  2   * (1/2)\NC   d \NC 1	  \NC float   \NC \NR
74\BC c \NC  2          \NC   d \NC 2   \NC integer \NC \NR
75\BC d \NC -2          \NC   d \NC 2	  \NC integer \NC \NR
76\BC e \NC  2   * (1/2)\NC   i \NC 1	  \NC float   \NC \NR
77\BC f \NC  2.1        \NC .0f \NC 2	  \NC float   \NC \NR
78\BC g \NC  2.0        \NC .0f \NC 2	  \NC float   \NC \NR
79\BC h \NC  2.1        \NC   G \NC 2.1 \NC float   \NC \NR
80\BC i \NC  2.0        \NC   G \NC 2	  \NC float   \NC \NR
81\BC j \NC  2          \NC .0f \NC 2	  \NC integer \NC \NR
82\BC k \NC -2          \NC .0f \NC 2	  \NC integer \NC \NR
83\stoptabulate
84
85This demonstrates that we have to be careful when we need numbers represented as
86strings. In \CONTEXT\ the places where we had to check for this was not that
87many: in fact, only some hashing related to font sizes had to be done using
88explicit rounding.
89
90Another surprising side effect is the following. Instead of:
91
92\starttyping
93local n = 2^6
94\stoptyping
95
96we now need to use:
97
98\starttyping
99local n = 0x40
100\stoptyping
101
102or just:
103
104\starttyping
105local n = 64
106\stoptyping
107
108because we don't want this to be serialized to \type {64.0} which is due to the
109fact that a power results in a float. One can wonder if this makes sense when we
110apply it to an integer.
111
112At any rate, once we were able to process a file, two standard documents were
113chosen for a performance test. Some experiments with loops and casts had
114demonstrated that we could expect a small performance hit and indeed, this was
115the case. Processing the \LUATEX\ manual takes 10.7 seconds with 5.2 on my
1165-year-old laptop and 11.6 seconds with 5.3. If we consider that \CONTEXT\ spends
117about 50\% of its time in \LUA, then we find here a 20\% performance penalty
118using the later version of \LUA. Processing the \METAFUN\ manual (which has lots
119of \METAPOST\ images) went from less than 20 seconds (and \LUAJITTEX\ does it in
12016 seconds) to up to more than 27 seconds. So there we lose more than 50\% on the
121\LUA\ end. When we observed these kinds of differences, Luigi and I immediately
122got into debugging mode, partly out of curiosity but also because consistent
123performance is always important to us.
124
125As these results made no sense, we traced different sub-mechanisms and eventually
126it became clear that the reason behind the speed penalty was in fact that the
127core \typ {string.format} function was behaving quite badly in the \type {mingw}
128cross|-|compiled binary, as can be seen by this test:
129
130\starttyping
131local t = os.clock()
132for i=1,1000*1000 do
133 -- local a = string.format("%.3f",1.23)
134 -- local b = string.format("%i",123)
135    local c = string.format("%s",123)
136end
137print(os.clock()-t)
138\stoptyping
139
140\starttabulate[|c|c|c|c|c|]
141\BC   \BC lua 5.3 \BC lua 5.2 \BC texlua 5.3  \BC texlua 5.2 \BC \NR
142\BC a \NC 0.43    \NC 0.54    \NC 3.71 (0.47) \NC 0.53       \NC \NR
143\BC b \NC 0.18    \NC 0.24    \NC 3.78 (0.17) \NC 0.22       \NC \NR
144\BC c \NC 0.26    \NC 0.68    \NC 3.67 (0.29) \NC 0.66       \NC \NR
145\stoptabulate
146
147Both 5.2 binaries perform the same but the 5.3 \LUA\ binary greatly outperforms
148the \LUATEX binary so we had to figure out why. After all, the integer
149optimization should bring some gain! It took us a while to figure out what was
150going wrong, and the numbers in parentheses are the results after fixing \LUATEX.
151
152Because font internals are specified in integers one would expect a gain
153in running the command:
154
155\starttyping
156mtxrun --script font --reload force
157\stoptyping
158
159and indeed that is the case. On my machine a scan results in 2561 registered
160fonts from 4906 read files and with 5.2 that takes 9.1 seconds while 5.3 needs a
161bit less: 8.6 seconds (with the bad cross|-|compiled format performance) and even
162less once that was fixed.
163
164For a test:
165
166\starttyping
167\setupbodyfont[modern]     \tf \bf \it \bs
168\setupbodyfont[pagella]    \tf \bf \it \bs
169\setupbodyfont[dejavu]     \tf \bf \it \bs
170\setupbodyfont[termes]     \tf \bf \it \bs
171\setupbodyfont[cambria]    \tf \bf \it \bs
172\starttext \stoptext
173\stoptyping
174
175This code needs 30\% more runtime using the newer version of \LUA\ so the
176question is: how often do we call \type {string.format} there? A first run (when
177we wipe the font cache) needs some 715\,000 calls while successive runs need
178115\,000 calls so the slow down definitely comes from the bad handling of \type
179{string.format}.
180
181When we drop in a \LUA\ or whatever other dependency update we don't want this
182kind of impact. In fact, when one uses external libraries that are or can be
183compiled under the \TEX\ Live infrastructure and the impact would be so dramatic,
184this would be very bad advertising, especially when one considers the occasional
185complaint about \LUATEX\ being slower than other engines.
186
187The good news is that eventually Luigi was able to nail down this issue and we
188got a binary that performed well. It looks like \LUA\ 5.3.4 (cross|)|compiles
189badly under both \GCC\ 5.3.0 and 6.3.0.
190
191So in the end loading the fonts takes:
192
193\starttabulate[||c|c|]
194\BC            \BC caching   \BC running \NC \NR
195\BC 5.2 stock  \NC  8.3      \NC 1.2     \NC \NR
196\BC 5.3 bugged \NC 12.6      \NC 2.1     \NC \NR
197\BC 5.3 fixed  \NC  6.3      \NC 1.0     \NC \NR
198\stoptabulate
199
200So indeed after an initial scare it looks like 5.3 is able to speed up \LUATEX\ a
201bit, given that one integrates it in the right way! The use of a recent compiler
202is needed here, although one can wonder when another bad case will show up again.
203One can also wonder why such a slow down can mostly go unnoticed, because for
204sure \LUATEX\ is not the only compiled program integrating the \LUA\ language.
205\footnote{We can only speculate that others do not pay such close attention to
206performance.}
207
208The next examples are some edge cases that show you need to be aware
209that
210\startitemize[n,text,nostopper]
211    \startitem an integer has its limits, \stopitem
212    \startitem that hexadecimal numbers are integers, and \stopitem
213    \startitem that \LUA\ 5.2 and \LUAJIT\ can differ in small details: \stopitem
214\stopitemize
215
216\starttabulate[||T|T|]
217\NC        \NC \tx print(0xFFFFFFFFFFFFFFFF) \NC \tx print(0x7FFFFFFFFFFFFFFF) \NC \NR
218\HL
219\BC lua 52 \NC 1.844674407371e+019 \NC 9.2233720368548e+018 \NC \NR
220\BC luajit \NC 1.844674407371e+19  \NC 9.2233720368548e+18  \NC \NR
221\BC lua 53 \NC -1                  \NC 9223372036854775807  \NC \NR
222\stoptabulate
223
224We see here that \LUA\ 5.3 clearly represents some progress.
225
226So, to summarize the migration, a quick test was relatively easy: move 5.3 into
227the code base, make slight adaptations to the internals (there were a few
228\LUATEX\ interfacing bits where explicit rounding was needed), run tests, and
229eventually fix some issues related to the Makefile (compatibility) and \CCODE\
230obscurities (the very slow \type {sprintf}). \footnote{This demonstrates the
231importance of compilers, or rather how one writes code with respect to each
232compiler.}
233
234Adapting \CONTEXT\ was also not much work, but the test suite uncovered some
235nasty side effects. For instance, the valid 5.2 solution:
236
237\starttyping
238local s = string.format("02X",u/1024)
239local s = string.char        (u/1024)
240\stoptyping
241
242now has to become (works with both 5.2 and 5.3):
243
244\starttyping
245local s = string.format("02X",math.floor(u/1024))
246local s = string.char        (math.floor(u/1024))
247\stoptyping
248
249or (with 5.2 and emulated or real 5.3):
250
251\starttyping
252local s = string.format("02X",bit32.rshift(u,10))
253local s = string.char        (bit32.rshift(u,10))
254\stoptyping
255
256or (5.3 only):
257
258\starttyping
259local s = string.format("02X",u >> 10))
260local s = string.char        (u >> 10)
261\stoptyping
262
263or (5.3 only):
264
265\starttyping
266local s = string.format("02X",u//1024)
267local s = string.char        (u//1024)
268\stoptyping
269
270Unfortunately, adapting a conditional section like:
271
272\starttyping
273if LUAVERSION >= 5.3 then
274    local s = string.format("02X",u >> 10))
275    local s = string.char        (u >> 10)
276else
277    local s = string.format("02X",bit32.rshift(u,10))
278    local s = string.char        (bit32.rshift(u,10))
279end
280\stoptyping
281
282will fail because (of course) the 5.2 parser doesn't like the 5.3 syntax
283elements. In \CONTEXT\ we have some experimental solutions for this but it is
284beyond the scope of this summary.
285
286In the process of this update a few \UTF\ helpers were added to the string
287library so that we have a common set for both \LUAJIT\ and \LUA\ (the \type
288{utf8} library that was added to 5.3 is not very useful for \LUATEX). For now we
289also keep the \type {bit32} library on board, of course, we'll not mention all
290the details here.
291
292When we consider a gain in speed of 5–10\% with 5.3 that also means that the gain
293obtained using \LUAJITTEX\ compared to \LUA\ 5.2 becomes less important. For
294instance in font processing both engines (\LUA\ 5.3 and \LUAJIT) now perform
295roughly to the same.
296
297As I write this, we've just entered 2018 and after a few months of testing
298\LUATEX\ with \LUA\ 5.3 we're confident that we can move the code to the
299experimental branch. This means that we will use this version in the \CONTEXT\
300distribution and likely will ship this as 1.10 in 2019 where \LUA\ 5.3 becomes
301the default. The 2018 version of \TEX~Live will have 1.07 with \LUA\ 5.2 while
302intermediate versions of the \LUA\ 5.3 binary will end up on the \CONTEXT\
303garden, probably with number 1.08 and 1.09 (who knows what else we will add or
304change in the meantime).
305
306\subsubject{Addendum}
307
308Around the 2018 meeting I also started what is to become the next major upgrade
309of \CONTEXT, this time using a new engine \LUAMETATEX. In working on that I
310decided to try \LUA\ 5.4 to see what consequences this new version would have for
311us. There are no real conceptual changes as were found with the number model in
3125.3, so the tests didn't reveal any real issues. But as an additional step
313towards a bit cleaner distinction between strings and numbers, I decided to
314disable the automatic casting so that mixing strings and numbers in expression
315for instance is no longer permitted. If I remember correctly, there was only in
316one place I had to adapt the source (and we're talking about a pretty large \LUA\
317code base).
318
319There is a new mechanism in \LUA\ for freezing constants but I'm not yet sure if
320it makes much sense to use it, although one of the intentions is to produce more
321efficient bytecode. \footnote {Mid July 2019 some quick tests indeed show a
322performance boost with the experimental code base, but if we want to benefit from
323using constants, the \CONTEXT\ codebase has to be adapted, which means that those
324parts no longer will work with stock \LUATEX.} It's use goes along with some
325other restrictions, like the possibility to adapt loop counters inside the loop.
326Inside the body of a loop one could always adapt such a variable, which (I can
327imagine) can come in handy. I haven't checked the source code for that, but
328probably I don't do this anywhere.
329
330Another new feature is an alternative garbage collector which seems to perform
331better when there are many variables with a short life spans. At least for now I
332have decided to default to this variant in future releases.
333
334Overall the performance of \LUA\ 5.4 is better than its predecessors which means
335that the gap between \LUATEX\ and \LUAJITTEX\ is closed or is closing. This is
336good because I have chosen not to support \LUAJIT\ in \LUAMETATEX.
337
338\stopcomponent
339
340% collectgarbage("count") -- two return values in 2
341