Source Browser ?

1% language=us
2
3\startcomponent about-calls
4
5\environment about-environment
6
7\logo[CRITED]{CritEd}
8
9\startchapter[title={\LUATEX\ 0.79}]
10
11% Hans Hagen, PRAGMA ADE, April 2014
12
13\startsection[title=Introduction]
14
15To some it might look as if not much has been done in \LUATEX\ development but
16this is not true. First of all, the 2013 versions (0.75-0.77) are quite stable
17and can be used for production so there is not much buzz about new things.
18\CONTEXT\ users normally won't even notice changes because most is encapsulated
19in functionality that itself won't change. The binaries on the \type
20{contextgarden.net} are always the latest so an update results in binaries that
21are in sync with the \LUA\ and \TEX\ code. Okay, behaviour might become better
22but that could also be the side effect of better coding. Of course some more
23fundamental changes can result in temporary bugs but those are normally easy to
24solve.
25
26Here I will only mention the most important work done. I'll leave out the
27technical details as they can be found in the manual and in articles that were
28written during development. The version discussed is 0.79.
29
30\stopsection
31
32\startsection[title=Speed]
33
34One of the things we spent a lot of time on is speed. This is of course of more
35importance for a system like \CONTEXT\ that can spend more than half its time in
36\LUA, but eventually we all benefit from it. For the average user it doesn't
37matter much if a run takes a few seconds but in automated workflows these
38accumulate and if a process has to produce 5 documents of 20 pages (each
39demanding a few runs) or a few documents of several hundreds of pages, it might
40make a difference. In the \CRITED\ project we aim for complex documents produced
41from \XML\ at a rate of 20 pages per second, at least for stock \LUATEX.
42\footnote {This might look slow but a lot is happening there. A simple 100 page
43document with one word per page processes at more that 500 pages per second but
44this is hard to match with more realistic documents. When processing data from
45bases using the \CLD\ interface getting 50 pages per seconds is no problem.} In
46an edit|-|preview cycle it feels better if we don't use more than half a second
47for a couple of pages: loading the \TEX\ format, initializing the \LUA\ modules,
48loading fonts, typesetting and producing a proper \PDF\ file. We also want to be
49prepared for the ultra portable computers where multiple cores compensate the
50lower frequency, which harms \TEX\ as sequential processor using one core only.
51
52An important aspect of speedup is that it must not obscure the code. This is why
53the easiest way to achieve it is to use a faster variant of \LUA, and \LUAJIT\
54with its faster virtual machine, is a solution for that. We are aware of the
55fact that processors not necessarily become faster, but that on the other hand
56memory becomes larger. Disk speed also got better with the arrival of
57flash based storage. Because \LUATEX\ should run smoothly on future portable
58devices, the more we can gain now, the better it gets in the future. A decent
59basic performance is possible and we don't have to focus too much on memory and
60disk access and mostly need to keep an eye on basic \CPU\ cycles. Although we
61have some ideas about improving performance, tests demonstrate that \LUATEX\
62is not doing that bad and we don't have to change it's internals. In fact, if we
63do it might as well result in a drastic slowdown!
64
65One interesting performance factor is console output. Because \TEX\ outputs
66immediately with hardly any buffering, it depends a lot on the speed of console
67output. This itself depends on what console is used. \UNIX\ consoles normally
68have some buffering and refresh delay built in. There the speed depends on what
69fonts are used and to what extend the output gets interpreted (escape sequences
70are an example). I've run into cases where a run took seconds more because of a
71bad choice of fonts. On \WINDOWS\ it's more complicated since there the standard
72console (like \TEX) is unbuffered. The good news is that there are several
73alternatives that perform quite well, like console2 and conemu. These
74alternatives buffer output and have refresh delays. But still, on a very high res
75screen, with a large console window logging has impact. Interesting is that when
76I run from the editor (SciTE) output is pretty fast, so normally I never notice
77much of a slowdown. Of course these kind of performance issues can hit you more
78when you work in a remote terminal.
79
80The reason why I mention this is that in order to provide a user feedback about
81issues, there has to be some logging and depending on the kind of use, more or
82less is needed. This means that on the \CONTEXT\ mailing list we sometimes get
83complaints about the amount of logging. It is for this reason that much logging is
84optional and all logging can be disabled as well. Because we go through \LUA\
85we have some control over efficiency too. In the current \LUATEX\ release most
86logging can now be intercepted, including error messages.
87
88Talking of a slowdown, in the \CRITED\ project we have to deal with real large
89indices (tens of thousands of entries) and we found out that in the case of
90interactive variants (register entry to text and back) the use of \LUAJITTEX\
91could bring down a run to a grinding halt. In the end, after much testing we
92figured out that a suboptimal string hashing function was the culprit and we did
93extensive tests with both the \LUAJIT, \LUA\ 5.1 and \LUA\ 5.2 variant. We ended
94up by replacing the \LUAJIT\ hash function by the the \LUA\ 5.1 one which is a
95relative easy operation. Because \LUAJIT\ can address less memory than regular
96\LUA\ it will always be a matter of testing if \LUAJITTEX\ can be used instead of
97\LUATEX. Standard document processing (reports and such) is normally no problem
98but processing large amounts of data from databases can be an issue.
99
100In the process of cleaning up the code base for sure we will also find ways to
101make things run even smoother. But, in any case, version 0.80 is already a good
102benchmark for what can be achieved.
103
104\stopsection
105
106\startsection[title=Nodes]
107
108One of the bottlenecks in the hybrid approach is crossing the so called C
109boundary. This is not really a bottleneck, unless we're talking of many millions
110of function calls. In practice this only happens in for instance more extreme
111font handling (Devanagari or sometimes Arabic). If performance is really an issue
112one can fallback on a more direct node access model. Of course the overhead of
113access should be compared to other related activities: one can gain .25 seconds
114on a run in using the direct access model, but if the whole runs takes 25
115seconds, it can be neglected. If the price paid for it is less readable code it
116should only be done deep down a macro package where no user even sees the code.
117We use this access model in the \CONTEXT\ core modules and so far it looks quite
118okay, especially for more extensive manipulations. The gain in speed is quite
119noticeable if you use the more advanced features of \CONTEXT.
120
121There can be some changes in the node model but not that drastic as the current
122model is quite ok and also stays close to original \TEX\ so that existing
123documentation still applies. One of the changes will be that glue spec (sub)nodes
124will disappear and glue nodes will carry that information. Direction whatsits
125will become first class nodes as they are part of the concept (whatsits
126normally relate to extensions) and the same might happen with image nodes. As a
127side effect we can restructure the code so that it becomes more readable. Some
128experimental \PDFTEX\ functionality will be removed as it can be done better with
129callbacks.
130
131\stopsection
132
133\startsection[title=The parbuilder and HZ]
134
135As we started from \PDFTEX\ we inherit also its experimental code and character.
136One of the objectives is to separate font- and backend as good as possible. We
137have already achieved a lot and apart from bringing consistency in the code, the
138biggest change has been a partial rewrite of the hz code, especially the way
139fonts are managed. Instead of making copies of fonts with different properties,
140we now carry information in the relevant nodes. The backend code already got away
141from multiple fonts by using transformation of the base font instead of
142additional font instances, so this was a natural adaptation. This was actually
143triggered by the fact that a \LUA\ based par builder demonstrated that this made
144sense. The new approach uses less memory and is a bit faster (at least in
145theory).
146
147In callbacks it makes life easier when a node list has a predictable structure.
148For instance, the result of a paragraph broken into lines still has discretionary
149nodes. Is that really needed? Lines can have left- or rightskip nodes, depending
150on the fact if they were set. Math nodes can disappear as part of a cleanup in
151the line break code, but this is unfortunate when one expects them to be
152somewhere in the list in a callback. All this will be made consistent. These are
153issues we will look into on the way to version 1.0.
154
155I occasionally play with the \LUA\ based par builder and it is quite compatible
156even if we take the floating point \LUA\ aspect into account. However when using
157hz the outcome is different: sometimes better, sometimes worse. Personally I
158don't care too much as long as it's consistent. Features like hz are for special
159cases anyway and can never be stable over years if only because fonts evolve. And
160we're talking of bordercase typesetting: narrow columns that no matter what method is
161used will never look okay. \footnote {Some people don't like larger spaces, others
162don't like stretched glyphs.}
163
164\stopsection
165
166\startsection[title=The backend]
167
168The separation of front- and backend is more a pet project. There is some
169experimental code that will get removed or integrated. We try to make the backend
170consistent from the \TEX\ as well as \LUA\ end and some is reflected in
171additional features and callbacks.
172
173Some of the variables that can be set (the \LUA\ counterparts of the \type {\pdf..}
174token registers at the \TEX\ end) are now consistent with each other and avoid
175going via pseudo tokenization.  Typical aspects of a backend that only a few users
176will notice but nevertheless needed work.
177
178The merge of engines also resulted in inconsistencies in function names, like using
179\type {pdf_} in function names where nothing \type {PDF} is involved.
180
181\stopsection
182
183\startsection[title=Backlinks]
184
185In callbacks we mostly deal with node lists. At the \TEX\ end of course we also
186have these lists but there it is quite clear what gets done with them. This means
187that there is no need for double linked lists. It also means that what is known
188as the head of a list can in fact be in the middle. The for \TEX\ characteristic
189nesting model has resulted in stacks and current pointers. The code uses so
190called temp nodes to point at the head node.
191
192As a consequence in \LUATEX, where we present a double linked list, before the
193current version one could run into cases where for instance a head node had a
194prev pointer, even one that made no sense. As said, no big deal in \TEX\ but in
195the hands of a user who manipulates the node list it can be dramatic. The current
196version has cleaned head nodes as well as consistent backlinks, but of course we
197keep the internals mostly unchanged because we stay close to the Knuthian
198original when possible. \footnote {Even with extensions the original
199documentation still covers most of what happens.}
200
201\stopsection
202
203\startsection[title=Properties]
204
205Sometimes you want to associate additional information to a node. A natural way
206to do this is attributes. These can be set at the \TEX\ and \LUA\ end and
207accessed at the \LUA\ end. At the \LUA\ end one can have tables with nodes as
208indices and store extra information but that has the disadvantage that one has no
209clue if such information is current: nodes come and go and are recycled.
210
211For this reason we now have a global properties table where each allocated node
212can have a table with whatever information users might like to store. This itself
213is not special, but the nice thing is that when a node is freed, that information
214is also freed. So, you cannot run into old data. When nodes are copied its
215properties are also copied. The overhead, when not used, is close to zero, which is
216always an objective when extending the core engine.
217
218Of course this model demands that macro package somehow controls consistent use
219but that is not different from what already has to be done. Also, simple
220extensions like this avoid hard codes solutions, which is also something we want
221to avoid.
222
223\stopsection
224
225\startsection[title=\LUA\ calls]
226
227We have so called user nodes that can carry a number, string, token list or node
228list. We now have added \LUA\ to this repertoire. In fact, we now could use only a
229\LUA\ variable and we might have done so in retrospect, but for the moment we we
230stick to the current model of several basic types. The \LUA\ variable can be
231anything and it is up to the user (in some callback) to deal with them.
232
233User nodes are not to be confused with late \LUA\ nodes. You can store a function
234call in a user node but that's about it. You can at a later moment decide to call
235that function but it's still an explicit action. The value of a late \LUA\ node
236on the other hand is dealt with automatically during shipout. When the value is a
237string it gets interpreted as \LUA, but new is that when the value is a function
238it will get called. At that moment we have access to some of the current backend
239properties, like locations.
240
241\stopsection
242
243\startsection[title=Artefacts]
244
245Because \LUATEX\ took code from \PDFTEX, that is built upon \ETEX, which in turn
246is an extension to \TEX, and \OMEGA, that also extends \TEX, there is code that
247no longer makes sense for us. Combine that with the fact that some code carries
248signatures of translated \PASCAL\ to \CCODE, we have some cleanup to do as follow
249up on the not to be underestimated move to \CCODE. This is an ongoing process but
250also fun doing. Luigi and I spend many hours exploring venues and have
251interesting Skype sessions that can easily sidetrack, and with Taco getting more
252time for \LUATEX\ we expect to get most of our (still growing) todo list done.
253
254Because \LUATEX\ started out as an experiment, there is some old code around. For
255instance, we used to have multiple instances and this still shows in some places.
256We can simplify the \LUA\ to \TEX\ interface a bit and clean up the \LUA\ global
257state handling, but we're not in a big hurry with this. Experiments have been
258done with some extensions to the writer code but they are hold back to after the
259cleanup.
260
261In a similar fashion we have sped up the way \LUA\ keyword and values get
262resolved. Already early in the development we did this for critical code like
263passing \LUA\ font tables to \TEX, followed by accessing nodes, but now we have
264done that for most code. There is still some to do but it has the side effect of
265not only consistency but also of helping to document the interface. Of course we
266learn a lot about the \LUA\ internals too. The C macro system is of great help
267here, although the mentioned pascal conversion (web2c) and merged engines have
268resulted in some inconsistency that needs to be cleaned up before we start
269documenting more of the internals (another subproject we want to finish before
270retirement).
271
272\stopsection
273
274\startsection[title=Callbacks]
275
276There are a few more callbacks and most of them come from the tracker. The
277backend now has page related callbacks, the \LUA\ error handler can be
278intercepted. Error messages that consist of multiple pieces are handled better
279too. When a file is opened and closed a callback is now possible. Technically we
280could have combined this with the already present callbacks but as in \TEX\
281synchronization matters these new callbacks relate to current message callbacks
282that show \type {[]}, \type {{}}, \type {<>} and|/|or \type {<<>>} fenced
283filenames, where the later were introduced in successive backend code.
284
285\stopsection
286
287\startsection[title=\LUA]
288
289We currently use \LUA\ 5.2 but a next version will show up soon. Because \LUA\
2905.3 introduces a hybrid number model, this will be one of the next things to play
291with. It could work out well, because \TEX\ is internally integer based (scaled
292points) but you never know. It could be that we need to check existing code for
293serialization and printing issues but normally that will not lead to
294compatibility issues. We could even decide to stick to \LUA\ 5.2 or at least wait
295till all has stabilized. There is some basic support for \UTF\ in 5.3 but in
296\CONTEXT\ we don't depend on that. In practice hardly any processing takes place
297that assumes that \UTF\ is more than a sequence of bytes and \LUA\ can handle
298bytes quite well.
299
300\stopsection
301
302\startsection[title=\CONTEXT]
303
304Of course the development of \LUATEX\ has consequences for \CONTEXT. For
305instance, existing code is used to test alternative solutions and sometimes these
306make it into the core. Some new features are used immediately, like the more
307consistent control over \PDF\ properties, but others have to wait till the new
308binary is more widespread. \footnote {Normally dissemination is rather fast
309because the contextgarden provides recent binaries. The new windows binaries
310often show up within hours after the repository has been updated.}
311
312Some of the improvement in the code base directly relate to \CONTEXT\ activities.
313For instance the \CRITED\ project (complex critical editions) uncovered some
314hashing issues with \LUAJIT\ that have been taken care of now. The (small)
315additions to the \PDF\ backend resulted in a partial cleanup of relatively old
316\CONTEXT\ backend code.
317
318Although some more complex mechanisms, like multi|-|columns are being reworked,
319it is still needed to open up a bit more of the \TEX\ internals, so we have some
320work to do. As usual, version 0.80 doesn't mean that only 0.20 has to be done to
321get to 1.00, as development is not a linear process. The jump from 0.77 to 0.79
322for instance involved a lot of work (exploration as well as testing). But as long
323as it's fun to do, time doesn't matter much. As we've said before: we're in no
324hurry.
325
326\stopsection
327
328\stopchapter
329
330\stopcomponent
331