Source Browser ?

1% language=us
2
3\startcomponent mk-luafitsin
4
5\environment mk-environment
6
7\chapter{How \LUA\ fits in}
8
9\subject{introduction}
10
11Here I will discuss a few of the experiments that drove the
12development of \LUATEX. It describes the state of affairs around
13the time that we were preparing for \TUG\ 2006. This development
14was pretty demanding for Taco and me but also much fun. We were in
15a kind of permanent Skype chat session, with binaries flowing in
16one direction and \TEX\ and \LUA\ code the other way. By gradually
17replacing (even critical) components of \CONTEXT\ we had a real
18test bed and torture tests helped us to explore and debug at the
19same time. Because Taco uses \LINUX\ as platform and I mostly use
20\MSWINDOWS, we could investigate platform dependent issues
21conveniently. While reading this text, keep in mind that this is
22just the beginning of the game.
23
24I will not provide sample code here. When possible, the \MKIV\
25code transparantly replaces \MKII\ code and users will seldom
26notices that something happens in different way. Of course the
27potential is there and future extensions may be unique to \MKIV.
28
29\subject{compatibility}
30
31The first experiments, already conducted with the experimental
32versions involved runtime conversion of one type of input into
33another. An example of this is the (TI) calculator math input
34handler that converts a rather natural math sequence into \TEX\
35and feeds that back into \TEX. This mechanism eventually will
36evolve into a configurable math input handler. Such applications
37are unique to \MKIV\ code and will not be backported to \MKII. The
38question is where downward compatibility will become a problem. We
39don't expect many problems, apart from occasional bugs that result
40from splitting the code base, mostly because new features will not
41affect older functionality. Because we have to reorganize the code
42base a bit, we also use this opportunity to start making a variant
43of \CONTEXT\ which consists of building blocks: \METATEX. This is
44less interesting for the average user, but may be of interest for
45those using \CONTEXT\ in workflows where only part of the
46functionality is needed.
47
48\subject{metapost}
49
50Of course, when I experiment with such new things, I cannot let
51\METAPOST\ leave untouched. And so, in the early stage of \LUATEX\
52development I decided to play with two \METAPOST\ related
53features: conversion and runtime processing.
54
55Conversion from \METAPOST\ output to \PDF\ is currently done in
56pure \TEX\ code. Apart from convenience, this has the advantage
57that we can let \TEX\ take care of font inclusions. The tricky
58part of this conversion is that \METAPOST\ output has some weird
59aspects, like \DVIPS\ specific linewidth snapping. Another nasty
60element in the conversion is that we need to transform paths when
61pens are used. Anyhow, the converter has reached a rather stable
62state by now.
63
64One of the ideas with \METAPOST\ version 1\high{+} is that we will
65have an alternative output mode. In the perspective of \LUATEX\ it
66makes sense to have a \LUA\ output mode. Whatever converter we
67use, it needs to deal with \METAFUN\ specials. These are
68responsible for special features like transparency, graphic
69inclusion, shading, and more. Currently we misuse colors to signal
70such features, but the new pre|/|post path hooks permit more
71advanced implementations. Experimenting with such new features is
72easier in \LUA\ than in \TEX.
73
74The \MKIV\ converter is a multi||pass converter. First we clean up the
75\METAPOST\ output, next we convert the \POSTSCRIPT\ code into \LUA\
76calls. We assume that this \LUA\ code eventually can be output directly
77from \METAPOST. We then evaluate this converted \LUA\ blob, which results
78in \TEX\ commands. Think of:
79
80\starttyping
811.2 setlinejoin
82\stoptyping
83
84turned into:
85
86\starttyping
87mp.setlinejoin(1.2)
88\stoptyping
89
90becoming:
91
92\starttyping
93\PDFcode{1.2 j}
94\stoptyping
95
96which is, when the \PDFTEX\ driver is active, equivalent to:
97
98\starttyping
99\pdfliteral{1.2 j}
100\stoptyping
101
102Of course, when paths are involved, more things happen behind the
103scenes, but in the end an \type {mp.path} enters the \LUA\
104machinery.
105
106When the \MKIV\ converter reached a stable state, tests
107demonstrated then the code was upto 20\% slower that the pure
108\TEX\ alternative on average graphics, and but faster when many
109complex path transformations (due to penshapes) need to be done.
110This slowdown was due to the cleanup (using expressions) and
111intermediate conversion. Because Taco develops \LUATEX\ as well as
112maintains and extends \METAPOST, we conducted experiments that
113combine features of these programs. As a result of this, shortcuts
114found their way into the \METAPOST\ output.
115
116\useMPlibrary[mis]
117
118\placefigure
119  []
120  [fig:mptopdf]
121  {converter test figure}
122  {\scale[width=\hsize]{\useMPgraphic{mptopdf-test}}}
123
124Cleaning up the \METAPOST\ output using \LUA\ expressions takes
125relatively much time. However, starting with version 0.970
126\METAPOST\ uses a preamble, which permits not only short commands,
127but also gets rid of the weird linewidth and filldraw related
128\POSTSCRIPT\ constructs. The moderately complex graphic that we
129use for testing (\in {figure} [fig:mptopdf]) takes over 16 seconds
130when converted 250 times. When we enable shortcuts we can avoid
131part of the cleanup and runtime goes down to under 7.5 seconds.
132This is significantly faster than the \MKII\ code. We did experiments
133with simulated \LUA\ output from \METAPOST\ and then the \MKIV\
134converter really flies. The values on Taco's system are given
135between parenthesis.
136
137\starttabulate[|||||]
138\HL
139\NC \bf prologues/mpprocset \NC \bf 1/0 \NC \bf 1/1 \NC \bf 2/0 \NC \bf 2/1 \NC \NR
140\HL
141\NC \MKII \NC ~8.5 (~5.7) \NC ~8.0 (5.5) \NC ~8.8 \NC ~8.5 \NC \NR
142\NC \MKIV \NC 16.1 (10.6) \NC ~7.2 (4.5) \NC 16.3 \NC ~7.4 \NC \NR
143\HL
144\stoptabulate
145
146The main reason for the huge difference in the \MKIV\ times is
147that we do a rigourous cleanup of the older \METAPOST\ output
148in order avoid messy the messy (but fast) code that we use in
149the \MKII\ converter. Think of:
150
151\starttyping
1520 0.5 dtransform truncate idtransform setlinewidth pop
153closepath gsave fill grestore stroke
154\stoptyping
155
156In the \MKII\ converter, we push every number or keyword on a
157stack and use keywords as trigger points. In the \MKIV\ code
158we convert the stack based \POSTSCRIPT\ calls to \LUA\
159function calls. Lines as shown are converted to single calls
160first. When \type {prologues} is set to~2, such line no longer
161show up and are replaced by simple calls accompanied by
162definitions in the preamble. Not only that, instead of verbose
163keywords, one or two character shortcuts are used. This means
164that the \MKII\ code can be faster when procsets are used
165because shorter strings end up in the stack and comparison
166happens faster. On the other hand, when no procsets are used,
167the runtime is longer because of the larger preamble.
168
169Because the converter is used outside \CONTEXT\ as well, we
170support all combinations in order not to get error messages, but
171the converter is supposed to work with the following settings:
172
173\starttyping
174prologues := 1 ;
175mpprocset := 1 ;
176\stoptyping
177
178We don't need to set \type {prologues} to~2 (font encodings
179in file) or~3 (also font resources in file). So, in the end, the
180comparison in speed comes down to 8.0 seconds for \MKII\ code and
1817.2 seconds for the \MKIV\ code when using the latest greatest
182\METAPOST. When we simulate \LUA\ output from \METAPOST, we end
183up with 4.2 seconds runtime and when \METAPOST\ could produce the
184converter's \TEX\ commands, we need only 0.3 seconds for embedding
185the 250 instances. This includes \TEX\ taking care of handling the
186specials, some of which demand building moderately complex \PDF\
187data structures.
188
189But, conversion is not the only factor in convenient \METAPOST\
190usage. First of all, runtime \METAPOST\ processing takes time. The
191actual time spent on handling embedded \METAPOST\ graphics is also
192dependent on the speed of starting up \METAPOST, which in turn
193depends on the size of the \TEX\ trees used: the bigger these are,
194the more time \KPSE\ spends on loading the \type {ls-R} databases.
195Eventually this bottleneck may go away when we have \METAPOST\ as
196a library. (In \CONTEXT\ one can also run \METAPOST\ between runs.
197Which method is faster, depends on the amount and complexity of
198the graphics.)
199
200Another factor in dealing with \METAPOST, is the usage of text in
201a graphic (\type {btex}, \type {textext}, etc.). Taco Hoekwater,
202Fabrice Popineau and I did some experiments with a persistent
203\METAPOST\ session in the background in order to simulate a
204library. The results look very promising: the overhead of embedded
205\METAPOST\ graphics goes to nearly zero, especially when we also
206let the parent \TEX\ job handle the typesetting of texts. A side
207effect of these experiments was a new mechanism in \CONTEXT\ (and
208\METAFUN) where \TEX\ did all typesetting of labels, and
209\METAPOST\ only worked with an abstract representation of the
210result. This way we can completely avoid nested \TEX\ runs (the
211ones triggered by \METAPOST). This also works ok in \MKII\ mode.
212
213Using a persistent \METAPOST\ run and piping data into it is not
214the final solution if only because the terminal log becomes messed
215up too much, and also because intercepting errors is real messy.
216In the end we need a proper library approach, but the experiments
217demonstrated that we needed to go this way: handling hundreds of
218complex graphics that hold typeset paragraphs (being slanted and
219rotated and more by \METAPOST), tooks mere seconds compared to
220minutes when using independent \METAPOST\ runs for each job.
221
222\subject{characters}
223
224Because \LUATEX\ is \UTF\ based, we need a different way to deal with
225input encoding. For this purpose there are callbacks that intercept
226the input and convert it as needed. For context this means that the
227regime related modules get a \LUA\ based counterparts. As a prelude to
228advanced character manipulations, we already load extensive unicode
229and conversion tables, with the benefit of being able to handle case
230handling with \LUA.
231
232The character tables are derived from unicode tables and \MKII\
233\CONTEXT\ data files and generated using \MTXTOOLS. The main
234character table is pretty large, and this made us experiment a bit
235with efficiency. It was in this stage that we realized that it
236made sense to use precompiled \LUA\ code (using \type {luac}).
237During format generation we let \CONTEXT\ keep track of used \LUA\
238files and compiled them on the fly. For a production run, the
239compiled files were loaded instead.
240
241Because at that stage \LUATEX\ was already a merge between
242\PDFTEX\ and \ALEPH, we had to deal with pretty large format
243files. About that moment the \CONTEXT\ format with the english
244user interface amounted to:
245
246\starttabulate[|c|c|c|c|c|]
247\NC \bf date   \NC \bf luatex \NC \bf pdftex \NC \bf xetex \NC \bf aleph \NC \NR
248\NC 2006-09-18 \NC 9 552 042  \NC 7 068 643  \NC 8 374 996 \NC 7 942 044 \NC \NR
249\stoptabulate
250
251One reason for the large size of the format file is that the
252memory footprint of a 32 bit \TEX\ is larger than that of good old
253\TEX, even with some of the clever memory allocation techniques as
254used in \LUATEX. After some experiments where size and speed were
255measured Taco decided to compress the format using a level~3 \ZIP\
256compression. This brilliant move lead to the following size:
257
258\starttabulate[|c|c|c|c|c|]
259\NC \bf date   \NC \bf luatex \NC \bf pdftex \NC \bf xetex \NC \bf aleph \NC \NR
260\NC 2006-10-23 \NC 3 135 568  \NC 7 095 775  \NC 8 405 764 \NC 7 973 940 \NC \NR
261\stoptabulate
262
263The first zipped versions were smaller (around 2.3 meg), but in
264the meantime we moved the \LUA\ code into the format and the
265character related tables take some space.
266
267\start \it How stable are the mentioned numbers? Ten months after writing the
268previous text we get the following numbers: \stop
269
270\starttabulate[|c|c|c|c|c|]
271\NC \bf date   \NC \bf luatex \NC \bf pdftex \NC \bf xetex \NC \bf aleph \NC \NR
272\NC 2007-08-16 \NC 5 603 676  \NC 7 505 925  \NC 8 838 538 \NC 8 369 206 \NC \NR
273\stoptabulate
274
275They are all some 400K larger, which is probably the result of changes in
276hyphenation patterns (we now load them all, some several times depending on the
277font encodings used). Also, some extra math support has been brought in the kernel
278and we predefine a few more things. However, \LUATEX's format has become much
279larger! Partly this is the result of more \LUA\ code, especially \OPENTYPE\ font
280handling and attributes related code. The extra \TEX\ code is probably compensated
281by the removal of obsolete (at least for \MKIV) code. However, the significantly
282larger number is mostly there because a different compression algorithm is used:
283speed is now favoured over efficiency.
284
285\subject{debugging}
286
287In the process of experimenting with callbacks I played a bit with
288handling \TEX\ error information. An option is to generate an
289\HTML\ page instead of spitting out the usual blob of into on the
290terminal. In \in {figure} [fig:error] and \in {figure} [fig:debug]
291you can see an example of this.
292
293\placefigure[][fig:error]{An example error screen.}{\externalfigure[mk-error.png][width=\textwidth]}
294\placefigure[][fig:debug]{An example debug screen.}{\externalfigure[mk-debug.png][width=\textwidth]}
295
296Playing with such features gives us an impression of what kind of
297access we need to \TEX's internals. It also formed a starting
298point for conversion routines and a mechanism for embedding \LUA\
299code in \HTML\ pages generated by \CONTEXT.
300
301\subject{file io}
302
303Replacing \TEX's in- and output handling is non||trival. Not only
304is the code quite interwoven in the \WEBC\ source, but there is also
305the \KPSE\ library to deal with. This means that quite some callbacks
306are needed to handle the different types of files. Also, there is output
307to the log and terminal to take care of.
308
309Getting this done took us quite some time and testing and
310debugging was good for some headaches. The mechanisms changed a
311few times, and \TEX\ and \LUA\ code was thrown away as soon as
312better solutions came around. Because we were testing on real
313documents, using a fully loaded \CONTEXT\ we could converge to a
314stable version after a while.
315
316Getting this \IO\ stuff done is tightly related to generating the
317format and starting up \LUATEX. If you want to overload the file
318searching and \IO\ handling, you need overload as soon as possible.
319Because \LUATEX\ is also supposed to work with the existing \KPSE\
320library, we still have that as fallback, but in principle one
321could think of a \KPSE\ free version, in which case the default
322file searching is limited to the local path and memory
323initialization also reverts to the hard coded defaults. A
324complication is that the soure code has \KPSE\ calls and
325references to \KPSE\ variables all over the place, so occasionally
326we run into interesting bugs.
327
328Anyhow, while Taco hacked his way around the code, I converted my
329existing \RUBY\ based \KPSE\ variant into \LUA\ and started working
330from that point. The advantage of having our own \IO\ handler is
331that we can go beyond \KPSE. For instance, since \LUATEX\ has,
332among a few others, the \ZIP\ libraries linked in, we can read from
333\ZIP\ files, and keep all \TEX\ related files in \TDS\ compliant \ZIP\
334files as well. This means that one can say:
335
336\starttyping
337\input zip:///somezipfile.zip?name=/somepath/somefile.tex
338\stoptyping
339
340and use similar references to access files. Of course we had to make
341sure that \KPSE\ like searching in the \TDS\ (standardized \TEX\ trees)
342works smoothly. There are plans to link the curl library into \LUATEX,
343so that we can go beyong this and access repositories.
344
345Of course, in order to be more or less \KPSE\ and \WEBC\
346compliant, we also need to support this paranoid file handling, so
347we provide mechanisms for that as well. In addition, we provide
348ways to create sandboxes for system calls.
349
350Getting to intercept all log output (well, most log output) was
351a problem in itself. For this I used a (preliminary) \XML\ based
352log format, which will make log parsing easier. Because we have
353full control over file searching, opening and closing, we can
354also provide more information about what files are loaded. For
355instance we can now easily trace what \TFM\ files \TEX\ reads.
356
357Implementing additional methods for locating and opening files is
358not that complex because the library that ships with \CONTEXT\
359is already prepared for this. For instance, implementing support
360for:
361
362\starttyping
363\input http://www.someplace.org/somepath/somefile.tex
364\stoptyping
365
366involved a few lines of code, most of which deals with caching the
367files. Because we overload the whole \IO\ handling, this means that
368the following works ok:
369
370% \bgroup \loggingall
371
372\startbuffer
373\placefigure
374  [][]
375  {http handling}
376  {\externalfigure
377     [http://www.pragma-ade.com/show-gra.pdf]
378     [page=1,width=\textwidth]}
379\stopbuffer
380
381\typebuffer \ifx\ctxlua \undefined \else \getbuffer \fi
382
383% \egroup
384
385Other protocols, like \FTP\ are also supported, so one can say:
386
387\starttyping
388\typefile {ftp://anonymous:@ctan.org/tex-archive/systems\
389    /knuth/lib/plain.tex}
390\stoptyping
391
392On the agenda is playing with database, but by the time that we enter
393that stage linking the \type {curl} libraries into \LUATEX\ should
394have taken place.
395
396\subject{verbatim}
397
398The advance of \LUATEX\ also permitted us to play with a long
399standing wish of catcode tables, a mechanism to quickly switch
400between different ways of treating input characters. An example of
401a place where such changes take place is verbatim (and in \CONTEXT\
402also when dealing with \XML\ input).
403
404We already had encountered the phenomena that when piping back
405results from \LUA\ to \TEX, we needed to take care of catcodes so
406that \TEX\ would see the input as we wished. Earlier experiments
407with applying \type {\scantokens} to a result and thereby
408interpreting the result conforming the current catcode regime was
409not sufficient or at least not handy enough, especially in the
410perspective of fully expandable \LUA\ results. To be honest, the \type
411{\scantokens} command was rather useless for this purposes due to its
412pseudo file nature and its end||of||file handling but in \LUATEX\
413we now have a convenient \type {\scantextokens} which has no side
414effects.
415
416Once catcode tables were in place, and the relevant \CONTEXT\ code
417adapted, I could start playing with one of the trickier parts of
418\TEX\ programming: typesetting \TEX\ using \TEX, or verbatim.
419Because in \CONTEXT\ verbatim is also related to buffering and
420pretty printing, all these mechanism were handled at once. It
421proved to be a pretty good testcase for writing \LUA\ results back
422to \TEX, because anything you can imagine can and will interfere
423(line endings, catcode changes, looking ahead for arguments, etc).
424This is one of the areas where \MKIV\ code will make things look
425more clean and understandable, especially because we could move
426all kind of postprocessing (needed for pretty printing, i.e.\
427syntax highlighting) to \LUA. Interesting is that the resulting
428code is not beforehand faster.
429
430Pretty printing 1000 small (one line) buffers and 5000 simple
431\type {\type} commands perform as follows:
432
433\starttabulate[|l|c|c|c|c|]
434\NC        \NC \TEX\ normal \NC \TEX\ pretty \NC \LUA\ normal \NC \LUA\ pretty \NC \NR
435\NC buffer \NC 2.5 (2.35)   \NC ~4.5 (3.05)  \NC 2.2 (1.8)    \NC ~2.5 (2.0)   \NC \NR
436\NC inline \NC 7.7 (4.90)   \NC 11.5 (7.25)  \NC 9.1 (6.3)    \NC 10.9 (7.5)   \NC \NR
437\stoptabulate
438
439Between braces the runtime on Taco's more modern machine is shown.
440It's not that easy to draw conclusions from this because \TEX\
441uses files for buffers and with \LUA\ we store buffers in memory.
442For inline verbatim, \LUA\ call's bring some overhead, but with
443more complex content, this becomes less noticable. Also, the \LUA\
444code is probably less optimized than the \TEX\ code, and we don't
445know yet what benefits a Just In Time \LUA\ compiler will bring.
446
447\subject{xml}
448
449Interesting is that the first experiments with \XML\ processing
450don't show the expected gain in speed. This is due to the fact
451that the \CONTEXT\ \XML\ parser is highly optimized. However, if
452we want to load a whole \XML\ file, for instance the formal
453\CONTEXT\ interface specification \type {cont-en.xml}, then we can
454bring down loading time (as well as \TEX\ memory usage) down from
455multiple seconds to a blink of the eyes. Experiments with internal
456mappings and manipulations demonstrated that we may not so much
457need an alternative for the current parser, but can add additional,
458special purpose ones.
459
460We may consider linking \XSLTPROC\ into \LUATEX, but this is yet
461undecided. After all, the problem of typesetting does not really
462change, so we may as well keep the process of manipulating and
463typesetting separated.
464
465\subject{multipass data}
466
467Those who know \CONTEXT\ a bit will know that it may need multiple
468passes to typeset a document. \CONTEXT\ not only keeps track of
469index entries, list entries, cross references, but also optimizes
470some of the output based on information gathered in previous
471passes. Especially so called two||pass data and positional
472information puts some demands on memory and runtime. Two||pass
473data is collapsed in lists because otherwise we would run out of
474memory (at least this was true years ago when these mechanisms
475were introduced). Positional information is stored in hashes and
476has always put a bit of a burden on the size of a so called
477utility file (\CONTEXT\ stores all information in one auxiliary
478file).
479
480These two datatypes were the first we moved to a \LUA\ auxiliary
481file and eventually all information will move there. The advantage
482is that we can use efficient hashes (without limitations) and only
483need to run over the file once. And \LUA\ is incredibly fast in
484loading the tables where we keep track of these things. For
485instance, a test file storing and reading 10.000 complex positions
486takes 3.2 seconds runtime with \LUATEX\ but 8.7 seconds with
487traditional \PDFTEX. Imagine what this will save when dealing with
488huge files (400 page 300 Meg files) that need three or more passes
489to be typeset. And, now we can without problems bump position
490tracking to milions of positions.
491
492\subject{resources}
493
494Finding files is somewhat tricky and has a history in the \TEX\
495community and its distributions. For reasons of packaging and
496searching files are organized in a tree and there are rules for
497locating files of given types in this tree. When we say
498
499\starttyping
500\input blabla.tex
501\stoptyping
502
503\TEX\ will look for this file by consulting the path specification
504associated with the filetype. When we say
505
506\starttyping
507\input blabla
508\stoptyping
509
510\TEX\ will add the \type {.tex} suffix itself. Most other filetypes
511are not seen by users but are dealt with in a similar way internally.
512
513As mentioned before, we support reading from other resources than
514the standard file system, for instance we can input files from
515websites or read from \ZIP\ archives. Although this works quite well,
516we need to keep in mind that there are some conflicting interests:
517structured search based on type related specifications versus more
518or less explicit requests.
519
520\starttyping
521\input zip:///archive.zip?name=blabla.tex
522\input zip:///archive.zip?name=/somepath/blabla.tex
523\stoptyping
524
525Here we need to be rather precise in defining the file location. We can
526of course build rather complex mechanisms for locating files here, but
527at some point that may backfire and result in unwanted matches.
528
529If you want to treat a \ZIP\ archive as a \TEX\ tree, then you need
530to register the file:
531
532\starttyping
533\usezipfile[archive.zip]
534\usezipfile[tex.zip][texmf-local]
535\usezipfile[tex.zip?tree=texmf-local]
536\stoptyping
537
538The first variant registers all files in the archive, but the
539next two are equivalent and only register a subtree. The registered
540tree is prepended to the \type {TEXMF} specification and thereby
541may overload existing trees.
542
543If an acrhive is not a real \TEX\ tree, you can access files anywhere
544in the tree by using wildcards
545
546\starttyping
547\input */blabla.tex
548\input */somepath/blabla.tex
549\stoptyping
550
551These mechanisms evolve over time and it may take a while before they
552stabelize. For instance, the syntax for the \ZIP\ inclusion has been
553adapted more than a year after this chapter was written (which is
554why this section is added).
555
556\stopcomponent
557