Source Browser ?

1% language=us runpath=texruns:manuals/luametatex
2
3\environment luametatex-style
4
5\startcomponent luametatex-lua
6
7\startchapter[reference=lua,title={Using \LUAMETATEX}]
8
9\startsection[title={Initialization},reference=init]
10
11\startsubsection[title={A bare bone engine}]
12
13Although the \LUAMETATEX\ engine will start up when you call the program it will
14not do much useful. You can compare it to computer hardware without (high level)
15operating system with a \TEX\ kernel being the bios. It can interpret \TEX\ code
16but for typesetting you need a reasonable setup. You also need to load fonts, and
17for output you need a backend, and both can be implemented in \LUA. If you don't
18like that and want to get up and running immediately, you will be more happy with
19\LUATEX, \PDFTEX\ or \XETEX, combined with your favorite macro package.
20
21If you just want to play around you can install the \CONTEXT\ distribution which
22(including manuals and some fonts) is tiny compared to a full \TEXLIVE\
23installation and can be run alongside it without problems. If there are issues
24you can go to the usual \CONTEXT\ support platforms and seek help where you can
25find the people who made \LUATEX\ and \LUAMETATEX.
26
27\stopsubsection
28
29\startsubsection[title={\LUAMETATEX\ as a \LUA\ interpreter}]
30
31\topicindex {initialization}
32\topicindex {\LUA+interpreter}
33
34Although \LUAMETATEX\ is primarily meant as a \TEX\ engine, it can also serve as
35a stand alone \LUA\ interpreter. There are two ways to make \LUAMETATEX\ behave
36like a standalone \LUA\ interpreter. The first method uses the command line
37option \type {--luaonly} followed by a filename. The second is more automatic: if
38the only non|-|option argument (file) on the commandline has the extension \type
39{lmt} or \type {lua}. The \type {luc} extension has been dropped because bytecode
40compiled files are not portable and one can always load indirect. The \type {lmt}
41suffix is more \CONTEXT\ specific and makes it possible to have files for
42\LUATEX\ and \LUAMETATEX\ alongside.
43
44In this mode, it will set \LUA's \type {arg[0]} to the found script name, pushing
45preceding options in negative values and the rest of the command line in the
46positive values, just like the \LUA\ interpreter does.
47
48\LUAMETATEX\ will exit immediately after executing the specified \LUA\ script and
49is, in effect, a somewhat bulky stand alone \LUA\ interpreter with a bunch of
50extra preloaded libraries. But we really want to keep the binary small, if
51possible below the 3MB which is okay for a script engine.
52
53When no argument is given, \LUAMETATEX\ will look for a \LUA\ file with the same
54name as the binary and run that one when present. This makes it possible to use
55the engine as a stub. For instance, in \CONTEXT\ a symlink from \type {mtxrun} to
56type {luametatex} will run the \type {mtxrun.lmt} or \type {mtxrun.lua} script
57when present in the same path as the binary itself. As mentioned before first
58checking for (\CONTEXT) \type {lmt} files permits different files for different
59engines in the same path.
60
61\stopsubsection
62
63\startsubsection[title={Other commandline processing}]
64
65\topicindex {command line}
66
67When the \LUAMETATEX\ executable starts, it looks for the \type {--lua} command line
68option. If there is no \type {--lua} option, the command line is interpreted in a
69similar fashion as the other \TEX\ engines. All options are accepted but only some
70are understood by \LUAMETATEX\ itself:
71
72\starttabulate[|l|p|]
73\DB commandline argument    \BC explanation \NC \NR
74\TB
75\NC \type{--credits}        \NC display credits and exit \NC \NR
76\NC \type{--fmt=FORMAT}     \NC load the format file \type {FORMAT} \NC\NR
77\NC \type{--help}           \NC display help and exit \NC\NR
78\NC \type{--ini}            \NC be \type {iniluatex}, for dumping formats \NC\NR
79\NC \type{--jobname=STRING} \NC set the job name to \type {STRING} \NC \NR
80\NC \type{--lua=FILE}       \NC load and execute a \LUA\ initialization script \NC\NR
81\NC \type{--version}        \NC display version and exit \NC \NR
82\LL
83\stoptabulate
84
85There are less options than with \LUATEX, because one has to deal with them in
86\LUA\ anyway. So for instance there are no options to enter a safer mode or
87control executing programs because this can easily be achieved with a startup
88\LUA\ script.
89
90Next the initialization script is loaded and executed. From within the script,
91the entire command line is available in the \LUA\ table \type {arg}, beginning
92with \type {arg[0]}, containing the name of the executable. As consequence
93warnings about unrecognized options are suppressed.
94
95Command line processing happens very early on. So early, in fact, that none of
96\TEX's initializations have taken place yet. The \LUA\ libraries that don't deal
97with \TEX\ are initialized rather soon so you have these available.
98
99\LUAMETATEX\ allows some of the command line options to be overridden by reading
100values from the \type {texconfig} table at the end of script execution (see the
101description of the \type {texconfig} table later on in this document for more
102details on which ones exactly).
103
104The value to use for \prm {jobname} is decided as follows:
105
106\startitemize
107\startitem
108    If \type {--jobname} is given on the command line, its argument will be the
109    value for \prm {jobname}, without any changes. The argument will not be
110    used for actual input so it need not exist. The \type {--jobname} switch only
111    controls the \prm {jobname} setting.
112\stopitem
113\startitem
114    Otherwise, \prm {jobname} will be the name of the first file that is read
115    from the file system, with any path components and the last extension (the
116    part following the last \type {.}) stripped off.
117\stopitem
118\startitem
119    There is an exception to the previous point: if the command line goes into
120    interactive mode (by starting with a command) and there are no files input
121    via \prm {everyjob} either, then the \prm {jobname} is set to \type
122    {texput} as a last resort.
123\stopitem
124\stopitemize
125
126So let's summarize this. The handling of what is called jobname is a bit complex.
127There can be explicit names set on the command line but when not set they can be
128taken from the \type {texconfig} table.
129
130\starttabulate[|l|T|T|T|]
131\NC startup filename \NC --lua     \NC a \LUA\ file  \NC                      \NC \NR
132\NC startup jobname  \NC --jobname \NC a \TEX\ tex   \NC texconfig.jobname    \NC \NR
133\NC startup dumpname \NC --fmt     \NC a format file \NC texconfig.formatname \NC \NR
134\stoptabulate
135
136These names are initialized according to \type {--luaonly} or the first filename
137seen in the list of options. Special treatment of \type {&} and \type {*} as well
138as interactive startup is gone but we still enter \TEX\ via an forced \type {\input}
139into the input buffer. \footnote {This might change at some point into an explicit
140loading triggered via \LUA.}
141
142When we are in \TEX\ mode at some point the engine needs a filename, for instance
143for opening a log file. At that moment the set jobname becomes the internal one
144and when it has not been set which internalized to jobname but when not set
145becomes \type {texput}. When you see a \type {texput.log} file someplace on your
146system it normally indicates a bad run.
147
148When running on \MSWINDOWS\ the command line, filenames, environment variable
149access etc.\ internally uses the current code page but to the user is exposed as
150\UTF8. Normally users won't notice this.
151
152% fileio_state     .jobname         : a tex string (set when a (log) file is opened)
153% engine_state     .startup_jobname : handles by option parser
154% environment_state.input_name      : temporary interceptor
155
156There is an extra options \type{--permitloadlib} that needs to be given when you
157load external libraries via \LUA. Although you could manage this via \LUA\ itself
158in a startup script, the reason for having this as option is the wish for
159security (at some point that became a demand for \LUATEX), so this might give an
160extra feeling of protection.
161
162\stopsubsection
163
164\stopsection
165
166\startsection[title={\LUA\ behaviour}]
167
168\startsubsection[title={The \LUA\ version}]
169
170\topicindex {\LUA+libraries}
171\topicindex {\LUA+extensions}
172
173We currently use \LUA\ 5.4 and will follow developments of the language but
174normally with some delay. Therefore the user needs to keep an eye on (subtle)
175differences in successive versions of the language. Here is an example of one
176aspect.
177
178\LUA s \type {tostring} function (and \type {string.format}) may return values in
179scientific notation, thereby confusing the \TEX\ end of things when it is used as
180the right|-|hand side of an assignment to a \prm {dimen} or \prm {count}. The
181output of these serializers also depend on the \LUA\ version, so in \LUA\ 5.3 you
182can get different output than from 5.2. It is best not to depend the automatic
183cast from string to number and vise versa as this can change in future versions.
184
185\stopsubsection
186
187\startsubsection[title={Locales}]
188
189\index {locales}
190
191In stock \LUA, many things depend on the current locale. In \LUAMETATEX, we can't
192do that, because it makes documents non|-|portable. While \LUAMETATEX\ is running
193if forces the following locale settings:
194
195\starttyping
196LC_CTYPE=C
197LC_COLLATE=C
198LC_NUMERIC=C
199\stoptyping
200
201There is no way to change that as it would interfere badly with the often
202language specific conversions needed at the \TEX\ end.
203
204\stopsubsection
205
206\stopsection
207
208\startsection[title={\LUA\ modules}]
209
210\topicindex {\LUA+libraries}
211\topicindex {\LUA+modules}
212
213Of course the regular \LUA\ modules are present. In addition we provide the \type
214{lpeg} library by Roberto Ierusalimschy, This library is not \UNICODE|-|aware,
215but interprets strings on a byte|-|per|-|byte basis. This mainly means that \type
216{lpeg.S} cannot be used with \UTF8 characters that need more than one byte, and
217thus \type {lpeg.S} will look for one of those two bytes when matching, not the
218combination of the two. The same is true for \type {lpeg.R}, although the latter
219will display an error message if used with multibyte characters. Therefore \type
220{lpeg.R('aä')} results in the message \type {bad argument #1 to 'R' (range must
221have two characters)}, since to \type {lpeg}, \type {ä} is two 'characters'
222(bytes), so \type {aä} totals three. In practice this is no real issue and with
223some care you can deal with \UNICODE\ just fine.
224
225There are some more libraries present. These are discussed on a later chapter.
226For instance we embed \type {luasocket} but contrary to \LUATEX\ don't embed the
227related \LUA\ code. The \type {luafilesystem} module has been replaced by a more
228efficient one that also deals with the \MSWINDOWS\ file and environment
229properties better (\UNICODE\ support in \MSWINDOWS\ dates from before \UTF8
230became dominant so we need to deal with wide \UNICODE16).
231
232There are more extensive math libraries and there are libraries that deal with
233encryption and compression. There are also some optional libraries that we do
234interface but that are loaded on demand. The interfaces are as minimal as can be
235because we so much in \LUA, which also means that one can tune behaviour to
236usage better.
237
238\stopsection
239
240\startsection[title={Files}]
241
242\startsubsection[title={File syntax}]
243
244\topicindex {files+names}
245
246\LUAMETATEX\ will accept a braced argument as a file name:
247
248\starttyping
249\input {plain}
250\openin 0 {plain}
251\stoptyping
252
253This allows for embedded spaces, without the need for double quotes. Macro
254expansion takes place inside the argument. Keep in mind that as side effect of
255delegating \IO\ to \LUA\ the \tex {openin} primitive is not provided by the
256engine and has to be implemented by the macro package. This also means that the
257limit on the number of open files is not enforced by the engine.
258
259\stopsubsection
260
261\startsubsection[title={Writing to file}]
262
263\topicindex {files+writing}
264
265Writing to a file in \TEX\ has two forms: delayed and immediate. Delayed writing
266means that the to be written text is anchored in the node list and flushed by the
267backend. As all \IO\ is delegated to \LUA, this also means that it has to deal
268with distinction. In \LUATEX\ the number of open files was already bumped to 127,
269but in \LUAMETATEX\ it depends on the macro package. The special meaning of
270channel 18 was already dropped in \LUATEX\ because we have \type {os.execute}.
271
272\stopsubsection
273
274\stopsection
275
276\startsection[title={Testing}]
277
278\topicindex {testing}
279
280For development reasons you can influence the used startup date and time. By
281setting the \type {start_time} variable in the \type {texconfig} table; as with
282other variables we use the internal name there. When Universal Time is needed,
283set the entry \type {use_utc_time} in the \type {texconfig} table.
284
285In \CONTEXT\ we provide the command line argument \type {--nodates} that does
286a bit more than disabling dates; it avoids time dependent information in the
287output file for instance.
288
289\stopsection
290
291\startsection[reference=internals,title={The internals}]
292
293\topicindex{nodes}
294\topicindex{boxes}
295\topicindex{\LUA}
296
297This is a reference manual and not a tutorial. This means that we discuss changes
298relative to traditional \TEX\ and also present new (or extended) functionality.
299As a consequence we will refer to concepts that we assume to be known or that
300might be explained later. Because the \LUATEX\ and \LUAMETATEX\ engines open up
301\TEX\ there's suddenly quite some more to explain, especially about the way a (to
302be) typeset stream moves through the machinery. However, discussing all that in
303detail makes not much sense, because deep knowledge is only relevant for those
304who write code not possible with regular \TEX\ and who are already familiar with
305these internals (or willing to spend time on figuring it out).
306
307So, the average user doesn't need to know much about what is in this manual. For
308instance fonts and languages are normally dealt with in the macro package that
309you use. Messing around with node lists is also often not really needed at the
310user level. If you do mess around, you'd better know what you're dealing with.
311Reading \quotation {The \TEX\ Book} by Donald Knuth is a good investment of time
312then also because it's good to know where it all started. A more summarizing
313overview is given by \quotation {\TEX\ by Topic} by Victor Eijkhout. You might
314want to peek in \quotation {The \ETEX\ manual} too.
315
316But \unknown\ if you're here because of \LUA, then all you need to know is that
317you can call it from within a run. If you want to learn the language, just read
318the well written \LUA\ book. The macro package that you use probably will provide
319a few wrapper mechanisms but the basic \prm {directlua} command that does the job
320is:
321
322\starttyping
323\directlua{tex.print("Hi there")}
324\stoptyping
325
326You can put code between curly braces but if it's a lot you can also put it in a
327file and load that file with the usual \LUA\ commands. If you don't know what
328this means, you definitely need to have a look at the \LUA\ book first.
329
330If you still decide to read on, then it's good to know what nodes are, so we do a
331quick introduction here. If you input this text:
332
333\starttyping
334Hi There ...
335\stoptyping
336
337eventually we will get a linked lists of nodes, which in \ASCII\ art looks like:
338
339\starttyping
340H <=> i <=> [glue] <=> T <=> h <=> e <=> r <=> e ...
341\stoptyping
342
343When we have a paragraph, we actually get something like this, where a \type
344{par} node stores some metadata and is followed by a \type {hlist} flagged as
345indent box:
346
347\starttyping
348[par] <=> [hlist] <=> H <=> i <=> [glue] <=> T <=> h <=> e <=> r <=> e ...
349\stoptyping
350
351Each character becomes a so called glyph node, a record with properties like the
352current font, the character code and the current language. Spaces become glue
353nodes. There are many node types and nodes can have many properties but that will
354be discussed later. Each node points back to a previous node or next node, given
355that these exist. Sometimes multiple characters are represented by one glyph
356(shape), so one can also get:
357
358\starttyping
359[par] <=> [hlist] <=> H <=> i <=> [glue] <=> Th <=> e <=> r <=> e ...
360\stoptyping
361
362And maybe some characters get positioned relative to each other, so we might
363see:
364
365\starttyping
366[par] <=> [hlist] <=> H <=> [kern] <=> i <=> [glue] <=> Th <=> e <=> r <=> e ...
367\stoptyping
368
369Actually, the above representation is one view, because in \LUAMETATEX\ we can
370choose for this:
371
372\starttyping
373[par] <=> [glue] <=> H <=> [kern] <=> i <=> [glue] <=> Th <=> e <=> r <=> e ...
374\stoptyping
375
376where glue (currently fixed) is used instead of an empty hlist (think of a \type
377{\hbox}). Options like this are available because want a certain view on these
378lists from the \LUA\ end and the result being predicable is part of that.
379
380It's also good to know beforehand that \TEX\ is basically centered around
381creating paragraphs and pages. The par builder takes a list and breaks it into
382lines. At some point horizontal blobs are wrapped into vertical ones. Lines are
383so called boxes and can be separated by glue, penalties and more. The page
384builder accumulates lines and when feasible triggers an output routine that will
385take the list so far. Constructing the actual page is not part of \TEX\ but done
386using primitives that permit manipulation of boxes. The result is handled back to
387\TEX\ and flushed to a (often \PDF) file.
388
389\starttyping
390\setbox\scratchbox\vbox\bgroup
391    line 1\par line 2
392\egroup
393
394\showbox\scratchbox
395\stoptyping
396
397The above code produces the next log lines that reveal how the engines sees a
398paragraph (wrapped in a \type {\vbox}):
399
400\starttyping[style=small]
4011:4: > \box257=
4021:4: \vbox[normal][16=1,17=1,47=1], width 483.69687, height 27.58083, depth 0.1416, direction l2r
4031:4: .\list
4041:4: ..\hbox[line][16=1,17=1,47=1], width 483.69687, height 7.59766, depth 0.1416, glue 455.40097fil, direction l2r
4051:4: ...\list
4061:4: ....\glue[left hang][16=1,17=1,47=1] 0.0pt
4071:4: ....\glue[left][16=1,17=1,47=1] 0.0pt
4081:4: ....\glue[parfillleft][16=1,17=1,47=1] 0.0pt
4091:4: ....\par[newgraf][16=1,17=1,47=1], hangafter 1, hsize 483.69687, pretolerance 100, tolerance 3000, adjdemerits 10000, linepenalty 10, doublehyphendemerits 10000, finalhyphendemerits 5000, clubpenalty 2000, widowpenalty 2000, brokenpenalty 100, emergencystretch 12.0, parfillskip 0.0pt plus 1.0fil, hyphenationmode 499519
4101:4: ....\glue[indent][16=1,17=1,47=1] 0.0pt
4111:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+00006C l
4121:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+000069 i
4131:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+00006E n
4141:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+000065 e
4151:4: ....\glue[space][16=1,17=1,47=1] 3.17871pt plus 1.58936pt minus 1.05957pt, font 30
4161:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+000031 1
4171:4: ....\penalty[line][16=1,17=1,47=1] 10000
4181:4: ....\glue[parfill][16=1,17=1,47=1] 0.0pt plus 1.0fil
4191:4: ....\glue[right][16=1,17=1,47=1] 0.0pt
4201:4: ....\glue[right hang][16=1,17=1,47=1] 0.0pt
4211:4: ..\glue[par][16=1,17=1,47=1] 5.44995pt plus 1.81665pt minus 1.81665pt
4221:4: ..\glue[baseline][16=1,17=1,47=1] 6.79396pt
4231:4: ..\hbox[line][16=1,17=1,47=1], width 483.69687, height 7.59766, depth 0.1416, glue 455.40097fil, direction l2r
4241:4: ...\list
4251:4: ....\glue[left hang][16=1,17=1,47=1] 0.0pt
4261:4: ....\glue[left][16=1,17=1,47=1] 0.0pt
4271:4: ....\glue[parfillleft][16=1,17=1,47=1] 0.0pt
4281:4: ....\par[newgraf][16=1,17=1,47=1], hangafter 1, hsize 483.69687, pretolerance 100, tolerance 3000, adjdemerits 10000, linepenalty 10, doublehyphendemerits 10000, finalhyphendemerits 5000, clubpenalty 2000, widowpenalty 2000, brokenpenalty 100, emergencystretch 12.0, parfillskip 0.0pt plus 1.0fil, hyphenationmode 499519
4291:4: ....\glue[indent][16=1,17=1,47=1] 0.0pt
4301:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+00006C l
4311:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+000069 i
4321:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+00006E n
4331:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+000065 e
4341:4: ....\glue[space][16=1,17=1,47=1] 3.17871pt plus 1.58936pt minus 1.05957pt, font 30
4351:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+000032 2
4361:4: ....\penalty[line][16=1,17=1,47=1] 10000
4371:4: ....\glue[parfill][16=1,17=1,47=1] 0.0pt plus 1.0fil
4381:4: ....\glue[right][16=1,17=1,47=1] 0.0pt
4391:4: ....\glue[right hang][16=1,17=1,47=1] 0.0pt
440\stoptyping
441
442The \LUATEX\ engine provides hooks for \LUA\ code at nearly every reasonable
443point in the process: collecting content, hyphenating, applying font features,
444breaking into lines, etc. This means that you can overload \TEX's natural
445behaviour, which still is the benchmark. When we refer to \quote {callbacks} we
446means these hooks. The \TEX\ engine itself is pretty well optimized but when you
447kick in much \LUA\ code, you will notices that performance drops. Don't blame and
448bother the authors with performance issues. In \CONTEXT\ over 50\% of the time
449can be spent in \LUA, but so far we didn't get many complaints about efficiency.
450Adding more callbacks makes no sense, also because at some point the performance
451hit gets too large. There are plenty of ways to achieve goals. For that reason:
452take remarks about \LUATEX, features, potential, performance etc.\ with a natural
453grain of salt.
454
455Where plain \TEX\ is basically a basic framework for writing a specific style,
456macro packages like \CONTEXT\ and \LATEX\ provide the user a whole lot of
457additional tools to make documents look good. They hide the dirty details of font
458management, language support, turning structure into typeset results, wrapping
459pages, including images, and so on. You should be aware of the fact that when you
460hook in your own code to manipulate lists, this can interfere with the macro
461package that you use. Each successive step expects a certain result and if you
462mess around to much, the engine eventually might bark and quit. It can even
463crash, because testing everywhere for what users can do wrong is no real option.
464
465When you read about nodes in the following chapters it's good to keep in mind
466what commands relate to them. Here are a few:
467
468\starttabulate[|l|l|p|]
469\DB command              \BC node          \BC explanation \NC \NR
470\TB
471\NC \prm {hbox}          \NC \nod {hlist} \NC horizontal box \NC \NR
472\NC \prm {vbox}          \NC \nod {vlist} \NC vertical box with the baseline at the bottom \NC \NR
473\NC \prm {vtop}          \NC \nod {vlist} \NC vertical box with the baseline at the top \NC \NR
474\NC \prm {hskip}         \NC \nod {glue}  \NC horizontal skip with optional stretch and shrink \NC \NR
475\NC \prm {vskip}         \NC \nod {glue}  \NC vertical skip with optional stretch and shrink \NC \NR
476\NC \prm {kern}          \NC \nod {kern}  \NC horizontal or vertical fixed skip \NC \NR
477\NC \prm {discretionary} \NC \nod {disc}  \NC hyphenation point (pre, post, replace) \NC \NR
478\NC \prm {char}          \NC \nod {glyph} \NC a character \NC \NR
479\NC \prm {hrule}         \NC \nod {rule}  \NC a horizontal rule \NC \NR
480\NC \prm {vrule}         \NC \nod {rule}  \NC a vertical rule \NC \NR
481\NC \prm {textdirection} \NC \nod {dir}   \NC a change in text direction \NC \NR
482\LL
483\stoptabulate
484
485Whatever we feed into \TEX\ at some point becomes a token which is either
486interpreted directly or stored in a linked list. A token is just a number that
487encodes a specific command (operator) and some value (operand) that further
488specifies what that command is supposed to do. In addition to an interface to
489nodes, there is an interface to tokens, as later chapters will demonstrate.
490
491Text (interspersed with macros) comes from an input medium. This can be a file,
492token list, macro body cq.\ arguments, some internal quantity (like a number),
493\LUA, etc. Macros get expanded. In the process \TEX\ can enter a group. Inside
494the group, changes to registers get saved on a stack, and restored after leaving
495the group. When conditionals are encountered, another kind of nesting happens,
496and again there is a stack involved. Tokens, expansion, stacks, input levels are
497all terms used in the next chapters. Don't worry, they loose their magic once you
498use \TEX\ a lot. You have access to most of the internals and when not, at least
499it is possible to query some state we're in or level we're at.
500
501When we talk about pack(ag)ing it can mean two things. When \TEX\ has consumed
502some tokens that represent text they are added to the current list. When the text
503is put into a so called \prm {hbox} (for instance a line in a paragraph) it
504(normally) first gets hyphenated, next ligatures are build, and finally kerns are
505added. Each of these stages can be overloaded using \LUA\ code. When these three
506stages are finished, the dimension of the content is calculated and the box gets
507its width, height and depth. What happens with the box depends on what macros do
508with it.
509
510The other thing that can happen is that the text starts a new paragraph. In that
511case some information is stored in a leading \type {par} node. Then indentation
512is appended and the paragraph ends with some glue. Again the three stages are
513applied but this time afterwards, the long line is broken into lines and the
514result is either added to the content of a box or to the main vertical list (the
515running text so to say). This is called par building. At some point \TEX\ decides
516that enough is enough and it will trigger the page builder. So, building is
517another concept we will encounter. Another example of a builder is the one that
518turns an intermediate math list into something typeset.
519
520Wrapping something in a box is called packing. Adding something to a list is
521described in terms of contributing. The more complicated processes are wrapped
522into builders. For now this should be enough to enable you to understand the next
523chapters. The text is not as enlightening and entertaining as Don Knuths books,
524sorry.
525
526\stopsection
527
528\stopchapter
529
530\stopcomponent
531