Source Browser ?

1% language=uk
2
3\environment luametatex-style
4
5\startcomponent luametatex-preamble
6
7\startchapter[reference=preamble,title={The internals}]
8
9\topicindex{nodes}
10\topicindex{boxes}
11\topicindex{\LUA}
12
13This is a reference manual and not a tutorial. This means that we discuss changes
14relative to traditional \TEX\ and also present new (or extended) functionality.
15As a consequence we will refer to concepts that we assume to be known or that
16might be explained later. Because the \LUATEX\ and \LUAMETATEX\ engines open up
17\TEX\ there's suddenly quite some more to explain, especially about the way a (to
18be) typeset stream moves through the machinery. However, discussing all that in
19detail makes not much sense, because deep knowledge is only relevant for those
20who write code not possible with regular \TEX\ and who are already familiar with
21these internals (or willing to spend time on figuring it out).
22
23So, the average user doesn't need to know much about what is in this manual. For
24instance fonts and languages are normally dealt with in the macro package that
25you use. Messing around with node lists is also often not really needed at the
26user level. If you do mess around, you'd better know what you're dealing with.
27Reading \quotation {The \TEX\ Book} by Donald Knuth is a good investment of time
28then also because it's good to know where it all started. A more summarizing
29overview is given by \quotation {\TEX\ by Topic} by Victor Eijkhout. You might
30want to peek in \quotation {The \ETEX\ manual} too.
31
32But \unknown\ if you're here because of \LUA, then all you need to know is that
33you can call it from within a run. If you want to learn the language, just read
34the well written \LUA\ book. The macro package that you use probably will provide
35a few wrapper mechanisms but the basic \lpr {directlua} command that does the job
36is:
37
38\starttyping
39\directlua{tex.print("Hi there")}
40\stoptyping
41
42You can put code between curly braces but if it's a lot you can also put it in a
43file and load that file with the usual \LUA\ commands. If you don't know what
44this means, you definitely need to have a look at the \LUA\ book first.
45
46If you still decide to read on, then it's good to know what nodes are, so we do a
47quick introduction here. If you input this text:
48
49\starttyping
50Hi There ...
51\stoptyping
52
53eventually we will get a linked lists of nodes, which in \ASCII\ art looks like:
54
55\starttyping
56H <=> i <=> [glue] <=> T <=> h <=> e <=> r <=> e ...
57\stoptyping
58
59When we have a paragraph, we actually get something like this, where a \type
60{par} node stores some metadata and is followed by a \type {hlist} flagged as
61indent box:
62
63\starttyping
64[par] <=> [hlist] <=> H <=> i <=> [glue] <=> T <=> h <=> e <=> r <=> e ...
65\stoptyping
66
67Each character becomes a so called glyph node, a record with properties like the
68current font, the character code and the current language. Spaces become glue
69nodes. There are many node types that we will discuss later. Each node points
70back to a previous node or next node, given that these exist. Sometimes
71multiple characters are represented by one glyphs, so one can also get:
72
73\starttyping
74[par] <=> [hlist] <=> H <=> i <=> [glue] <=> Th <=> e <=> r <=> e ...
75\stoptyping
76
77And maybe some characters get positioned relative to each other, so we might
78see:
79
80\starttyping
81[par] <=> [hlist] <=> H <=> [kern] <=> i <=> [glue] <=> Th <=> e <=> r <=> e ...
82\stoptyping
83
84Actually, the above representation is one view, because in \LUAMETATEX\ we can
85choose for this:
86
87\starttyping
88[par] <=> [glue] <=> H <=> [kern] <=> i <=> [glue] <=> Th <=> e <=> r <=> e ...
89\stoptyping
90
91where glue (currently fixed) is used instead of an empty hlist (think of a \type
92{\hbox}). Options like this are available because want a certain view on these
93lists from the \LUA\ end and the result being predicable is part of that.
94
95It's also good to know beforehand that \TEX\ is basically centered around
96creating paragraphs and pages. The par builder takes a list and breaks it into
97lines. At some point horizontal blobs are wrapped into vertical ones. Lines are
98so called boxes and can be separated by glue, penalties and more. The page
99builder accumulates lines and when feasible triggers an output routine that will
100take the list so far. Constructing the actual page is not part of \TEX\ but done
101using primitives that permit manipulation of boxes. The result is handled back to
102\TEX\ and flushed to a (often \PDF) file.
103
104The \LUATEX\ engine provides hooks for \LUA\ code at nearly every reasonable
105point in the process: collecting content, hyphenating, applying font features,
106breaking into lines, etc. This means that you can overload \TEX's natural
107behaviour, which still is the benchmark. When we refer to \quote {callbacks} we
108means these hooks. The \TEX\ engine itself is pretty well optimized but when you
109kick in much \LUA\ code, you will notices that performance drops. Don't blame and
110bother the authors with performance issues. In \CONTEXT\ over 50\% of the time
111can be spent in \LUA, but so far we didn't get many complaints about efficiency.
112Adding more callbacks makes no sense, also because at some point the performance
113hit gets too large. There are plenty ways to achieve one goals. For that reason:
114take remarks about \LUATEX, features, potential, performance etc.\ with a natural
115grain of salt.
116
117Where plain \TEX\ is basically a basic framework for writing a specific style,
118macro packages like \CONTEXT\ and \LATEX\ provide the user a whole lot of
119additional tools to make documents look good. They hide the dirty details of font
120management, language demands, turning structure into typeset results, wrapping
121pages, including images, and so on. You should be aware of the fact that when you
122hook in your own code to manipulate lists, this can interfere with the macro
123package that you use. Each successive step expects a certain result and if you
124mess around to much, the engine eventually might bark and quit. It can even
125crash, because testing everywhere for what users can do wrong is no real option.
126
127When you read about nodes in the following chapters it's good to keep in mind
128what commands relate to them. Here are a few:
129
130\starttabulate[|l|l|p|]
131\DB command              \BC node          \BC explanation \NC \NR
132\TB
133\NC \prm {hbox}          \NC \nod {hlist} \NC horizontal box \NC \NR
134\NC \prm {vbox}          \NC \nod {vlist} \NC vertical box with the baseline at the bottom \NC \NR
135\NC \prm {vtop}          \NC \nod {vlist} \NC vertical box with the baseline at the top \NC \NR
136\NC \prm {hskip}         \NC \nod {glue}  \NC horizontal skip with optional stretch and shrink \NC \NR
137\NC \prm {vskip}         \NC \nod {glue}  \NC vertical skip with optional stretch and shrink \NC \NR
138\NC \prm {kern}          \NC \nod {kern}  \NC horizontal or vertical fixed skip \NC \NR
139\NC \prm {discretionary} \NC \nod {disc}  \NC hyphenation point (pre, post, replace) \NC \NR
140\NC \prm {char}          \NC \nod {glyph} \NC a character \NC \NR
141\NC \prm {hrule}         \NC \nod {rule}  \NC a horizontal rule \NC \NR
142\NC \prm {vrule}         \NC \nod {rule}  \NC a vertical rule \NC \NR
143\NC \prm {textdirection} \NC \nod {dir}   \NC a change in text direction \NC \NR
144\LL
145\stoptabulate
146
147Whatever we feed into \TEX\ at some point becomes a token which is either
148interpreted directly or stored in a linked list. A token is just a number that
149encodes a specific command (operator) and some value (operand) that further
150specifies what that command is supposed to do. In addition to an interface to
151nodes, there is an interface to tokens, as later chapters will demonstrate.
152
153Text (interspersed with macros) comes from an input medium. This can be a file,
154token list, macro body cq.\ arguments, \ some internal quantity (like a number),
155\LUA, etc. Macros get expanded. In the process \TEX\ can enter a group. Inside
156the group, changes to registers get saved on a stack, and restored after leaving
157the group. When conditionals are encountered, another kind of nesting happens,
158and again there is a stack involved. Tokens, expansion, stacks, input levels are
159all terms used in the next chapters. Don't worry, they loose their magic once you
160use \TEX\ a lot. You have access to most of the internals and when not, at least
161it is possible to query some state we're in or level we're at.
162
163When we talk about pack(ag)ing it can mean two things. When \TEX\ has consumed
164some tokens that represent text. When the text is put into a so called \type
165{\hbox} it (normally) first gets hyphenated (even in an horizontal list), next
166ligatures are build, and finally kerns are added. Each of these stages can be
167overloaded using \LUA\ code. When these three stages are finished, the dimension
168of the content is calculated and the box gets its width, height and depth. What
169happens with the box depends on what macros do with it.
170
171The other thing that can happen is that the text starts a new paragraph. In that
172case some information is stored in a leading \type {par} node. Then indentation
173is appended and the paragraph ends with some glue. Again the three stages are
174applied but this time, afterwards, the long line is broken into lines and the
175result is either added to the content of a box or to the main vertical list (the
176running text so to say). This is called par building. At some point \TEX\ decides
177that enough is enough and it will trigger the page builder. So, building is
178another concept we will encounter. Another example of a builder is the one that
179turns an intermediate math list into something typeset.
180
181Wrapping something in a box is called packing. Adding something to a list is
182described in terms of contributing. The more complicated processes are wrapped
183into builders. For now this should be enough to enable you to understand the next
184chapters. The text is not as enlightening and entertaining as Don Knuths books,
185sorry.
186
187\stopchapter
188
189\stopcomponent
190