1
2
3
4
5\environment luametatexstyle
6
7\startcomponent luametatexenhancements
8
9\startchapter[reference=enhancements,title={Enhancements}]
10
11\startsection[title={Introduction}]
12
13\startsubsection[title={Primitive behaviour}]
14
15From day one, \LUATEX\ has offered extra features compared to the superset of
16\PDFTEX, which includes \ETEX, and \ALEPH. This has not been limited to the
17possibility to execute \LUA\ code via \prm {directlua}, but \LUATEX\ also adds
18functionality via new \TEXside primitives or extensions to existing ones. The
19same is true for \LUAMETATEX. Some primitives have \type {luatex} in their name
20and there will be no \type {luametatex} variants. This is because we consider
21\LUAMETATEX\ to be \LUATEX 2\high{}.
22
23Contrary to the \LUATEX\ engine \LUAMETATEX\ enables all its primitives. You can
24clone (a selection of) primitives with a different prefix, like this:
25
26\starttyping
27\directlua { tex.enableprimitives('normal',tex.extraprimitives()) }
28\stoptyping
29
30The \type {extraprimitives} function returns the whole list or a subset,
31specified by one or more keywords \type {tex}, \type {etex} or \type {luatex}.
32When you clone all primitives you can also do this:
33
34\starttyping
35\directlua { tex.enableprimitives('normal',true) }
36\stoptyping
37
38But be aware that the curly braces may not have the proper \prm {catcode}
39assigned to them at this early time (giving a \quote {Missing number} error), so
40it may be needed to put these assignments before the above line:
41
42\starttyping
43\catcode \{ = 1
44\catcode \} = 2
45\stoptyping
46
47More finegrained primitives control is possible and you can look up the
48details in \in {section} [luaprimitives]. There are only three kinds of
49primitives: \type {tex}, \type {etex} and \type {luatex} but a future version
50might drop this and no longer make that distinction as it no longer serves a
51purpose apart from the fact that it reveals some history.
52
53\stopsubsection
54
55\startsubsection[title={Rationale}]
56
57One can argue that \TEX\ should stay as it is but over decades usage of this
58program has evolved and resulted in large macro packages that often need to rely
59on what the \TEX\ books calls \quote {dirty tricks}. When you look deep down in
60the code of \CONTEXT\ \MKII, \MKIV\ and \MKXL\ (aka \LMTX) you will see plenty of
61differences but quite a bit of the functionality in the most recent versions is
62also available in \MKII. Of course more has been added over time, and some
63mechanisms could be made more efficient and reliable but plenty was possible.
64
65So, when you see something done in \CONTEXT\ \LMTX\ using new \LUAMETATEX\
66primitives you can assume that somehow the same is done in \CONTEXT\ \MKIV. We
67dont really need \LUAMETATEX\ instead of \LUATEX. Among the main reasons for
68still going for this new engine are:
69
70\startitemize[packed]
71\startitem
72 some new primitives make for less tracing and tracing has become rather
73 verbose over years (just try \type {tracingall}); examples are the new macro
74 argument handling and some new hooks
75\stopitem
76\startitem
77 some new primitives permits more efficient coding and have a positive impact
78 on performance (this sort of compensates a performance hit due to delegating
79 work to \LUA)
80\stopitem
81\startitem
82 other primitives are there because they make the code look better; good
83 examples are the extensions to conditionals; they remove the necessity for
84 all kind of (somewhat unnatural) middle layers; take local control as example
85\stopitem
86\startitem
87 a few primitives make complex and demanding mechanism a bit easier to grasp
88 and explain; think of alignments, inserts and marks
89\stopitem
90\startitem
91 more access from the \LUA\ end to \TEX\ internals: a few more callbacks, more
92 options, more robust interfaces, etc
93\stopitem
94\startitem
95 some mechanisms are very specific but can be made more generic (and powerful),
96 like inserts, marks, adjusts and local boxes
97\stopitem
98\stopitemize
99
100I realize that new primitives also can make some \TEX\ code look less threatening
101to new users. It removes a bit of hackery and limits the level of guru that comes
102with showing off the mastery of expansion and lookahead. So be it. I wonder if
103those objecting to some of the extensions (with the argument that they are not
104needed, and \CONTEXT\ \MKIV\ is proof of that) can resist using them. I admit
105that it sometimes hurt to throw away good working but cumbersome code that took a
106while to evolve, but I also admit that I favor long distance traveling by bike or
107car over riding horseback.
108
109It took a few years for \LUAMETATEX\ to evolve to what it is now and most
110extensions are not there \quotation {because they were easy} or \quotation {could
111be done}. If that were the case, there would be plenty more. In many aspects it
112has been a balancing act and much also relates to looking at the \CONTEXT\ source
113code (\TEX\ as well as \LUA) and wondering why it looks that way. It is also
114driven by the fact that I want to be able to explain to users why things are done
115in a certain way. In fact, I want users to be able to look at the code and
116understand it (apart from maybe a few real dirty low level helpers that are also
117dirty because of performance reasons). Just take this into account when reading on.
118
119And yes, there are still a few possibilities I want to explore \unknown\ some might
120show up temporarily so dont be surprised. Im also aware that some new features can
121have bugs or side effects that didnt show up in \CONTEXT, which after all is the
122benchmark and environment in which all this evolves.
123
124Over time, the other \TEX\ engines might have an occasional feature (primitive)
125added and it is very unlikely that \LUAMETATEX\ will follow up on that. First of
126all we have different internals but most of all because plenty of time went into
127considering what got added and what not, apart from the fact that we have
128callbacks. Decades of \TEX\ development never really have lead to an extensive
129wish list so there is no real need why there should be a demand on anything other
130than we offer here. If \TEX\ worked well for ages, it can as well do for more, so
131there is no need to cripple the code base simply in order to be compatible with
132other engines; \LUAMETATEX\ is already quite different anyway.
133
134\stopsubsection
135
136\startsubsection[title={Version information}]
137
138\topicindex{version}
139\topicindex{banner}
140
141There are three primitives to test the version of \LUATEX\ (and \LUAMETATEX):
142
143\unexpanded\def\VersionHack#1
144 {\ctxlua{
145 local banner = "\luatexbanner"
146 local banner = string.match(banner,"(.+)\letterpercent(") or banner
147 context(string.gsub(banner ,"jit",""))
148 }}
149
150\starttabulate[llpl]
151\DB primitive \BC value
152 \BC explanation \NC \NR
153\TB
154\NC \prm {luatexbanner} \NC \VersionHack{\luatexbanner}
155 \NC the banner reported on the console \NC \NR
156\NC \prm {luatexversion} \NC \the\luatexversion
157 \NC major and minor number combined \NC \NR
158\NC \prm {luatexrevision} \NC \the\luatexrevision
159 \NC the revision number \NC \NR
160\LL
161\stoptabulate
162
163A version is defined as follows:
164
165\startitemize
166\startitem
167 The major version is the integer result of \prm {luatexversion} divided by
168 100. The primitive is an \quote {internal variable}, so you may need to prefix
169 its use with \prm {the} or \prm {number} depending on the context.
170\stopitem
171\startitem
172 The minor version is a number running from 0 upto 99.
173\stopitem
174\startitem
175 The revision is reported by \prm {luatexrevision}. Contrary to other engines
176 in \LUAMETATEX\ is also a number so one needs to prefix it with \prm {the} or
177 \prm {number}. \footnote {In the past it always was good to prefix the
178 revision with \prm {number} anyway, just to play safe, although there have
179 for instance been times that \PDFTEX\ had funny revision indicators that at
180 some point ended up as letters due to the internal conversions.}
181\stopitem
182\startitem
183 The full version number consists of the major version (\type {X}), minor
184 version (\type {YY}) and revision (\type {ZZ}), separated by dots, so \type
185 {X.YY.ZZ}.
186\stopitem
187\stopitemize
188
189The \LUATEX\ binary has companions like \LUAJITTEX\ and a version that has a font
190rendering library on board. Both introduce dependencies that dont fit into the
191\LUAMETATEX\ agenda: compilation should be easy and future proof and not depend
192on code outside the source tree. It means that for instance the \CONTEXT\ runners
193dont really need to check much more than the basic name. It also means that the
194\type {context} and \type {mtxrun} stubs can be symbolic links to the main
195program that itself is about 3MB, so we can keep the binary footprint small. For
196normal \CONTEXT\ \LMTX\ processing no other binaries are needed because whatever
197support we need is done in \LUA.
198
199The \LUAMETATEX\ version number starts at2 in order to prevent a clash with
200\LUATEX, and the version commands are the same. This is a way to indicate that
201these projects are related.
202
203The \type {status} library also provides some information including what we get
204with the three mentioned primitives:
205
206\starttabulate[ll]
207\DB field \BC value \NC \NR
208\TB
209\NC \type {filename} \NC \cldcontext{status.filename} \NC \NR
210\NC \type {banner} \NC \cldcontext{status.banner} \NC \NR
211\NC \type {luatexengine} \NC \cldcontext{status.luatex_engine} \NC \NR
212\NC \type {luatexversion} \NC \cldcontext{status.luatex_version} \NC \NR
213\NC \type {luatexrevision} \NC \cldcontext{status.luatex_revision} \NC \NR
214\NC \type {luatexverbose} \NC \cldcontext{status.luatex_verbose} \NC \NR
215\NC \type {copyright} \NC \cldcontext{status.copyright} \NC \NR
216\NC \type {developmentid} \NC \cldcontext{status.development_id} \NC \NR
217\NC \type {formatid} \NC \cldcontext{status.format_id} \NC \NR
218\NC \type {usedcompiler} \NC \cldcontext{status.used_compiler} \NC \NR
219\LL
220\stoptabulate
221
222\stopsubsection
223
224\stopsection
225
226\startsection[title={\UNICODE\ text support}]
227
228\startsubsection[title={Extended ranges}]
229
230\topicindex{\UNICODE}
231
232Text input and output is now considered to be \UNICODE\ text, so input characters
233can use the full range of \UNICODE\ ($2{20}2{16}1 = \hbox{0x10FFFF}$). Later
234chapters will talk of characters and glyphs. Although these are not
235interchangeable, they are closely related. During typesetting, a character is
236always converted to a suitable graphic representation of that character in a
237specific font. However, while processing a list of tobetypeset nodes, its
238contents may still be seen as a character. Inside the engine there is no clear
239separation between the two concepts. Because the subtype of a glyph node can be
240changed in \LUA\ it is up to the user. Subtypes larger than 255 indicate that
241font processing has happened.
242
243A few primitives are affected by this, all in a similar fashion: each of them has
244to accommodate for a larger range of acceptable numbers. For instance, \prm
245{char} now accepts values between0 and $1{,}114{,}111$. This should not be a
246problem for wellbehaved input files, but it could create incompatibilities for
247input that would have generated an error when processed by older \TEXbased
248engines. The affected commands with an altered initial (left of the equal sign)
249or secondary (right of the equal sign) value are: \prm {char}, \prm {lccode},
250\prm {uccode}, \prm {hjcode}, \prm {catcode}, \prm {sfcode}, \prm {efcode}, \prm
251{cfcode}, \prm {lpcode}, \prm {rpcode}, \prm {chardef}.
252
253As far as the core engine is concerned, all input and output to text files is
254\UTF8 encoded. Input files can be preprocessed using the \type {reader}
255callback. This will be explained in \in {section} [iocallback]. Normalization of
256the \UNICODE\ input is on purpose not builtin and can be handled by a macro
257package during callback processing. We have made some practical choices and the
258user has to live with those.
259
260
261
262
263
264
265
266Contrary to other \TEX\ engines, the output to the terminal is asis so there
267is no escaping with \type {}. We operate in a \UTF\ universe. Because we
268operate in a \CCODE\ universum, zero characters are special but because we also
269live in a \UNICODE\ galaxy that is no real problem.
270
271\stopsubsection
272
273\startsubsection[title={\prm {tocharacter}}]
274
275\topicindex{\UNICODE}
276
277The expandable command \prm {tocharacter} reads a number between0 and $1{,}114{,}111$
278and expands to the associated \UNICODE\ character.
279
280\stopsubsection
281
282\startsubsection[title={Extended tables}]
283
284All traditional \TEX\ and \ETEX\ registers can be 16bit numbers. The affected
285commands are:
286
287\startfourcolumns
288\startlines
289\prm {count}
290\prm {dimen}
291\prm {skip}
292\prm {muskip}
293\prm {marks}
294\prm {toks}
295\prm {countdef}
296\prm {dimendef}
297\prm {skipdef}
298\prm {muskipdef}
299\prm {toksdef}
300\prm {insert}
301\prm {box}
302\prm {unhbox}
303\prm {unvbox}
304\prm {copy}
305\prm {unhcopy}
306\prm {unvcopy}
307\prm {wd}
308\prm {ht}
309\prm {dp}
310\prm {setbox}
311\prm {vsplit}
312\stoplines
313\stopfourcolumns
314
315Fonts are loaded via \LUA\ and a minimal amount of information is kept at the
316\TEX\ end. Sharing resources is up to the loaders. The engine doesnt really care
317about what a character (or glyph) number represents (a \UNICODE\ or index) as it
318only is interested in dimensions.
319
320In \TEX\ the number of registers is 256 and \ETEX\ bumped that to 32K. One reason
321for a fixed number is that these registers are fast ways to store data and
322therefore are part of the main lookup table (used for data and pointers to data
323as well as save and restore housekeeping). In \LUATEX\ the number was bumped to
32464K but one can argue that less would also do. In order to keep the default
325memory footprint reasonable, in \LUAMETATEX\ the number of languages, fonts and
326marks is limited. The size of some tables can be limited by configuration
327settings, so they can start out small and grow till configured maximum which is
328smaller than the absolute maximum.
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343Because we have additional ways to store integers, dimensions and glue, we might
344actually decide to decrease the maximum of the registers: if 64K is not enough,
345and you work around it, then likely 32K might do as well. Also, we have \LUA\ to
346store massive amounts of data. One can argue that saving some 1.5MB memory (when
347we go halfway) is not worth the effort in a time when you have to close a browser
348in order to free the gigabytes it consumes, but there is no reason not to be lean
349and mean: a more conservative approach to start with creates headroom for going
350wild later.
351
352\stopsubsection
353
354\stopsection
355
356\startsection[title={Attributes}]
357
358\startsubsection[title={Nodes}]
359
360\topicindex {nodes}
361
362When \TEX\ reads input it will interpret the stream according to the properties
363of the characters. Some signal a macro name and trigger expansion, others open
364and close groups, trigger math mode, etc. Whats left over becomes the typeset
365text. Internally we get a linked list of nodes. Characters become \nod {glyph}
366nodes that have for instance a \type {font} and \type {char} property and \typ
367{\kern 10pt} becomes a \nod {kern} node with a \type {width} property. Spaces are
368alien to \TEX\ as they are turned into \nod {glue} nodes. So, a simple paragraph
369is mostly a mix of sequences of \nod {glyph} nodes (words) and \nod {glue} nodes
370(spaces). A node can have a subtype so that it can be recognized as for instance
371a space related glue.
372
373The sequences of characters at some point are extended with \nod {disc} nodes
374that relate to hyphenation. After that font logic can be applied and we get a
375list where some characters can be replaced, for instance multiple characters can
376become one ligature, and font kerns can be injected. This is driven by the
377font properties.
378
379Boxes (like \prm {hbox} and \prm {vbox}) become \nod {hlist} or \nod {vlist}
380nodes with \type {width}, \type {height}, \type {depth} and \type {shift}
381properties and a pointer \type {list} to its actual content. Boxes can be
382constructed explicitly or can be the result of subprocesses. For instance, when
383lines are broken into paragraphs, the lines are a linked list of \nod {hlist}
384nodes, possibly with glue and penalties in between.
385
386Internally nodes have a number. This number is actually an index in the memory
387used to store nodes.
388
389So, to summarize: all that you enter as content eventually becomes a node, often
390as part of a (nested) list structure. They have a relative small memory footprint
391and carry only the minimal amount of information needed. In traditional \TEX\ a
392character node only held the font and slot number, in \LUATEX\ we also store some
393language related information, the expansion factor, etc. Now that we have access
394to these nodes from \LUA\ it makes sense to be able to carry more information
395with a node and this is where attributes kick in.
396
397It is important to keep in mind that there are situations where nodes get created
398in the current context. For instance, when \TEX\ builds a paragraph or page or
399constructs math formulas, it does add nodes and giving these the current
400attributes makes no sense and can even give weird side effects. In these cases,
401the attributes are inherited from neighbouring nodes.
402
403\stopsubsection
404
405\startsubsection[title={Attribute registers}]
406
407\topicindex {attributes}
408
409Attributes are a completely new concept in \LUATEX. Syntactically, they behave a
410lot like counters: attributes obey \TEXs nesting stack and can be used after
411\prm {the} etc.\ just like the normal \prm {count} registers.
412
413\startsyntax
414\attribute <16bit number> <optional equals> <32bit number>!crlf
415\attributedef <csname> <optional equals> <16bit number>
416\stopsyntax
417
418Conceptually, an attribute is either \quote {set} or \quote {unset}. Unset
419attributes have a special negative value to indicate that they are unset, that
420value is the lowest legal value: \type {"7FFFFFFF} in hexadecimal, a.k.a.
421$2147483647$ in decimal. It follows that the value \type {"7FFFFFFF} cannot be
422used as a legal attribute value, but you {\it can\/} assign \type {"7FFFFFFF} to
423\quote {unset} an attribute. All attributes start out in this \quote {unset}
424state in \INITEX.
425
426Attributes can be used as extra counter values, but their usefulness comes mostly
427from the fact that the numbers and values of all \quote {set} attributes are
428attached to all nodes created in their scope. These can then be queried from any
429\LUA\ code that deals with node processing. Further information about how to use
430attributes for node list processing from \LUA\ is given in\in {chapter}[nodes].
431
432Attributes are stored in a sorted (sparse) linked list that are shared when
433possible. This permits efficient testing and updating. You can define many
434thousands of attributes but normally such a large number makes no sense and is
435also not that efficient because each node carries a (possibly shared) link to a
436list of currently set attributes. But they are a convenient extension and one of
437the first extensions we implemented in \LUATEX.
438
439In \LUAMETATEX\ we try to minimize the memory footprint and creation of these
440attribute lists more aggressive sharing them. This feature is still somewhat
441experimental.
442
443\stopsubsection
444
445\startsubsection[title={Box attributes}]
446
447\topicindex {attributes}
448\topicindex {boxes}
449\topicindex {vcentering}
450
451Nodes typically receive the list of attributes that is in effect when they are
452created. This moment can be quite asynchronous. For example: in paragraph
453building, the individual line boxes are created after the \prm {par} command has
454been processed, so they will receive the list of attributes that is in effect
455then, not the attributes that were in effect in, say, the first or third line of
456the paragraph.
457
458Similar situations happen in \LUATEX\ regularly. A few of the more obvious
459problematic cases are dealt with: the attributes for nodes that are created
460during hyphenation, kerning and ligaturing borrow their attributes from their
461surrounding glyphs, and it is possible to influence box attributes directly.
462
463When you assemble a box in a register, the attributes of the nodes contained in
464the box are unchanged when such a box is placed, unboxed, or copied. In this
465respect attributes act the same as characters that have been converted to
466references to glyphs in fonts. For instance, when you use attributes to implement
467color support, each node carries information about its eventual color. In that
468case, unless you implement mechanisms that deal with it, applying a color to
469already boxed material will have no effect. Keep in mind that this
470incompatibility is mostly due to the fact that separate specials and literals are
471a more unnatural approach to colors than attributes.
472
473It is possible to finetune the list of attributes that are applied to a \type
474{hbox}, \type {vbox} or \type {vtop} by the use of the keyword \type {attr}. The
475\type {attr} keyword(s) should come before a \type {to} or \type {spread}, if
476that is also specified. An example is:
477
478\startbuffer[tex]
479\attribute997=123
480\attribute998=456
481\setbox0=\hbox {Hello}
482\setbox2=\hbox attr 999 = 789 attr 998 = "7FFFFFFF{Hello}
483\stopbuffer
484
485\startbuffer[lua]
486 for b=0,2,2 do
487 for a=997, 999 do
488 tex.sprint("box ", b, " : attr ",a," : ",tostring(tex.box[b] [a]))
489 tex.sprint("\\quad\\quad")
490 tex.sprint("list ",b, " : attr ",a," : ",tostring(tex.box[b].list[a]))
491 tex.sprint("\\par")
492 end
493 end
494\stopbuffer
495
496\typebuffer[tex]
497
498Box 0 now has attributes 997 and 998 set while box 2 has attributes 997 and 999
499set while the nodes inside that box will all have attributes 997 and 998 set.
500Assigning the maximum negative value causes an attribute to be ignored.
501
502To give you an idea of what this means at the \LUA\ end, take the following
503code:
504
505\typebuffer[lua]
506
507Later we will see that you can access properties of a node. The boxes here are so
508called \nod {hlist} nodes that have a field \type {list} that points to the
509content. Because the attributes are a list themselves you can access them by
510indexing the node (here we do that with \type {[a]}). Running this snippet gives:
511
512\start
513 \getbuffer[tex]
514 \startpacked \tt
515 \ctxluabuffer[lua]
516 \stoppacked
517\stop
518
519Because some values are not set we need to apply the \type {tostring} function
520here so that we get the word \type {nil}.
521
522A special kind of box is \prm {vcenter}. This one also can have attributes. When
523one or more are set these plus the currently set attributes are bound to the
524resulting box. In regular \TEX\ these centered boxes are only permitted in math
525mode, but in \LUAMETATEX\ there is no error message and the box the height and
526depth are equally divided. Of course in text mode there is no math axis related
527offset applied.
528
529It is possible to change or add to the attributes assigned to a box with \prm
530{boxattribute}:
531
532\starttyping
533\boxattribute 0 123 456
534\stoptyping
535
536You can set attributes of the current paragraph specification node with \prm
537{parattribute}:
538
539\starttyping
540\parattribute 123 456
541\stoptyping
542
543\stopsubsection
544
545\stopsection
546
547\startsection[title={\LUA\ related primitives}]
548
549\startsubsection[title={\prm {directlua}}]
550
551\topicindex{scripting}
552\topicindex{luadirect}
553
554In order to merge \LUA\ code with \TEX\ input, a few new primitives are needed.
555The primitive \prm {directlua} is used to execute \LUA\ code immediately. The
556syntax is
557
558\startsyntax
559\directlua <general text>
560\stopsyntax
561
562The \syntax {<general text>} is expanded fully, and then fed into the \LUA\
563interpreter. After reading and expansion has been applied to the \syntax
564{<general text>}, the resulting token list is converted to a string as if it was
565displayed using \type {\the\toks}. On the \LUA\ side, each \prm {directlua} block
566is treated as a separate chunk. In such a chunk you can use the \type {local}
567directive to keep your variables from interfering with those used by the macro
568package.
569
570The conversion to and from a token list means that you normally can not use \LUA\
571line comments (starting with \type {}) within the argument. As there typically
572will be only one \quote {line} the first line comment will run on until the end
573of the input. You will either need to use \TEXstyle line comments (starting
574with \%), or change the \TEX\ category codes locally. Another possibility is to
575say:
576
577\starttyping
578\begingroup
579\endlinechar=10
580\directlua ...
581\endgroup
582\stoptyping
583
584Then \LUA\ line comments can be used, since \TEX\ does not replace line endings
585with spaces. Of course such an approach depends on the macro package that you
586use.
587
588The \prm {directlua} command is expandable. Since it passes \LUA\ code to the
589\LUA\ interpreter its expansion from the \TEX\ viewpoint is usually empty.
590However, there are some \LUA\ functions that produce material to be read by \TEX,
591the so called print functions. The most simple use of these is \type
592{tex.print(<string> s)}. The characters of the string \type {s} will be placed on
593the \TEX\ input buffer, that is, \quote {before \TEXs eyes} to be read by \TEX\
594immediately. For example:
595
596\startbuffer
597\count10=20
598a\directlua{tex.print(tex.count[10]+5)}b
599\stopbuffer
600
601\typebuffer
602
603expands to
604
605\getbuffer
606
607Here is another example:
608
609\startbuffer
610$\pi = \directlua{tex.print(math.pi)}$
611\stopbuffer
612
613\typebuffer
614
615will result in
616
617\getbuffer
618
619Note that the expansion of \prm {directlua} is a sequence of characters, not of
620tokens, contrary to all \TEX\ commands. So formally speaking its expansion is
621null, but it collects material in a new level on the input stack to be
622immediately read by \TEX\ after the \LUA\ call as finished. It is a bit like
623\ETEXs \prm {scantokens}, which now uses the same mechanism. For a description
624of print functions look at \in {section} [sec:luaprint].
625
626Because the \syntax {<general text>} is a chunk, the normal \LUA\ error handling
627is triggered if there is a problem in the included code. The \LUA\ error messages
628should be clear enough, but the contextual information is often suboptimal
629because it can come from deep down, and \TEX\ has no knowledge about what you do
630in \LUA. Often, you will only see the line number of the right brace at the end
631of the code.
632
633While on the subject of errors: some of the things you can do inside \LUA\ code
634can break up \LUAMETATEX\ pretty bad. If you are not careful while working with
635the node list interface, you may even end up with errors or even crashes from
636within the \TEX\ portion of the executable.
637
638\stopsubsection
639
640\startsubsection[title={\prm {luaescapestring}}]
641
642\topicindex {escaping}
643
644This primitive converts a \TEX\ token sequence so that it can be safely used as
645the contents of a \LUA\ string: embedded backslashes, double and single quotes,
646and newlines and carriage returns are escaped. This is done by prepending an
647extra token consisting of a backslash with category code12, and for the line
648endings, converting them to \type {n} and \type {r} respectively. The token
649sequence is fully expanded.
650
651\startsyntax
652\luaescapestring <general text>
653\stopsyntax
654
655Most often, this command is not actually the best way to deal with the
656differences between \TEX\ and \LUA. In very short bits of \LUA\ code it is often
657not needed, and for longer stretches of \LUA\ code it is easier to keep the code
658in a separate file and load it using \LUAs \type {dofile}:
659
660\starttyping
661\directlua { dofile("mysetups.lua") }
662\stoptyping
663
664\stopsubsection
665
666\startsubsection[title={\prm {luafunction}, \prm {luafunctioncall} and \prm {luadef}}]
667
668\topicindex{functions}
669\topicindex{luafunctions}
670
671The \prm {directlua} commands involves tokenization of its argument (after
672picking up an optional name or number specification). The tokenlist is then
673converted into a string and given to \LUA\ to turn into a function that is
674called. The overhead is rather small but when you have millions of calls it can
675have some impact. For this reason there is a variant call available: \prm
676{luafunction}. This command is used as follows:
677
678\starttyping
679\directlua {
680 local t = lua.get_functions_table()
681 t[1] = function() tex.print("!") end
682 t[2] = function() tex.print("?") end
683}
684
685\luafunction1
686\luafunction2
687\stoptyping
688
689Of course the functions can also be defined in a separate file. There is no limit
690on the number of functions apart from normal \LUA\ limitations. Of course there
691is the limitation of no arguments but that would involve parsing and thereby give
692no gain. The function, when called in fact gets one argument, being the index, so
693in the following example the number \type {8} gets typeset.
694
695\starttyping
696\directlua {
697 local t = lua.get_functions_table()
698 t[8] = function(slot) tex.print(slot) end
699}
700\stoptyping
701
702The \prm {luafunctioncall} primitive does the same but is unexpandable, for
703instance in an \prm {edef}. In addition \LUATEX\ provides a definer:
704
705\starttyping
706 \luadef\MyFunctionA 1
707 \global\luadef\MyFunctionB 2
708\protected\global\luadef\MyFunctionC 3
709\stoptyping
710
711You should really use these commands with care. Some references get stored in
712tokens and assume that the function is available when that token expands. On the
713other hand, as we have tested this functionality in relative complex situations
714normal usage should not give problems.
715
716{\em It makes sense to delegate the implementation of the primitives to \LUA.}
717
718\stopsubsection
719
720\startsubsection[title={\prm {luabytecode} and \prm {luabytecodecall}}]
721
722\topicindex{luabytecode}
723\topicindex{bytecode}
724
725Analogue to the function callers discussed in the previous section we have byte
726code callers. Again the call variant is unexpandable.
727
728\starttyping
729\directlua {
730 lua.bytecode[9998] = function(s)
731 tex.sprint(s*token.scan_int())
732 end
733 lua.bytecode[5555] = function(s)
734 tex.sprint(s*token.scan_dimen())
735 end
736}
737\stoptyping
738
739This works with:
740
741\starttyping
742\luabytecode 9998 5 \luabytecode 5555 5sp
743\luabytecodecall9998 5 \luabytecodecall5555 5sp
744\stoptyping
745
746The variable \type {s} in the code is the number of the byte code register that
747can be used for diagnostic purposes. The advantage of bytecode registers over
748function calls is that they are stored in the format (but without upvalues).
749
750{\em It makes sense to delegate the implementation of the primitives to \LUA.}
751
752\stopsubsection
753
754\stopsection
755
756\startsection[title={Catcode tables}]
757
758\startsubsection[title={Catcodes}]
759
760\topicindex {catcodes}
761
762Catcode tables are a new feature that allows you to switch to a predefined
763catcode regime in a single statement. You can have lots of different tables, but
764if you need a dozen you might wonder what youre doing. This subsystem is
765backward compatible: if you never use the following commands, your document will
766not notice any difference in behaviour compared to traditional \TEX. The contents
767of each catcode table is independent from any other catcode table, and its
768contents is stored and retrieved from the format file.
769
770\stopsubsection
771
772\startsubsection[title={\prm {catcodetable}}]
773
774The primitive \prm {catcodetable} switches to a different catcode table. Such a
775table has to be previously created using one of the two primitives below, or it
776has to be zero. Table zero is initialized by \INITEX.
777
778\startsyntax
779\catcodetable <15bit number>
780\stopsyntax
781
782\stopsubsection
783
784\startsubsection[title={\prm {initcatcodetable}}]
785
786\startsyntax
787\initcatcodetable <15bit number>
788\stopsyntax
789
790The primitive \prm {initcatcodetable} creates a new table with catcodes
791identical to those defined by \INITEX. The new catcode table is allocated
792globally: it will not go away after the current group has ended. If the supplied
793number is identical to the currently active table, an error is raised. The
794initial values are:
795
796\starttabulate[ccll]
797\DB catcode \BC character \BC equivalent \BC category \NC \NR
798\TB
799\NC 0 \NC \tttf \letterbackslash \NC \NC \type {escape} \NC \NR
800\NC 5 \NC \tttf \letterhat\letterhat M \NC return \NC \type {carret} \NC \NR
801\NC 9 \NC \tttf \letterhat\letterhat @ \NC null \NC \type {ignore} \NC \NR
802\NC 10 \NC \tttf <space> \NC space \NC \type {spacer} \NC \NR
803\NC 11 \NC {\tttf a} \endash\ {\tttf z} \NC \NC \type {letter} \NC \NR
804\NC 11 \NC {\tttf A} \endash\ {\tttf Z} \NC \NC \type {letter} \NC \NR
805\NC 12 \NC everything else \NC \NC \type {other} \NC \NR
806\NC 14 \NC \tttf \letterpercent \NC \NC \type {comment} \NC \NR
807\NC 15 \NC \tttf \letterhat\letterhat ? \NC delete \NC \type {invalidchar} \NC \NR
808\LL
809\stoptabulate
810
811\stopsubsection
812
813\startsubsection[title={\prm {savecatcodetable}}]
814
815\startsyntax
816\savecatcodetable <15bit number>
817\stopsyntax
818
819\prm {savecatcodetable} copies the current set of catcodes to a new table with
820the requested number. The definitions in this new table are all treated as if
821they were made in the outermost level. Again, the new table is allocated globally:
822it will not go away after the current group has ended. If the supplied number is
823the currently active table, an error is raised.
824
825\stopsubsection
826
827\startsubsection[title={\prm {letcharcode}}]
828
829This primitive can be used to assign a meaning to an active character, as in:
830
831\starttyping
832\def\foo{bar} \letcharcode123=\foo
833\stoptyping
834
835This can be a bit nicer than using the uppercase tricks (using the property of
836\prm {uppercase} that it treats active characters special).
837
838\stopsubsection
839
840\stopsection
841
842\startsection[title={Tokens and expansion}]
843
844\startsubsection[title={\prm {scantextokens}, \prm {tokenized} and \prm {retokenized}}]
845
846\topicindex {tokensscanning}
847
848The syntax of \prm {scantextokens} is identical to \prm {scantokens}. This
849primitive is a slightly adapted version of \ETEXs \prm {scantokens}. The
850differences are:
851
852\startitemize
853\startitem
854 The last (and usually only) line does not have a \prm {endlinechar} appended.
855\stopitem
856\startitem
857 \prm {scantextokens} never raises an EOF error, and it does not execute
858 \prm {everyeof} tokens.
859\stopitem
860\startitem
861 There are no \quote {\unknown\ while end of file \unknown} error tests
862 executed. This allows the expansion to end on a different grouping level or
863 while a conditional is still incomplete.
864\stopitem
865\stopitemize
866
867The implementation in \LUAMETATEX\ is different in the sense that it uses the same
868methods as printing from \LUA\ to \TEX\ does. Therefore, in addition to the two
869commands we also have this expandable command:
870
871\startsyntax
872\tokenized {...}
873\tokenized catcodetable <number> {...}
874\stopsyntax
875
876The \prm {retokenized} variant differs in that it doesnt check for a keyword and
877just used the current catcode regime.
878
879The \ETEX\ command \type {\tracingscantokens} has been dropped in the process as
880that was interwoven with the old code.
881
882\stopsubsection
883
884\startsubsection[title={\prm {toksapp}, \prm {tokspre}, \prm {etoksapp}, \prm {etokspre},
885\prm {gtoksapp}, \prm {gtokspre}, \prm {xtoksapp}, \prm {xtokspre}}]
886
887Instead of:
888
889\starttyping
890\toks0\expandafter{\the\toks0 foo}
891\stoptyping
892
893you can use:
894
895\starttyping
896\etoksapp0{foo}
897\stoptyping
898
899The \type {pre} variants prepend instead of append, and the \type {e} variants
900expand the passed general text. The \type {g} and \type {x} variants are global.
901
902\stopsubsection
903
904\startsubsection[title={\prm {etoks} and \prm {xtoks}}]
905
906A mix between the previously discussed append and prepend primitives and simple
907toks register assignments are these two. They act like \prm {toks} but expand
908their content first. The \type {x} variant does a global assignment.
909
910\stopsubsection
911
912\startsubsection[title={\prm {expanded}, \prm {expandedafter}, \prm {localcontrol}, \prm
913{localcontrolled}, \prm {beginlocalcontrol} and \prm {endlocalcontrol}}]
914
915\topicindex {expansion}
916
917The \prm {expanded} primitive takes a token list and expands its content which
918can come in handy: it avoids a tricky mix of \prm {expandafter} and \prm
919{noexpand}. You can compare it with what happens inside the body of an \prm
920{edef}. The \tex {immediateassignment} and \tex {immediateassigned} commands are
921gone because we have the more powerful local control commands. They are a tad
922slower but this mechanism isnt used that much anyway.
923
924\starttyping
925\let\immediateassigned\localcontrolled
926\stoptyping
927
928Say that we define:
929
930\startbuffer
931\edef\TestA
932 {\advance\scratchcounter\plusone}
933\edef\TestB
934 {\localcontrol\TestA
935 \the\scratchcounter}
936\edef\TestC
937 {\localcontrolled{\advance\scratchcounter\plusone}
938 \the\scratchcounter}
939\edef\TestD
940 {\beginlocalcontrol\advance\scratchcounter\plusone\endlocalcontrol
941 \the\scratchcounter}
942\stopbuffer
943
944\typebuffer \getbuffer
945
946With this example:
947
948\startbuffer
949\scratchcounter 10 \meaningasis\TestA
950\scratchcounter 20 \meaningasis\TestB
951\scratchcounter 30 \meaningasis\TestC
952\scratchcounter 40 \meaningasis\TestD
953\stopbuffer
954
955\typebuffer
956
957We get this:
958
959\startlines
960\tttf \getbuffer
961\stoplines
962
963These local control primitives are a bit tricky and error message can be
964confusing. Future versions might have a bit better recovery but in practice it
965works as expected.
966
967An \prm {expandedafter} primitive is also provided as an variant on \prm
968{expandafter} that takes a token list instead of a single token.
969
970\stopsubsection
971
972\startsubsection[title={\prm {semiprotected}, \prm {semiexpanded}, \prm {expand},
973\prm {semiexpand} and \prm {expandactive}}]
974
975These primitives can best be explained with a few examples. The semi boils down to
976a bit more controlled usage of \prm {protected} macros.
977
978\startbuffer
979 \def\Test {test}
980 \def\TestA{\Test}
981\protected \def\TestB{\Test}
982\semiprotected \def\TestC{\Test}
983 \edef\TestD{\Test}
984 \edef\TestE{\TestA}
985 \edef\TestF{\TestB}
986 \edef\TestG{\TestC}
987 \edef\TestH{\normalexpanded{\TestB\TestC}}
988 \edef\TestI{\semiexpanded{\TestB\TestC}}
989 \edef\TestJ{\expand\TestB\expand\TestC}
990 \edef\TestK{\semiexpand\TestB\semiexpand\TestC}
991\stopbuffer
992
993\typebuffer \getbuffer
994
995The effective meanings are given next (we use \prm {meaningasis} for this):
996
997\startlines \tttf
998\meaningasis\Test
999\meaningasis\TestA
1000\meaningasis\TestB
1001\meaningasis\TestC
1002\meaningasis\TestD
1003\meaningasis\TestE
1004\meaningasis\TestF
1005\meaningasis\TestG
1006\meaningasis\TestH
1007\meaningasis\TestI
1008\meaningasis\TestJ
1009\meaningasis\TestK
1010\stoplines
1011
1012I admit that is not yet applied much in \CONTEXT\ as we have no real need for it
1013and I implemented it more out for nostalgic reasons: the kind of selective
1014protect mechanism we have in \MKII.
1015
1016Assuming that \type {} is made active:
1017
1018\starttyping
1019\protected\def{!}
1020
1021\edef\xxxx{}
1022\edef\xxxx{\expandactive}
1023\stoptyping
1024
1025In both cases the meaning will show \type {} so its kind of subtle because in reality
1026they have the following internal representation:
1027
1028\starttyping
1029active char 126
1030protected call
1031\stoptyping
1032
1033\stopsubsection
1034
1035\startsubsection[title={Going ahead with \prm {expandafterpars} and \prm {expandafterspaces}}]
1036
1037\topicindex{expansionafter}
1038
1039Here are again some convenience primitives that simplify coding, remove the need
1040to show off with multistep macros and are nicely expandable. They fit in the
1041repertoire of additional primitives that make macro code look somewhat easier.
1042Here are a few examples:
1043
1044\startbuffer
1045\def\foo{!!} [\expandafterpars \foo \par test]
1046\def\foo{!!} [\expandafterspaces\foo test]
1047
1048\def\foo{!!} \def\oof{\foo} [{\oof} test]
1049\def\foo{!!} \def\oof{\expandafterspaces\foo} [{\oof}test]
1050\stopbuffer
1051
1052\typebuffer
1053
1054These are typically used when building high level interfaces so not many users
1055will see them in document sources.
1056
1057\startlines
1058\getbuffer
1059\stoplines
1060
1061\stopsubsection
1062
1063\startsubsection[title={\prm {afterassigned}}]
1064
1065\topicindex{assignmentsafter}
1066
1067This primitive is a multiple token variant of \prm {afterassignment} and it takes
1068a token list. It might look better in some cases than multiple single token
1069\quote {calls}.
1070
1071\stopsubsection
1072
1073\startsubsection[title={\prm {detokenized}}]
1074
1075\topicindex{serializing}
1076
1077The \prm {string} primitive serializes what comes next, a control sequence or
1078something more primitive string representation or just the (\UTF) character so it
1079does look at what it sees next in some detail. This can give confusing results
1080when the next token is for instance a new line. The \prm {detokenized} is less
1081picky and just serializes the token, so in the next examples an empty lines is
1082what we normally expect it to become: a serialized par token.
1083
1084\def\oof{s\expandafter\foo\string}
1085\def\ofo{d\expandafter\foo\detokenized}
1086\def\foo#1{:[#1]}
1087
1088\startbuffer
1089\oof test
1090\ofo test
1091\oof \relax
1092\ofo \relax
1093\oof \par
1094\ofo \par
1095
1096\oof
1097
1098\ofo
1099
1100done
1101\stopbuffer
1102
1103\typebuffer
1104
1105We need the empty lines and \quote {done} to make sure we see the effect:
1106
1107{\tttf \getbuffer}
1108
1109\stopsubsection
1110
1111\startsubsection[title={\prm {expandtoken} and \prm {expandcstoken}}]
1112
1113\topicindex{expansiontokens}
1114
1115These two are not really needed but can make code look less weird (and
1116impressive) because there are no catcode changes involved. The next example
1117illustrates what they do:
1118
1119\startbuffer
1120\edef\foo{\expandtoken 12 123 } \meaning\foo
1121\edef\oof{\bgroup \egroup} \meaning\oof
1122\edef\oof{\expandcstoken \bgroup\expandcstoken \egroup} \meaning\oof
1123\edef\oof{\expandcstoken \foo } \meaning\oof
1124\stopbuffer
1125
1126\typebuffer
1127
1128So \prm {expandtoken} expects two arguments: a catcode and a character number.
1129The \prm {expandcstoken} will only look at control sequences representing a
1130character.
1131
1132\startlines
1133\getbuffer
1134\stoplines
1135
1136\stopsubsection
1137
1138\stopsection
1139
1140\startsection[title=Grouping]
1141
1142\startsubsection[title={\prm {endsimplegroup}}]
1143
1144\topicindex{groupingending}
1145
1146This feature might look somewhat weird so just ignore that it is there. It is one
1147of these features that might never make it in a engine when discussed in
1148committee but it comes in handy in \CONTEXT, so:
1149
1150\startbuffer
1151\def\foo{\beginsimplegroup\bf\let\next}
1152
1153\foo{test}
1154\foo{test\endgroup
1155\foo{test\endsimplegroup
1156\foo{test\egroup
1157\stopbuffer
1158
1159\typebuffer
1160
1161These lines typeset as:
1162
1163\startlines \getbuffer \stoplines
1164
1165The \prm {beginsimplegroup} primitives signals that any end group command, except
1166\prm {endmathgroup} will wrap up the current group. The \prm {endsimplegroup} is
1167sort of redundant but fits in anyway.
1168
1169The also \LUAMETATEX\ specific \prm {beginmathgroup} and \prm {endmathgroup}
1170commands are like \prm {begingroup} and \prm {endgroup} but restore the mathstyle
1171when it has been changed in the group.
1172
1173\startsubsection[title={\prm {aftergrouped}}]
1174
1175\topicindex{groupingafter}
1176
1177There is a new experimental feature that can inject multiple tokens to after the group
1178ends. An example demonstrate its use:
1179
1180\startbuffer
1181{
1182 \aftergroup A \aftergroup B \aftergroup C
1183test 1 : }
1184
1185{
1186 \aftergrouped{What comes next 1}
1187 \aftergrouped{What comes next 2}
1188 \aftergrouped{What comes next 3}
1189test 2 : }
1190
1191
1192{
1193 \aftergroup A \aftergrouped{What comes next 1}
1194 \aftergroup B \aftergrouped{What comes next 2}
1195 \aftergroup C \aftergrouped{What comes next 3}
1196test 3 : }
1197
1198{
1199 \aftergrouped{What comes next 1} \aftergroup A
1200 \aftergrouped{What comes next 2} \aftergroup B
1201 \aftergrouped{What comes next 3} \aftergroup C
1202test 4 : }
1203\stopbuffer
1204
1205\typebuffer
1206
1207This gives:
1208
1209\startpacked\getbuffer\stoppacked
1210
1211\stopsubsection
1212
1213\startsubsection[title={\prm {atendofgroup} and \prm {atendofgrouped}}]
1214
1215\topicindex{groupingending}
1216
1217These are variants of \prm {aftergroup} and \prm {aftergrouped} but they happen
1218{\em before} the groups is closed. It is one of these primitives that is not
1219really needed but that can make code (and tracing) cleaner, which is one of the
1220objectives (at least for \CONTEXT).
1221
1222\stopsubsection
1223
1224\stopsection
1225
1226\startsection[title=Conditions]
1227
1228\startsubsection[title={\prm{ifabsnum} and \prm {ifabsdim}}]
1229
1230\topicindex{conditions}
1231
1232There are two tests that we took from \PDFTEX:
1233
1234\startbuffer
1235\ifabsnum 10 = 10
1236 the same number
1237\fi
1238\ifabsdim 10pt = 10pt
1239 the same dimension
1240\fi
1241\stopbuffer
1242
1243\typebuffer
1244
1245This gives
1246
1247\blank {\tt \getbuffer} \blank
1248
1249\stopsubsection
1250
1251\startsubsection[title={Comparing}]
1252
1253When comparing (for instance) to numbers the a \type {=}, \type {<} or \type {>}
1254is used. In \LUAMETATEX\ you can negate such a comparison by \type {!}, as in
1255\type {!=}, \type {!<} or \type {!>}. Multiple \type {!} flip that state.
1256
1257In addition to these \ASCII\ combinations, we also support some \UNICODE\
1258variants. The extra comparison options are:
1259
1260\starttabulate[lccl]
1261\DB character \BC \BC \BC operation \NC \NR
1262\TB
1263\NC \type {0x2208} \NC $\tocharacter"2208$ \NC \NC element of \NC \NR
1264\NC \type {0x2209} \NC $\tocharacter"2209$ \NC \NC not element of \NC \NR
1265\NC \type {0x2260} \NC $\tocharacter"2260$ \NC \type {!=} \NC not equal \NC \NR
1266\NC \type {0x2264} \NC $\tocharacter"2264$ \NC \type {!>} \NC less equal \NC \NR
1267\NC \type {0x2265} \NC $\tocharacter"2265$ \NC \type {!<} \NC greater equal \NC \NR
1268\NC \type {0x2270} \NC $\tocharacter"2270$ \NC \NC not less equal \NC \NR
1269\NC \type {0x2271} \NC $\tocharacter"2271$ \NC \NC not greater equal \NC \NR
1270\LL
1271\stoptabulate
1272
1273\stopsubsection
1274
1275\startsubsection[title={\prm{ifzeronum}, \prm {ifzerodim}}]
1276
1277\topicindex {conditionsnumbers}
1278\topicindex {conditionsdimensions}
1279
1280Their name tells what they test for: zero (point) values.
1281
1282\stopsubsection
1283
1284\startsubsection[title={\prm{ifcmpnum}, \prm {ifcmpdim}, \prm {ifnumval}, \prm
1285{ifdimval}, \prm {ifchknum} and \prm {ifchkdim}}]
1286
1287\topicindex {conditionsnumbers}
1288\topicindex {conditionsdimensions}
1289\topicindex {numbers}
1290\topicindex {dimensions}
1291
1292New are the ones that compare two numbers or dimensions:
1293
1294\startbuffer
1295\ifcmpnum 5 8 less \or equal \else more \fi
1296\ifcmpnum 5 5 less \or equal \else more \fi
1297\ifcmpnum 8 5 less \or equal \else more \fi
1298\stopbuffer
1299
1300\typebuffer \blank {\tt \getbuffer} \blank
1301
1302and
1303
1304\startbuffer
1305\ifcmpdim 5pt 8pt less \or equal \else more \fi
1306\ifcmpdim 5pt 5pt less \or equal \else more \fi
1307\ifcmpdim 8pt 5pt less \or equal \else more \fi
1308\stopbuffer
1309
1310\typebuffer \blank {\tt \getbuffer} \blank
1311
1312There are also some number and dimension tests. All four expose the \type {\else}
1313branch when there is an error, but two also report if the number is less, equal
1314or more than zero.
1315
1316\startbuffer
1317\ifnumval 123 \or < \or = \or > \or ! \else ? \fi
1318\ifnumval 0 \or < \or = \or > \or ! \else ? \fi
1319\ifnumval 123 \or < \or = \or > \or ! \else ? \fi
1320\ifnumval abc \or < \or = \or > \or ! \else ? \fi
1321
1322\ifdimval 123pt \or < \or = \or > \or ! \else ? \fi
1323\ifdimval 0pt \or < \or = \or > \or ! \else ? \fi
1324\ifdimval 123pt \or < \or = \or > \or ! \else ? \fi
1325\ifdimval abcpt \or < \or = \or > \or ! \else ? \fi
1326\stopbuffer
1327
1328\typebuffer \blank {\tt \getbuffer} \blank
1329
1330\startbuffer
1331\ifchknum 123 \or okay \else bad \fi
1332\ifchknum 0 \or okay \else bad \fi
1333\ifchknum 123 \or okay \else bad \fi
1334\ifchknum abc \or okay \else bad \fi
1335
1336\ifchkdim 123pt \or okay \else bad \fi
1337\ifchkdim 0pt \or okay \else bad \fi
1338\ifchkdim 123pt \or okay \else bad \fi
1339\ifchkdim abcpt \or okay \else bad \fi
1340\stopbuffer
1341
1342\typebuffer \blank {\tt \getbuffer} \blank
1343
1344The last checked values are available in \prm {lastchknum} and \prm {lastchkdim}.
1345These dont obey grouping.
1346
1347The two primitives \prm {ifchkdimension} and \prm {ifchknumber} are like \prm
1348{ifchkdimen} and \prm {ifchknum}but are more rigorous: the short ones quit
1349scanning at a match where after the match there can be anything, while the long
1350variants dont accept following crap.
1351
1352\stopsubsection
1353
1354\startsubsection[title={\prm {ifmathstyle} and \prm {ifmathparameter}}]
1355
1356These two are variants on \prm {ifcase} where the first one operates with values
1357in ranging from zero (display style) to seven (cramped script script style) and
1358the second one can have three values: a parameter is zero, has a value or is
1359unset. The \type {\ifmathparameter} primitive takes a proper parameter name and a
1360valid style identifier (a primitive identifier or number). The \type
1361{\ifmathstyle} primitive is equivalent to \type {\ifcase \mathstyle}.
1362
1363\stopsubsection
1364
1365\startsubsection[title={\prm {ifempty}}]
1366
1367This primitive tests for the following token (control sequence) having no
1368content. Assuming that \type {\empty} is indeed empty, the following two are
1369equivalent:
1370
1371\starttyping
1372\ifempty\whatever
1373\ifx\whatever\empty
1374\stoptyping
1375
1376There is no real performance gain here, its more one of these extensions that
1377lead to less clutter in tracing.
1378
1379\stopsubsection
1380
1381\startsubsection[title={\prm {ifrelax}}]
1382
1383This primitive complements \type {\ifdefined}, \type {\ifempty} and \type
1384{\ifcsname} so that we have all reasonable tests directly available.
1385
1386\stopsubsection
1387
1388\startsubsection[title={\prm {ifboolean}}]
1389
1390This primitive tests for nonzero, so the next variants are similar
1391
1392\starttyping
1393 \ifcase <integer>.F.\else .T.\fi
1394\unless\ifcase <integer>.T.\else .F.\fi
1395 \ifboolean<integer>.T.\else .F.\fi
1396\stoptyping
1397
1398\stopsubsection
1399
1400\startsubsection[title={\prm {iftok} and \prm {ifcstok}}]
1401
1402\topicindex {conditionstokens}
1403\topicindex {tokens}
1404
1405Comparing tokens and macros can be done with \type {\ifx}. Two extra test are
1406provided in \LUAMETATEX:
1407
1408\startbuffer
1409\def\ABC{abc} \def\DEF{def} \def\PQR{abc} \newtoks\XYZ \XYZ {abc}
1410
1411\iftok{abc}{def}\relax (same) \else [different] \fi
1412\iftok{abc}{abc}\relax [same] \else (different) \fi
1413\iftok\XYZ {abc}\relax [same] \else (different) \fi
1414
1415\ifcstok\ABC \DEF\relax (same) \else [different] \fi
1416\ifcstok\ABC \PQR\relax [same] \else (different) \fi
1417\ifcstok{abc}\ABC\relax [same] \else (different) \fi
1418\stopbuffer
1419
1420\typebuffer \startpacked[blank] {\tt\nospacing\getbuffer} \stoppacked
1421
1422You can check if a macro is defined as protected with \type {\ifprotected} while
1423frozen macros can be tested with \type {\iffrozen}. A provisional \type
1424{\ifusercmd} tests will check if a command is defined at the user level (and this
1425one might evolve).
1426
1427\stopsubsection
1428
1429\startsubsection[title={\prm {ifhastok}, \prm {ifhastoks}, \prm {ifhasxtoks} and \prm {ifhaschar}}]
1430
1431\topicindex {conditionstokens}
1432\topicindex {tokens}
1433
1434The first three test primitives run over a token list in order to encounter a
1435single token or a sequence. The \type {x} variants applies expansion.
1436
1437\startbuffer
1438\def\ab {ab}
1439\def\abc{abc}
1440\ifhastok 1 {12} Y\else N\fi
1441\ifhastoks {ab} {abc}Y\else N\fi
1442\ifhastoks {ab} {\abc}Y\else N\fi
1443\ifhastoks {\ab}{\abc}Y\else N\fi
1444\ifhasxtoks{ab} {\abc}Y\else N\fi
1445\ifhastok 3 {12} Y\else N\fi
1446\ifhastoks {de} {abc}Y\else N\fi
1447\stopbuffer
1448
1449\typebuffer \startpacked[blank] {\tt\nospacing\getbuffer} \stoppacked
1450
1451The \prm {ifhaschar} primitive differs from \prm {ifhastok} in that it handles
1452nested balanced \quote {lists}, as in:
1453
1454\startbuffer
1455\ifhastok a {abc}Y\else N\fi
1456\ifhaschar a {abc}Y\else N\fi
1457\ifhastok a{{a}bc}Y\else N\fi
1458\ifhaschar a{{a}bc}Y\else N\fi
1459\stopbuffer
1460
1461\typebuffer \startpacked[blank] {\tt\nospacing\getbuffer} \stoppacked
1462
1463\stopsubsection
1464
1465\startsubsection[title={\prm {ifarguments}, \prm {ifparameters} and \prm {ifparameter}}]
1466
1467These are part of the extended macro argument parsing features. The \prm
1468{ifarguments} condition is like an \prm {ifcase} where the number is the
1469picked up number of arguments. The number reflects the {\em last} count, so
1470successive macro expansions will adapt the value. The \prm {ifparameters}
1471counts till the first empty parameter and the \prm {ifparameter} (singular)
1472takes a parameter reference (like \type {#2}) and again is an \prm {ifcase}
1473where zero means a bad reference, one a nonempty argument and two an empty
1474one. A typical usage is:
1475
1476\starttyping
1477\def\foo#1#2
1478 {\ifparameter#1\or one\fi
1479 \ifparameter#2\or two\fi}
1480\stoptyping
1481
1482No expansion of arguments takes place here but you can use a test like this:
1483
1484\starttyping
1485\def\foo#1#2
1486 {\iftok{#1}{}\else one\fi
1487 \iftok{#2}{}\else two\fi}
1488\stoptyping
1489
1490
1491\stopsubsection
1492
1493\startsubsection[title={\prm {ifcondition}}]
1494
1495\topicindex {conditions}
1496
1497This is a somewhat special one. When you write macros conditions need to be
1498properly balanced in order to let \TEXs fast branch skipping work well. This new
1499primitive is basically a noop flagged as a condition so that the scanner can
1500recognize it as an iftest. However, when a real test takes place the work is
1501done by what follows, in the next example \tex {something}.
1502
1503\starttyping
1504\unexpanded\def\something#1#2
1505 {\edef\tempa{#1}
1506 \edef\tempb{#2}
1507 \ifx\tempa\tempb}
1508
1509\ifcondition\something{a}{b}
1510 \ifcondition\something{a}{a}
1511 true 1
1512 \else
1513 false 1
1514 \fi
1515\else
1516 \ifcondition\something{a}{a}
1517 true 2
1518 \else
1519 false 2
1520 \fi
1521\fi
1522\stoptyping
1523
1524If you are familiar with \METAPOST, this is a bit like \type {vardef} where the macro
1525has a return value. Here the return value is a test.
1526
1527Experiments with something \type {\ifdef} actually worked ok but were rejected
1528because in the end it gave no advantage so this generic one has to do. The \type
1529{\ifcondition} test is basically is a noop except when branches are skipped.
1530However, when a test is expected, the scanner gobbles it and the next test result
1531is used. Here is an other example:
1532
1533\startbuffer
1534\def\mytest#1
1535 {\ifabsdim#1>0pt\else
1536 \expandafter \unless
1537 \fi
1538 \iftrue}
1539
1540\ifcondition\mytest{10pt}\relax nonzero \else zero \fi
1541\ifcondition\mytest {0pt}\relax nonzero \else zero \fi
1542\stopbuffer
1543
1544\typebuffer \blank {\tt \getbuffer} \blank
1545
1546The last expansion in a macro like \type {\mytest} has to be a condition and here
1547we use \type {\unless} to negate the result.
1548
1549\stopsubsection
1550
1551\startsubsection[title={\prm {orelse} and \prm {orunless}}]
1552
1553Sometimes you have successive tests that, when laid out in the source lead to
1554deep trees. The \type {\ifcase} test is an exception. Experiments with \type
1555{\ifcasex} worked out fine but eventually were rejected because we have many
1556tests so it would add a lot. As \LUAMETATEX\ permitted more experiments,
1557eventually an alternative was cooked up, one that has some restrictions but is
1558relative lightweight. It goes like this:
1559
1560\starttyping
1561\ifnum\count0<10
1562 less
1563\orelse\ifnum\count0=10
1564 equal
1565\else
1566 more
1567\fi
1568\stoptyping
1569
1570The \type {\orelse} has to be followed by one of the if test commands, except
1571\type {\ifcondition}, and there can be an \type {\unless} in front of such a
1572command. These restrictions make it possible to stay in the current condition
1573(read: at the same level). If you need something more complex, using \type
1574{\orelse} is probably unwise anyway. In case you wonder about performance, there
1575is a little more checking needed when skipping branches but that can be
1576neglected. There is some gain due to staying at the same level but that is only
1577measurable when you runs tens of millions of complex tests and in that case it is
1578very likely to drown in the real action. Its a convenience mechanism, in the
1579sense that it can make your code look a bit easier to follow.
1580
1581There is a nice side effect of this mechanism. When you define:
1582
1583\starttyping
1584\def\quitcondition{\orelse\iffalse}
1585\stoptyping
1586
1587you can do this:
1588
1589\starttyping
1590\ifnum\count0<10
1591 less
1592\orelse\ifnum\count0=10
1593 equal
1594 \quitcondition
1595 indeed
1596\else
1597 more
1598\fi
1599\stoptyping
1600
1601Of course it is only useful at the right level, so you might end up with cases like
1602
1603\starttyping
1604\ifnum\count0<10
1605 less
1606\orelse\ifnum\count0=10
1607 equal
1608 \ifnum\count2=30
1609 \expandafter\quitcondition
1610 \fi
1611 indeed
1612\else
1613 more
1614\fi
1615\stoptyping
1616
1617The \prm {orunless} variant negates the next test, just like \prm {unless}. In
1618some cases these commands look at the next token to see if it is an iftest so
1619a following negation will not work (read: making that work would complicate the
1620code and hurt efficiency too). Side note: interesting is that in \CONTEXT\ we
1621hardly use this kind of negation.
1622
1623\stopsubsection
1624
1625\startsubsection[title={\prm {ifflags}}]
1626
1627This checker deal with control sequences. You can check if a command is a
1628protected one, that is, defined with the \type {\protected} prefix. A command is
1629frozen when it has been defined with the \type {\frozen} prefix. Beware: only
1630macros can be frozen. A user command is a command that is not part of the
1631predefined set of commands. This is an experimental command. The flag values can
1632be queried with \typ {tex.getflagvalues}.
1633
1634\stopsubsection
1635
1636\stopsection
1637
1638\startsection[title={Control and debugging}]
1639
1640\startsubsection[title={Tracing}]
1641
1642\topicindex {tracing}
1643
1644If \prm {tracingonline} is larger than2, the node list display will also print
1645the node number of the nodes as well as set attributes (these can be made verbose
1646by a callback). We have only a generic whatsit but again a callback can be used
1647to provide detail. So, when a box is shown in \CONTEXT\ you will see quite a lot
1648more than in other engines. Because nodes have more fields, more is shown anyway,
1649and for nodes that have sublists (like discretionaries) these are also shown. All
1650that could have been delegated to \LUA\ but it felt wrong to not made that a core
1651engine feature.
1652
1653The \prm {tracingpenalties} parameter triggers the line break routine to report
1654the applied interline penalties to the output.
1655
1656When \prm {tracingcommands} is larger than 3 the mode switch will be not be
1657prefixed to the \type {{command}} but get its own \type {[line]}.
1658
1659When \prm {tracinghyphenation} is set to 1 duplicate patterns are reported (in
1660\CONTEXT\ we default to that) and higher values will also show details about the
1661\LUA\ hyphenation (exception) feedback loop discussed elsewhere.
1662
1663When set to 1 the \prm {tracingmath} variable triggers the reporting of the mode
1664(inline or display) an mlist is processed. Other new tracing commands are
1665discussed where the mechanisms that they relate to are introduced.
1666
1667The \prm {tracingnodes} variable makes that when a node list is reported the node
1668numbers are also shown. This is only useful when you have callbacks that access
1669nodes.
1670
1671\starttabulate[lp]
1672\DB value \BC effect \NC \NR
1673\TB
1674\NC 1 \NC show node numbers in lists \NC \NR
1675\NC 2 \NC also show numbers of attribute nodes \NC \NR
1676\NC 3 \NC also show glue spec node numbers \NC \NR
1677\LL
1678\stoptabulate
1679
1680When the \prm {shownodedetails} variable is set to a value larger than zero and a
1681node is shown (in a list) then more details will be revealed. This can be rather
1682verbose because in \LUAMETATEX\ node carry more properties than in traditional
1683\TEX\ and \LUATEX. A value larger than one will also show details of attributes
1684that are bound to nodes.
1685
1686The \prm {tracinglevels} variable is a bitset and offers the following features:
1687
1688\starttabulate[lp]
1689\DB value \BC effect \NC \NR
1690\TB
1691\NC 1 \NC show group level \NC \NR
1692\NC 2 \NC show input level \NC \NR
1693\NC 4 \NC show catcode regime \NC \NR
1694\LL
1695\stoptabulate
1696
1697So a value of7 shows them all. In \CONTEXT\ we set this variable to3 which
1698gives a rather verbose log when tracing is on but in the end itsnot that bad
1699because using some of the newer programming related primitive can save tracing.
1700
1701The \prm {tracinglists} variable will show some of the (intermediate) lists that
1702get processed. It is there mainly for development but might evolve.
1703
1704Because in \LUATEX\ the saving and restoring of locally redefined macros and set
1705variables is optimized a bit in order to prevent redundant stack usage, there
1706will be less tracing visible.
1707
1708Also, because we have a more extensive macro argument parser, a fast path (and
1709less storage demands) for macros with no arguments, and flags that can be set for
1710macros the way macros are traced can be different in details (we therefore have
1711for instance \prm {meaningfull} (double ls indeed) and \prm {meaningless} as
1712variants of \prm {meaning} as well as \prm {meaningasis} for more literal
1713alternative). The \prm {meaningful} and \prm {meaningles} variants show no body
1714but do show the preamble when we have arguments.
1715
1716\stopsubsection
1717
1718
1719
1720
1721
1722
1723
1724
1725
1726
1727
1728
1729\startsubsection[title={\prm {lastnodetype}, \prm {lastnodesubtype}, \prm
1730{currentiftype}}]
1731
1732The \ETEX\ command \prm {lastnodetype} returns the node codes as used in the
1733engine. You can query the numbers at the \LUA\ end if you need the actual values.
1734The parameter \type {\internalcodesmode} is no longer provided as compatibility
1735switch because \LUATEX\ has more cq. some different nodes and it makes no sense
1736to be incompatible with the \LUA\ end of the engine. The same is true for \prm
1737{currentiftype}, as we have more conditionals and also use a different order.
1738The \prm {lastnodesubtype} is a bonus and again reports the codes used
1739internally. During development these might occasionally change, but eventually
1740they will be stable.
1741
1742\stopsubsection
1743
1744\startsubsection[title={\prm {lastboundary} and \prm {unboundary}}]
1745
1746There are \prm {lastpenalty}, \prm {lastskip}, \prm {lastkern} and \prm {lastbox}
1747primitives and \LUAMETATEX\ also offers \prm {lastboundary} which gives the value
1748assigned to a user boundary node. This means that we also have a \prm
1749{unboundary} to complement the other \tex {un...} primitives.
1750
1751\stopsubsection
1752
1753\startsubsection[title=Nodes]
1754
1755\topicindex {nodes}
1756
1757The \ETEX\ primitive \prm {lastnodetype} is not honest in reporting the
1758internal numbers as it uses its own values. But you can set \type
1759{\internalcodesmode} to a nonzero value to get the real ids instead. In
1760addition there is \prm {lastnodesubtype}.
1761
1762Another last one is \prm {lastnamedcs} which holds the last match but this one
1763should be used with care because one never knows if in the meantime something
1764else \quote {last} has been seen.
1765
1766\stopsubsection
1767
1768\stopsection
1769
1770\startsection[title=Kerns and penalties]
1771
1772\startsubsection[title=\prm {hkern} and \prm {vkern}]
1773
1774\topicindex {kerns}
1775
1776These two primitives complement \prm {hskip} and \prm {vskip} and force the right
1777mode when issued. Contrary to the skips, internally we still have a common kern
1778command code but that is not something the user has to worry about.
1779
1780\stopsubsection
1781
1782\startsubsection[title=\prm {hpenalty} and \prm {vpenalty}]
1783
1784\topicindex {penalties}
1785
1786As the kern and skip related primitives mentioned in the in the previous section
1787these two primitives are there fort consistency: they force the right (related)
1788mode. (Sometimes being a bit more explicit is cleaner.)
1789
1790\stopsubsection
1791
1792\stopsection
1793
1794\startsection[title=Scanning]
1795
1796\startsubsection[title=Keywords]
1797
1798\topicindex {keywords}
1799\topicindex {scanningkeywords}
1800
1801Some primitives accept one or more keywords and \LUAMETATEX\ adds some more. In
1802order to deal with this efficiently the keyword scanner has been optimized, where
1803even the context was taken into account. As a result the scanner was quite a bit
1804faster. This kind of optimization was a graduate process the eventually ended up
1805in what we have now. In traditional \TEX\ (and also \LUATEX) the order of
1806keywords is sometimes mixed and sometimes prescribed. In most cases only one
1807occurrence is permitted. So, for instance, this is valid in \LUATEX:
1808
1809\starttyping
1810\hbox attr 123 456 attr 123 456 spread 10cm { }
1811\hrule width 10cm depth 3mm
1812\hskip 3pt plus 2pt minus 1pt
1813\stoptyping
1814
1815The \type {attr} comes before the \type {spread}, rules can have multiple mixed
1816dimension specifiers, and in glue the optional \type {minus} part always comes
1817last. The last two commands are famous for look ahead side effects which is why
1818macro packages will end them with something not keyword, like \type {\relax},
1819when needed.
1820
1821In \LUAMETATEX\ the following is okay. Watch the few more keywords in box and
1822rule specifications.
1823
1824\starttyping
1825\hbox reverse to 10cm attr 123 456 orientation 4 xoffset 10pt spread 10cm { }
1826\hrule xoffset 10pt width 10cm depth 3mm
1827\hskip 3pt minus 1pt plus 2pt
1828\stoptyping
1829
1830Here the order is not prescribed and, as demonstrated with the box specifier, for
1831instance dimensions (specified by \type {to} or \type {spread} can be overloaded
1832by later settings. In case you wonder if that breaks compatibility: in some way
1833it does but bad or sloppy keyword usage breaks a run anyway. For instance \type
1834{minuscule} results in \type {minus} with no dimension being seen. So, in the end
1835the user should not noticed it and when a user does, the macro package already
1836had an issue that had to be fixed.
1837
1838\stopsubsection
1839
1840\startsubsection[title={\prm {norelax}}]
1841
1842\topicindex{relaxing}
1843
1844There are a few cases where the \TEX\ scanned skips over spaces and \prm {relax} as
1845well as quits on a \prm {relax} in which case it gets pushed back. An example is
1846given below:
1847
1848\startbuffer
1849\edef\TestA{\ifnum1=1\relax Y\else N\fi} \meaningasis\TestA
1850\edef\TestB{\ifnum1=1\norelax Y\else N\fi} \meaningasis\TestB
1851\stopbuffer
1852
1853\typebuffer
1854
1855The second line also contains a sentinel but this time we use \prm {norelax}
1856which will not be pushed back. So, this feature is just a trick to get rid of (in
1857itself reasonable) side effects.
1858
1859\startlines\getbuffer \stoplines
1860
1861\stopsubsection
1862
1863\startsubsection[title={\prm {ignorepars}}]
1864
1865This primitive is like \prm {ignorespaces} but also skips paragraph ending
1866commands (normally \prm {par} and empty lines).
1867
1868\stopsubsection
1869
1870\startsubsection[title={\prm {futureexpand}, \prm {futureexpandis}, \prm {futureexpandisap}}]
1871
1872\topicindex{expansionfuture}
1873
1874These commands are used as:
1875
1876\starttyping
1877\futureexpand\sometoken\whenfound\whennotfound
1878\stoptyping
1879
1880When there is no match and a space was gobbled a space will be put back. The
1881\type {is} variant doesnt do that while the \type {isap} even skips \type
1882{\pars}, These characters stand for \quote {ignorespaces} and \quote
1883{ignorespacesandpars}.
1884
1885\stopsubsection
1886
1887\stopsection
1888
1889\startsection[title=Macros]
1890
1891\startsubsection[title={\prm {lettonothing} and \prm {glettonothing}}]
1892
1893This primitive is equivalent to:
1894
1895\starttyping
1896\protected\def\lettonothing#1{\def#1{}}
1897\stoptyping
1898
1899and although it might feel faster (only measurable with millions of calls) its
1900mostly there because it is easier on tracing (less clutter). An advantage over
1901letting to an empty predefined macro is also that in tracing we keep seeing the
1902name (relaxing would show the relax equivalent).
1903
1904\stopsubsection
1905
1906\startsubsection[title={\prm {glet}}]
1907
1908This primitive is similar to:
1909
1910\starttyping
1911\protected\def\glet{\global\let}
1912\stoptyping
1913
1914but faster (only measurable with millions of calls) and probably more convenient
1915(after all we also have \type {\gdef}).
1916
1917\stopsubsection
1918
1919\startsubsection[title={\prm {defcsname}, \prm {edefcsname}, \prm {gdefcsname} and \prm {xdefcsname}}]
1920
1921Although we can implement these primitives easily using macros it makes sense,
1922given the popularity of \prm {csname} to have these as primitives. It also saves
1923some \prm {expandafter} usage and it looks a bit better in the source.
1924
1925\starttyping
1926\gdefcsname foo\endcsname{oof}
1927\stoptyping
1928
1929\stopsubsection
1930
1931\startsubsection[title={\prm {letcsname} and \prm {gletcsname}}]
1932
1933These can also be implemented using macros but again they are natively provided
1934by the engine for the same reasons: less code and less tracing clutter.
1935
1936\starttyping
1937\gletcsname foo\endcsname \relax
1938\stoptyping
1939
1940\stopsubsection
1941
1942\startsubsection[title={\prm{cdef}, \prm {cdefcsname} and \prm {constant}}]
1943
1944These primitives are like \prm {edef} and \prm {edefcsname} but tag the macro as
1945being kind of simple, which means that in some scenarios they are serialized in a
1946fast way, not going through the expansion machinery. This is actually an
1947experiment but it will stay. The \prm {constant} prefix can be used with other
1948definition primitives instead.
1949
1950\stopsubsection
1951
1952\startsubsection[title={\prm {csstring}, \prm {begincsname} and \prm {lastnamedcs}}]
1953
1954These are somewhat special. The \prm {csstring} primitive is like
1955\prm {string} but it omits the leading escape character. This can be
1956somewhat more efficient than stripping it afterwards.
1957
1958The \prm {begincsname} primitive is like \prm {csname} but doesnt create
1959a relaxed equivalent when there is no such name. It is equivalent to
1960
1961\starttyping
1962\ifcsname foo\endcsname
1963 \csname foo\endcsname
1964\fi
1965\stoptyping
1966
1967The advantage is that it saves a lookup (dont expect much speedup) but more
1968important is that it avoids using the \prm {if} test. The \prm {lastnamedcs}
1969is one that should be used with care. The above example could be written as:
1970
1971\starttyping
1972\ifcsname foo\endcsname
1973 \lastnamedcs
1974\fi
1975\stoptyping
1976
1977This is slightly more efficient than constructing the string twice (deep down in
1978\LUATEX\ this also involves some \UTF8 juggling), but probably more relevant is
1979that it saves a few tokens and can make code a bit more readable.
1980
1981Active characters are stored in the hash with a special prefix sequence prepended
1982to the character: \prm {csactive} or the never used \UTF\ representation of \type
1983{UFFFF}.
1984
1985\stopsubsection
1986
1987\startsubsection[title={\prm {futuredef} and \prm {futurecsname}}]
1988
1989This is just the definition variant of \prm {futurelet} and a simple example
1990shows the difference:
1991
1992\startbuffer
1993\def\whatever{[\next:\meaning\next]}
1994\futurelet\next\whatever A
1995\futuredef\next\whatever B
1996\stopbuffer
1997
1998\typebuffer
1999
2000\getbuffer
2001
2002The next one was more an experiment that then stayed around, just to see what
2003surprising abuse of this primitive will happen:
2004
2005\startbuffer
2006\def\whateveryes{[YES]}
2007\def\whatevernop{[NOP]}
2008\let\whatever\undefined
2009\futurecsname\whatevernop whatever\endcsname
2010\futurecsname\whatevernop whateveryes\endcsname
2011\stopbuffer
2012
2013\typebuffer
2014
2015When the assembles control sequence is undefined the given one will be expanded,
2016a weird one, right? I will probably apply it some day in cases where I want less
2017tracing and a more direct expansion of an assembled name.
2018
2019\getbuffer
2020
2021Here is a usage example:
2022
2023\starttyping
2024\xdef\Whatever{\futurecsname\whatevernop whatever\endcsname}
2025\xdef\Whatever{\futurecsname\whateveryes whateveryes\endcsname}
2026\xdef\Whatever{\ifcsname whatever\endcsname\lastnamedcs\else\whatevernop\fi}
2027\xdef\Whatever{\ifcsname whateveryes\endcsname\lastnamedcs\else\whatevernop\fi}
2028\xdef\Whatever{\ifcsname whatever\endcsname\csname whatever\endcsname\else\whatevernop\fi}
2029\xdef\Whatever{\ifcsname whateveryes\endcsname\csname whateveryes\endcsname\else\whatevernop\fi}
2030\stoptyping
2031
2032The timings for one million times defining each of these definitions are 0.277,
20330.313, 0.310, 0.359, 0.352 and 0.573 seconds (on a 2018 Dell 7250 Precision
2034laptop with mobile E31505M v6 processor), so there is a little gain here, but of
2035course in practice no one will notice that because not that many such macros are
2036defined (or used).
2037
2038\stopsubsection
2039
2040\startsubsection[title=Prefixes]
2041
2042Quite some primitive usage can be preceded by a prefix. For instance assignments
2043can be \prm {global} and macros can be defined \prm {protected}. In \LUAMETATEX\
2044we have more prefixes, like \prm {tolerant} that signals a different way of
2045interpreting macro arguments and \type {permanent} that flags a definition in
2046away that, when overload protection is enabled, will prevent redefinition.
2047Prefixes like \prm {immediate} and its counterpart \prm {deferred} are backend
2048related and, as we dont have one in the engine, are not doing much. They are
2049just intercepted and passed to e.g.\ some \LUA\ function called so that one can
2050use them to construct additional (\LUA\ based) pseudo primitives. Various
2051prefixes are discussed elsewhere.
2052
2053\stopsubsection
2054
2055\startsubsection[title=Arguments]
2056
2057\topicindex {macrosarguments}
2058
2059Again this is experimental and (used and) discussed in document that come with the
2060\CONTEXT\ distribution. When defining a macro you can do this:
2061
2062\starttyping
2063\def\foo(#1)#2{...}
2064\stoptyping
2065
2066Here the first argument between parentheses is mandate. But the magic
2067prefix \prm {tolerant} makes that limitation go away:
2068
2069\starttyping
2070\tolerant\def\foo(#1)#2{...}
2071\stoptyping
2072
2073A variant is this:
2074
2075\starttyping
2076\tolerant\def\foo(#1)#*(#2){...}
2077\stoptyping
2078
2079Here we have two optional arguments, possibly be separated by spaces. There are
2080more parsing options:
2081
2082\starttabulate[Ti2l]
2083\FL
2084\NC \NC keep the braces \NC \NR
2085\NC \NC discard and dont count the argument \NC \NR
2086\NC \NC remove leading an trailing spaces and pars \NC \NR
2087\NC = \NC braces are mandate \NC \NR
2088\NC \NC braces are mandate and kept \NC \NR
2089\NC \NC keep leading spaces \NC \NR
2090\ML
2091\NC 19 \NC an argument \NC \NR
2092\NC 0 \NC discard but count the argument \NC \NR
2093\ML
2094\NC * \NC ignore spaces \NC \NR
2095\NC . \NC ignore pars and spaces \NC \NR
2096\NC , \NC push back space when no match \NC \NR
2097\ML
2098\NC : \NC pick up scanning here \NC \NR
2099\NC ; \NC quit scanning \NC \NR
2100\LL
2101\stoptabulate
2102
2103For the moment we leave it to your fantasy what these options do. Most probably
2104only make sense when you write a bit more complex macros. Just try to imagine
2105what this does:
2106
2107\starttyping
2108\permanent\tolerant\global\protected\def\foo(#1)#*#;[#2]#:#3{...}
2109\stoptyping
2110
2111Of course complex combinations can be confusing because after all \TEX\ is
2112parsing for (multitoken) delimiters and will happily gobble the whole file if
2113you are not careful. You can quit scanning with \prm {ignorearguments} if you
2114want:
2115
2116\starttyping
2117\mymacro 123\ignorearguments
2118\stoptyping
2119
2120which of course only makes sense when used in a nested call where an already
2121picked up arguments is processed further. A not (yet) discussed feature of the
2122parser is that it will happily skip tokens that have the (probably seldom used)
2123ignored characters property.
2124
2125When you use tracing or see error messages arguments defined using for instance
2126\type {#=} will have their usual number in the macro body, so you need to keep
2127track of the numbers.
2128
2129All this is rather easy on the engine and although it might have a little impact
2130on performance this has been compensated by some more efficiency in the macro
2131parser and engine in general and of course you can gain back some by using these
2132features.
2133
2134\stopsubsection
2135
2136\startsubsection[title={\prm {parametermark}}]
2137
2138The meaning of primitive \prm {parametermark} is equivalent to \type {#} in a macro
2139definition, just like \prm {alignmark} is in an alignment. It can be used to circumvent
2140catcode issues. The normal \quotation {duplicate them when nesting} rules apply.
2141
2142\startbuffer
2143\def\foo\parametermark1
2144 {\def\oof\parametermark\parametermark1
2145 {[\parametermark1:\parametermark\parametermark1]}}
2146\stopbuffer
2147
2148\typebuffer \getbuffer
2149
2150Here \type {\foo{X}\oof{Y}} gives: \foo{X}\oof{Y}.
2151
2152\stopsubsection
2153
2154\startsubsection[title={\prm {lastarguments} and \prm {parametercount}}]
2155
2156\topicindex{argumentsnumberof}
2157
2158There are two state variables that refer to the number of read arguments. An
2159example can show the difference:
2160
2161\startbuffer
2162\tolerant\def\foo[#1]#*[#2]{[\the\lastarguments,\the\parametercount]}
2163
2164\foo[1][2]
2165\foo[1]
2166\foo
2167
2168x: \foo[1][2]
2169x: \foo[1]
2170x: \foo
2171\stopbuffer
2172
2173\typebuffer
2174
2175What you get actually depends on the macro package. When for instance \prm
2176{everypar} has some value that results in a macro being expanded, the numbers
2177reported can refer to the most recent macro because serializing the number can
2178result in entering horizontal mode.
2179
2180\startlines
2181\getbuffer
2182\stoplines
2183
2184The \prm {lastarguments} returns the most recent global state variable as with
2185any \type {\last...} primitives. Because it actually looks at the parameter stack
2186of the currently expanded macro \prm {parametercount} is more reliable but also
2187less efficient.
2188
2189\stopsubsection
2190
2191\startsubsection[title=Overload protection]
2192
2193\topicindex {macrosoverloading}
2194
2195There is an experimental overload protection mechanism that we will test for a
2196while before declaring it stable. The reason for that is that we need to adapt
2197the \CONTEXT\ code base in order to test its usefulness. Protection is achieved
2198via prefixes. Depending on the value of the \prm {overloadmode} variable
2199warnings or errors will be triggered. Examples of usage can be found in some
2200documents that come with \CONTEXT, so here we just stick to the basics.
2201
2202\starttyping
2203\mutable \def\foo{...}
2204\immutable\def\foo{...}
2205\permanent\def\foo{...}
2206\frozen \def\foo{...}
2207\aliased \def\foo{...}
2208\stoptyping
2209
2210A \prm {mutable} macro can always be changed contrary to an \prm {immutable} one.
2211For instance a macro that acts as a variable is normally \prm {mutable}, while a
2212constant can best be immutable. It makes sense to define a public core macro as
2213\prm {permanent}. Primives start out a \prm {permanent} ones but with a primitive
2214property instead.
2215
2216\startbuffer
2217 \let\relaxone \relax 1: \meaningfull\relaxone
2218\aliased \let\relaxtwo \relax 2: \meaningfull\relaxtwo
2219\permanent\let\relaxthree\relax 3: \meaningfull\relaxthree
2220\stopbuffer
2221
2222\typebuffer
2223
2224The \prm {meaningfull} primitive is like \prm {meaning} but report the
2225properties too. The \prm {meaningless} companion reports the body of a macro.
2226Anyway, this typesets:
2227
2228\startlines \tttf \getbuffer \stoplines
2229
2230So, the \prm {aliased} prefix copies the properties. Keep in mind that a macro
2231package can redefine primitives, but \prm {relax} is an unlikely candidate.
2232
2233There is an extra prefix \prm {noaligned} that flags a macro as being valid
2234for \prm {noalign} compatible usage (which means that the body must contain that
2235one. The idea is that we then can do this:
2236
2237\starttyping
2238\permanent\protected\noaligned\def\foo{\noalign{...}}
2239\stoptyping
2240
2241that is: we can have protected macros that dont trigger an error in the parser
2242where there is a look ahead for \prm {noalign} which is why normally protection
2243doesnt work well. So: we have macro flagged as permanent (overload protection),
2244being protected (that is, not expandable by default) and a valid equivalent of
2245the noalign primitive. Of course we can also apply the \prm {global} and \prm
2246{tolerant} prefixes here. The complete repertoire of extra prefixes is:
2247
2248\starttabulate
2249\HL
2250\NC \type {frozen} \NC a macro that has to be redefined in a managed way \NC \NR
2251\NC \type {permanent} \NC a macro that had better not be redefined \NC \NR
2252\NC \type {primitive} \NC a primitive that normally will not be adapted \NC \NR
2253\NC \type {immutable} \NC a macro or quantity that cannot be changed, it is a constant \NC \NR
2254\NC \type {mutable} \NC a macro that can be changed no matter how well protected it is \NC \NR
2255\HL
2256\NC \type {instance} \NC a macro marked as (for instance) be generated by an interface \NC \NR
2257\HL
2258\NC \type {noaligned} \NC the macro becomes acceptable as \type {\noalign} alias \NC \NR
2259\HL
2260\NC \type {overloaded} \NC when permitted the flags will be adapted \NC \NR
2261\NC \type {enforced} \NC all is permitted (but only in zero mode or ini mode) \NC \NR
2262\NC \type {aliased} \NC the macro gets the same flags as the original \NC \NR
2263\HL
2264\NC \type {untraced} \NC the macro gets a different treatment in tracing \NC \NR
2265\HL
2266\stoptabulate
2267
2268The not yet discussed \prm {instance} is just a flag with no special meaning
2269which can be used as classifier. The \prm {frozen} also protects against overload
2270which brings amount of blockers to four.
2271
2272To what extent the engine will complain when a property is changed in a way that
2273violates the flags depends on the parameter \prm {overloadmode}. When this
2274parameter is set to zero no checking takes place. More interesting are values
2275larger than zero. If that is the case, when a control sequence is flagged as
2276mutable, it is always permitted to change. When it is set to immutable one can
2277never change it. The other flags determine the kind of checking done. Currently
2278the following overload values are used:
2279
2280\starttabulate[llccccc]
2281 \NC \NC \BC immutable \BC permanent \BC primitive \BC frozen \BC instance \NC \NR
2282 \NC 1 \NC warning \NC \star \NC \star \NC \star \NC \NC \NC \NR
2283 \NC 2 \NC error \NC \star \NC \star \NC \star \NC \NC \NC \NR
2284 \NC 3 \NC warning \NC \star \NC \star \NC \star \NC \star \NC \NC \NR
2285 \NC 4 \NC error \NC \star \NC \star \NC \star \NC \star \NC \NC \NR
2286 \NC 5 \NC warning \NC \star \NC \star \NC \star \NC \star \NC \star \NC \NR
2287 \NC 6 \NC error \NC \star \NC \star \NC \star \NC \star \NC \star \NC \NR
2288\stoptabulate
2289
2290The even values (except zero) will abort the run. A value of 255 will freeze this
2291parameter. At level five and above the \prm {instance} flag is also checked but
2292no drastic action takes place. We use this to signal to the user that a specific
2293instance is redefined (of course the definition macros can check for that too).
2294
2295The \prm {overloaded} prefix can be used to overload a frozen macro. The \prm
2296{enforced} is more powerful and forces an overload but that prefix is only
2297effective in ini mode or when its embedded in the body of a macro or token list
2298at ini time unless of course at runtime the mode is zero.
2299
2300So far for a short explanation. More details can be found in the \CONTEXT\
2301documentation where we can discuss it in a more relevant perspective. It must be
2302noted that this feature only makes sense a controlled situation, that is: user
2303modules or macros of unpredictable origin will probably suffer from warnings and
2304errors when de mode is set to non zero. In \CONTEXT\ were okay unless of course
2305users redefine instances but there a warning or error is kind of welcome.
2306
2307There is an extra prefix \prm {untraced} that will suppress the meaning when
2308tracing so that the macro looks more like a primitive. It is still somewhat
2309experimental so what gets displayed might change.
2310
2311The \prm {letfrozen}, \prm {unletfrozen}, \prm {letprotected} and \prm
2312{unletprotected} primitives do as their names advertise. Of course the \prm
2313{overloadmode} must be set so that it is permitted.
2314
2315\stopsubsection
2316
2317\startsubsection[title={Swapping meaning}]
2318
2319\topicindex {macrosswapping}
2320
2321The \prm {swapcsvalues} will swap the values of two control sequences of the same
2322type. This is a somewhat tricky features because it can interfere with grouping.
2323
2324\startbuffer
2325\scratchcounterone 1 \scratchcountertwo 2
2326(\the\scratchcounterone,\the\scratchcountertwo)
2327\swapcsvalues \scratchcounterone \scratchcountertwo
2328(\the\scratchcounterone,\the\scratchcountertwo)
2329\swapcsvalues \scratchcounterone \scratchcountertwo
2330(\the\scratchcounterone,\the\scratchcountertwo)
2331
2332\scratchcounterone 3 \scratchcountertwo 4
2333(\the\scratchcounterone,\the\scratchcountertwo)
2334\bgroup
2335\swapcsvalues \scratchcounterone \scratchcountertwo
2336(\the\scratchcounterone,\the\scratchcountertwo)
2337\egroup
2338(\the\scratchcounterone,\the\scratchcountertwo)
2339\stopbuffer
2340
2341\typebuffer
2342
2343We get similar results:
2344
2345\startlines
2346\getbuffer
2347\stoplines
2348
2349\stopsubsection
2350
2351\stopsection
2352
2353\startsection[title=Quantities]
2354
2355\startsubsection[title={Constants with \prm{integerdef}, \prm {dimensiondef},
2356\prm {gluespecdef} and \prm {mugluespecdef}}]
2357
2358It is rather common to store constant values in a register or character
2359definition.
2360
2361\starttyping
2362\newcount\MyConstantA \MyConstantA 123
2363\newdimen\MyConstantB \MyConstantB 123pt
2364\chardef \MyConstantC \MyConstantC 123
2365\stoptyping
2366
2367But in \LUAMETATEX\ we also can do this:
2368
2369\starttyping
2370\integerdef \MyConstantI 456
2371\dimensiondef \MyConstantD 456pt
2372\gluespecdef \MyConstantG 987pt minus 654pt plus 321pt
2373\mugluespecdef \MyConstantG 3mu plus 2mu minus 1mu
2374\stoptyping
2375
2376These two are stored as efficient as a register but dont occupy a register slot.
2377They can be set as above, need \prm {the} for serializations and are seen as
2378valid number or dimension when needed. They do behave like registers so one can
2379use for instance \prm {advance} and assign values but keep in mind that an alias
2380(made by for instance \prm {let} clones the value and that clone will not follow
2381a change in the original. For that, registers can be used because there we use an
2382indirect reference.
2383
2384Experiments with constant strings made the engine source more complex than I
2385wanted so that features was rejected. Of course we can use the prefixes mentioned
2386in a previous section.
2387
2388\stopsubsection
2389
2390\startsubsection[title={Getting internal indices with \prm {indexofcharacter} and \prm {indexofregister}}]
2391
2392When you have defined a register with one of the \tex {...def} primitives but for
2393some reasons needs to know the register index you can query that:
2394
2395\startbuffer
2396\the\indexofregister \scratchcounterone,
2397\the\indexofregister \scratchcountertwo,
2398\the\indexofregister \scratchwidth,
2399\the\indexofregister \scratchheight,
2400\the\indexofregister \scratchdepth,
2401\the\indexofregister \scratchbox
2402\stopbuffer
2403
2404\typebuffer
2405
2406We lie a little here because in \CONTEXT\ the box index \tex {scratchbox} is
2407actually defined as: \normalexpanded {\typ {\meaningasis \scratchbox}} but it
2408still is a number so it fits in.
2409
2410\getbuffer
2411
2412A similar primitive gives us the (normally \UNICODE) value of a character:
2413
2414\startbuffer
2415\chardef\MyCharA=65
2416\the\indexofcharacter A
2417\the\indexofcharacter \MyCharA
2418\stopbuffer
2419
2420\typebuffer
2421
2422The result is equivalent to \type {\number A} but avoids the back quote:
2423\inlinebuffer.
2424
2425\stopsubsection
2426
2427\startsubsection[title={Serialization with \prm {todimension}, \prm {toscaled}, \prm {tohexadecimal} and \prm {tointeger}}]
2428
2429These serializers take a verbose or symbolic quantity:
2430
2431\starttyping
2432\todimension 10pt \todimension \scratchdimen
2433\toscaled 10pt \toscaled \scratchdimen
2434\tointeger 10 \tointeger \scratchcounter
2435\tohexadecimal 10 \tohexadecimal \scratchcounter
2436\stoptyping
2437
2438This is particularly handy in cases where you dont know what you deal with, for instance
2439when a value is stored in a macro. Using \type {\the} could fail there while:
2440
2441\starttyping
2442\the\dimexpr10pt\relax
2443\stoptyping
2444
2445is often overkill and gives more noise in a trace.
2446
2447\stopsubsection
2448
2449\startsubsection[title={Serialization with \prm {thewithoutunit}, \prm {tosparsedimension} and \prm {tosparsescaled}}]
2450
2451\topicindex {units}
2452
2453By default \TEX\ lets \type {1pt} come out as \type {1.0pt} which is why we also have
2454two sparse variants:
2455
2456\startbuffer
2457\todimension 10pt\quad\tosparsedimension 10pt
2458\todimension 1.2pt\quad\tosparsedimension 1.2pt
2459\toscaled 10pt\quad\tosparsescaled 10pt
2460\toscaled 1.2pt\quad\tosparsescaled 1.2pt
2461\stopbuffer
2462
2463\typebuffer
2464
2465This time trailing zeros (and a trailing period) will be dropped:
2466
2467\startlines \getbuffer \stoplines
2468
2469The \prm {thewithoutunit} primitive is like \prm {the} on a dimension but it
2470omits the unit.
2471
2472\stopsubsection
2473
2474\startsubsection[title={Units}]
2475
2476The familiar \TEX\ units like \type {pt} and \type {cm} are supported but since
2477the 2021 \CONTEXT\ meeting we also support the Knuthian Potrzebie, cf.\ \typ
2478{en.wikipedia.orgwikiPotrzebie}. The two character acronym is \type {dk}. One
2479\type {dk} is 6.43985pt. This unit is particularly suited for offsets in framed
2480examples.
2481
2482In 2023 we added the Edith (\type {es}) and Tove (\type {ts}) as metric
2483replacements for the inch (\type {in}). As with the \type {dk} more background
2484information can be found in documents that come with \CONTEXT\ and user group
2485journals. The \type {eu} unit starts out as one \type {es} but can be scaled with
2486\prm {eufactor}.
2487
2488\startbuffer
2489\localcontrolledloop 5 55 5 {
2490 \eufactor=\currentloopiterator
2491 \dontleavehmode\strut
2492 \vrule height .1es depth .25ts width 1dk\relax\quad
2493 \vrule height .1es depth .25ts width 1eu\relax\quad
2494 \the\currentloopiterator
2495 \par
2496}
2497\stopbuffer
2498
2499\typebuffer
2500
2501This example code shows all four new units. Watch how \prm {eufactor} is clipped
2502to a value in the range $150$. The default factor of $10$ makes the European
2503Unit equivalent to ten Toves or one Edith.
2504
2505\startpacked
2506\startcolor[darkgray]
2507\getbuffer
2508\stopcolor
2509\stoppacked
2510
2511\stopsubsection
2512
2513\stopsection
2514
2515\startsection[title=Registers]
2516
2517
2518\startsubsection[title={32 bit floats}]
2519
2520The engine has native float registers which means that we have a similar set of
2521primitives as for dimensions and integers: \prm {float}, \prm {floatdef}, \prm
2522{floatexpr} and \prm {iffloat}. In the context if an integer a rounded value is
2523used, in the context of a dimensions the floating point number is interpreted as
2524points. Internally floats are stored as so called posits, which gives more accuracy
2525for smaller values.
2526
2527\stopsubsection
2528
2529\startsubsection[title={32 bit posits}]
2530
2531{\em This is a playground. It will stay but might evolve.}
2532
2533\stopsubsection
2534
2535\startsubsection[title={Extra features}]
2536
2537There are \prm {ifabsnum}, \prm {ifabsdim} and \prm {ifabsfloat} that compare
2538absolute values of quantities. The primitives \prm{ifzeronum}, \prm{ifzerodim},
2539\prm{ifzerofloat} do a fast test for zero. The \prm {ifintervalnum}, \prm
2540{ifintervaldim} and \prm {ifintervalfloat} primitives take a delta and two values
2541and check if these values overlap within the ranges dedicated by the delta.
2542
2543\stopsubsection
2544
2545\stopsection
2546
2547\startsection[title=Expressions]
2548
2549\startsubsection[title={Rounding and scaling}]
2550
2551\topicindex {expressionstraditional}
2552
2553The \type {*expr} parsers now accept \type {:} as operator for integer division
2554(the \type {} operators does rounding. This can be used for division compatible
2555with \type {\divide}. Im still wondering if adding a couple of bit operators
2556makes sense (for integers).
2557
2558The \prm{numericscale} parser is kind of special (and might evolve). For now it
2559converts a following number in a scale value as often used in \TEX, where 1000
2560means scaling by1.0. The trick is in the presence of a digit (or comma): 1.234
2561becomes 1234 but 1234 stays 1234 and from this you can deduce that 12.34 becomes
2562123400. Internally \TEX\ calculates with integers, but this permits the macro
2563package to provide an efficient mix.
2564
2565\stopsubsection
2566
2567\startsubsection[title={Enhanced expressions}]
2568
2569\topicindex {expressionsenhanced}
2570
2571The \ETEX\ expression primitives are handy but have some limitations. Although
2572the parsers have been rewritten in \LUAMETATEX\ and somewhat more efficient the
2573only extension we have is support for an integer division with \type {:}. After
2574experimenting for a while and pondering how to make \prm {dimexpr} and \prm
2575{numexpr} more powerful I decided to come up with alternatives in order not to
2576introduce incompatibilities.
2577
2578The \prm {numexpression} and \prm {dimexpression} primitives are equivalent but
2579offer more. The first one operates in the integer domain and the second one
2580assumes scaled values. Often the second one can act like the first when
2581serialized with \prm {number} in front. This is because when \TEX\ sees a
2582symbolic reference to an integer or dimension it can treat them as it likes.
2583
2584The set of operators that we have to support is the following. Most have
2585alternatives so that we can get around catcode issues.
2586
2587\starttabulate[cTcT]
2588\DB action \BC symbol \BC keyword \NC \NR
2589\TB
2590\NC add \NC \NC \NC \NR
2591\NC subtract \NC \NC \NC \NR
2592\NC multiply \NC * \NC \NC \NR
2593\NC divide \NC : \NC \NC \NR
2594\NC mod \NC \letterpercent \NC mod \NC \NR
2595\NC band \NC \NC band \NC \NR
2596\NC bxor \NC \NC bxor \NC \NR
2597\NC bor \NC \letterbar \space v \NC bor \NC \NR
2598\NC and \NC \NC and \NC \NR
2599\NC or \NC \letterbar\letterbar \NC or \NC \NR
2600\NC setbit \NC <undecided> \NC bset \NC \NR
2601\NC resetbit \NC <undecided> \NC breset \NC \NR
2602\NC left \NC << \NC \NC \NR
2603\NC right \NC >> \NC \NC \NR
2604\NC less \NC < \NC \NC \NR
2605\NC lessequal \NC <= \NC \NC \NR
2606\NC equal \NC = == \NC \NC \NR
2607\NC moreequal \NC >= \NC \NC \NR
2608\NC more \NC > \NC \NC \NR
2609\NC unequal \NC <> != \lettertilde = \NC \NC \NR
2610\NC not \NC ! \lettertilde \NC not \NC \NR
2611\LL
2612\stoptabulate
2613
2614Here are some things that \prm {numexpr} is not suitable for:
2615
2616\starttyping
2617\scratchcounter = \numexpression
2618 "00000 bor "00001 bor "00020 bor "00400 bor "08000 bor "F0000
2619\relax
2620
2621\ifcase \numexpression
2622 (\scratchcounterone > 5) (\scratchcountertwo > 5)
2623\relax yes\else nop\fi
2624\stoptyping
2625
2626You can get an idea what the engines sees by setting \prm {tracingexpressions}
2627to a value larger than zero. It shows the expression in rpn form.
2628
2629\starttyping
2630\dimexpression 4pt * 2 6pt \relax
2631\dimexpression 2 * 4pt 6pt \relax
2632\dimexpression 4pt * 2.5 6pt \relax
2633\dimexpression 2.5 * 4pt 6pt \relax
2634\numexpression 2 * 4 6 \relax
2635\numexpression (1 2) * (3 4) \relax
2636\stoptyping
2637
2638The \prm {relax} is mandate simply because there are keywords involved so the
2639parser needs to know where to stop scanning. It made no sense to be more clever
2640and introduce fuzziness (so there is no room for exposing indepth \TEX\
2641insight and expertise here). In case you wonder: the difference in performance
2642between the \ETEX\ expression mechanism and the more extended variant will
2643normally not be noticed, probably because they both use a different approach and
2644because the \ETEX\ variant also has been optimized. \footnote {I might add some
2645features in the future.}
2646
2647The iftest shown before can be done using the new primitives \prm
2648{ifdimexpression} and \prm {ifnumexpression} which are boolean tests with zero
2649being \type {false}.
2650
2651\stopsubsection
2652
2653\startsubsection[title={Calculations with \prm {advanceby}, \prm {multiplyby} and
2654\prm {divideby}.}]
2655
2656The \prm {advance}, \prm {multiply} and \prm {divide} primitives accept an
2657optional keyword \type {by}. In \CONTEXT\ we never use that feature and as a
2658consequence the scanner has to push back a scanned token after checking for the
2659\type {b} or \type {B}. These three new primitives avoid that and therefore
2660perform better, but that is (as usual with such enhancements) only noticeable in
2661demanding scenarios.
2662
2663The original three plus these new ones also operate on the \quote {constant}
2664integers, dimensions etc.
2665
2666\stopsubsection
2667
2668\stopsection
2669
2670\startsection[title=Loops]
2671
2672\topicindex {loops}
2673
2674There is actually not that much to tell about the three loop primitives \prm
2675{expandedloop}, \prm {unexpandedloop} and \prm {localcontrolledloop}. They are
2676used like:
2677
2678\startbuffer
2679\unexpandedloop 1 10 1 {
2680 [!]
2681}
2682\stopbuffer
2683
2684\typebuffer
2685
2686This will give 10 snippets.
2687
2688\getbuffer
2689
2690So what will the next give?
2691
2692\startbuffer
2693\edef\TestA{\unexpandedloop 1 10 1 {!}}\meaning\TestA
2694\edef\TestB{\expandedloop 1 10 1 {!}}\meaning\TestB
2695\stopbuffer
2696
2697\typebuffer
2698
2699We see no difference in results between the two loops:
2700
2701\startlines \tt
2702\getbuffer
2703\stoplines
2704
2705But the the next variant shows that they do:
2706
2707\startbuffer
2708\edef\TestA{\unexpandedloop 1 10 1 {\the\currentloopiterator}}\meaning\TestA
2709\edef\TestB{\expandedloop 1 10 1 {\the\currentloopiterator}}\meaning\TestB
2710\stopbuffer
2711
2712\typebuffer
2713
2714The unexpanded variants sort of delays:
2715
2716\startlines \tt
2717\getbuffer
2718\stoplines
2719
2720You can nest loops and query the nesting level:
2721
2722\startbuffer
2723\expandedloop 1 10 1 {
2724 \ifodd\currentloopiterator\else
2725 [\expandedloop 1 \currentloopiterator 1 {
2726 \the\currentloopnesting
2727 }]
2728 \fi
2729}
2730\stopbuffer
2731
2732\typebuffer
2733
2734Here we use the two numeric state primitives \prm {currentloopiterator} and \prm
2735{currentloopnesting}. This results in:
2736
2737\getbuffer
2738
2739The \prm {quitloop} primitive makes it possible to prematurely exit a loop (at
2740the next step), although of course in the next case one can just adapt the final
2741iterator value instead. Here we step by 2:
2742
2743\startbuffer
2744\expandedloop 1 20 2 {
2745 \ifnum\currentloopiterator>10
2746 \quitloop
2747 \else
2748 [!]
2749 \fi
2750}
2751\stopbuffer
2752
2753\typebuffer
2754
2755This results in:
2756
2757\getbuffer
2758
2759The \prm {lastloopiterator} primitive keeps the last iterator value and is a global
2760one as all \type {\last...} primitives. The loops also work with negative values.
2761
2762A special case is \prm {localcontrolledloop} which fits into the repertoire of
2763local control primitives. In that case the loop body gets expanded in a nested
2764main loop which can come in handy in tricky cases where full expansion is mixed
2765with for instance assignments but of course users should then be aware of
2766outoforder side effects when you push back something in the input. Consider
2767it a playground.
2768
2769\stopsection
2770
2771\stopchapter
2772
2773\stopcomponent
2774
2775
2776
2777
2778
2779
2780
2781
2782
2783
2784
2785
2786
2787
2788
2789
2790
2791
2792
2793
2794
2795 |