Source Browser ?

1% language=us
2
3\startcomponent onandon-expansion
4
5\environment onandon-environment
6
7\startchapter[title={More (new) expansion trickery}]
8
9Contrary to what one might expect when looking at macro definitions, \TEX\ is
10pretty efficient. Occasionally I wonder if some extra built in functionality
11could help me write better code but when you program with a bit care there is
12often not much to gain in terms of tokens and performance. \footnote {The long
13trip to the yearly Bacho\TeX\ meeting is always a good opportunity to ponder
14\TEX\ and its features. The new functionality discussed here is a side effect of
15the most recent trip.} Also, some possible extensions probably only would be
16applied a few times which makes them low priority. When you look at the
17extensions brought by \ETEX\ the number is not that large, and \LUATEX\ only
18added a few that deal with the language, for instance \tex {expanded} which is
19like an \tex {edef} without the defining a macro and acts on a token list wrapped
20in (normally) curly braces. Just as reference we mention some of the expansion
21related helpers.
22
23\starttabulate[|l|l|p|]
24\BC command \BC argument \BC
25    comment
26\NC \NR
27\HL
28\NC \tex {expandafter} \NC \type {token} \NC
29    The token after the next token gets expanded (one level only). In tricky
30    \TEX\ code you can often see multiple such commands in sequence which makes a
31    nice puzzle.
32\NC \NR
33\NC \tex {noexpand} \NC \type {token} \NC
34    The token after this command is not expanded in the context of expansion.
35\NC \NR
36\NC \tex {expanded} \NC \type {{tokens}} \NC
37    The given token list is expanded. This command showed up early in \LUATEX\
38    development and was taken from \ETEX\ follow|-|ups. I have mails from 2011
39    mentioning its presence in \PDFTEX\ 1.50 (which was targeted in 2008) but
40    somehow it never ended up in a production version at that time (and we're
41    still not at that version). In \CONTEXT\ we already had a command with that
42    name so there we use \tex {normalexpanded}. Users normally can just use the
43    \CONTEXT\ variant of \type {\expanded}.
44\NC \NR
45\NC \tex {unexpanded} \NC \type {{tokens}} \NC
46    The given token list is hidden from expansion. Again, in \CONTEXT\ we already
47    had a command serving as prefix for definitions so instead we use \tex
48    {normalunexpanded}. In the core of \CONTEXT\ this new \ETEX\ command is hardly
49    used.
50\NC \NR
51\NC \tex {detokenize} \NC \type {{tokens}} \NC
52    The given tokenlist becomes (basically) verbatim \TEX\ code. We had something
53    like that in \CONTEXT\ but have no nameclash. It is used in a few places. It's
54    also an \ETEX\ command.
55\NC \NR
56\NC \tex {scantokens} \NC \type {{tokens}} \NC
57    This primitive interprets its argument as a pseudo file. We don't really use it.
58\NC \NR %
59\NC \tex {scantextokens} \NC \type {{tokens}} \NC
60    This \LUATEX\ primitive does the same but has no end|-|of|-|file side
61    effects. This one is also not really used in \CONTEXT.
62\NC \NR
63\NC \tex {protected} \NC \type {\.def} \NC
64    The definition following this prefix, introduced in \ETEX, is unexpandable in
65    the context of expansion. We already used such a command in \CONTEXT\ but
66    with a completely different meaning so use \tex {normalprotected} as prefix
67    or \tex {unexpanded} which is an alias.
68\NC \NR
69\stoptabulate
70
71Here I will present two other extensions in \LUATEX\ that can come in handy, and
72they are there simply because their effect can hardly be realized otherwise
73(never say never in \TEX). One has to do with immediately applying a definition,
74the other with user defined conditions. The first one relates directly to
75expansion, the second one concerns conditions and relates more to parsing
76branches which on purpose avoids expansion.
77
78{\em In the meantime \LUAMETATEX\ has a slightly different implementation which
79goes under the umbrella \quote {local control}. We show both ways here. The
80example where two token lists are compared can be done easier with \type
81{\iftok}.}
82
83For the first one I use some silly examples. I must admit that although I can
84envision useful application, I really need to go over the large amount of
85\CONTEXT\ source code to really find a place where it is making things better.
86Take the following definitions:
87
88\startbuffer
89\newcount\NumberOfCalls
90
91\def\TestMe{\advance\NumberOfCalls1 }
92
93\edef\Tested{\TestMe foo:\the\NumberOfCalls}
94\edef\Tested{\TestMe foo:\the\NumberOfCalls}
95\edef\Tested{\TestMe foo:\the\NumberOfCalls}
96
97\meaning\Tested
98\stopbuffer
99
100\typebuffer
101
102The result is a macro \tex {Tested} that not only has the unexpanded incrementing
103code in its body but also hasn't done any advancing:
104
105\getbuffer
106
107Of course when you're typesetting something, this kind of expansion normally is
108not needed. Instead of the above definition we can define \tex {TestMe} in a way
109that expands the assignment immediately. You need of course to be aware of
110preventing look ahead interference by using a space or \tex {relax} (often an
111expression works better as it doesn't leave an \tex {relax}).
112
113\startbuffer
114% luatex
115
116\def\TestMe{\immediateassignment\advance\NumberOfCalls1 }
117
118% luametatex
119
120\def\TestMe{\localcontrolled{\advance\NumberOfCalls1 }}
121
122\edef\Tested{\TestMe bar:\the\NumberOfCalls}
123\edef\Tested{\TestMe bar:\the\NumberOfCalls}
124\edef\Tested{\TestMe bar:\the\NumberOfCalls}
125
126\meaning\Tested
127\stopbuffer
128
129\typebuffer
130
131This time the counter gets updated and we don't see interference in the resulting
132\tex {Tested} macro:
133
134\getbuffer
135
136Here is a somewhat silly example of an expanded comparison of two \quote
137{strings}:
138
139\startbuffer
140% luatex
141
142\def\ExpandedDoifElse#1#2#3#4%
143  {\immediateassignment\edef\tempa{#1}%
144   \immediateassignment\edef\tempb{#2}%
145   \ifx\tempa\tempb
146     \immediateassignment\def\next{#3}%
147   \else
148     \immediateassignment\def\next{#4}%
149   \fi
150   \next}
151
152% luametatex
153
154\def\ExpandedDoifElse#1#2#3#4%
155  {\localcontrolled{\edef\tempa{#1}}%
156   \localcontrolled{\edef\tempb{#2}}%
157   \ifx\tempa\tempb
158     \localcontrolled{\def\next{#3}}%
159   \else
160     \localcontrolled{\def\next{#4}}%
161   \fi
162   \next}
163
164\edef\Tested
165  {(\ExpandedDoifElse{abc}{def}{yes}{nop}/%
166    \ExpandedDoifElse{abc}{abc}{yes}{nop})}
167
168\meaning\Tested
169\stopbuffer
170
171\typebuffer
172
173I don't remember many cases where I needed such an expanded comparison. We have a
174variant in \CONTEXT\ that uses \LUA\ but that one is not really used in the core.
175Anyway, the above code gives:
176
177\getbuffer
178
179You can do the same assignments as in preambles of \tex {halign} and after \tex
180{accent} which means that assignments to box registers are blocked (boxing
181involves grouping and delayed assignments and so). The error you will get when
182you use a non||assignment command refers to a prefix, because internally such
183commands are called prefixed commands. Leading spaces and \tex {relax} are
184ignored.
185
186In addition to this one|-|time immediate assignment a pseudo token list variant
187is provided, so the above could be rewritten to:
188
189\starttyping
190% luatex
191
192\def\ExpandedDoifElse#1#2#3#4%
193  {\immediateassigned {
194     \edef\tempa{#1}
195     \edef\tempb{#2}
196   }%
197   \ifx\tempa\tempb
198     \immediateassignment\def\next{#3}%
199   \else
200     \immediateassignment\def\next{#4}%
201   \fi
202   \next}
203
204% luametatex
205
206\def\ExpandedDoifElse#1#2#3#4%
207  {\beginlocalcontrol
208     \edef\tempa{#1}
209     \edef\tempb{#2}
210   \endlocalcontrol
211   \ifx\tempa\tempb
212     \localcontrolled{\def\next{#3}}%
213   \else
214     \localcontrolled{\def\next{#4}}%
215   \fi
216   \next}
217\stoptyping
218
219While \tex {expanded} first builds a token lists that then gets used, the \tex
220{immediateassigned} primitive just walls over the list delimited by curly braces.
221
222A next extension concerns conditions. If you have done a bit of extensive \TEX\
223programming you know that nested conditions need to be properly constructed in
224for instance macro bodies. This is because (for good reason) \TEX\ goes into a
225fast scanning mode when there is a match and it has to skip the \tex {else} upto
226\tex {fi} branch. In order to do that properly a nested \tex {if} in there needs
227to have a matching \tex {fi}.
228
229In practice this is no real problem and careful coding will never give a problem
230here: you can either hide nested code in a macro or somehow jump over nested
231conditions if really needed. Actually you only need to care when you pickup a
232token inside the branch because likely you don't want to pick up for instance a
233\tex {fi} but something that comes after it. Say that we have a sane conditional
234setup like this:
235
236\starttyping
237\newif\iffoo \foofalse
238\newif\ifbar \bartrue
239
240\ifoo
241  \ifbar \else \fi
242\else
243  \ifbar \else \fi
244\fi
245\stoptyping
246
247Here the \tex {iffoo} and \tex {ifbar} need to be equivalent to \tex {iftrue} or
248\tex {iffalse} in order to succeed well and that is what for instance \tex
249{footrue} and \tex {foofalse} will do: change the meaning of \tex {iffoo}.
250
251But imagine that you want something more complex. You want for instance to let
252\tex {ifbar} do some calculations. In that case you want it to behave a bit like
253what a so called \type {vardef} in \METAPOST\ does: the end result is what
254matters. Now, because \TEX\ macros often are a complex mix of expandable and
255non|-|expandable this is not that trivial. One solution is a dedicated definer,
256say \tex {cdef} for defining a macro with conditional properties. I actually
257implemented such a definer a few years ago but left it so long in a folder with
258ideas that I only found it back after I had come up with another solution. It was
259probably proof that it was not that good an idea.
260
261The solution implemented in \LUATEX\ is just a special case of a test: \tex
262{ifcondition}. When looking at the next example, keep in mind that from the
263perspective of \TEX's scanner it only needs to know if something is a token that
264does some test and has a matching \tex {fi}. For that purpose you can consider
265\tex {ifcondition} to be \tex {iftrue}. When \TEX\ actually wants to do a test,
266which is the case in the true branch, then it will simply ignore this \tex
267{ifcondition} primitive and expands what comes after it (which is \TEX's natural
268behaviour). Effectively \tex {ifcondition} has no meaning except from when it has
269to be skipped, in which case it's a token flagged as \tex {if} kind of command.
270
271\starttyping
272\unexpanded\def\something#1#2%
273  {\edef\tempa{#1}%
274   \edef\tempb{#2}
275   \ifx\tempa\tempb}
276
277\ifcondition\something{a}{b}%
278    \ifcondition\something{a}{a}%
279        true 1
280    \else
281        false 1
282    \fi
283\else
284    \ifcondition\something{a}{a}%
285        true 2
286    \else
287        false 2
288    \fi
289\fi
290\stoptyping
291
292Wrapped in a macro you can actually make this fully expandable when you use the
293previously mentioned immediate assignment. Here is another example:
294
295\starttyping
296\unexpanded\def\onoddpage
297  {\ifodd\count0 }
298
299\ifcondition\onoddpage odd \else even \fi page
300\stoptyping
301
302The previously defined comparison macro can now be rewritten as:
303
304\starttyping
305% luatex
306
307\def\EqualTokens#1#2%
308  {\immediateassignment\edef\tempa{#1}%
309   \immediateassignment\edef\tempb{#2}%
310   \ifx\tempa\tempb}
311
312\def\ExpandedDoifElse#1#2#3#4%
313  {\ifcondition\EqualTokens{#1}{#2}%
314     \immediateassignment\def\next{#3}%
315   \else
316     \immediateassignment\def\next{#4}%
317   \fi
318   \next}
319
320% luametatex
321
322\def\EqualTokens#1#2%
323  {\localcontrolled{\edef\tempa{#1}}%
324   \localcontrolled{\edef\tempb{#2}}%
325   \ifx\tempa\tempb}
326
327\def\ExpandedDoifElse#1#2#3#4%
328  {\ifcondition\EqualTokens{#1}{#2}%
329     \localcontrolled{\def\next{#3}}%
330   \else
331     \localcontrolled{\def\next{#4}}%
332   \fi
333   \next}
334\stoptyping
335
336When used this way it will of course also work without the \tex {ifcondition} but
337when used nested it can be like this. This last example also demonstrates that
338this feature probably only makes sense in more complicated cases where more work
339is done in the \tex {onoddpage} or \tex {equaltokens} macro. And again, I am not
340sure if for instance in \CONTEXT\ I have a real use for it because there are only
341a few cases where nesting like this could benefit. I did some tests with a low
342level macro where it made the code look nicer. It was actually a bit faster but
343most core macros are not called that often. Although the overhead of this feature
344can be neglected, performance should not be the reason for using it: in \CONTEXT\
345for instance one can often only measure such possible speed|-|ups on macros that
346are called tens or hundreds of thousands of times and that seldom happens in a
347real run end even then a change from say 0.827 seconds to 0.815 seconds for 10K
348calls of a complex case is just noise as the opposite can also happen.
349
350Although not strictly necessary these extensions might make some code look better
351so that is why they officially will be available in the 1.09 release of \LUATEX\
352in fall 2018. It might eventually inspire me to go over some code and see where I
353can improve the look and feel.
354
355The last few years I have implemented some more ideas as local experiments, for
356instance \tex {futurelet} variant or a simple (one level) \tex {expand}, but in
357the end rejected them because there is no real benefit in them (no better looking
358code, no gain in performance, hard to document, possible side effects, etc.), so
359it is very unlikely that we will have more extensions like this. After all, we
360could do more than 40 years without them. Although \unknown\ who knows what we
361will provide in \LUATEX\ version~2.
362
363\stopchapter
364
365\stopcomponent
366