Source Browser ?

1% language=us runpath=texruns:manuals/luametatex
2
3\environment luametatex-style
4
5\startdocument[title=principles]
6
7\startsection[title={Introduction}]
8
9This is a bit odd manual but needed anyway. In the process of adding features to
10\LUAMETATEX\ and adapting \CONTEXT\ accordingly some decisions were made. On the
11one hand generic flexibility is a criterion used when the extending engine, on
12the other hand practical usability in \CONTEXT\ is used to decide where to draw
13a line or make some choices. It makes no sense to complicate the already complex
14engine even more, or cripple \CONTEXT\ when cleaner (low level) solutions are
15possible.
16
17Here I will collect some of the considerations and mention the choices made.
18These are mostly mine but some result from discussions and experiments. This
19overview is not complete, new primitives are discussed elsewhere and the
20\CONTEXT\ low level manuals explain how to use these. Consider this to be a
21teaser.
22
23{\em This summary is work in progress.}
24
25\stopsection
26
27\startsection[title=Text fonts]
28
29Plenty has been written about fonts in \TEX, so here I will only mention a few
30aspects. Traditionally the \TEX\ engines works with copies of fonts at given
31sizes. For large fonts that is kind of inefficient. This is why in \LUAMETATEX\
32we can scale a font on|-|the|-|fly using \typ {\glyphscale}, \typ {\glyphxscale}
33and \typ {\glyphyscale}. This feature is also used to implement a more efficient
34(although not 100\percent\ metric compatible) compact font mode. It works okay in
35text as well as math although it comes at a price: many more calculations are
36needed at the engine end.
37
38One way to get an expanded, squeezed, emboldened or slanted font in \CONTEXT\ is
39to use the effects mechanism. It is quite flexible but again comes at a price
40because the backend has to do more work which is measurable, especially because
41effects can apply to the font or individual glyphs. However, the advantages
42out|-|weight the disadvantages. At the cost of yet a bit more performance a more
43native variant is also available using \typ {\glyphslant} and \typ
44{\glyphweight}.
45
46\definefontfeature[extendtest] [default][extend=1.25]
47\definefontfeature[squeezetest][default][squeeze=1.25]
48\definefontfeature[slanttest]  [default][slant=0.5]
49\definefontfeature[weighttesta][default][weight=0.2] % 0.4pt is bad for the g in lm
50\definefontfeature[weighttestb][default][effect={weight=0.2,auto=yes}]
51
52\starttexdefinition demo #1
53    \startcombination[nx=6,ny=1,distance=12.5mm]
54      {\scale[scale=8000]{\definedfont[Serif*default     @ 10bp]\ruledhbox{\color[maincolor]{#1}}}} {}
55      {\scale[scale=8000]{\definedfont[Serif*extendtest  @ 10bp]\ruledhbox{\color[maincolor]{#1}}}} {}
56      {\scale[scale=8000]{\definedfont[Serif*squeezetest @ 10bp]\ruledhbox{\color[maincolor]{#1}}}} {}
57      {\scale[scale=8000]{\definedfont[Serif*slanttest   @ 10bp]\ruledhbox{\color[maincolor]{#1}}}} {}
58      {\scale[scale=8000]{\definedfont[Serif*weighttesta @ 10bp]\ruledhbox{\color[maincolor]{#1}}}} {}
59      {\scale[scale=8000]{\definedfont[Serif*weighttestb @ 10bp]\ruledhbox{\color[maincolor]{#1}}}} {}
60    \stopcombination
61\stoptexdefinition
62
63\startlinecorrection \demo{a} \stoplinecorrection
64\startlinecorrection \demo{x} \stoplinecorrection
65\startlinecorrection \demo{p} \stoplinecorrection
66%startlinecorrection \demo{(} \stoplinecorrection
67\startlinecorrection \demo{g} \stoplinecorrection
68\startlinecorrection \demo{-} \stoplinecorrection
69
70Extending, as seen in the second renderings, scales the shapes horizontally,
71while squeezing, in the third renderings, does it in the vertical direction. In
72both cases the dimensions have to be adapted. This is not the case when we slant.
73The last two samples in a row have an increased weight, and these are the more
74tricky cases because here one can argue how to scale and reposition a shape. When
75a shape is above the baseline we increase the height, and when it goes below we
76increase the depth. The engine is capable to increase the width, height and depth
77and shift the shape a little. It only makes sense to adapt the height and depth
78when they are non|-|zero. It will never be perfect, but this feature is not
79perfect anyway.
80
81The way fonts are set up in a \TEX\ macro package often originates in the past,
82if only because it came with fonts. The Computer Modern fonts are among the few
83that have multiple design sizes. However, the collection is pretty much based on
84a ten point design. For math there are seven and five point variants for the
85script sizes, for footnotes an eight point makes sense and section heads can use
86the larger twelve point plus the few larger sizes. Setting up a twelve point
87body font environment, as we have in \CONTEXT, is quite doable with the fonts but
88for an eleven point body font more compromised have to be made.
89
90One can wonder why in \CONTEXT\ the ten point math setup of 10/7/5 became 12/9/7
91instead if 12/8.4/6 and the reason is just that when there were still bitmap
92fonts one didn't want too many (intermediate) sizes. Anyway, we're sort of stuck
93with this default setup now, but nothing prevents users to redefine a body font
94environment.
95
96Another speciality of \TEX\ (fonts) is that they have italic correction, something
97that lacks in \OPENTYPE\ fonts (apart from math btu there it serves a different
98purpose). We can however emulate it, and in \CONTEXT\ that is an option. Given that
99we have to make choices it is clear that the engine can only be supportive here,
100especially when we use the \typ {\glyphslant} method.
101
102A curious case is the following: in Computer Modern we find italic correction in
103the upright fonts, for instance between an \quote {f} and \quote {h}. Dealing with
104this automatically is impossible because italic correction is not to be applied
105between glyph runs of the same font.
106
107% quite some italics with not angle and uprights with angles
108% why not proper boundingbox (right side bearing) with (now also small) width
109% instead of unpredictable either or not urx
110
111\stopsection
112
113\startsection[title=Math fonts]
114
115Support for math in an \UNICODE\ aware engine is also driven by the repertoire of
116characters and their organization in \UNICODE, as well as by \OPENTYPE\ math as
117cooked up by \MICROSOFT\ with a bit of input from \TEX\ folk.
118
119The engine is agnostic when it comes to \UNICODE: there are no character codes
120interpreted in special ways. There are math alphabets but these are not special:
121in a traditional eight bit engine we have families to deal with them, in a
122\UNICODE\ aware engine there are several solutions. The most important character
123property that has some consequence is the math class but for dealing with that
124we're on our own anyway. Everything \UNICODE\ related is up to \CONTEXT\ to deal
125with, and it is the macro package that drives the engine, using the constructs
126that are available, like atoms with specific classes, fractions, accents,
127delimiters, fences, radicals, operators etc.
128
129When it comes to fonts it is more complex. The \OPENTYPE\ math standard is driven
130by the fact that \MSWORD\ provides a math editor and therefore needs a font. That
131font is Cambria and it is (at the time of writing this) the only font that comes
132from the origin. It has not been extended, nor fixed so basically what is in
133there kind of has become the standard. The other \OPENTYPE\ math fonts are a
134curious mix of old and new technology and again not much has happened there.
135
136Now, when it comes to choices here, a few can be made based on conclusions drawn
137during decades of dealing with these fonts and the assumed technology.
138
139\startitemize
140
141\startitem
142    There has be no real developments so we can just assume that what we got is
143    what we will have forever. Cambria is and remains the standard, quite some
144    fonts shipped with \TEX\ have issues that will stay, and new fonts,
145    especially when developed outside \TEX's scope likely also have issues,
146    because, after all, what is used for testing them?
147\stopitem
148
149\startitem
150    Only a few renders support the new technology. It is unlikely that \MSWORD\
151    will \ change and basically \XETEX\ and \LUATEX\ are also frozen. On the web
152    old school fonts are used, at least till 2023. Plenty of time went by since
153    the beginning of the century and nothing improved.
154\stopitem
155
156\startitem
157    The most important font properties that play a role are parameters, italic
158    correction, variants and extensibles, anchors for accents, stylistic
159    alternates, script alternates and staircase kerns. There are some rules of
160    how to apply italic correction, but many fonts make them unapplicable. The
161    same is true for anchors and kerns. There are only top kerns.
162\stopitem
163
164\startitem
165    Italic correction is a flawed concept and we decided to just ignore them:
166    when specified we add it to the width and discard them afterwards. The value
167    is translated into a bottom right corner kern. For large operators we
168    translate them to top and bottom accents.
169\stopitem
170
171\startitem
172    Top accents can be flawed so in many cases we can just ignore them. They
173    only make sense for italic shapes anyway.
174\stopitem
175
176\startitem
177    Staircase kerns are a nice idea but make no sense. First of all they concern
178    two characters, nucleus and script, but we can also have accents, fraction,
179    fenced stuff and other constructs in scripts so instead we prefer a system of
180    corner kerns. Also, we noticed that staircase kerns are often implemented
181    partially and even then not that well, probably because there was no way to
182    test them. Even worse is that when they are inconsistent formulas can look
183    rather inconsistent. So, we translate staircase kerns into corner kerns and
184    add and|/| overload them by corner kerns. These kerns can then be applied for
185    any reasonable combination.
186\stopitem
187
188\startitem
189    Extensible are mixed breed. Rules should be extensibles but aren't. Some snippets
190    have \UNICODE\ points so they can be used to construct missing glyphs but the
191    repertoire is inconsistent. Because we don't expect \UNICODE\ to adapt we
192    therefore provide alternative solutions.
193\stopitem
194
195\startitem
196    The repertoire of math parameters is on the one hand incomplete and on the
197    other hand less dependent on the font and more on intended usage. So, apart
198    from a few, we end up with adapting to our needs. It is part of the more granular
199    control that we wish.
200\stopitem
201
202\startitem
203    Gaps in alphabet vectors are a pain but the engine is agnostic of them. For some
204    reason the \TEX\ community let itself down on this so it has to cope at the macro
205    level. It is by now an old problem.
206\stopitem
207
208\stopitemize
209
210So, to summarize the font part, an alternative standard could discard the concept
211of italic correction and go for proper widths, a simplified corner kern model,
212provide top and bottom accents, prescribe a repertoire of extensibles and
213snippets and at least fill the gaps in alphabets instead of relying on shared
214glyphs. It won't happen any time soon, but still we do follow that approach and
215have the engine ready for it. Because we adapt the fonts runtime to this, we can
216eventually remove all the code related to italic correction and staircase kerns,
217simply because it is not used.
218
219\stopsection
220
221\startsection[title=Rules]
222
223The original \TEX\ engine actually has only two graphical elements: glyphs and
224rules. These have a width, height and depth and when decisions are made, for
225instance when deciding where to break a line, or when boxes are constructed these
226dimensions have to be known. Actually, \TEX\ doesn't really care what these
227elements are, it's the dimensions that matter most. Graphics for instance can be
228abstract objects, traditionally injected via so called specials wrapped into a
229box of given dimensions. The \PDFTEX\ and later engines added a native
230representation but basically it acted like a box (or rule if you like). It's the
231backend that turns glyphs, rules and these special boxes into something that one
232can see and print.
233
234Rules have the three dimensions we mentioned. There are horizontal and vertical rules,
235but only at the primives level. Once you specified an \type {\hrule} or \type {\vrule}
236it became a generic rule with the main difference being the default dimensions. A rule
237initializes with so called running dimensions, think of signals that the final dimension
238comes from the encapsulating box.
239
240Here we have a vertical rule: {\darkblue \vrule width 3cm height 5mm depth 2mm}
241with width 3cm, height 5mm and depth 2mm. If we don't specify a width we get the
242default thickness of 0.4pt, as in {\darkblue \vrule height 5mm depth 2mm} and when we
243prefix it with \type {\leaders} and let it follow by a \type {\hfill} we get this:
244{\darkblue \leaders \vrule height 5mm depth 2mm\hfill}.
245
246When we put on an \type {\hrule} on an empty line the running width kicks in:
247\crlf {\darkblue \hrule height 5mm depth 2mm}\crlf which is a feature that one
248can use in for instance tables. However the fact that we only talk rectangles
249means that there is only a limited repertoire of applications. In order to frame
250some text you need four (disconnected) rules, For a background fill you can use a
251single rule. There is also an application for rules that have height and depth
252but no width: these so called struts that can enforce vertical spacing and
253dimensions.
254
255So what does \LUAMETATEX\ bring to the rules? Because the engine itself is only
256interested in dimensions it's more about passing information to the backend. For
257this we have a few more fields in the rule nodes that can be set from \LUA. This
258permits for instance to hook in \METAPOST\ graphics that adapt like rules. There
259are a few more primitives, one for making struts: they can take their dimensions
260from a character. In math mode they're invisible and don't influence inter-atom
261spacing but still take their role in determining dimensions. Then there are the virtual
262rules that have dimensions (to be used in the backend) but don't contribute in the
263frontend. The \type {\novrule} and \type {\nohrule} do contribute but are ignores in the
264backend so they are cheap alternatives for empty boxes with specific dimensions set.
265
266Some rule subtypes are set by the engine, for instance the math engine marks
267over, under, fraction and radical rules. In \LUA\ one can mark outline, user, box
268and image rules so that node list processors can take their properties into
269accounT when needed, the frontend is only interested in the dimensions and sees
270them as normal rules.
271
272\startbuffer
273\hrule height \strutht depth \strutdp on 0.04tw off 0.01tw \relax
274\stopbuffer
275
276\bgroup \nowhitespace
277\strut\darkblue \getbuffer
278\egroup
279
280Here we have the following call:
281
282\typebuffer
283
284The \type {on} and \type {off} are among the new keys and they do nothing at the
285\TEX\ end. It is the backend that will create the dash pattern. You can achieve
286the same effect with leaders but while here we have a single rule, for a leader
287the engine will make as many rules as are needed for this dash pattern. This is a
288good example of adding little to the fontend in order to make the backend do the
289job. In a similar fashion outlines are delegated. Other tricks involve offsets
290and there is room for some additional features but for now they are on the
291\quotation {Only when I need it.} list, after all we need something to wish for.
292
293\stopsection
294
295\startsection[title=Paragraphs]
296
297A lot can be said about paragraphs but we keep it short here. Much more can be
298found in for instance the articles that we wrote on the subject. When you enter
299(or generate) text it will be added to a list (of nodes). That list can become a
300horizontal box, vertical box, or end up in the main vertical list. When we go
301vertical the list will be split in lines and the process is called line breaking.
302Between the lines we can get penalties that tell the machinery how a paragraph of
303lines can be split over page boundaries.
304
305When breaking the engine can use up to three passes: a first pass that uses \type
306{\pretolerance} as criterion, a tolerant pass with hyphenation enabled using
307\type {\tolerance} and an emergency pass that kicks in \type {\emergencystretch}
308when set. In \LUAMETATEX\ we can have additional passes that come online
309depending on criteria and|/|or thresholds; search for \type {\parpasses} to learn
310more about this.
311
312The par builder in \LUAMETATEX\ has more features that users can control and also
313normalized the resulting lines so that later on from the \LUA\ end they can be
314manipulated easier. There are also ways to let embedded inserts, marks and (v)adjusts
315migrate to the outer level. All this takes more runtime than in original \TEX\ but
316in practice one won't really notice this because we gain in other places.
317
318Most or what is new is available as features in \CONTEXT, most noticeably in
319extra keys to \type {\setupalign}. It is also good to know that we have ways to
320hook specific features in what is called \quote {wrapping up paragraph}. Also,
321contrary to traditional \TEX\ we configured \CONTEXT\ to use the mechanism that
322freezes paragraph specific parameters with the current paragraph so that there is
323no (or at least less) interference with grouping.
324
325\stopsection
326
327\startsection[title=Pages]
328
329{\em todo}
330
331\stopsection
332
333\startsection[title=Alignments]
334
335{\em todo}
336
337\stopsection
338
339\startsection[title=Adjusts]
340
341You can put stuff before and after lines using \type {\vadjust} and at the edges
342using \type {\localleftbox} and alike. Both are seen in the par builder, where the
343boxes contribute to the dimensions and the adjusted material is inserted when the
344paragraph is wrapped up and contributed to the current list. In \LUAMETATEX\ these
345mechanism have been extended so that we can actually uses them in am meaningful
346way.
347
348\stopsection
349
350
351\startsection[title=Marks]
352
353These signals in the text are used for managing (for instance) running headers
354and a few extra features have been added, like migration to an outer level and
355resets. In \MKIV\ we handled marks in \LUA\ but with \LUAMETATEX\ it makes sense
356to use the engine.
357
358\stopsection
359
360\startsection[title=Inserts]
361
362Inserts are signals that end up in lines and migrate to the outer level, that is
363the main vertical list. An example of usage is footnotes. In the main vertical
364list they are bound to the line they relate to so that the page builder can make
365sure that they end up on the same page. In \LUAMETATEX\ they can bubble up from
366deeply nested boxes. Contrary to the traditional binding of an insert class to
367various registers in \LUAMETATEX\ they can be managed independently which means
368that they have more properties.
369
370\stopsection
371
372\startsection[title=Boxes]
373
374{\em todo}
375
376\stopsection
377
378\startsection[title=Language]
379\stopsection
380
381\startsection[title=Math]
382
383Plenty has been written about the multi-year project of opening up and extending
384the math engine. Opening up and providing full control is part of supporting and
385experimenting with \OPENTYPE\ math fonts but we already discussed this in a
386previous section. Another aspect of opening up is making hard coded properties
387configureable, even if that feature will hardly be used, simply because the
388built-in defaults make sense. Then there is all kind of control over rendering
389that can be controlled by keywords to the math specific elements like atoms,
390fractions, operators, accents, radicals and fences.
391
392Because traditional fonts are phased out in favour of (often flawed) \OPENTYPE\
393variants much of what is new is also controlled by fonts, be it that we have our
394own extensions. In \CONTEXT\ mathfonts are tweaked to fit our model. Inter atom
395spacing, penalties, discretionaries, continuation scripts (think multiscripts,
396pre and post), additional classes, dictionaries, linebreaks, carrying properties
397over math groups, are all features that make it possible to renderer more precise
398math without the need for manual intervention. It often looks, for instance from
399posts on support platforms, that the more or less standard math has to come with
400tweaking your source; it has become an accepted practice. In \CONTEXT\ we always
401had structure and we added some more of that and because the math engine carries
402more information around we could eventually simplify some code otherwise done in
403\LUA.
404
405By looking at what \CONTEXT\ actually needs, we could decide to strip down the
406math engine (old as well as new features). We can also decide to eventually just
407assume wide fonts to be used and drop old font support. After all, because one
408has to load the fonts with \LUA, it's not hard to map traditional fonts to
409(extended) \OPENTYPE\ alternatives, which is actually what we do anyway with for
410instance Antykwa.
411
412\stopsection
413
414\startsection[title=Programming]
415
416{\em todo}
417
418\stopsection
419
420\startsection[title=Protection]
421
422The idea behind \TEX\ is that users define macros. However, when they do so in
423the perspective of a macro package there is the danger that core functionality
424can be overwritten. Now, one can of course make all primitives less accessible,
425for instance by some prefix. But that makes no real sense for features that
426belong to the language. When users use CamelCase for their names they're unlikely
427to run into issues, so while internal macros are actually prefixed, we don't
428do that with the primitives, so you can write code that looks \TEX.
429
430Over time \CONTEXT\ has been ridiculed by non users for prefixing with \type
431{\do} or \type {\dodo} but that's by folk who love long (cryptic) names with many
432underscores and other inaccessible characters. The way we protect users from
433accidental overloading is by using the \LUAMETATEX\ overload protection system.
434Macros (and primitives) can be tagged in way so that the engine can issue warning
435or even error in case of an undesirable definition.
436
437There is of course some overhead involved in for instance every \type {\def} or
438\type {\let} but it is little and the engine is fast anyway.
439
440\stopsection
441
442\startsection[title=Optimization]
443
444There are many places where the engine could be optimized without getting
445obscure. One reason is that the memory layout is somewhat different because we
446snap to 8, 16, 32 or 64 bits and the engine being a \UNICODE\ capable one already
447has more memory available in some places than what was needed. Also, knowing
448usage patterns, it was possible to identify possible bottlenecks and widen the
449necks.
450
451Furthermore, it was possible to improve input handling, logging, save stack
452usage, keyword parsing, expressions, and much more. On the other hand nodes
453became larger so there we loose some. The \LUAMETATEX\ engine is faster than
454\LUATEX, although some of the gain is lost on the fact that one needs to use
455\LUA\ backend.
456
457\stopsection
458
459\startsection[title=Input]
460
461The input can come from files, token lists, macros and \LUA\ which means many
462places. When it comes from \LUA\ it can be tokens, nodes, string, and each has
463its special way of handling and the engine has to keep track of this when it
464accumulated the input that pops up after a \LUA\ call. This is done as efficient
465as possible without sacrificing performance. The fact that we have \UTF\ should
466not have too much impact.
467
468\stopsection
469
470\startsection[title=Nesting]
471
472When you enter a group a stack boundary is set and when some value changes the
473original value is pushed on the stack. After leaving the group values are
474restored. The engine tries to avoid redundant actions which improves memory usage
475and runtime.
476
477Every macro expansion, opened file, expanded token list, etc.\ pushes the input
478stack and that comes with overhead. Again we have tried to minimize the impact
479and thereby gain a bit over \LUATEX.
480
481Other stacks like those used by math, alignment, conditionals, expressions etc.\
482have also been improved some. On the other hand, by unweaving some shared code
483there can be a price to pay, but as with everything usage patterns indicate no
484penalty here.
485
486\stopsection
487
488\startsection[title=conditions]
489
490We already had more conditionals in \LUATEX\ but again the repertoire of
491conditionals has been extended. This permits us to remove some middle-layer
492helpers and stay closer to the core language. It also helps to improve
493performance.
494
495Another important addition has been \type {\orelse} than permits us to write test
496in a way similar to what other language provide with for instance \type {elseif}
497or \type {else if}.
498
499\stopsection
500
501\startsection[title=macros]
502
503Expanding macros happens a lot especially in a more complex macro package. This
504means that adding features in that area can have a large impact on runtime.
505Nevertheless the argument parser now provides a few handfuls of variants in
506picking up arguments with out noticeable degradation, especially because these
507new features can gain performance.
508
509At the same time there have been some optimizations in storing macro related
510states, checking and accessing parameters. There are additional (internal)
511classes of macros that make for a more natural implementation; for instance
512\type {\protected} macros are now first class commands.
513
514\stopsection
515
516\startsection[title=Keywords]
517
518Some primitives accept one or more keywords and \LUAMETATEX\ adds some more. In
519order to deal with this efficiently the keyword scanner has been optimized, where
520even the context was taken into account. As a result the scanner was quite a bit
521faster. This kind of optimization was a graduate process the eventually ended up
522in what we have now. In traditional \TEX\ (and also \LUATEX) the order of
523keywords is sometimes mixed and sometimes prescribed. In most cases only one
524occurrence is permitted. So, for instance, this is valid in \LUATEX:
525
526\starttyping
527\hbox attr 123 456 attr 123 456 spread 10cm { }
528\hrule width 10cm depth 3mm
529\hskip 3pt plus 2pt minus 1pt
530\stoptyping
531
532The \type {attr} comes before the \type {spread}, rules can have multiple mixed
533dimension specifiers, and in glue the optional \type {minus} part always comes
534last. The last two commands are famous for look ahead side effects which is why
535macro packages will end them with something not keyword, like \type {\relax},
536when needed.
537
538In \LUAMETATEX\ the following is okay. Watch the few more keywords in box and
539rule specifications.
540
541\starttyping
542\hbox reverse to 10cm attr 123 456 orientation 4 xoffset 10pt spread 10cm { }
543\hrule xoffset 10pt width 10cm depth 3mm
544\hskip 3pt minus 1pt plus 2pt
545\stoptyping
546
547Here the order is not prescribed and, as demonstrated with the box specifier, for
548instance dimensions (specified by \type {to} or \type {spread} can be overloaded
549by later settings. In case you wonder if that breaks compatibility: in some way
550it does but bad or sloppy keyword usage breaks a run anyway. For instance \type
551{minuscule} results in \type {minus} with no dimension being seen. So, in the end
552the user should not noticed it and when a user does, the macro package already
553had an issue that had to be fixed.
554
555\stopsection
556
557\startsection[title=Directions]
558
559The directional model in \LUAMETATEX\ is a simplified version the the model used
560in \LUATEX. In fact, not much is happening at all: we only register a change in
561direction. The approach is that we try to make node lists balanced but also try
562to avoid some side effects. What happens is quite intuitive if we forget about
563spaces (turned into glue) but even there what happens makes sense if you look at
564it in detail. However that logic makes in|-|group switching kind of useless when
565no properly nested grouping is used: switching from right to left several times
566nested, results in spacing ending up after each other due to nested mirroring. Of
567course a sane macro package will manage this for the user but here we are
568discussing the low level injection of directional information.
569
570This is what happens:
571
572\starttyping
573\textdirection 1 nur {\textdirection 0 run \textdirection 1 NUR} nur
574\stoptyping
575
576This becomes stepwise:
577
578\startnarrower
579\starttyping
580injected: [push 1]nur {[push 0]run [push 1]NUR} nur
581balanced: [push 1]nur {[push 0]run [pop 0][push 1]NUR[pop 1]} nur[pop 0]
582result  : run {RUNrun } run
583\stoptyping
584\stopnarrower
585
586And this:
587
588\starttyping
589\textdirection 1 nur {nur \textdirection 0 run \textdirection 1 NUR} nur
590\stoptyping
591
592becomes:
593
594\startnarrower
595\starttyping
596injected: [+TRT]nur {nur [+TLT]run [+TRT]NUR} nur
597balanced: [+TRT]nur {nur [+TLT]run [-TLT][+TRT]NUR[-TRT]} nur[-TRT]
598result  : run {run RUNrun } run
599\stoptyping
600\stopnarrower
601
602Now, in the following examples watch where we put the braces:
603
604\startbuffer
605\textdirection 1 nur {{\textdirection 0 run} {\textdirection 1 NUR}} nur
606\stopbuffer
607
608\typebuffer
609
610This becomes:
611
612\startnarrower
613\getbuffer
614\stopnarrower
615
616Compare this to:
617
618\startbuffer
619\textdirection 1 nur {{\textdirection 0 run }{\textdirection 1 NUR}} nur
620\stopbuffer
621
622\typebuffer
623
624Which renders as:
625
626\startnarrower
627\getbuffer
628\stopnarrower
629
630So how do we deal with the next?
631
632\startbuffer
633\def\ltr{\textdirection 0\relax}
634\def\rtl{\textdirection 1\relax}
635
636run {\rtl nur {\ltr run \rtl NUR \ltr run \rtl NUR} nur}
637run {\ltr run {\rtl nur \ltr RUN \rtl nur \ltr RUN} run}
638\stopbuffer
639
640\typebuffer
641
642It gets typeset as:
643
644\startnarrower
645\startlines
646\getbuffer
647\stoplines
648\stopnarrower
649
650We could define the two helpers to look back, pick up a skip, remove it and
651inject it after the dir node. But that way we loose the subtype information that
652for some applications can be handy to be kept as|-|is. This is why we now have a
653variant of \prm {textdirection} which injects the balanced node before the skip.
654Instead of the previous definition we can use:
655
656\startbuffer[def]
657\def\ltr{\linedirection 0\relax}
658\def\rtl{\linedirection 1\relax}
659\stopbuffer
660
661\typebuffer[def]
662
663and this time:
664
665\startbuffer[txt]
666run {\rtl nur {\ltr run \rtl NUR \ltr run \rtl NUR} nur}
667run {\ltr run {\rtl nur \ltr RUN \rtl nur \ltr RUN} run}
668\stopbuffer
669
670\typebuffer[txt]
671
672comes out as a properly spaced:
673
674\startnarrower
675\startlines
676\getbuffer[def,txt]
677\stoplines
678\stopnarrower
679
680Anything more complex that this, like combination of skips and penalties, or
681kerns, should be handled in the input or macro package because there is no way we
682can predict the expected behavior. In fact, the \prm {linedirection} is just a
683convenience extra which could also have been implemented using node list parsing.
684
685Directions are complicated by the fact that they often need to work over groups
686so a separate grouping related stack is used. A side effect is that there can be
687paragraphs with only a local par node followed by direction synchronization
688nodes. Paragraphs like that are seen as empty paragraphs and therefore ignored.
689Because \prm {noindent} doesn't inject anything but a \prm {indent} injects
690an box, paragraphs with only an indent and directions are handles and paragraphs
691with content. When indentation is normalized a paragraph with an indentation
692skip is seen as content.
693
694\stopsection
695
696\startsection[title=Hooks]
697
698{\em todo}
699
700\stopsection
701
702\startsection[title=Units]
703
704The familiar \TEX\ units like \type {pt} and \type {cm} are supported but since
705the 2021 \CONTEXT\ meeting we also support the Knuthian Potrzebie, cf.\ \typ
706{en.wikipedia.org/wiki/Potrzebie}. The two character acronym is \type {dk}. One
707\type {dk} is 6.43985pt. This unit is particularly suited for offsets in framed
708examples.
709
710In 2023 we added the Edith (\type {es}) and Tove (\type {ts}) as metric
711replacements for the inch (\type {in}). As with the \type {dk} more background
712information can be found in documents that come with \CONTEXT\ and user group
713journals. The \type {eu} unit starts out as one \type {es} but can be scaled with
714\prm {eufactor}.
715
716\startbuffer
717\localcontrolledloop -5 55 5 {
718    \eufactor=\currentloopiterator
719    \dontleavehmode\strut
720    \vrule height .1es depth .25ts width 1dk\relax\quad
721    \vrule height .1es depth .25ts width 1eu\relax\quad
722    \the\currentloopiterator
723    \par
724}
725\stopbuffer
726
727\typebuffer
728
729This example code shows all four new units. Watch how \prm {eufactor} is clipped
730to a value in the range $1-50$. The default factor of $10$ makes the European
731Unit equivalent to ten Toves or one Edith.
732
733\startpacked
734\startcolor[darkgray]
735\getbuffer
736\stopcolor
737\stoppacked
738
739In addition to these there can be user units but because these are macro package
740dependent they are not discussed here.
741
742\stopsection
743
744\startsection[title=Local control]
745
746There are a few new primitives that permit what we call local controlled
747expansion. This permits for instance expanding non expandable macros and even
748typesetting inside an expansion context like \type {\edef}. Regular \TEX has a
749main loop to where it returns after every primitive action, but local control let
750the engine go into a nested main loop.
751
752\stopsection
753
754\startsection[title=Overload protection]
755
756Protection is achieved via prefixes. Depending on the value of the \prm
757{overloadmode} variable warnings or errors will be triggered. Examples of usage
758can be found in some documents that come with \CONTEXT, so here we just stick to
759the basics.
760
761\starttyping
762\mutable  \def\foo{...}
763\immutable\def\foo{...}
764\permanent\def\foo{...}
765\frozen   \def\foo{...}
766\aliased  \def\foo{...}
767\stoptyping
768
769A \prm {mutable} macro can always be changed contrary to an \prm {immutable} one.
770For instance a macro that acts as a variable is normally \prm {mutable}, while a
771constant can best be immutable. It makes sense to define a public core macro as
772\prm {permanent}. Primitives start out a \prm {permanent} ones but with a primitive
773property instead.
774
775\startbuffer
776          \let\relaxone  \relax 1: \meaningfull\relaxone
777\aliased  \let\relaxtwo  \relax 2: \meaningfull\relaxtwo
778\permanent\let\relaxthree\relax 3: \meaningfull\relaxthree
779\stopbuffer
780
781\typebuffer
782
783The \prm {meaningfull} primitive is like \prm {meaning} but report the
784properties too. The \prm {meaningless} companion reports the body of a macro.
785Anyway, this typesets:
786
787\startlines \tttf \getbuffer \stoplines
788
789So, the \prm {aliased} prefix copies the properties. Keep in mind that a macro
790package can redefine primitives, but \prm {relax} is an unlikely candidate.
791
792There is an extra prefix \prm {noaligned} that flags a macro as being valid
793for \prm {noalign} compatible usage (which means that the body must contain that
794one. The idea is that we then can do this:
795
796\starttyping
797\permanent\protected\noaligned\def\foo{\noalign{...}} % \foo is unexpandable
798\stoptyping
799
800that is: we can have protected macros that don't trigger an error in the parser
801where there is a look ahead for \prm {noalign} which is why normally protection
802doesn't work well. So: we have macro flagged as permanent (overload protection),
803being protected (that is, not expandable by default) and a valid equivalent of
804the noalign primitive. Of course we can also apply the \prm {global} and \prm
805{tolerant} prefixes here. The complete repertoire of extra prefixes is:
806
807\starttabulate
808\FL
809\NC \type {frozen}     \NC a macro that has to be redefined in a managed way \NC \NR
810\NC \type {permanent}  \NC a macro that had better not be redefined \NC \NR
811\NC \type {primitive}  \NC a primitive that normally will not be adapted \NC \NR
812\NC \type {immutable}  \NC a macro or quantity that cannot be changed, it is a constant \NC \NR
813\NC \type {mutable}    \NC a macro that can be changed no matter how well protected it is \NC \NR
814\ML
815\NC \type {instance}   \NC a macro marked as (for instance) be generated by an interface \NC \NR
816\ML
817\NC \type {noaligned}  \NC the macro becomes acceptable as \type {\noalign} alias \NC \NR
818\ML
819\NC \type {overloaded} \NC when permitted the flags will be adapted \NC \NR
820\NC \type {enforced}   \NC all is permitted (but only in zero mode or ini mode) \NC \NR
821\NC \type {aliased}    \NC the macro gets the same flags as the original \NC \NR
822\ML
823\NC \type {untraced}   \NC the macro gets a different treatment in tracing \NC \NR
824\LL
825\stoptabulate
826
827The not yet discussed \prm {instance} is just a flag with no special meaning
828which can be used as classifier. The \prm {frozen} also protects against overload
829which brings amount of blockers to four.
830
831To what extent the engine will complain when a property is changed in a way that
832violates the flags depends on the parameter \prm {overloadmode}. When this
833parameter is set to zero no checking takes place. More interesting are values
834larger than zero. If that is the case, when a control sequence is flagged as
835mutable, it is always permitted to change. When it is set to immutable one can
836never change it. The other flags determine the kind of checking done. Currently
837the following overload values are used:
838
839\starttabulate[|l|l|c|c|c|c|c|]
840    \NC   \NC         \BC immutable \BC permanent \BC primitive \BC frozen \BC instance \NC \NR
841    \NC 1 \NC warning \NC \star     \NC \star     \NC \star     \NC        \NC          \NC \NR
842    \NC 2 \NC error   \NC \star     \NC \star     \NC \star     \NC        \NC          \NC \NR
843    \NC 3 \NC warning \NC \star     \NC \star     \NC \star     \NC \star  \NC          \NC \NR
844    \NC 4 \NC error   \NC \star     \NC \star     \NC \star     \NC \star  \NC          \NC \NR
845    \NC 5 \NC warning \NC \star     \NC \star     \NC \star     \NC \star  \NC \star    \NC \NR
846    \NC 6 \NC error   \NC \star     \NC \star     \NC \star     \NC \star  \NC \star    \NC \NR
847\stoptabulate
848
849The even values (except zero) will abort the run. A value of 255 will freeze this
850parameter. At level five and above the \prm {instance} flag is also checked but
851no drastic action takes place. We use this to signal to the user that a specific
852instance is redefined (of course the definition macros can check for that too).
853
854The \prm {overloaded} prefix can be used to overload a frozen macro. The \prm
855{enforced} is more powerful and forces an overload but that prefix is only
856effective in ini mode or when it's embedded in the body of a macro or token list
857at ini time unless of course at runtime the mode is zero.
858
859So far for a short explanation. More details can be found in the \CONTEXT\
860documentation where we can discuss it in a more relevant perspective. It must be
861noted that this feature only makes sense a controlled situation, that is: user
862modules or macros of unpredictable origin will probably suffer from warnings and
863errors when de mode is set to non zero. In \CONTEXT\ we're okay unless of course
864users redefine instances but there a warning or error is kind of welcome.
865
866There is an extra prefix \prm {untraced} that will suppress the meaning when
867tracing so that the macro looks more like a primitive. It is still somewhat
868experimental so what gets displayed might change.
869
870The \prm {letfrozen}, \prm {unletfrozen}, \prm {letprotected} and \prm
871{unletprotected} primitives do as their names advertise. Of course the \prm
872{overloadmode} must be set so that it is permitted.
873
874\stopsection
875
876\startsection[title=Tracing]
877
878There is are more tracing options, like in math, alignments and inserts, and
879tracing can be more detailed. This is partly a aide effect of the need for
880exploring new features. Tracing is not always compatible, if only because there
881are more possibilities, for instance in the way macros are defined and can handle
882arguments.
883
884\stopsection
885
886\stopdocument
887
888% \setupbodyfont[pagella]
889% \starttext
890% \definefont[TestA][SerifBold*default,oldstyle]
891% \def\TestB  {{\infofont[\the\glyphdatafield]}}
892% \def\TestC#1{{\infofont(#1)}\csname#1\endcsname}
893% \bgroup
894%     \crlf abc 123\crlf
895%     \bgroup
896%         \TestA
897%         \TestB
898%         \crlf \strut \kern10pt abc 123 \crlf
899%     \egroup
900%     \TestB
901%     \TestC{smallcaps}%
902%     \TestB
903%     \crlf abc 123 \crlf
904%     \bgroup
905%         \TestA
906%         \TestB
907%         \crlf \strut \kern10pt abc 123 \crlf
908%     \egroup
909%     \TestB
910%     \bgroup
911%         \TestC{oldstyle}%
912%         \TestB
913%         \crlf \strut \kern10pt abc 123 \crlf
914%     \egroup
915%     \TestB
916%     \crlf abc 123 \crlf
917% \egroup
918% \stoptext
919