% language=us runpath=texruns:manuals/luametatex \environment luametatex-style \startdocument[title=principles] \startsection[title={Introduction}] This is a bit odd manual but needed anyway. In the process of adding features to \LUAMETATEX\ and adapting \CONTEXT\ accordingly some decisions were made. On the one hand generic flexibility is a criterion used when the extending engine, on the other hand practical usability in \CONTEXT\ is used to decide where to draw a line or make some choices. It makes no sense to complicate the already complex engine even more, or cripple \CONTEXT\ when cleaner (low level) solutions are possible. Here I will collect some of the considerations and mention the choices made. These are mostly mine but some result from discussions and experiments. This overview is not complete, new primitives are discussed elsewhere and the \CONTEXT\ low level manuals explain how to use these. Consider this to be a teaser. {\em This summary is work in progress.} \stopsection \startsection[title=Text fonts] Plenty has been written about fonts in \TEX, so here I will only mention a few aspects. Traditionally the \TEX\ engines works with copies of fonts at given sizes. For large fonts that is kind of inefficient. This is why in \LUAMETATEX\ we can scale a font on|-|the|-|fly using \typ {\glyphscale}, \typ {\glyphxscale} and \typ {\glyphyscale}. This feature is also used to implement a more efficient (although not 100\percent\ metric compatible) compact font mode. It works okay in text as well as math although it comes at a price: many more calculations are needed at the engine end. One way to get an expanded, squeezed, emboldened or slanted font in \CONTEXT\ is to use the effects mechanism. It is quite flexible but again comes at a price because the backend has to do more work which is measurable, especially because effects can apply to the font or individual glyphs. However, the advantages out|-|weight the disadvantages. At the cost of yet a bit more performance a more native variant is also available using \typ {\glyphslant} and \typ {\glyphweight}. \definefontfeature[extendtest] [default][extend=1.25] \definefontfeature[squeezetest][default][squeeze=1.25] \definefontfeature[slanttest] [default][slant=0.5] \definefontfeature[weighttesta][default][weight=0.2] % 0.4pt is bad for the g in lm \definefontfeature[weighttestb][default][effect={weight=0.2,auto=yes}] \starttexdefinition demo #1 \startcombination[nx=6,ny=1,distance=12.5mm] {\scale[scale=8000]{\definedfont[Serif*default @ 10bp]\ruledhbox{\color[maincolor]{#1}}}} {} {\scale[scale=8000]{\definedfont[Serif*extendtest @ 10bp]\ruledhbox{\color[maincolor]{#1}}}} {} {\scale[scale=8000]{\definedfont[Serif*squeezetest @ 10bp]\ruledhbox{\color[maincolor]{#1}}}} {} {\scale[scale=8000]{\definedfont[Serif*slanttest @ 10bp]\ruledhbox{\color[maincolor]{#1}}}} {} {\scale[scale=8000]{\definedfont[Serif*weighttesta @ 10bp]\ruledhbox{\color[maincolor]{#1}}}} {} {\scale[scale=8000]{\definedfont[Serif*weighttestb @ 10bp]\ruledhbox{\color[maincolor]{#1}}}} {} \stopcombination \stoptexdefinition \startlinecorrection \demo{a} \stoplinecorrection \startlinecorrection \demo{x} \stoplinecorrection \startlinecorrection \demo{p} \stoplinecorrection %startlinecorrection \demo{(} \stoplinecorrection \startlinecorrection \demo{g} \stoplinecorrection \startlinecorrection \demo{-} \stoplinecorrection Extending, as seen in the second renderings, scales the shapes horizontally, while squeezing, in the third renderings, does it in the vertical direction. In both cases the dimensions have to be adapted. This is not the case when we slant. The last two samples in a row have an increased weight, and these are the more tricky cases because here one can argue how to scale and reposition a shape. When a shape is above the baseline we increase the height, and when it goes below we increase the depth. The engine is capable to increase the width, height and depth and shift the shape a little. It only makes sense to adapt the height and depth when they are non|-|zero. It will never be perfect, but this feature is not perfect anyway. The way fonts are set up in a \TEX\ macro package often originates in the past, if only because it came with fonts. The Computer Modern fonts are among the few that have multiple design sizes. However, the collection is pretty much based on a ten point design. For math there are seven and five point variants for the script sizes, for footnotes an eight point makes sense and section heads can use the larger twelve point plus the few larger sizes. Setting up a twelve point body font environment, as we have in \CONTEXT, is quite doable with the fonts but for an eleven point body font more compromised have to be made. One can wonder why in \CONTEXT\ the ten point math setup of 10/7/5 became 12/9/7 instead if 12/8.4/6 and the reason is just that when there were still bitmap fonts one didn't want too many (intermediate) sizes. Anyway, we're sort of stuck with this default setup now, but nothing prevents users to redefine a body font environment. Another speciality of \TEX\ (fonts) is that they have italic correction, something that lacks in \OPENTYPE\ fonts (apart from math btu there it serves a different purpose). We can however emulate it, and in \CONTEXT\ that is an option. Given that we have to make choices it is clear that the engine can only be supportive here, especially when we use the \typ {\glyphslant} method. A curious case is the following: in Computer Modern we find italic correction in the upright fonts, for instance between an \quote {f} and \quote {h}. Dealing with this automatically is impossible because italic correction is not to be applied between glyph runs of the same font. % quite some italics with not angle and uprights with angles % why not proper boundingbox (right side bearing) with (now also small) width % instead of unpredictable either or not urx \stopsection \startsection[title=Math fonts] Support for math in an \UNICODE\ aware engine is also driven by the repertoire of characters and their organization in \UNICODE, as well as by \OPENTYPE\ math as cooked up by \MICROSOFT\ with a bit of input from \TEX\ folk. The engine is agnostic when it comes to \UNICODE: there are no character codes interpreted in special ways. There are math alphabets but these are not special: in a traditional eight bit engine we have families to deal with them, in a \UNICODE\ aware engine there are several solutions. The most important character property that has some consequence is the math class but for dealing with that we're on our own anyway. Everything \UNICODE\ related is up to \CONTEXT\ to deal with, and it is the macro package that drives the engine, using the constructs that are available, like atoms with specific classes, fractions, accents, delimiters, fences, radicals, operators etc. When it comes to fonts it is more complex. The \OPENTYPE\ math standard is driven by the fact that \MSWORD\ provides a math editor and therefore needs a font. That font is Cambria and it is (at the time of writing this) the only font that comes from the origin. It has not been extended, nor fixed so basically what is in there kind of has become the standard. The other \OPENTYPE\ math fonts are a curious mix of old and new technology and again not much has happened there. Now, when it comes to choices here, a few can be made based on conclusions drawn during decades of dealing with these fonts and the assumed technology. \startitemize \startitem There has be no real developments so we can just assume that what we got is what we will have forever. Cambria is and remains the standard, quite some fonts shipped with \TEX\ have issues that will stay, and new fonts, especially when developed outside \TEX's scope likely also have issues, because, after all, what is used for testing them? \stopitem \startitem Only a few renders support the new technology. It is unlikely that \MSWORD\ will \ change and basically \XETEX\ and \LUATEX\ are also frozen. On the web old school fonts are used, at least till 2023. Plenty of time went by since the beginning of the century and nothing improved. \stopitem \startitem The most important font properties that play a role are parameters, italic correction, variants and extensibles, anchors for accents, stylistic alternates, script alternates and staircase kerns. There are some rules of how to apply italic correction, but many fonts make them unapplicable. The same is true for anchors and kerns. There are only top kerns. \stopitem \startitem Italic correction is a flawed concept and we decided to just ignore them: when specified we add it to the width and discard them afterwards. The value is translated into a bottom right corner kern. For large operators we translate them to top and bottom accents. \stopitem \startitem Top accents can be flawed so in many cases we can just ignore them. They only make sense for italic shapes anyway. \stopitem \startitem Staircase kerns are a nice idea but make no sense. First of all they concern two characters, nucleus and script, but we can also have accents, fraction, fenced stuff and other constructs in scripts so instead we prefer a system of corner kerns. Also, we noticed that staircase kerns are often implemented partially and even then not that well, probably because there was no way to test them. Even worse is that when they are inconsistent formulas can look rather inconsistent. So, we translate staircase kerns into corner kerns and add and|/| overload them by corner kerns. These kerns can then be applied for any reasonable combination. \stopitem \startitem Extensible are mixed breed. Rules should be extensibles but aren't. Some snippets have \UNICODE\ points so they can be used to construct missing glyphs but the repertoire is inconsistent. Because we don't expect \UNICODE\ to adapt we therefore provide alternative solutions. \stopitem \startitem The repertoire of math parameters is on the one hand incomplete and on the other hand less dependent on the font and more on intended usage. So, apart from a few, we end up with adapting to our needs. It is part of the more granular control that we wish. \stopitem \startitem Gaps in alphabet vectors are a pain but the engine is agnostic of them. For some reason the \TEX\ community let itself down on this so it has to cope at the macro level. It is by now an old problem. \stopitem \stopitemize So, to summarize the font part, an alternative standard could discard the concept of italic correction and go for proper widths, a simplified corner kern model, provide top and bottom accents, prescribe a repertoire of extensibles and snippets and at least fill the gaps in alphabets instead of relying on shared glyphs. It won't happen any time soon, but still we do follow that approach and have the engine ready for it. Because we adapt the fonts runtime to this, we can eventually remove all the code related to italic correction and staircase kerns, simply because it is not used. \stopsection \startsection[title=Rules] The original \TEX\ engine actually has only two graphical elements: glyphs and rules. These have a width, height and depth and when decisions are made, for instance when deciding where to break a line, or when boxes are constructed these dimensions have to be known. Actually, \TEX\ doesn't really care what these elements are, it's the dimensions that matter most. Graphics for instance can be abstract objects, traditionally injected via so called specials wrapped into a box of given dimensions. The \PDFTEX\ and later engines added a native representation but basically it acted like a box (or rule if you like). It's the backend that turns glyphs, rules and these special boxes into something that one can see and print. Rules have the three dimensions we mentioned. There are horizontal and vertical rules, but only at the primives level. Once you specified an \type {\hrule} or \type {\vrule} it became a generic rule with the main difference being the default dimensions. A rule initializes with so called running dimensions, think of signals that the final dimension comes from the encapsulating box. Here we have a vertical rule: {\darkblue \vrule width 3cm height 5mm depth 2mm} with width 3cm, height 5mm and depth 2mm. If we don't specify a width we get the default thickness of 0.4pt, as in {\darkblue \vrule height 5mm depth 2mm} and when we prefix it with \type {\leaders} and let it follow by a \type {\hfill} we get this: {\darkblue \leaders \vrule height 5mm depth 2mm\hfill}. When we put on an \type {\hrule} on an empty line the running width kicks in: \crlf {\darkblue \hrule height 5mm depth 2mm}\crlf which is a feature that one can use in for instance tables. However the fact that we only talk rectangles means that there is only a limited repertoire of applications. In order to frame some text you need four (disconnected) rules, For a background fill you can use a single rule. There is also an application for rules that have height and depth but no width: these so called struts that can enforce vertical spacing and dimensions. So what does \LUAMETATEX\ bring to the rules? Because the engine itself is only interested in dimensions it's more about passing information to the backend. For this we have a few more fields in the rule nodes that can be set from \LUA. This permits for instance to hook in \METAPOST\ graphics that adapt like rules. There are a few more primitives, one for making struts: they can take their dimensions from a character. In math mode they're invisible and don't influence inter-atom spacing but still take their role in determining dimensions. Then there are the virtual rules that have dimensions (to be used in the backend) but don't contribute in the frontend. The \type {\novrule} and \type {\nohrule} do contribute but are ignores in the backend so they are cheap alternatives for empty boxes with specific dimensions set. Some rule subtypes are set by the engine, for instance the math engine marks over, under, fraction and radical rules. In \LUA\ one can mark outline, user, box and image rules so that node list processors can take their properties into accounT when needed, the frontend is only interested in the dimensions and sees them as normal rules. \startbuffer \hrule height \strutht depth \strutdp on 0.04tw off 0.01tw \relax \stopbuffer \bgroup \nowhitespace \strut\darkblue \getbuffer \egroup Here we have the following call: \typebuffer The \type {on} and \type {off} are among the new keys and they do nothing at the \TEX\ end. It is the backend that will create the dash pattern. You can achieve the same effect with leaders but while here we have a single rule, for a leader the engine will make as many rules as are needed for this dash pattern. This is a good example of adding little to the fontend in order to make the backend do the job. In a similar fashion outlines are delegated. Other tricks involve offsets and there is room for some additional features but for now they are on the \quotation {Only when I need it.} list, after all we need something to wish for. \stopsection \startsection[title=Paragraphs] A lot can be said about paragraphs but we keep it short here. Much more can be found in for instance the articles that we wrote on the subject. When you enter (or generate) text it will be added to a list (of nodes). That list can become a horizontal box, vertical box, or end up in the main vertical list. When we go vertical the list will be split in lines and the process is called line breaking. Between the lines we can get penalties that tell the machinery how a paragraph of lines can be split over page boundaries. When breaking the engine can use up to three passes: a first pass that uses \type {\pretolerance} as criterion, a tolerant pass with hyphenation enabled using \type {\tolerance} and an emergency pass that kicks in \type {\emergencystretch} when set. In \LUAMETATEX\ we can have additional passes that come online depending on criteria and|/|or thresholds; search for \type {\parpasses} to learn more about this. The par builder in \LUAMETATEX\ has more features that users can control and also normalized the resulting lines so that later on from the \LUA\ end they can be manipulated easier. There are also ways to let embedded inserts, marks and (v)adjusts migrate to the outer level. All this takes more runtime than in original \TEX\ but in practice one won't really notice this because we gain in other places. Most or what is new is available as features in \CONTEXT, most noticeably in extra keys to \type {\setupalign}. It is also good to know that we have ways to hook specific features in what is called \quote {wrapping up paragraph}. Also, contrary to traditional \TEX\ we configured \CONTEXT\ to use the mechanism that freezes paragraph specific parameters with the current paragraph so that there is no (or at least less) interference with grouping. \stopsection \startsection[title=Pages] {\em todo} \stopsection \startsection[title=Alignments] {\em todo} \stopsection \startsection[title=Adjusts] You can put stuff before and after lines using \type {\vadjust} and at the edges using \type {\localleftbox} and alike. Both are seen in the par builder, where the boxes contribute to the dimensions and the adjusted material is inserted when the paragraph is wrapped up and contributed to the current list. In \LUAMETATEX\ these mechanism have been extended so that we can actually uses them in am meaningful way. \stopsection \startsection[title=Marks] These signals in the text are used for managing (for instance) running headers and a few extra features have been added, like migration to an outer level and resets. In \MKIV\ we handled marks in \LUA\ but with \LUAMETATEX\ it makes sense to use the engine. \stopsection \startsection[title=Inserts] Inserts are signals that end up in lines and migrate to the outer level, that is the main vertical list. An example of usage is footnotes. In the main vertical list they are bound to the line they relate to so that the page builder can make sure that they end up on the same page. In \LUAMETATEX\ they can bubble up from deeply nested boxes. Contrary to the traditional binding of an insert class to various registers in \LUAMETATEX\ they can be managed independently which means that they have more properties. \stopsection \startsection[title=Boxes] {\em todo} \stopsection \startsection[title=Language] \stopsection \startsection[title=Math] Plenty has been written about the multi-year project of opening up and extending the math engine. Opening up and providing full control is part of supporting and experimenting with \OPENTYPE\ math fonts but we already discussed this in a previous section. Another aspect of opening up is making hard coded properties configureable, even if that feature will hardly be used, simply because the built-in defaults make sense. Then there is all kind of control over rendering that can be controlled by keywords to the math specific elements like atoms, fractions, operators, accents, radicals and fences. Because traditional fonts are phased out in favour of (often flawed) \OPENTYPE\ variants much of what is new is also controlled by fonts, be it that we have our own extensions. In \CONTEXT\ mathfonts are tweaked to fit our model. Inter atom spacing, penalties, discretionaries, continuation scripts (think multiscripts, pre and post), additional classes, dictionaries, linebreaks, carrying properties over math groups, are all features that make it possible to renderer more precise math without the need for manual intervention. It often looks, for instance from posts on support platforms, that the more or less standard math has to come with tweaking your source; it has become an accepted practice. In \CONTEXT\ we always had structure and we added some more of that and because the math engine carries more information around we could eventually simplify some code otherwise done in \LUA. By looking at what \CONTEXT\ actually needs, we could decide to strip down the math engine (old as well as new features). We can also decide to eventually just assume wide fonts to be used and drop old font support. After all, because one has to load the fonts with \LUA, it's not hard to map traditional fonts to (extended) \OPENTYPE\ alternatives, which is actually what we do anyway with for instance Antykwa. \stopsection \startsection[title=Programming] {\em todo} \stopsection \startsection[title=Protection] The idea behind \TEX\ is that users define macros. However, when they do so in the perspective of a macro package there is the danger that core functionality can be overwritten. Now, one can of course make all primitives less accessible, for instance by some prefix. But that makes no real sense for features that belong to the language. When users use CamelCase for their names they're unlikely to run into issues, so while internal macros are actually prefixed, we don't do that with the primitives, so you can write code that looks \TEX. Over time \CONTEXT\ has been ridiculed by non users for prefixing with \type {\do} or \type {\dodo} but that's by folk who love long (cryptic) names with many underscores and other inaccessible characters. The way we protect users from accidental overloading is by using the \LUAMETATEX\ overload protection system. Macros (and primitives) can be tagged in way so that the engine can issue warning or even error in case of an undesirable definition. There is of course some overhead involved in for instance every \type {\def} or \type {\let} but it is little and the engine is fast anyway. \stopsection \startsection[title=Optimization] There are many places where the engine could be optimized without getting obscure. One reason is that the memory layout is somewhat different because we snap to 8, 16, 32 or 64 bits and the engine being a \UNICODE\ capable one already has more memory available in some places than what was needed. Also, knowing usage patterns, it was possible to identify possible bottlenecks and widen the necks. Furthermore, it was possible to improve input handling, logging, save stack usage, keyword parsing, expressions, and much more. On the other hand nodes became larger so there we loose some. The \LUAMETATEX\ engine is faster than \LUATEX, although some of the gain is lost on the fact that one needs to use \LUA\ backend. \stopsection \startsection[title=Input] The input can come from files, token lists, macros and \LUA\ which means many places. When it comes from \LUA\ it can be tokens, nodes, string, and each has its special way of handling and the engine has to keep track of this when it accumulated the input that pops up after a \LUA\ call. This is done as efficient as possible without sacrificing performance. The fact that we have \UTF\ should not have too much impact. \stopsection \startsection[title=Nesting] When you enter a group a stack boundary is set and when some value changes the original value is pushed on the stack. After leaving the group values are restored. The engine tries to avoid redundant actions which improves memory usage and runtime. Every macro expansion, opened file, expanded token list, etc.\ pushes the input stack and that comes with overhead. Again we have tried to minimize the impact and thereby gain a bit over \LUATEX. Other stacks like those used by math, alignment, conditionals, expressions etc.\ have also been improved some. On the other hand, by unweaving some shared code there can be a price to pay, but as with everything usage patterns indicate no penalty here. \stopsection \startsection[title=conditions] We already had more conditionals in \LUATEX\ but again the repertoire of conditionals has been extended. This permits us to remove some middle-layer helpers and stay closer to the core language. It also helps to improve performance. Another important addition has been \type {\orelse} than permits us to write test in a way similar to what other language provide with for instance \type {elseif} or \type {else if}. \stopsection \startsection[title=macros] Expanding macros happens a lot especially in a more complex macro package. This means that adding features in that area can have a large impact on runtime. Nevertheless the argument parser now provides a few handfuls of variants in picking up arguments with out noticeable degradation, especially because these new features can gain performance. At the same time there have been some optimizations in storing macro related states, checking and accessing parameters. There are additional (internal) classes of macros that make for a more natural implementation; for instance \type {\protected} macros are now first class commands. \stopsection \startsection[title=Keywords] Some primitives accept one or more keywords and \LUAMETATEX\ adds some more. In order to deal with this efficiently the keyword scanner has been optimized, where even the context was taken into account. As a result the scanner was quite a bit faster. This kind of optimization was a graduate process the eventually ended up in what we have now. In traditional \TEX\ (and also \LUATEX) the order of keywords is sometimes mixed and sometimes prescribed. In most cases only one occurrence is permitted. So, for instance, this is valid in \LUATEX: \starttyping \hbox attr 123 456 attr 123 456 spread 10cm { } \hrule width 10cm depth 3mm \hskip 3pt plus 2pt minus 1pt \stoptyping The \type {attr} comes before the \type {spread}, rules can have multiple mixed dimension specifiers, and in glue the optional \type {minus} part always comes last. The last two commands are famous for look ahead side effects which is why macro packages will end them with something not keyword, like \type {\relax}, when needed. In \LUAMETATEX\ the following is okay. Watch the few more keywords in box and rule specifications. \starttyping \hbox reverse to 10cm attr 123 456 orientation 4 xoffset 10pt spread 10cm { } \hrule xoffset 10pt width 10cm depth 3mm \hskip 3pt minus 1pt plus 2pt \stoptyping Here the order is not prescribed and, as demonstrated with the box specifier, for instance dimensions (specified by \type {to} or \type {spread} can be overloaded by later settings. In case you wonder if that breaks compatibility: in some way it does but bad or sloppy keyword usage breaks a run anyway. For instance \type {minuscule} results in \type {minus} with no dimension being seen. So, in the end the user should not noticed it and when a user does, the macro package already had an issue that had to be fixed. \stopsection \startsection[title=Directions] The directional model in \LUAMETATEX\ is a simplified version the the model used in \LUATEX. In fact, not much is happening at all: we only register a change in direction. The approach is that we try to make node lists balanced but also try to avoid some side effects. What happens is quite intuitive if we forget about spaces (turned into glue) but even there what happens makes sense if you look at it in detail. However that logic makes in|-|group switching kind of useless when no properly nested grouping is used: switching from right to left several times nested, results in spacing ending up after each other due to nested mirroring. Of course a sane macro package will manage this for the user but here we are discussing the low level injection of directional information. This is what happens: \starttyping \textdirection 1 nur {\textdirection 0 run \textdirection 1 NUR} nur \stoptyping This becomes stepwise: \startnarrower \starttyping injected: [push 1]nur {[push 0]run [push 1]NUR} nur balanced: [push 1]nur {[push 0]run [pop 0][push 1]NUR[pop 1]} nur[pop 0] result : run {RUNrun } run \stoptyping \stopnarrower And this: \starttyping \textdirection 1 nur {nur \textdirection 0 run \textdirection 1 NUR} nur \stoptyping becomes: \startnarrower \starttyping injected: [+TRT]nur {nur [+TLT]run [+TRT]NUR} nur balanced: [+TRT]nur {nur [+TLT]run [-TLT][+TRT]NUR[-TRT]} nur[-TRT] result : run {run RUNrun } run \stoptyping \stopnarrower Now, in the following examples watch where we put the braces: \startbuffer \textdirection 1 nur {{\textdirection 0 run} {\textdirection 1 NUR}} nur \stopbuffer \typebuffer This becomes: \startnarrower \getbuffer \stopnarrower Compare this to: \startbuffer \textdirection 1 nur {{\textdirection 0 run }{\textdirection 1 NUR}} nur \stopbuffer \typebuffer Which renders as: \startnarrower \getbuffer \stopnarrower So how do we deal with the next? \startbuffer \def\ltr{\textdirection 0\relax} \def\rtl{\textdirection 1\relax} run {\rtl nur {\ltr run \rtl NUR \ltr run \rtl NUR} nur} run {\ltr run {\rtl nur \ltr RUN \rtl nur \ltr RUN} run} \stopbuffer \typebuffer It gets typeset as: \startnarrower \startlines \getbuffer \stoplines \stopnarrower We could define the two helpers to look back, pick up a skip, remove it and inject it after the dir node. But that way we loose the subtype information that for some applications can be handy to be kept as|-|is. This is why we now have a variant of \prm {textdirection} which injects the balanced node before the skip. Instead of the previous definition we can use: \startbuffer[def] \def\ltr{\linedirection 0\relax} \def\rtl{\linedirection 1\relax} \stopbuffer \typebuffer[def] and this time: \startbuffer[txt] run {\rtl nur {\ltr run \rtl NUR \ltr run \rtl NUR} nur} run {\ltr run {\rtl nur \ltr RUN \rtl nur \ltr RUN} run} \stopbuffer \typebuffer[txt] comes out as a properly spaced: \startnarrower \startlines \getbuffer[def,txt] \stoplines \stopnarrower Anything more complex that this, like combination of skips and penalties, or kerns, should be handled in the input or macro package because there is no way we can predict the expected behavior. In fact, the \prm {linedirection} is just a convenience extra which could also have been implemented using node list parsing. Directions are complicated by the fact that they often need to work over groups so a separate grouping related stack is used. A side effect is that there can be paragraphs with only a local par node followed by direction synchronization nodes. Paragraphs like that are seen as empty paragraphs and therefore ignored. Because \prm {noindent} doesn't inject anything but a \prm {indent} injects an box, paragraphs with only an indent and directions are handles and paragraphs with content. When indentation is normalized a paragraph with an indentation skip is seen as content. \stopsection \startsection[title=Hooks] {\em todo} \stopsection \startsection[title=Units] The familiar \TEX\ units like \type {pt} and \type {cm} are supported but since the 2021 \CONTEXT\ meeting we also support the Knuthian Potrzebie, cf.\ \typ {en.wikipedia.org/wiki/Potrzebie}. The two character acronym is \type {dk}. One \type {dk} is 6.43985pt. This unit is particularly suited for offsets in framed examples. In 2023 we added the Edith (\type {es}) and Tove (\type {ts}) as metric replacements for the inch (\type {in}). As with the \type {dk} more background information can be found in documents that come with \CONTEXT\ and user group journals. The \type {eu} unit starts out as one \type {es} but can be scaled with \prm {eufactor}. \startbuffer \localcontrolledloop -5 55 5 { \eufactor=\currentloopiterator \dontleavehmode\strut \vrule height .1es depth .25ts width 1dk\relax\quad \vrule height .1es depth .25ts width 1eu\relax\quad \the\currentloopiterator \par } \stopbuffer \typebuffer This example code shows all four new units. Watch how \prm {eufactor} is clipped to a value in the range $1-50$. The default factor of $10$ makes the European Unit equivalent to ten Toves or one Edith. \startpacked \startcolor[darkgray] \getbuffer \stopcolor \stoppacked In addition to these there can be user units but because these are macro package dependent they are not discussed here. \stopsection \startsection[title=Local control] There are a few new primitives that permit what we call local controlled expansion. This permits for instance expanding non expandable macros and even typesetting inside an expansion context like \type {\edef}. Regular \TEX has a main loop to where it returns after every primitive action, but local control let the engine go into a nested main loop. \stopsection \startsection[title=Overload protection] Protection is achieved via prefixes. Depending on the value of the \prm {overloadmode} variable warnings or errors will be triggered. Examples of usage can be found in some documents that come with \CONTEXT, so here we just stick to the basics. \starttyping \mutable \def\foo{...} \immutable\def\foo{...} \permanent\def\foo{...} \frozen \def\foo{...} \aliased \def\foo{...} \stoptyping A \prm {mutable} macro can always be changed contrary to an \prm {immutable} one. For instance a macro that acts as a variable is normally \prm {mutable}, while a constant can best be immutable. It makes sense to define a public core macro as \prm {permanent}. Primitives start out a \prm {permanent} ones but with a primitive property instead. \startbuffer \let\relaxone \relax 1: \meaningfull\relaxone \aliased \let\relaxtwo \relax 2: \meaningfull\relaxtwo \permanent\let\relaxthree\relax 3: \meaningfull\relaxthree \stopbuffer \typebuffer The \prm {meaningfull} primitive is like \prm {meaning} but report the properties too. The \prm {meaningless} companion reports the body of a macro. Anyway, this typesets: \startlines \tttf \getbuffer \stoplines So, the \prm {aliased} prefix copies the properties. Keep in mind that a macro package can redefine primitives, but \prm {relax} is an unlikely candidate. There is an extra prefix \prm {noaligned} that flags a macro as being valid for \prm {noalign} compatible usage (which means that the body must contain that one. The idea is that we then can do this: \starttyping \permanent\protected\noaligned\def\foo{\noalign{...}} % \foo is unexpandable \stoptyping that is: we can have protected macros that don't trigger an error in the parser where there is a look ahead for \prm {noalign} which is why normally protection doesn't work well. So: we have macro flagged as permanent (overload protection), being protected (that is, not expandable by default) and a valid equivalent of the noalign primitive. Of course we can also apply the \prm {global} and \prm {tolerant} prefixes here. The complete repertoire of extra prefixes is: \starttabulate \FL \NC \type {frozen} \NC a macro that has to be redefined in a managed way \NC \NR \NC \type {permanent} \NC a macro that had better not be redefined \NC \NR \NC \type {primitive} \NC a primitive that normally will not be adapted \NC \NR \NC \type {immutable} \NC a macro or quantity that cannot be changed, it is a constant \NC \NR \NC \type {mutable} \NC a macro that can be changed no matter how well protected it is \NC \NR \ML \NC \type {instance} \NC a macro marked as (for instance) be generated by an interface \NC \NR \ML \NC \type {noaligned} \NC the macro becomes acceptable as \type {\noalign} alias \NC \NR \ML \NC \type {overloaded} \NC when permitted the flags will be adapted \NC \NR \NC \type {enforced} \NC all is permitted (but only in zero mode or ini mode) \NC \NR \NC \type {aliased} \NC the macro gets the same flags as the original \NC \NR \ML \NC \type {untraced} \NC the macro gets a different treatment in tracing \NC \NR \LL \stoptabulate The not yet discussed \prm {instance} is just a flag with no special meaning which can be used as classifier. The \prm {frozen} also protects against overload which brings amount of blockers to four. To what extent the engine will complain when a property is changed in a way that violates the flags depends on the parameter \prm {overloadmode}. When this parameter is set to zero no checking takes place. More interesting are values larger than zero. If that is the case, when a control sequence is flagged as mutable, it is always permitted to change. When it is set to immutable one can never change it. The other flags determine the kind of checking done. Currently the following overload values are used: \starttabulate[|l|l|c|c|c|c|c|] \NC \NC \BC immutable \BC permanent \BC primitive \BC frozen \BC instance \NC \NR \NC 1 \NC warning \NC \star \NC \star \NC \star \NC \NC \NC \NR \NC 2 \NC error \NC \star \NC \star \NC \star \NC \NC \NC \NR \NC 3 \NC warning \NC \star \NC \star \NC \star \NC \star \NC \NC \NR \NC 4 \NC error \NC \star \NC \star \NC \star \NC \star \NC \NC \NR \NC 5 \NC warning \NC \star \NC \star \NC \star \NC \star \NC \star \NC \NR \NC 6 \NC error \NC \star \NC \star \NC \star \NC \star \NC \star \NC \NR \stoptabulate The even values (except zero) will abort the run. A value of 255 will freeze this parameter. At level five and above the \prm {instance} flag is also checked but no drastic action takes place. We use this to signal to the user that a specific instance is redefined (of course the definition macros can check for that too). The \prm {overloaded} prefix can be used to overload a frozen macro. The \prm {enforced} is more powerful and forces an overload but that prefix is only effective in ini mode or when it's embedded in the body of a macro or token list at ini time unless of course at runtime the mode is zero. So far for a short explanation. More details can be found in the \CONTEXT\ documentation where we can discuss it in a more relevant perspective. It must be noted that this feature only makes sense a controlled situation, that is: user modules or macros of unpredictable origin will probably suffer from warnings and errors when de mode is set to non zero. In \CONTEXT\ we're okay unless of course users redefine instances but there a warning or error is kind of welcome. There is an extra prefix \prm {untraced} that will suppress the meaning when tracing so that the macro looks more like a primitive. It is still somewhat experimental so what gets displayed might change. The \prm {letfrozen}, \prm {unletfrozen}, \prm {letprotected} and \prm {unletprotected} primitives do as their names advertise. Of course the \prm {overloadmode} must be set so that it is permitted. \stopsection \startsection[title=Tracing] There is are more tracing options, like in math, alignments and inserts, and tracing can be more detailed. This is partly a aide effect of the need for exploring new features. Tracing is not always compatible, if only because there are more possibilities, for instance in the way macros are defined and can handle arguments. \stopsection \stopdocument % \setupbodyfont[pagella] % \starttext % \definefont[TestA][SerifBold*default,oldstyle] % \def\TestB {{\infofont[\the\glyphdatafield]}} % \def\TestC#1{{\infofont(#1)}\csname#1\endcsname} % \bgroup % \crlf abc 123\crlf % \bgroup % \TestA % \TestB % \crlf \strut \kern10pt abc 123 \crlf % \egroup % \TestB % \TestC{smallcaps}% % \TestB % \crlf abc 123 \crlf % \bgroup % \TestA % \TestB % \crlf \strut \kern10pt abc 123 \crlf % \egroup % \TestB % \bgroup % \TestC{oldstyle}% % \TestB % \crlf \strut \kern10pt abc 123 \crlf % \egroup % \TestB % \crlf abc 123 \crlf % \egroup % \stoptext