1
2
3
4
5
6
7
8
9
10
11\startcomponent ontargetmath
12
13\environment ontargetstyle
14
15\startchapter[title={A different approach to math spacing}]
16
17\startsubject[title=Introduction]
18
19The \TEX\ engine is famous for its rendering of math and even after decades there
20is no real contender. And so there also is no real pressure to see if we can do
21better. However, when Mikael Sundqvist ran into a Swedish math rendering
22specification and we started discussing a possible support for that in \CONTEXT,
23it quickly became clear that the way \TEX\ does spacing is a bit less flexible
24than one wishes for. We already have much of what is needed in place but it also
25has to work well with how \TEX\ sees things:
26
27\startitemize[packed,n]
28\startitem
29 Math is made from a sequence of atoms: a quantity with a nucleus, superscript
30 subscript. \footnote {I suddenly realize why in the engine noads have a
31 nucleus field: they are atoms \unknown\ but what does that make super and
32 subscripts.} Atoms are spaced by \type {\thinmuskip}, \type {\medmuskip} and
33 \type {\thickmuskip} or nothing, and that is sort of hard coded.
34\stopitem
35\startitem
36 Atoms are organized by class and there are seven (or eight, depending on how
37 you look at it) of them visible: binary symbols, relations, etc. The
38 invisible ones, composites like fractions and fenced material (we call them
39 molecules) are at some point mapped onto the core set. Molecules like fences
40 have a different class left and right of the fenced material.
41\stopitem
42\startitem
43 In addition the engine itself has all kind of spacing related parameters and
44 these kick in automatically and sometimes have side effects. The same is
45 true for penalties.
46\stopitem
47\stopitemize
48
49The normal approach to spacing other than imposed by the engine is to use
50correction space, like \type {\,} and I think that quite some \TEX\ users think
51that this is how it is supposed to be. The standard way to enter math relates to
52scientific publishing and there the standards are often chiseled in stone so why
53should users tweak anyway. However, in \CONTEXT\ we tend to start from the users
54and not the publishers end so there we can decide to follow different routes.
55Users can always work around something they dont like but we focus on reliable
56input giving predictable output. Also, when reading on, it is good to realize
57that it is all about the user experience here: it should look nice (which then of
58course makes one become aware of issues elsewhere) and we dont care much about
59specific demands of publishers in the scientific field: the fact that they often
60rekey content doesnt go well with users paying attention themselves, let
61alone the fact that nowadays they can demand word processor formats.
62
63The three mentioned steps are fine for the average case but sometimes make no
64sense. It was definitely the best approach given time and resources but when
65\LUATEX\ went \OPENTYPE\ a lot of parameters were added and at that time we
66therefore added spacing by class pair. That not only decoupled the relation
67between the three (configurable) muskip parameters but also made it possible to
68use plenty of them. Now it must be said that for consistency having these three
69skips works great but given the tweaking expected from users consistency is
70not always what comes out.
71
72This situation is very well comparable to the proclaimed qualities of the
73typesetting of text by \TEX. Yes, it can do a great job, and often does, but users
74can mess up quite well. I remember that when we did tests with \HZ\ the outcomes
75were pretty unimpressive. When you give an audience a set of sample renderings,
76where each sample is slightly different and each user gets a randomized subset,
77the sudden lack of being able to compare (and agree) with another \TEX ie makes
78for interesting conclusions. They look for the opposites of what is claimed to be
79perfect. So, two lines with hyphens rate low, even if not doing it would look
80worse. The same for a few short words in the last line of a paragraph. Excessive
81spacing is also seen as bad. So, when asked why some paragraphs looked okay
82noticing (excessive and troublesome) expansion was not seen as a problem; instead
83it were hyphens that got the attraction.
84
85The same is probably true for math: the input with lots of correction spaces or
86commands where characters would do can be horrible but its just the way it is
87supposed to be. The therefore expected output can only be perfect, right,
88independent of how one actually messed up spacing. But personally I think that it
89is often spacing messed up by users that make a \TEX\ document recognizable. It
90compares to word processor results that one can sometimes identify by multiple
91consecutive spaces in the typeset text instead of using a glue model like \TEX.
92Reaching perfection is not always trivial, but fortunately we can also find
93plenty of nice looking documents done with \TEX.
94
95The \TEX book has an excellent and intriguing chapter on the fine points of math
96and it definitely shows why Don Knuth wrote \TEX\ as a tool for his books. He
97pays a lot of attention to detail and that is also why it all works out so well.
98If you need to render from unseen sources (as happens in an \XML\ workflow)
99coming from several authors and have time nor money to check everything, youre
100off worse. And Im not even talking of input where invisible \UNICODE\ spacing
101characters are injected. It is the \TEX\ book(s) that has drawn me to this
102program and believe it or not, in the first project I was involved in that
103demanded typeset (quantum mechanics) math the \IBM\ typewriter with changing
104bulbs ruled the scenery. In fact, our involvement was quickly cut off when we
105dared to show a chapter done in \TEX\ that looked better.
106
107Apart from an occasional tweak, in \CONTEXT\ we never really used this opened up
108math atom pair spacing mechanism available in \LUATEX\ extensively. So, when I
109was pondering how to proceed it stroke me that it would make sense to generalize
110this mechanism. It was already possible (via a mode parameter) to bypass the
111second step mentioned above, but we definitely needed more than the visible
112classes that the engine had. In \CONTEXT\ we already had more classes but those
113were meant for assigning characters and commands to specific math constructs
114(think of fences, fractions and radicals) so in the end they were not really
115classes. Considering this option was made easier by the fact that Mikael would do
116the testing and help configuring the defaults, which all will result in a new
117math user manual.
118
119There are extensions introduced in \LUATEX\ and later \LUAMETATEX\ that are not
120discussed here. In this expose we concentrate on the features that were explored,
121extended and introduced while we worked on updating math support in \LMTX.
122
123\stopsubject
124
125\startsubject[title=An example]
126
127Before we go into details, lets give an example of unnoticed spacing effects. We
128use three simple formulas all using fractions:
129
130\startbuffer[one]
131\ruledhbox{$\frac{x2}{a1}$}
132\stopbuffer
133
134\startbuffer[two]
135\ruledhbox{$x \frac{x2}{a1} = 10$}
136\stopbuffer
137
138\startbuffer[three]
139\ruledhbox{$\frac{1}{2}\frac{1}{2}x$}
140\stopbuffer
141
142\typebuffer[one]
143
144and:
145
146\typebuffer[two]
147
148as well as:
149
150\typebuffer[three]
151
152\startbuffer[all]
153\dontleavehmode
154\scale[scale=4000]{\getbuffer[one]}
155\scale[scale=4000]{\getbuffer[two]}
156\scale[scale=4000]{\getbuffer[three]}
157\stopbuffer
158
159\startlinecorrection
160\nulldelimiterspace=1.2pt\getbuffer[all]
161\stoplinecorrection
162
163If you look closely you see that the fraction has a little space at the left and
164right. Where does that come from? Because we normally dont put a tight frame
165around a fraction, we are not really aware of it. The spacing between what are
166called ordinary, operator, binary, relation and other classes of atoms is
167explained in the \TEX book (or \quotation {\TEX\ by Topic} if you want a summary)
168and basically we have a class by class matrix that is built into \TEX. The
169engine looks at successive items and spacing depends on their (perceived) class.
170Because the number of classes is limited, and because the spacing pairs are hard
171coded, the engine cheats a little. Depending on what came before or comes next
172the class of an atom is adapted to suit the spacing matrix. One can say that a
173\quotation {reading mathematician} is built in. And most of the decisions are
174okay. If needed one can always wrap something in e.g.\ \type {\mathrel} but of
175course that also can interfere with grouping. All this is true for \TEX, \PDFTEX,
176\XETEX\ and \LUATEX, but a bit different in \LUAMETATEX\ as we will see.
177
178The little spacing on both edges of the fraction is a side effect of the way they
179are built internally: fractions are actually a generalized form of \quotation
180{stuff put on top of other stuff} and they can have left andor right
181delimiters: this is driven by primitives that have names like \type {\atop} and
182\type {\atopwithdelims}. The way the components are placed is (especially in the
183case of \OPENTYPE) driven by lots of parameters and I will leave that out of the
184discussion.
185
186When there are no delimiters, a so called \type {\nulldelimiterspace} will be
187injected. That parameter is set to 1.2 points and I have to admit that in
188\CONTEXT\ I never considered letting that one adapt to the body font size, which
189means that, as we default to a 12 point body font, the value there should have
190been 1.44 points: mea culpa. When we set this parameter to zero point, we get
191this:
192
193\startlinecorrection
194\nulldelimiterspace=0pt\getbuffer[all]
195\stoplinecorrection
196
197As intermezzo and moment of contemplation I show some examples of fractions mixed
198into text. When we have the delimiter space set we get this:
199
200\start \nulldelimiterspace=1.2pt \dorecurse{100}{test $\frac{1}{#1}$ } \stop
201
202While with zero it looks like this, quite a different outcome:
203
204\start \nulldelimiterspace=0pt \dorecurse{100}{test $\frac{1}{#1}$ } \stop
205
206A little tracing shows it more clearly:
207
208\start \showboxes \nulldelimiterspace=1.2pt \dorecurse{100}{test $\frac{1}{#1}$ } \stop
209
210You can zoom in and see where it interferes with margin alignment.
211
212\start \showboxes \nulldelimiterspace=0pt \dorecurse{100}{test $\frac{1}{#1}$ } \stop
213
214So, if you ever meet a user who claims perfection and superiority of typesetting,
215check out herhis work which might have inline fractions done the spacy way. It
216might make other visually typesetting claims less trustworthy. And yes, one can
217wonder if margin kerning could help here but as this content is wrapped in boxes it
218is unlikely to work out well (and not worth the effort).
219
220In order to get a better picture of the spacing, two more renderings are shown.
221This time we show the bounding boxes of the characters too (you might need to zoom
222in to see it):
223
224\startlinecorrection
225\showglyphs\nulldelimiterspace=1.2pt\getbuffer[all]
226\stoplinecorrection
227
228Again we also show the zero case
229
230\startlinecorrection
231\showglyphs\nulldelimiterspace=0pt\getbuffer[all]
232\stoplinecorrection
233
234This makes clear why there actually is this extra space around a fraction:
235regular operators have side bearings and thereby have some added space. And when
236we put a fraction in front of a symbol we need that little extra space. Of course
237a proper class pair spacing value could do the job but there is no fraction
238class. The engine cheats by changing the class depending on what follows or came
239before and this is why on the average it looks okay. However, these examples
240demonstrate that there are some assumptions with regard to for instance fonts
241and this is one of the reasons why the more or less official expected \OPENTYPE\
242behavior as dictated by the Cambria font doesnt always work out well for fonts
243that evolved from the ones used in the \TEX\ community. Also imagine how this
244interferes with the fact that traditional \TEX\ fonts and the machinery do magic
245with cheating about width combined with italic correction (all plausible and
246quite clever but somewhat tricky with respect to \OPENTYPE).
247
248Because here we discuss the way \LUAMETATEX\ and \CONTEXT\ deal with this, the
249following examples show a probably unexpected outcome. Again first the nonzero
250case:
251
252\startlinecorrection
253\showglyphs\showmakeup[mathglue]\nulldelimiterspace=1.2pt\getbuffer[all]
254\stoplinecorrection
255
256And here the zero case:
257
258\startlinecorrection
259\showglyphs\showmakeup[mathglue]\nulldelimiterspace=0pt\getbuffer[all]
260\stoplinecorrection
261
262I will not go into details about the way fractions are supported in the engine
263because some extensions are already around for quite a while. The main
264observation here is that in \LUAMETATEX\ we have alternative primitives that
265assume forward scanning, as if the numerator and denominator are arguments. The
266engine also supports skewed (vulgar) fractions natively where numerator and
267denominator are raised and lowered relative to the (often) slash. Many aspects
268of the rendering can be tuned in the so called font goodie files, which is also
269the place where we define the additional font parameters.
270
271\stopsubject
272
273\startsubject[title=Atom spacing]
274
275If you are familiar with traditional \TEX\ you know that there is some built in
276\type {ordbin} spacing. But there is no such pair for a fraction and a relation,
277simply because there is no fraction class. However, in \LUAMETATEX\ there is one,
278and wed better set it up if we zero the margins of a fraction.
279
280It is worth noticing that fractions are sort of special anyway. The official
281syntax is \type {n \over m} and numerator and denominator can be sub formulas.
282This is the one case where the parser sort of has to look back, which is tricky
283because the machinery is a forward looking one. Therefore, in order to get the
284expected styling (or avoid unexpected side effects) one will normally wrap all in
285braces as in: \type {{ {n} \over {m} }} which of course kind defeats the simple
286syntax which probably is supported for \type {1\over2} kind of usage, so a next
287challenge is to make \type {12} come out right. All this means that in practice
288we have wrappers like \type {\frac} which accidentally in \LUAMETATEX\ can be
289defined using forward looking primitives with plenty extra properties driven by
290keywords. It also means that fractions as expected by the engine due to wrapping
291actually can be a different kind of atom, which can have puzzling side effects
292with respect to spacing (because the remapping happens unseen).
293
294Interesting is that adapting \LUAMETATEX\ to a more extensive model was quite
295doable, also because the code base had already been made more configurable. Of
296course it involved quite a bit of tedious editing and throwing out already nice
297and clean code that had taken some effort, but thats the way it is. Of course
298more classes also means that some storage properties had to be adapted within the
299available space but by sacrificing families that was possible. With 64 potential
300classes we now are back to 64 families compared to 7 classes and 256 families in
301\LUATEX\ and 7 classes and 16 families in traditional \TEX.
302
303Also interesting is that the new implementation is actually somewhat simpler and
304therefore the binary is a tad smaller too. But does all that mean that there were
305no pitfalls? Sure there were! It is worth noticing that doing all this reminded
306me of the early days of \LUATEX\ development, where Taco and I exchanged binaries
307and \TEX\ code in a more or less constant way using Skype. For \LUAMETATEX\ we
308used good old mail for files and Mojcas build farm for binaries and Mikael and I
309spent many months exchanging information and testing out alternatives on a daily
310basis: it is in my opinion the only way to do this and its fun too. It has been
311a lot of work but once we got going there was nothing that could stop us. A side
312effect was that there were no updates during this period, which was something
313users noticed.
314
315In the spacing matrix there is \type {inner} and internally theres also some
316care to be taken of \type {vcenter}. The \type {inner} class is actually shared
317with the \type {variable} class which is not so much a real class but more a
318signal to the engine that when an alphabetic or numeric character is included
319it has to come from a specific family: upright family zero or math italic family
320one in traditional speak. But, what if we dont have that setup? Well, then one
321has to make sure that this special class number is not associated (which is no
322big deal). It does mean that when we extend the repertoire of classes we
323cannot use slot seven. Always keep in mind that classes (and thereby signals) get
324assigned to characters (some defaults by the engine, others by the macro
325package). It is why in \CONTEXT\ we use abstract class numbers, just in case the
326engine gets adapted.
327
328We also cannot use slot eight because that one is a signal too: for a possible
329active math character, a feature somewhat complicated by the fact that it should
330not interfere with passing around such active characters in arguments. In math
331mode where we have lots of macros passing around content, this special class works
332around these side effects. We dont need this feature in \CONTEXT\ because
333contrary to other macro packages we dont handle primes, pseudo superscripts
334potentially followed by other super and subscripts by making the \type {} an
335active character and thereby a macro in math mode. This trickery again closely
336relates to preferable input, font properties, and limitations of memory and such
337at the time \TEX\ showed up (much has to fit into 8, 16 or 32 bits, so there is
338not much room for e.g.\ more than 8 classes). Since we started with \MKIV\ the way
339math is dealt with is a bit different than normally done in \TEX\ anyway.
340
341\stopsubject
342
343\startsubject[title=Atom rules]
344
345We can now control the spacing between every atom but unfortunately that is not
346good enough. Therefore, we arrive at yet another feature built into the engine:
347turning classes into other classes depending on neighbors. And this is precisely
348why we have certain classes. Lets quote \quotation {\TEX\ by Topic}: {\em The
349cases \type {*} (in the atom spacing matrix) cannot occur, because a \type {bin}
350object is converted to \type {ord} if it is the first in the list, preceded by
351\type {bin}, \type {op}, \type {open}, \type {punct}, \type {rel}, or followed by
352\type{close}, \type {punct}, or \type {rel}; also, a \type{rel} is converted to
353\type{ord} when it is followed by \type {close} or \type {punct}.}
354
355We can of course keep these hard coded heuristics but can as well make that bit
356of code configurable, which we did. Below is demonstrated how one can set up the
357defaults at the \TEX\ end. We use symbolic names for the classes.
358
359\starttyping
360\setmathatomrule \mathbegincode \mathbinarycode
361 \allmathstyles \mathordinarycode \mathordinarycode
362
363\setmathatomrule \mathbinarycode \mathbinarycode
364 \allmathstyles \mathbinarycode \mathordinarycode
365\setmathatomrule \mathoperatorcode \mathbinarycode
366 \allmathstyles \mathoperatorcode \mathordinarycode
367\setmathatomrule \mathopencode \mathbinarycode
368 \allmathstyles \mathopencode \mathordinarycode
369\setmathatomrule \mathpunctuationcode \mathbinarycode
370 \allmathstyles \mathpunctuationcode \mathordinarycode
371\setmathatomrule \mathrelationcode \mathbinarycode
372 \allmathstyles \mathrelationcode \mathordinarycode
373
374\setmathatomrule \mathbinarycode \mathclosecode
375 \allmathstyles \mathordinarycode \mathclosecode
376\setmathatomrule \mathbinarycode \mathpunctuationcode
377 \allmathstyles \mathordinarycode \mathpunctuationcode
378\setmathatomrule \mathbinarycode \mathrelationcode
379 \allmathstyles \mathordinarycode \mathrelationcode
380
381\setmathatomrule \mathrelationcode \mathclosecode
382 \allmathstyles \mathordinarycode \mathclosecode
383\setmathatomrule \mathrelationcode \mathpunctuationcode
384 \allmathstyles \mathordinarycode \mathpunctuationcode
385\stoptyping
386
387Watch the special class with \type {\mathbegincode}. This is actually class 62 so
388you dont need much fantasy to imagine that class 63 is \type {\mathendcode}, but
389that one is not yet used. In a similar fashion we can initialize the spacing
390itself: \footnote {Constant, engine specific, numbers like these are available in
391tables at the \LUA\ end so we can change them and users can check that.}
392
393\starttyping
394\setmathspacing \mathordcode \mathopcode \allmathstyles \thinmuskip
395\setmathspacing \mathordcode \mathbincode \allsplitstyles \medmuskip
396\setmathspacing \mathordcode \mathrelcode \allsplitstyles \thickmuskip
397\setmathspacing \mathordcode \mathinnercode \allsplitstyles \thinmuskip
398
399\setmathspacing \mathopcode \mathordcode \allmathstyles \thinmuskip
400\setmathspacing \mathopcode \mathopcode \allmathstyles \thinmuskip
401\setmathspacing \mathopcode \mathrelcode \allsplitstyles \thickmuskip
402\setmathspacing \mathopcode \mathinnercode \allsplitstyles \thinmuskip
403
404\setmathspacing \mathbincode \mathordcode \allsplitstyles \medmuskip
405\setmathspacing \mathbincode \mathopcode \allsplitstyles \medmuskip
406\setmathspacing \mathbincode \mathopencode \allsplitstyles \medmuskip
407\setmathspacing \mathbincode \mathinnercode \allsplitstyles \medmuskip
408
409\setmathspacing \mathrelcode \mathordcode \allsplitstyles \thickmuskip
410\setmathspacing \mathrelcode \mathopcode \allsplitstyles \thickmuskip
411\setmathspacing \mathrelcode \mathopencode \allsplitstyles \thickmuskip
412\setmathspacing \mathrelcode \mathinnercode \allsplitstyles \thickmuskip
413
414\setmathspacing \mathclosecode \mathopcode \allmathstyles \thinmuskip
415\setmathspacing \mathclosecode \mathbincode \allsplitstyles \medmuskip
416\setmathspacing \mathclosecode \mathrelcode \allsplitstyles \thickmuskip
417\setmathspacing \mathclosecode \mathinnercode \allsplitstyles \thinmuskip
418
419\setmathspacing \mathpunctcode \mathordcode \allsplitstyles \thinmuskip
420\setmathspacing \mathpunctcode \mathopcode \allsplitstyles \thinmuskip
421\setmathspacing \mathpunctcode \mathrelcode \allsplitstyles \thinmuskip
422\setmathspacing \mathpunctcode \mathopencode \allsplitstyles \thinmuskip
423\setmathspacing \mathpunctcode \mathclosecode \allsplitstyles \thinmuskip
424\setmathspacing \mathpunctcode \mathpunctcode \allsplitstyles \thinmuskip
425\setmathspacing \mathpunctcode \mathinnercode \allsplitstyles \thinmuskip
426
427\setmathspacing \mathinnercode \mathordcode \allsplitstyles \thinmuskip
428\setmathspacing \mathinnercode \mathopcode \allmathstyles \thinmuskip
429\setmathspacing \mathinnercode \mathbincode \allsplitstyles \medmuskip
430\setmathspacing \mathinnercode \mathrelcode \allsplitstyles \thickmuskip
431\setmathspacing \mathinnercode \mathopencode \allsplitstyles \thinmuskip
432\setmathspacing \mathinnercode \mathpunctcode \allsplitstyles \thinmuskip
433\setmathspacing \mathinnercode \mathinnercode \allsplitstyles \thinmuskip
434\stoptyping
435
436And because we have a few more atom classes this also needs to happen:
437
438\starttyping[style=\tt]
439\letmathspacing \mathactivecode \mathordinarycode
440\letmathspacing \mathvariablecode \mathordinarycode
441\letmathspacing \mathovercode \mathordinarycode
442\letmathspacing \mathundercode \mathordinarycode
443\letmathspacing \mathfractioncode \mathordinarycode
444\letmathspacing \mathradicalcode \mathordinarycode
445\letmathspacing \mathmiddlecode \mathopencode
446\letmathspacing \mathaccentcode \mathordinarycode
447
448\letmathatomrule \mathactivecode \mathordinarycode
449\letmathatomrule \mathvariablecode \mathordinarycode
450\letmathatomrule \mathovercode \mathordinarycode
451\letmathatomrule \mathundercode \mathordinarycode
452\letmathatomrule \mathfractioncode \mathordinarycode
453\letmathatomrule \mathradicalcode \mathordinarycode
454\letmathatomrule \mathmiddlecode \mathopencode
455\letmathatomrule \mathaccentcode \mathordinarycode
456\stoptyping
457
458With \type {\resetmathspacing} we get an allzero state but that might become
459more refined in the future. What is not clear from the above is that there is
460also an inheritance mechanism. The three special muskip registers are actually
461shortcuts so that changing the register value is reflected in the spacing. When a
462regular muskip value is (verbose or as register) that value is sort of frozen.
463However, the \type {\inherited} prefix will turn references to registers and
464constants into a delayed value: as with the predefined we now have a more dynamic
465behavior which means that we can for instance use reserved muskip registers as we
466can use the predefined. A bonus is that one can also use regular glue or
467dimensions, just in case one wants the same spacing in all styles (a muskip
468adapts to the size).
469
470When you look at all of the above you might wonder how users are supposed to deal
471with math spacing. The answer is that often they can just assume that \TEX\ does
472the right thing. If something somehow doesnt feel right, looking at solutions by
473others will probably lead a new user to just copy a trick, like injecting a \type
474{\thinmuskip}. But it can be that atoms depend on the already applied (or not)
475spacing, which in turn depends on values in the atom spacing matrix that probably
476only a few users have seen. So, in the end it all boils down to trust in the
477engine and ones eyesight combined with hopefully some consistency in adding
478space directives and often with \TEX\ it is consistency that makes documents look
479right. In \CONTEXT\ we have many more classes even if only a few characters fit in,
480like differential, exponential and imaginary.
481
482\stopsubject
483
484\startsubject[title=Fractions again]
485
486We now return to the fraction molecule. With the mechanisms at our disposal we
487can change the fixed margins to more adaptive ones:
488
489\starttyping
490\inherited\setmathspacing \mathbinarycode \mathfractioncode
491 \allmathstyles \thickermuskip
492\inherited\setmathspacing \mathfractioncode \mathbinarycode
493 \allmathstyles \thickermuskip
494\nulldelimiterspace\zeropoint
495$x \frac{1}{x2} x$
496\stoptyping
497
498Here \typ {\thickermuskip} is defined as \type {7mu plus 5mu} where the stretch
499is the same as a \typ {\thickmuskip} and the width \type {2mu} more. We start out
500with three variants, where the last two have \typ {\nulldelimiterspace} set to
501\type {0pt} and the first one uses the \type {1.2pt}.
502
503\definecolor[tred] [r=1,a=1,t=.5]
504\definecolor[tgreen][g=1,a=1,t=.5]
505\definecolor[tblue] [b=1,a=1,t=.5]
506
507\startlinecorrection
508\scale[scale=4000]\bgroup\vbox\bgroup \forgetall
509 \bgroup \tgreen \nulldelimiterspace=1.2pt $x \frac{1}{x2} x$\egroup\par
510 \bgroup \tred \nulldelimiterspace=0.0pt $x \frac{1}{x2} x$\egroup\par
511 \bgroup \tblue \nulldelimiterspace=0.0pt $x \frac{1}{x2} x$\egroup\par
512\egroup\egroup
513\stoplinecorrection
514
515When we now apply the new settings to the last one, and overlay them we get the
516following output: the first and last case are rather similar which is why this
517effort was started in the first place.
518
519\startlinecorrection
520\scale[scale=4000]\bgroup
521 \startoverlay
522 \bgroup
523 \inherited\setmathspacing \mathbinarycode \mathfractioncode \allmathstyles \thickmuskip
524 \inherited\setmathspacing \mathfractioncode \mathbinarycode \allmathstyles \thickmuskip
525 \tgreen \nulldelimiterspace=1.2pt $x \frac{1}{x2} x$
526 \egroup
527 \bgroup
528 \inherited\setmathspacing \mathbinarycode \mathfractioncode \allmathstyles \thickmuskip
529 \inherited\setmathspacing \mathfractioncode \mathbinarycode \allmathstyles \thickmuskip
530 \tred \nulldelimiterspace=0.0pt $x \frac{1}{x2} x$
531 \egroup
532 \bgroup
533 \inherited\setmathspacing \mathbinarycode \mathfractioncode \allmathstyles \thickermuskip
534 \inherited\setmathspacing \mathfractioncode \mathbinarycode \allmathstyles \thickermuskip
535 \tblue \nulldelimiterspace=0.0pt $x \frac{1}{x2} x$
536 \egroup
537 \stopoverlay
538\egroup
539\stoplinecorrection
540
541Of course these changes are not upward compatible but as they are tiny they are
542not that likely to change the number of lines in a paragraph. In display mode
543changes in horizontal dimensions also have little effect.
544
545\startsubject[title=Penalties]
546
547An inline formula can be broken across lines, and for sure there are places where
548you dont want to break or prefer to break. In \TEX\ line breaks can be
549influenced by using penalties. At the outer level of an inline math formula, we
550can have a specific penalty before and after a binary andor relation. The
551defaults are such that there are no penalties set, but most macro packages set
552the so called \type {\relpenalty} and \type {\binoppenalty} (the \type {op} in
553this name does not relate to the operator class) so a value between zero and
5541000. In \LUATEX\ we also have \type{\pre} variants of these, so we have four
555penalties that can be set, but that is not enough in our new approach.
556
557These penalties are class bound and dont relate to styles, like atom spacing
558does. That means that while atom spacing involves $64 \times 64 \times 8$
559potential values, an amount that we can manage by using the discussed
560inheritance. The inheritance takes less values because which store 4 style values
561per class in one number. For penalties we only need to keep $64 \times 2$ in
562mind, plus a range of inheritance numbers. Therefore it was decided to also
563generalize penalties so that each class can have them. The magic commands are
564shown with some useless examples:
565
566\starttyping
567\letmathparent \mathdigitcode
568 \mathbincode
569 \mathbincode
570 \mathdigitcode
571 \mathdigitcode
572\stoptyping
573
574By default the penalties are on their own, like:
575
576\starttyping
577\letmathparent \mathdigitcode
578 \mathdigitcode
579 \mathdigitcode
580 \mathdigitcode
581 \mathdigitcode
582\stoptyping
583
584The options and reserved parent mapping are not (yet) discussed here. Unless
585values are assigned they are ignored.
586
587\starttyping
588\setmathprepenalty \mathordcode 100
589\setmathpostpenalty \mathordcode 600
590\setmathprepenalty \mathbincode 200
591\setmathpostpenalty \mathbincode 700
592\setmathprepenalty \mathrelcode 300
593\setmathpostpenalty \mathrelcode 800
594\stoptyping
595
596As with spacing, when there is no known value, the parent will be consulted. An unset
597penalty has a value of 10000.
598
599After discussing the implications of inline math crossing lines, Mikael and I
600decided there can be two solutions. Both can of course be implemented in \LUA,
601but on the other hand, they make good extensions, also because it sort of
602standardized it. The first advanced control feature tweaks penalties:
603
604\starttyping
605\mathforwardpenalties 2 200 100
606\mathbackwardpenalties 2 100 50
607\stoptyping
608
609This will add 200 and 100 to the first two math related penalties, and 100 and 50
610to the last two (watch out: the 100 will be assigned to the last one found, the
61150 to the one before it). As with all things penalty and line break related, you
612need to have some awareness of how nonlinear the badness calculation is as
613well of the fact that the tolerance and stretch related parameters play a role
614here.
615
616The second tweak is setting \type {\maththreshold} to some value. When set to for
617instance \type {40pt}, formulas that take less space than this will be wrapped in
618a \type {\hbox} and thereby will never break across a page. \footnote {A future
619version might inject severe penalties instead, time will learn.} Actually that
620second tweak has a variant so we have three tweaks! Say that we have this sample
621formula wrapped in some bogus text and repeat that snippet a lot of times:
622
623\startbuffer[demo]
624x xx xxx xxxx $1 x$ x xx xxx xxxx
625\stopbuffer
626
627\typebuffer[demo]
628
629Now look at the example on the next page. You will notice that the red and blue
630text have different line breaks. This is because we have given the threshold some
631stretch and shrink. The red text has a zero threshold so it doesnt do any magic
632at all, while the second has this setup:
633
634\startbuffer
635\setupmathematics[threshold=medium]
636\stopbuffer
637
638\typebuffer
639
640That setting set the threshold to \typ {4em plus 0.75em minus 0.50em} and when
641the formula size exceeds the four quads the line break code will use the real
642formula width but with the given stretch and shrink. Eventually the calculated
643size will be used to repackage the formula. In the future we will also provide a
644way to define slack more relative to the size andor number of atoms.
645
646\startpostponing
647\startlinecorrection \small
648\startoverlay
649 {\setupmathematics[threshold=none]
650 \vbox\bgroup
651 \darkred
652 \dorecurse {40} {\getbuffer[demo]}
653 \egroup}
654 {\setupmathematics[threshold=medium]
655 \vbox\bgroup
656 \darkblue
657 \dorecurse {40} {\getbuffer[demo]}
658 \egroup}
659\stopoverlay
660\stoplinecorrection
661\stoppostponing
662
663Another way to influence line breaks is to use the two inline math related
664penalties that have been added at Mikaels suggestion:
665
666\startbuffer \setupalign[verytolerant]
667{\dorecurse{25}{test $\darkred #1{#1} x{#1}{#1}$ test }\blank}
668{\preinlinepenalty 500 \postinlinepenalty 500
669 \dorecurse{25}{test $\darkgreen #1{#1} x{#1}{#1}$ test }\blank}
670{\postinlinepenalty 500 \preinlinepenalty 500
671 \dorecurse{25}{test $\darkblue #1{#1} x{#1}{#1}$ test }\blank}
672\stopbuffer
673
674\typebuffer
675
676To get an example that shows the effect takes a bit of trial and error because
677\TEX\ does a very good job in line breaking. This is why weve set the tolerance
678and also use negative penalties.
679
680In addition to the \type {\mathsurround} (kern) and \type {\mathsurroundskip}
681(glue) parameters this is a property of the nodes that mark the beginning and end
682of an inline math formula.
683
684\blank \getbuffer \blank
685
686\stopsubject
687
688\startsubject[title=Flattening]
689
690The traditional engine has some code for flattening math constructs that in the
691end are just one character. So in the end, \type {\tilde {u}} and \type {\tilde
692{uu}} become different objects even if both are in fact accents. In fact, when an
693accent is constructed there is a special code path for single characters so that
694script placement adapts to the shape of that character.
695
696However because of interaction with primes, which themselves are sort of
697superscripts and due to the somewhat weird way fonts provide them when it comes
698to positioning and sizes, in \CONTEXT\ we already are fooling around a bit with
699these characters. For understandable reasons of memory usage, complexity and
700eightbitness primes are not a native \TEX\ thing but more something that is
701handled at the macro level (although not in \MKIV\ and \LMTX).
702
703In the end it was script placements on (widely) accented math characters that
704made us introduce a dedicated \type {\Umathprime} primitive that adds a prime to
705a math atom. It permits an uninterupted treatment of scripts while in the final
706assembly of the molecule the prime, superscript, subscript and maybe even
707prescripts that prime gets squeezed in. Because the concept of primes is missing
708in \OPENTYPE\ math an additional font parameter \type {PrimeTopRaisePercent} has
709been introduced as well as an \type {\Umathprimeraise} primitive. In retrospect I
710should have done that earlier but one tends to stick to the original as much as
711possible. However, at some point Mikael and I reached a state where we decided
712that proper (clean) engine extensions make way more sense than struggling with
713border cases and explaining users why things are so complicated.
714
715The input \type {$ X \Uprimescript{} 2 3 $} gives this:
716
717\startlinecorrection
718\startcombination[nx=4,distance=3em,inbetween={\blank[3*medium]}]
719 {\switchtobodyfont [modern]\removeunwantedspaces\scale[s=5]{\strut$\showboxes\showglyphs X\Uprimescript{}23$}} {Latin Modern}
720 {\switchtobodyfont[cambria]\removeunwantedspaces\scale[s=5]{\strut$\showboxes\showglyphs X\Uprimescript{}23$}} {Cambria}
721 {\switchtobodyfont[pagella]\removeunwantedspaces\scale[s=5]{\strut$\showboxes\showglyphs X\Uprimescript{}23$}} {Pagella}
722 {\switchtobodyfont [dejavu]\removeunwantedspaces\scale[s=5]{\strut$\showboxes\showglyphs X\Uprimescript{}23$}} {Dejavu}
723\stopcombination
724\stoplinecorrection
725
726With \type {\tracingmath = 1} this nicely traces as:
727
728\starttyping
729> \inlinemath=
730\noad[ord][...]
731.\nucleus
732..\mathchar[ord] family "0, character "58
733.\superscript
734..\mathchar[dig] family "0, character "32
735.\subscript
736..\mathchar[dig] family "0, character "32
737.\primescript
738..\mathchar[ord] family "0, character "27
739\stoptyping
740
741Of course this feature can also be used for other prime like ornaments and who
742knows how it will evolve over time.
743
744You can influence the positioning with \type {\Umathprimesupshift} which adds
745some kern between a prime and superscript. The \type {\Umathextraprimeshift}
746moves a prime up. The \type {\Umathprimeraise} is a font parameter that defaults
747to 25 which means a raise of 25\percent of the height. These are all (still)
748experimental parameters.
749
750\stopsubject
751
752\startsubject[title=Fences]
753
754Fences can be good for headaches. Because the math that I (or actually my
755colleague) deal with is mostly school math encoded in presentation \MATHML\ (sort
756or predictable) or some form of sequential \ASCII\ based input (often rather
757messy and therefore unpredictable due to ambiguity) fences are a pain. A \TEX ie
758can make sure that left and right fences are matched. A \TEX ie also knows when
759something is an inline parenthesis or when a more high level structure is needed,
760for instance when parentheses have to scale with what they wrap. In that case the
761\type {\left} and \type {\right} mechanism is used. In arbitrary input missing
762one of those is fatal. Therefore, handling of fences in \CONTEXT\ is one of the
763more complex sub mechanisms: we not only need to scale when needed, but also
764catch asymmetrical usage.
765
766A side effect of the encapsulating fencing construct is that it wraps the content
767in a so called inner (as in \type {\mathinner}) which means that we get a box,
768and it is a well known property of boxes that they dont break across lines. With
769respect to fences, a way out is to not really fence content, but do something
770like this:
771
772\starttyping
773\left(\strut\right. x 1 \left.\strut\right)
774\stoptyping
775
776and hope for the best. Both pairs are coupled in the sense that their sizes will
777match and the strut is what determines the size. So, as long as there is a proper
778match of struts all is well, but it is definitely a decent hack. The drawback is
779in the size of the strut: if a formula needs a higher one, larger struts have to
780be used. This is why in plain \TEX\ we have these commands:
781
782\starttyping[style=\small\tt]
783\def\bigl {\mathopen \big } \def\bigm {\mathrel\big } \def\bigr {\mathclose\big }
784\def\Bigl {\mathopen \Big } \def\Bigm {\mathrel\Big } \def\Bigr {\mathclose\Big }
785\def\biggl{\mathopen \bigg} \def\biggm{\mathrel\bigg} \def\biggr{\mathclose\bigg}
786\def\Biggl{\mathopen \Bigg} \def\Biggm{\mathrel\Bigg} \def\Biggr{\mathclose\Bigg}
787
788\def\big #1{{\hbox{$\left#1\vbox to 8.5pt{}\right.\nomathspacing$}}}
789\def\Big #1{{\hbox{$\left#1\vbox to 11.5pt{}\right.\nomathspacing$}}}
790\def\bigg#1{{\hbox{$\left#1\vbox to 14.5pt{}\right.\nomathspacing$}}}
791\def\Bigg#1{{\hbox{$\left#1\vbox to 17.5pt{}\right.\nomathspacing$}}}
792
793\def\nomathspacing{\nulldelimiterspace0pt\mathsurround0pt}
794\stoptyping
795
796The middle is kind of interesting because it has relation properties, while the
797\type {\middle} introduced in \ETEX\ got open properties, but we leave that
798aside.
799
800In \CONTEXT\ we have plenty of alternatives, including these commands, but they
801are defined differently. For instance they adapt to the font size. The hard coded
802point sizes in the plain \TEX\ code relates to the font and steps available in
803there (either by next larger or by extensible). The values thereby need to be
804adapted to the chosen body font as well as the body font size. In \MKIV\ and even
805better in \LMTX\ we can actually consult the font and get more specific sizes.
806
807But, this section is not about how to get these fixed sizes. Actually, the need
808to choose explicitly is not what we want, especially because \TEX\ can size
809delimiters so well. So, take this code snippet:
810
811\startbuffer
812$ x = \left( \dorecurse{40}{\frac{x}{x#1} } x \right) $
813\stopbuffer
814
815\typebuffer
816
817
818
819
820
821
822
823When we typeset this inline, as in \inlinebuffer, we get nicely scaled fences but
824in a way that permits line breaks. The reason is that the engine has been
825extended with a \type {fenced} class so that we can recognize later on,
826when \TEX\ comes to injecting spaces and penalties, that we need to unpack the
827construct. It is another beneficial side effect of the generalization.
828
829The Plain \TEX\ code can be used to illustrate some of what we discussed before
830about fractions. In the next code we use excessive delimiter spacing:
831
832\starttyping
833\def\Bigg#1{
834 {
835 \hbox {
836 $\normalleft#1\vbox to 17.5pt{}\normalright.\nomathspacing$
837 }
838 }
839}
840
841\nulldelimiterspace0pt
842\def\nomathspacing{\nulldelimiterspace0pt\mathsurround0pt}
843
844$\Bigg( 1 x\Bigg) \quad \Bigg( \frac{1}{x}\Bigg)$\par
845
846\nulldelimiterspace10pt
847\def\nomathspacing{\nulldelimiterspace0pt\mathsurround0pt}
848
849$\Bigg( 1 x\Bigg) \quad \Bigg( \frac{1}{x}\Bigg)$\par
850
851\nulldelimiterspace10pt
852\def\nomathspacing{\mathsurround0pt}
853
854$\Bigg( 1 x\Bigg) \quad \Bigg( \frac{1}{x}\Bigg)$\par
855\stoptyping
856
857This renders as follows. We explicitly set \type {\nulldelimiterspace} to values
858because in \CONTEXT\ it is now zero by default.
859
860\startlinecorrection
861 \pushoverloadmode
862 \def\Bigg#1{{\hbox{$\normalleft#1\vbox to 17.5pt{}\normalright.\nomathspacing$}}}
863 \popoverloadmode
864 \startcombination[nx=3,ny=1,distance=3em]
865 {\showboxes
866 \nulldelimiterspace0pt
867 \def\nomathspacing{\nulldelimiterspace0pt\mathsurround0pt}
868 $\Bigg( 1 x\Bigg) \quad \Bigg( \frac{1}{x}\Bigg)$}
869 {\tttf 0pt with reset at end}
870 {\showboxes
871 \nulldelimiterspace10pt
872 \def\nomathspacing{\nulldelimiterspace0pt\mathsurround0pt}
873 $\Bigg( 1 x\Bigg) \quad \Bigg( \frac{1}{x}\Bigg)$}
874 {\tttf 10pt with reset at end}
875 {\showboxes
876 \nulldelimiterspace10pt
877 \def\nomathspacing{\mathsurround0pt}
878 $\Bigg( 1 x\Bigg) \quad \Bigg( \frac{1}{x}\Bigg)$}
879 {\tttf 10pt without reset at end}
880 \stopcombination
881\stoplinecorrection
882
883\stopsubject
884
885\startsubject[title=Radicals]
886
887In traditional \TEX\ a radical with degree is defined as macro. That macro does
888some measurements and typesets the result in four sizes for a choice. The macro
889typesets the degree in a box that contains the degree as formula. There is a less
890guesswork going on than with respect to how the radical symbol is shaped but as
891were talking plain \TEX\ here it works out okay because the default font is well
892known.
893
894Radicals are a nice example of a two dimensional \quote {extender} but only the
895vertical dimension uses the extension mechanism, which itself operates either
896horizontally or vertically, although in principle it could go both ways. The
897horizontal extension is a rule and the fact that the shape is below the baseline
898(as are other large symbols) will make the rule connect well: the radical shape
899sticks out a little, so one can think of the height reflecting the rule height.
900\footnote {When you zoom in you will notice that this is not always optimal
901because of the way the slope touched the rule.} In \OPENTYPE\ fonts there is a
902parameter and in \LUATEX\ we use the default rule thickness for traditional
903fonts, which is correct for Latin Modern. There are more places in the fonts
904where the design relates to this thickness, for instance fraction rules are
905supposed to match the minus, but this is a bit erratic if you compare fonts. This
906is one of the corrections we apply in the goodie files.
907
908In \OPENTYPE\ the specification of the radical also includes spacing properties
909of the degree and that is why we have a primitive in \LUATEX\ that also handles
910the degree. It is what we used in \CONTEXT\ \MKIV. But \unknown\ we actually end
911up with a situation that compares to the already discussed fraction: there is
912space added before a radical when there is a degree. However, because we now have
913a radical atom class, we can avoid using that one and use the new pairwise
914spacing. Some fuzzy spacing logic in the engine could therefore be removed and we
915assume that \type {\Umathradicaldegreebefore} is zero. For the record: the \type
916{\Umathradicaldegreeafter} sort of tells how much space there is above the low
917part of the root, which means that we can compensate for multidigit degrees.
918
919Zeroing a parameter is something that relates to a font which means that it has
920to happen for each math font which in turn can mean a familystyle combination. In
921order to avoid that complication (or better: to avoid tracing clutter) we have a
922way to disable a parameter:
923
924\startbuffer
925\ruledhbox{$x \sqrt[123]{b}12$}
926\ruledhbox{$x \sqrt[12] {b}12$}
927\ruledhbox{$x \sqrt[1] {b}12$}
928\ruledhbox{$x \sqrt {b}12$}
929\stopbuffer
930
931\typebuffer
932
933\startlinecorrection
934\startcombination[1*4]
935 {\switchtobodyfont[modern]\scale[height=10mm]{\showglyphs\setmathignore\Umathradicaldegreebefore0\getbuffer}}{}
936 {\type {\setmathignore \Umathradicaldegreebefore 0}}{}
937 {\switchtobodyfont[modern]\scale[height=10mm]{\showglyphs\setmathignore\Umathradicaldegreebefore1\getbuffer}}{}
938 {\type {\setmathignore \Umathradicaldegreebefore 1}}{Latin Modern}
939\stopcombination
940\stoplinecorrection
941
942One problem with these spacing parameters is that they are inconsistent across
943fonts. The Latin Modern has a rather large space before the degree, while Cambria
944and Pagella have little. That means that when you prototype a mechanism the
945chosen solution can look great but not so much when at some point you use another
946font.
947
948\startlinecorrection
949\startcombination[1*4]
950 {\switchtobodyfont[pagella]\scale[height=10mm]{\showglyphs\setmathignore\Umathradicaldegreebefore0\getbuffer}}{}
951 {\type {\setmathignore \Umathradicaldegreebefore 0}}{}
952 {\switchtobodyfont[pagella]\scale[height=10mm]{\showglyphs\setmathignore\Umathradicaldegreebefore1\getbuffer}}{}
953 {\type {\setmathignore \Umathradicaldegreebefore 1}}{Cambria}
954\stopcombination
955\stoplinecorrection
956
957\stopsubject
958
959\startsubject[title=More fences]
960
961One of the reasons why the \MKII\ and \MKIV\ fence related mechanism is somewhat
962complex is that we want a clean solution for filtering fences like parenthesis by
963size, something that in the traditional happens via a fake fence pair that
964encapsulates a strut of a certain size. In \LMTX\ we use the same approach but
965have made the sequence more configurable. In practice that means that the values
9661 up to 4 are just that but for some fonts we use the sequence \type {1 3 5 7}.
967There was no need to adapt the engine as it already worked quite well.
968
969\stopsubject
970
971\startsubject[title=Integrals]
972
973The Latin Modern fonts have only one size of big operators and one reason can be
974that there is no need for more. Another reason can be that there was just no
975space in the font. However, an \OPENTYPE\ font has plenty slots available and the
976reference font Cambria has integral signs in sizes as well as extensibles.
977
978In \LUATEX\ we already have generic vertical extensibles but that only works well
979with specified sizes. And, cheating with delimiters has the side effect that we
980get the wrong spacing. In \LUAMETATEX\ however we have ways to adapt the size to
981what came or what comes. In fact, it is a mechanism that is available for any atom
982that we support. However, it doesnt play well with script and this whole \type {\limits}
983and \type {\nolimits} is a bit clumsy so Mikael and I decided that different route had
984to be followed. For adaptive large operators we provide this interface:
985
986\startbuffer[sample1]
987$ x \integral [color=darkred,top={t},bottom={b}] {\frac{1}{x}} = 10 $
988\stopbuffer
989
990\startbuffer[sample2]
991$ x \startintegral [color=darkblue,top={t},bottom={b}]
992 \frac{1}{x}
993\stopintegral = 10 $
994\stopbuffer
995
996\startbuffer[sample3]
997$ x \startintegral [color=darkgreen,top={t},bottom={b},method=vertical]
998 \frac{1}{x}
999\stopintegral= 10 $
1000\stopbuffer
1001
1002\typebuffer [sample1,sample2,sample3]
1003
1004This text is not about the user interface so we wont discuss how to define additional
1005large operators using oneliners.
1006
1007\startlinecorrection
1008\startcombination[nx=3,ny=1,distance=2em]
1009 {\scale[s=2]{\getbuffer[sample1]}} {}
1010 {\scale[s=2]{\getbuffer[sample2]}} {}
1011 {\scale[s=2]{\getbuffer[sample3]}} {}
1012\stopcombination
1013\stoplinecorrection
1014
1015The low level \LUAMETATEX\ implementation handles this input:
1016
1017\starttyping
1018\Uoperator \Udelimiter "0 \fam "222B {top} {bottom} {...}
1019\Uoperator limits \Udelimiter "0 \fam "222B {top} {bottom} {...}
1020\Uoperator nolimits \Udelimiter "0 \fam "222B {top} {bottom} {...}
1021\stoptyping
1022
1023plus the usual keywords that fenced accept, because after all, this is just a
1024special case of fencing.
1025
1026Currently these special left operators are implemented as a special case of
1027fences because that mechanism does the scaling. It means that we need a (bogus)
1028right fence, or need to brace the content (basically create an atom). When no
1029right fence is found one is added automatically. Because there is no real
1030fencing, right fences are removed when processing takes place. When you specify a
1031\type {class} that one will be used for the left and right spacing, otherwise we
1032have openclose spacing.
1033
1034\stopsubject
1035
1036\startsubject[title=Going details]
1037
1038When the next feature was explored Mikael tagged it as math micro typography and
1039the reason is that you need not only to set up the engine for it but also need to
1040be aware of this kind of spacing. Because we wanted to get rid of this script spacing
1041that the font imposes we configured \CONTEXT\ with:
1042
1043\starttyping
1044\setmathignore\Umathspacebeforescript\plusone
1045\setmathignore\Umathspaceafterscript \plusone
1046\stoptyping
1047
1048This basically nils all these tiny spaces. But the latest configuration has this
1049instead:
1050
1051\starttyping
1052
1053
1054
1055\mathslackmode \plusone
1056
1057\setmathoptions\mathopcode \plusthree
1058\setmathoptions\mathbinarycode \plusthree
1059\setmathoptions\mathrelationcode\plusthree
1060\setmathoptions\mathopencode \plusthree
1061\setmathoptions\mathclosecode \plusthree
1062\setmathoptions\mathpunctcode \plusthree
1063\stoptyping
1064
1065This tells the engine to convert these spaces into what we call slack: disposable
1066kerns at the edges. But it also converts these kerns into a glue component when
1067possible. As with all these extensions it complicates the machinery but users
1068will never see that. Now, the last six lines do the magic that made us return to
1069honoring the spaces: we can tell the engine to ignore this slack when there are
1070specific classes at the edges. These options are a bitset and \type {1} means
1071\quotation {no slack left} and \type {2} means \quotation {no slack right} so
1072\type {3} sets both.
1073
1074\startbuffer
1075\def\TestSlack#1
1076 {\vbox\bgroup
1077 \mathslackmode\zerocount
1078 \hbox\bgroup
1079 \setmathignore\Umathspacebeforescript\zerocount
1080 \setmathignore\Umathspaceafterscript \zerocount
1081 #1
1082 \egroup
1083 \vskip.9\lineheight
1084 \hbox\bgroup\red
1085 \setmathignore\Umathspacebeforescript\plusone
1086 \setmathignore\Umathspaceafterscript \plusone
1087 #1
1088 \egroup
1089 \egroup}
1090
1091\startcombination[nx=3]
1092 {\showglyphs\TestSlack{$f2 > $}} {}
1093 {\showglyphs\TestSlack{$ > f2$}} {}
1094 {\showglyphs\TestSlack{$f2 > f2$}} {}
1095\stopcombination
1096\stopbuffer
1097
1098\typebuffer
1099
1100\startlinecorrection
1101\scale[width=\textwidth]{\getbuffer}
1102\stoplinecorrection
1103
1104Because this overall removal of slack is not granular enough a while later we
1105introduced a way to set this per class, as is demonstrated in the following
1106example.
1107
1108\startbuffer
1109\def\TestSlack#1
1110 {\vbox\bgroup
1111 \mathslackmode\plusone
1112 \hbox\bgroup\red
1113 \setmathignore\Umathspacebeforescript\zerocount
1114 \setmathignore\Umathspaceafterscript \zerocount
1115 #1
1116 \egroup
1117 \vskip.9\lineheight
1118 \hbox\bgroup\green
1119 \setmathoptions\mathrelationcode \zerocount
1120 #1
1121 \egroup
1122 \vskip.9\lineheight
1123 \hbox\bgroup\blue
1124 \setmathoptions\mathrelationcode \plusthree
1125 #1
1126 \egroup
1127 \egroup}
1128
1129\startcombination[nx=3]
1130 {\showglyphs\TestSlack{$f2 > $}} {}
1131 {\showglyphs\TestSlack{$ > f2$}} {}
1132 {\showglyphs\TestSlack{$f2 > f2$}} {}
1133\stopcombination
1134\stopbuffer
1135
1136\typebuffer
1137
1138\startlinecorrection
1139\scale[width=\textwidth]{\getbuffer}
1140\stoplinecorrection
1141
1142Of course we need to experiment a lot with real documents and it might take a
1143while before all this is stable (in the engine and in \CONTEXT). And as we dont
1144need to conform to the decades old golden \TEX\ math standards we have some
1145degrees of freedom in this: for Mikael and me it is pretty much a visual thing
1146where we look closely at large samples. Of course in practice details get lost
1147when we print at 10 point but that doesnt mean we cant provide the best
1148experience. \footnote {Whenever I look at (my) old (math) school books I realize
1149that Don Knuth had very good reasons to come up with \TEX\ and, it being hard to
1150beat, \TEX\ still sets the standard!}
1151
1152As we mention class specific options, we also need to mention the special case
1153where we have for instance simple formulas like single atoms (for instance
1154digits) are preceded by a sign (binary). These special spacing cases are handled
1155by a lookahead flag that can be set \typ {\setmathoptions <class>}, like the
1156slack flags. More options might become available in due time. When set the
1157lookahead will check for the automatically injected end class atom and use that
1158for spacing when found. The mentioned lookahead is one of the hard coded
1159heuristics in the traditional engine but here we need to explicitly configure it.
1160
1161\stopsubject
1162
1163\startsubject[title=Ghosts]
1164
1165As plain \TEX\ has macros like \type {\vphantom} you also find them in macro
1166packages that came later. These create a boxes that have their content removed
1167after the dimensions are set. They take space and are invisible but theres also
1168nothing there.
1169
1170A variant in the upgraded math machinery are ghosts: these are visible in the
1171sense that they show up but ignored when it comes to spacing. Here is an example.
1172The option bit set here tells the engine that we ghost at the right, so we have
1173ghosts around the relation (it controls where the spacing ends up).
1174
1175\startbuffer
1176$
1177 x
1178 \mathatom class \mathghostcode {!!}
1179 >
1180 \mathatom class \mathghostcode options "00000020 {!!}
1181 1
1182 \quad
1183 x
1184 \mathatom class \mathghostcode {\hbox{\smallinfofont ord}}
1185 >
1186 \mathatom class \mathghostcode options "00000020 {\hbox{\smallinfofont dig}}
1187 1
1188$
1189\stopbuffer
1190
1191\typebuffer
1192
1193You never know when this comes in handy but it fits in the new, more granular
1194approach to spacing. The code above shows that its just a class, this time with
1195number \number\mathghostcode.
1196
1197\startlinecorrection
1198 \scale[width=\textwidth]{\showglyphs\showmakeup[mathglue]\getbuffer}
1199\stoplinecorrection
1200
1201\stopsubject
1202
1203\startsubject[title=Struts]
1204
1205In order to get consistent spacing the \CONTEXT\ macro package makes extensive
1206use of struts in text mode as well as math mode. The normal way to implement that is
1207either an empty box or a zero width rule, both with a specifically set height
1208and depth. In \CONTEXT\ \MKII\ and \MKIV\ (and for a long time in \LMTX\ too) they
1209were rules so that we could visualize them: they get some width and kerns
1210around them to compensate for that.
1211
1212In order to not let them interfere with spacing we could wrap them into a ghost
1213atom but it is kind of ugly. Anyway, before we had these ghost atoms an
1214alternative to struts was already implemented: a special kind of rule. The reason is
1215that I wanted a cleaner and more predictable way to adapt struts to the math style
1216uses and sometimes predicting that is fragile. What we want is a delayed assignment of
1217dimensions.
1218
1219We have two solutions. The first one uses two math parameters that themselves
1220adapt to the style, as do other parameters: \type {\Umathruleheight} and \type
1221{\Umathruledepth}. The other solution relates a font (or family) and character
1222with the strut rule which is then used as measure for the height and depth. Just
1223for the record: this also works in text mode, which is why a recent \LMTX\ also
1224does use that for struts now. The optional visualization is just part of the
1225regular visualization mechanism in \CONTEXT\ which already had provisions for
1226struts. A side effect of this is that the rule primitives now accept three more
1227keywords: \type {font}, \type {fam} and \type {char}, in addition to the already
1228present traditional ones \type {width}, \type {height} and \type {depth}, the
1229(backend) margin ones \type {left} (or \type {top}) and \type {right} (or \type
1230{bottom}) options, as well as \type {xoffset} and \type {yoffset}). The command
1231that creates a rule with subtype \type {strut} is simply \type {\srule}. Because
1232struts are rather macro package specific I leave it to this.
1233
1234One positive side effect is that we could simplify the \CONTEXT\ fraction mechanism
1235a bit. Over time control over the (font driven) gaps was introduced but that is
1236not really needed because we zero the gaps anyway. There was also a tolerance
1237mechanism which again was not used. However, for skewed fractions we do use the new
1238tolerance mechanism as well as gap control.
1239
1240\stopsubject
1241
1242\startsubject[title=Atoms]
1243
1244Now that we have generic atoms (\type {\mathatom}) another, sometimes confusing aspect
1245of the math parsing can be solved. Take this:
1246
1247\starttyping
1248\def\MyBin{\mathbin{\tt mybin}}
1249$ x \MyBin \MyBin $
1250\stoptyping
1251
1252The parser just doesnt like that which means that one has to use
1253
1254\starttyping
1255\def\MyBin{\mathbin{\tt mybin}}
1256$ x {\MyBin} {\MyBin} $
1257\stoptyping
1258
1259or:
1260
1261\starttyping
1262\def\MyBin{{\mathbin{\tt mybin}}}
1263$ x \MyBin \MyBin $
1264\stoptyping
1265
1266But the later has side effects: it creates a list that can influence spacing. It
1267is for that reason that we do accept atoms where they were not accepted before.
1268Of course that itself can have side effects but at least we dont get an error
1269message. It fits well into the additional (user) classes model. And, given that
1270in \CONTEXT\ the \type {\frac} command is actually wrapped as \type {\mathfrac}
1271the next will work too:
1272
1273\starttyping
1274$ x\frac{1}{2} x{\frac{1}{2}} $
1275\stoptyping
1276
1277but in practice you should probably use the braced version here for clarity.
1278
1279\stopsubject
1280
1281\startsubject[title=The \type {vcenter} primitive]
1282
1283Traditionally this primitive is bound to math but it had already been adapted to also
1284work in text mode. As part of the upgrade of math we can now also pass all the options
1285that normal boxed take and we can also cheat with the axis. Here is an example:
1286
1287\startbuffer
1288\def\TEST{\hbox\bgroup
1289 \darkred \vrule width 2pt height 4pt
1290 \darkgreen \vrule width 10pt depth 2pt
1291\egroup}
1292$
1293 x \mathatom \mathvcentercode {!!!}
1294 \ruledvcenter {\TEST}
1295 \ruledvcenter {\TEST}
1296 \ruledvcenter axis 1 {\TEST}
1297 \ruledvcenter xoffset 2pt yoffset 2pt {\TEST}
1298 \ruledvcenter xoffset 2pt yoffset 2pt {\TEST}
1299 x
1300$
1301\stopbuffer
1302
1303\typebuffer
1304
1305There was already a vcenter class available before we did this:
1306
1307\startlinecorrection
1308 \scale[width=\textwidth]{\showglyphs\showmakeup[mathglue]\getbuffer}
1309\stoplinecorrection
1310
1311\stopsubject
1312
1313\startsubject[title=Text]
1314
1315Sometimes you want text in math, for instance \type {sin} or \type {cos} but
1316text in math is not really text:
1317
1318\startbuffer
1319$ \setmathspacing\mathordinarycode\mathordinarycode\textstyle 10mu fin(x) $
1320\stopbuffer
1321
1322\typebuffer
1323
1324The result demonstrates that what looks like a word actually becomes three
1325math atoms:
1326
1327\startlinecorrection
1328 \scale[width=\textwidth]{\showglyphs\showmakeup[mathglue]\getbuffer}
1329\stoplinecorrection
1330
1331Okay, so how about then wrapping it into a text box:
1332
1333\startbuffer
1334$
1335 \setmathspacing\mathordinarycode\mathordinarycode\textstyle 10mu
1336 fin(x) \quad \hbox{fin}(x)
1337$
1338\stopbuffer
1339
1340\typebuffer
1341
1342Here we get:
1343
1344\startlinecorrection
1345 \scale[width=\textwidth]{\showglyphs\showmakeup[mathglue]\getbuffer}
1346\stoplinecorrection
1347
1348We even get a ligature which might be an indication that were not using a math
1349font which indeed is the case: the box is typeset in the regular text font.
1350
1351\startbuffer
1352\def\Test#1
1353 {\setmathspacing\mathordinarycode\mathordinarycode\textstyle 5mu
1354 $\showglyphs
1355 #1
1356 {\tf fin} \quad
1357 \hbox{fin} \quad
1358 \mathatom class \mathordinarycode textfont {fin}
1359 \mathatom class \mathordinarycode textfont {\tf fin}
1360 \mathatom class \mathordinarycode textfont {\hbox{fin}}
1361 \mathatom class \mathordinarycode mathfont {\hbox{fin}}
1362 $}
1363\stopbuffer
1364
1365\typebuffer \getbuffer
1366
1367When we feed this macro with the \type {\textstyle}, \type {\scriptstyle} and \type
1368{\scriptscriptstyle} we get:
1369
1370\startlinecorrection
1371 \startcombination[1*3]
1372 {\scale[scale=3000]{\Test\textstyle }} {text}
1373 {\scale[scale=3000]{\Test\scriptstyle }} {script}
1374 {\scale[scale=3000]{\Test\scriptscriptstyle}} {scriptscript}
1375 \stopcombination
1376\stoplinecorrection
1377
1378Here you see a new atom option action: \type {textfont} which does as much as
1379setting the font to the current family font and the size to the one used in the
1380style. For the record: you only get ligatures when they are configured and
1381provided by the font (and as math is a script itself it is unlikely to work).
1382\footnote {The existing mechanisms in \CONTEXT\ already dealt with this but it is
1383nevertheless nice to have it as a clean engine feature.}
1384
1385\stopsubject
1386
1387\startsubject[title=Tracing]
1388
1389I wont discuss the tracing features in \CONTEXT\ here but for sure the
1390visualizer helps a lot in figuring out all this. In \LUAMETATEX\ we carry a bit
1391more information with the resulting nodes so we can provide more details, for
1392instance about the applied spacing and penalties. Some is shown in the examples.
1393A more recent tracing feature is this:
1394
1395\starttyping
1396\tracingmath 1
1397\tracingonline 1
1398$
1399 \mathord (
1400 \mathord {(}
1401 \mathord \Udelimiter"4 0 (
1402 \Udelimiter"4 0 (
1403$
1404\stoptyping
1405
1406That gives us on the console (the dots represent detailed attribute info that we
1407omit here):
1408
1409\starttyping
14107:3: > \inlinemath=
14117:3: \noad[ord][...]
14127:3: .\nucleus
14137:3: ..\mathchar[open] family "0, character "28
14147:3: \noad[ord][...]
14157:3: .\nucleus
14167:3: ..\mathlist
14177:3: ...\noad[open][...]
14187:3: ....\nucleus
14197:3: .....\mathchar[open] family "0, character "28
14207:3: \noad[ord][...]
14217:3: .\nucleus
14227:3: ..\mathchar[open] family "0, character "28
14237:3: \noad[open][...]
14247:3: .\nucleus
14257:3: ..\mathchar[open] family "0, character "28
1426\stoptyping
1427
1428A tracing level of 2 will spit out some information about applied spacing and
1429penalties between atoms (when set) and level 3 will show the math list before the
1430first and second pass (a mix of nodes and noads) we well as the result (nodes) plus
1431return some details about rules, spacing and penalties applied.
1432
1433\stopsubject
1434
1435\startsubject[title=Is there more?]
1436
1437The engine already provides the option to circumvent the side effect of a change
1438in a parameter acting sort of global: the last value given is also the one that a
1439second pass starts with. The \type {\frozen} prefix will turn settings into local
1440ones but thats another (already old) topic. There are many such improvements and
1441options not mentioned here but you can find them mentioned and explained in older
1442development stories. A lot has been around for a while but not been applied in
1443\CONTEXT\ yet.
1444
1445When \TEX\ was written one important property (likely related to memory
1446consumption) is that node lists have only forward pointers. That means that the
1447state of preceding material has to be kept track of: there is no going (or
1448looking) back. In \LUATEX\ we have double linked lists so in principle we can try
1449to be more clever but so far I decided not to touch the math machinery in that
1450way. But who knows what comes next.
1451
1452\stopsubject
1453
1454\startsubject[title=Those italics]
1455
1456Right from the start of \LUATEX\ it became clear that the fact that \TEX\ assumes
1457the actual width of glyphs to be incremented by the italic correction that then
1458selectively is removed has been an issue. It made for successive attempts to
1459improve spacing in \CONTEXT\ by providing pseudo features. But, when we moved
1460from assembled \UNICODE\ math fonts to \quote {real} ones that became messy: what
1461trick to apply when and even worse where? In the end there are only a very few
1462shapes that actually are affected in the sense that when we dont deal with them
1463it looks bad. It also happens that one of those shapes is the italic \quote {f},
1464a letter that is used frequently in math. It might even be safe to say that the
1465simple fact that the math italic f has this excessively wrong width and thereby
1466pretty large italic correction is the cause of many problems.
1467
1468In the \LMTX\ approach Mikael and I settled on patching shapes in the so called
1469font goodie files, aka \type {lfg} files and only a handful of entries needed a
1470treatment. This makes a good case for removing the traditional font code path
1471from \LUAMETATEX.
1472
1473\startbuffer
1474 \dontleavehmode {\tt\fontclass}:
1475 \start
1476 \showglyphs \setupinterlinespace[1.2]
1477 \dostepwiserecurse{a}{z}{1}{$\Uchar#112$ }
1478 \dostepwiserecurse{a}{z}{1}{$\bi\Uchar#112$ }
1479 \par
1480 \stop
1481\stopbuffer
1482
1483\start
1484 \switchtobodyfont[modern] \getbuffer\blank
1485 \switchtobodyfont[cambria] \getbuffer\blank
1486 \switchtobodyfont[pagella] \getbuffer\blank
1487 \switchtobodyfont[termes] \getbuffer\blank
1488 \switchtobodyfont[bonum] \getbuffer\blank
1489\stop
1490
1491One of the other very sloped symbol is the integral, although most fonts have
1492them more upright than tex has. Of course there are many variants of these
1493integrals in a math font. Here we also have some font parameters that we can
1494tune, which is what we do.
1495
1496\stopsubject
1497
1498\startsubject[title=Accents]
1499
1500Accents are common in languages other than English and its English that \TEX\
1501was made for. Although the seven bit variant became eight bit handling accents
1502never was sophisticated and one of the main reasons is of course that one could
1503use prebuilt composed characters. The \OPENTYPE\ format brought proper
1504anchoring (aka marks) to font formats and when \LUATEX\ deals with text those
1505kick in. In \OPENTYPE\ math however, anchoring is kind of limited to the top
1506position only. Because the \TEXGYRE\ fonts are based on traditional \TEX\ fonts,
1507their accents have not become better suited.
1508
1509\startbuffer
1510$ \hat{x} \enspace \widehat{x} \enspace \widehat{xx} \enspace \widehat{xxx}
1511 \enspace \hat{f} \enspace \widehat{f} $
1512\stopbuffer
1513
1514\typebuffer
1515
1516When looking at examples you need to be aware of the fact hat fonts can have been
1517adapted in the goodie files. \footnote {Extreme examples can be found for Lucida
1518Bright where we not only have to fix the extensible parts of horizontal braces
1519but also have to provide horizontal brackets.} So, for instance bounding boxes
1520and such can differ from the original. Anyway, the previous code in Cambria looks
1521as follows.
1522
1523\startlinecorrection
1524\scale[height=18mm]{\showglyphs\switchtobodyfont[cambria]\inlinebuffer}
1525\stoplinecorrection
1526
1527With Latin Modern we get:
1528
1529\startlinecorrection
1530\scale[height=18mm]{\showglyphs\switchtobodyfont[modern]\inlinebuffer}
1531\stoplinecorrection
1532
1533And Dejavu comes out as:
1534
1535\startlinecorrection
1536\scale[height=18mm]{\showglyphs\switchtobodyfont[dejavu]\inlinebuffer}
1537\stoplinecorrection
1538
1539As you can see there are some differences. In for instance Latin Modern the shape
1540of the hat and smallest wide hat are different and the first wide one has zero
1541dimensions combined with a negative anchor. When an accented character is
1542followed by a superscript or prime the italic correction of the base kicks in but
1543that cannot be enough to not let this small wide hat overflow into the script. We
1544could compensate for it but then we need to know the dimensions. Of course we can
1545consult the bounding box but it makes no sense to let heuristics enter the
1546machinery here while were in the process generalization. One option is to have
1547two extra parameters that can be used when the width of the accent comes close to
1548the width of the base (we then assume that zero accent width means that it has
1549base width) we add an additional kern. In the end we settled for a (semi automatic)
1550correction option in the goodie files.
1551
1552There are actually three categories of extensible accents to consider: those that
1553resemble the ones used in text (like tildes and hats), those wrapping something
1554(like braces and bracket but also arrows) and rules (that in traditional \TEX\
1555indeed are rules). In \CONTEXT\ we have different interfaces for each of these
1556in order to have a more extensive control. The text related ones are the
1557simplest and closest to what the engine supports out of the box but even there we
1558use tweaked glyphs to get better spacing because (of course) fonts have different
1559and inconsistent spacing in the boundingbox above and below the real shape. This
1560is again some tweak that we moved from being {\em automatic} to being {\em under
1561goodie file control}. But this is all too \CONTEXT\ specific to discuss here in
1562more detail.
1563
1564\stopsubject
1565
1566\startsubject[title=Decision time]
1567
1568After lots of tests Mikael and I came to the conclusion that were facing the
1569following situation. When typesetting math most single characters are italic and
1570we already knew from the start of the \LUATEX\ project that the italics shapes
1571are problematic when it comes to typesetting math. But it looks like even some
1572upright characters can have italic correction: in TexGyreBonum for instance the
1573bold upright \type {f} has italic correction, probably because it then can
1574(somehow) kern with a following \type {i}. It anyhow assumes no italic correction
1575to be applied between these characters.
1576
1577In the end the mixed math font model model got more and more stressed so one
1578decision was to simply assume fonts to be used that are either Cambria like
1579\OPENTYPE, or mostly traditional in metrics, or a hybrid of both. It then made
1580more sense to change the engine control options that we have into ones that
1581simply enable certain code paths, independent of the fact if a font is \OPENTYPE\
1582or not. It then become a bit \quotation {crap in, crap out}, but because we
1583already tweak fonts in the goodie files its quite okay. Some fonts have bad
1584metrics anyway or miss characters and it makes no sense to support abandoned
1585fonts either. Also, when a traditional font is assembled one can set up the
1586engine with different flags and we can deal with it as we wish. In the end it is
1587all up to the macro package to configure things right, which is what we tried to
1588do for months when rooting out all the artifacts that fonts bring. \footnote {In
1589previous versions one could configure this per font but that has been dropped.}
1590
1591That said, the reason why some (fuzzy) mixed model works out okay (also in
1592\LUATEX) is that proper \OPENTYPE\ fonts use staircase kerns instead of italic
1593correction. They also have no ligatures and kerns. We also suspect that not that
1594much attention is paid to the rendering. Its a bit like these \quotation {How
1595many fs do you count in this sentence?} tests where people tend to overlook
1596\type {of}, \type {if} and similar short words. Mathematicians loves \type {f}s
1597but probably also overlook the occasionally weird spacing and kerning.
1598
1599A side effect is that mixing \OPENTYPE\ and traditional fonts is also no longer
1600assumed which in turn made a few (newly introduced) state variables obsolete. Once
1601everything is stable (including extensions discussed before) some further cleanup
1602can happen. Another side effect is that one needs to tell the engine what to apply
1603and where, like this:
1604
1605\starttyping
1606\mathfontcontrol\numexpr \zerocount
1607 \overrulemathcontrolcode
1608 \underrulemathcontrolcode
1609 \fractionrulemathcontrolcode
1610 \radicalrulemathcontrolcode
1611 \accentskewhalfmathcontrolcode
1612 \accentskewapplymathcontrolcode
1613
1614 \applyverticalitalickernmathcontrolcode
1615 \applyordinaryitalickernmathcontrolcode
1616 \staircasekernmathcontrolcode
1617
1618
1619 \applyboxeditalickernmathcontrolcode
1620 \applytextitalickernmathcontrolcode
1621 \checktextitalickernmathcontrolcode
1622
1623 \applyscriptitalickernmathcontrolcode
1624 \italicshapekernmathcontrolcode
1625\relax
1626\stoptyping
1627
1628There might be more control options (also for tracing purposes) and some of the
1629symbolic (\CONTEXT) names might change for the better. As usual it will take some
1630years before all is stable but because most users use the latest greatest version
1631it will be tested well.
1632
1633After this was decided and effective I also decided to drop the mapping from
1634traditional font parameters to the \OPENTYPE\ derives engine ones: we now assume
1635that the latter ones are set. After all, we already did that in \CONTEXT\ for the
1636virtual assemblies that we started out with in the beginning of \LUATEX\ and
1637\MKIV.
1638
1639\stopsubject
1640
1641\startsubject[title=Dirty tricks]
1642
1643Once you start playing with edge cases you also start wondering if some otherwise
1644complex things can be done easier. The next macro brings together a couple of
1645features discussed in previous sections. It also uses two state variables:
1646\type {\lastleftclass} and \type {\lastrightclass} that hold the most recent
1647edge classes.
1648
1649\startbuffer
1650\tolerant\permanent\protected\def\NiceHack[#1]#:#2
1651 {\begingroup
1652 \setmathatomrule
1653 \mathbegincode\mathbincode
1654 \allmathstyles
1655 \mathbegincode\mathbincode
1656 \normalexpanded
1657 {\setbox\scratchbox\hpack
1658 ymove \Umathaxis\Ustyle\mathstyle
1659 \bgroup
1660 \framed
1661 [location=middle,#1]
1662 {$\Ustyle\mathstyle#2$}
1663 \egroup}
1664 \mathatom
1665 class 32
1666 \ifnum\lastleftclass <\zerocount\else leftclass \lastleftclass\fi
1667 \ifnum\lastrightclass<\zerocount\else rightclass \lastrightclass\fi
1668 \bgroup
1669 \box\scratchbox
1670 \egroup
1671 \endgroup}
1672\stopbuffer
1673
1674\typebuffer \getbuffer
1675
1676\startbuffer
1677\def\MyTest#1
1678 {$ x #1 x $\quad
1679 $ x \NiceHack[offset=0pt]{#1} x $\quad
1680 $\displaystyle x #1 x $\quad
1681 $\displaystyle x \NiceHack[offset=0pt]{#1} x $}
1682
1683\scale[scale=2000]{\MyTest{>}} \blank
1684\scale[scale=2000]{\MyTest{}} \blank
1685\scale[scale=2000]{\MyTest{!}} \blank
1686\scale[scale=2000]{\MyTest{\frac{1}{2}}}\blank
1687\scale[scale=2000]{\MyTest{\frac{1}{2}}} \blank
1688\stopbuffer
1689
1690\typebuffer
1691
1692Of course this is not code you immediately come up with after reading this text,
1693also because you need to know a bit of \CONTEXT.
1694
1695\startlinecorrection
1696\showmakeup[mathglue]\getbuffer
1697\stoplinecorrection
1698
1699There are a few control options, like \type {\noatomruling} that can be used to
1700prevent rules being applied to the next atom. We can use these in order to
1701achieve more advanced alignment results, but discussing math alignments would
1702demand many more pages than make sense here.
1703
1704\stopsubject
1705
1706\startsubject[title=Tuned kerning]
1707
1708The \CONTEXT\ distribution has dedicated code for typesetting units that dates
1709back to the mid nineties of the previous century but was (code wise) upgraded
1710from \MKII\ to \MKIV\ which made it end up in the physics name space. There is
1711not much reason to redo that code but when we talk new spacing classes it might
1712make sense at some point to see if we can use less code for spacing by using a
1713\quote {unit} class. When Mikael pointed out that, for instance in Pagella:
1714
1715\startlinecorrection
1716\switchtobodyfont[pagella]\showglyphs
1717\scale[height=1cm]{\setmathoptions\mathdivisioncode\zerocount\m{m3s2}}
1718\stoplinecorrection
1719
1720doesnt space well the obvious answer is: use the units mechanism because this
1721kind of rendering was why it was made in the first place. However, the question
1722is of course, can we do better anyway. The chosen solution uses a combination
1723of class options and tweaked shape kerning:
1724
1725\startlinecorrection
1726\switchtobodyfont[pagella]\showglyphs
1727\scale[height=1cm]{\m{m3s2}}
1728\stoplinecorrection
1729
1730An example of a class setup in \CONTEXT\ is:
1731
1732\starttyping
1733\setmathoptions\mathdivisioncode\numexpr
1734 \nopreslackclassoptioncode \nopostslackclassoptioncode
1735 \lefttopkernclassoptioncode \righttopkernclassoptioncode
1736 \leftbottomkernclassoptioncode \rightbottomkernclassoptioncode
1737\relax
1738\stoptyping
1739
1740and, although we dont go into the details of tweaking here, this is the kind
1741if code you will find in the goodie file:
1742
1743\starttyping
1744{
1745 tweak = "kerns",
1746 list = {
1747 [0x2F] = {
1748 topleft = 0.3,
1749 bottomright = 0.2,
1750 }
1751 }
1752}
1753\stoptyping
1754
1755where the numbers are a percentage of the width. This specification translates in
1756a math staircase kerning recipe.
1757
1758\stopsubject
1759
1760\startsubject[title=More font tweaks]
1761
1762Once you start looking into the details of these fonts you are likely to notice
1763more issues. For instance, in the nice looking Lucida math fonts the relations
1764have inconsistent widths and even shapes. This can partially be corrected by
1765using a stylistic alternate but even that forced us to come up with a mechanism
1766to selectively replace \quote {bad} shapes because there is not that much
1767granularity in the alternates. And once we looked at these alternates we noticed
1768that the definition of of script versus calligraphic is also somewhat fuzzy and
1769font dependent. That made for yet another tweak where we can swap alphabets and
1770let the math machinery choose the expected shape. In \UNICODE\ this is handled by
1771variant selectors which is rather cumbersome. Because these two styles are used
1772mixed in the same document, a proper additional alphabet would have made more
1773sense. As we already support variant selectors it was no big deal to combine that
1774mechanism with a variant selector features over a range of calligraphic or script
1775characters, which indeed is what mathematicians use (Mikael can be very
1776convincing). With this kind of tweaks the engine doesnt really play a role: we
1777always could and did deal with it. Its just that upgrading the engine made us
1778look again at this.
1779
1780\stopsubject
1781
1782\startsubject[title=Normalization]
1783
1784Once we had all these spacing related features upgraded it was time to move to
1785other aspects math typesetting. Most of that is not handled in the engine but at
1786the macro level. Examples of this are making sure that math spacing obeys the
1787rules across alignment cells, breaking long formulas into lines with various
1788alignment schemes. The first is accommodated by using the primitives that set the
1789states at the beginning and end of a formula so that is definitely something that
1790the engine facilitates. The second was already possible in \MKIV\ but is somewhat
1791more transparent now by using tagged boundary nodes.
1792
1793But for this summary we stick to discussing the more low level features and where
1794most of what we discussed here concerns horizontal spacing we also have some
1795vertical magic like the mentioned scaled fences and operators but they sort of
1796behave as expected given the traditional \TEX\ approach. We have some more:
1797
1798\startbuffer
1799\definemathradical[esqrt][sqrt][height=\maxdimen,depth=\maxdimen]
1800\definemathradical[ssqrt][sqrt][height=3ex,depth=2ex]
1801
1802\def\TestSqrt#1
1803 {test $ #1{x} #1{\sin(x)} $ test\quad
1804 test $ #1{x} #1{\sin(x)} #1{\frac{1}{x}} $ test\quad
1805 test $ #1{x} #1{x2} $ test\quad
1806 test $ \left(#1{x} #1{x2} \right) $ test\par}
1807
1808\TestSqrt \sqrt \blank
1809\TestSqrt \esqrt \blank
1810\TestSqrt \ssqrt \blank
1811\stopbuffer
1812
1813\typebuffer \getbuffer
1814
1815In the above example you see that square roots can be made to adapt themselves to
1816other such roots. For this we had to add an additional pass. Originally there are
1817just two passes: a first typesetting pass where the maximum height and depth are
1818collected so that in the second pass the fences can be generated and injected.
1819That second pass also handles the spacing and penalties. In \LUAMETATEX\ we now
1820have (1) radical body typesetting, (2) radical typesetting, (3) atom typesetting
1821with height and depth analysis, (4) fence typesetting, and finally (5) inject
1822spacing, penalties, remove slack, etc.
1823
1824In the examples above we set the height and depth and these are passed by
1825keywords to the radical primitive (most atoms and math structures accept keywords
1826that control rendering). Here the special values \type {\maxdimen} signal that we
1827have to make radicals of equal height and depth.
1828
1829In \MKII\ we had ways to snap formulas so that we got consistent line spacing.
1830For a while I wondered if the engine could help with that but in the end no
1831specific engine features are needed, but is is definitely an area that I keep an
1832eye on because consistent spacing is important. After all one has to draw aline
1833somewhere and we always have the \LUA\ callback mechanism available.
1834
1835\stopsubject
1836
1837\startsubject[title=More goodies]
1838
1839This summary will never be complete because we keep improving the rendering of
1840math. For instance, when Mikael checked some less used math alphabets of Latin
1841Modern and Bonum, as part of the goodie file completion, we were a bit horrified
1842by the weird top accent anchoring, inconsistent dimensions and stale italic
1843correction present in some glyphs. For instance there was a italic correction
1844after an upright blackboard lowercase \quote {f}, the upright digits had somewhat
1845random top accent anchors, and due to the lack of granularity in for instance
1846wide hats, characters that are often seen together got inconsistent wide hats.
1847Also clashing with scripts was possible. All this resulted in yet another bunch
1848of features:
1849
1850\startitemize
1851\startitem
1852 In the goodie files we added efficient options to remove anchors from
1853 alphabets (or individual characters).
1854\stopitem
1855\startitem
1856 In the goodie files we added similar options to remove italic correction.
1857\stopitem
1858\startitem
1859 Characters got a few extra fields: margins that can be used to cheat with
1860 dimensions so that we can get more consistent wide accents.
1861\stopitem
1862\startitem
1863 The engine also got the possibility to compensate for accents when
1864 superscripts need to be anchored (by diminishing the height of accents as
1865 well as via an offsets).
1866\stopitem
1867\stopitemize
1868
1869We expect to add (and use) some more options like this when we run into other
1870persistent issues. For sure there are some already that are not discussed here.
1871Of course one can argue why we spend time on this: in 15 years of \UNICODE\ math
1872usage in the \TEX\ community no one ever bothered about a wide hat over the
1873digit7 and no one wondered about the bad spacing after a lowercase blackboardf,
1874but as we go on we do run into these phenomena and it has become a bit of an
1875obsession to get it all right. \footnote {Of course all this puts the usual
1876bashing of Microsoft Word by users in a different perspective: limited control in
1877the \TEX\ engine, faulty fonts that come with \TEX\ distributions, lack of
1878testing and quality control, and probably the believe that all gets done well
1879automatically plays a role here.}
1880
1881By closely looking at default positioning of accents on top of characters Mikael
1882noticed that the anchor points are actually always in the middle of the topmost
1883left and right points of shapes. It looks like these points are calculates
1884automatically and therefore you end up with an anchor on top of the highest part
1885of the seven in Latin Modern Serif but in the middle of a seven with a flat top.
1886You also get anchors at the top of the vertical line in b, d, and on the sticky
1887bit of the g. It is a good example of being careful with automating font design.
1888In our case, removing most anchors and adding a few later on was the solution.
1889\footnote {In the original fonts and traditional \TEX\ engine a kerning pair
1890between a so called skew char and the character at hand is used.}
1891
1892\stopsubject
1893
1894\startsubject[title=Untold stories]
1895
1896There are of course more features but not all make sense to discuss here. For instance,
1897all hard coded properties are now configurable. Take for instance:
1898
1899\startbuffer
1900\Umathsuperscriptvariant\textstyle 1
1901\Umathsubscriptvariant \textstyle 1
1902
1903$ 123 \quad {\scriptstyle 123} \quad {\displaystyle 123}$
1904\stopbuffer
1905
1906\typebuffer
1907
1908This gives us:
1909
1910\startlinecorrection
1911\scale[scale=3000]{\getbuffer}
1912\stoplinecorrection
1913
1914Here the number refers to one of the build in variants, that themselves are a
1915range of styles. In the next table the narrow variants are cramped:
1916
1917\def\Cramped#1{{\glyphyscale\numexpr6*\glyphyscale10\relax#1}}
1918
1919\starttabulate[clcccccccc]
1920\BC 0 \BC normal \NC
1921 D \NC \Cramped{D} \NC
1922 T \NC \Cramped{T} \NC
1923 S \NC \Cramped{S} \NC
1924 SS \NC \Cramped{SS} \NC
1925\NR
1926\BC 1 \BC cramped \NC
1927 \Cramped{D} \NC \Cramped{D} \NC
1928 \Cramped{T} \NC \Cramped{T} \NC
1929 \Cramped{S} \NC \Cramped{S} \NC
1930 \Cramped{SS} \NC \Cramped{SS} \NC
1931\NR
1932\BC 2 \BC subscript \NC
1933 \Cramped{S} \NC \Cramped{S} \NC
1934 \Cramped{S} \NC \Cramped{S} \NC
1935 \Cramped{SS} \NC \Cramped{SS} \NC
1936 \Cramped{SS} \NC \Cramped{SS} \NC
1937\NR
1938\BC 3 \BC superscript \NC
1939 S \NC S \NC
1940 S \NC S \NC
1941 SS \NC SS \NC
1942 SS \NC SS \NC
1943\NR
1944\BC 4 \BC small \NC
1945 S \NC S \NC
1946 S \NC S \NC
1947 SS \NC SS \NC
1948 SS \NC SS \NC
1949\NR
1950\BC 5 \BC smaller \NC
1951 S \NC \Cramped{S} \NC
1952 S \NC \Cramped{S} \NC
1953 SS \NC \Cramped{SS} \NC
1954 SS \NC \Cramped{SS} \NC
1955\NR
1956\BC 6 \BC numerator \NC
1957 S \NC \Cramped{S} \NC
1958 S \NC \Cramped{S} \NC
1959 SS \NC \Cramped{SS} \NC
1960 SS \NC \Cramped{SS} \NC
1961\NR
1962\BC 7 \BC denominator \NC
1963 \Cramped{S} \NC \Cramped{S} \NC
1964 \Cramped{S} \NC \Cramped{S} \NC
1965 \Cramped{SS} \NC \Cramped{SS} \NC
1966 \Cramped{SS} \NC \Cramped{SS} \NC
1967\NR
1968\BC 8 \BC double (superscript) \NC
1969 S \NC \Cramped{S} \NC
1970 S \NC \Cramped{S} \NC
1971 SS \NC \Cramped{SS} \NC
1972 SS \NC \Cramped{SS} \NC
1973\NR
1974\stoptabulate
1975
1976If you want you can change these values but of course were then basically
1977changing some of logic behind math rendering and for sure Don Knuth had good
1978reasons for these defaults.
1979
1980Another untold story relates to multi scripts. When a double script is seen,
1981\TEX\ injects an ordinary recovery atom, issues an error message, and when told
1982so just continues. In order to always continue \LUAMETATEX\ introduces a mode
1983variable that default to minus one, as negative values will trigger the error.
1984Zero of positive values are interpreted as a class and bypass the error. Here is
1985an example of usage:
1986
1987\startbuffer
1988\mathdoublescriptmode
1989 "\tohexadecimal\mathexperimentalcode
1990 \tohexadecimal\mathexperimentalcode
1991
1992\setmathspacing \mathexperimentalcode \mathexperimentalcode \allmathstyles 20mu
1993\setmathspacing \mathordinarycode \mathexperimentalcode \allmathstyles 20mu
1994
1995$x12345678$
1996\stopbuffer
1997
1998\typebuffer
1999
2000We get this:
2001
2002\startlinecorrection
2003\scale[scale=3000]{\getbuffer}
2004\stoplinecorrection
2005
2006\stopsubject
2007
2008\startsubject[title=Final words]
2009
2010One can argue that all these new features can make a document look better. But
2011you only have to look at what Don Knuth produces himself to see that one always
2012could do a good job with \TEX, although maybe at the cost of some extra spacing
2013directives. It is the fact that \OPENTYPE\ showed up as well as many more math
2014fonts, all with their own (sometimes surprising) special effects, that made us
2015adapt the engine. Of course there are also new possibilities that permit better
2016and more robust macro support. The \TEX book has a chapter on \quotation {the
2017fine points of mathematics typesetting} for a reason.
2018
2019There has never been an excuse to produce bad looking documents. It is all
2020about care. For sure there is a category of users who are forced to use \TEX, so
2021they are excused. There are also those who have no eye for typography and rely on
2022the macro package, so there we can to some extent blame the authors of those
2023packages. And there are of course the sloppy users, those who dont enter a
2024revision loop at all. They could as well use any system that in some way can
2025handle math. One can also wonder in what way massive remote editing as well as
2026collaborative working on documents make things better. It probably becomes less
2027personal. At meetings and platforms \TEX\ users like to bash the alternatives but
2028in the end they are part of the same landscape and when it comes to math they
2029dominate. Maybe there is less to brag about then we like: just do your thing and
2030try to do it as good as possible. Rely on your eyes and pay attention to the
2031details, which is possible because the engine provided the means. The previous
2032text shows a few things to pay attention to.
2033
2034Now that all the basics that have to do with proper dimensions, spacing,
2035penalties and logic are dealt with, we moved on to the more high level
2036constructs. We also havent applied some features in the \CONTEXT\ code base yet
2037and are now experimenting with the more high level constructs. For instance the
2038frequently used math alignment mechanism has been overhauled to support advanced
2039inter atom spacing across rows, and in practice one will now more often not even
2040use this alignment mechanism and use the alignment features in multiline
2041display math, if only because they offer advanced annotation. As a nice side
2042effect some of the mechanism that we use for this (like the improved \type
2043{\vadjust} primitive engine feature) also became somewhat more powerful in
2044regular text mode and well see where that brings us.
2045
2046Given the time we spend on this and given the numerous new features it will take
2047a while before all that got added to the engine will be documented. Of course the
2048usage in \CONTEXT\ also serves as documentation. This is not really a problem
2049because most users will happily rely on the goodie files being okay and
2050maintained, and usage other than \CONTEXT\ is unlikely to use these new features,
2051if only because it will break away from the established long term standards and
2052habits.
2053
2054\stopsubject
2055
2056
2057
2058
2059\stopchapter
2060
2061\stopcomponent
2062
2063
2064
2065
2066
2067
2068
2069
2070
2071
2072
2073
2074
2075
2076
2077
2078
2079
2080
2081
2082
2083
2084
2085
2086
2087
2088
2089
2090
2091
2092
2093
2094
2095
2096
2097
2098
2099
2100
2101
2102
2103
2104
2105
2106
2107
2108
2109
2110
2111
2112
2113
2114
2115
2116
2117
2118
2119
2120
2121
2122
2123
2124
2125
2126
2127
2128
2129
2130
2131
2132
2133
2134
2135
2136
2137
2138
2139
2140
2141
2142
2143
2144
2145
2146
2147
2148
2149
2150
2151
2152
2153
2154
2155
2156
2157
2158
2159
2160
2161
2162
2163
2164
2165
2166
2167
2168
2169
2170
2171
2172
2173
2174
2175
2176
2177
2178
2179
2180 |