Source Browser ?

1% language=us
2
3\environment still-environment
4
5\starttext
6
7\startchapter[title=Math new style: are we better off?]
8
9\startsection[title=Introduction]
10
11In this article I will summarize the state of upgrading math support in \CONTEXT\
12per mid 2013 in the perspective of demand, usability, font development and
13\LUATEX. There will be some examples, but don't consider this a manual: there are
14enough articles in the \type {mkiv}, \type {hybrid} and \type {about} series
15about specific topics; after all, we started with this many years ago. Where
16possible I will draw some conclusions with respect to the engine. Some comments
17might sound like criticism, but you should keep in mind that I wouldn't spend so
18much time on \TEX\ if I would not like it that much. It's just that the
19environment wherein \TEX\ is and can be used is not always as perfect as one
20likes it to be, i.e.\ bad habits and decisions once made can be pretty persistent
21and haunt us forever. I'm not referring to \TEX\ the language and program here,
22but more to its use in scientific publishing: in an early stage standards were
23set and habits were nurtured which meant that to some extent the coding resembles
24the early days of computing and the look and feel got frozen in time, in spite of
25developments in coding and evolving typographic needs. I think that the community
26has missed some opportunities to influence and improve matters which means that
27we're stuck with suboptimal situations and, although they are an improvement,
28\UNICODE\ math and \OPENTYPE\ math have their flaws.
29
30This is not a manual. Some aspects will be explained with examples, others are
31just mentioned. I've written down enough details in the documents that describe
32the history of \LUATEX\ and \MKIV\ and dedicated manuals and repeating myself
33makes not much sense. Even if you think that I talk nonsense, some of the
34examples might set you thinking. This article was written for the \TUG\ 2013
35conference in Japan. Many thanks to Barbara Beeton for proofreading and providing
36feedback.
37
38\stopsection
39
40\startsection[title=Some basic questions]
41
42Is there still a need for a program like \TEX ? Those who typeset math will argue
43that there is. After all, one of the reasons why \TEX\ showed up is typesetting
44math. In this perspective we should ask ourselves a few questions:
45
46\startitemize[packed]
47\startitem Is \TEX\ still the most adequate tool? \stopitem
48\startitem Does it make sense to invest in better machinery? \stopitem
49\startitem Have we learned from the past and improved matters? \stopitem
50\startitem What drives development and choices to be made? \stopitem
51\stopitemize
52
53The first question is not that easy to answer, unless you see proof in the fact
54that \TEX\ is still popular for typesetting a wide range of complex content (with
55critical editions being among the most complex). Indeed the program still
56attracts new users and developers. But we need to be realistic. First of all,
57there is some bias involved: if you have used a tool for many years, it becomes
58the one and only and best tool. But that doesn't necessarily make it the best
59tool for everyone.
60
61In this internet world finding a few thousand fellow users gives the impression
62that there is a wide audience but there can be of course thousandfold more users
63of other systems that don't fall into your scope. This is fine: I always wonder
64why there is not more diversity; for instance, we have only a few operating
65systems to choose from, and in communities around computer languages there is a
66tendency to evangelize (sometimes quite extreme). We should also take into
67account that a small audience can have a large impact so size doesn't matter
68much.
69
70As \TEX\ is still popular among mathematicians, we can assume that it hasn't lost
71its charm yet and often it is their only option. We have a somewhat curious
72situation that scientific publishers still want to receive \TEX\ documents |<|a
73demand that is not much different from organizations demanding \MSWORD\
74documents|>| but at the same time don't care too much about \TEX\ at all. Their
75involvement in user groups has started degrading long ago, compared to their
76profits; they don't invest in development; they are mostly profit driven, i.e.\
77those who submit their articles don't even own their sources any more, etc.\
78
79On the other hand, we have users who make their own books (self|-|publishing) and
80who go, certainly in coding and style, beyond what publishers do: they want to
81use all kinds of fonts (and mixtures), color, nicely integrated graphics, more
82interesting layouts, experiment with alternative presentations. But especially
83for documents that contain math that also brings a price: you have to spend more
84time on thinking about presenting the content and coding of the source. This all
85means that if we look at the user side, alternative input is an option,
86especially if they want to publish on different media. I know that there are
87\CONTEXT\ users who make documents (or articles) with \CONTEXT, using whatever
88coding suits best, and do some conversion when it has to be submitted to a
89journal. Personally I think that the lack of interest of (commercial) publishers,
90and their rather minimal role in development, no longer qualifies them to come up
91with requirements for the input, if only because in the end all gets redone
92anyway (in Far Far Away).
93
94It means that, as long as \TEX\ is feasible, we are relatively free to move on
95and experiment with alternative input. Therefore the other two questions become
96relevant. The \TEX\ engines are adapted to new font technology and a couple of
97math fonts are being developed (funded by the user groups). Although the \TEX\
98community didn't take the lead in math font technology we are catching up. At the
99same time we're investing much time in new tools, but given the fact that much
100math is produced for publishers it doesn't get much exposure. Scientific
101publishing is quite traditional and like other publishing lags behind and
102eventually will disappear in its current form. It could happen that one morning
103we find out that all that \quote {publishers want it this or that way} gets
104replaced by ways of publishing where authors do all themselves. A publisher (or
105his supplier) can keep using a 20-year old \TEX\ ecosystem without problems and
106no one will notice, but users can go on and come up with more modern designs and
107output formats and in that perspective the availability of modern engines and
108fonts is good. I've said it before: for \CONTEXT\ user demand drives development.
109
110In the next sections I will focus on different aspects of math and how we went
111from \MKII\ to \MKIV. I will also discuss some (pending) issues. For each aspect
112I will try to answer the third question: did matters improve and if not, and how
113do we cope with it (in \CONTEXT).
114
115\stopsection
116
117\startsection[title=The math script]
118
119All math starts with symbols and|/|or characters that have some symbolic meaning
120and in \TEX\ speak this can be entered in a rather natural way:
121
122\startbuffer
123$ y = 2x + b $
124\stopbuffer
125
126\typebuffer
127
128In order to let \TEX\ know it's math (the equivalent of) two dollar signs are
129used as triggers. The output of this input is: \inlinebuffer. But not all is that
130simple, for instance if we want to square the x, we need to use a superscript
131signal:
132
133\startbuffer
134$ y = x^2 + ax + b $
135\stopbuffer
136
137\typebuffer
138
139The \type {^} symbol results in a smaller \type {2} raised after the \type {x} as
140in \inlinebuffer. Ok, this \type {^} and its cousin \type {_} are well known
141conventions so we stick to this kind of input.
142
143\startbuffer
144$ y = \sqrt { x^2 + ax + b } $
145\stopbuffer
146
147A next level of complexity introduces special commands, for instance a command
148that will wrap its argument in a square root symbol: \inlinebuffer.
149
150\typebuffer
151
152It is no big deal to avoid the backslash and use this kind of coding:
153
154\startbuffer
155\asciimath { y = sqrt ( x^2 + ax + b ) }
156\stopbuffer
157
158\typebuffer
159
160In fact, we have been supporting scientific calculator input for over a decade in
161projects where relatively simple math had to be typeset. In one of our
162longest|-|running math related projects the input went from \TEX, to content
163\MATHML\ to \OPENMATH\ and via presentation \MATHML\ ended up as a combination of
164some kind of encoding that web browsers can deal with. This brings us to reality:
165it's web technology that drives (and will drive math) coding. Unfortunately
166content driven coding (like content \MATHML) does not seem to be the winner here,
167even if it renders easier and is more robust.
168
169Later I will discuss fences, like parentheses. Take this dummy formula:
170
171\starttyping
172$ (x + 1) / a = (x - 1) / b $
173\stoptyping
174
175In a sequential (inline) rendering this will come out okay. A more display mode
176friendly variant can be:
177
178\starttyping
179$ \frac{x + 1}{a} = \frac{x - 1}{b} $
180\stoptyping
181
182which in pure \TEX\ would have been:
183
184\starttyping
185$ {x + 1} \over {a} = {x - 1} \over {b} $
186\stoptyping
187
188The main difference between these two ways of coding is that in the second
189(plain) variant the parser doesn't know in advance what it is dealing with. There
190are a few cases in \TEX\ where this kind of parsing is needed and it complicates
191not only the parser but also is not too handy at the macro level. This is why the
192\type {\frac} macro is often used instead. In \LUATEX\ we didn't dare to get rid
193of \type {\over} and friends, even if we're sure they are not used that often by
194users.
195
196In inline or in more complex display math, the use of fences is quite normal.
197
198\startbuffer
199$ ( \frac{x + 1}{a}  + 1 )^2  = \frac{x - 1}{b} $
200\stopbuffer
201
202\typebuffer
203
204Here we have a problem. The parentheses don't come out well.
205
206\blank \noindentation \getbuffer \blank
207
208We have to do this:
209
210\startbuffer
211$ \left( \frac{x + 1}{a}  + 1 \right)^2  = \frac{x - 1}{b} $
212\stopbuffer
213
214\typebuffer
215
216in order to get:
217
218\blank \noindentation \getbuffer \blank
219
220Doing that \type{\left}|-|\type{\right} trick automatically is hard, although in
221\MATHML, where we have to interpret operators anyway it is somewhat easier. The
222biggest issue here is that these two directives need to be paired. In \ETEX\ a
223\type {\middle} primitive was added to provide a way to have bars adapt their
224height to the surroundings. Interesting is that where at the character level a
225\type {(} has a math property \type {open} and \type {)} has \type {close}. The
226bar, as we will see later, can also act as separator but this property does not
227exist. Because properties (classes in \TEX\ speak) determine spacing we have a
228problem here. So far we didn't extend the repertoire of properties in \LUATEX\ to
229suit our needs (although in \CONTEXT\ we do have more properties).
230
231If you are a \TEX\ user typesetting math, you can without doubt come up with more
232cases of source coding that have the potential of introducing complexities. But
233you will also have noticed that in most cases \TEX\ does a pretty good job on
234rendering math out of the box. And macro packages can provide additional
235constructs that help to hide the details of fine tuning (because there is a lot
236that {\em can} be fine tuned).
237
238In \TEX\ there are a couple of special cases that we can reconsider in the
239perspective of (for instance) faster machines. Normally a macro cannot have a
240\type {\par} in one of its arguments. By defining them as \type {\long} this
241limitation goes away. This default limitation was handy in times when a run was
242relatively slow and grabbing a whole document source as argument due to a missing
243brace had a price. Nowadays this is no real issue which is why in \LUATEX\ we can
244disable \type {\long} which indeed we do in \CONTEXT. On the agenda is to also
245permit \type {\par} in a math formula, as currently \TEX\ complains loudly.
246Permitting a bit more spacy formula definitions (by using empty lines) would be a
247good thing.
248
249Another catch is that in traditional \TEX\ math characters cannot be used outside
250math. That restriction has been lifted. Of course users need to be aware of the
251fact that a mix of math and text symbols can be visually incompatible.
252
253In the examples we used \type {^} and \type {_} and in math mode these have
254special meanings. Traditionally in text mode they trigger an error message. In
255\CONTEXT\ \MKIV\ we have made these characters regular characters but in math
256mode they still behave as expected. \footnote {In an intermediate version \type
257{\nonknuthmode} and \type {\donknuthmode} controlled this.} In a similar fashion
258the \type {&} is an ampersand and when you enable \type {\asciimode} the dollar
259and percent signs also become regular. \footnote {Double percent signs act as
260comments then which is comparable to comments in some programming languages.} In
261\LUATEX\ we have introduced primitives for all characters (or more precisely:
262catcodes) that \TEX\ uses for special purposes like opening and closing math
263mode, scripts, table alignment, etc.
264
265In projects that involve \XML\ we use \MATHML. In \TEX\ many characters can be
266inserted using commands that are tuned for some purpose. The same character can
267be associated with several commands. In \MATHML\ entities and \UNICODE\
268characters are used instead of commands. Interesting is that whenever we get math
269coded that way, there is a good chance that the coding is inconsistent. Of course
270there are ways in \MATHML\ to make sure that a character gets interpreted in the
271right way. For instance, the \type {mfenced} element drives the process of
272(matching) parenthesis, brackets, etc.\ and a renderer can use this property to
273make sure these symbols stretch vertically when needed. However, using \type {mo}
274in an \type {mrow} for a fence is also an option, but that demands some more
275(fuzzy) analysis. I will not go into details here, but some of the more obscure
276options and flags in \CONTEXT\ relate to overcoming issues with such cases.
277
278I have no experience with how \MSWORD\ handles math input, apart from seeing some
279demos. But I know that there is some input parsing involved that is a mixture
280between \TEX\ and analysis. Just as word processing has driven math font
281technology it might be that at some point users expect more clever processing of
282input. To a large extent \TEX\ users already expect that. Where till now \TEX\
283could inspire the way word processers do math, word processors can inspire \TEX
284ies way of inputting text.
285
286So, we have \MATHML, which, in spite of being structured, is still providing
287users a lot of freedom. Then there are word processors, where mouse clicks and
288interpretation does the job. And of course we have \TEX, with its familiar
289backslashes. Let us consider math, when seen in print, as a script to express the
290math language. And indeed, in \OPENTYPE, math is one of the official scripts
291although one where a rather specific kind of machinery is needed in order to get
292output.
293
294I could show more complex math formulas but no matter what notation is used,
295coding will always be somewhat cumbersome and handywork. Math formula coding and
296typesetting remains a craft in itself and \TEX\ notation will keep its place for
297a while. So, with that aspect settled we can continue to discuss rendering.
298
299% So what drives development? I tend to forget about publishers, who, if \TEX\ is
300% known at all in the organization, outsource anyway, and focus on users. One of
301% these users is me, and we do some work for publishers, but they seldom know or
302% care what tools we use. Users also contribute to development: for instance user
303% groups spend considerable money on font development. Interesting is that given
304% the substantial profits of publishers who indirectly still benefit from this it
305% are the users who invest in the tools. In my opinion this also puts them in
306% charge. And of course, developments with respect to input, output and fonts are a
307% driving force behind engine development. There are some more factors: control, as
308% \TEX\ is a programming language, and joy, as manipulating look and feel can be
309% fun. In the future these two will probably dominate over the others, when
310% typesetting and print become more specialized.
311
312\stopsection
313
314\startsection[title=Alphabets]
315
316I have written about math alphabets before so let's keep it simple here. I think
317we can safely say that most math support mechanisms in macro packages are
318inspired by plain \TEX. In traditional \TEX\ we have fonts with a limited number
319of glyphs and an eight|-|bit engine, so in order to get the thousands of possible
320characters mapped onto glyphs the right one has to be picked from some font. In
321addition to characters that you find in \UNICODE, there are also variants,
322additional sizes and bits and pieces that are used in constructing large
323characters, so in practice a math font is quite large. But it is unlikely that we
324will ever run into a situation where fonts pose limits.
325
326The easiest way is of course a direct mapping: an \quote {a} entered in math mode
327becomes an \quote{$a$} simply because the current font at that time has an italic
328shape in the slot referenced by the character. If we want a bold shape instead,
329we can switch to another font and still input an \quote {a}. The 16 families
330available are normally enough for the alphabets that we need. Because symbols can
331be collected in any font, they are normally accessed by name, like \type {\oplus}
332or $\oplus$.
333
334In \UNICODE\ math the math italic \quote {$a$} has slot \type {U+1D44E} and
335directly entering this character in a \UNICODE\ aware \TEX\ engine also has to
336give that \quote {$a$}. In fact, it is the only official way to get that
337character and the fact that we can enter the traditional \ASCII\ characters and
338get an italic shape is a side effect of the macro package, for instance the way
339it defines math fonts and families. \footnote {Our experience is that even when
340for instance \MATHML\ permits coding of math in \XML, copy editors have no
341problem with abusing regular italic font switches to simulate math. This can
342result is a weird mix of math rendering.}
343
344\definefont[mathdemo][file:texgyrepagellamath*mathematics]
345
346Before we move on, let's stress a limitation in \UNICODE\ with respect to math
347alphabets. It has always been a principle of \UNICODE\ committees to never
348duplicate entries. So, thanks to the availability of some characters in
349traditional (font) encodings, we ended up with some symbols that are used for
350math in the older regions of \UNICODE. As a consequence some alphabets have gaps.
351The only real reason I can come up with for accepting these gaps is that old
352documents using these symbols would be not compatible with gapfull \UNICODE\ math
353but I could argue that a document that uses those old codepoints uses commands
354(and needs some special fonts) to get the other symbols anyway, so it's unlikely
355to be a real math document. On the other hand, once we start using \UNICODE\ math
356we could benefit from gapless alphabets simply because otherwise each application
357would have to deal with the exceptions. One can come up with arguments like
358\quotation {just use this or that library} but that assumes persistence, and also
359forces everyone to use the same approach. In fact, if we hide behind a library we
360could as well have hidden the vectors (alphabets) as well. But as they are
361exposed, the gaps stand out as an anomaly. \footnote {One good reason for not
362having the gaps is that when users cut and paste there is no way to know if \type
363{U+210E} is used as Planck constant or variable of some sort, i.e.\ the not
364existing \type {0x1D455}. There is no official way to tag it as something math,
365and even then, as it has no code point it so has lost it's meaning, contrary to a
366copied $i$.} Let's illustrate this with an example. Say that we load the \TEX
367Gyre Pagella math font and call up a few characters:
368
369\startbuffer
370\definefont[mathdemo][file:texgyrepagellamath*mathematics]
371\mathdemo \char"0211C \char"1D507 \char"1D515
372\stopbuffer
373
374\typebuffer
375
376The \UNICODE\ fraktur math alphabet is continuous but the \quote {MATHEMATICAL
377FRAKTUR CAPITAL R} is missing as we already have the \type {BLACK-LETTER CAPITAL
378R} instead. So, this is why we only see two characters show up. It means that in
379the input we cannot have a \type {U+1D515}.
380
381\blank \start \getbuffer \stop \blank
382
383Of course we can cheat and fill in the gap:
384
385\startbuffer
386\definefontfeature
387  [mymathematics]
388  [mathematics]
389  [mathgaps=yes]
390\stopbuffer
391
392\typebuffer \getbuffer
393
394This feature will help us cheat:
395
396\startbuffer
397\definefont[mathdemo][file:texgyrepagellamath*mymathematics]
398\mathdemo \char"0211C \char"1D507 \char"1D515
399\stopbuffer
400
401\typebuffer
402
403This time we can use the character. I wonder what would happen if the \TEX\
404community would simply state that slot \type {U+1D515} is valid. I bet that math
405related applications would support it, as they also support more obscure
406properties of \TEX\ input encoding.
407
408\blank \start \getbuffer \stop \blank
409
410If you still wonder why I bother about this, here is a practical example. The
411\SCITE\ editor that I use is rather flexible and permits me to implement advanced
412lexers for \CONTEXT\ (and especially hybrid usage). It also permits to hook in
413\LUA\ code and that way the editor can (within bounds) be extended. As an example
414I've added some button bars that permit entering math alphabets. Of course the
415appearance depends on the font used but operating systems tend to consult
416multiple fonts when the core font of the editor doesn't provide a glyph.
417
418\startlinecorrection
419    \externalfigure[still-math-stripe.png][width=\textwidth]
420\stoplinecorrection
421
422Here I show a small portion of the stripe with buttons that inject the shown
423characters. What happens in the rendering is that first the used font is
424consulted and that one has a couple of \quote {BLACK LETTER CAPITAL}s so they get
425used. The others are \quote {MATHEMATICAL FRAKTUR CAPITAL}s and since the font is
426not a math font the renderer takes them from (in this case) Cambria Math, which
427is why they look so different, especially in proportion. Of course we could start
428out with Cambria but it has no monospace (which I want for editing) and is a less
429complete text font, so we have a chicken||egg problem here. It is one reason why
430as part of the math font project we extend the Dejavu Sans Mono with proper
431(consistent) math symbols. Anyhow, it illustrates why gaps are kind of evil from
432the application point of view.
433
434\startluacode
435local data     = characters.data
436
437local bold     = context.bold
438local verbatim = context.formatted.type
439local small    = context.small
440local normal   = context
441
442local NC, NR, HL = context.NC, context.NR, context.HL
443
444context.start()
445
446context.definefont(
447    { "mathdemo"},
448    { "file:texgyrepagellamath*mymathematics" }
449)
450
451context.starttabulate { "||c||||" }
452    NC() bold("gap")
453    NC() bold("char")
454    NC() bold("meant")
455    NC() bold("unicode")
456    NC() bold("used")
457    NR() HL()
458    for k, v in table.sortedhash(mathematics.gaps) do
459        local description = data[v].description
460        local surrogate = string.match(description,".- (.)$")
461        if not surrogate then
462            surrogate = "H"
463        end
464        for i=k-1,1,-1 do
465            local d = data[i].description
466            if d ~= "PRIVATE SLOT" then
467                surrogate = string.gsub(d,"(.)$",surrogate)
468                break
469            end
470        end
471        NC() verbatim("%U",k)
472        NC() normal  ("\\mathdemo %c",k)
473        NC() small   (surrogate)
474        NC() verbatim("%U",v)
475        NC() small   (description)
476        NR()
477    end
478context.stoptabulate()
479
480context.stop()
481\stopluacode
482
483Barbara Beeton told me that, although it took some convincing arguments in the
484discussions about math in \UNICODE, we have at least one hole less than to be
485expected: slot \type {U+1D4C1} has not been seen as already covered by \type
486{U+02113}. So is there really this distinction between a \typ {MATHEMATICAL
487SCRIPT SMALL L} and \typ {SCRIPT SMALL L} (usually \type {\ell} in macro
488packages? Indeed there is, although at the time of this writing interestingly
489Latin Modern fonts lacked the mathematical one (which in \CONTEXT\ math mode
490normally results in an upright drop||in). Such details become important when math
491is edited by someone not familiar with the distinction between a variable (or
492whatever) represented by a script shape and the length operator. There seems not
493to be agreement by font designers about the shapes being upright or italic, so
494some confusion will remain, although this does not matter as long as within the
495font they differ.
496
497\definefont[SampleMathLatinModern][file:latinmodern-math]
498\definefont[SampleMathStixXits]   [file:xits-math]
499\definefont[SampleMathBonum]      [file:texgyrebonum-math]
500\definefont[SampleMathTermes]     [file:texgyretermes-math]
501\definefont[SampleMathPagella]    [file:texgyrepagella-math]
502\definefont[SampleMathLucida]     [file:lucidabrightmathot]
503
504\starttabulate[||||]
505    \NC \bf font     \NC \bf \type {U+1D4C1} \NC \bf \type {U+02113} \NC \NR
506    \HL
507    \NC latin modern \NC \SampleMathLatinModern \char"1D4C1 \NC \SampleMathLatinModern \char"02113 \NC \NR
508    \NC stix/xits    \NC \SampleMathStixXits    \char"1D4C1 \NC \SampleMathStixXits    \char"02113 \NC \NR
509    \NC bonum        \NC \SampleMathBonum       \char"1D4C1 \NC \SampleMathBonum       \char"02113 \NC \NR
510    \NC termes       \NC \SampleMathTermes      \char"1D4C1 \NC \SampleMathTermes      \char"02113 \NC \NR
511    \NC pagella      \NC \SampleMathPagella     \char"1D4C1 \NC \SampleMathPagella     \char"02113 \NC \NR
512    \NC lucida       \NC \SampleMathLucida      \char"1D4C1 \NC \SampleMathLucida      \char"02113 \NC \NR
513\stoptabulate
514
515As math uses greek and because greek was already present in \UNICODE\ when math
516was recognized as script and got its entries, you can imagine that there are some
517issues there too, but let us move on to using alphabets.
518
519In addition to a one||to||one mapping from a font slot onto a glyph, you can
520assign properties to characters that map them onto a slot in some family (which
521itself relates to a font). This means that in a traditional approach you can
522choose among two methods:
523
524\startitemize[packed]
525
526    \startitem
527        You define several fonts (or instances of the same font) where the
528        positions of regular characters point to the relevant shape. So, when an
529        italic family is active the related font maps character \type {U+61} as
530        well as \type {U+1D44E} to the same italic shape \quote {$ \utfchar
531        {0x1D44E} $}. A switch from italic to bold italic is then a switch in
532        family and in that family the \type {U+61} as well as \type {U+1D482}
533        become bold italic \quote {$ \utfchar {0x1D482} $}.
534    \stopitem
535
536    \startitem
537        You define just one font. The alphabet (uppercase, lowercase and sometimes
538        digits and a few symbols) gets codes that point to the right shape. When we
539        switch from italic to bold italic, these codes get reassigned.
540    \stopitem
541
542\stopitemize
543
544The first method has some additional overhead in defining fonts (you can use
545copies but need to make sure that the regular \ASCII\ slots are overloaded) but
546the switch from italic to bold italic is fast, while in the second variant there
547is less overhead in fonts but reassigning the codes with a style switch has some
548overhead (although in practice this overhead is can be neglected because not that
549many alphabet switches take place). In fact, many \TEX\ users will probably stick
550to traditional approaches where verbose names are used and these can directly
551point to the right shape.
552
553In \CONTEXT, when we started with \MKIV, we immediately decided to follow another
554approach. We only have one family and we assume \UNICODE\ math input. Ok, we do
555have a few more families, but these relate to a full bold math switch and
556right||to||left math. We cannot expect users to enter \UNICODE\ math, if only
557because support in editors is not that advanced, so we need to support the
558\ASCII\ input method as well.
559
560We have one family and don't redefine character codes, but set properties
561instead. We don't switch fonts, but properties. These properties (often a
562combination) translates into the remapping of a specific character in the input
563onto a \UNICODE\ math code point that then directly maps onto a shape. This
564approach is quite clean and efficient at the \TEX\ end but carries quite a lot of
565overhead at the \LUA\ end. So far users never complained about it, maybe because
566\CONTEXT\ math support is rather optimized. Also, dealing with characters is only
567part of math typesetting and we have subsystems that use far more processing
568power.
569
570Because math characters are organized in classes, we need to set them up. Because
571for several reasons we collect character properties in a database we also define
572these character properties in \LUA. This means that the \type {math-*} files are
573relatively small. So we have much less code at the \TEX\ end, but quite a lot at
574the \LUA\ end. This assumes a well managed \LUA\ subsystem because as soon as
575users start plugging in their code, we have to make sure that the core system
576still functions well. The amount of code involved in virtual math fonts is also
577relatively large but most of that is becoming sort of obsolete.
578
579Relatively new in \CONTEXT\ is the possibility in some mathematical constructs to
580configure the math style (text, script, etc.) and in some cases math classes can
581be influenced. Control over styles is somewhat more convenient in \LUATEX,
582because we can consult the current style in some cases. I expect more of this
583kind of control in \CONTEXT, although most users probably never need it. These
584kinds of features are meant for users like Aditya Mahajan, who likes to explore
585such features and also takes advantage of the freedom to experiment with the look
586and feel of math.
587
588The font code that relates to math is not the easiest to understand but this is
589because it has to deal with bold as well as bidirectional math in efficient ways.
590Because in \CONTEXT\ we have additional sizes (\type {x}, \type {xx}, \type {a},
591\type {b}, \type {c}, \type {d}, \unknown) we also have some delayed additional
592defining going on. This all might sound slower to set up but in the end we win
593some back by the fact that we have fewer fonts to load. The price that a
594\CONTEXT\ user pays in terms of runtime is more influenced by the by now large
595sequence of math list manipulators than by loading a font.
596
597An unfortunate shortcoming of \UNICODE\ math is that some alphabets have gaps.
598This is because characters can only end up once in the standard. Given the number
599of weird characters showing up in recent versions, I think this condition is
600somewhat over the top. It forces applications that deal with \UNICODE\ math to
601implement exceptions over and over again. In \CONTEXT\ we assume no gaps and
602compensate for that.
603
604There are several ways that characters can become glyphs. An \quote {a} can
605become an italic, bold, bold italic but also end up sans serif or monospace.
606Because there are several artistic interpretations possible, some fonts provide a
607so|-|called alternate. In the case of for instance greek we can also distinguish
608upright or slanted (italic). A less well known transformation is variants driven
609by \UNICODE\ modified directives. If we forget about bidirectional math and full
610bold (heavy) math we can (currently) identify 6 axes:
611
612\starttabulate[|c|l|l|]
613\HL
614\NC \bf axis \NC \bf use  \NC \bf choices \NC \NR
615\HL
616\NC 1        \NC type     \NC digits, lowercase \& uppercase latin \& greek, symbols \NC \NR
617\NC 2        \NC alphabet \NC regular, sans serif, monospace, blackboard, fraktur, script \NC \NR
618\NC 3        \NC style    \NC upright, italic, bold, bolditalic \NC \NR
619\NC 4        \NC variant  \NC alternative rendering provided by font \NC \NR
620\NC 5        \NC shape    \NC unchanged, upright, italic \NC \NR
621\NC 6        \NC \UNICODE \NC alternative rendering driven by \UNICODE\ modifier \NC \NR
622\HL
623\stoptabulate
624
625Apart from the last one, this is not new, but it is somewhat easier to support
626this consistently. It's one of the areas where \UNICODE\ shines, although the
627gaps in vectors are a bad thing. One thing that I decided early in the \MKIV\
628math development is that all should fit into the same model: it makes no sense to
629cripple a whole system because of a few exceptions.
630
631Users expect their digits to be rendered upright and letters to be rendered with
632italic shapes, but use regular \ASCII\ input. This means that we need to relocate
633the letters to the relevant alphabet in \UNICODE. In \CONTEXT\ this happens as
634part of several analysis steps that more or less are the same as the axis
635mentioned. In addition there is collapsing, remapping, italic correction,
636boldening, checking, intercepting of special input, and more going on. Currently
637there are (depending on what gets enabled) some 10 to 15 manipulation passes over
638the list and there will be more.
639
640So how does the situation compare to the old one? I think we can safely say that
641we're better off now and that \LUATEX\ behaves quite okay. There is not much that
642can be improved, apart from more complete fonts (especially bold). A nice bonus
643of \LUATEX\ is that math characters can be used in text mode as well (given that
644the current font provides them).
645
646It will be clear that by following this route we moved far away from the \MKII\
647approach and the dependency on \LUA\ has become rather large in this case. The
648benefit is that we have rather clean code with hardly any exceptions. It came at
649the price of lots of experiments and (re)coding but I think it pays off for
650users.
651
652\stopsection
653
654\startsection[title=Bold]
655
656Bold is sort of special. There are bold symbols and some bold alphabets and that
657{\em is} basically what bold math is: just a different rendering. In a proper
658\OPENTYPE\ math fonts these bold characters are covered.
659
660Section titles or captions are often typeset bolder and when they contain math
661all of it needs to be bolder too. So, a regular italic shape becomes a bold
662italic shape but a bold shape becomes heavy. This means that we need a full blown
663bold font for that purpose. And although some are on the agenda of the font team,
664often we need to fake it. This is seldom an issue as (at least in the documents
665that I deal with) section titles are not that loaded with math.
666
667A proper implementation of such a mechanism involves two aspects: first there
668needs to be a complete bold math font with heavy bold included, and second the
669macro package must switch to bold math in a bold context. When no real bold font
670is available, some automatic mapping can take place, but that might give
671interpretation issues if bold is used in a formula. For the average highschool
672math that we render this is not an issue. Currently there are no full bold math
673fonts that have enough coverage. (The \XITS\ font, derived from \STIX, has a bold
674companion that does provide for instance bold radicals but lacks many bolder
675alphabets and symbols.)
676
677\startbuffer
678\startimath
679       \sqrt{x^2\over 4x}  \qquad
680  {\bf \sqrt{x^2\over 4x}} \qquad
681  {\mb \sqrt{x^2\over 4x}} \qquad
682       \sqrt{x^2 + 4x}     \qquad
683  {\bf \sqrt{x^2 + 4x}}    \qquad
684  {\mb \sqrt{x^2 + 4x}}
685\stopimath
686\stopbuffer
687
688\typebuffer
689
690This gives:
691
692\blank \getbuffer \blank
693
694Here it is always a bit of a guess if bold extensibles are (already) supported so
695it's dangerous to go wild with full bold/heavy combinations unless you check
696carefully what results you get. Another aspect you need to be aware of is that
697there is an extensive fallback mechanism present. When possible a proper alphabet
698will be used, but when one is not present there is a fallback on another. This
699ensures that we get at least something.
700
701There is not much that an engine can do about it, apart from providing enough
702families to implement it. In a \TYPEONE\ universe indeed we need lots of families
703already so the traditional 16-family pool is drained soon. In \LUATEX\ we can
704have 256 families which means that additional \TYPEONE\ bases family sets are no
705issue any longer. But as in \MKIV\ we no longer follow that route, bold math can
706be set up relatively easy, given that we have a bold font. If we don't have such
707a font, we have an intermediate mode where a bold font is simulated. Keep in mind
708that this always will need checking, at least as long as don't have complete
709enough bold fonts with heavy bold included.
710
711\stopsection
712
713\startsection[title=Radicals]
714
715In most cases a \TEX\ user is not that aware of what happens in order to get a
716nicely wrapped up root on paper. In traditional \TEX\ this is an interplay
717between rather special font properties and macros. In \LUATEX\ it has become a
718bit more simple because we introduced a primitive for it. Also, in \OPENTYPE\
719fonts, the radical is provided in a somewhat more convenient way. In an
720\OPENTYPE\ math font there are some variables that control the rendering:
721
722\starttyping
723RadicalExtraAscender
724RadicalRuleThickness
725RadicalVerticalGap
726RadicalDisplayStyleVerticalGap
727\stoptyping
728
729The engine will use these to construct the symbol. The root symbols can grow in two
730dimensions: the left bit grows vertically but due to the fact that there is a slope
731involved it happens in steps using different symbols.
732
733\blank
734$ \dorecurse{10}{\rootradical{}{\blackrule[height=#1ex,depth=0pt,width=0pt]}} $
735\blank
736
737Compare this to for instance how a bracket grows:
738
739\blank
740$ \dorecurse{10}{\left[\blackrule[height=#1ex,depth=0pt,width=0pt]\right.} $
741\blank
742
743The bracket is a so|-|called vertical extensible character. It grows in steps
744using different glyphs and when we run out of variants a last resort kicks in: a
745symbol gets constructed from three pieces, a top and bottom piece and in between
746a repeated middle segment. The root symbol is also vertically extensible but
747there the change to the stretched variant is visually rather distinct. This has a
748reason: the specification cannot deal with slopes. So, in order to stretch the
749last resort, as with the bracket, goes vertical and provides a middle segment.
750
751The root can also grow horizontally; just watch this:
752
753\blank
754$ \dorecurse{10}{\rootradical{}{\blackrule[height=#1ex,depth=0pt,width=#1ex,color=gray]}} $
755\blank
756
757The font specification can handle vertical as well as horizontal extensibles but
758surprise: it cannot handle a combination. Maybe the reason is that there is only
759one such symbol: the radical. So, instead of expecting a symmetrical engine, an
760exception is made that is controlled by the mentioned variables. So, while we go
761upwards with a proper middle glyph, we go horizontal using a rule.
762
763One can argue that the traditional \TEX\ machinery is complex because it uses
764special font properties and macros, but once you start looking into the modern
765variant it becomes clear that although we can have a somewhat cleaner
766implementation, it still is a kludge. And, because rendering on paper no longer
767drives development it is not to be expected that this will change. The \TEX\
768community didn't come up with a better approach and there is no reason to believe
769that it will in the future.
770
771One of the reasons for users to use \TEX\ is control over the output: instead of
772some quick and dirty job authors can spend time on making their documents look
773the way they want. Even in these internet times with dynamic rendering, there is
774still a place for a more frozen rendering, explicitly driven by the author. But,
775that only makes sense when the author can influence the rendering, maybe even
776without bounds.
777
778So, because in \CONTEXT\ I really want to provide control, as one of the last
779components, math radicals were made configurable too. In fact, the code involved
780is not that complex because most was already in place. What is interesting is
781that when I rewrapped radicals once again I realized that instead of delegating
782something to the engine and font one could as well forget about it and do all in
783dedicated code. After all, what is a root symbol more that a variation of a
784framed bit of text. Here are some examples.
785
786\startbuffer[demo]
787$
788  y = \sqrt   { x^2 + ax + b } \quad
789  y = \sqrt[2]{ x^2 + ax + b } \quad
790  y = \sqrt[3]{ \frac{x^2 + ax + b }{c} }
791$
792\stopbuffer
793
794\typebuffer[demo]
795
796By default this gets rendered as follows:
797
798\blank \start \getbuffer[demo] \stop \blank
799
800We can change the rendering alternative to one that permits some additional
801properties (like color):
802
803\startbuffer[setup]
804\setupmathradical[sqrt][alternative=normal,color=maincolor]
805\stopbuffer
806
807\typebuffer[setup]
808
809This looks more or less the same:
810
811\blank \start \getbuffer[setup,demo] \stop \blank
812
813\startbuffer[setup]
814\setupmathradical
815  [sqrt]
816  [alternative=mp,
817   color=darkgreen]
818\stopbuffer
819
820We can go a step further and instead of a font use a symbol that adapts itself:
821
822\typebuffer[setup]
823
824Now we get this:
825
826\blank \start \getbuffer[setup,demo] \stop \blank
827
828Such a variant can be more subtle, as we not only can adapt the slope
829dynamically, but also add a nice finishing touch to the end of the horizontal
830line. Take this variant:
831
832\startbuffer
833\startuniqueMPgraphic{math:radical:extra}
834  draw
835    math_radical_simple(OverlayWidth,OverlayHeight,OverlayDepth,OverlayOffset)
836    withpen pencircle
837      xscaled (2OverlayLineWidth)
838      yscaled (3OverlayLineWidth/4)
839      rotated 30
840    dashed evenly
841    withcolor OverlayLineColor ;
842\stopuniqueMPgraphic
843\stopbuffer
844
845\typebuffer \getbuffer
846
847\startbuffer[setup-extra]
848\setupmathradical
849  [sqrt]
850  [alternative=mp,
851   mp=math:radical:extra,
852   color=darkred]
853\stopbuffer
854
855We hook this graphic into the macro:
856
857\typebuffer[setup-extra]
858
859And this time we see a dashed line:
860
861\blank \start \getbuffer[setup-extra,demo] \stop \blank
862
863Of course one can argue about esthetics but let's face it: much ends up in print,
864also by publishers, that doesn't look pretty at all, so I tend to provide the
865author the freedom to make what he or she likes most. If someone is willing to
866spend time on typesetting (using \TEX), let's at least make it a pleasant
867experience.
868
869\blank
870$ \getbuffer[setup]\dostepwiserecurse{1}{13}{2}{\sqrt{\blackrule[height=#1ex,depth=0pt,width=#1ex,color=gray]}\quad} $
871\blank
872
873Here we see the symbol adapt. We can think of alternative symbols, for instance
874the first part becomes wider dependent on the height, but this can be made less
875prominent. Depending on user input I will provide some more variants as it's
876relatively easy to implement.
877
878Before I wrap up, let's see what exactly we have in stock deep down.
879Traditionally \TEX\ provides a \type {\surd} command which is just the root
880symbol. Then there is a macro \type {\root..\of..} that wraps the last argument
881in a root and typesets a degree as well (of given). In \CONTEXT\ we now provide
882this:
883
884\startbuffer
885$\surd x \quad \surdradical x  \quad \rootradical{3}{x} \quad \sqrt[3]{x}$
886\stopbuffer
887
888\typebuffer
889
890I don't remember ever having used the \type {\surd} command, but this is what
891it renders:
892
893\blank \noindentation \getbuffer \blank
894
895Only the last command, \type {\sqrt} is a macro defined in one of the math
896modules, the others are automatically defined from the database:
897
898\starttyping
899[0x221A] = { -- there are a few more properties set
900  unicodeslot = 0x221A,
901  description = "SQUARE ROOT",
902  adobename   = "radical",
903  category    = "sm",
904  mathspec    = {
905    { class = "root",     name = "rootradical" },
906    { class = "radical",  name = "surdradical" },
907    { class = "ordinary", name = "surd"        },
908  },
909}
910\stoptyping
911
912So we get the following definitions:
913
914\testpage[4]
915
916\starttabulate[||||]
917\FL
918\NC \bf command         \NC \bf meaning                \NC \bf usage \NC \SR
919\FL
920\NC \type{\surd}        \NC \tttf \meaning\surd        \NC \type{\surd} \NC \FR
921\NC \type{\surdradical} \NC \tttf \meaning\surdradical \NC \type{\surdradical {body}} \NC \MR
922\NC \type{\rootradical} \NC \tttf \meaning\rootradical \NC \type{\rootradical {degree} {body}} \NC \LR
923\LL
924\stoptabulate
925
926So, are we better off? Given that a font sticks to how Cambria does it, we only
927need a minimal amount of code to implement roots. This is definitely an
928improvement at the engine level. However, in the font there are no fundamental
929differences between the traditional and more modern approach, but we've lost the
930opportunity to make a proper two||dimensional extensible. Eventually the user
931won't care as long as the macro package wraps it all up in useable macros.
932
933\stopsection
934
935\startsection[title=Primes]
936
937Another rather disturbing issue is with primes. A prime is an accent|-|like
938symbol that as a kind of superscript is attached to a variable or function. In
939good old \TEX\ tradition this is entered as follows:
940
941\startbuffer
942$ f'(x) $ and $ f''(x) $
943\stopbuffer
944
945\typebuffer
946
947which produces: \inlinebuffer. The upright quote symbols are never used for
948anything else than primes and magically get remapped onto a prime symbol. This
949might look trivial, but there are several aspects to deal with, especially when
950using traditional fonts. In the eight|-|bit \type {lmsy10} math symbol font,
951which is derived from the original \type {cmsy10} the prime symbol looks like
952this:
953
954\startlinecorrection
955\ruledhbox{\definedfont[file:lmsy10.afm]\getnamedglyphdirect{file:lmsy10.afm}{prime}}
956\stoplinecorrection
957
958The bounding box is rather tight and the reason for this becomes clear when we put
959it alongside another character:
960
961\startlinecorrection
962$x\ruledhbox{\definedfont[file:lmsy10.afm]\getnamedglyphdirect{file:lmsy10.afm}{prime}}$
963\stoplinecorrection
964
965The prime is not only pretty large, it also sits on the baseline. It means that
966in order to make it a real prime (basically an operator pointing back to the
967preceding symbol), we need to raise it. Of course we can define a \type {\prime}
968command that takes care of this, and indeed that is what happens in plain \TEX\
969and derived formats. The more direct \type {'} input is supported by making that
970character an active character in math mode. Active characters behave like
971commands and in this case the \type {\prime} command.
972
973In the \OPENTYPE\ latin modern fonts the prime (\type{U+2032}) looks like this:
974
975\startlinecorrection
976$x\ruledhbox{\definedfont[file:latin-modernmath]\utfchar{0x2032}}$
977\stoplinecorrection
978
979So here we have an already raised and also smaller prime symbol. And, because we
980also have double (\type{U+2033}) and triple primes (\type{U+2034}) a few more
981characters are available
982
983\startlinecorrection
984$x\ruledhbox{\definedfont[file:latin-modernmath]\utfchar{0x2032}}$
985$x\ruledhbox{\definedfont[file:latin-modernmath]\utfchar{0x2033}}$
986$x\ruledhbox{\definedfont[file:latin-modernmath]\utfchar{0x2034}}$
987\stoplinecorrection
988
989In the traditional approach these second and third order primes are built from
990the first order primes. And this introduces, in addition to the raising, another
991complexity: the \type {\prime} command has to look ahead and intercept future
992primes. And as there can also be a following raised symbol (or number) it needs
993to take a superscript trigger into account as well. So, let's look at some
994possible input:
995
996\def\ShowPrime#1{\NC \type{$#1$} \NC $#1$ \NC \NR}
997
998\starttabulate[|||]
999\ShowPrime{f'(x)}
1000\ShowPrime{f''(x)}
1001\ShowPrime{f'''(x)}
1002\ShowPrime{f\prime^2}
1003\ShowPrime{f\prime\prime^2}
1004\ShowPrime{f\prime\prime\prime^2}
1005\ShowPrime{f'\prime'^2}
1006\ShowPrime{f^'(x)}
1007\ShowPrime{f'^2}
1008\ShowPrime{f{\prime}^2}
1009\stoptabulate
1010
1011Now imagine that you have this big prime character sitting on the baseline and
1012you need to turn \type {'''} into a a triple prime, but don't want \type {^'} to
1013be double raised, while on the other hand \type {^2} should be. This is of course
1014doable with some macro juggling but how about supporting traditional fonts in
1015combination with \OPENTYPE, where the primes are already raised.
1016
1017When we started with \LUATEX\ and \CONTEXT\ \MKIV, one of the first decisions I
1018made was to go \UNICODE\ math and drop eight|-|bit. In order to compensate for
1019the lack of fonts, a mechanism was provided to construct virtual \UNICODE\ math
1020fonts, as a prelude to the lm/gyre \OPENTYPE\ math fonts. In the meantime we have
1021these fonts and the virtual variants are only kept as historic reference and for
1022further experiments.
1023
1024As a starter I wrote a variant of the traditional \CONTEXT\ \type {\prime}
1025command that could recognize somehow if it was dealing with a \TYPEONE\ or
1026\OPENTYPE\ font. As a consequence it also had the traditional raise and look
1027ahead mess on board. However, there was also some delegation to the \LUA\
1028enhanced math support code, so the macro was not that complex. When the real
1029\OPENTYPE\ math fonts showed up the macro was dropped and the virtual fonts were
1030adapted to the raised|-|by|-|default situation, which in itself was somewhat
1031complicated by the fact that a smaller symbol had to be used, i.e.\ some more
1032information about the current set of defined math sizes has to be passed around.
1033\footnote {The actual solution for this qualifies as a dirty trick so we are not
1034freed from tricks yet.}
1035
1036Anyhow, the current implementation is rather clean and supports collapsing of
1037combinations rather well. There are four prime symbols but only three reverse
1038prime symbols. If needed I can provide a virtual \typ {REVERSED TRIPLE PRIME} if
1039needed, but I guess it's not needed.
1040
1041\def\Nsprime{\ruledmbox{\prime}}
1042\def\Ndprime{\ruledmbox{\doubleprime}}
1043\def\Ntprime{\ruledmbox{\tripleprime}}
1044\def\Nqprime{\ruledmbox{\quadrupleprime}}
1045
1046\def\Rsprime{\ruledmbox{\reversedprime}}
1047\def\Rdprime{\ruledmbox{\reverseddoubleprime}}
1048\def\Rtprime{\ruledmbox{\reversedtripleprime}}
1049
1050\starttabulate[|lT|lT|lM|lM|]
1051\NC U+2032 \NC \chardescription{"2032} \NC \prime               \NC \Nsprime \NC \NR
1052\NC U+2033 \NC \chardescription{"2033} \NC \doubleprime         \NC \Nsprime \Nsprime \quad
1053                                                                    \Ndprime \NC \NR
1054\NC U+2034 \NC \chardescription{"2034} \NC \tripleprime         \NC \Nsprime \Nsprime \Nsprime \quad
1055                                                                    \Nsprime \Ndprime \quad
1056                                                                    \Ndprime \Nsprime \quad
1057                                                                    \Ntprime \NC \NR
1058\NC U+2057 \NC \chardescription{"2057} \NC \quadrupleprime      \NC \Nsprime \Nsprime \Nsprime \Nsprime \quad
1059                                                                    \Nsprime \Nsprime \Ndprime \quad
1060                                                                    \Nsprime \Ndprime \Nsprime \quad
1061                                                                    \Ndprime \Nsprime \Nsprime \quad
1062                                                                    \Ndprime \Ndprime \quad
1063                                                                    \Ntprime \Nsprime \quad
1064                                                                    \Nsprime \Ntprime \quad
1065                                                                    \Nqprime \NC \NR
1066\NC U+2035 \NC \chardescription{"2035} \NC \reversedprime       \NC \Rsprime \NC \NR
1067\NC U+2036 \NC \chardescription{"2036} \NC \reverseddoubleprime \NC \Rsprime \Rsprime \quad
1068                                                                    \Rdprime \NC \NR
1069\NC U+2037 \NC \chardescription{"2037} \NC \reversedtripleprime \NC \Rsprime \Rsprime \Rsprime \quad
1070                                                                    \Rsprime \Rdprime \quad
1071                                                                    \Rdprime \Rsprime \quad
1072                                                                    \Rtprime \NC \NR
1073\stoptabulate
1074
1075Of course no one will use this ligature approach but I've learned to be prepared
1076as it wouldn't be the first time when we encounter input that is cut and paste
1077from someplace or clicked|-|till|-|it|-|looks|-|okay.
1078
1079There is one big complication and that is that where in \TEX\ there is only one
1080big prime that gets raised and repeated in case of multiple primes, in \OPENTYPE\
1081the primes are already raised. They are in fact not supposed to be superscripted,
1082as they are already. In plain \TEX\ the prime is entered using an upright single
1083quote and that one is made active: it is in fact a macro. That macro looks ahead
1084and intercepts following primes as well as subscripts. In the end, a superscript
1085(the prime) and optional subscripts are attached to the preceding symbol. If we
1086want to benefit from the \UNICODE\ primes as well as support collapsing, such a
1087macro quickly becomes messy. Therefore, in \MKIV\ the optional subscript is
1088handled in the collapser. We cheat a bit by relocating super- and subscripts and
1089at the same time remap the primes to virtual characters that are smashed to a
1090smaller height, lowered to the baseline, and eventually superscripted. Indeed, it
1091sounds somewhat complex and it is. In a next version I will also provide ways to
1092influence the size as one might want larger of smaller primes to show up. This is
1093one case where the traditional \TEX\ fonts have a benefit as the primes are
1094superscriptable characters, but we have to admit that the \UNICODE\ and
1095\OPENTYPE\ approach is conceptually more correct. The only way out of this is to
1096have a primitive operation for primes just as we have for radicals but that also
1097has some drawbacks. Eventually I might come up with a cleaner solution for this
1098dilemma.
1099
1100Let us summarize the situation and solution used in \MKIV\ now:
1101
1102\startitemize[packed]
1103    \startitem
1104        When (still) using the virtual \UNICODE\ math fonts, we construct a
1105        virtual glyph that has properties similar to proper \OPENTYPE\ math
1106        fonts.
1107    \stopitem
1108    \startitem
1109        We collapse a sequence of primes into proper double and triple
1110        primes.
1111    \stopitem
1112    \startitem
1113        We unraise primes so that users who (for some reason) superscript them
1114        (maybe because they still assume big ones sitting on the baseline) get
1115        the desired outcome.
1116    \stopitem
1117    \startitem
1118        We accept mixtures of \type {'} and \type {\prime}.
1119    \stopitem
1120\stopitemize
1121
1122We can do this because in \CONTEXT\ \MKIV\ we don't care too much about exact
1123visual compatibility as long as we can make users happy with clean mechanisms.
1124So, this is one of the situations where the new situation is better, thanks to on
1125the one hand the way primes are provided in fonts, and on the other hand the
1126enhanced math machinery in \MKIV.
1127
1128\stopsection
1129
1130\startsection[title=Accents]
1131
1132There are a few special character types in math and accents are one of them.
1133Personally I think that the term accent is somewhat debatable but as they are
1134symbols drawn on top of or below something we can stick to that description for
1135the moment. In addition to some regular fixed width variants, we have adaptive
1136versions: \type {\hat} as well as \type {\widehat} and more.
1137
1138\startlinecorrection
1139\dorecurse{6}{$\widehat{\blackrule[width=#1ex,color=gray]}$ }
1140\stoplinecorrection
1141
1142I have no clue if wider variants are needed but such a partial coverage
1143definitely looks weird. So, as an escape users can kick in their own code. After
1144all, who says that a user cannot come up with a new kind of math. The following
1145example demonstrates how this is done:
1146
1147\startbuffer
1148\startMPextensions
1149    vardef math_ornament_hat(expr w,h,d,o,l) text t =
1150        image (
1151            fill
1152                (w/2,10l) -- (w + o/2,o/2) --
1153                (w/2, 7l) -- (  - o/2,o/2) --
1154                cycle shifted (0,h-o) t ;
1155            setbounds
1156                currentpicture
1157            to
1158                unitsquare xysized(w,h) enlarged (o/2,0)
1159        )
1160    enddef ;
1161\stopMPextensions
1162\stopbuffer
1163
1164\typebuffer \getbuffer
1165
1166This defines a hat|-|like symbol. Once the sources of the math font project are
1167published I can imagine that an ambitious user defines a whole set of proper
1168shapes. Next we define an adaptive instance:
1169
1170\startbuffer
1171\startuniqueMPgraphic{math:ornament:hat}
1172    draw
1173        math_ornament_hat(
1174            OverlayWidth,
1175            OverlayHeight,
1176            OverlayDepth,
1177            OverlayOffset,
1178            OverlayLineWidth
1179        )
1180    withpen
1181        pencircle
1182            xscaled (2OverlayLineWidth)
1183            yscaled (3OverlayLineWidth/4)
1184            rotated 30
1185    withcolor
1186        OverlayLineColor ;
1187\stopuniqueMPgraphic
1188\stopbuffer
1189
1190\typebuffer \getbuffer
1191
1192Last we define a symbol:
1193
1194\startbuffer
1195\definemathornament [mathhat] [mp=math:ornament:hat,color=darkred]
1196\stopbuffer
1197
1198\typebuffer \getbuffer
1199
1200And use it as \type {\mathhat{...}}:
1201
1202\startlinecorrection
1203\dorecurse{8}{$\mathhat{\blackrule[width=#1ex,color=gray]}$ }
1204\stoplinecorrection
1205
1206Of course this completely bypasses the accent handler and in fact even writing
1207the normal stepwise one is not that hard to do in macros. But, there is a
1208built||in mechanism that helps us for those cases and it can even deal with font
1209based stretched alternatives of which there are a few: curly braces, brackets and
1210parentheses. The reason that these can stretch is that they don't have slopes and
1211therefore can be constructed out of pieces: in the case of a curly brace we have
12124 snippets: begin, end, middle and repeated rules, and in the case of braces and
1213brackets 3 snippets will do. But, if we really want we can use \METAPOST\ code
1214similar to the code shown above to get a nicer outcome.
1215
1216There are in good \TEX\ tradition four accents that can also stretch
1217horizontally: bar, brace, parenthesis and bracket. When using fonts such an
1218accent looks like this:
1219
1220% \setupmathstackers[vfenced][color=darkyellow]
1221
1222\startbuffer
1223$ \overbrace{a+b+c+d} \quad \underbrace{a+b+c+d} \quad \doublebrace{a+b+c+d} $
1224\stopbuffer
1225
1226\blank \start \setupmathstackers[vfenced][color=darkyellow] \getbuffer \stop \blank
1227
1228this is coded like:
1229
1230\typebuffer
1231
1232As with radicals, for more fancy math you can plug in \METAPOST\ variants. Of
1233course this kind of rendering should fit into the layout of the document but I
1234can imagine that for schoolbooks this makes sense.
1235
1236\startbuffer[setup]
1237\useMPlibrary[mat]
1238
1239\setupmathstackers
1240  [vfenced]
1241  [color=darkred,
1242   alternative=mp]
1243\stopbuffer
1244
1245\typebuffer[setup]
1246
1247Applied in an example we get:
1248
1249\startbuffer[demo]
1250$\overbracket{a+b+c+d} \quad \underbracket{a+b+c+d} \quad \doublebracket{a+b+c+d}$ \blank
1251$\overparent {a+b+c+d} \quad \underparent {a+b+c+d} \quad \doubleparent {a+b+c+d}$ \blank
1252$\overbrace  {a+b+c+d} \quad \underbrace  {a+b+c+d} \quad \doublebrace  {a+b+c+d}$ \blank
1253$\overbar    {a+b+c+d} \quad \underbar    {a+b+c+d} \quad \doublebar    {a+b+c+d}$ \blank
1254\stopbuffer
1255
1256\start \getbuffer[setup] \startlines\getbuffer[demo]\stoplines \stop
1257
1258This kind of magic is partly possible because in \LUATEX\ (and therefore \MKIV)
1259we can control matters a bit better. And of course the fact that we have
1260\METAPOST\ embedded means that the impact of using graphics is not that large.
1261
1262We used the term \quote {stackers} in the setup command so although these are
1263officially accents, in \CONTEXT\ we implement them as instances of a more generic
1264mechanism: things stacked on top of each other. We will discuss these in the next
1265section.
1266
1267\stopsection
1268
1269\startsection[title=Stackers]
1270
1271In plain \TEX\ and derived work you will find lots of arrow builders. In most
1272cases we're talking of a combination of one or more single or double arrow heads
1273combined with a rule. In any case it is something that is not so much font driven
1274but macro magic. Optionally there can be text before and|/|or after as well as
1275text above and|/|or below them. The later is for instance the case in chemistry.
1276This text is either math or upright properly kerned and spaced non||mathematical
1277text so we're talking of some mixed math and text usage. The size is normally
1278somewhat smaller.
1279
1280Arrows can also go on top or below regular math so in the end we end up with
1281several cases:
1282
1283\startitemize[packed]
1284    \startitem
1285        Something stretchable on top of or centered around the baseline, optionally
1286        with text above or below.
1287    \stopitem
1288    \startitem
1289        Something stretchable on top of a running (piece of) text or math.
1290    \stopitem
1291    \startitem
1292        Something stretchable below a running (piece of) text or math.
1293    \stopitem
1294    \startitem
1295        Something stretchable on top as well as below a running (piece of) text
1296        or math.
1297    \stopitem
1298\stopitemize
1299
1300These have in common that the symbol gets stretched. In fact the last three cases
1301are quite similar to accents but in traditional \TEX\ and its fonts arrows and
1302alike never made it to accents. One reason is probably that because a macro
1303language was available and because fonts were limited, it was rather easy to use
1304rules to extend an arrowhead.
1305
1306In \CONTEXT\ this kind of vertically stacked stretchable material is implemented
1307as stackers. In the chapter \type {mathstackers} of \type {about.pdf} you can
1308read more about the details so here I stick to a short summary to illustrate what
1309we're dealing with. Say that you want an arrow that stretches over a given width.
1310
1311\starttyping
1312\hbox to 4cm{\leftarrowfill}
1313\stoptyping
1314
1315In traditional \TEX\ with traditional fonts the definition of this arrow
1316looks as follows:
1317
1318\starttyping
1319\def\leftarrowfill {$
1320  \mathsurround=0pt
1321  \mathord{\mathchar"2190}
1322  \mkern-7mu
1323  \cleaders
1324   \hbox {$
1325     \mkern-2mu
1326     \mathchoice
1327      {\setbox0\hbox{$\displaystyle     -$}\ht0=0pt\dp0=0pt\box0}
1328      {\setbox0\hbox{$\textstyle        -$}\ht0=0pt\dp0=0pt\box0}
1329      {\setbox0\hbox{$\scriptstyle      -$}\ht0=0pt\dp0=0pt\box0}
1330      {\setbox0\hbox{$\scriptscriptstyle-$}\ht0=0pt\dp0=0pt\box0}
1331     \mkern-2mu
1332    $}
1333   \hfill
1334  \mkern-7mu
1335  \mathchoice
1336   {\setbox0\hbox{$\displaystyle     -$}\ht0=0pt\dp0=0pt\box0}
1337   {\setbox0\hbox{$\textstyle        -$}\ht0=0pt\dp0=0pt\box0}
1338   {\setbox0\hbox{$\scriptstyle      -$}\ht0=0pt\dp0=0pt\box0}
1339   {\setbox0\hbox{$\scriptscriptstyle-$}\ht0=0pt\dp0=0pt\box0}
1340$}
1341\stoptyping
1342
1343When using \TYPEONE\ fonts we don't use a \type {\mathchar} but
1344more something like this:
1345
1346\starttyping
1347\leftarrow = \mathchardef\leftarrow="3220
1348\stoptyping
1349
1350What we see in this macro is a left arrow head at the start and as minus sign at
1351the end. In between the \type {\cleaders} will take care of filling up the
1352available hsize with more minus signs. The overlap is needed in order to avoid
1353gaps due to rounding in the renderer and also obscures the rounded caps of the
1354used minus sign.
1355
1356The minus sign is used because it magically connects well to the arrow head. This
1357is of course a property of the design but even then you can consider it a dirty
1358trick. We don't specify a width here as this macro adapts itself to the current
1359width due to the leader. But if we do know the width an easier approach becomes
1360possible. Take this combination of a left and right arrow on top of each other:
1361
1362\starttyping
1363\mathstylehbox{\Umathaccent\fam\zerocount"21C4{\hskip4cm}}
1364\stoptyping
1365
1366The \type {\mathstylehbox} macro is a \CONTEXT\ helper. When we take a closer
1367look at the result (scaled up a bit) we see again snippets being used: \footnote
1368{We cheat a bit here: as we use \XITS\ in this document, and that font doesn't
1369yet provide this magic we switch temporarily to the Pagella font}.
1370
1371\startlinecorrection
1372\showglyphs \switchtobodyfont[pagella]
1373\scale[width=\textwidth]{\mathstylehbox{\Umathaccent\fam\zerocount"21C4{\hskip4cm}}}
1374\stoplinecorrection
1375
1376But this time the engine itself deals with the filling. Unfortunately for the
1377accent approach to work we need to specify the width. Given how these arrows are
1378used, this is no problem: because we often put text on top and|/|or below, we
1379need to do some packaging and therefore know the dimensions, but a generic
1380alternative would be nice. This is why for \LUATEX\ we have on the low priority
1381agenda:
1382
1383\starttyping
1384\leaders"2190\hfill
1385\stoptyping
1386
1387or a similar primitive. This way we can let the engine do some work and keep
1388macros simple. Normally \type {\leaders} delegate part of repeating to the
1389backend but in the case of math it has to be part of constructing the formula
1390because the extensible constructor has to be used.
1391
1392If you've looked into the \LUATEX\ manual you might have noticed that there is a
1393new primitive that permits this:
1394
1395\starttyping
1396\mathstylehbox{\Uoverdelimiter\fam"21C4{\hskip4cm}}
1397\stoptyping
1398
1399However, it is hardly useable for our purpose for several reasons. First of all,
1400when the argument is narrower than the smallest possible delimiter both get left
1401aligned, so the delimiter sticks out (this can be considered a bug). But also,
1402the placement is influenced by a couple of parameters that we then need to force
1403to zero values, which might interfere. Another property of this mechanism is that
1404the style is influenced and so we need to mess more with that. These are enough
1405reasons to ignore this extension for a while. Maybe at some point, when really
1406needed, I will write a proper wrapper for this primitive.
1407
1408When we started with \MKIV\ we stuck with the leaders approach for a while if
1409only because there was no real need to redefine the old macros. But after a while
1410one starts wondering if this is still the way to go, especially when
1411reimplementing the chemistry macros didn't lead to nicer looking code. Part of
1412the problem was that putting two arrows on top of each other where each one goes
1413into another direction gave issues due to the fact that we don't have the right
1414snippets to do it nicely. A way out was to create virtual characters for
1415combinations of begin and end snippets as well as middle pieces, construct a
1416proper virtual extensible and use the \LUATEX\ extensible constructor. Although
1417we still have a character that gets built out of snippets, at least the begin and
1418end snippet indicate that we have to do with one codepoint, contrary to two
1419independent stacked arrows.
1420
1421This was also the moment that I realized that it was somewhat weird that
1422\OPENTYPE\ math fonts didn't have that kind of support. After discussing this
1423with Bogus{\l}aw Jackowski of the math font project we decided that it made sense
1424to add proper native extensibles to the upcoming math fonts. Of course I still
1425had to support other math fonts but at least we had a conceptually clean example
1426font now. So, from that moment on the implementation used extensibles when
1427possible and falls back on the fake approach when needed.
1428
1429In \CONTEXT\ all these vertically stacked items are now handled by the math
1430stacker subsystem, including a decent set of configuration options. As said, the
1431symbols that need to stretch currently use the accent primitives which is okay
1432but somewhat messy because that mechanism is hard to control (after all it wants
1433to put stuff on top or below something). For (mostly) chemistry we can put text
1434on top or below arrows and control offsets of the text as well as the axis of the
1435arrows. We can use color and set the style. In addition there are constructs
1436where there is text in the middle and arrows (or other symbols that need to
1437adapt) on top or at the bottom.
1438
1439Many arrows come in sizes. For instance there are two sizes of right pointing
1440arrows as well as stretched variants, and use as top and bottom accents.
1441
1442\starttabulate[|T||]
1443\NC \detokenize {$\rightarrow     \quad \char"2192$} \NC $\rightarrow     \quad \char"2192$ \NC \NR
1444\NC \detokenize {$\longrightarrow \quad \char"27F6$} \NC $\longrightarrow \quad \char"27F6$ \NC \NR
1445\TB
1446\NC \detokenize {\hbox to 2cm{$\rightarrowfill$}}    \NC \hbox to 2cm{$\rightarrowfill$}    \NC \NR
1447\NC \detokenize {\hbox to 4cm{$\rightarrowfill$}}    \NC \hbox to 4cm{$\rightarrowfill$}    \NC \NR
1448\TB
1449\NC \detokenize {$\overrightarrow{a+b+c}$}           \NC $\overrightarrow{a+b+c}$           \NC \NR
1450\NC \detokenize {$\underrightarrow{a+b+c}$}          \NC $\underrightarrow{a+b+c}$          \NC \NR
1451\stoptabulate
1452
1453The first two arrows are just characters. The boxed ones are extensibles using
1454leaders that build the arrow from snippets (a hack till we have proper character
1455leaders) and the last two are implemented by abusing the accent mechanism and
1456thereby use the native extensibles of the first character.
1457
1458The problem here is in names and standards. The first characters have a fixed
1459size while the later are composed. The short ones have the extensibles and can
1460therefore be used as accents (or when supported as character leader). However
1461from the user's perspective, the distinction between the two \UNICODE\ characters
1462might be less clear, not so much when they are used as character, but when used
1463on top of or below something. As a coincidence, while writing this section, a
1464colleague dropped a snippet of \MATHML\ on my desk:
1465
1466\starttyping
1467<m:math>
1468  <m:mrow>
1469    <m:mover accent='true'>
1470      <m:mrow>
1471        <m:mi>A</m:mi>
1472        <m:mi>S</m:mi>
1473      </m:mrow>
1474      <m:mo stretchy='true'>→</m:mo>
1475    </m:mover>
1476  </m:mrow>
1477</m:math>
1478\stoptyping
1479
1480However, instead of {<m:mo>→</m:mo>} there was used \type
1481{<m:mo>&xrarr;</m:mo>} and that entity is the long arrow. As is often the case in
1482\MATHML\ the rendering is supposed to be quite tolerant and here both should
1483stretch over the row. When a \TEX\ user renders his or her source and sees
1484something wrong, the search for what character or command should be used instead
1485starts. A \MATHML\ user probably just expects things to work. This means that in
1486a system like \CONTEXT\ there will always be hacks and kludges to deal with such
1487matters. It is again one of these areas where optimally the \TEX\ community could
1488have influenced proper and systematic coding, but it didn't happen. So, no matter
1489now good we make an engine or macro package, we always need to be prepared to
1490adapt to what users expect. Let's face it: it's not that trivial to explain why
1491one should favor one or the other arrow as accent: the more it has to cover, the
1492longer it gets and the more we think of long arrows, but adding a whole bunch of
1493\type {\longrightarrow...} commands to \CONTEXT\ makes no sense.
1494
1495Nevertheless, we might eventually provide more \MATHML\ compliant commands at the
1496\TEX\ end. Just consider the following \MATHML\ snippets: \footnote {These
1497examples are variations on what we run into in Dutch school math (age 14\endash
149816).}
1499
1500\startbuffer[mathml]
1501<m:math xmlns:m="http://www.w3.org/1998/Math/MathML">
1502    <m:mrow>
1503        <m:mi>a</m:mi>
1504        <m:mover>
1505            <m:mo>&xrarr;</m:mo>
1506            <m:ms>arrow + text</m:ms>
1507        </m:mover>
1508        <m:mi>b</m:mi>
1509        <m:mover>
1510            <m:ms>text + arrow</m:ms>
1511            <m:mo>&xrarr;</m:mo>
1512        </m:mover>
1513        <m:mi>c</m:mi>
1514    </m:mrow>
1515</m:math>
1516\stopbuffer
1517
1518\typebuffer[mathml]
1519
1520This renders as:
1521
1522\blank \xmlprocessbuffer{main}{mathml}{} \blank
1523
1524Here the same construct is being used for two purposes: put an arrow on top of
1525content that sits on the math axis or put text on an arrow that sits on the math
1526axis. In \TEX\ we have different commands for these:
1527
1528\startbuffer[tex]
1529$ a \overrightarrow{b+c} d $ and $ a \mrightarrow{b+c} d $
1530\stopbuffer
1531
1532\typebuffer[tex]
1533
1534or
1535
1536\blank \getbuffer[tex] \blank
1537
1538The same is the case for:
1539
1540\startbuffer[mathml]
1541<m:math xmlns:m="http://www.w3.org/1998/Math/MathML">
1542    <m:mrow>
1543        <m:mi>a</m:mi>
1544        <m:munder>
1545            <m:mo>&xrarr;</m:mo>
1546            <m:ms>arrow + text</m:ms>
1547        </m:munder>
1548        <m:mi>b</m:mi>
1549        <m:munder>
1550            <m:ms>text + arrow</m:ms>
1551            <m:mo>&xrarr;</m:mo>
1552        </m:munder>
1553        <m:mi>c</m:mi>
1554    </m:mrow>
1555</m:math>
1556\stopbuffer
1557
1558\typebuffer[mathml]
1559
1560or:
1561
1562\blank \xmlprocessbuffer{main}{mathml}{} \blank
1563
1564When no arrow (or other stretchable character) is used, we still need to put one
1565on top of the other, but in any case we need to recognize the two cases that need
1566the special stretch treatment. There is also a combination of over and under:
1567
1568\startbuffer[mathml]
1569<m:math xmlns:m="http://www.w3.org/1998/Math/MathML">
1570    <m:mrow>
1571        <m:mi>a</m:mi>
1572        <m:munderover>
1573            <m:mo>&xrarr;</m:mo>
1574            <m:ms>text 1</m:ms>
1575            <m:ms>text 2</m:ms>
1576        </m:munderover>
1577        <m:mi>b</m:mi>
1578    </m:mrow>
1579</m:math>
1580\stopbuffer
1581
1582\typebuffer[mathml]
1583
1584\blank \xmlprocessbuffer{main}{mathml}{} \blank
1585
1586And again we need to identify the special stretchable characters from anything
1587otherwise.
1588
1589\startbuffer[mathml]
1590<m:math xmlns:m="http://www.w3.org/1998/Math/MathML">
1591    <m:mrow>
1592        <m:mi>a</m:mi>
1593        <m:munderover>
1594            <m:ms>text 1</m:ms>
1595            <m:ms>text 2</m:ms>
1596            <m:ms>text 3</m:ms>
1597        </m:munderover>
1598        <m:mi>b</m:mi>
1599    </m:mrow>
1600</m:math>
1601\stopbuffer
1602
1603\typebuffer[mathml]
1604
1605or:
1606
1607\blank \xmlprocessbuffer{main}{mathml}{} \blank
1608
1609And we even can have this:
1610
1611\startbuffer[mathml]
1612<m:math xmlns:m="http://www.w3.org/1998/Math/MathML">
1613    <m:mrow>
1614        <m:mi>a</m:mi>
1615        <m:munderover>
1616            <m:ms>text 1</m:ms>
1617            <m:mo>&xrarr;</m:mo>
1618            <m:ms>text 2</m:ms>
1619        </m:munderover>
1620        <m:mi>b</m:mi>
1621    </m:mrow>
1622</m:math>
1623\stopbuffer
1624
1625\typebuffer[mathml]
1626
1627\blank \xmlprocessbuffer{main}{mathml}{} \blank
1628
1629We have been supporting \MATHML\ in \CONTEXT\ for a long time and will continue
1630doing it. I will probably reimplement the converter (given a good reason) using
1631more recent subsystems. It doesn't change the fact that in order to support it,
1632we need to have some robust analytical support macros (functions) to deal with
1633situations as mentioned. The \TEX\ engine is not made for that but in the
1634meantime it has become more easy thanks to a combination of \TEX, \LUA\ and data
1635tables. Consistent availability of extensibles (either or not virtual) helps too.
1636
1637Among the conclusions we can draw is that quite a lot of development (font as
1638well as engine) is driven by what we have had for many years. A generic
1639multi||dimensional glyph handler could have covered all odd cases that used to be
1640done with macros but for historic reasons we could still be stuck with several
1641slightly different and overlapping mechanisms. Nevertheless we can help macro
1642writers by providing for instance leaders that accept characters as well in which
1643case in math mode extensibles can be used.
1644
1645\stopsection
1646
1647\startsection[title=Fences]
1648
1649Fences are symbols that are put left and|/|or right of a formula. They adapt
1650their height and depth to the content they surround, so they are vertical
1651extensibles. Users tend to minimize their coding but this is probably not a good
1652idea with fences as there is some magic involved. For instance, \TEX\ always
1653wants a matching left and right fence, even if one is a phantom. So you will
1654normally have something like this:
1655
1656\starttyping
1657\left\lparent x \right\rparent
1658\stoptyping
1659
1660and when you don't want one of them you use a period:
1661
1662\starttyping
1663\left\lparent x \right.
1664\stoptyping
1665
1666The question is, can we make the users live easier by magically turning braces,
1667brackets and parentheses etc.\ into growing ones. As with much in \MKIV, it could
1668be that \LUA\ can be of help. However, look at the following cases:
1669
1670\startbuffer
1671\startformula (x) \stopformula
1672\stopbuffer
1673
1674\typebuffer \getbuffer
1675
1676This internally becomes something like this:
1677
1678\starttyping
1679open  noad : nucleus : mathchar : U+00028
1680ord   noad : nucleus : mathchar : U+00078
1681close noad : nucleus : mathchar : U+00029
1682\stoptyping
1683
1684We get a linked list of three so|-|called noads where each nucleus is a math
1685character. In addition to a nucleus there can be super- and subscripts.
1686
1687\startbuffer
1688\startformula \mathinner { (x) } \stopformula
1689\stopbuffer
1690
1691\typebuffer \getbuffer
1692
1693\starttyping
1694inner noad : nucleus : submlist :
1695  open  noad : nucleus : mathchar : U+00028
1696  ord   noad : nucleus : mathchar : U+00078
1697  close noad : nucleus : mathchar : U+00029
1698\stoptyping
1699
1700This is still simple, although the inner primitive results in three extra levels.
1701
1702\startbuffer
1703\startformula \left( x \right) \stopformula
1704\stopbuffer
1705
1706\typebuffer \getbuffer
1707
1708Now it becomes more complex, although we can still quite well recognize the
1709input. The question is: how easily can we translate the previous examples into
1710this structure.
1711
1712\starttyping
1713inner noad : nucleus : submlist :
1714  left  fence : delim   : U+00028
1715  ord   noad  : nucleus : mathchar U+00078
1716  right fence : delim   : U+00029
1717\stoptyping
1718
1719\startbuffer
1720\startformula ||x|| \stopformula
1721\stopbuffer
1722
1723\typebuffer \getbuffer
1724
1725Again, we can recognize the sequence in the input:
1726
1727\starttyping
1728ord noad : nucleus : mathchar : U+0007C
1729ord noad : nucleus : mathchar : U+0007C
1730ord noad : nucleus : mathchar : U+00078
1731ord noad : nucleus : mathchar : U+0007C
1732ord noad : nucleus : mathchar : U+0007C
1733\stoptyping
1734
1735Here we would have to collapse the two bars into one. Now, say that we manage to
1736do this, even if it will cost a lot of code to check all border cases, then how
1737about this?
1738
1739\startbuffer
1740\startformula \left|| x \right|| \stopformula
1741\stopbuffer
1742
1743\typebuffer \getbuffer
1744
1745\starttyping
1746inner noad : nucleus : submlist noad :
1747  left  fence : delim : U+00028
1748  ord   noad  : nucleus : mathchar : U+0007C
1749  ord   noad  : nucleus : mathchar : U+00078
1750  right fence : delim : U+00029
1751ord noad : nucleus : mathchar : U+0007C
1752\stoptyping
1753
1754This time we have to look over the sublist and compare the last fence with the
1755character following the sublist. If you keep in mind that there can be all kind
1756of nodes in between, like glue, and that we can have multiple nested fences, it
1757will be clear that this is a no|-|go. Maybe for simple cases it could work out
1758but for a bit more complex math one ends up in constantly fighting asymmetrical
1759input at the \LUA\ end and occasionally fighting the heuristics at the \TEX\ end.
1760
1761It is for this reason that we provide a mechanism that users can use to avoid the
1762primitives \type {\left} and \type {\right}.
1763
1764\startbuffer
1765\setupmathfences
1766  [color=red]
1767
1768\definemathfence
1769  [fancybracket]
1770  [bracket]
1771  [command=yes,
1772   color=blue]
1773
1774\startformula
1775  a \fenced[bar]      {\frac{1}{b}} c \qquad
1776  a \fenced[doublebar]{\frac{1}{b}} c \qquad
1777  a \fenced[triplebar]{\frac{1}{b}} c \qquad
1778  a \fenced[bracket]  {\frac{1}{b}} c \qquad
1779  a \fancybracket     {\frac{1}{b}} c
1780\stopformula
1781\stopbuffer
1782
1783\typebuffer
1784
1785So, you can either use a generic instance of fences (\type {\fenced}) or you
1786can define your own commands. There can be several classes of fences and they
1787can inherit and be cloned.
1788
1789\getbuffer
1790
1791As a bonus \CONTEXT\ provides a few wrappers:
1792
1793\startbuffer
1794\startformula
1795\Lparent    \frac{1}{a} \Rparent    \quad
1796\Lbracket   \frac{1}{b} \Rbracket   \quad
1797\Lbrace     \frac{1}{c} \Rbrace     \quad
1798\Langle     \frac{1}{d} \Rangle     \quad
1799\Lbar       \frac{1}{e} \Rbar       \quad
1800\Ldoublebar \frac{1}{f} \Rdoublebar \quad
1801\Ltriplebar \frac{1}{f} \Rtriplebar \quad
1802\Lbracket   \frac{1}{g} \Rparent    \quad
1803\Langle     \frac{1}{h} \Rnothing
1804\stopformula
1805\stopbuffer
1806
1807\typebuffer
1808
1809which gives:
1810
1811\getbuffer
1812
1813For bars, the same applies as for primes: we collapse them into proper \UNICODE\
1814characters when applicable:
1815
1816\def\Nsbar{\ruledmbox{\singleverticalbar}}
1817\def\Ndbar{\ruledmbox{\doubleverticalbar}}
1818\def\Ntbar{\ruledmbox{\tripleverticalbar}}
1819
1820\starttabulate[|lT|lT|lM|lM|]
1821\NC U+007C \NC \chardescription{"007C} \NC \singleverticalbar \NC \Nsbar \NC \NR
1822\NC U+2016 \NC \chardescription{"2016} \NC \doubleverticalbar \NC \Nsbar \Nsbar \quad
1823                                                                  \Ndbar \NC \NR
1824\NC U+2980 \NC \chardescription{"2980} \NC \tripleverticalbar \NC \Nsbar \Nsbar \Nsbar \quad
1825                                                                  \Nsbar \Ndbar \quad
1826                                                                  \Ndbar \Nsbar \quad
1827                                                                  \Ntbar \NC \NR
1828\stoptabulate
1829
1830The question is always: to what extent do users want to structure their input.
1831For instance, you can define this:
1832
1833\startbuffer
1834\definemathfence [weirdrange] [left="0028,right="005D]
1835\stopbuffer
1836
1837\typebuffer \getbuffer
1838
1839and use it as:
1840
1841\startbuffer
1842$ (a,b] = \fenced[weirdrange]{a,b}$
1843\stopbuffer
1844
1845\typebuffer
1846
1847This gives \inlinebuffer\ and unless you want to apply color or use specific
1848features there is nothing wrong with the direct way. Interesting is that the
1849complications are seldom in regular \TEX\ input, but \MATHML\ is a different
1850story. There is an \type {mfenced} element but as users can also use the more
1851direct route, a bit more checking is needed in order to make sure that we have
1852matching open and close symbols. For reasons mentioned before we cannot delegate
1853this to \LUA\ but have to use special versions of the \type {\left} and \type
1854{\right} commands.
1855
1856One complication of making a nice mechanism for this is that we cannot use the
1857direct characters. For instance curly braces are also used for grouping and the
1858less and equal signs serve different purposes. So, no matter what we come up
1859with, these cases remain special. However, in \CONTEXT\ the following is valid:
1860
1861\startbuffer
1862\setupmathfences[color=darkgreen]
1863\setupmathfences[mirrored][color=darkred]
1864
1865\startformula
1866\left { \frac{1}{a} \right } \quad
1867\left [ \frac{1}{b} \right ] \quad
1868\left ( \frac{1}{c} \right ) \quad
1869\left < \frac{1}{d} \right > \quad
1870\left ⟨ \frac{1}{d} \right ⟩ \quad
1871\left | \frac{1}{e} \right | \quad
1872\left ⟪ \frac{1}{e} \right ⟫ \quad
1873\left ⟫ \frac{1}{e} \right ⟪ \quad
1874\left [ \frac{1}{d} \right [ \quad
1875\left ] \frac{1}{d} \right [ \quad
1876\stopformula
1877\stopbuffer
1878
1879\typebuffer
1880
1881In the background mapping onto the mentioned left and right commands happens so
1882we do get color support as well. And, it doesn't look that bad in your document
1883source either. Of course other combinations are also possible.
1884
1885\start \getbuffer \stop
1886
1887As there are many ways to get fences and users can come from other macro packages
1888(or use them mixed) we support them all as well as possible.
1889
1890\startbuffer
1891\left (       \frac{1}{x} \right )       =
1892      (       \frac{1}{x}        )       =
1893\left\(       \frac{1}{x} \right\)       =
1894     \(       \frac{1}{x}       \)       =
1895\left\lparent \frac{1}{x} \right\rparent =
1896     \lparent \frac{1}{x}       \rparent =
1897     \Lparent \frac{1}{x}       \Rparent
1898\stopbuffer
1899
1900\typebuffer
1901
1902\blank \noindentation $\getbuffer$ \blank
1903
1904Unfortunately \UNICODE\ math doesn't free us from some annoyances with respect to
1905paired fences. On the one hand coding math is a symbolic, abstract matter: a left
1906parenthesis opens something and a right one closes something. The same is true
1907for brackets and braces. However, the bar is used for left and right fencing as
1908well as separating pieces of a formula (e.g.\ in conditions). Because
1909traditionally these left and right bars were purely vertical with no slope, or
1910hooks, or other thingies attached, in \UNICODE\ there is only one slot for it.
1911Where paired fences can play a role in analyzing content, bars are rather useless
1912for that. It also means that when coding a formula one cannot rely on the bar
1913symbol to determine a left or right property. Normally this is no problem as we
1914can use symbolic names (that include the \type {\left} or \type {\right}
1915directive) but for instance in rendering \MATHML\ it demands some fuzzy logic to
1916be applied. It would have been nice to have code points for the three cases.
1917
1918\startbuffer
1919\ruledhbox{$\left|x\right|$}
1920\ruledhbox{$\left(x\middle|x\right)$}
1921\ruledhbox{$\startcheckedfences\left(x\leftorright|x\right)\stopcheckedfences$}
1922\ruledhbox{$\startcheckedfences\leftorright|x\leftorright|\stopcheckedfences$}
1923\ruledhbox{$\startcheckedfences\leftorright|x\stopcheckedfences$}
1924\ruledhbox{$\startcheckedfences\left(x\leftorright|\stopcheckedfences$}
1925\stopbuffer
1926
1927\typebuffer
1928
1929Believe me: we run into any combination of these bars and parentheses. And we're
1930no longer surprised to see code like this (generated from applications):
1931
1932\starttyping
1933<math>
1934  <mrow>
1935    <mo>(</mo>
1936    <mi>y</mi>
1937    <mrow>
1938      <mo>|</mo>
1939    </mrow>
1940    <mi>y</mi>
1941    <mo>)</mo>
1942  </mrow>
1943</math>
1944\stoptyping
1945
1946Here the bar sits in its own group, so what is it? A lone left, right or middle
1947symbol, meant to stretch with the surroundings or not?
1948
1949To summarize: there is no real difference (or progress) with respect to fences in
1950\LUATEX\ compared to traditional \TEX. We still need matching \type {\left} and
1951\type {\right} usage and catching mismatches automatically is hard. By adding
1952some hooks at the \TEX\ end we can easily check for a missing \type {\right} but
1953a missing \type {\left} needs a two|-|pass approach. Maybe some day in \CONTEXT\
1954we will end up with multipass math processing and then I'll look into this again.
1955
1956\stopsection
1957
1958\startsection[title=Directions]
1959
1960The first time I saw right|-|to|-|left math was at a Dante and later at a TUG
1961meeting hosted in Morocco where Azzeddine Lazrek again demonstrated
1962right|-|to|-|left math. It was only after Khaled Hosny added some support to the
1963\XITS\ font that I came to supporting it in \CONTEXT. Apart from some
1964housekeeping nothing special is needed: the engine is ready for it. Of course it
1965would be nice to extend the lm and gyre fonts as well but currently it's not on
1966the agenda. I expect to add some more control and features in the future, if only
1967because it is a nice visual experience. And writing code for such features is
1968kind of fun.
1969
1970As this is about as complex as it can gets, it makes a nice example of how we
1971control math font definitions, so let's see how we can define a \XITS\ use case.
1972Because we have a bold (heavy) font too, we define that as well. First we define
1973the two fonts.
1974
1975\starttyping
1976\starttypescript [math] [xits,xitsbidi] [name]
1977  \loadfontgoodies [xits-math]
1978  \definefontsynonym
1979    [MathRoman]
1980    [file:xits-math.otf]
1981    [features=math\mathsizesuffix,goodies=xits-math]
1982  \definefontsynonym
1983    [MathRomanBold]
1984    [file:xits-mathbold.otf]
1985    [features=math\mathsizesuffix,goodies=xits-math]
1986\stoptypescript
1987\stoptyping
1988
1989Discussing font goodies is beyond this article so I stick to a simple
1990explanation. We use so|-|called goodie files for setting special properties of
1991fonts, but also for defining special treatment, for instance runtime patches. The
1992current \type {xits-math} goodie file looks as follows:
1993
1994\starttyping
1995return {
1996  name = "xits-math",
1997  version = "1.00",
1998  comment = "Goodies that complement xits (by Khaled Hosny).",
1999  author = "Hans Hagen",
2000  copyright = "ConTeXt development team",
2001  mathematics = {
2002    italics = {
2003      ["xits-math"] = {
2004         defaultfactor = 0.025,
2005         disableengine = true,
2006         corrections   = {
2007           [0x1D453] = -0.0375, -- f
2008        },
2009      },
2010    },
2011    alternates = {
2012      cal       = { feature = 'ss01', value = 1,
2013        comment = "Mathematical Calligraphic Alphabet" },
2014      greekssup = { feature = 'ss02', value = 1,
2015        comment = "Mathematical Greek Sans Serif Alphabet" },
2016      greekssit = { feature = 'ss03', value = 1,
2017        comment = "Mathematical Italic Sans Serif Digits" },
2018      monobfnum = { feature = 'ss04', value = 1,
2019        comment = "Mathematical Bold Monospace Digits" },
2020      mathbbbf  = { feature = 'ss05', value = 1,
2021        comment = "Mathematical Bold Double-Struck Alphabet" },
2022      mathbbit  = { feature = 'ss06', value = 1,
2023        comment = "Mathematical Italic Double-Struck Alphabet" },
2024      mathbbbi  = { feature = 'ss07', value = 1,
2025        comment = "Mathematical Bold Italic Double-Struck Alphabet" },
2026      upint     = { feature = 'ss08', value = 1,
2027        comment = "Upright Integrals" },
2028      vertnot   = { feature = 'ss09', value = 1,
2029        comment = "Negated Symbols With Vertical Stroke" },
2030    },
2031  }
2032}
2033\stoptyping
2034
2035There can be many more entries but here the most important one is the \type
2036{alternates} table. It defines the additional styles available in the font.
2037Alternaties are chosen using commands like
2038
2039\starttyping
2040\mathalternate{cal}\cal
2041\stoptyping
2042
2043and of course shortcuts for this can be defined.
2044
2045Of course there is more than math, so we define a serif collection too:
2046
2047\starttyping
2048\starttypescript [serif] [xits] [name]
2049  \setups[font:fallback:serif]
2050  \definefontsynonym[Serif]          [xits-regular.otf]    [features=default]
2051  \definefontsynonym[SerifBold]      [xits-bold.otf]       [features=default]
2052  \definefontsynonym[SerifItalic]    [xits-italic.otf]     [features=default]
2053  \definefontsynonym[SerifBoldItalic][xits-bolditalic.otf] [features=default]
2054\stoptypescript
2055\stoptyping
2056
2057If needed you can redefine the \type {default} feature before this typescript is
2058used. Once we have the fonts defined we can start building a typeface:
2059
2060\starttyping
2061\starttypescript[xits]
2062  \definetypeface [xits] [rm] [serif] [xits]   [default]
2063  \definetypeface [xits] [ss] [sans]  [heros]  [default] [rscale=0.9]
2064  \definetypeface [xits] [tt] [mono]  [modern] [default] [rscale=1.05]
2065  \definetypeface [xits] [mm] [math]  [xits]   [default]
2066\stoptypescript
2067\stoptyping
2068
2069We can now switch to this typeface with:
2070
2071\starttyping
2072\setupbodyfont[xits]
2073\stoptyping
2074
2075But, as we wanted bidirectional math, something more is needed. Instead of the
2076two fonts we define six. We could have a more abstract reference to the \XITS\
2077fonts but in cases like this we prefer file names because then at least we can be
2078sure that we get what we ask for.
2079
2080\starttypescript [math] [xits,xitsbidi] [name]
2081  \loadfontgoodies[xits-math]
2082  \definefontsynonym[MathRoman]       [xits-math.otf]    [features=math\mathsizesuffix,goodies=xits-math]
2083  \definefontsynonym[MathRomanL2R]    [xits-math.otf]    [features=math\mathsizesuffix-l2r,goodies=xits-math]
2084  \definefontsynonym[MathRomanR2L]    [xits-math.otf]    [features=math\mathsizesuffix-r2l,goodies=xits-math]
2085  \definefontsynonym[MathRomanBold]   [xits-mathbold.otf][features=math\mathsizesuffix,goodies=xits-math]
2086  \definefontsynonym[MathRomanBoldL2R][xits-mathbold.otf][features=math\mathsizesuffix-l2r,goodies=xits-math]
2087  \definefontsynonym[MathRomanBoldR2L][xits-mathbold.otf][features=math\mathsizesuffix-r2l,goodies=xits-math]
2088\stoptypescript
2089
2090So, we use the same fonts several times but apply different features to them.
2091This time the typeface definition explicitly turns on both directions. When we
2092don't do that we get only left to right support, which is of course more
2093efficient in terms of font usage.
2094
2095\starttypescript[xitsbidi]
2096  \definetypeface [xitsbidi] [rm] [serif] [xits]     [default]
2097  \definetypeface [xitsbidi] [ss] [sans]  [heros]    [default] [rscale=0.9]
2098  \definetypeface [xitsbidi] [tt] [mono]  [modern]   [default] [rscale=1.05]
2099  \definetypeface [xitsbidi] [mm] [math]  [xitsbidi] [default] [direction=both]
2100\stoptypescript
2101
2102We can now switch to the bidirectional typeface with:
2103
2104\starttyping
2105\setupbodyfont[xitsbidi]
2106\stoptyping
2107
2108However, in order to get bidirectional math indeed, we need to turn it on.
2109
2110\starttyping
2111\setupmathematics[align=r2l]
2112\stoptyping
2113
2114You might have wondered what this special way of defining the features using
2115\type {\mathsizesuffix} means? The value of this macro is set at font definition
2116time, and can be one of three values: \type {text}, \type {script} and \type
2117{scriptscript}. At this moment the features are defined as follows:
2118
2119\starttyping
2120\definefontfeature
2121  [mathematics]
2122  [mode=base,
2123   liga=yes,
2124   kern=yes,
2125   tlig=yes,
2126   trep=yes,
2127   mathalternates=yes,
2128   mathitalics=yes,
2129 % nomathitalics=yes, % don't pass to tex
2130   language=dflt,
2131   script=math]
2132\stoptyping
2133
2134From this we clone:
2135
2136\starttyping
2137\definefontfeature
2138  [mathematics-l2r]
2139  [mathematics]
2140  []
2141
2142\definefontfeature
2143  [mathematics-r2l]
2144  [mathematics]
2145  [language=ara,
2146   rtlm=yes,
2147   locl=yes]
2148\stoptyping
2149
2150Watch how we enable two specific features, where \type {rtlm} is a \XITS|-|specific
2151one. The eventually used features are defined as follows.
2152
2153\starttyping
2154\definefontfeature[math-text]            [mathematics]    [ssty=no]
2155\definefontfeature[math-script]          [mathematics]    [ssty=1,mathsize=yes]
2156\definefontfeature[math-scriptscript]    [mathematics]    [ssty=2,mathsize=yes]
2157
2158\definefontfeature[math-text-l2r]        [mathematics-l2r][ssty=no]
2159\definefontfeature[math-script-l2r]      [mathematics-l2r][ssty=1,mathsize=yes]
2160\definefontfeature[math-scriptscript-l2r][mathematics-l2r][ssty=2,mathsize=yes]
2161
2162\definefontfeature[math-text-r2l]        [mathematics-r2l][ssty=no]
2163\definefontfeature[math-script-r2l]      [mathematics-r2l][ssty=1,mathsize=yes]
2164\definefontfeature[math-scriptscript-r2l][mathematics-r2l][ssty=2,mathsize=yes]
2165\stoptyping
2166
2167Even if it is relatively simple to do, it makes no sense to build complex mixed
2168mode system, so currently we have to decide before we typeset a formula:
2169
2170\startbuffer
2171\setupmathematics[align=l2r]
2172\startformula
2173       \sqrt{x^2\over 4x}  \qquad
2174  {\bf \sqrt{x^2\over 4x}} \qquad
2175  {\mb \sqrt{x^2\over 4x}}
2176\stopformula
2177\stopbuffer
2178
2179\typebuffer
2180
2181This gives a left to right formula:
2182
2183\getbuffer
2184
2185\startbuffer
2186\setupmathematics[align=r2l]
2187\startformula
2188       \sqrt{ف^2\over 4ب}  \qquad
2189  {\bf \sqrt{ف^2\over 4ب}} \qquad
2190  {\mb \sqrt{ف^2\over 4ب}}
2191\stopformula
2192\stopbuffer
2193
2194\typebuffer
2195
2196And here we get an Arabic formula, where the quality of course is determined
2197by the completeness of the font.
2198
2199\start
2200\switchtobodyfont[xitsbidi]
2201\getbuffer
2202\stop
2203
2204The bold font has a partial bold implementation so unless I implement a more
2205complex pseudo|-|bold mechanism you should not expect results. Because we have no
2206official Arabic math alphabets they are not seen by the \CONTEXT\ \MKIV\
2207analyzers that normally take care of this. It's all a matter of demand and supply
2208(combined with a dose of motivation). For instance while a base size might be
2209covered, the extensibles might be missing.
2210
2211About the time of writing this another variation was requested at the mailing
2212list. For Persian math we keep the direction from left to right but the digits
2213have to be in an Arabic font. We cannot use the bidirectional handler for this so
2214we need to swap regular and bold digits in another way. We can use the fallback
2215mechanism for this and a definition roughly boils down to this:
2216
2217\starttyping
2218\definefontfallback
2219  [mathdigits]
2220  [dejavusansmono]
2221  [digitsarabicindic]
2222  [check=yes,
2223   force=yes,
2224   offset=digitsnormal]
2225\stoptyping
2226
2227This is used in:
2228
2229\starttyping
2230\definefontsynonym
2231  [MathRoman]
2232  [file:xits-math.otf]
2233  [features=math\mathsizesuffix,
2234   goodies=xits-math,
2235   fallbacks=mathdigits]
2236\stoptyping
2237
2238The problem with this kind of feature is not so much in the implementation,
2239because by now in \CONTEXT\ we have plenty of ways to deal with such issues in a
2240convenient way. The biggest challenge is to come up with an interface that
2241somehow fits in the model of typescripts and with a couple of predefined
2242typescripts we now have:
2243
2244\starttyping
2245\usetypescriptfile[mathdigits]
2246\usetypescript [mathdigits] [xits-dejavu] [arabicindic]
2247\setupbodyfont[dejavu]
2248\stoptyping
2249
2250\startbuffer[pefama]
2251\definefontfeature [persian-fake-math] [arabic] [anum=yes]
2252
2253\definefont[persianfakemath][dejavusans*persian-fake-math]
2254\stopbuffer
2255
2256\getbuffer[pefama]
2257
2258\def\PeFaMa#1{\mathord{\hbox{\persianfakemath#1}}}
2259
2260After that a formula like \type {$2 + 3 = 5$} comes out as $ \PeFaMa2 + \PeFaMa3
2261= \PeFaMa5 $. In fact, if you want that in text mode, you can just use the
2262\CONTEXT\ \MKIV\ font feature \type {anum}:
2263
2264\typebuffer[pefama]
2265
2266But of course you won't have proper math then. But as right|-|to|-|left math is
2267still under construction, in due time we might end up with more advanced
2268rendering. Currently you can exercise a little control. For instance by using the
2269\type {align} parameter in combination with the \type {bidi} parameter. Of course
2270support for special symbols like square roots depends on the font as well. We
2271probably need to mirror a few more characters.
2272
2273\startbuffer
2274  \m{   (  1 =   1)   }\quad
2275  \m{   (123 = 123)   }\quad
2276  \m{ a (  1 =   1) b }\quad
2277  \m{ a (123 = 123) b }\quad
2278  \m{ x = 123 y + (1 / \sqrt {x}) }
2279\stopbuffer
2280
2281\typebuffer
2282
2283As in math we can assume sane usage of fences, we don't need extensive tests on
2284pairing.
2285
2286\starttabulate[|T|T||]
2287\HL
2288\NC \rm\bf align \NC \rm\bf bidi \NC                                   \NC \NR
2289\HL
2290\NC l2r   \NC no   \NC \setupmathematics          [bidi=no]\getbuffer  \NC \NR
2291\NC l2r   \NC yes  \NC \setupmathematics          [bidi=yes]\getbuffer \NC \NR
2292\NC r2l   \NC no   \NC \setupmathematics[align=r2l,bidi=no]\getbuffer  \NC \NR
2293\NC r2l   \NC yes  \NC \setupmathematics[align=r2l,bidi=yes]\getbuffer \NC \NR
2294\HL
2295\stoptabulate
2296
2297\stopsection
2298
2299\startsection[title=Structure]
2300
2301At some point publishers started asking for tagged \PDF\ and as a consequence a
2302typeset math formula suddenly becomes more than a blob of ink. There are several
2303arguments for tagging content. One is accessibility and another is reflow.
2304Personally I think that both arguments are not that relevant. For instance, if
2305you want to help a visually impaired reader, it's far better to start from a well
2306structured original and ship that along with the typeset version. And, if you
2307want reflow, you can better provide a (probably) simplified version in for
2308instance \HTML\ format.
2309
2310We are surrounded by all kinds of visualizations, and text on paper or some
2311medium is one. We don't make a painting accessible either. If accessibility is a
2312demand, it should be done as best as can be, and the source is then the starting
2313point. Of course publishers don't like that because when a source is available,
2314it's one step closer to reuse by others. But that problem can simply be ignored
2315as we consider publishers to be some kind of facilitating organization that
2316deliver content from others. Alas publishers don't play that humble role so as
2317long as they're around they can demand from their suppliers tagging of something
2318visual.
2319
2320Of course when you use \TEX\ tagging is no real issue as you can make the input
2321as verbose and structured as you like. But authors don't always want to be
2322verbose, take this:
2323
2324\startbuffer
2325$ f(x) = x^2 + 3x + 7 $
2326\stopbuffer
2327
2328\typebuffer
2329
2330This enters \TEX\ as a sequence of characters: \enabletrackers [math.classes]
2331\inlinebuffer \disabletrackers[math.classes]. These characters can have
2332properties, for instance they can represent a relation or be an opening or
2333closing symbol, but in most cases they are just classified as ordinary. These
2334properties to some extent control spacing and interplay between math elements.
2335They are not structure. If you have seen presentation \MATHML\ you have noticed
2336that there are operators (\type {mo}), identifiers (\type {mi}) and numbers
2337(\type {mn}), as well as some structural elements like fences (\type {mfenced}),
2338superscripts (\type {msup}), subscripts (\type {msub}). Because it is a
2339presentational encoding, there is no guarantee about the quality of the input as
2340well as the rendering, but it somehow made it into a standard that is also used
2341for tagging \PDF\ content.
2342
2343Going from mostly unstructured \TEX\ math input to more structured output is
2344complicated by the fact that the intermediate somewhat structured math lists
2345eventually become regular boxes, glyphs, kerns, glue etc. In \CONTEXT\ we carry
2346some persistent information around so that we can still reverse engineer the
2347output to structured input but this can be improved by more explicit tagging. We
2348plan to add some more of that to future versions but here is an example:
2349
2350\starttyping
2351$ \apply{f}{(x)} = x^2 + 3x + 7 $
2352\stoptyping
2353
2354You can go over the top too:
2355
2356\starttyping
2357$ \apply{f}{(x)} = \mi{x}^\mi{2} + \mi{3}\mi{x} + \mi{7} $
2358\stoptyping
2359
2360The trick is to find an optimal mix of structure and readability. For instance,
2361in \type {\sin} we already have the apply done by default, so often extra tagging
2362is only needed in situations where there are several ways to interpret the text.
2363Of course we're not enforcing this, but by providing some structure related
2364features, at least we hope to make users aware of the issue. Directly inputting
2365\MATHML\ is also an option but has never become popular.
2366
2367All this is mostly a macro package issue, and \CONTEXT\ has the basics on board.
2368Because there is no need to adapt \LUATEX\ the most we will do is add a bit more
2369consistency in building the lists (two way pointers) and carrying over properties
2370(like attributes). We also have on the agenda a math table model that suits
2371\MATHML, because some of those tables are somewhat hard to deal with.
2372
2373How the export and tagging evolves depends on demand. I must admit that I
2374implemented it as an exercise mostly because these are features I don't need
2375myself (and no one really asked for it anyway).
2376
2377\stopsection
2378
2379\startsection[title=Italic correction]
2380
2381Here we face a special situation. In regular \OPENTYPE\ italic correction is not
2382part of the game, although one can cook up some positioning feature that does a
2383similar job. In \OPENTYPE\ math there is italic correction, but also a more
2384powerful sharpe|-|related kerning which is to be preferred. In traditional \TEX\
2385the italic correction was present but since it is a font specific feature there
2386is no way to make it work across fonts, and \TYPEONE\ based math has lots of
2387them.
2388
2389At some point we have discussed throwing italic correction out of the engine, if
2390only because it was unclear how and when to apply it. In the meantime there is
2391some compromise reached. Because \CONTEXT\ is always in sync with the latest
2392\LUATEX, we oscillated between solutions and this was complicated by the fact
2393that we had to support a mix of \OPENTYPE\ math fonts and virtualized \TYPEONE\
2394legacy fonts.
2395
2396The italic correction related code is still somewhat experimental, but we have
2397several options. \footnote {In text mode we also have an advanced mechanism for
2398italic correction but this operates independent from math.} In most cases we
2399insert the italic correction ourselves and as the engine then sees a kern already
2400it will not add another one. This has the advantage that we can be more
2401consistent if only because not all fonts have these corrections and not all cases
2402are considered by the engine.
2403
2404\startitemize[n]
2405    \startitem
2406        A math font can have italic correction per glyph. The engine gets
2407        this passed but before it can apply them we already inject them into
2408        the mathlist where needed.
2409    \stopitem
2410    \startitem
2411        This is a variant of the first one, but is always applied, and not
2412        controlled by the font. This makes it possible to add additional
2413        corrections. This method is kind of obsolete as we no longer generate
2414        missing corrections at font definition time. \footnote {Because the
2415        font loader is also used for the generic code, we don't want to add
2416        such features there.}
2417    \stopitem
2418    \startitem
2419        This variant looks at the shape and if it is italic (or bolditalic) then
2420        correction is applied. Here the correction is related to the emwidth
2421        and controlled by a factor. We use this method by default.
2422    \stopitem
2423    \startitem
2424        The fourth variant is a mixture of the first (font driven) and the third
2425        (emwidth driven).
2426    \stopitem
2427\stopitemize
2428
2429Are we better off? I honestly don't know. It is a bit of a mess and will always
2430be, simply because the reference font (cambria) and reference implementation
2431(msword) is not clear about it and we follow them. In that respect I consider it
2432a macro package issue mostly. In \CONTEXT\ at least we can offer some options.
2433
2434\startsection[title=Big]
2435
2436When migrating math to \MKIV\ I couldn't resist looking into some functionality
2437that currently uses macro magic. An example is big delimiters.
2438
2439\startbuffer[bigs]
2440$ ( \big( \Big( \bigg( \Bigg( x $
2441\stopbuffer
2442
2443\typebuffer[bigs]
2444
2445\blank \getbuffer[bigs] \blank
2446
2447Personally I never use these, I just trust \type {\left} and \type {\right} to do
2448the right job, but I'm no reference at all when it comes to math. The reason for
2449looking into the bigs is that in plain \TEX\ there are some magic numbers
2450involved. The macros, when translated to \CONTEXT\ boil down to this:
2451
2452\starttyping
2453\left<delimiter>\vbox to 0.85\bodyfontsize{}\right.
2454\left<delimiter>\vbox to 1.15\bodyfontsize{}\right.
2455\left<delimiter>\vbox to 1.45\bodyfontsize{}\right.
2456\left<delimiter>\vbox to 1.75\bodyfontsize{}\right.
2457\stoptyping
2458
2459Knowing that we have a chain of sizes in the font, I was tempted to go for a
2460solution where a specific size is chosen from the linked list of next sizes.
2461There are several strategies possible when we delegate this to \LUA\ but we don't
2462provide a high level interface yet. Personally I'd like to set the low level
2463configuration options as:
2464
2465\starttyping
2466\setconstant\bigmathdelimitermethod \plusone
2467\setconstant\bigmathdelimitervariant\plusthree
2468\stoptyping
2469
2470But as users might expect plain||like behaviour, \CONTEXT\ also provides the command
2471
2472\starttyping
2473\plainbigdelimiters
2474\stoptyping
2475
2476which sets the method to~2. Currently that is the default. When method~1 is
2477chosen there are four variants and the reason for keeping them all is that they
2478are part of experiments and explorations.
2479
2480\starttabulate[|||]
2481\NC 1 \NC choose size $ \tf  n $ from the available sizes \NC \NR
2482\NC 2 \NC choose size $ \tf 2n $ from the available sizes \NC \NR
2483\NC 3 \NC choose the first variant that has $ \tf 1.33^n \times (ht + dp) > size $\NC \NR
2484\NC 4 \NC choose the first variant that has $ \tf 1.33^n \times bodyfontsize > size $\NC \NR
2485\stoptabulate
2486
2487The last three variants give similar results but they are not always the same as
2488the plain method. This is because not all fonts provide the same range.
2489
2490\def\SetBig#1#2%
2491  {\setnewconstant\bigmathdelimitermethod#1\relax
2492   \setnewconstant\bigmathdelimitervariant#2\relax
2493   \getbuffer[bigs]}
2494
2495\starttabulate[|l|l|l|l|]
2496\HL
2497\NC           \NC pagella       \NC \switchtobodyfont[modern] latin modern  \NC \switchtobodyfont[cambria] cambria       \NC \NR
2498\HL
2499\NC plain     \NC \SetBig{2}{0} \NC \switchtobodyfont[modern] \SetBig{2}{0} \NC \switchtobodyfont[cambria] \SetBig{2}{0} \NC \NR
2500\NC variant 1 \NC \SetBig{1}{1} \NC \switchtobodyfont[modern] \SetBig{1}{1} \NC \switchtobodyfont[cambria] \SetBig{1}{1} \NC \NR
2501\NC variant 2 \NC \SetBig{1}{2} \NC \switchtobodyfont[modern] \SetBig{1}{2} \NC \switchtobodyfont[cambria] \SetBig{1}{2} \NC \NR
2502\NC variant 3 \NC \SetBig{1}{3} \NC \switchtobodyfont[modern] \SetBig{1}{3} \NC \switchtobodyfont[cambria] \SetBig{1}{3} \NC \NR
2503\NC variant 4 \NC \SetBig{1}{4} \NC \switchtobodyfont[modern] \SetBig{1}{4} \NC \switchtobodyfont[cambria] \SetBig{1}{4} \NC \NR
2504\HL
2505\stoptabulate
2506
2507So, we are somewhat unpredictable but at least we have several ways to control
2508the situation and better solutions might show up.
2509
2510% \dontleavehmode\dostepwiserecurse{0}{6}{1}{\ruledhbox{$\mathdelimiterstep{#1}($} }
2511
2512\stopsection
2513
2514\startsection[title=Macros]
2515
2516I already discussed roots and the traditional \type {\root} command is a nice
2517example of one that can be simplified in \LUATEX\ thanks to a new primitive. A
2518macro package often has quite a lot of macros related to math that deal with
2519tables and \LUATEX\ doesn't change that. But there is a category of commands that
2520became obsolete: the ones that are used to construct characters that are not in
2521the fonts. Keep in mind that the number of fonts as well as their size was
2522limited at the time \TEX\ was written, so by providing building blocks additional
2523characters could be made. Think of for instance the negated symbols: a new symbol
2524could be made by overlaying a slash. The same is true for arrows: by prepending
2525or appending minus signs, arrows of arbitrary length could be constructed.
2526
2527Here I will stick to another example: dots. In plain \TEX\ we have this definition:
2528
2529\starttyping
2530\def\vdots
2531  {\vbox
2532     {\baselineskip4pt
2533      \lineskiplimit0pt
2534      \kern6pt
2535      \hbox{.}%
2536      \hbox{.}%
2537      \hbox{.}}}
2538\stoptyping
2539
2540This will typeset vertical dots, while the next does them diagonally:
2541
2542\starttyping
2543\def\ddots
2544  {\mathinner
2545     {\mkern1mu
2546      \raise7pt\vbox{\kern7pt\hbox{.}}%
2547      \mkern2mu
2548      \raise4pt\hbox{.}%
2549      \mkern2mu
2550      \raise1pt\hbox{.}%
2551      \mkern1mu}}
2552\stoptyping
2553
2554Of course these dimensions relate to the font size of plain \TEX\ so in \CONTEXT\
2555\MKII\ we have something like this:
2556
2557\startbuffer
2558\def\vdots
2559  {\vbox
2560     {\baselineskip4\points
2561      \lineskiplimit\zeropoint
2562      \kern6\points
2563      \hbox{$\mathsurround\zeropoint.$}%
2564      \hbox{$\mathsurround\zeropoint.$}%
2565      \hbox{$\mathsurround\zeropoint.$}}}
2566
2567\def\ddots
2568  {\mathinner
2569     {\mkern1mu
2570      \raise7\points\vbox{\kern 7\points\hbox{$\mathsurround\zeropoint.$}}%
2571      \mkern2mu
2572      \raise4\points\hbox{$\mathsurround\zeropoint.$}%
2573      \mkern2mu
2574      \raise \points\hbox{$\mathsurround\zeropoint.$}%
2575      \mkern1mu}}
2576\stopbuffer
2577
2578\typebuffer
2579
2580These two symbols are rendered (in \MKII) as follows:
2581
2582\start \getbuffer
2583
2584\startlinecorrection[blank]
2585\dontleavehmode \quad \ruledhbox{$\vdots$} \quad \ruledhbox{$\ddots$}
2586\stoplinecorrection
2587
2588\stop
2589
2590I must admit that I only noticed the rather special height when I turned these
2591macros into virtual characters for the initial virtual \UNICODE\ math that we
2592needed in the first versions of \MKIV. This is a side effect of their use in
2593matrices. However, in \MKIV\ we just use the characters in the font and get:
2594
2595\startlinecorrection[blank]
2596\dontleavehmode \quad \ruledhbox{$\vdots$} \quad \ruledhbox{$\ddots$}
2597\stoplinecorrection
2598
2599These characters look different because instead of three text periods a real
2600symbol is used. The fact that we have more complete fonts and rely less on
2601special font properties to achieve effects is a good thing, and in this respect
2602it cannot be denied that \LUATEX\ triggered the development of more complete
2603fonts. Of course from the user's perspective the outcome is often the same,
2604although \unknown\ using a single character instead of three has the advantage of
2605smaller files (neglectable), less runtime (really neglectable) and cleaner output
2606files (undeniable) from where such characters can now be copied as one.
2607
2608\stopsection
2609
2610\startsection[title=Unscripting]
2611
2612If you ever looked into plain \TEX\ you might have noticed this following
2613section. The symbols are more related to programming languages than to math.
2614
2615\starttyping
2616% The following changes define internal codes as recommended
2617% in Appendix C of The TeXbook:
2618\mathcode`\^^@="2201 % \cdot
2619\mathcode`\^^A="3223 % \downarrow
2620\mathcode`\^^B="010B % \alpha
2621\mathcode`\^^C="010C % \beta
2622\mathcode`\^^D="225E % \land
2623\mathcode`\^^E="023A % \lnot
2624\mathcode`\^^F="3232 % \in
2625\mathcode`\^^G="0119 % \pi
2626\mathcode`\^^H="0115 % \lambda
2627\mathcode`\^^I="010D % \gamma
2628\mathcode`\^^J="010E % \delta
2629\mathcode`\^^K="3222 % \uparrow
2630\mathcode`\^^L="2206 % \pm
2631\mathcode`\^^M="2208 % \oplus
2632\mathcode`\^^N="0231 % \infty
2633\mathcode`\^^O="0140 % \partial
2634\mathcode`\^^P="321A % \subset
2635\mathcode`\^^Q="321B % \supset
2636\mathcode`\^^R="225C % \cap
2637\mathcode`\^^S="225B % \cup
2638\mathcode`\^^T="0238 % \forall
2639\mathcode`\^^U="0239 % \exists
2640\mathcode`\^^V="220A % \otimes
2641\mathcode`\^^W="3224 % \leftrightarrow
2642\mathcode`\^^X="3220 % \leftarrow
2643\mathcode`\^^Y="3221 % \rightarrow
2644\mathcode`\^^Z="8000 % \ne
2645\mathcode`\^^[="2205 % \diamond
2646\mathcode`\^^\="3214 % \le
2647\mathcode`\^^]="3215 % \ge
2648\mathcode`\^^^="3211 % \equiv
2649\mathcode`\^^_="225F % \lor
2650\stoptyping
2651
2652This means as much as: when I hit \type {Ctrl-Z} on my keyboard and my editor
2653honors that by injecting character \type {U+1A} into the input then \TEX\ will
2654turn that into $\ne$, given that you're in math mode. I'm not sure how many
2655keyboards and editors there are around that still do that but it illustrates that
2656inputting in some kind of \WYSIWYG\ is not alien to \TEX. \footnote {There are
2657more such hidden features, for instance, in some fonts special ligatures can be
2658implemented that no one ever uses.}
2659
2660One of the subprojects of the ongoing \TEX\ user group font project is to extend
2661the already extensive Dejavu font with all relevant math characters so that we
2662can edit a document in a more \UNICODE\ savvy way. So, after more than three
2663decades we might arrive where Don Knuth started: you see what you input and a
2664similar shape will end up on paper.
2665
2666Does this mean that all such input is good? Definitely not, because in \UNICODE\
2667we find all kinds of characters that somehow ended up there as a result of
2668merging existing encodings. At work we're accustomed to getting input that is a
2669mix of everything a word processor can produce and often we run into characters
2670that users find normal but are not that handy from a \TEX\ perspective. It's the
2671main reason why in math mode we intercept some of them, for instance in:
2672
2673\startbuffer
2674$ y = x² + x³ + x²³ + x²ᵃ $ % not all characters are in monospace
2675\stopbuffer
2676
2677\typebuffer
2678
2679These superscripts are an inconsistent bunch so they will never be real
2680substitutes for the \type {^} syntax, simply because a mix like above looks bad.
2681But fortunately it comes out well: \inlinebuffer. This is because \CONTEXT\ will
2682transform such super- and subscripts into real ones and in the process also
2683collapse multiple scripts into a group. This is typically one of the features
2684that already showed up early in \MKIV.
2685
2686Here we have a feature that doesn't relate to fonts, the math machinery or the
2687engine, but is just a macro package goodie. It's a way to respond to the
2688variation in input, although probably hardly any \TEX\ math user will need it.
2689It's one of those features that comes in handy when you use \TEX\ as invisible
2690backend where the input is never seen by humans.
2691
2692\stopsection
2693
2694\startsection[title=Combining fonts]
2695
2696I already mentioned that we started out with virtual math fonts. Defining them is
2697not that hard and boils down to defining what fonts make up the desired math
2698font. Normally one starts out with a decent complete \OPENTYPE\ math font
2699followed by mapping \TYPEONE\ fonts onto specific alphabets and symbols. On top
2700of this there are additional virtual characters constructed (including
2701extensibles). However, this method will become kind of obsolete (read: not used)
2702when all relevant \OPENTYPE\ math fonts are available.
2703
2704Does this mean that we have only simple font setups? In practice yes: you can set
2705up a math font in a few lines in a regular typescript. There are of course a few
2706more lines needed when defining bold and|/|or right|-|to|-|left math but users
2707don't need to bother about it. All is predefined. There are signals that users
2708want to combine fonts so the already present fallback mechanism for text fonts
2709has been made to work with math fonts as well. This permits for instance to
2710complement the not|-|yet|-|finished \OPENTYPE\ Euler math fonts with Pagella. Of
2711course you always need to keep consistency into account, but in principle you can
2712overload for instance specific alphabets, something that can make sense when
2713simple math is mixed with a font that has no math companion. In that case using
2714the text italic in math mode might look better. For the at the time of this
2715writing incomplete Euler font we can add characters like this:
2716
2717\starttyping
2718\loadtypescriptfile[texgyre]
2719\loadtypescriptfile[dejavu]
2720
2721\resetfontfallback  [euler]
2722
2723\definefontfallback [euler] [texgyrepagella-math] [0x02100-0x02BFF]
2724\definefontfallback [euler] [texgyrepagella-math] [0x1D400-0x1D7FF]
2725
2726\starttypescript [serif] [euler] [name]
2727  \setups[font:fallback:serif]
2728  \definefontsynonym [Serif] [euler] [features=default]
2729\stoptypescript
2730
2731\starttypescript [math] [euler] [name]
2732  \definefontsynonym [MathRoman] [euler] [features=math\mathsizesuffix,fallbacks=euler]
2733\stoptypescript
2734
2735\starttypescript [euler]
2736  \definetypeface [\typescriptone] [rm] [serif] [euler]  [default]
2737  \definetypeface [\typescriptone] [tt] [mono]  [dejavu] [default] [rscale=0.9]
2738  \definetypeface [\typescriptone] [mm] [math]  [euler]  [default]
2739\stoptypescript
2740\stoptyping
2741
2742If needed one can use names instead of code ranges (like \type {uppercasescript})
2743as well as map one range onto another. This last option is handy for merging a
2744regular text font into an alphabet (in which case the \UNICODE's don't match).
2745
2746We expect math fonts to be rather complete because after all, a font designer has
2747a large repertoire of free alphabets to choose from. So, in practice combining
2748math fonts will happen seldom. In text mode this is more common, especially when
2749multiple scripts are mixed. There is a whole bunch of modules that can generate
2750all kind of tables and overviews for testing.
2751
2752\stopsection
2753
2754\startsection[title=Experiments]
2755
2756I won't describe all experiments here. An example of an experiment is a better
2757way of dealing with punctuation, especially the cultural determined
2758period|/|comma treatment. I still have the code somewhere but the heuristics are
2759too messy to keep around.
2760
2761There are also some planned experiments, like breaking and aligning display math,
2762but they have a low priority. It's not that hard to do, but I need a good reason.
2763The same is true for equation number placement where primitives are used that can
2764sometimes interfere or not be used in all cases. Currently that placement in
2765combination with alignments is implemented with quite a lot of fuzzy macro code.
2766
2767One of the areas where experimenting will continue is with fonts. Early in the
2768development of \MKIV\ font goodies showed up. A font (or collection of fonts) can
2769have a file (or more files) that control functionality and can have fixes. There
2770are some in place for math fonts. It is a convenient way to use the latest
2771greatest fonts as we have ways to circumvent issues, for instance with math
2772parameters. The virtual math fonts are also defined as goodies.
2773
2774Some mechanisms will probably be made accessible from the \TEX\ end so that users
2775can exercise more control. And because we're not done yet, additional features
2776will show up for sure. There are some math related subsystems like physics and
2777chemistry and these already demanded some extensions and might need more.
2778Introducing math symbol (and property) dictionaries as in \OPENMATH\ is probably
2779a next step.
2780
2781I already mentioned that typesetting and rendering related technology is driven
2782by the web. This also reflects on \UNICODE\ and \OPENTYPE. For instance, we find
2783not only emoticons like \type {U+1F632} (ASTONISHED FACE) in the standard but
2784also \quote {MOUNT FUJI}, \type {TOKYO TOWER}, \type {STATUE OF LIBERTY}, \type
2785{SILHOUETTE OF JAPAN}. On the other hand, in one of our older projects we still
2786have to provide some tweak for the unary minus (as when discussing scientific
2787calculators used in math lessons) a distinction has to be made with a regular
2788minus sign. And there are no symbols to refer to use of media (simulation,
2789applet, etc.) and there is as far as I know no emoticon for a student asking a
2790question. Somehow it's hard to defend that the Planck constant is as different
2791from a math italic~h as a \quote {GRINNING FACE} is from a \quote {GRINNING FACE
2792WITH SMILING EYES}, but the last both got a code point. I wonder with an \type
2793{UNAMUSED FACE}.
2794
2795Of course we can argue that this is all too visual to end up in \UNICODE, but the
2796main point that I want to make is that as a \TEX\ community (which is also
2797related to education) we are of not that much importance and influence. Maybe it
2798is because we always had a programmable system at hand, and folks who could make
2799fonts, and were already extending and exploring before the web became a factor.
2800Anyhow, in \CONTEXT\ we solve these issues by making mechanisms extensible. For
2801instance we can extend fonts with virtual glyphs and add features to existing
2802fonts on the fly. Simple examples are adding some glyphs and properties to math
2803fonts or adding color properties to whatever font. More complex examples are
2804implementing paragraph optimizers using feature sets of fonts (most noticeably
2805the upcoming Husayni font for advanced arabic typesetting). And, math typesetting
2806is a speciality anyway.
2807
2808Upcoming extensions to \UNICODE\ and \OPENTYPE\ will demonstrate that the \TEX\
2809community could have been a bit more demanding and innovative, given that it had
2810known what to demand. Interesting is that some innovation already happened by
2811providing special fonts and macros and engines, but I guess much gets unnoticed.
2812On the other hand, I must admit that experimenting and providing solutions
2813independent of evolving technology also has benefits: it made (and makes) some
2814user group meetings interesting to go to and creates interesting niches of users.
2815Without this experimental playground I for sure would not be around.
2816
2817\stopsection
2818
2819\startsection[title=Tracing]
2820
2821Tracing is available for nearly all mechanisms and math is no exception. Most
2822tracing happens at the \LUA\ end and can be enabled with the tracker mechanism.
2823Users will seldom use this, but for development the situation is definitely more
2824comfortable in \MKIV. Of course it helps that the penalty of tracing and logging
2825has become less in recent times because memory as well as runtime is hardly
2826influenced.
2827
2828We provide several styles (modules) for generating lists and tables of characters
2829and extensibles, visualizing features and comparing fonts. Here we benefit from
2830\LUA\ because we can use the database embedded in \CONTEXT\ and looping and
2831testing is more convenient in this language. Of course the rendering is done by
2832\TEX, so this is a typical example of hybrid usage.
2833
2834\stopsection
2835
2836\startsection[title=Conclusion]
2837
2838It is somewhat ironic that while \CONTEXT\ is sometimes tagged as \quote {not to
2839be used when you need to do math typesetting} it is this macro package that
2840drives the development of \LUATEX\ with its updated math engine, which in turn
2841influences the updated math engine in \XETEX, that is used by other macro
2842packages. In a similar fashion the possibility to process \OPENTYPE\ math fonts
2843in \LUATEX\ triggered the development of such fonts as follow up on the Latin
2844Modern and \TEX\ Gyre projects. So, the fact that in \CONTEXT\ we have a bit more
2845freedom in experimenting with math (and engines) has some generic benefits as
2846well.
2847
2848I think that overall we're better off. The implementation at the \TEX\ end is
2849much cleaner because we no longer have to deal with different math encodings and
2850multiple families. Because in \CONTEXT\ we're less bound to traditional
2851approaches and don't need to be code compatible with other engines we can follow
2852different routes than usual. After all, that was also one of the main motivations
2853behind starting the \LUATEX\ project: clean (better understandable code), less
2854mean (no more hacks at the \TEX\ end), even if that means to be less lean (quite
2855a lot of \LUA\ code). Between the lines above you can read that I think that
2856we've missed some opportunities but that's a side effect of the community not
2857being that innovative which in turn is probably driven by more or less standard
2858expectations of publishers, as they are more served by good old stability instead
2859of progress. Therefore, we're probably stuck for a while, if not forever, with
2860what we have now. And a decent \CONTEXT\ math implementation is not going to
2861change that. What matters is that we can (still) keep up with developments
2862outside our sphere of influence.
2863
2864I don't claim that the current implementation of math in \MKIV\ is flawless, but
2865eventually we will get there.
2866
2867\stopsection
2868
2869% \blank[2*big,samepage]
2870
2871% \startlines
2872% Hans Hagen
2873% PRAGMA ADE
2874% Hasselt NL
2875% June-August 2013
2876% \stoplines
2877
2878\stopchapter
2879
2880\stoptext
2881