% language=us

\startcomponent hybrid-math

\environment hybrid-environment

\startchapter[title={Handling math: A retrospective}]

{This is \TUGBOAT\ article .. reference needed.}

% In this article I will reflect on how the plain \TEX\ approach to math
% fonts influenced the way math has been dealt with in \CONTEXT\ \MKII\
% and why (and how) we divert from it in its follow up \MKIV, now that
% \LUATEX\ and \OPENTYPE\ math have come around.

When you start using \TEX, you cannot help but notice that math plays an
important role in this system. As soon as you dive into the code you will see
that there is a concept of families that is closely related to math typesetting.
A family is a set of three sizes: text, script and scriptscript.

\startformula
a^{b^{c}} = \frac{d}{e}
\stopformula

The smaller sizes are used in superscripts and subscripts and in more complex
formulas where information is put on top of each other.

It is no secret that the latest math font technology is not driven by the \TEX\
community but by Microsoft. They have taken a good look at \TEX\ and extended the
\OPENTYPE\ font model with the information that is needed to do things similar to
\TEX\ and beyond. It is a firm proof of \TEX's abilities that after some 30 years
it is still seen as the benchmark for math typesetting. One can only speculate
what Don Knuth would have come up with if today's desktop hardware and printing
technology had been available in those days.

As a reference implementation of a font Microsoft provides Cambria Math. In the
specification the three sizes are there too: a font can provide specifically
designed script and scriptscript variants for text glyphs where that is relevant.
Control is exercised with the \type {ssty} feature.

Another inheritance from \TEX\ and its fonts is the fact that larger symbols can
be made out of snippets and these snippets are available as glyphs in the font,
so no special additional (extension) fonts are needed to get for instance really
large parentheses. The information of when to move up one step in size (given
that there is a larger shape available) or when and how to construct larger
symbols out of snippets is there as well. Placement of accents is made easy by
information in the font and there are a whole lot of parameters that control the
typesetting process. Of course you still need machinery comparable to \TEX's math
subsystem but Microsoft Word has such capabilities.

I'm not going to discuss the nasty details of providing math support in \TEX, but
rather pay some attention to an (at least for me) interesting side effect of
\TEX's math machinery. There are excellent articles by Bogus\l{}aw Jackowski and
Ulrik Vieth about how \TEX\ constructs math and of course Knuth's publications
are the ultimate source of information as well.

Even if you only glance at the implementation of traditional \TEX\ font support,
the previously mentioned families are quite evident. You can have 16 of them but
4 already have a special role: the upright roman font, math italic, math symbol
and math extension. These give us access to some 1000 glyphs in theory, but when
\TEX\ showed up it was mostly a 7-bit engine and input of text was often also
7-bit based, so in practice many fewer shapes are available, and subtracting the
snippets that make up the large symbols brings down the number again.

Now, say that in a formula you want to have a bold character. This character is
definitely not in the 4 mentioned families. Instead you enable another one, one
that is linked to a bold font. And, of course there is also a family for bold
italic, slanted, bold slanted, monospaced, maybe smallcaps, sans serif, etc. To
complicate things even more, there are quite a few symbols that are not covered
in the foursome so we need another 2 or 3 families just for those. And yes, bold
math symbols will demand even more families.

\startformula
a + \bf b + \bi c = \tt d + \ss e + \cal f
\stopformula

Try to imagine what this means for implementing a font system. When (in for
instance \CONTEXT) you choose a specific body font at a certain size, you not
only switch the regular text fonts, you also initialize math. When dealing with
text and a font switch there, it is no big deal to delay font loading and
initialization till you really need the font. But for math it is different. In
order to set up the math subsystem, the families need to be known and set up and
as each one can have three members you can imagine that you easily initialize
some 30 to 40 fonts. And, when you use several math setups in a document,
switching between them involves at least some re-initialization of those
families.

When Taco Hoekwater and I were discussing \LUATEX\ and especially what was needed
for math, it was sort of natural to extend the number of families to 256. After
all, years of traditional usage had demonstrated that it was pretty hard to come
up with math font support where you could freely mix a whole regular and a whole
bold set of characters simply because you ran out of families. This is a side
effect of math processing happening in several passes: you can change a family
definition within a formula, but as \TEX\ remembers only the family number, a
later definition overloads a previous one. The previous example in a traditional
\TEX\ approach can result in:

\starttyping
a + \fam7 b + \fam8 c = \fam9 d + \fam10 e + \fam11 f
\stoptyping

Here the \type{a} comes from the family that reflects math italic (most likely
family~1) and \type {+} and \type {=} can come from whatever family is told to
provide them (this is driven by their math code properties). As family numbers
are stored in the identification pass, and in the typesetting pass resolve to
real fonts you can imagine that overloading a family in the middle of a
definition is not an option: it's the number that gets stored and not what it is
bound to. As it is unlikely that we actually use more than 16 families we could
have come up with a pool approach where families are initialized on demand but
that does not work too well with grouping (or at least it complicates matters).

So, when I started thinking of rewriting the math font support for \CONTEXT\
\MKIV, I still had this nicely increased upper limit in mind, if only because I
was still thinking of support for the traditional \TEX\ fonts. However, I soon
realized that it made no sense at all to stick to that approach: \OPENTYPE\ math
was on its way and in the meantime we had started the math font project. But
given that this would easily take some five years to finish, an intermediate
solution was needed. As we can make virtual fonts in \LUATEX, I decided to go
that route and for several years already it has worked quite well. For the moment
the traditional \TEX\ math fonts (Computer Modern, px, tx, Lucida, etc) are
virtualized into a pseudo|-|\OPENTYPE\ font that follows the \UNICODE\ math
standard. So instead of needing more families, in \CONTEXT\ we could do with
less. In fact, we can do with only two: one for regular and one for bold,
although, thinking of it, there is nothing that prevents us from mixing different
font designs (or preferences) in one formula but even then a mere four families
would still be fine.

To summarize this, in \CONTEXT\ \MKIV\ the previous example now becomes:

\starttyping
U+1D44E + U+1D41B + 0x1D484 = U+1D68D + U+1D5BE + U+1D4BB
\stoptyping

For a long time I have been puzzled by the fact that one needs so many fonts for
a traditional setup. It was only after implementing the \CONTEXT\ \MKIV\ math
subsystem that I realized that all of this was only needed in order to support
alphabets, i.e.\ just a small subset of a font. In \UNICODE\ we have quite a few
math alphabets and in \CONTEXT\ we have ways to map a regular keyed-in (say)
\quote{a} onto a bold or monospaced one. When writing that code I hadn't even
linked the \UNICODE\ math alphabets to the family approach for traditional \TEX.
Not being a mathematician myself I had no real concept of systematic usage of
alternative alphabets (apart from the occasional different shape for an
occasional physics entity).

Just to give an idea of what \UNICODE\ defines: there are alphabets in regular
(upright), bold, italic, bold italic, script, bold script, fraktur, bold fraktur,
double|-|struck, sans|-|serif, sans|-|serif bold, sans|-|serif italic,
sans|-|serif bold italic and monospace. These are regular alphabets with upper-
and lowercase characters complemented by digits and occasionally Greek.

It was a few years later (somewhere near the end of 2010) that I realized that a
lot of the complications in (and load on) a traditional font system were simply
due to the fact that in order to get one bold character, a whole font had to be
loaded in order for families to express themselves. And that in order to have
several fonts being rendered, one needed lots of initialization for just a few
cases. Instead of wasting one font and family for an alphabet, one could as well
have combined 9 (upper and lowercase) alphabets into one font and use an offset
to access them (in practice we have to handle the digits too). Of course that
would have meant extending the \TEX\ math machinery with some offset or
alternative to some extensive mathcode juggling but that also has some overhead.

If you look at the plain \TEX\ definitions for the family related matters, you
can learn a few things. First of all, there are the regular four families
defined:

\starttyping
\textfont0=\tenrm \scriptfont0=\sevenrm \scriptscriptfont0=\fiverm
\textfont1=\teni  \scriptfont1=\seveni  \scriptscriptfont1=\fivei
\textfont2=\tensy \scriptfont2=\sevensy \scriptscriptfont2=\fivesy
\textfont3=\tenex \scriptfont3=\tenex   \scriptscriptfont3=\tenex
\stoptyping

Each family has three members. There are some related definitions
as well:

\starttyping
\def\rm      {\fam0\tenrm}
\def\mit     {\fam1}
\def\oldstyle{\fam1\teni}
\def\cal     {\fam2}
\stoptyping

So, with \type {\rm} you not only switch to a family (in math mode) but you also
enable a font. The same is true for \type {\oldstyle} and this actually brings us
to another interesting side effect. The fact that oldstyle numerals come from a
math font has implications for the way this rendering is supported in macro
packages. As naturally all development started when \TEX\ came around, package
design decisions were driven by the basic fact that there was only one math font
available. And, as a consequence most users used the Computer Modern fonts and
therefore there was never a real problem in getting those oldstyle characters in
your document.

However, oldstyle figures are a property of a font design (like table digits) and
as such not specially related to math. And, why should one tag each number then?
Of course it's good practice to tag extensively (and tagging makes switching
fonts easy) but to tag each number is somewhat over the top. When more fonts
(usable in \TEX) became available it became more natural to use a proper oldstyle
font for text and the \type {\oldstyle} more definitely ended up as a math
command. This was not always easy to understand for users who primarily used
\TEX\ for anything but math.

Another interesting aspect is that with \OPENTYPE\ fonts oldstyle figures are
again an optional feature, but now at a different level. There are a few more
such traditional issues: bullets often come from a math font as well (which works
out ok as they have nice, not so tiny bullets). But the same is true for
triangles, squares, small circles and other symbols. And, to make things worse,
some come from the regular \TEX\ math fonts, and others from additional ones,
like the \AMS\ symbols. Again, \OPENTYPE\ and \UNICODE\ will change this as now
these symbols are quite likely to be found in fonts as they have a larger
repertoire of shapes.

From the perspective of going from \MKII\ to \MKIV\ it boils down to changing old
mechanisms that need to handle all this (dependent on the availability of fonts)
to cleaner setups. Of course, as fonts are never completely consistent, or
complete for that matter, and features can be implemented incorrectly or
incompletely we still end up with issues, but (at least in \CONTEXT) dealing with
that has been moved to runtime manipulation of the fonts themselves (as part of
the so-called font goodies).

Back to the plain definitions, we now arrive at some new families:

\starttyping
\newfam\itfam \def\it{\fam\itfam\tenit}
\newfam\slfam \def\sl{\fam\slfam\tensl}
\newfam\bffam \def\bf{\fam\bffam\tenbf}
\newfam\ttfam \def\tt{\fam\ttfam\tentt}
\stoptyping

The plain \TEX\ format was never meant as a generic solution but instead was an
example of a macro set and serves as a basis for styles used by Don Knuth for his
books. Nevertheless, in spite of the fact that \TEX\ was made to be extended,
pretty soon it became frozen and the macros and font definitions that came with
it became the benchmark. This might be the reason why \UNICODE\ now has a
monospaced alphabet. Once you've added monospaced you might as well add more
alphabets as for sure in some countries they have their own preferences.
\footnote {At the Dante 2011 meeting we had interesting discussions during dinner
about the advantages of using Sütterlinschrift for vector algebra and the
possibilities for providing it in the upcoming \TeX\ Gyre math fonts.}

As with \type {\rm}, the related commands are meant to be used in text as well.
More interesting is to see what follows now:

\starttyping
\textfont        \itfam=\tenit
\textfont        \slfam=\tensl

\textfont        \bffam=\tenbf
\scriptfont      \bffam=\sevenbf
\scriptscriptfont\bffam=\fivebf

\textfont        \ttfam=\tentt
\stoptyping

Only the bold definition has all members. This means that (regular) italic,
slanted, and monospaced are not actually that much math at all. You will probably
only see them in text inside a math formula. From this you can deduce that
contrary to what I said before, these variants were not really meant for
alphabets, but for text in which case we need complete fonts. So why do I still
conclude that we don't need all these families? In practice text inside math is
not always done this way but with a special set of text commands. This is a
consequence of the fact that when we add text, we want to be able to do so in
each language with even language|-|specific properties supported. And, although a
family switch like the above might do well for English, as soon as you want
Polish (extended Latin), Cyrillic or Greek you definitely need more than a family
switch, if only because encodings come into play. In that respect it is
interesting that we do have a family for monospaced, but that \type {\Im} and
\type {\Re} have symbolic names, although a more extensive setup can have a
blackboard family switch.

By the way, the fact that \TEX\ came with italic alongside slanted also has some
implications. Normally a font design has either italic or something slanted (then
called oblique). But, Computer Modern came with both, which is no surprise as
there is a metadesign behind it. And therefore macro packages provide ways to
deal with those variants alongside. I wonder what would have happened if this had
not been the case. Nowadays there is always this regular, italic (or oblique),
bold and bold italic set to deal with, and the whole set can become lighter or
bolder.

In \CONTEXT\ \MKII, however, the set is larger as we also have slanted and bold
slanted and even smallcaps, so most definition sets have 7~definitions instead
of~4. By the way, smallcaps is also special. if Computer Modern had had smallcaps
for all variants, support for them in \CONTEXT\ undoubtedly would have been kept
out of the mentioned~7 but always been a new typeface definition (i.e.\ another
fontclass for insiders). So, when something would have to be smallcaps, one would
simply switch the whole lot to smallcaps (bold smallcaps, etc.). Of course this
is what normally happens, at least in my setups, but nevertheless one can still
find traces of this original Computer Modern|-|driven approach. And now we are at
it: the whole font system still has the ability to use design sizes and combine
different ones in sets, if only because in Computer Modern you don't have all
sizes. The above definitions use ten, seven and five, but for instance for an
eleven point set up you need to creatively choose the proper originals and scale
them to the right family size. Nowadays only a few fonts ship with multiple
design sizes, and although some can be compensated with clever hinting it is a
pity that we can apply this mechanism only to the traditional \TEX\ fonts.

Concerning the slanting we can remark that \TEX ies are so fond of this that they
even extended the \TEX\ engines to support slanting in the core machinery (or
more precisely in the backend while the frontend then uses adapted metrics). So,
slanting is available for all fonts.

This brings me to another complication in writing a math font subsystem: bold.
During the development of \CONTEXT\ \MKII\ I was puzzled by the fact that user
demands with respect to bold were so inconsistent. This is again related to the
way a somewhat simple setup looks: explicitly switching to bold characters or
symbols using a \type {\bf} (alike) switch. This works quite well in most cases,
but what if you use math in a section title? Then the whole lot should be in bold
and an embedded bold symbol should be heavy (i.e.\ more bold than bold). As a
consequence (and due to limited availability of complete bold math fonts) in
\MKII\ there are several bold strategies implemented.

However, in a \UNICODE\ universe things become surprisingly easy as \UNICODE\
defines those symbols that have bold companions (whatever you want to call them,
mostly math alphanumerics) so a proper math font has them already. This limited
subset is often available in a font collection and font designers can stick to
that subset. So, eventually we get one regular font (with some bold glyphs
according to the \UNICODE\ specification) and a bold companion that has heavy
variants for those regular bold shapes.

The simple fact that \UNICODE\ distinguishes regular and bold simplifies an
implementation as it's easier to take that as a starting point than users who for
all their goodwill see only their small domain of boldness.

It might sound like \UNICODE\ solves all our problems but this is not entirely
true. For instance, the \UNICODE\ principle that no character should be there
more than once has resulted in holes in the \UNICODE\ alphabets, especially
Greek, blackboard, fraktur and script. As exceptions were made for non|-|math I
see no reason why the few math characters that now put holes in an alphabet could
not have been there. As with more standards, following some principles too
strictly eventually results in all applications that follow the standard having
to implement the same ugly exceptions explicitly. As some standards aim for
longevity I wonder how many programming hours will be wasted this way.

This brings me to the conclusion that in practice 16 families are more than
enough in a \UNICODE|-|aware \TEX\ engine especially when you consider that for a
specific document one can define a nice set of families, just as in plain \TEX.
It's simply the fact that we want to make a macro package that does it all and
therefore has to provide all possible math demands into one mechanism that
complicates life. And the fact that \UNICODE\ clearly demonstrates that we're
only talking about alphabets has brought (at least) \CONTEXT\ back to its basics:
a relatively simple, few|-|family approach combined with a dedicated alphabet
selection system. Of course eventually users may come up with new demands and we
might again end up with a mess. After all, it's the fact that \TEX\ gives us
control that makes it so much fun.

\stopchapter

\stopcomponent