mk-tracking.tex /size: 13 Kb    last modification: 2023-12-21 09:43
1% language=us
2
3% \enabletrackers[otf.loading]
4% \enabletrackers[otf.lookups]
5
6\startcomponent mk-track
7
8\environment mk-environment
9
10\startbuffer[latin-default-features]
11\definefontfeature
12  [latin-default]
13  [mode=node,language=dflt,script=latn,
14   liga=yes,calt=yes,clig=yes,
15   kern=yes]
16\stopbuffer
17
18\startbuffer[arabtype-default-features]
19\definefontfeature
20  [arabtype-default]
21  [mode=node,language=dflt,script=arab,
22   init=yes,medi=yes,fina=yes,isol=yes,
23   ccmp=yes,locl=yes,calt=yes,
24   liga=yes,clig=yes,dlig=yes,rlig=yes,
25   mark=yes,mkmk=yes,kern=yes,curs=yes]
26\stopbuffer
27
28\startbuffer[zapfino-default-features]
29\definefontfeature
30   [zapfino-default]
31   [mode=node,language=dflt,script=latn,
32    calt=yes,clig=yes,rlig=yes,tlig=yes,
33    kern=yes,curs=yes]
34\stopbuffer
35
36\getbuffer[latin-default-features]
37\getbuffer[arabtype-default-features]
38\getbuffer[zapfino-default-features]
39
40\chapter{Tracking}
41
42We entered 2009 with a partial reimplementation of the \OPENTYPE\
43feature handler. One of the reasons was an upgrade of the
44\FONTFORGE\ libraries that \LUATEX\ uses.
45
46The specification of \OPENTYPE\ is kind of vague. Apart from a
47lack of a proper free specifications there's also the problem that
48Microsoft and Adobe may have their own interpretation of how and
49in what order to apply features. In general the Microsoft website
50has more detailed specifications and is a better reference. There
51is also some information in the \FONTFORGE\ help files.
52
53Because there is so much possible, fonts might contain bugs and/or
54be made to work with certain renderers. These may evolve over time
55which may have the side effect that suddenly fonts behave
56differently.
57
58After a lot of experiments (mostly by Taco, me and Idris) we're
59now at yet another implementation. Of course all errors are mine
60and of course the code can be improved. There are quite some
61optimization going on here and processing speed is currently
62acceptable. Not all functions are implemented yet, often because I
63lack the fonts for testing. Many scripts are not yet supported
64either, but I will look into them as soon as \CONTEXT\ users ask
65for it.
66
67The data provided by the \FONTFORGE\ library has organized lookups
68(which relate to features) in a certain way. A first
69implementation of this code was organized featurewise: information
70related to features was collected and processing boiled down to a
71run over the features. The current implementation honours the order
72in the main feature table. Since we can reorder this table as we
73want, we can eventually support several models of processing. We
74kept the static as well as dynamic feature processing, because it
75had proved to be rather useful. The formerly three loop variants
76have been discarded but might reappear at some time.
77
78One reason for this change is that the interactive version of
79\FONTFORGE\ now provides a more detailed overview of the way
80lookups are supposed to be handled. When you consult the
81information of a font and in particular a glyph in a font, you now
82get quite some information about what features can be applied and
83in what order this takes place.
84
85In \CONTEXT\ \MKIV\ we deal with this as follows. Keep in mind
86that we start with characters but stepwise these can become more
87abstract representation, named glyphs. For instance a letter~a can
88be represented by a shape (glyph) that is similar to an uppercase~A.
89
90\startitemize
91
92\item We loop over all lookups. Normally there are only a few
93lookups but fonts that deal with scripts that resemble
94handwriting, like arabic of Zapfino, might have hundreds of them.
95Each lookup has a detailed specification of what language and/or
96scripts it applies to.
97
98\item For each lookup we do a run over the list of glyphs. So, if
99we have 50 lookups, and a paragraph has 500 glyphs, we do some
10025000 loops. Keep in mind that for arab we start with a sequence
101of characters and vowels, and during a run, these might be
102replaced by for instance ligatures and combined vowels, so the 500
103stepwise becomes less.
104
105\item We only process the features that are enabled. Normally the
106lookups are organized in such a way that features take place in a
107similar way: (de)composition, replacement of initial, medial,
108final and isolated forms, specific replacements by one or more
109variant, composition of ligatures, mark positioning, cursive
110corrections and kerning. The font itself does not contain
111information about what features are to be enabled by default. Some
112applications have built in presets, others might extend their
113repertoire over time.
114
115\item A lookup can be a contextual lookup, which means that
116treatment takes place on a match of a sequence of characters
117(glyphs), either of not preceded or followed by specific other
118characters (glyphs). We we loop over all contexts till we have a
119match. Some fonts have lots of contextual lookups, which in turn
120might increase the number of loops over the list of characters
121(glyphs). If we have a match, we process the associated list of
122sublookups. Technically it is possible to replace (say) five
123characters by first a ligature (that replaces the first two by
124one), then a multiple substitution (resulting in an extra three
125glyphs replacing one) and then discarding the other rest (being
126two characters). Because by that time characters (say, unicode
127points) might have been replaced by glyphs (an index in the font)
128a contextual lookup can involve quite some match points.
129
130\stopitemize
131
132In \CONTEXT\ we do this for each font that is used in a list, so
133in practice we have quite some nested loops. Each font can have
134its own set of features enables of features might be applied
135dynamically, independent of font related settings. So, around the
136mentioned loops there is another one: a loop over the fonts used
137in a list (paragraph).
138
139We process the whole list and then consult the glyph nodes. An
140alternative approach is to collect strings of characters using the
141same font including spaces (because some lookups involve spaces).
142However, we then need to reconstruct the list which is no fun.
143Also, we need to carry quite some information, like attributes, so
144eventually we don't gain much (if we gain something at all).
145
146Another consideration has been to operate on sublists of font
147usage (using a subhead and subtail) but again this would
148complicate matters as we then neext to keep track of a changing
149subhead and subtail. On the other hand, this might save some
150runtime. The number of changes in the code needed to do this is
151not that large but it only makes sense when we have many fonts
152in a list and don't change fonts to frequently.
153
154This whole treatment is rather extensively optimized and so the
155process is reasonable fast (you really don't want to know how much
156time was spent on figuring out fast methods, testing and
157reimplementing this). While I was implementing the \LUA\ code,
158Taco made sure that access to the information in nodes was as fast
159as possible and in our usual chat sessions we compared the output
160with the one produced by the \FONTFORGE\ preview.
161
162It was for this reason that more and more debugging code was added
163but even that made tracking of what really happened cumbersome.
164Therefore a more visual method was written, which will be shown
165laster on.
166
167You can enable tracing using the designated commands:
168
169\starttyping
170\enabletracker[otf.ligatures,otf.singles]
171\stoptyping
172
173and disable them for instance with:
174
175\starttyping
176\disabletracker[otf.*]
177\stoptyping
178
179Or you can pass directives to the command line:
180
181\starttyping
182context --track=otf.ligatures myfile.tex
183\stoptyping
184
185With regards to \OPENTYPE\ handling we have the following tracker
186keys available:
187
188\starttabulate
189\NC \type{otf.actions}       \NC show all replacements and positioning \NC \NR
190\NC \type{otf.alternatives}  \NC show what glyph is replaced by what alternative \NC \NR
191\NC \type{otf.analyzing}     \NC color glyphs according to script specific analysis \NC \NR
192\NC \type{otf.applied}       \NC applied features per font instance \NC \NR
193\NC \type{otf.bugs}          \NC show diagnostic information \NC \NR
194\NC \type{otf.contexts}      \NC show what contextual lookups take place \NC \NR
195\NC \type{otf.cursive}       \NC show cursive anchoring when applied \NC \NR
196\NC \type{otf.details}       \NC show more details about lookup handling \NC \NR
197\NC \type{otf.dynamics}      \NC show dynamic feature definitions \NC \NR
198\NC \type{otf.features}      \NC show what features are a applied \NC \NR
199\NC \type{otf.kerns}         \NC show kerning between glyphs when applied \NC \NR
200\NC \type{otf.ligatures}     \NC show what glyphs are replaced by one other \NC \NR
201\NC \type{otf.loading}       \NC show more information when loading (caching) a font \NC \NR
202\NC \type{otf.lookups}       \NC keep track of what lookups are consulted \NC \NR
203\NC \type{otf.marks}         \NC show mark anchoring when applied \NC \NR
204\NC \type{otf.multiples}     \NC show what glyph is replaced by multiple others \NC \NR
205%NC \type{otf.normal_chain}  \NC \NC \NR
206\NC \type{otf.positions}     \NC show what glyphs are positioned (combines other trackers) \NC \NR
207\NC \type{otf.preparing}     \NC show what information is collected for later usage in lookups \NC \NR
208\NC \type{otf.replacements}  \NC show what glyphs are replaced (combines other trackers) \NC \NR
209\NC \type{otf.sequences}     \NC \NC \NR
210\NC \type{otf.singles}       \NC show what glyph is replaced by one other \NC \NR
211%NC \type{otf.steps}         \NC \NC \NR
212%NC \type{otf.verbose_chain} \NC \NC \NR
213\stoptabulate
214
215Some other trackers might also come in handy:
216
217\starttabulate
218%NC \type{fonts.collecting} \NC  \NC \NR
219\NC \type{fonts.combining}  \NC show what extra characters are added when forcing combined shapes \NC \NR
220\NC \type{fonts.defining}   \NC show what fonts are defined \NC \NR
221\NC \type{fonts.loading}    \NC show more details when a font is loaded (and cached) for the first time \NC \NR
222%NC \type{fonts.names}      \NC \NC \NR
223%NC \type{fonts.scaling}    \NC \NC \NR
224\stoptabulate
225
226We now show another way to track what happens with your text.
227Because this is rather verbose, you should only apply it to words.
228The second argument can be \type {-1} (right to left), \type {0}
229(default) or \type {1} (left to right). The third argument can
230be invisible in the code because the font used for verbatim might
231lack the shapes. A font has a different ordering than \UNICODE\
232because after all one character can have multiple
233representations, one shape can be used for multiple characters,
234or shapes might not have a \UNICODE\ point at all. In \MKIV\ we
235push all shapes that have no direct relationship with \UNICODE\ to
236the private area so that \TEX\ still sees them (hence the large
237numbers in the following examples).
238
239The next example uses Latin Modern. Here we apply the following
240features:
241
242\typebuffer[latin-default-features]
243
244\startbuffer
245\showotfcomposition
246  {name:lmroman12regular*latin-default at 24pt}
247  {0}
248  {flinke fietser}
249\stopbuffer
250
251\typebuffer \start \veryraggedright \getbuffer \stop
252
253The next example uses Arabtype. Here we apply the following features:
254
255\typebuffer[arabtype-default-features]
256
257\startbuffer
258\showotfcomposition
259  {arabtype*arabtype-default at 48pt}
260  {-1}
261  {الضَّرَّ}
262\stopbuffer
263
264\typebuffer \start \veryraggedright \getbuffer \stop
265
266\startbuffer
267\showotfcomposition
268  {arabtype*arabtype-default at 48pt}
269  {-1}
270  {لِلّٰهِ}
271\stopbuffer
272
273\typebuffer \start \veryraggedright \getbuffer \stop
274
275Another arabic example (after all, fonts that support arabic have
276lots of nice features) is the following. First we define a bunch
277of feature collections
278
279\startbuffer
280\definefontfeature
281  [salt-n]
282  [analyze=yes,mode=node,
283   language=dflt,script=arab,
284   init=yes,medi=yes,fina=yes,isol=yes,
285   liga=yes,calt=yes,ccmp=yes,
286   kern=yes,curs=yes,mark=yes,mkmk=yes]
287
288\definefontfeature[salt-y][salt-n][salt=yes]
289\definefontfeature[salt-1][salt-n][salt=1]
290\definefontfeature[salt-2][salt-n][salt=2]
291\definefontfeature[salt-3][salt-n][salt=3]
292\definefontfeature[salt-r][salt-n][salt=random]
293\stopbuffer
294
295\typebuffer \getbuffer
296
297Next we show a few traced examples. Watch the reported alternatives.
298
299\startbuffer
300\showotfcomposition{scheherazaderegot*salt-n at 36pt}{-1}{\char"6DD}
301\showotfcomposition{scheherazaderegot*salt-y at 36pt}{-1}{\char"6DD}
302\showotfcomposition{scheherazaderegot*salt-1 at 36pt}{-1}{\char"6DD}
303\showotfcomposition{scheherazaderegot*salt-2 at 36pt}{-1}{\char"6DD}
304\showotfcomposition{scheherazaderegot*salt-3 at 36pt}{-1}{\char"6DD}
305\showotfcomposition{scheherazaderegot*salt-r at 36pt}{-1}{\char"6DD}
306\showotfcomposition{scheherazaderegot*salt-r at 36pt}{-1}{\char"6DD}
307\showotfcomposition{scheherazaderegot*salt-r at 36pt}{-1}{\char"6DD}
308\stopbuffer
309
310\typebuffer \start \veryraggedright \getbuffer \stop
311
312The font that we use here can be downloaded from the website of
313Sil International.
314
315For a Zapfino example we use the following feature set:
316
317\typebuffer[zapfino-default-features]
318
319\startbuffer
320\showotfcomposition
321  {zapfinoextraltpro*zapfino-default at 48pt}
322  {0}
323  {Prof. Dr. Donald E. Knuth}
324\stopbuffer
325
326\typebuffer \start \veryraggedright \getbuffer \stop
327
328When dealing with features, we may run into problems due to
329characters that are in the input stream but have no associated
330glyph in the font. Although we test for this a user might want to
331intercept side effect.
332
333\starttyping
334\checkcharactersinfont
335\removemissingcharacters
336\stoptyping
337
338The first command only checks and reports missing characters,
339while the second one also removes them.
340
341\stopcomponent
342