hybrid-goodies.tex /size: 20 Kb    last modification: 2023-12-21 09:43
1% language=us
2
3\usetypescriptfile[type-husayni]
4
5\startcomponent hybrid-goodies
6
7\environment hybrid-environment
8
9% this will change
10
11\definefontfeature
12  [husayni-none]
13  [analyze=yes,mode=node,
14   language=dflt,script=arab,
15   ccmp=yes]
16
17\definefontfeature
18  [husayni-default]
19  [analyze=yes,mode=node,
20   language=dflt,script=arab,
21   ccmp=yes,init=yes,medi=yes,fina=yes,
22   rlig=yes,calt=yes,salt=yes,anum=yes,
23   kern=yes,curs=yes,mark=yes,mkmk=yes,
24   ss01=yes,ss03=yes,ss10=yes,ss12=yes,ss15=yes,ss16=yes,
25   ss19=yes,ss24=yes,ss25=yes,ss26=yes,ss27=yes,ss31=yes,
26   ss34=yes,ss35=yes,ss36=yes,ss37=yes,ss38=yes,ss41=yes,
27   ss43=yes]
28
29\definefontfeature
30  [husayni-first-order]
31  [script=arab,ss01=yes,ss03=yes,ss05=yes,
32   ss10=yes,ss12=yes,ss15=yes,ss16=yes,ss19=yes,ss24=yes,
33   ss25=yes,ss26=yes,ss27=yes,ss31=yes,ss34=yes,ss35=yes,
34   ss36=yes,ss37=yes,ss38=yes,ss41=yes,ss42=yes,ss43=yes]
35
36\definefontfeature
37  [husayni-stack-jiim-multi-level]
38  [script=arab,ss05=yes]
39
40\definefontfeature
41  [husayni-minimal-stretching]
42  [script=arab,
43   ss05=yes,ss09=yes,ss06=yes,ss13=yes,ss17=yes,ss40=yes,
44   js11=yes,js14=yes,js16=yes]
45
46\definefontfeature
47  [husayni-maximal-stretching]
48  [script=arab,
49   ss05=yes,ss09=yes,ss06=yes,ss13=yes,ss17=yes,ss40=yes,
50   js13=yes,js14=yes,js16=yes]
51
52\definefontfeature
53  [husayni-chop-haa]
54  [script=arab,
55   ss05=yes,ss09=yes,ss06=yes,ss13=yes,ss17=yes,ss54=yes]
56
57\definefontfeature
58  [husayni-normal]
59  [goodies=husayni,
60   featureset=default]
61
62\definefont[ArabicFontNone][husayni*husayni-none    at 40pt]
63\definefont[ArabicFontFull][husayni*husayni-default at 40pt] % husayni-normal
64
65\startchapter[title={Font Goodies}]
66
67\startsection[title={Introduction}]
68
69The Oriental \TEX\ project is one of the first and more ambitious users of
70\LUATEX. A major undertaking in this project is the making of a rather full
71features and complex font for typesetting Arabic. As the following text will show
72some Arabic, you might get the impression that I'm an expert but be warned that
73I'm far from that. But as Idris compensates this quite well the team has a lot of
74fun in figuring out how to achieve our goals using \OPENTYPE\ technology in
75combination with \LUATEX\ and \MKIV. A nice side effect of this is that we end up
76with some neat tricks in the \CONTEXT\ core.
77
78Before we come to some of these goodies, an example of Arabic is given that
79relates quite well to the project. It was first used at the euro\TEX\ 2009
80meeting. Take the following 6 shapes:
81
82\starttabulate[|c|c|c|c|c|c|]
83\NC \ArabicFontFull ل \NC \ArabicFontFull و \NC \ArabicFontFull ا \NC \ArabicFontFull ت \NC \ArabicFontFull ي \NC \ArabicFontFull خ \NC \NR
84\NC \type{l} \NC \type{w} \NC \type{ā} \NC \type{t} \NC \type{ī} \NC \type{kh} \NC \NR
85\stoptabulate
86
87With these we can make the name \LUATEX\ and as we use a nice script we can
88forget about the lowered~E. Putting these characters in sequence is not enough as
89Arabic typesetting has to mimick the subtle aspects of scribes.
90
91In Latin scripts we have mostly one|-|to|-|one and many|-|to|-|one substitutions.
92These can happen in sequence which in in practice boils down to multiple passes
93over the stream of characters. In this process sometimes surrounding characters
94(or shapes) play a role, for instance ligatures are not always wanted and their
95coming into existence might depend on neighbouring characters. In some cases
96glyphs have to be (re)positioned relative to each other. While in Latin scripts
97the number of substitutions and positioning is not that large but in advanced
98Arabic fonts it can be pretty extensive.
99
100With \OPENTYPE\ we have some machinery available, so we try to put as much logic
101in the font as possible. However, in addition we have some dedicated optimizing
102routines. The whole process is split into a couple if stages.
103
104The so called First|-|Order Analysis puts a given character into isolated,
105initial, middle, or final state. Next, the Second|-|Order Analysis looks at the
106characters and relates this state to what characters precede or succeed it. Based
107on that state we do character substitutions. There can be multiple analysis and
108replacements in sequence. We can do some simple aesthetic stretching and
109additional related replacements. We need to attach identity marks and vowels in
110proper but nice looking places. In most cases we're then done. Contrary to other
111fonts we don't use many ligatures but compose characters.
112
113The previous steps already give reasonable results and implementing it also
114nicely went along with the development of \LUATEX\ and \CONTEXT\ \MKIV. Currently
115we're working on extending and perfecting the font to support what we call
116Third|-|Order Contextual Analysis. This boils down to an interplay between the
117paragraph builder and additional font features. In order to get pleasing spacing
118we apply further substitutions, this time with wider or narrower shapes. When
119this is done we need to reattach identity marks and vowels. Optionally we can
120apply \HZ\ like stretching as a finishing touch but so far we didn't follow that
121route yet.
122
123So, let's see how we can typeset the word \LUATEX\ in Arabic using some of these
124techniques.
125
126\startlines
127no order (kh ī t ā w [u] l)\hfilll {\righttoleft\ArabicFontNone لُواتيخ}
128first order \hfilll {\subff{husayni-first-order}\righttoleft\ArabicFontFull لُواتيخ}
129second order \hfilll {\righttoleft\ArabicFontFull لُواتيخ}
130second order (Jiim-stacking) \hfilll {\addff{husayni-stack-jiim-multi-level}\righttoleft\ArabicFontFull لُواتيخ}
131minimal stretching \hfilll {\addff{husayni-minimal-stretching}\righttoleft\ArabicFontFull لُواتيخ}
132maximal stretching (level 3) \hfilll {\addff{husayni-maximal-stretching}\righttoleft\ArabicFontFull لُواتيخ}
133chopped letter khaa (for e.g.\ underlining) \hfilll {\addff{husayni-chop-haa}\righttoleft\ArabicFontFull لُواتيخ}
134\stoplines
135
136As said, this font is quite complex in the sense that it has many features and
137associated lookups. In addition to the usual features we have stylistic and
138justification variants. As these are not standardized (after all, each font can
139have its own look and feel and associated treatments) we store some information
140in the goodies files that ship with this font.
141
142\startbuffer[stylistics]
143\startluacode
144  local goodies = fonts.goodies.load("husayni")
145  local stylistics = goodies and goodies.stylistics
146  if stylistics then
147    local col, row, type = context.NC, context.NR, context.type
148    context.starttabulate { "|l|pl|" }
149    col() context("feature") col() context("meaning") col() row()
150    for feature, meaning in table.sortedpairs(stylistics) do
151      col() type(feature) col() type(meaning) col() row()
152    end
153    context.stoptabulate()
154  end
155\stopluacode
156\stopbuffer
157
158\getbuffer[stylistics]
159
160It is highly unlikely that a user will remember all these features, which is why
161there will be a bunch of predefined combinations. These are internalized as
162follows:
163
164\startbuffer[featuresets]
165\startluacode
166  local goodies = fonts.goodies.load("husayni")
167  local featuresets = goodies and goodies.featuresets
168  if featuresets then
169    local col, row, type = context.NC, context.NR, context.type
170    context.starttabulate { "|l|pl|" }
171    col() context("featureset") col() context("definitions") col() row()
172    for featureset, definitions in table.sortedpairs(featuresets) do
173      col() type(featureset) col()
174      for k, v in table.sortedpairs(definitions) do
175        type(string.format("%s=%s",k,tostring(v)))
176        context.quad()
177      end
178      col() row()
179    end
180    context.stoptabulate()
181  end
182\stopluacode
183\stopbuffer
184
185\getbuffer[featuresets]
186
187\stopsection
188
189\startsection[title={Color}]
190
191One of the objectives of the oriental \TEX\ project is to bring color to typeset
192Arabic. When Idris started making samples with much manual intervention it was
193about time to figure out if it could be supported by a bit of \LUA\ code.
194
195As the colorization concerns classes of glyphs (like vowels) this is something
196that can best be done after all esthetics have been sorted out. Because things
197like coloring are not part of font technology and because we don't want to misuse
198the \OPENTYPE\ feature mechanisms for that, the solution lays in an extra file
199that describes these goodies.
200
201\startbuffer[goodies-1]
202\definefontfeature
203  [husayni-colored]
204  [goodies=husayni,
205   colorscheme=default,
206   featureset=default]
207\stopbuffer
208
209\startbuffer[goodies-2]
210\start
211  \definedfont[husayni*husayni-colored at 72pt]
212  \righttoleft
213  \resetfontcolorscheme   لُواتيخ ألف ليلة وليلة \par
214  \setfontcolorscheme  [1]لُواتيخ ألف ليلة وليلة \crlf
215  \setfontcolorscheme  [2]لُواتيخ ألف ليلة وليلة \crlf
216\stop
217\stopbuffer
218
219\getbuffer[goodies-1,goodies-2]
220
221The second and third of these three lines have colored vowels and identity marks.
222So how did we get the colors? There are actually two mechanisms involved in this:
223
224\startitemize[packed]
225\startitem we need to associate colorschemes with classed of glyphs \stopitem
226\startitem we need to be able to turn on and off coloring \stopitem
227\stopitemize
228
229The first is done by loading goodies and selecting a colorscheme:
230
231\typebuffer[goodies-1]
232
233Turning on and off coloring is done with two commands (we might provide a proper
234environment for this) as shown in:
235
236\typebuffer[goodies-2]
237
238If you look closely at the feature definition you'll notice that we also choose a
239default featureset. For most (latin) fonts the regular feature definitions are
240convenient, but for fonts that are used for Arabic there are preferred
241combinations of features as there can be many.
242
243Currently the font we use here has the following colorschemes:
244
245\startbuffer[colorschemes]
246\startluacode
247  local goodies = fonts.goodies.load("husayni")
248  local colorschemes = goodies and goodies.colorschemes
249  if colorschemes then
250    local col, row, type = context.NC, context.NR, context.type
251    context.starttabulate { "|l|pl|" }
252    col() context("colorscheme") col() context("numbers") col() row()
253    for colorscheme, numbers in table.sortedpairs(colorschemes) do
254      col() type(colorscheme) col()
255      for i=1,#numbers do
256        type(i)
257        context.quad()
258      end
259      col() row()
260    end
261    context.stoptabulate()
262  end
263\stopluacode
264\stopbuffer
265
266\getbuffer[colorschemes]
267
268\stopsection
269
270\startsection[title={The goodies file}]
271
272In principle a goodies files can contain anuy data that makes sense but in order
273to be useable some entries have a prescribed structure. A goodies file looks as
274follows:
275
276\starttyping
277return {
278  name = "husayni",
279  version = "1.00",
280  comment = "Goodies that complement the Husayni font by Idris Samawi Hamid.",
281  author = "Idris Samawi Hamid and Hans Hagen",
282  featuresets = {
283    default = {
284      key = value, <table>, ...
285    },
286    ...
287  },
288  stylistics = {
289    key = value, ...
290  },
291  colorschemes = {
292    default = {
293      [1] = {
294        "glyph_a.one", "glyph_b.one", ...
295      },
296      ...
297    }
298  }
299}
300\stoptyping
301
302We already saw the list of special features and these are defined in the \type
303{stylistics} stable. In this document, that list was typeset using the following
304(hybrid) code:
305
306\typebuffer[stylistics]
307
308The table with colorscheme that we showed is generated with:
309
310\getbuffer[colorschemes]
311
312In a similar fashion we typeset the featuresets:
313
314\typebuffer[featuresets]
315
316The unprocessed \type {featuresets} table can contain one or more
317named sets and each set can be a mixture of tables and key value
318pairs. Say that we have:
319
320\starttyping
321  default = {
322    kern = "yes", { ss01 = "yes" }, { ss02 = "yes" }, "mark"
323  }
324\stoptyping
325
326Given the previous definition, the order of processing is as follows.
327
328\startitemize[packed,n]
329\startitem \type {{ ss01 = "yes" }} \stopitem
330\startitem \type {{ ss02 = "yes" }} \stopitem
331\startitem \type {mark} (set to \type {"yes"}) \stopitem
332\startitem \type {kern = "yes"} \stopitem
333\stopitemize
334
335So, first we process the indexed part if the list, and next the hash. Already set
336values are not set again. The advantage of using a \LUA\ table is that you can
337simplify definitions. Before we return the table we can define local variables,
338like:
339
340\starttyping
341local one = { ss01 = "yes" }
342local two = { ss02 = "yes" }
343local pos = { kern = "yes", mark = "yes" }
344\stoptyping
345
346and use them in:
347
348\starttyping
349default = {
350  one, two, pos
351}
352\stoptyping
353
354That way we we can conveniently define all kind of interesting combinations
355without the need for many repetitive entries.
356
357The \type {colorsets} table has named subtables that are (currently) indexed by
358number. Each number is associated with a color (at the \TEX\ end) and is coupled
359to a list of glyphs. As you can see here, we use the name of the glyph. We prefer
360this over an index (that can change during development of the font). We cannot
361use \UNICODE\ points as many such glyphs are just variants and have no unique
362code.
363
364\stopsection
365
366\startsection[title={Optimizing Arabic}]
367
368\usemodule[abr-01,narrowtt]
369
370\enabletrackers[fonts.goodies,nodes.optimizer]
371
372The ultimate goal of the Oriental \TEX\ project is to improve the look and feel
373of a paragraph. Because \TEX\ does a pretty good job on breaking the paragraph
374into lines, and because complicating the paragraph builder is not a good idea, we
375finally settled on improving the lines that result from the par builder. This
376approach is rather close to what scribes do and the advanced Husayni font
377provides features that support this.
378
379In principle the current optimizer can replace character expansion but that would
380slow down considerably. Also, for that we first have to clean up the experimental
381\LUA\ based par builder.
382
383After several iterations the following approach was chosen.
384
385\startitemize
386
387\startitem
388    We typeset the paragraph with an optimal feature set. In our case this is
389    \type {husayni-default}.
390\stopitem
391
392\startitem
393    Next we define two sets of additional features: one that we can apply to
394    shrink words, and one that does the opposite.
395\stopitem
396
397\startitem
398    When the line has a badness we don't like, we either stepwise shrink words or
399    stretch them, depending on how bad things are.
400\stopitem
401
402\stopitemize
403
404The set that takes care of shrinking is defined as:
405
406\starttyping
407\definefontfeature
408  [shrink]
409  [husayni-default]
410  [flts=yes,js17=yes,ss05=yes,ss11=yes,ss06=yes,ss09=yes]
411\stoptyping
412
413Stretch has a few more variants:
414
415\starttyping
416\definefontfeature
417  [minimal_stretching]
418  [husayni-default]
419  [js11=yes,js03=yes]
420\definefontfeature
421  [medium_stretching]
422  [husayni-default]
423  [js12=yes,js05=yes]
424\definefontfeature
425  [maximal_stretching]
426  [husayni-default]
427  [js13=yes,js05=yes,js09=yes]
428\definefontfeature
429  [wide_all]
430  [husayni-default]
431  [js11=yes,js12=yes,js13=yes,js05=yes,js09=yes]
432\stoptyping
433
434Next we define a font solution:
435
436\starttyping
437\definefontsolution
438  [FancyHusayni]
439  [goodies=husayni,
440   less=shrink,
441   more={minimal_stretching,medium_stretching,maximal_stretching,wide_all}]
442\stoptyping
443
444Because these featuresets relate quite closely to the font design we don't use
445this way if defining but put the definitions in the goodies file:
446
447\startntyping
448    .....
449    featuresets = { -- here we don't have references to featuresets
450        default = {
451            default,
452        },
453        minimal_stretching = {
454            default, js11 = yes, js03 = yes,
455        },
456        medium_stretching = {
457            default, js12=yes, js05=yes,
458        },
459        maximal_stretching= {
460            default, js13 = yes, js05 = yes, js09 = yes,
461        },
462        wide_all = {
463            default, js11 = yes, js12 = yes, js13 = yes, js05 = yes, js09 = yes,
464        },
465        shrink = {
466            default, flts = yes, js17 = yes, ss05 = yes, ss11 = yes, ss06 = yes, ss09 = yes,
467        },
468    },
469    solutions = { -- here we have references to featuresets, so we use strings!
470        experimental = {
471            less = { "shrink" },
472            more = { "minimal_stretching", "medium_stretching", "maximal_stretching", "wide_all" },
473        },
474    },
475    .....
476\stopntyping
477
478Now the definition looks much simpler:
479
480\startbuffer
481\definefontsolution
482  [FancyHusayni]
483  [goodies=husayni,
484   solution=experimental]
485\stopbuffer
486
487% unhbox to show stretch - shrink
488
489\typebuffer \getbuffer
490
491{\em I want some funny text (complete with translation). Actually I want all
492examples translated.}
493
494\startbuffer[sample]
495قد صعدنا
496ذرى الحقائق بأقدام النبوة و الولاية و نورنا
497سبع طبقات أعلام الفتوى بالهداية فنحن ليوث
498الوغى و غيوث الندى و طعان العدى و فينا السيف و
499القلم في العاجل و لواء الحمد
500و الحوض في الآجل و أسباطنا حلفاء
501الدين و خلفاء النبيين و مصابيح الأمم و مفاتيح
502الكرم فالكليم ألبس حلة الاصطفاء لما عهدنا
503منه الوفاء و روح القدس في جنان الصاقورة ذاق من
504حدائقنا الباكورة و شيعتنا الفئة الناجية و
505الفرقة الزاكية و صاروا لنا ردءا و صونا و على
506الظلمة ألبا و عونا و سينفجر لهم ينابيع
507الحيوان بعد لظى النيران لتمام آل حم و طه و
508الطواسين من السنين و هذا الكتاب درة من درر
509الرحمة و قطرة من بحر الحكمة و كتب الحسن بن
510علي العسكري في سنة أربع و خمسين و مائتين
511\stopbuffer
512
513\startbuffer
514\definedfont[husayni*husayni-default at 24pt]
515% todo: factor ivm grid, so the next line looks hackery:
516\expanded{\setuplocalinterlinespace[line=\the\dimexpr2\lineheight]}
517\setfontsolution[FancyHusayni]% command will change
518\enabletrackers[builders.paragraphs.solutions.splitters.colors]
519\righttoleft \getbuffer[sample] \par
520\disabletrackers[builders.paragraphs.solutions.splitters.colors]
521\resetfontsolution
522\stopbuffer
523
524In the following example the yellow words are stretched and the green ones are
525shrunken.\footnote {Make sure that the paragraph is finished (for instance using
526\type {\par} before resetting it.)}
527
528\typebuffer
529
530\start \getbuffer \stop
531
532% \setfontsolution[FancyHusayni]x\par\resetfontsolution
533% \setfontsolution[FancyHusayni]x\par\resetfontsolution
534% \setfontsolution[FancyHusayni]x\par\resetfontsolution
535% \setfontsolution[FancyHusayni]x\par\resetfontsolution
536% \setfontsolution[FancyHusayni]x\par\resetfontsolution
537% \setfontsolution[FancyHusayni]x\par\resetfontsolution
538% \setfontsolution[FancyHusayni]x\par\resetfontsolution
539% \setfontsolution[FancyHusayni]x\par\resetfontsolution
540
541% \startbuffer[sample]
542% \dorecurse{50}{الحمد \recurselevel\space}
543% \stopbuffer
544
545This mechanism is somewhat experimental as is the (user) interface. It is also
546rather slow compared to normal processing. There is room for improvement but I
547will do that when other components are more stable so that simple variants (that
548we can use here) can be derived.
549
550When criterium~0 used above is changed into for instance~5 processing is faster.
551When you enable a preroll processing is more time consuming. Examples of settings
552are:
553
554\starttyping
555\setupfontsolutions[method={preroll,normal},criterium=2]
556\setupfontsolutions[method={preroll,random},criterium=5]
557\setupfontsolutions[method=reverse,criterium=8]
558\setupfontsolutions[method=random,criterium=2]
559\stoptyping
560
561Using a preroll is slower because it first tries all variants and then settles
562for the best; otherwise we process the first till the last solution till the
563criterium is satisfied.
564
565% {\em Todo: show normal, reverse and random.}
566% {\em Todo: bind setting to paragraph.}
567
568\stopsection
569
570\startsection[title={Protrusion and expansion}]
571
572There are two entries in the goodies file that relate to advanced parbuilding:
573\type {protrusions} and \type {expansions}.
574
575\starttyping
576protrusions = {
577  vectors = {
578    pure = {
579      [0x002C] = { 0, 1 }, -- comma
580      [0x002E] = { 0, 1 }, -- period
581      .....
582    }
583  }
584}
585\stoptyping
586
587These vectors are similar to the ones defined globally but the vectors defined in
588a goodie file are taken instead when present.
589
590\stopsection
591
592\startsection[title={Filenames and properties}]
593
594As filenames and properties of fonts are somewhat of an inconsistent mess, we can
595use the goodies to provide more information:
596
597\starttyping
598files = {
599  name = "antykwapoltawskiego", -- shared
600  list = {
601      ["AntPoltLtCond-Regular.otf"] = {
602       -- name   = "antykwapoltawskiego",
603          style  = "regular",
604          weight = "light",
605          width  = "condensed",
606      },
607      .....
608    }
609  }
610}
611\stoptyping
612
613Internally this will become a lookup tree so that we can have a predictable
614specifier:
615
616\starttyping
617\definefont[MyFontA][antykwapoltawskiego-bold-italic]
618\definefont[MyFontB][antykwapoltawskiego-normal-italic-condensed]
619\definefont[MyFontC][antykwapoltawskiego-light-regular-semicondensed]
620\stoptyping
621
622Of course one needs to load the goodies. One way to force that is:
623
624\starttyping
625\loadfontgoodies[antykwapoltawskiego]
626\stoptyping
627
628The Antykwa Poltawskiego family is rather large and provides all kind of
629combinations.
630
631\startbuffer
632\usemodule[fonts-goodies]
633\showfontgoodiesfiles[name=antykwapoltawskiego]
634\stopbuffer
635
636\startpacked
637\getbuffer
638\stoppacked
639
640This list is generated with:
641
642\typebuffer
643
644\stopsection
645
646\stopchapter
647
648\stopcomponent
649