Source Browser ?

1% language=us
2
3% This only works in MKIV because in LMTX we don't load the font-vir module
4% any longer. No one used it anyway.
5
6\environment mk-environment
7
8\startcomponent mk-punk
9
10\page[right] \start
11
12% the opentype one:
13%
14% \setupbodyfont[punknova]
15
16% the mp based runtime one:
17
18
19\usemodule[m][punk]
20\usetypescript[punk]
21\switchtobodyfont[punk,12pt]
22
23\StartRandomPunk
24
25\definesymbol[1][--]
26\setupsorting[logo][style=]
27\setupcapitals[title=no]
28\setuptype[style=\tf]
29\setuptyping[style=\tf]
30\logo[METAPOST] {MetaPost}
31\logo[METAFONT] {MetaFont}
32
33\chapter{How to convince Don and Hermann to use \LUATEX}
34
35{\em The code shown here should look a bit different in versions
36of \MKIV\ after March 2011. This is because the font system was
37cleaned up and upgraded. The prinicples remain the same. You can
38have a look at \type {m-punk.mkiv} in the \CONTEXT\ distribution.}
39
40Odds are pretty low that Don Knuth will use \LUATEX\ for
41typesetting the next update of his opus magnum, and odds are even
42lower that Hermann Zapf will use \MPLIB\ for Melior Nova. However,
43the next example of combining \METAFONT\ and \TEX\ may draw their
44interest in this new variant: \METATEX.
45
46The font used here is called \quote {punk} and is designed by
47Donald Knuth. There is a note in the file that says: \quotation
48{Font inspired by Gerard and Marjan Unger's lectures, February
491985}. If you didn't notice it yet: punk is a random font.
50
51You may wonder why we started looking into this masterpiece of
52font design. Well, there are a few reasons:
53
54\startitemize
55
56\item  We always liked this font, but after the rise of outline
57       fonts it was not a natural candidate for using in
58       documents. Fun is always a good motive.
59
60\item  For many years we have been suggesting that special glyphs
61       and/or aspects of typesetting could be realized by runtime
62       generation of graphics, and we need this testbed for the
63       Oriental \TEX\ project: Idris needs stretchable inter|-|glyph
64       connections.
65
66\item  Taco likes using tricky \METAPOST\ backgrounds for his
67       presentations that demonstrate this programming language.
68
69\item  Hartmut loves to tweak the backend and runtime font generation
70       will demand some extensions to the font inclusion and literal
71       handlers.
72
73\item  Because Hans attends many \TEX\ conferences together with Volker
74       Schaa, he has promised him to avoid repeating talk and
75       presentation layouts, and so a new presentation style was needed.
76
77\stopitemize
78
79To this we can add an already mentioned motivation: convince Don and
80Hermann to use \LUATEX\ \unknown\ who knows. And, if that fails, maybe
81they can team up for an extensions to this font: more style variants,
82proper math and the full range of \UNICODE\ glyphs.
83
84The punk font is written in \METAFONT\ and there are multiple
85sources. These are merged into one file which is to be processed
86using the \type {mfplain} format. Definitions of characters in
87this font look like:
88
89\starttyping
90beginpunkchar ("A",13,1,2) ;
91    z1 = pp(1.5u,0) ; z2 = (.5w,1.1h) ; z3 = pp(w-1.5u,0) ;
92    pd z1 ; pd z3 ; draw z1 -- z2 -- z3 ;
93    z4 = pp .3[z1,z2] ; z5 = pp .3[z3,z2] ;
94    pd z4 ; pd z5 ; draw z4 -- z5;
95endchar ;
96\stoptyping
97
98When \TEX\ needs a font, i.e.\ when we have something like this:
99
100\starttyping
101\font\somefont=whatever at 12pt
102\stoptyping
103
104in \CONTEXT\ control is delegated to a font loader written in
105\LUA\ that is hooked into \TEX. This loader interprets the name
106and if needed filters the specification from it. Think of this:
107
108\starttyping
109\font\somefont=whatever*smallcaps at 16pt
110\stoptyping
111
112This means: load font \type {whatever} and enable the smallcaps features.
113However this mechanism is mostly geared towards \TYPEONE\ and \OPENTYPE\
114fonts. But punk is neither: it's a \METAFONT, and we need to treat it as
115such. We will use \LUATEX's powerful virtual font technology
116because that way we can smuggle the proper shapes in the final
117file. And \unknown\ no bitmaps and no funny encoding.
118
119In \CONTEXT\ \MKIV\ there is a preliminary virtual font definition
120mechanism. There is no advanced \TEX\ interface yet so we need to do it in
121\LUA. Fortunately we do have access to this from the font mechanism:
122
123\starttyping
124\font\somefont=mypunk@punk at 20pt
125\stoptyping
126
127This is a rather valid directive to create a font that internally
128will be called \type {mypunk}. For this the virtual font creation
129command \type {punk} will be used, and in a moment we will see what
130this triggers.
131
132Of course, users will never see such low level definitions. They will
133use proper typescript, which set up a whole font system. For instance,
134in this document we use:
135
136\typebuffer[fontdefinition]
137
138Now, using punk in inself is not that much of a challenge, but how about
139using multiple instances of this font and then typeset the text chosing
140variants of a glyph at random. Of course this will have some trade-off in
141terms of runtime. In this document we use punk as the bodyfont and
142therefore it comes in several sizes. On Hans's laptop generating the
143glyphs takes a while:
144
145\starttyping
1467500 glyphs, 12.887 seconds runtime, 581 glyphs/second
147\stoptyping
148
149Fortunately \MKIV\ provides a caching mechanism so once the fonts
150are generated, a next run will be more comfortable. This time we
151get reported:
152
153\starttyping
1540.187 seconds, 60 instances, 320.856 instances/second
155\stoptyping
156
157which is not that bad for loading 60 files of 5 megabytes \PDF\
158literals each. The reason why the files are large is that although
159these glyphs look simple, in fact they are rather complex: each
160glyph at least one paths and several knots, and since a special
161pen is used, conversion results in a larger than normal description
162of a shape.
163
164Since we use the standard converter from \METAPOST\ to \PDF, we
165can gain some generation time by using a dedicated converter for
166glyphs. Eventually the \MPLIB\ library may even provide a proper
167charstring generator so that we can construct real fonts at
168runtime.
169
170So, how does this work behind the screens? Because we can use some
171of the mechanisms already present in \CONTEXT\ it is not even that complex.
172
173\startitemize
174
175\item The \type {punk} directive tells \CONTEXT\ to create a virtual
176      font. Such a font can be made out of real fonts; we use this
177      for instance in the font feature \type {combine}, where we
178      add virtually composed characters that are missing by combining
179      characters present. However, here we have no real font.
180
181\item And so this virtual font is not build on top of an existing font, but
182      spawns a \MPLIB\ process that will build the font, unless it is
183      present in the cache on disk. The shapes are converted to \PDF\ literals
184      and for each character a proper definition table is made.
185
186\item In total 10 such fonts are made, but only one is returned to the
187      font callback that asked us to provide the font. The list of
188      the alternatives is stored in the \LUA\ table that represents
189      the font and kept at the \LUA\ end. So, for each size used,
190      a unique set of 10 variants is generated.
191
192\item The randomizer operates on the node list. Instead of using a
193      dedicated mechanism for this, we hijack one of the attribute values
194      of the case swapper already present in \MKIV. After that we can selectively
195      turn on and off the randomizer.
196
197\item At some point \TEX\ will hand over the node lists to \CONTEXT. At
198      that moment a lot of things can happen to the list, and one of
199      them is a sequence of character handlers, of which the mentioned case
200      handler is one. The handler sweeps over the nodelist
201      and for each glyph node triggers a function that is bound to the
202      attribute value.
203
204\item This function is rather trivial: it looks at the font id of the
205      glyph, and resolves it to the font table. If that table has a
206      list of alternatives, it will randomly choose one and assign it to
207      the font attribute of the glyph. That's all.
208
209\item Eventually the backend routines will inject the \PDF\ literals that
210      were collected in the commands table of the virtual glyph.
211
212\stopitemize
213
214It will not come as a surprise that our resulting file is larger
215than what we get when using traditional outline fonts or just one
216instance of punk. However, this is just an experiment, and
217eventually a proper font constructor will be provided, so that the
218glyph drawing is delegated to the font renderer. An intermediate
219optimization can be to use so called \PDF\ xforms, but a properly
220runtime generated font is best because then we can search in the
221file too.
222
223Because by now reading the punk font should go fluently we can now
224move on to the code. We already have a \type {fonts} namespace,
225which we now extend with an \METAPOST\ sub namespace:
226
227\starttyping
228fonts.mp = fonts.mp or { }
229\stoptyping
230
231We set a version number and define a cache on disk. When the number changes
232fonts stored in the cache will be regenerated when needed. The
233\type {containers} module provides the relevant function.
234
235\starttyping
236fonts.mp.version = 1.01
237fonts.mp.cache = containers.define("fonts", "mp", fonts.mp.version, true)
238\stoptyping
239
240We already have a \type {metapost} namespace, and within it we define a
241sub namespace:
242
243\starttyping
244metapost.characters = metapost.characters or { }
245\stoptyping
246
247Now we're ready for the real action: we define a dedicated flusher
248that will be passed to the \METAPOST\ converter. A next version of
249\MPLIB\ will provide the \TFM\ font information which gives better
250glyph dimensions, plus additional kerning information. All this code
251is defined in a closure (\type {do ... end}) which
252nicely hides the local variables.
253
254\starttyping
255local characters, descriptions = { }, { }
256local factor, total, variants = 100, 0, 0
257local l, n, w, h, d  = { }, 0, 0, 0, 0
258
259local flusher = {
260    startfigure = function(chrnum,llx,lly,urx,ury)
261        l, n = { }, chrnum
262        w, h, d = urx - llx, ury, -lly
263        total = total + 1
264    end,
265    flushfigure = function(t)
266        for i=1, #t do
267            l[#l+1] = t[i]
268        end
269    end,
270    stopfigure = function()
271        local cd = characters.data[n]
272        descriptions[n] = {
273            unicode = n,
274            name = cd and cd.adobename,
275            width = w*100,
276            height = h*100,
277            depth = d*100,
278        }
279        characters[i] = {
280            commands = {
281                { "special", "pdf: " .. table.concat(l," ") },
282            }
283        }
284    end
285}
286\stoptyping
287
288In the normal converter, the start and stop function do the
289packaging in a box. The flush function is called when literals
290need to be flushed. This threesome does as much as collecting
291glyph information in the \type {list} table. Intermediate literals
292are stored in the \type {l} table. Each glyph has a description and
293(in this case) one command that defines the virtual shape. The name
294is picked up from the character data table that is present in \MKIV.
295
296As told before we generate multiple instances per requested font
297and here is how it happens. We initialize the \type {mfplain}
298format and reset it afterwards. The punk definition file is
299adapted for multiple runs. Scaling happens here because later on
300the scaler has no knowledge about what is present in the commands.
301We use a few helpers for processing the \METAPOST\ code and format
302the final font table in a way \CONTEXT\ \MKIV\ likes. Currently
303the parameters (font dimensions) are rather hard coded, but this
304will change when \MPLIB\ can provide them.
305
306\starttyping
307function metapost.characters.process(mpxformat, name, instances, scalefactor)
308    statistics.starttiming(metapost.characters)
309    scalefactor = scalefactor or 1
310    instances = instances or 10
311    local fontname = file.removesuffix(file.basename(name))
312    local hash = file.robustname(string.format(
313        "%s %04i %04i", fontname, scalefactor*1000, instances))
314    local lists = containers.read(fonts.mp.cache, hash)
315    if not lists then
316        statistics.starttiming(flusher)
317        local data = io.loaddata(resolvers.findfile(name))
318        metapost.reset(mpxformat)
319        lists = { }
320        for i=1,instances do
321            characters, descriptions = { }
322            metapost.process(
323                mpxformat,
324                {
325                    "randomseed := " .. i*10 .. ";",
326                    "scale_factor := " .. scalefactor .. " ;",
327                    data
328                },
329                false,
330                flusher
331            )
332            lists[#lists+1] = {
333                designsize = 655360,
334                name = string.format("%s-%03i",hash,i),
335                parameters = {
336                    slant         =    0,
337                    space         =  333   * scalefactor,
338                    space_stretch =  166.5 * scalefactor,
339                    space_shrink  =  111   * scalefactor,
340                    x_height      =  431   * scalefactor,
341                    quad          = 1000   * scalefactor,
342                    extra_space   =    0
343                },
344                ["type"] = "virtual",
345                characters = characters,
346                descriptions = descriptions,
347            }
348        end
349        metapost.reset(mpxformat) -- saves memory
350        lists = containers.write(fonts.mp.cache, hash, lists)
351        statistics.stoptiming(flusher)
352    end
353    variants = variants + #lists
354    statistics.stoptiming(metapost.characters)
355    return lists
356end
357\stoptyping
358
359We're not yet there. This was just a font generator that returns
360a list of fonts defined in a format liked by \MKIV\ and not that
361far from what \TEX\ wants back from us. Next we define the
362main definition function, the one that is called when the font
363is defined as virtual font. The special number \type {-1000}
364tells the scaler to honour the designsize, which boils down to
365no scaling, but just copying to the final  table that is passed
366to \TEX. The \type {define} function returns an id which we will
367use later.
368
369The scaler uses the \type {descriptions} to add dimensions (and other data
370needed) in the \type {characters} table. This is something \MKIV\ specific.
371
372\starttyping
373function fonts.handlers.vf.combiner.commands.metafont(g,v)
374    local size = g.specification.size
375    local data = metapost.characters.process(v[2],v[3],v[4],size/655360)
376    local list, t = { }, { }
377    for d=1,#data do
378        t = data[d]
379        t = fonts.constructors.scale(t, -1000)
380        t.id = font.define(t)
381        list[#list+1] = t.id
382    end
383    for k, v in pairs(t) do
384        g[k] = v -- kind of replace, when not present, make nil
385    end
386    g.variants = list
387end
388\stoptyping
389
390We hook this into the \CONTEXT\ font handler and from now on
391the \type {@punk} is recognized:
392
393\starttyping
394fonts.definers.methods.install( "punk", { { "metafont", "mfplain", "punkfont.mp", 10 } } )
395\stoptyping
396
397Now that we can define the font, we need to deal with
398the randomizer. This is optional fun. The mentioned case swappers
399are implemented in the \type {cases} namespace:
400
401\starttyping
402local fontdata = fonts.hashes.identifiers
403
404cases.actions[99] = function(current)
405    local c = current.char
406    local used = fontdata[current.font].variants
407    if used then
408        local f = math.random(1,#used)
409        current.font = used[f]
410        return current, true
411    else
412        return current, false
413    end
414end
415\stoptyping
416
417This function is called in one of the passes over the node
418list. Thanks to this framework we don't need that much code.
419We didn't show two statistics functions. They are the reason why
420we keep track of the total number of glyphs defined.
421
422This leaves us defining the interface, so here we go:
423
424\starttyping
425\def\StartRandomPunk{\begingroup\setcharactercasing[99]}
426\def\StopRandomPunk {\endgroup}
427\stoptyping
428
429The set command just sets the attribute that we associated
430with casing (one of the many attributes). The number 99 is
431rather arbitrary.
432
433If you follow the development of \LUATEX\ and \MKIV\ (we do talks at
434conferences, keep track of the development history in \type {mk.pdf},
435and report on the \CONTEXT\ mailing list) you will have noticed  that
436we often use somewhat extreme examples to explore and test the
437functionality and this is no exception. As usual it helped us to improve
438the code and extend our todo list. Can the previous code convince
439the grand wizards to start using \LUATEX ? Probably not. Let's
440anyway hope that they will put the addition of punk math to their todo
441list. In the meantime we've already started adding missing characters:
442
443\startlinecorrection[blank]
444    \hbox to \hsize \bgroup \hss % { ' \ " }
445        \dorecurse{6}{\hbox{\char123\enspace\char39\enspace\char92\enspace\char34\enspace\char125}\quad}\unskip
446    \hss \egroup
447\stoplinecorrection
448
449Also, because we can be sure that Mojca Miklavec's first test will
450be if her favourite characters \color [mkcolor] {\ccaron}, \color
451[mkcolor] {\scaron} and \color [mkcolor] {\zcaron} are supported,
452we made sure that we composed those accented characters as well.
453\footnote {This is accomplished by adding \type
454{composecharacters(t)} at an undisclosed location in
455the previous code.}
456
457\StopRandomPunk \page[right] \stop
458
459\stopcomponent
460