Source Browser ?

1% language=us
2
3\useMPlibrary[dum]
4
5\startcomponent mk-mplib
6
7\environment mk-environment
8
9\definetextbackground
10    [sideline]
11    [mp=mpos:par:sideline,
12     framecolor=mkcolor,
13     frameoffset=5mm]
14
15\startuseMPgraphic{mpos:par:sideline}{linecolor,lineoffset}
16    for i=1 upto nofmultipars :
17        fill leftboundary multipars[i]
18            shifted (-\MPvar{lineoffset},0)
19            rightenlarged 1mm withcolor \MPvar{linecolor} ;
20    endfor ;
21\stopuseMPgraphic
22
23\chapter{The MetaPost Library}
24
25% \subject{Hans Hagen \& Taco Hoekwater} \blank[3*big]
26
27This chapter is written by Taco and Hans around the time
28that \MPLIB\ was integrated into \LUATEX.
29It is part of our torture test.
30
31\subject{introduction}
32
33If \METAPOST\ support had not been as tightly integrated into
34\CONTEXT\ as it is, at least half of the projects \PRAGMA\ has
35been doing in the last decade could not have been done at all.
36\starttextbackground[sideline]Take for instance
37backgrounds behind text or graphic markers alongside text. These are
38probably the most complex mechanisms in \CONTEXT: positions are
39stored, and positional information is passed on to
40\METAPOST, where intersections between the text areas and the running
41text are converted into graphics that are then positioned in the
42background of the text.\stoptextbackground{} Underlining of text
43(sometimes used in the educational documents that we typeset) and
44change bars (in the margins) are implemented using the same
45mechanism because those are basically a background with only one of the
46frame sides drawn.
47
48You can probably imagine that a 300 page document with several such
49graphics per page takes a while to process. A nice example of such
50integrated graphics is the \LUATEX\ reference manual, that has an
51unique graphic at each page: a stylized image of a revolving moon.
52
53\startuseMPgraphic{lualogo-x}{angle}
54color   luaplanetcolor ; luaplanetcolor := .5blue  ;
55color   luaholecolor   ; luaholecolor   :=   white ;
56numeric luaextraangle  ; luaextraangle  := \MPvar{angle} ;
57
58vardef lualogo = image (
59    % Graphic design by A. Nakonechnyj. Copyright (c) 1998, All rights reserved.
60    save luaorbitcolor, d, r, p ; color luaorbitcolor ; numeric d, r, p ;
61    luaorbitcolor := .5luaholecolor ; d := sqrt(2)/4 ; r := 1/4 ; p := r/8 ;
62    fill fullcircle scaled 1 withcolor luaplanetcolor ;
63    draw fullcircle rotated 40.5 scaled (1+r) dashed evenly scaled p withpen pencircle scaled (p/2) withcolor luaorbitcolor ;
64    fill fullcircle scaled r shifted (d+1/8,d+1/8) rotated luaextraangle withcolor luaplanetcolor ;
65    fill fullcircle scaled r shifted (d-1/8,d-1/8) withcolor luaholecolor   ;
66    setbounds currentpicture to fullsquare scaled 1.3;
67)  enddef ;
68
69draw lualogo scaled 1cm ;
70\stopuseMPgraphic
71
72\startlinecorrection[blank]
73\hbox to \hsize
74  {\hss\dostepwiserecurse{0}{360}{60}{\useMPgraphic{lualogo-x}{angle=\recurselevel}\hss}}
75\stoplinecorrection
76
77Most of the running time integrating such graphics seemed to be
78caused by the mechanics of the process: starting the separate
79\METAPOST\ interpreter and having to deal with a number of
80temporary files. Therefore our expectations were high with regards
81to integrating \METAPOST\ more tightly into \LUATEX. Besides the
82speed gain, it also true that the simpler the process of using
83such use of graphics becomes, the more modern a \TEX\ runs looks
84and the less problems new users will have with understanding how
85all the processes cooperate.
86
87This article will not discuss the application interface of the
88\MPLIB\ library in detail, for that there is the \LUATEX\ manual. In short,
89using the embedded \METAPOST\ interpreter in \LUATEX\ boils
90down to the following:
91
92\startitemize[packed]
93\item Open an instance using \type {mplib.new}, either to process
94      images with a format to be loaded, or to create such a format.
95      This function returns a library object.
96\item Execute sequences of \METAPOST\ commands, using the object's
97      \type{execute} method. This returns a result.
98\item Check if the result is valid and (if it is okay) request the list
99      of objects. Do whatever you want with them, most probably
100      convert them to some output format. You can also request a
101      string representation of a graphic in \POSTSCRIPT\ format.
102\stopitemize
103
104There is no need to close the library object. As long as you didn't make
105any fatal errors, the library recovers well and can stay alive during
106the entire \LUATEX\ run.
107
108Support for \MPLIB\ depends on a few components: integration,
109conversion and extensions. This article shows some of the code
110involved in supporting the library. Let's start with the conversion.
111
112\subject{conversion}
113
114The result of a \METAPOST\ run traditionally is a \POSTSCRIPT\
115language description of the generated graphic(s). When
116\PDF\ is needed, that \POSTSCRIPT\ code has to be converted to the target
117format. This includes embedded text as well as penshapes used for
118drawing. To demonstrate, here is a simple example graphic:
119
120\startluacode
121document.sample_mp_graphic = [[
122    draw fullcircle
123        scaled 2cm
124        withpen pencircle xscaled 1mm yscaled .5mm rotated 30
125        withcolor .75red ;
126]]
127\stopluacode
128
129\startbuffer[mpexample]
130draw fullcircle
131  scaled 2cm
132  withpen pencircle xscaled 1mm yscaled .5mm rotated 30
133  withcolor .75red ;
134\stopbuffer
135
136\placefigure
137  [left]
138  {}
139  {\startMPcode \ctxlua{tex.sprint(document.sample_mp_graphic)}\stopMPcode}
140
141\typebuffer[mpexample]
142
143Notice how the pen is not a circle but a rotated ellipse. Later on it
144will become clear what the consequences of that are for the conversion.
145
146How does this output look in \POSTSCRIPT ? If the preamble is left out
147it looks like this:
148
149\startbuffer
150\startluacode
151do
152  local mpx = metapost.format("metafun")
153  local result = mpx:execute(string.format("beginfig(0);%s;endfig;",document.sample_mp_graphic))
154  local str = result.fig[1]:postscript()
155  str = str:gsub("%%BeginProlog.-%%EndProlog","%% <<stripped preamble was here>>")
156  tex.sprint(tex.ctxcatcodes,"\\starttyping\n")
157  tex.sprint((str:gsub("[\n\r]$","")))
158  tex.sprint("\\stoptyping")
159end
160\stopluacode
161\stopbuffer
162
163\getbuffer
164
165The most prominent code here concerns the path. The numbers in brackets define
166the transformation matrix for the pen we used. The \PDF\ variant looks as follows:
167
168\startbuffer
169\startluacode
170do
171  local mpx = metapost.format("metafun")
172  local result = mpx:execute(string.format("beginfig(0);%s;endfig;",document.sample_mp_graphic))
173  local list = metapost.pdfliterals(result)
174  tex.sprint(tex.ctxcatcodes,"\\starttyping")
175  tex.sprint(table.concat(list,"\n"))
176  tex.sprint("\\stoptyping")
177end
178\stopluacode
179\stopbuffer
180
181\getbuffer
182
183The operators don't look much different from the \POSTSCRIPT, which is
184mostly due to the fact that in the \POSTSCRIPT\ code, the preamble
185defines shortcuts like \type {c} for \type {curveto}.  Again, most code
186involves the path. However, this time the numbers are different and
187the transformation comes before the path.
188
189In the case of \PDF\ output, we could use \TEX\ itself to do the
190conversion: a generic converter is implemented in \type
191{supp-pdf.tex}, while a converter optimized for \CONTEXT\ \MKII\ is
192defined in the files whose names start with \type {meta-pdf}.  But in
193\CONTEXT\ \MKIV\ we use \LUA\ code for the conversion instead. Thanks to
194\LUA's powerful \LPEG\ parsing library, this gives cleaner code and is
195also faster. This converter currently lives in \type {mlib-pdf.lua}.
196
197Now, with the embedded \METAPOST\ library, conversion goes different
198still because now it is possible to request the drawn result and
199associated information in the form of \LUA\ tables.
200
201\startbuffer
202\startluacode
203do
204  local mpx = metapost.format("metafun")
205  local result = mpx:execute(string.format("beginfig(0);%s;endfig;",document.sample_mp_graphic))
206  local str = table.serialize(metapost.totable(result), "figure")
207  tex.sprint(tex.ctxcatcodes,"\\starttyping")
208  tex.sprint((str:gsub("[\n\r]$","")))
209  tex.sprint("\\stoptyping")
210end
211\stopluacode
212\stopbuffer % $<-emacs font-lock
213
214% \startsimplecolumns[n=3,width=18.5cm]
215% \getbuffer
216% \stopsimplecolumns
217
218\getbuffer
219
220This means that instead of parsing \POSTSCRIPT\ output, we now can operate on
221a proper datastructure and get code like the following:
222
223\starttyping
224function convertgraphic(result)
225  if result then
226    local figures = result.fig
227    if figures then
228      for fig in ipairs(figures) do
229        local llx, lly, urx, ury = unpack(fig:boundingbox())
230        if urx > llx then
231          startgraphic(llx, lly, urx, ury)
232          for object in ipairs(fig:objects()) do
233            if object.type == "..." then
234              ...
235              flushgraphic(...)
236              ...
237            else
238              ...
239            end
240          end
241          finishgraphic()
242        end
243      end
244    end
245  end
246end
247\stoptyping
248
249Here \type {result} is what the library returns when one or more
250graphics are processed. As you can deduce from this snippet, a
251result can contain multiple figures. Each figure corresponds with a
252\type {beginfig} \type {...} \type {endfig}. The graphic operators that
253the converter generates (so called \PDF\ literals) have to be
254encapsulated in a proper box so this is why we have:
255
256\startitemize[packed]
257\item \type {startgraphic}: start packaging the graphic
258\item \type {flushgraphic}: pipe literals to \TEX
259\item \type {finishgraphic}: finish packaging the graphic
260\stopitemize
261
262It does not matter what number you passed to \type {beginfig}, the
263graphics come out in the natural order.
264
265Little over half a dozen different object types are possible. The example
266\METAPOST\ \type{draw} command from above results in an \type {outline} object.
267This object contains not only path information but also carries
268rendering data, like the color and the pen. So, in the end we will
269flush code like \type {1 M} which sets the \type {miterlimit} to one
270or \type {.5 g} which sets the color to 50\% gray, in addition to a
271path.
272
273Because objects are returned in a way that closely resembles a
274\METAPOST's internals, some extra work needs to be done in order to
275calculate paths with elliptical pens. An example of a helper function
276in somewhat simplified form is shown next:
277
278\starttyping
279function pen_characteristics(object)
280  local p = object.pen[1]
281  local wx, wy, width
282  if p.right_x == p.x_coord and p.left_y == p.y_coord then
283    wx = abs(p.left_x  - p.x_coord)
284    wy = abs(p.right_y - p.y_coord)
285  else -- pyth: sqrt(a^2 +b^2)
286    wx = pyth(p.left_x - p.x_coord, p.right_x - p.x_coord)
287    wy = pyth(p.left_y - p.y_coord, p.right_y - p.y_coord)
288  end
289  if wy/coord_range_x(object.path, wx) >=
290                    wx/coord_range_y(object.path, wy) then
291    width = wy
292  else
293    width = wx
294  end
295  local sx, sy = p.left_x, p.right_y
296  local rx, ry = p.left_y, p.right_x
297  local tx, ty = p.x_coord, p.y_coord
298  if width ~= 1 then
299    if width == 0 then
300      sx, sy = 1, 1
301    else
302      rx, ry, sx, sy = rx/width, ry/width, sx/width, sy/width
303    end
304  end
305  if abs(sx) < eps then sx = eps end
306  if abs(sy) < eps then sy = eps end
307  return sx, rx, ry, sy, tx, ty, width
308end
309\stoptyping
310
311If \type {sx} and \type {sy} are 1, there is no need to transform
312the path, otherwise a suitable transformation matrix is calculated
313and returned. The function itself uses a few helpers that make the
314calculations even more obscure. This kind of code does not fall in
315the category trivial and as already mentioned, these basic
316algorithms were derived from the \METAPOST\ sources. Even so,
317these snippets demonstrate that interfacing using \LUA\ does not
318look that bad.
319
320In the actual \MKIV\ code things look a bit different because it does
321a bit more and uses optimized code. There you will also find the code
322dealing with the actual transformation, of which these helpers are
323just a portion.
324
325If you compare the \POSTSCRIPT\ and the \PDF\ code you will notice
326that the paths looks different. This is because the use and
327application of a transformation matrix in \PDF\ is different from how
328it is handled in \POSTSCRIPT. In \PDF\ more work is assumed to be
329done by the \PDF\ generating application. This is why in both the
330\TEX\ and the \LUA\ based converters you will find transformation code
331and the library follows the same pattern. In that respect \PDF\
332differs fundamentally from \POSTSCRIPT.
333
334Within the \TEX\ based converter there was the problem of keeping the
335needed calculations within \TEX's accuracy, which fortunately permits
336larger values that \METAPOST\ can produce. This plus the parsing code
337resulted in a not-that-easy to follow bunch of \TEX\ code.  The \LUA\
338based parser is more readable, but since it also operates on
339\POSTSCRIPT\ code it is kind of unnatural too, but at least there are
340less problems with keeping the calculations sane. The \MPLIB\ based
341converter is definitely the cleanest and least sensitive to future
342changes in the \POSTSCRIPT\ output. Does this mean that there is no
343ugly code left? Alas, as we will see in the next section, dealing with
344extensions is still somewhat messy. In practice users will not be
345bothered with such issues, because writing a converter is a one time
346job by macro package writers.
347
348\subject{extensions}
349
350In \METAFUN, which is the \METAPOST\ format used with \CONTEXT,
351a few extensions are provided, like:
352
353\startitemize[packed]
354\item cmyk, spot and multitone colors
355\item including external graphics
356\item lineair and circulair shades
357\item texts converted to outlines
358\item inserting arbitrary texts
359\stopitemize
360
361Until now, most of these extensions have been implemented by using
362specially coded colors and by injecting so called specials (think of
363them as comments) into the output. On one of our trips to a \TEX\
364conference, we discussed ways to pass information along with
365paths and eventually we arrived at associating text strings with paths
366as a simple and efficient solution.  As a result, recently \METAPOST\
367was extended by \type {withprescript} and \type {withpostscript}
368directives. For those who are unfamiliar with these new scripts,
369they are used as follows:
370
371\starttyping
372draw fullcircle withprescript "hello" withpostscript "world" ;
373\stoptyping
374
375In the \POSTSCRIPT\ output these scripts end up before and after the
376path, but in the \PDF\ converter they can be overloaded to implement
377extensions, and that works reasonably well.  However, at the moment
378there cannot be multiple pre- and postscripts associated with a single
379path inside the \METAPOST\ internals. This means that for the moment,
380the scripts mechanism is only used for a few of the extensions.
381Future versions of \MPLIB\ may provide more sophisticated methods
382for carrying information around.
383
384The \MKIV\ conversion mechanism uses scripts for graphic inclusion,
385shading and text processing but unfortunately cannot use them for
386more advanced color support.
387
388A nasty complication is that the color spaces in \METAPOST\ don't
389cast, which means that one cannot assign any color to a color
390variables: each colorspace has it's own type of variable.
391
392\starttyping
393color     one ; one := (1,1,0)   ; % correct
394cmykcolor two ; two := (1,0,0,1) ; % correct
395one := two ; % error
396fill fullcircle scaled 1cm withcolor .5[one,two] ; % error
397\stoptyping
398
399In \CONTEXT\ we use constructs like this:
400
401\starttyping
402\startreusableMPgraphic{test}
403  fill fullcircle scaled 1cm withcolor \MPcolor{mycolor} ;
404\stopreusableMPgraphic
405
406\reuseMPgraphic{test}
407\stoptyping
408
409Because \type {withcolor} is clever enough to understand what color
410type it receives, this is ok, but how about:
411
412\starttyping
413\startreusableMPgraphic{test}
414  color c ; c := \MPcolor{mycolor} ;
415  fill fullcircle scaled 1cm withcolor c ;
416\stopreusableMPgraphic
417\stoptyping
418
419Here the color variable only accepts an \RGB\ color and because in
420\CONTEXT\ there is mixed color space support combined with
421automatic colorspace conversions, it doesn't know in advance what type
422it is going to get. By implementing color spaces other than \RGB\
423using special colors (as before) such type mismatches can be avoided.
424
425The two techniques (coding specials in colors and pre|/|postscripts)
426cannot be combined because a script is associated with a path and
427cannot be bound to a variable like \type{c}. So this again is an argument
428for using special colors that remap onto \CMYK\, spot or
429multi|-|tone colors.
430
431Another area of extensions is text. In previous versions of
432\CONTEXT\ the text processing was already isolated: text ended
433up in a separate file and was processed in an separate run. More
434recent versions of \CONTEXT\ use a more abstract model of boxes that
435are preprocessed before a run, which avoids the external run(s). In
436the new approach everything can be kept internal. The conversion
437even permits constructs like:
438
439\starttyping
440for i=1 upto 100 :
441  draw btex oeps etex rotated i ;
442endfor ;
443\stoptyping
444
445but since this construct is kind of obsolete (at least in the library
446version of \METAPOST) it is better to use:
447
448\starttyping
449for i=1 upto 100 :
450  draw textext("cycle " & decimal i) rotated i ;
451endfor ;
452\stoptyping
453
454Internally a trial pass is done so that indeed 100 different texts will
455be drawn. The throughput of texts is so high that in practice one will
456not even notice that this happens.
457
458Dealing with text is yet another example of using \LPEG. The following
459snippet of code sheds some light on how text in graphics is dealt with.
460Actually this is a variation on a previous implementation. That one
461was slightly faster but looked more complex. It was also not robust for
462complex texts defined in macros in a format.
463
464\starttyping
465local P, S, V, Cs = lpeg.P, lpeg.S, lpeg.V, lpeg.Cs
466
467local btex    = P("btex")
468local etex    = P(" etex")
469local vtex    = P("verbatimtex")
470local ttex    = P("textext")
471local gtex    = P("graphictext")
472local spacing = S(" \n\r\t\v")^0
473local dquote  = P('"')
474
475local found = false
476
477local function convert(str)
478  found = true
479  return "textext(\"" .. str .. "\")"
480end
481local function ditto(str)
482  return "\" & ditto & \""
483end
484local function register()
485  found = true
486end
487
488local parser = P {
489    [1] = Cs((V(2)/register + V(3)/convert + 1)^0),
490    [2] = ttex + gtex,
491    [3] = (btex + vtex) * spacing *
492                Cs((dquote/ditto + (1-etex))^0) * etex,
493}
494
495function metapost.check_texts(str)
496  found = false
497  return parser:match(str), found
498end
499\stoptyping
500
501If you are unfamiliar with \LPEG\ it may take a while to see what
502happens here: we replace the text between \type {btex} and \type {etex}
503by a call to \type {textext}, a macro. Special care is given to
504embedded double quotes.
505
506When text is found, the graphic is processed two times. The definition
507of \type {textext} is different for each run. The first run we have:
508
509\starttyping
510vardef textext(expr str) =
511    image (
512        draw unitsquare
513            withprescript "tf"
514            withpostscript str ;
515    )
516enddef ;
517\stoptyping
518
519After the first run the result is not really converted, but just
520the outlines with the \type {tf} prescript are filtered. In the
521loop over the object there is code like:
522
523\starttyping
524local prescript = object.prescript
525if prescript then
526  local special = metapost.specials[prescript]
527  if special then
528    special(object.postscript,object)
529  end
530end
531\stoptyping
532
533Here, \type {metapost} is just the namespace used by the
534converter. The prescript tag \type {tf} triggers a function:
535
536\starttyping
537function metapost.specials.tf(specification,object)
538  tex.sprint(tex.ctxcatcodes,format("\\MPLIBsettext{%s}{%s}",
539    metapost.textext_current,specification))
540  if metapost.textext_current < metapost.textext_last then
541    metapost.textext_current = metapost.textext_current + 1
542  end
543  ...
544end
545\stoptyping
546
547Again, you can forget about the details of this function. Important is
548that there is a call out to \TEX\ that will process the text. Each snippet
549gets the number of the box that holds the content. The macro that is
550called just puts stuff in a box:
551
552\starttyping
553\def\MPLIBsettext#1#2%
554  {\global\setbox#1\hbox{#2}}
555\stoptyping
556
557In the next processing cycle of the \METAPOST\ code, the \type {textext}
558macro does something different :
559
560\starttyping
561vardef textext(expr str) =
562    image (
563        _tt_n_ := _tt_n_ + 1 ;
564        draw unitsquare
565            xscaled _tt_w_[_tt_n_]
566            yscaled (_tt_h_[_tt_n_] + _tt_d_[_tt_n_])
567            withprescript "ts"
568            withpostscript decimal _tt_n_ ;
569    )
570enddef ;
571\stoptyping
572
573This time the by then known dimensions of the box that is used to
574store the snippet are used. These are stored in the \type {_tt_w_},
575\type {_tt_h_} and \type {_tt_d_} arrays. The arrays are defined by
576\LUA\ using information about the boxes, and passed to the library
577before the second run. The result from the second \METAPOST\ run
578is converted, and again the prescript is used as trigger:
579
580\starttyping
581function metapost.specials.ts(specification,object,result)
582    local op = object.path
583    local first, second, fourth  = op[1], op[2], op[4]
584    local tx, ty = first.x_coord      , first.y_coord
585    local sx, sy = second.x_coord - tx, fourth.y_coord - ty
586    local rx, ry = second.y_coord - ty, fourth.x_coord - tx
587    if sx == 0 then sx = 0.00001 end
588    if sy == 0 then sy = 0.00001 end
589    metapost.flushfigure(result)
590    tex.sprint(tex.ctxcatcodes,format(
591        "\\MPLIBgettext{%f}{%f}{%f}{%f}{%f}{%f}{%s}",
592        sx,rx,ry,sy,tx,ty,metapost.textext_current))
593    ...
594end
595\stoptyping
596
597At this point the converter is actually converting the graphic and
598passing \PDF\ literals to \TEX. As soon as it encounters a text,
599it flushes the \PDF\ code collected so far and injects some \TEX\
600code. The \TEX\ macro looks like:
601
602\starttyping
603\def\MPLIBgettext#1#2#3#4#5#6#7%
604  {\ctxlua{metapost.sxsy(\number\wd#7,\number\ht#7,\number\dp#7)}%
605   \pdfliteral{q #1 #2 #3 #4 #5 #6 cm}%
606   \vbox to \zeropoint{\vss\hbox to \zeropoint
607     {\scale[sx=\sx,sy=\sy]{\raise\dp#7\box#7}\hss}}%
608   \pdfliteral{Q}}
609\stoptyping
610
611Because text can be transformed, it needs to be scale back to the right
612dimensions, using both the original box dimensions and the transformation
613of the unitquare associated with the text.
614
615\starttyping
616local factor = 65536*(7200/7227)
617
618function metapost.sxsy(wd,ht,dp) -- helper for text
619  commands.edef("sx",(wd ~= 0 and 1/( wd    /(factor))) or 0)
620  commands.edef("sy",(wd ~= 0 and 1/((ht+dp)/(factor))) or 0)
621end
622\stoptyping
623
624So, in fact there are the following two processing alternatives:
625
626\startitemize[packed]
627\item tex: calls a \LUA\ function that processed the graphic
628\item lua: parse the \METAPOST\ code for texts and decide if two
629      runs are needed
630\stopitemize
631
632Now, if there was no text to be found, the continuation is:
633
634\startitemize[packed]
635\item lua: process the code using the library
636\item lua: convert the resulting graphic (if needed) and check
637      if texts are used
638\stopitemize
639
640Otherwise, the next steps are:
641
642\startitemize[packed]
643\item lua: process the code using the library
644\item lua: parse the resulting graphic for texts (in the postscripts)
645      and signal \TEX\ to process these texts afterwards
646\item tex: process the collected text and put the result in boxes
647\item lua: process the code again using the library but this time let
648      the unitsquare be transformed using the text dimensions
649\item lua: convert the resulting graphic and replace the transformed
650      unitsquare by the boxes with text
651\stopitemize
652
653The processor itself is used in the \MKIV\ graphic function that takes
654care of the multiple passes mentioned before. To give you an idea of
655how it works, here is how the main graphic processing function roughly
656looks.
657
658\starttyping
659local current_format, current_graphic
660
661function metapost.graphic_base_pass(mpsformat,str,preamble)
662    local prepared, done = metapost.check_texts(str)
663    metapost.textext_current = metapost.first_box
664    if done then
665        current_format, current_graphic = mpsformat, prepared
666        metapost.process(mpsformat, {
667            preamble or "",
668            "beginfig(1); ",
669            "_trial_run_ := true ;",
670            prepared,
671            "endfig ;"
672        }, true ) -- true means: trialrun
673        tex.sprint(tex.ctxcatcodes,
674            "\\ctxlua{metapost.graphic_extra_pass()}")
675    else
676        metapost.process(mpsformat, {
677            preamble or "",
678            "beginfig(1); ",
679            "_trial_run_ := false ;",
680            str,
681            "endfig ;"
682        } )
683    end
684end
685
686function metapost.graphic_extra_pass()
687    metapost.textext_current = metapost.first_box
688    metapost.process(current_format, {
689        "beginfig(0); ",
690        "_trial_run_ := false ;",
691        table.concat(metapost.texttextsdata()," ;\n"),
692        current_graphic,
693        "endfig ;"
694    })
695end
696\stoptyping
697
698The box information is generated as follows:
699
700\starttyping
701function metapost.texttextsdata()
702    local t, n = { }, 0
703    for i = metapost.first_box, metapost.last_box do
704        n = n + 1
705        local box_i = tex.box[i]
706        if box_i then
707            t[#t+1] = format(
708                "_tt_w_[%i]:=%f;_tt_h_[%i]:=%f;_tt_d_[%i]:=%f;",
709                n, box_i.width /factor,
710                n, box_i.height/factor,
711                n, box_i.depth /factor
712            )
713        else
714            break
715        end
716    end
717    return t
718end
719\stoptyping
720
721This is a typical example of accessing information available inside
722\TEX\ from \LUA, in this case information about boxes.
723
724The \type {trial_run} flag is used at the \METAPOST\ end, in fact the
725\type {textext} macro looks as follows:
726
727\starttyping
728vardef textext(expr str) =
729    if _trial_run_ :
730        % see first variant above
731    else :
732        % see second variant above
733    fi
734enddef ;
735\stoptyping
736
737This trickery is not new. We used it already in \CONTEXT\ for some
738time, but until now the multiple runs took way more time and from
739the perspective of the user this all looked much more complex.
740
741It may not be that obvious, but: in the case of a trial run (for
742instance when texts are found), after the first processing stage,
743and during the parsing of the result, the commands that typeset the
744content will be printed to \TEX. After processing, the command to do
745an extra pass is printed to \TEX\ also. So, once control is passed
746back to \TEX, at some point \TEX\ itself will pass control back to
747\LUA\ and do the extra pass.
748
749The base function is called in:
750
751\starttyping
752function metapost.graphic(mpsformat,str,preamble)
753    local mpx = metapost.format(mpsformat or "metafun")
754    metapost.graphic_base_pass(mpx,str,preamble)
755end
756\stoptyping
757
758The \type {metapost.format} function is part of \type {mlib-run}.
759It loads the \type{metafun} format, possibly after (re|)|generating it.
760
761Now, admittedly all this looks a bit messy, but in pure \TEX\ macros
762it would be even more so. Sometime in the future, the postponed calls to
763\tex{ctxlua} and the explicit \tex{pdfliteral}s can and will be
764replaced by using direct node generation, but that requires a rewrite
765of the internal \LUATEX\ support for \PDF\ literals.
766
767The snippets are part of the \type {mlib-*} files of \MKIV. These files are
768tagged as experimental and will stay that way for a while yet. This is
769proved by the fact that by now we use a slightly different approach.
770
771Summarizing the impact of \MPLIB\ on extensions, we can conclude that
772some are done better and some more or less the same. There are some
773conceptual problems that prohibit using pre- and postscripts for
774everything (at least currently).
775
776\subject{integrating}
777
778The largest impact of \MPLIB\ is processing graphics at runtime.
779In \MKII\ there are two methods: real runtime processing (each
780graphic triggered a call to \METAPOST) and collective processing
781(between \TEX\ runs). The first method slows down the \TEX\ run,
782the second method generates a whole lot of intermediate \POSTSCRIPT\
783files. In both cases there is a lot of file \IO\ involved.
784
785In \MKIV, the integrated library is capable of processing
786thousands of graphics per second, including conversion. The
787preliminary tests (which involved no extensions) involved graphics
788with 10 random circles drawn with penshapes in random colors, and
789the thoughput was around 2000 such graphics per second on a
7902.3 MHz Core Duo:
791
792\startlinecorrection[blank]
793\hbox to \textwidth{\strut\dorecurse{5}{\startMPcode
794numeric w ; w := \the\textwidth/6 ;
795for k:=1 upto 10 :
796    draw fullcircle
797        scaled uniformdeviate(w)
798        withpen pencircle xscaled (w/20) yscaled (w/40) rotated 30
799        withcolor (red/(k/4)) ;
800endfor ;
801setbounds currentpicture to fullsquare scaled w ;
802\stopMPcode\hss}\unskip}
803\stoplinecorrection
804
805In practice there will be some more overhead involved than in the
806tests. For instance, in \CONTEXT\ information about the current state
807of \TEX\ has to be passed on also: page dimensions, font information,
808typesetting related parameters, preamble code, etc.
809
810The whole \TEX\ interface is written around one process function:
811
812\starttyping
813metapost.graphic(metapost.format("metafun"),"mp code")
814\stoptyping
815
816optionally a preamble can be passed as the third argument.
817This one function is used in several other macros, like:
818
819\starttyping
820\startMPcode                  ... \stopMPcode
821\startMPpage                  ... \stopMPpage
822\startuseMPgraphic     {name} ... \stopuseMPgraphic
823\startreusableMPgraphic{name} ... \stopreusableMPgraphic
824\startuniqueMPgraphic  {name} ... \stopuniqueMPgraphic
825
826\useMPgraphic{name}
827\reuseMPgraphic{name}
828\uniqueMPgraphic{name}
829\stoptyping
830
831The user interface is downward compatible: in \MKIV\ the same
832top-level commands are provided as in \MKII. However, the
833(previously required) configuration macros and flags are obsolete.
834
835This time, the conclusion is that the impact on \CONTEXT\ is immense:
836The code for embedding graphics is very clean, and the running time
837for graphics inclusion is now negligible. Support for text in graphics is
838more natural now, and takes no runtime either (in \MKII\ some
839parsing in \TEX\ takes place, and if needed long lines are split;
840all this takes time).
841
842In the styles that \PRAGMA\ uses internally, there is support for the
843generation of placeholders for missing graphics. These placeholders
844are \METAPOST\ graphics that have some 60 randomly scaled circles with randomized
845colors. The time involved in generating 50 such graphics is (on Hans'
846machine) some 14 seconds, while in \LUATEX\ only half a second is needed.
847
848\startlinecorrection[blank]
849\hbox to \textwidth \bgroup
850\hss\dorecurse{4}{\externalfigure[dummy][width=.2\textwidth]\hss}%
851\egroup
852\stoplinecorrection
853
854Because \LUATEX\ needs more startup time and deals with larger fonts
855resources, \PDFTEX\ is generally faster, but now that we have \MPLIB,
856\LUATEX\ suddenly is the winner.
857
858% We end this article by showing a few graphics. \CONTEXT\ ships with a module
859% that permits tracking of resource usage. Users can add the following line
860% to their document:
861
862% \starttyping
863% \usemodule[timing]
864% \stoptyping
865
866% After that one can use commands like
867
868% \starttyping
869% \ShowNamedUsage{\jobname-luatex-progress}{luastate_bytes}{elapsed_time}
870% \ShowNamedUsage{\jobname-luatex-progress}{dyn_used}{elapsed_time}
871% \ShowNamedUsage{\jobname-luatex-progress}{str_ptr}{elapsed_time}
872% \ShowNamedUsage{\jobname-luatex-progress}{pdf_literal}{elapsed_time}
873% \ShowNamedUsage{\jobname-luatex-progress}{glyph}{elapsed_time}
874% \stoptyping
875
876% In this document, \LUA\ memory usage is as follows. The blue lines
877% represent runtime, on Hans' machine some 3.5 seconds, including
878% startuptime, which clearly shows in the graphic. Keep in mind that
879% garbage collection only happens at certain moment, so \LUA\ memory
880% usage is normally a sawtooth graphic. The horizontal axis reflects the
881% number of pages.
882
883% \startlinecorrection[blank]
884% \ShowNamedUsage{\jobname-luatex-progress}{luastate_bytes}{elapsed_time}
885% \stoplinecorrection
886
887% Dynamic memory usage of \TEX\ grows a bit because at the \TEX\ end we need
888% to store data as well.
889
890% \startlinecorrection[blank]
891% \ShowNamedUsage{\jobname-luatex-progress}{dyn_used}{elapsed_time}
892% \stoplinecorrection
893
894% Currently we use \type {\pdfliteral}, which means that all
895% literals go through \TEX's tokenizer and string pool. With large
896% graphics this can be a real memory hog.
897
898% Unfortunately not all this memory is freed, so eventually Hartmut
899% Henkel (the third member of the \LUATEX\ team) will reimplement
900% this part of the backend.
901
902% \startlinecorrection[blank]
903% \ShowNamedUsage{\jobname-luatex-progress}{str_ptr}{elapsed_time}
904% \stoplinecorrection
905
906% Just to complete this picture, we show the usage of literals next. Although
907% the converter collects as much data as possible before flushing, we see
908% some peaks.
909
910% \startlinecorrection[blank]
911% \ShowNamedUsage{\jobname-luatex-progress}{pdf_literal}{elapsed_time}
912% \stoplinecorrection
913
914% Compare this to the glyph usage. Indeed: the more graphics we use, the
915% less glyphs we encounter.
916
917% \startlinecorrection[blank]
918% \ShowNamedUsage{\jobname-luatex-progress}{glyph}{elapsed_time}
919% \stoplinecorrection
920
921% Of course these graphics were integrated, generated and converted
922% using \MPLIB.
923
924\stopcomponent
925