1
2
3\environment lowlevelstyle
4
5\startdocument
6 [title=lines,
7 coauthor=Mikael Sundqvist,
8 color=middleorange]
9
10\startsectionlevel[title=Introduction]
11
12There is no doubt that \TEX\ does an amazing job of \quotation {breaking
13paragraphs into lines} where a paragraph is a sequence of words in the input
14separated by spaces or its equivalents (single line endings turned space). The
15best descriptions of how that is done can be found in Don Knuths \quotation {The
16\TEX\ Book}, \quotation {\TEX\ The Program} and \quotation {Digital Typography}.
17Reading and rereading the relevant portions of those texts is a good exercise in
18humility.
19
20That said, whatever follows here builds upon what Knuth gave us and in no way we
21pretend to do better than that. It started out as a side track of improving
22rendering math in combination with more control over breaking inline math. It
23pretty much about having fun with the par builder but in the end can also help
24make your results look better. This is especially true for proze.
25
26Trying to describe the inner working of the par builder makes no sense. Not only
27is it kind of complex, riddled with magic constants and heuristics, but there is
28a good chance for us to talk nonsense thanks to misunderstanding. However, some
29curious aspects will be brought up. Consider what follows a somewhat naive
30approach and whatever goes wrong, blame the authors, not \TEX.
31
32If youre one of those reader who love to complain about the bad manuals, you can
33stop reading here. There is plenty said in the mentioned books but you can also
34consult Viktor Eijkhouts excellent \quotation {\TEX\ by Topic} (just search the
35web for how the get these books). If youre curious and in for some adventure,
36keep reading.
37
38\stopsectionlevel
39
40\startsectionlevel[title=Warning]
41
42This is a first version. What is described here will stay but is still
43experimental and how it evolves also depends on what demands we get from the
44users. We have defined some experimental setups in \CONTEXT. We wil try to
45improve the explanations in ways that (we hope) makes clear what happens deep
46down but that takes time. These might change depending on feedback. We assume
47that were in granular mode:
48
49\starttyping
50\setupalign[granular]
51\stoptyping
52
53We will explain below what that means, but let us already now make clear that
54this will likely become the default! As far as we can see, due to the larger
55solution space, the interword spacing is more even but that also means that some
56paragraphs can become one line less or more.
57
58\stopsectionlevel
59
60\startsectionlevel[title=Constructing paragraphs]
61
62There are several concepts at work when \TEX\ breaks a paragraph into lines. Here
63we assume that we talk about text: words separated by spaces. We also assume that
64the text starts at the left edge and nicely runs till the right edge, with the
65exception of the last line.
66
67\startitemize
68\startitem
69 The spaces between words can stretch or shrink. We dont want that to be too
70 inconsistent (visible) between two lines. This is where the terms loose and
71 tight come into play.
72\stopitem
73\startitem
74 Words can be hyphenated but we dont want that to happen too often. We also
75 discourage neighboring lines to have hyphens. Hyphenating the (pre) final line
76 is also sort of bad.
77\stopitem
78\startitem
79 We definitely dont want words to stick out in the margin. If we have to choose,
80 stretching is preferred over shrinking. If spaces become too small words,
81 start to blur.
82\stopitem
83\startitem
84 If needed glyphs can stretch or shrink a little in order to get rid of
85 excessive spacing. But we really want to keep it minimal, and avoid it when
86 possible. Usually we permit more stretch than shrink. Not all scripts (and
87 fonts for that matter) might work well with this feature.
88\stopitem
89\startitem
90 As a last resort we can stretch spaces so that we get rid of any still
91 sticking out word. When \TEX\ reports an overfull box (often a line) you have
92 to pay attention!
93\stopitem
94\stopitemize
95
96When \TEX\ decides where to break and when to finish doing so it uses a system of
97penalties and demerits and at some point makes decisions with regards to how bad
98a breakpoint (and eventually a paragraph) is. The penalties are normally
99relatively small unless we really want to penalize. When \TEX\ is in the process
100of breaking a paragraph it calculates badness values for each line. This can be
101seen as a measure on how bad looking a line is; a badness of zero is good, but
102the larger the badness becomes, the worse the line is.
103
104Here we shortly summarize the parameters that play a role in calculating what
105\TEX\ calls the costs of breaking a line at some point: its a combination of
106weighting penalties as well as over or undershooting the line with, where the
107amount (dimension) and kind of (fillers) stretch and shrink determien the final
108verdict.
109
110\startbuffer
111\ruledhbox to 20 ts{left \hss right}
112\ruledhbox to 40 ts{left \hss right}
113\ruledhbox to 5 ts{left \hss right}
114\ruledhbox to 5 ts{left right}
115\ruledhbox to 5 es{
116 left
117 \hskip 1ts plus 0.5ts\relax
118 middle
119 \hskip 1ts plus 1.5ts\relax
120 right
121}
122\stopbuffer
123
124\typebuffer
125
126These boxes show a bit what happens with spacing that can stretch of shrink. The
127first three cases are not bad because its what we ask for with the wildcard
128\type {\hss}. \footnote {We use this opportunity to promote the new \type {ts} and
129\type {es} units.}
130
131\startlines \getbuffer \stoplines
132
133\TEX\ will run over each paragraph at most three times. On each such run, it will
134choose different breakpoints, calculate badness of each possible line, combine
135that with eventual penalties, and calculate a certain demerit value for each
136possible paragraph. It creats a set of solutions as it progresses, discards the
137worse cases so far and eventually ends of what it thinks is best.
138
139The process is primarily controlled by these parameters:
140
141\startitemize
142\startitem
143 \type {\pretolerance}: This number determines the success of the first, not
144 hyphenated pass. Often the value is set to the plain \TEX\ value of 100.
145 If \TEX\ finds a possible division of a paragraph such that no line has
146 a badness higher than \type {\pretolerance}, the algorithm quits here and
147 that line is chosen.
148\stopitem
149\startitem
150 \type {\tolerance}: This number determines the success of the second,
151 hyphenated pass. Often the value is set to the plain \TEX\ value of 200.
152\stopitem
153\startitem
154 \type {\emergencystretch}: This dimension kicks in when the second pass is
155 not successful. In \CONTEXT\ we often set it to \type {2\bodyfontsize}.
156\stopitem
157\stopitemize
158
159When we are (in \CONTEXT\ speak) \type {tolerant}, we have a value of 3000, while
160\type {verytolerant} bumps it to 4500. These are pretty large values compared to
161the default 100 and 200 that seem to cover most cases well, especially when we
162have short words, a reasonable width and lots of opportunities for hyphenation.
163Keep in mind that a macro package has to default to values that make sense for
164the average case.
165
166We now come to the other relevant parameters. You need to keep in mind that the
167demerits are made from penalty values that get squared which is why parameters
168with demerits in their name have high values: a penalty of $50$ squared has to
169relate to a demerit of $5000$, so we might have $2500 5000$ at some point.
170
171The formula (most often) used to calculate the demerits \im{d} is
172
173\startformula
174 d = (l b p)2 e
175\stopformula
176
177Here \im {l} is the \type {\linepenalty}, set to \im {10} in plain, \im {b} is the
178badness of the line, and \im {p} is the penalty of the current break (for example,
179added by hyphenation, or by breaking an inline formula). The \im {e} stands for
180extra nonlocal demerits, that do not depend on only the current line, like
181the \type {\doublehyphendemerits} that is added if two lines in a row are
182hyphenated.
183
184The badness reflects how the natural linewidth relates to the target width and
185uses a cubic function. A badness of zero is of course optimal, but a badness of
18699 is pretty bad. A magic threshold is 12 (around that value a line is considered
187decent). If you look at the formula above you can now understand why the line
188penalty defaults to the low value of 10.
189
190\startitemize
191\startitem
192 \type {\hyphenpenalty}: When a breakpoint occurs at a discretionary this one
193 gets added. In \LUAMETATEX\ we store penalties in the discretionary nodes but
194 user defined \typ {\discretionary}s can carry dedicated penalties. This
195 value is set to 50, which is not that much. Large values reduce the solution
196 space so best keep this one reasonable.
197\stopitem
198\startitem
199 \type {\linepenalty}: Normally this is set to 10 and it is the baseline for a
200 breakpoint. This is again a small value compared to for instance the
201 penalties that you find in inline math. There we need some breakpoints and
202 after binary and relation symbols such an opportunity is created. The
203 specific penalties are normally 500 and 700. One has to keep in mind, as
204 shown in the formula above, that the penalties are not acting on a linear
205 scale when the demerits are calculated. Math spacing and penalty control is
206 discussed in the (upcoming) math manual.
207\stopitem
208\startitem
209 \type {\doublehyphendemerits}: Because it is considered bad to have two
210 hyphens in a row this is often set pretty high, many thousands. These are
211 treated as demerits (so outside of the squared part of the above formula).
212\stopitem
213\startitem
214 \type {\finalhyphendemerits}: The final (pre last) line having a hyphen is
215 also considered bad. The last line is handled differently anyway, just
216 because it gets normally flushed left.
217\stopitem
218\startitem
219 \type {\adjdemerits}: lines get rated in terms of being loose, decent, tight,
220 etc. When two lines have a different rating we bump the total demerits.
221\stopitem
222\startitem
223 \type {\looseness}: it is possible to force less or more lines but to what
224 extend this request is honored depends on for instance the possible
225 (emergency) stretch in the spaces (or any glue for that matter).
226
227\stopitem
228\stopitemize
229
230It is worth noticing that you can set \typ {\lastlinefit} such that the spaces in
231the last line will be comparable to those in the preceding line. This is a
232feature that \ETEX\ brought us. Anyways, keep in mind normally penalties are
233either small, or when we want to be tough, pretty high. Demerits are often
234relatively large.
235
236The next one is a flag that triggers expansion (or compression) of glyphs to kick
237in. Those get added to the available stretch andor shrink of a line:
238
239\startitemize[continue]
240\startitem
241 \type {\adjustspacing}: Its value determines if expansion kicks in: glyphs
242 basically get a stretch and shrink value, something that helps filling our
243 lines. We only have zero, two and three (and not the \PDFTEX\ value of two):
244 three means \quote {only glyphs} and two means \quote {font kerns and
245 glyphs}.
246\stopitem
247\stopitemize
248
249In \LUAMETATEX\ we also have:
250
251\startitemize
252\startitem
253 \typ {\linebreakcriterion}: The normal distinction between loose, decent and
254 tight in \TEX\ uses 12 for 0.5 and 99 for about 1.0, but because we have more
255 granularity (.25) we can set four values instead. The default of zero (\type
256 {"0C0C0C63}) then becomes \type {"020C2A63}. When set that way the default
257 \typ {\adjdemerits} has to be halved 5000 so that we compare the more
258 granular distances. Dont worry if you \quote {dont get it}, hardly any user
259 will change these values. One can think of the 100 squared becomes a 10000
260 (at least this helps relating these numbers) and 10000 is pretty bad in \TEX s
261 perception.
262\stopitem
263\startitem
264 \type {\adjustspacingstep}: When set this one is are used instead of the font
265 bound value which permits local control without defining a new font instance.
266\stopitem
267\startitem
268 \type {\adjustspacingstretch}: idem.
269\stopitem
270\startitem
271 \type {\adjustspacingshrink}: idem.
272\stopitem
273
274
275
276\startitem
277 \type {\orphanpenalty}: This penalty will be injected before the last word of a paragraph.
278\stopitem
279\startitem
280 \type {\orphanpenalties}: Alternatively a series of penalties can be defined.
281 This primitive expects a count followed by that number of penalties. These
282 will be injected starting from the end.
283\stopitem
284\stopitemize
285
286The shape of a paragraph is determined by \typ {\hangindent}, \type {\hangafter},
287\typ {\parshape} and \typ {\parindent}. The width is controlled by \typ {\hsize},
288\typ {\leftskip}, \typ {\rightskip}. In addition there are \typ
289{\parinitleftskip}, \typ {\parinitrightskip}, \typ {\parfillleftskip} and \typ
290{\parfillrightskip} that control first and last lines.
291
292We also have these:
293
294\startitemize
295\startitem
296 \type {\linebreakpasses}: When set to one, the currently set \type {\parpasses}
297 will be applied.
298\stopitem
299\startitem
300 \type {\parpasses}: This primitive defined a set of sub passes that kick in
301 when the second pass is finished. This basically opens up the par builder. It
302 is still experimental and will be improved based upon user feedback. Although
303 it is a side effect of improving the breaking of extensive mixes of math and
304 text, it is also quite useful for text only (think novels).
305\stopitem
306\stopitemize
307
308In the next sections we will explain how these can improve the look and feel of
309what you typeset.
310
311\stopsectionlevel
312
313\startsectionlevel[title=Subpasses]
314
315In \LUATEX\ and therefore also in \LUAMETATEX\ a paragraph is constructed in steps:
316
317\startitemize
318\startitem
319 The list of nodes that makes the paragraph is hyphenated: words become a
320 mixture of glyphs and discretionaries.
321\stopitem
322\startitem
323 That list is processed by a font handler that can remove, add or change glyphs
324 depending on how glyphs interact. This depends on the language and scripts used.
325\stopitem
326\startitem
327 The result is fed into the parbuilder that applies upto three passes as mentioned
328 before.
329\stopitem
330\stopitemize
331
332In traditional \TEX\ these three actions are combined into one and the overhead
333is shared. In the split case the processing time gets distributed and in practice
334the last action is not the one that takes most time. This is why the mechanism
335that we discuss next has little impact on the run: calling the par builder a few
336times more seldom results in more runtime. This is why in we support so called
337sub passes between the second and third one.
338
339Here is an example of a setup. We set a low tolerance for the first pass and second
340pass. We can do that because we dont need to play safe nor need to compromise.
341
342\starttyping
343\pretolerance 75
344\tolerance 150
345\parpasses 3
346 threshold 0.025pt
347 classes \indecentparpassclasses
348 tolerance 150
349 next
350 threshold 0.025pt
351 classes \indecentparpassclasses
352 tolerance 200
353 emergencystretch .25\bodyfontsize
354 next
355 threshold 0.025pt
356 classes \indecentparpassclasses
357 tolerance 200
358 optional 1
359 emergencystretch .5\bodyfontsize
360\relax
361\linebreakpasses 1
362\stoptyping
363
364Because we want to retain performance we need to test efficiently if we really
365need the (here upto three) additional passes, so lets see how it is done. When a
366pass list is defined, and line break passes are enabled, the engine will check {\em after}
367the second pass if some more work is needed. For that it will do a quick analysis and
368calculate four values:
369
370\startitemize[packed]
371\startitem overflow : the maximum value found, this is something really bad. \stopitem
372\startitem underflow : the maximum value found, this is something we can live with. \stopitem
373\startitem verdict : what is the worst badness of lines in this paragraph. \stopitem
374\startitem classified : what classes are assigned to lines, think looseness, decent and tight. \stopitem
375\stopitemize
376
377There are two cases where the engine will continue with the applying passes:
378there is an overflow or there is a verdict (max badness) larger than zero. When we
379tested this on some large documents we noticed that this is nearly always true,
380but by checking we save a few unnecessary passes.
381
382Next we test if a pass is really needed, and if not we check the next pass. When
383a pass is done, we pick up where we left, but we test for the overflow or badness
384every sub pass. The next checks make us run a pass:
385
386\startitemize[packed]
387\startitem overfull exceeds threshold \stopitem
388\startitem verdict exceeds badness \stopitem
389\startitem classified overlaps classes \stopitem
390\stopitemize
391
392Here \typ {threshold}, \typ {badness} and \typ {classes} are options in a pass
393section. Which test makes sense depends a bit on how \TEX\ sees the result.
394Internally \TEX\ uses numbers for its classification (0..5) but we map that onto
395a bitset because we want an overview:
396
397\starttabulate[rlcccc]
398\NC \NC \BC indecent \BC almostdecent \BC loose \BC tight \NC \NR
399\NC 1 \BC veryloose \NC \NC \NC \NC \NC \NR
400\NC 2 \BC loose \NC \NC \NC \NC \NC \NR
401\NC 4 \BC semiloose \NC \NC \NC \NC \NC \NR
402\NC 8 \BC decent \NC \NC \NC \NC \NC \NR
403\NC 16 \BC semitight \NC \NC \NC \NC \NC \NR
404\NC 32 \BC tight \NC \NC \NC \NC \NC \NR
405\stoptabulate
406
407The semiloose and semitight values are something \LUAMETATEX. In \CONTEXT\ we
408have these four variants predefined as \typ {\indecentparpassclasses} and such.
409
410The sections in a par pass setup are separated by \type {next}. For testing
411purposes you can add \type {skip} and \type {quit}. The \type {threshold} tests
412against the overfull value, the \type {badness} against the verdict and \type
413{classes} checks for overlap with encountered classes, the classification.
414
415You can specify an \typ {identifier} in the first segment that then will be used
416in tracing but it is also passed to callbacks that relate to this feature.
417Discussing these callback is outside the scope fo this wrapup.
418
419You need to keep in mind that parameters are not reset to their original values
420between two subpasses of a paragraph.
421We have \typ {tolerance} and \typ {emergencystretch} which are handy for simple
422setups. When we start with a small tolerance we often need to bump that one. The
423stretch is likely a last resort. The usual demerits can be set too: \typ
424{doublehyphendemerits}, \typ {finalhyphendemerits} and \typ {adjdemerits}. We
425have \typ {extrahyphenpenalty} that gets added to the penalty in a discretionary.
426You can also set \typ {linepenalty} to a different value than it normally gets.
427
428The \typ {looseness} can be set but keep in mind that this only makes sense in
429very special cases. Its hard to be loose when there is not much stretch or shrink
430available. The \typ {linebreakcriterion} parameter can best be left untouched and is
431mostly there for testing purposes.
432
433The \LUAMETATEX\ specific \typ {orphanpenalty} gets injected before the last word
434in a paragraph. High values can lead to overfull boxes but when used in text that
435hyphenate well or with languages that have short words it might work out well.
436
437The next four parameters are related to expansion: \typ {adjustspacing}, \typ
438{adjustspacingstep}, \typ {adjustspacingshrink} and \typ {adjustspacingstretch}.
439Here we have several scenarios.
440
441\startitemize
442\startitem
443 Fonts are set up for expansion (in \CONTEXT\ for instance with the quality
444 specifier). When \type {hz} is then enabled it will always kick in.
445\stopitem
446\startitem
447 When we dont enable it, the par pass can do it by setting \typ {adjustspacing} (to 3).
448\stopitem
449\startitem
450 When the other parameters are set these will overload the ones in the font,
451 but used with the factors in there, so different characters get scaled
452 differently. You can set the step to one to get more granular results.
453\stopitem
454\startitem
455 When expansion is {\em not} set on the font, setting the options in a pass will activate
456 expansion but with the factors set to 1000. This means all characters are treated equal,
457 which is less subtle.
458\stopitem
459\stopitemize
460
461When a font is not set up to use expansion, you can do something like this:
462
463\starttyping
464\parpasses 6
465 classes \indecentparpassclasses
466 threshold 0.025pt
467 tolerance 250
468 extrahyphenpenalty 50
469 orphanpenalty 5000
470
471 next ifadjustspacing
472 threshold 0.025pt
473 classes \tightparpassclasses
474 tolerance 300
475 adjustspacing 3
476 orphanpenalty 5000
477 next ifadjustspacing
478 threshold 0.025pt
479 tolerance 350
480 adjustspacing 3
481 adjustspacingstep 1
482 adjustspacingshrink 20
483 adjustspacingstretch 40
484 orphanpenalty 5000
485 emergencystretch .25\bodyfontsize
486
487 next
488 threshold 0.025pt
489 classes \tightparpassclasses
490 tolerance 300
491 adjustspacing 3
492 adjustspacingstep 1
493 adjustspacingshrink 10
494 adjustspacingstretch 15
495 orphanpenalty 5000
496 next
497 threshold 0.025pt
498 tolerance 350
499 adjustspacing 3
500 adjustspacingstep 1
501 adjustspacingshrink 20
502 adjustspacingstretch 40
503 orphanpenalty 5000
504 emergencystretch .25\bodyfontsize
505
506 next
507 threshold 0.025pt
508 tolerance 3000
509 orphanpenalty 5000
510 emergencystretch .25\bodyfontsize
511\relax
512\stoptyping
513
514With \typ {ifadjustspacing} you ignore steps that expect the font to be setup, so
515you dont waste time if that is not the case.
516
517There is also a \typ {callback} parameter but that one is experimental and used
518for special purposes and testing. We dont expect users to mess with that.
519
520A really special feature is optional content. Here we use as example a quote from
521Digital Typography:
522
523\starttyping[paragraph=yes,align=flushleft]
524Many readers will skim over formulas on their first reading
525of your exposition. Therefore, your sentences should flow
526smoothly when all but the simplest formulas are replaced by
527\quotation {blah} or some other \optionalword {1} {grunting }noise.
528\stoptyping
529
530Here the \type {grunting} (with embedded space) is considered optional. When you
531set \typ {\linebreakoptional} to1 this word will be typeset. However, when you
532set the pass parameter \typ {linebreakoptional} to0 it will be skipped. There
533can be multiple optional words with different numbers. The numbers are actually
534bits in a bit set so plenty is possible. However, normally these two values are
535enough, if used at all.
536
537\stopsectionlevel
538
539\startsectionlevel[title=Definitions]
540
541The description above is rather low level and in practice users will use a bit
542higher level interface. Also, in practice only a subset of the parameters makes
543sense in general usage. It is not that easy to decide on what parameter subset
544will work out well but it can be fun to play with variants. After all, this is
545also what \TEX\ is about: look, feel and fun.
546
547Some users praise the ability of recent \TEX\ engines to provide expansion and
548protrusion. This feature is a bit demanding because not only does it add to
549runtime (although in \LUAMETATEX\ that normally can be neglected), it also makes
550the output files larger. Some find it hard to admit, but it even can result in
551bad looking documents when applied with extremes.
552
553The traditional (\MKIV) way to set up expansion is to add this to the top of the
554document, or at least before fonts get loaded.
555
556\startbuffer[passa]
557\definefontfeature
558 [default]
559 [default]
560 [expansion=quality,protrusion=quality]
561\stopbuffer
562
563\typebuffer[a]
564
565and later on to enable it with:
566
567\starttyping
568\setupalign[hz]
569\stoptyping
570
571However, par passes make it possible to be more selective. Take the following two
572definitions:
573
574\startbuffer[passb]
575\startsetups align:pass:quality:1
576 \pretolerance 50
577 \tolerance 150
578 \parpasses 6
579 identifier \parpassidentifier{quality:1}
580 threshold 0.025pt
581 tolerance 175
582 next
583 threshold 0.025pt
584 tolerance 200
585 next
586 threshold 0.025pt
587 tolerance 250
588 next
589 classes \almostdecentparpassclasses
590 tolerance 300
591 emergencystretch .25\bodyfontsize
592 next ifadjustspacing
593 classes \indecentparpassclasses
594 tolerance 300
595 adjustspacing 3
596 emergencystretch .25\bodyfontsize
597 next
598 threshold 0.025pt
599 tolerance 3000
600 emergencystretch 2\bodyfontsize
601 \relax
602\stopsetups
603
604\startsetups align:pass:quality:2
605 \pretolerance 50
606 \tolerance 150
607 \parpasses 5
608 identifier \parpassidentifier{quality:2}
609 threshold 0.025pt
610 tolerance 175
611 next
612 threshold 0.025pt
613 tolerance 200
614 next
615 threshold 0.025pt
616 tolerance 250
617 next ifadjustspacing
618 classes \indecentparpassclasses
619 tolerance 300
620 adjustspacing 3
621 emergencystretch .25\bodyfontsize
622 next
623 threshold 0.025pt
624 tolerance 3000
625 emergencystretch 2\bodyfontsize
626 \relax
627\stopsetups
628\stopbuffer
629
630\typebuffer[passb]
631
632You can now enable one of these:
633
634\starttyping
635\setupalignpass[quality:1]
636\stoptyping
637
638\startbuffer[passc]
639\starttext
640 \showmakeup[expansion,space]
641 \startTEXpage[offset=1ts]
642 \startcombination[3*3]
643 {\vtop{\hsize 8cm\setupalignpass[none] \samplefile{tufte}}} {none}
644 {\vtop{\hsize 9cm\setupalignpass[none] \samplefile{tufte}}} {none}
645 {\vtop{\hsize12cm\setupalignpass[none] \samplefile{tufte}}} {none}
646 {\vtop{\hsize 8cm\setupalignpass[quality:1]\samplefile{tufte}}} {quality:1}
647 {\vtop{\hsize 9cm\setupalignpass[quality:1]\samplefile{tufte}}} {quality:1}
648 {\vtop{\hsize12cm\setupalignpass[quality:1]\samplefile{tufte}}} {quality:1}
649 {\vtop{\hsize 8cm\setupalignpass[quality:2]\samplefile{tufte}}} {quality:2}
650 {\vtop{\hsize 9cm\setupalignpass[quality:2]\samplefile{tufte}}} {quality:2}
651 {\vtop{\hsize12cm\setupalignpass[quality:2]\samplefile{tufte}}} {quality:2}
652 \stopcombination
653 \stopTEXpage
654\stoptext
655\stopbuffer
656
657The result is shown in \in {figure} [fig:passes:expansion] where you can see that
658expansion is applied selectively; you have to zoom in to see where.
659
660\startplacefigure[location=page,reference=fig:passes:expansion,title={Two different passes applied to \type {tufte.tex}.}]
661 \typesetbuffer[passa,passb,passc][width=\textwidth]
662\stopplacefigure
663
664\stopsectionlevel
665
666\startsectionlevel[title=Tracing]
667
668There are several ways to see what goes on. The engine has a tracing option that
669is set with \type {\tracingpasses}. Setting it to \type {1} reports the passes on
670the console, and a value of \type {2} also gives some details.
671
672There is a also a tracker, \type {paragraphs.passes} that can be enabled. This gives
673a bit more information:
674
675\starttyping
676\enabletrackers[paragraphs.passes]
677\enabletrackers[paragraphs.passes=summary]
678\enabletrackers[paragraphs.passes=details]
679\stoptyping
680
681If you want to see where expansion kicks in, you can use:
682
683\starttyping
684\showmakeup[expansion]
685\stoptyping
686
687This is just one of the options, \type {spaces}, \type {penalties}, \type {glue}
688are are useful when you play with passes, but if you are really into the low level
689details, this is what you want:
690
691\startbuffer
692\startnarrower[5*right]
693\startshowbreakpoints[option=margin,offset=\dimexpr{.5\emwidth\rightskip}]
694\samplefile{tufte}
695\stopshowbreakpoints
696\stopnarrower
697\stopbuffer
698
699\typebuffer \getbuffer
700
701You can see the chosen solutions with
702
703\startbuffer
704\showbreakpoints[n=1]
705\stopbuffer
706
707\typebuffer \getbuffer
708
709When we started playing with the par builder in the perspective of
710math, we side tracked and ended up with a feature that can ge used
711in controlled situations. Currently we only have a low level
712\CONTEXT\ interface for this (see \in {figure} [fig:passes:lousiness]).
713
714\startbuffer[lousiness]
715 \startTEXpage[offset=1ts]
716 \startcombination[3*1]
717 \bgroup \vtop\bgroup
718 \hsize8cm
719 \setupalign[verytolerant]
720 \tracinglousiness 1
721 \lousiness 0
722 \samplefile{ward}
723 \egroup \egroup
724 {\type {\tracinglousiness 1}
725 \type {\lousiness 0}}
726 \bgroup \vtop\bgroup
727 \hsize8cm
728 \setupalign[verytolerant]
729 \lousiness 1 11 0
730 \samplefile{ward}
731 \egroup \egroup
732 {\type {\lousiness 1 11 0}}
733 \bgroup \vtop\bgroup
734 \hsize8cm
735 \setupalign[verytolerant]
736 \silliness 11
737 \samplefile{ward}
738 \egroup \egroup
739 {\type {\silliness 11}}
740 \stopcombination
741\stopTEXpage
742\stopbuffer
743
744\startplacefigure[location=here,reference=fig:passes:lousiness,title={Influencing the way \TEX\ breaks lines applied to \type {ward.tex}.}]
745 \typesetbuffer[lousiness][width=\textwidth]
746\stopplacefigure
747
748\stopsectionlevel
749
750\startsectionlevel[title=Criterion]
751
752The \type {granular} alignment option will configure the linebreakcriterion to
753work with $0.25$ steps instead of $0.50$ steps which means that successive lines
754can become a bit closer in spacing. There is no real impact on performance
755because testing happens anyway. In \in {figure}[fig:criterion] you see some
756examples, where in some it indeed makes a difference.
757
758\startbuffer[criterion]
759\starttext
760\definecolor[ttest][a=1,t=.5]
761\definecolor[rtest][a=1,t=.5,r=1]
762\dostepwiserecurse{80}{120}{1}{
763 \startTEXpage[offset=1ts]
764 \startcombination[3*1]
765 {\startoverlay
766 {\vtop{\showmakeup[space]\hsize #1mm\ttest \samplefile{tufte}}}
767 {\vtop{\showmakeup[space]\hsize #1mm\rtest \setupalign[granular]\samplefile{tufte}}}
768 \stopoverlay} {}
769 {\vtop{\showmakeup[space]\hsize #1mm\ttest \samplefile{tufte}}} {}
770 {\vtop{\showmakeup[space]\hsize #1mm\rtest \setupalign[granular]\samplefile{tufte}}} {}
771 \stopcombination
772 \stopTEXpage
773}
774\stoptext
775\stopbuffer
776
777\startplacefigure[location=here,reference=fig:criterion,title={More granular interline criteria.}]
778 \startcombination[1*4]
779 {\typesetbuffer[criterion][width=\textwidth,page=35]} {}
780 {\typesetbuffer[criterion][width=\textwidth,page=36]} {}
781 {\typesetbuffer[criterion][width=\textwidth,page=37]} {}
782 {\typesetbuffer[criterion][width=\textwidth,page=38]} {}
783 \stopcombination
784\stopplacefigure
785
786\stopsectionlevel
787
788\startsectionlevel[title=Examples]
789
790\start \em
791
792 The \CONTEXT\ distribution comes with a few test setups: \typ
793 {spacimptests.mkxl}. Once we have found a suitable set of values and sample
794 texts we might discuss them here.
795
796 Currently we provide the following predefined passes that you can enable with
797 \typ {\setupalignpass}: \type {decent}, \type {quality}, \type {test1}, \type
798 {test2}, \type {test3}, \type {test4}, \type {test5}. We hope that users are
799 willing to test these.
800
801\stop
802
803\stopsectionlevel
804
805\startsectionlevel[title=Pages]
806
807While the par builder does multiple passes, the page builder is a single pass
808progressive routine. Every time something gets added to the (so called) main
809vertical list the page state gets updated and when the page overflows what has
810been collected gets passed to the output routine. It is to a large extend driven
811by glue (with stretch and shrink) and penalties and when content (boxes) is added
812the process is somewhat complicated by inserts as these needs to be taken into
813account too.
814
815You can get pages that run from top to bottom by adding stretch between lines but
816by default in \CONTEXT\ we prefer to fill up the bottom with white space.
817
818It can be hard to make decisions at the \TEX\ end around a potential page break
819because in order to get an idea how much space is left, one needs to trigger the
820page builder which can have side effects.
821
822Penalties play an important role and because these are used to control for
823instance widows and clubs high values can lead to underfull pages so if we want
824to influence that we need to cheat. For this we have three experimental
825mechanisms:
826
827\startitemize[packed]
828\startitem tweaking the page goal: \type {\pageextragoal} \stopitem
829\startitem initializing the state quantities: \type {\initialpageskip} \stopitem
830\startitem adapting the state quantities as we go: \type {\additionalpageskip} \stopitem
831\stopitemize
832
833The first tweak is for me to play with, and when a widow or club is seen the
834extra amount can kick in. This feature is likely to be replaced by a more
835configurable one.
836
837The second tweak lets the empty page start out with some given height, stretch
838and shrink. This variable is persistent over pages. This is not true for the
839third tweak: it kicks in when the page gets initialized {\em or} as we go, but
840after it has been applied the value is reset. That makes it a feature like \type
841{\looseness}. We could combine these into one (because one can set up a
842persistent one in the macro package at well defined spots) but having an initial
843one also nicely can compensate the usual topskip glue hackery with a more natural
844control option.
845
846\startbuffer[pagelooseness1]
847\starttext
848 \showframe[text]
849 \setuplayout[width=middle,headerdistance=5mm]
850 \setupalign[vertical,height]
851 \dorecurse{10}{
852 \samplefile{tufte}\par
853 \setpagelooseness[lines=2]
854 \dorecurse{5}{
855 {\red \samplefile{knuth}}\par
856 {\green \samplefile {ward}}\par
857 {\blue \samplefile{davis}}\par
858 }
859 \page
860 }
861\stoptext
862\stopbuffer
863
864\startbuffer[pagelooseness2]
865\starttext
866 \showframe[text]
867 \setuplayout[width=middle,headerdistance=5mm]
868 \setupalign[vertical,height]
869 \dorecurse{10}{
870 \samplefile{tufte}\par
871 \setpagelooseness[3]
872 \dorecurse{5}{
873 {\red \samplefile{knuth}}\par
874 {\green \samplefile {ward}}\par
875 {\blue \samplefile{davis}}\par
876 }
877 \page
878 }
879\stoptext
880\stopbuffer
881
882Adapting the layout (within the regular text area) is done with \typ
883{\setpagelooseness} an demonstrated in \in {figure} [fig:pagelooseness1] and \in
884{figure} [fig:pagelooseness2]. Possible parameters are \type {lines}, \type
885{height}, \type {stretch} and \type {shrink}. You can also directly specify the
886number of lines. The other two features are not (yet) interfaced.
887
888\startplacefigure[location=here,reference=fig:pagelooseness1,title={Cheating with page dimensions: \type {[lines=2]}.}]
889 \startcombination[4*1]
890 {\typesetbuffer[pagelooseness1][width=\combinationwidth,page=1,frame=on]} {}
891 {\typesetbuffer[pagelooseness1][width=\combinationwidth,page=2,frame=on]} {}
892 {\typesetbuffer[pagelooseness1][width=\combinationwidth,page=3,frame=on]} {}
893 {\typesetbuffer[pagelooseness1][width=\combinationwidth,page=4,frame=on]} {}
894 \stopcombination
895\stopplacefigure
896
897\startplacefigure[location=here,reference=fig:pagelooseness2,title={Cheating with page dimensions: \type {[3]}.}]
898 \startcombination[4*1]
899 {\typesetbuffer[pagelooseness2][width=\combinationwidth,page=1,frame=on]} {}
900 {\typesetbuffer[pagelooseness2][width=\combinationwidth,page=2,frame=on]} {}
901 {\typesetbuffer[pagelooseness2][width=\combinationwidth,page=3,frame=on]} {}
902 {\typesetbuffer[pagelooseness2][width=\combinationwidth,page=4,frame=on]} {}
903 \stopcombination
904\stopplacefigure
905
906It is not that trivial to fulfill the wide range of user demands but over time
907the \typ {\setupalign} commands has gotten plenty of features. Getting for
908instance windows and clubs right in the kind of mixed usage that is common in
909\CONTEXT\ is not always easy. One can experiment with scenarios (also to get some
910understanding of matters) but none is probably perfect (unless one does something
911close to manual tweaking). There is also the butterfly effect: a change here
912might trigger na issue there.
913
914The examples in \in {figure} [fig:vz1], \in [fig:vz2] and \in [fig:vz3] scale
915vertically in order ti fill up the text area; the \type {vz} parameter is set
916with \typ {setuplayout}. In the example the widow and club penalties are set to
917$10000$. In these examples we have enabled the \typ {layout.vz} trackers that
918shows a small black rule indicating the amount of stretch.
919
920\startbuffer[vz1]
921\starttext
922 \showframe[text]
923 \enabletrackers[layout.vz]
924 \setuplayout[width=middle,headerdistance=5mm,vz=no]
925 \clubpenalty 10000
926 \widowpenalty 10000
927 \dostepwiserecurse{0}{30}{1}{
928 \dorecurse{#1}{\strut dummy line ##1\par}
929 \dorecurse{4}{\samplefile{tufte}\par}
930 }
931\stoptext
932\stopbuffer
933
934\startbuffer[vz2]
935\starttext
936 \showframe[text]
937 \enabletrackers[layout.vz]
938 \setuplayout[width=middle,headerdistance=5mm,vz=yes]
939 \clubpenalty 10000
940 \widowpenalty 10000
941 \dostepwiserecurse{0}{30}{1}{
942 \dorecurse{#1}{\strut dummy line ##1\par}
943 \dorecurse{4}{\samplefile{tufte}\par}
944 }
945\stoptext
946\stopbuffer
947
948\startbuffer[vz3]
949\starttext
950 \showframe[text]
951 \enabletrackers[layout.vz]
952 \setuplayout[width=middle,headerdistance=5mm,vz=2]
953 \clubpenalty 10000
954 \widowpenalty 10000
955 \dostepwiserecurse{0}{30}{1}{
956 \dorecurse{#1}{\strut dummy line ##1\par}
957 \dorecurse{4}{\samplefile{tufte}\par}
958 }
959\stoptext
960\stopbuffer
961
962\startplacefigure[location=here,reference=fig:vz1,title={Cheating with vertical expansion: \type {[vz=no]}.}]
963 \startcombination[4*1]
964 {\typesetbuffer[vz1][width=\combinationwidth,page=1,frame=on]} {}
965 {\typesetbuffer[vz1][width=\combinationwidth,page=2,frame=on]} {}
966 {\typesetbuffer[vz1][width=\combinationwidth,page=3,frame=on]} {}
967 {\typesetbuffer[vz1][width=\combinationwidth,page=4,frame=on]} {}
968 \stopcombination
969\stopplacefigure
970
971\startplacefigure[location=here,reference=fig:vz2,title={Cheating with vertical expansion: \type {[vz=yes]}.}]
972 \startcombination[4*1]
973 {\typesetbuffer[vz2][width=\combinationwidth,page=1,frame=on]} {}
974 {\typesetbuffer[vz2][width=\combinationwidth,page=2,frame=on]} {}
975 {\typesetbuffer[vz2][width=\combinationwidth,page=3,frame=on]} {}
976 {\typesetbuffer[vz2][width=\combinationwidth,page=4,frame=on]} {}
977 \stopcombination
978\stopplacefigure
979
980\startplacefigure[location=here,reference=fig:vz3,title={Cheating with vertical expansion: \type {[vz=2]}.}]
981 \startcombination[4*1]
982 {\typesetbuffer[vz3][width=\combinationwidth,page=1,frame=on]} {}
983 {\typesetbuffer[vz3][width=\combinationwidth,page=2,frame=on]} {}
984 {\typesetbuffer[vz3][width=\combinationwidth,page=3,frame=on]} {}
985 {\typesetbuffer[vz3][width=\combinationwidth,page=4,frame=on]} {}
986 \stopcombination
987\stopplacefigure
988
989There are a few other tweaks but these one can wonder about these. We can add
990stretch and shrink to the baseline skip, something that can also be triggered
991with the \quote {spread} option to \typ {\setupalign}, assuming that also \typ
992{height} is given). An alternative is to permit an extra line and accept a visual
993overflow, assuming that the layout is set up to make sure that the footer line
994doesnt overlap. None of this guarantees that a whole document with plenty of
995graphics and special constructs will come out well, but for text only it might
996work okay. \in {Figures} [fig:extra1], \in [fig:extra2] and \in [fig:extra3]
997show some of this.
998
999\startbuffer[extra1]
1000\starttext
1001 \showframe[text]
1002 \setuplayout[width=middle,headerdistance=15mm,vz=no]
1003 \setupalign[height]
1004 \clubpenalty 10000
1005 \widowpenalty 10000
1006 \dorecurse{10}{
1007 \samplefile{tufte}\par
1008 \samplefile{knuth}\par
1009 \samplefile{ward}\par
1010 \samplefile{davis}\par
1011 }
1012\stoptext
1013\stopbuffer
1014
1015\startbuffer[extra2]
1016\starttext
1017 \showframe[text]
1018 \setuplayout[width=middle,headerdistance=15mm,vz=no]
1019 \setupalign[height]
1020 \clubpenalty 10000
1021 \widowpenalty 10000
1022 \baselineskip 1\baselineskip plus 1pt minus .1pt
1023 \dorecurse{10}{
1024 \samplefile{tufte}\par
1025 \samplefile{knuth}\par
1026 \samplefile{ward}\par
1027 \samplefile{davis}\par
1028 }
1029\stoptext
1030\stopbuffer
1031
1032\startbuffer[extra3]
1033\starttext
1034 \showframe[text]
1035 \setuplayout[width=middle,headerdistance=15mm,vz=no]
1036 \setupalign[height]
1037 \clubpenalty 10000
1038 \widowpenalty 10000
1039 \pageextragoal1\lineheight
1040 \dorecurse{10}{
1041 \samplefile{tufte}\par
1042 \samplefile{knuth}\par
1043 \samplefile{ward}\par
1044 \samplefile{davis}\par
1045 }
1046\stoptext
1047\stopbuffer
1048
1049\page
1050
1051\startplacefigure[location=here,reference=fig:extra1,title={Cheating: just high penalties.}]
1052 \startcombination[4*2]
1053 {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra1][width=\combinationwidth,page=1,frame=on]}} {}
1054 {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra1][width=\combinationwidth,page=2,frame=on]}} {}
1055 {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra1][width=\combinationwidth,page=3,frame=on]}} {}
1056 {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra1][width=\combinationwidth,page=4,frame=on]}} {}
1057 {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra1][width=\combinationwidth,page=5,frame=on]}} {}
1058 {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra1][width=\combinationwidth,page=6,frame=on]}} {}
1059 {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra1][width=\combinationwidth,page=7,frame=on]}} {}
1060 {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra1][width=\combinationwidth,page=8,frame=on]}} {}
1061 \stopcombination
1062\stopplacefigure
1063
1064\startplacefigure[location=here,reference=fig:extra2,title={Cheating: \typ {\baselineskip 1\baselineskip plus 1pt minus .1pt}.}]
1065 \startcombination[4*2]
1066 {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra2][width=\combinationwidth,page=1,frame=on]}} {}
1067 {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra2][width=\combinationwidth,page=2,frame=on]}} {}
1068 {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra2][width=\combinationwidth,page=3,frame=on]}} {}
1069 {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra2][width=\combinationwidth,page=4,frame=on]}} {}
1070 {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra2][width=\combinationwidth,page=5,frame=on]}} {}
1071 {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra2][width=\combinationwidth,page=6,frame=on]}} {}
1072 {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra2][width=\combinationwidth,page=7,frame=on]}} {}
1073 {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra2][width=\combinationwidth,page=8,frame=on]}} {}
1074 \stopcombination
1075\stopplacefigure
1076
1077\startplacefigure[location=here,reference=fig:extra3,title={Cheating: \typ {\pageextragoal \lineheight}.}]
1078 \startcombination[4*2]
1079 {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra3][width=\combinationwidth,page=1,frame=on]}} {}
1080 {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra3][width=\combinationwidth,page=2,frame=on]}} {}
1081 {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra3][width=\combinationwidth,page=3,frame=on]}} {}
1082 {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra3][width=\combinationwidth,page=4,frame=on]}} {}
1083 {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra3][width=\combinationwidth,page=5,frame=on]}} {}
1084 {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra3][width=\combinationwidth,page=6,frame=on]}} {}
1085 {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra3][width=\combinationwidth,page=7,frame=on]}} {}
1086 {\clip[ny=12,sy=1,y=10]{\typesetbuffer[extra3][width=\combinationwidth,page=8,frame=on]}} {}
1087 \stopcombination
1088\stopplacefigure
1089
1090\stopsectionlevel
1091
1092\startsectionlevel[title=Profiles]
1093
1094You can have a paragraph with lines that exceed the maximum height andor depth
1095or where spaces end up in a way that create so called rivers. Rivers are more a
1096curiosity than an annoyance because any attempt to avoid them is likely to result
1097in a worse looking result. The unequal line distances can be annoying too but
1098these can be avoided when bringing lines closer together doesnt lead to clashes.
1099This can be done without reformatting the paragraph by passing the \type
1100{profile} option to \typ {\setupalign}. It comes at the cost of a little more
1101runtime and (as far as we observed) it kicks in seldom, for instance when inline
1102math is used that has super or subscripts, radicals, fractions or other slightly
1103higher constructs.
1104
1105\stopsectionlevel
1106
1107\page
1108
1109\stopdocument
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179 |