followingup-bitmaps.tex /size: 9353 b    last modification: 2021-10-28 13:50
1% language=us runpath=texruns:manuals/followingup
2
3\startcomponent followingup-bitmaps
4
5\environment followingup-style
6
7\startchapter[title={Bitmap images}]
8
9\startsection[title={Introduction}]
10
11In \TEX\ image inclusion is traditionally handled by specials. Think of a signal
12added someplace in the page stream that says:
13
14\starttyping
15\special{image: foo.png 2000 3000}
16\stoptyping
17
18Here the number for instance indicate a scale factor to be divided by 1000.
19Because \TEX\ has no floating point numbers, normally one uses an integer and the
20magic multiplier 1000 representing 1.000. Such a special is called a \quote
21{whatsit} and is one reason why \TEX\ is so flexible and adaptive.
22
23In \PDFTEX\ instead of a \type {\special} the command \type {\pdfximage} and its
24companions are used. In \LUATEX\ this concept has been generalized to \type
25{\useimageresource} which internally is not a so called whatsit (an extension
26node) but a special kind of rule. This makes for nicer code as now we don't need
27to check if a certain whatsit node is actually one with dimensions, while rules
28already are part of calculating box dimensions, so no extra overhead in checking
29for whatsits is added. In retrospect this was one of the more interesting
30conceptual changes in \LUATEX.
31
32In \LUAMETATEX\ we don't have such primitives but we do have these special rule
33nodes; we're talking of subtypes and the frontend doesn't look at those details.
34Depending on what the backend needs one can easily define a scanner that
35implements a primitive. We already did that in \CONTEXT. More important is that
36inclusion is not handled by the engine simply because there is no backend. This
37means that we need to do it ourselves. There are two steps involved in this that
38we will discuss below.
39
40\stopsection
41
42\startsection[title={Identifying}]
43
44There is only a handful of image formats that makes sense in a typesetting
45workflow. Because \PDF\ inclusion is supported (but not discussed here) one can
46actually take any format as long as it converts to \PDF, and tools like graphic
47magic do a decent job on that. \footnote {Although one really need to check a
48converted image. When we moved to pplib, I found out that lots of converted
49images in a project had invalid \PDF\ objects, but apart from a warning nothing
50bad resulted from this because those objects were not used.} The main bitmap
51formats that we care about are \JPEG, \JPEG2000, and \PNG. We could deal with
52\JBIG\ files but I never encountered them so let's forget about them for now.
53
54One of the problems with a built|-|in analyzer (and embedder) is that it can
55crash or just abort the engine. The main reason is that when the used libraries
56run into some issue, the engine is not always able to recover from it: a
57converter just aborts which then cleans up (potentially messed up) memory. In
58\LUATEX\ we also abort, simply because we have no clue to what extend further on
59the libraries are still working as expected. We play safe. For the average user
60this is quite ok as it signals that an image has to be fixed.
61
62In a workflow that runs unattended on a server and where users push images to a
63resource tree, there is a good change that a \TEX\ job fails because of some
64problem with images. A crash is not really an option then. This is one reason why
65converting bitmaps to \PDF\ makes much sense. Another reason is that some color
66profiling might be involved. Runtime manipulations make no sense, unless there is
67only one typesetting run.
68
69Because in \LMTX\ we do the analyzing ourselves \footnote {Actually, in \MKIV\
70this was also possible but not widely advertised, but we now exclusively keep
71this for \LMTX.} we can recover much easier. The main reason is of course that
72because we use \LUA, memory management and garbage collection happens pretty well
73controlled. And crashing \LUA\ code can easily be intercepted by a \type {pcall}.
74
75Most (extensible) file formats are based on tables that gets accessed from an
76index of names and offsets into the file. This means that filtering for instance
77metadata like dimensions and resolutions is no big deal (we always did that). I
78can extend analyzing when needed without a substantial change in the engine that
79can affect other macro packages. And \LUA\ is fast enough (and often faster) for
80such tasks.
81
82\stopsection
83
84\startsection[title={Embeding}]
85
86Once identified the frontend can use that information for scaling and (if needed)
87reuse of the same image. Embedding of the image resource happens when a page is
88shipped out. For \JPEG\ images this is actually quite simple: we only need to
89create a dictionary with the right information and push the bitmap itself into
90the associated stream.
91
92For \PNG\ images it's a bit different. Unfortunately \PDF\ only supports certain
93formats, for instance masks are separated and transparency needs to be resolved.
94This means that there are two routes: either pass the bitmap blob to the stream,
95or convert it to a suitable format supported by \PDF. In \LUATEX\ that is
96normally done by the backend code, which uses a library for this. It is a typical
97example of a dependency of something much larger than actually needed. In
98\LUATEX\ the original poppler library used for filtering objects from a \PDF\
99file as well as the \PNG\ library also have tons of code on board that relates to
100manipulating (writing) data. But we don't need those features. As a side note:
101this is something rather general. You decide to use a small library for a simple
102task only to find out after a decade that it has grown a lot offering features
103and having extra dependencies that you really don't want. Even worse: you end up
104with constant updates due to fixed security (read: bug) fixes.
105
106Passing the \PNG\ blob unchanged in itself to the \PDF\ file is trivial, but
107massaging it into an acceptable form when it doesn't suit the \PDF\ specification
108takes a bit more code. In fact, \PDF\ does not really support \PNG\ as format,
109but it supports \PNG\ compression (aka filters).
110
111Trying to support more complex \PNG\ files is a nice way to test if you can
112transform a public specification into a program as for instance happens with
113\PDF, \OPENTYPE, and font embedding in \CONTEXT. So this again was a nice
114exercise in coding. After a while I was able to process the \PNG\ test suite
115using \LUA. Optimizing the code came with understanding the specification.
116However, for large images, especially interlaced ones, runtime was definitely not
117to be ignored. It all depended on the tasks at hand:
118
119\startitemize
120
121\startitem
122    A \PNG\ blob is compressed with \ZIP\ compression, so first it needs to be
123    decompressed. This takes a bit of time (and in the process we found out that
124    the \type {zlib} library used in \LUATEX\ had a bug that surfaced when a
125    mostly zero byte image was uncompressed and we can then hit a filled up
126    buffer condition.
127\stopitem
128
129\startitem
130    The resulting uncompressed stream is itself compressed with a so called
131    filter. Each row starts with a filter byte that indicates how to convert
132    bytes into other bytes. The most commonly used methods are deltas with
133    preceding pixels and/or pixels on a previous row. When done the filter bytes
134    can go away.
135\stopitem
136
137\startitem
138    Sometimes an image uses 1, 2 or 4 bits per pixel, in which case the rows
139    needs to be expanded. This can involve a multiplication factor per pixel (it
140    can also be an index in a palette).
141\stopitem
142
143\startitem
144    An image can be interlaced which means that there are seven parts of the
145    image that stepwise build up the whole. In professional workflows with high
146    res images interlacing makes no sense as transfer over the internet is not an
147    issue and the overhead due to reassembling the image and the potentially
148    larger file size (due to independent compression of the seven parts) are not
149    what we want either.
150\stopitem
151
152\startitem
153    There can be an image mask that needs to be separated from the main blob. A
154    single byte gray scale image then has two bytes per pixel, and a double byte
155    pixel has four bytes of information. An \RGB\ image has three bytes per pixel
156    plus an alpha byte, and in the case of double byte pixels we get eight bytes
157    per pixel.
158\stopitem
159
160\startitem
161    Finally the resulting blob has to be compressed again. The current amount of
162    time involved in that suggests that there is room for improvement.
163\stopitem
164
165\stopitemize
166
167The process is controlled by number of rows and columns, the number of bytes per
168pixel (one or two) and the color space which effectively means one or three
169bytes. These numbers get fed into the filter, deinterlacer, expander and|/|or
170mask separator. In order to speed up the embedding these basic operations can be
171assisted by a helpers written in \CCODE. Because \LUA\ is quite good with
172strings, we pass strings and get back strings. So, most of the logic stays at the
173\LUA\ end.
174
175\stopsection
176
177\startsection[title=Conclusion]
178
179Going for a library|-|less solution for bitmap inclusion is quite doable and in
180most cases as efficient. Because we have a pure \LUA\ implementation for testing
181and an optimized variant for production, we can experiment as we like. A positive
182side effect is that we can more robustly intercept bad images and inject a
183placeholder instead.
184
185\stopsection
186
187\stopchapter
188
189\stopcomponent
190