Source Browser ?

1% language=us runpath=texruns:manuals/evenmore
2
3\startcomponent evenmore-libraries
4
5\environment evenmore-style
6
7\startchapter[title={Libraries}]
8
9\startsection[title={Introduction}]
10
11The \LUAMETATEX\ binary comes with a couple of libraries built in. These normally
12provide enough functionality to get a \TEX\ job done. But take the case where
13need to manipulate (or convert) an image before we can include it? It would be
14nice if \CONTEXT\ does that for you so having some features in the binary that
15handle it make sense. However, given that such a conversion only happens once it
16makes more sense to just call an external program and let that deal with it. It
17is for that reason that the \CONTEXT\ code base has hardly any library related
18code: most of what one wants to do can be done by calling a program. Some callers
19are built in, others can be dealt with using the Adityas filter module. The most
20significant runtime exception is probably accessing \SQL\ databases where it
21might be more efficient to use a library call instead of calling a client. And
22even then the main reason for that interface being present is the simple fact
23that I (ab)use the engine to serve requests that need some kind of database
24access. Another example of where we need some external program is in generating
25barcodes. Here one can argue that it does make sense to do that runtime, for
26instance because they change or because one doesn't like to have dozens of cached
27barcode images on disk.
28
29In this chapter I will explain how we deal with libraries in \LUAMETATEX. Because
30libraries create a dependency an approach is chosen that tries to avoid bloating
31the source tree with additional header and source files. This is made easy by the
32fact that we don't need full blown interfaces to libraries where all methods are
33exposed. We know what we need and most of these tasks somehow relate to
34typesetting which is a limited application with known demands in terms of input,
35output and performance. We don't need to serve every possible scenario.
36
37\stopsection
38
39\startsection[title={Using \LUA\ libraries}]
40
41One approach is to use a \LUA\ library that sits between the embedded \LUA\ instance
42and the external library. Say that one does this:
43
44\starttyping
45local mylib = require("mylib")
46\stoptyping
47
48This can locate and load the file \type {mylib.lua} which implements a bunch of
49(\LUA) functions. But, it can also load a library, for instance \type
50{mylib.dll}, a binary that provides functions that themselves can call external
51ones. Often such a library is also responsible for some resource management which
52is then done via userdata objects. Such a connector library on the one hand
53refers to \LUA\ library methods (like \type {const char * str = lua_tostring (L,
541);} for fetching a \LUA\ string variable from the argument list) and on the
55other hand to those in the external library (like passing that string \type {str}
56to a function and passing the result back to \LUA\ with \type {lua_pushstring (L,
57result);}). If we would follow that approach in \LUAMETATEX\ it means that in
58addition to the main binary (on \MSWINDOWS\ that is \type {luametatex.exe}) there
59is also an extra intermediate binary (on \MSWINDOWS\ that is \type {mylib.dll})
60plus the external library (on \MSWINDOWS\ that could be \type {foolib.dll}) which
61itself can depend on other libraries.
62
63In this approach we need to compile the extra intermediate libraries alongside
64the main \LUAMETATEX\ binary. Quite likely we then need access to the header
65files of the external libraries too. We might even decide to put the dependencies
66in our source tree. But, this is not what we like to do: it adds extra work, we
67need to keep an eye on updates and operating specific patches, we complicate the
68compilation, etc. This all contradicts the fact that we want \LUAMETATEX\ to be
69simple. There is no need to complicate the setup just because a very few users
70want to use some library. Add to this the fact that quite likely we need to
71provide a version of \LUAMETATEX\ that exposes its \LUA\ related symbols which
72makes for a larger binary. So, this approach is not really an option because
73at the same time we like to keep the binary (and memory footprint) as small
74as possible (think of running in a container or on a low energy device).
75
76\stopsection
77
78\startsection[title=A variant]
79
80There are a few issues when you use \LUA\ libraries from elsewhere. First of all,
81you need to get hold of one that matches the version of \LUA\ that you use. There
82are not that many and some only can be set up as part of a larger framework.
83Also, you can find plenty of modules that seem not to be maintained (or maybe
84they are just very stable and I'm wrong here). Also, not all platforms are
85supported equally well. Then there is the question to what extend libraries are
86to stay. What is considered to be the standard today might not be tomorrow. Even
87in the rather stable \TEX\ ecosystem we see them come and go. These are all
88reasons to avoid hard coded dependencies. Ideally we like users to be able to
89compile \LUAMETATEX\ in the future without too must hassle.
90
91A couple of years after we started the \LUATEX\ project, a solution for using
92libraries was implemented, called \SWIGLIB, because it uses the swig
93infrastructure. It was an attempt to come up with a more or less standard
94approach, a rather one|-|to|-|one mapping so that basically any library could be
95interfaced. But, probably because no one really needs libraries, it never catched
96on. In \MKIV\ we still support loading libraries made that way but in \LMTX\ that
97code has been removed.
98
99As a side note: the code that deals with this in \MKIV\ also deals with version
100specific loading. When we were playing with for instance \MYSQL\ libs we found
101out that it made sense to be able to support different \API s, but in the end,
102given the rare usage of libraries, that made no sense either. Therefore in \LMTX\
103locating libraries has version support removes and as a consequence is much
104simpler (code|-|wise).
105
106\stopsection
107
108\startsection[title=Foreign function interfaces]
109
110Then there is a \FFI\ interface, first introduced in \LUAJITTEX\ as it is part of
111\LUAJIT, and later a similar library was built|-|in \LUATEX. But \LUAJIT\ doesn't
112conceptually follow \LUA\ upgrades and its future is unsure so in \LUAMETATEX\
113there is no \JIT\ variant (the \JIT\ part was never used anyway as it only slowed
114down a run; we just used the \FFI\ part plus the fact that the restricted virtual
115machine performs better). The \FFI\ library used in \LUATEX\ also comes from
116elsewhere and it doesn't seem to be maintained any longer, so that code is to be
117kept working in the perspective of \LUATEX. Both technologies hook into the
118processor architecture and are somewhat complex so when their maintenance becomes
119unsure we have to reconsider using them. Not all hardware platforms are supported
120\footnote {As I write this only Intel works while ARM doesn't and only on
121\MSWINDOWS, \LINUX\ and \OSX\ I can compile without alignment warnings} and the
122functionality can differ in details per platform. To some extend we can keep
123using \FFI\ in \LUATEX\ because Luigi takes care of it, but who knows when it
124becomes too problematic. Does it make sense to adopt a library that needs tweaks
125depending on architectures? For now we're good for \LUATEX, so for a while we're
126also okay (in \MKIV).
127
128The nice thing about \FFI\ is that one can define the interface at runtime. Of
129course this interface has to fit the current version of the library \API, but
130that is doable. It is up to a user of a library to determine where it comes from.
131It can be put in the \TEX\ tree but also being taken from wherever the operating
132system put it in the path. Of course that can then be a bit of an issue when
133there are different versions because programs can ship their own variants, but
134when you use a library you probably are aware of that and know what you're doing.
135A drawback of \FFI\ is that it opens up the whole machinery pretty low level,
136which can be considered a risk. Some can consider that to be a security threat.
137It for these reasons that \LUAMETATEX\ doesn't provide the \FFI\ feature; users
138who depend on it can of course use \MKIV\ with \LUATEX.
139
140\stopsection
141
142\startsection[title=So how to proceed?]
143
144When a library and its \LUA\ interface are kept external the main binary has to
145be compiled in a way that permits loading libraries (read: symbols need to be
146known). When we use \FFI\ that is not needed. And when a library is internal we
147have the disadvantage that we mentioned at the start of this chapter.
148
149So, how do we combine the advantages of \FFI\ (runtime binding), external
150libraries (no need to have all that code in the code base) and internal libraries
151(no loading issues)? At some point it stroke me that we actually can do that with
152not that much effort. The solution was probably subconsciously implanted by
153noticing the fact that the \LUAMETATEX\ machinery uses function pointers in some
154places and the fact that when a \LUA\ library is loaded by \LUA\ itself, a
155specific initialization function is called to initialize it: by combining these
156concepts we can delay the binding till when a library is needed.
157
158In \LUAMETATEX\ we can therefore have some optional libraries that offer a
159minimal interface because after all we can do a lot at the \LUA\ end. Optional
160libraries register themselves in the global \type {optional} table. We're talking
161of a couple of hundred lines of \CCODE\ for a simple binding. The functions in an
162optional library table can be used (accessed) without loading the library and
163then just do nothing useful. So, before using them you need to load the third
164party library but we can safely assume that the \LUA\ wrapper code calls an
165initializer when it needs some feature. That initializer, which by the way is
166located at the \LUA\ end, loads the external library, and when that is successful
167the needed helpers are bound by resolving function pointers. There is no
168dependency when nothing is used: the main binary stays lean and mean because the
169binding normally only adds a few \KB. Users can compile without dependencies and
170when used performance is quite okay (no \FFI\ overhead).
171
172The \LUAMETATEX\ distribution only ships a few such bindings but these can serve
173as example. What is shipped has a proper \LUA\ companion file and these are then
174the standard one used in the \CONTEXT\ distribution. Think of \MYSQL\ and
175\SQLITE\ (for databases), \ZINT\ (for barcodes), simple \CURL\ (for fetching
176stuff), \GHOSTSCRIPT\ and \GRAPHICSMAGICK\ (for some conversions) bindings . When
177compiled into \LUAMETATEX\ these will add some interfacing code to the main
178binary but that gets compensated by the removal of the \FFI\ library. The \LUA\
179interfaces provide just enough to get us going. At some point we can consider
180providing libraries as optional part of an installation because we can generate
181them using the buildbot infrastructure managed by Mojca, but the core
182distribution (source code) is kept clean.
183
184\stopsection
185
186\stopchapter
187
188\stopcomponent
189