libraries-mkiv.tex /size: 11 Kb    last modification: 2023-12-21 09:43
1% language=us runpath=texruns:manuals/libraries
2
3% author    : Hans Hagen
4% copyright : ConTeXt Development Team
5% license   : Creative Commons Attribution ShareAlike 4.0 International
6% reference : pragma-ade.nl | contextgarden.net | texlive (related) distributions
7% origin    : the ConTeXt distribution
8%
9% comment   : Because this manual is distributed with TeX distributions it comes with a rather
10%             liberal license. We try to adapt these documents to upgrades in the (sub)systems
11%             that they describe. Using parts of the content otherwise can therefore conflict
12%             with existing functionality and we cannot be held responsible for that. Many of
13%             the manuals contain characteristic graphics and personal notes or examples that
14%             make no sense when used out-of-context.
15%
16% comment   : Some chapters might have been published in TugBoat, the NTG Maps, the ConTeXt
17%             Group journal or otherwise. Thanks to the editors for corrections. Also thanks
18%             to users for testing, feedback and corrections.
19
20\usemodule[art-01,abr-02]
21
22\definecolor
23  [maincolor]
24  [r=.4]
25
26\definecolor
27  [extracolor]
28  [g=.4]
29
30\setupbodyfont
31  [11pt]
32
33\setuptype
34  [color=maincolor]
35
36\setuptyping
37  [color=maincolor]
38
39\definefontsynonym
40  [TitlePageMono]
41  [file:lmmonoproplt10-bold*default]
42
43\setuphead
44  [color=maincolor]
45
46\usesymbols
47  [cc]
48
49\setupinteraction
50  [hidden]
51
52\loadfontgoodies[lm]
53
54\startdocument
55  [metadata:author=Hans Hagen,
56   metadata:title=Libraries in ConTeXt,
57   author=Hans Hagen,
58   affiliation=PRAGMA ADE,
59   location=Hasselt NL,
60   title=Libraries in \ConTeXt,
61   support=www.contextgarden.net,
62   website=www.pragma-ade.nl]
63
64\startluasetups[libraries]
65    context.nohyphens()
66    for i=1,640 do
67        context.definedfont { string.formatters["TitlePageMono at %p"](65536*(10+math.random(5))) }
68        context("Libraries ")
69    end
70    context.removeunwantedspaces()
71\stopluasetups
72
73\startMPpage
74
75StartPage ;
76
77    fill Page enlarged 1cm withcolor \MPcolor{extracolor} ;
78
79    draw textext("\framed[loffset=2pt,roffset=2pt,frame=off,width=\paperwidth,align={normal,paragraph,verytolerant,stretch}]{\luasetup{libraries}}")
80        xysized (PaperWidth,PaperHeight)
81        shifted center Page
82        withcolor .8white ;
83
84    draw textext.ulft("\definedfont[TitlePageMono]basics")
85        xsized .75PaperWidth
86        shifted lrcorner Page
87        shifted (-1cm,2cm)
88        withcolor \MPcolor{maincolor} ;
89
90  % draw textext.ulft("\definedfont[TitlePageMono]in context mkiv")
91  %     xsized .6PaperWidth
92  %     shifted lrcorner Page
93  %     shifted (-1cm,6cm)
94  %     withcolor \MPcolor{maincolor} ;
95
96StopPage ;
97
98\stopMPpage
99
100\dontcomplain
101
102\startsubject[title=Contents]
103
104\placelist[section][alternative=a]
105
106\stopsubject
107
108\startsection[title=Introduction]
109
110Since we started developing \LUATEX\ several methods have been explored with
111regards to external libraries. Now, before we go into details, it must be said
112that in practice this feature is hardly needed. If someone really needs it, it is
113likely to be in a setting where one can also write some library interface and
114compile \LUATEX\ to suit it. So, what we're talking of here is probably not of
115interest for all of you.
116
117In the perspective of \CONTEXT\ a dependency on a library is not what we have in
118mind when we advocate long term usage (and stability) of a workflow. If you see
119\TEX\ as an independent component but still depend on its use of libraries you
120might consider having a backup plan in case a library is no longer maintained, of
121when it gets replaced by another favourite, as happens. There are several ways to
122use a library, and we mention a few here.
123
124The first one is to use \LUA\ {\em wrapper} libraries that interface to some {\em
125specific} library. This is what you do when you use \LUA\ as stand alone program.
126In that case you depend on someone cooking up an efficient and reliable
127interface. Then you depend on the author or others to provide the binaries. If
128there is only one target platform you can wonder if you like that additional
129dependency. Another aspect to keep in mind is that such a wrapper library has to
130match both the used library and the version of \LUA\ that you use. When we use
131this method in \LUATEX, one also has also to make sure that the \LUATEX\ binary
132is compiled in a way that permits loading (this has to do with exposing symbols
133that need a runtime binding).
134
135Because this didn't work out well in practice, already before version 1.0 showed
136up we explored a flexible way to create libraries suited for \LUATEX. This
137project was tagged \quotation {\SWIGLIB}. An infrastructure was created and a
138couple of example library interfaces were provided. However, in practice this
139never catched on and we don't expect distributions to provide libraries in
140addition to the main \LUATEX\ binary, but the framework is there for those who
141want to play with it.
142
143Not long after we released \LUATEX\ 1.0, we started experimenting a bit more with
144so called foreign function interface: \FFI. Originally that interface to external
145libraries was only available in \LUAJITTEX, but a good and compatible alternative
146is now also available in the normal engine too. For users it is not that relevant
147to know how it works, as long as it works. It means that in addition to \SWIGLIB\
148we have a method that doesn't demand compilation as it uses normal (public)
149libraries.
150
151A last alternative we mention is to just add the libraries to the \LUATEX\
152engine. In fact, this happens: the \METAPOST\ binary has been provided this way
153for quite a while now. In \LUAMETATEX\ for instance some more math related
154libraries were added, simply because the overhead is not that large and because
155it is a way to extend \METAPOST\ beyond what it provided out of the box.
156
157Before we proceed, ask yourself this questions: \quotation {Do I really need more
158libraries?} For instance does it really make sense to use \CURL, \GHOSTSCRIPT\ or
159\GRAPHICSMAGICK\ libraries while the command line version is (normally) just as
160efficient and avoids a dependency. This is even more true if you realizes that
161for instance a fetch or conversion only needs to happen once per run or in fact
162only when there is some change in the resource. On the other hand, when accessing
163databases one can avoid the often slower command line calls and save the hassle
164of intermediate files. Here efficiency wins. Also, when \CONTEXT\ is used in a
165high performance database backend application a distribution and the used
166libraries are not updated on a daily basis. Anyway, if the answer to the
167question, and it can depend on what variant of \CONTEXT\ you use: \MKIV\ or
168\LMTX, is \quotation {Yes!} then read on.
169
170\stopsection
171
172\startsection[title=Usage in \MKIV]
173
174Although there are (still) some examples of \SWIGLIB\ in \MKIV\ we now assume
175that the \FFI\ interface is used. Apart from some experiments we currently can
176use \FFI\ interfaced libraries in:
177
178\starttabulate[|l|l|l|l|l|]
179\FL
180\BC module               \BC library        \BC windows     \BC unix           \NC \NR
181\ML
182\NC util-crl             \NC curl           \NC libcurl     \NC libcurl        \NC \NR % todo: client and ffi
183\NC util-sql-imp-ffi     \NC mysql          \NC libmysql    \NC libmysqlclient \NC \NR
184\NC util-sql-imp-sqlite  \NC sqlite         \NC sqlite3     \NC sqlite3        \NC \NR
185%NC font-phb-imp-library \NC harfbuzz       \NC libharfbuzz \NC libharfbuzz    \NC \NR % old, for testing uniscribe (idris fonts)
186%NC                      \NC ghostscript    \NC             \NC                \NC \NR % only a few experiments
187%NC                      \NC graphicsmagick \NC             \NC                \NC \NR % only a few experiments
188\LL
189\stoptabulate
190
191The profiler that we occasionally use to identify bottlenecks in the engine (for
192instance when we upgrade \LUA) uses \FFI\ to provide access to the high resolution
193timers but this is typically different per platform.
194
195One problem with libraries, especially on \WINDOWS\ is that the library is found on
196some system path and it can happen that multiple programs ship the same library but in
197different versions. You can try to play safe and put libraries in the \TEX\ tree, for
198instance on my system they are in:
199
200\starttyping
201c:/data/tex-context/tex/texmf-win64/bin/lib/luatex/foo/libfoo.dll
202\stoptyping
203
204You can trace where libraries are looked for with:
205
206\starttyping
207\enabletrackers[resolvers.ffilib]
208\stoptyping
209
210or in \LUA\ with:
211
212\starttyping
213trackers.enable("resolvers.ffilib")
214\stoptyping
215
216The library is first located on one of the valid \TDS\ paths (these are sort of
217standardized in \TEX\ distributions) and then using the normal \FFI\ loader.
218
219\stopsection
220
221\startsection[title=Usage in \LMTX]
222
223In \CONTEXT\ \LMTX\ we only have so called \quote {optional} libraries. Actually
224they are just simple interfaces that register themselves in the \type {optional}
225namespace and that can load the real library at runtime. The approach is rather
226lightweight in the sense that compilation of the \LUAMETATEX\ binary doesn't
227depend on additional resources (there is no need to have the libraries on the
228system or their source code to be present) and that at runtime there is no need
229to have the libraries present. This sounds similar to delayed loading of \LUA\
230wrapper libraries but the difference is that there is no potential \LUA\ version
231clash and we also still have one single (relatively small) binary.
232
233This approach works ok because in practice only very few users need a library and
234when they do they have to compile the interface anyway. And, as they compile,
235they can as well add a few lines that are needed to integrate their library to
236the optional mechanism. If you depend on such extensions, you quite certainly
237know what you're doing and need to maintain a code base very careful.
238
239There are some optional libraries present, like:
240
241\starttabulate[|l|l|l|l|l|]
242\FL
243\BC module                  \BC library        \BC windows      \BC unix           \NC \NR
244\ML
245\NC libs-imp-curl           \NC curl           \NC libcurl      \NC libcurl        \NC \NR
246\NC libs-imp-mysql          \NC mysql          \NC libmysql     \NC libmysqlclient \NC \NR
247\NC libs-imp-sqlite         \NC sqlite         \NC sqlite3      \NC sqlite3        \NC \NR
248%NC libs-imp-postgress      \NC libpg          \NC libpg        \NC libpg          \NC \NR % untested
249\NC libs-imp-zint           \NC libzint        \NC libzint      \NC libzint        \NC \NR
250\NC libs-imp-mujs           \NC libmujs        \NC libmujs      \NC libmujs        \NC \NR
251\NC libs-imp-kpse           \NC kpse           \NC kpathsea*w64 \NC libkpathsea    \NC \NR % left-over for mkii runner
252\NC libs-imp-ghostscript    \NC ghostscript    \NC gswin64      \NC libgs          \NC \NR
253\NC libs-imp-graphicsmagick \NC graphicsmagick \NC several      \NC unknown        \NC \NR
254%NC font-phb-imp-internal   \NC hb             \NC libharfbuzz  \NC libharfbuzz    \NC \NR % maybe, for idris, testing uniscribe
255\LL
256\stoptabulate
257
258It is still to be decided how these will be used. An example of a library that
259can be used out of the box is \ZINT\ for using barcodes: see \type {m-zint.mkxl}
260for an example of its usage. The \SQL\ libraries have their own manuals and have
261been around for a while. They have a common encapsulating infrastructure written
262in \LUA.
263
264It is best to keep libraries in a place where you can keep an eye on them being
265updated, like
266
267\starttyping
268c:/data/tex-context/tex/texmf-win64/bin/lib/luametatex/foo/libfoo.dll
269\stoptyping
270
271You could of course use the ones provided by the system or maybe use symbolic
272links so that you still keep en eye on changes. The \CONTEXT\ distribution might
273at some point provide the libraries that we support.
274
275\stopsection
276
277\startsection[title=Colofon]
278
279\starttabulate[|B|p|]
280\NC author    \NC \documentvariable{author}, \documentvariable{affiliation}, \documentvariable{location} \NC \NR
281\NC version   \NC \currentdate \NC \NR
282\NC website   \NC \documentvariable{website} \endash\ \documentvariable{support} \NC \NR
283\NC comment   \NC many thanks to Luigi Scarso for taking care of \FFI\ support in the engines \NC \NR
284\stoptabulate
285
286\stopsection
287
288\stopdocument
289