% language=us runpath=texruns:manuals/libraries

% author    : Hans Hagen
% copyright : ConTeXt Development Team
% license   : Creative Commons Attribution ShareAlike 4.0 International
% reference : pragma-ade.nl | contextgarden.net | texlive (related) distributions
% origin    : the ConTeXt distribution
%
% comment   : Because this manual is distributed with TeX distributions it comes with a rather
%             liberal license. We try to adapt these documents to upgrades in the (sub)systems
%             that they describe. Using parts of the content otherwise can therefore conflict
%             with existing functionality and we cannot be held responsible for that. Many of
%             the manuals contain characteristic graphics and personal notes or examples that
%             make no sense when used out-of-context.
%
% comment   : Some chapters might have been published in TugBoat, the NTG Maps, the ConTeXt
%             Group journal or otherwise. Thanks to the editors for corrections. Also thanks
%             to users for testing, feedback and corrections.

\usemodule[art-01,abr-02]

\definecolor
  [maincolor]
  [r=.4]

\definecolor
  [extracolor]
  [g=.4]

\setupbodyfont
  [11pt]

\setuptype
  [color=maincolor]

\setuptyping
  [color=maincolor]

\definefontsynonym
  [TitlePageMono]
  [file:lmmonoproplt10-bold*default]

\setuphead
  [color=maincolor]

\usesymbols
  [cc]

\setupinteraction
  [hidden]

\loadfontgoodies[lm]

\startdocument
  [metadata:author=Hans Hagen,
   metadata:title=Libraries in ConTeXt,
   author=Hans Hagen,
   affiliation=PRAGMA ADE,
   location=Hasselt NL,
   title=Libraries in \ConTeXt,
   support=www.contextgarden.net,
   website=www.pragma-ade.nl]

\startluasetups[libraries]
    context.nohyphens()
    for i=1,640 do
        context.definedfont { string.formatters["TitlePageMono at %p"](65536*(10+math.random(5))) }
        context("Libraries ")
    end
    context.removeunwantedspaces()
\stopluasetups

\startMPpage

StartPage ;

    fill Page enlarged 1cm withcolor \MPcolor{extracolor} ;

    draw textext("\framed[loffset=2pt,roffset=2pt,frame=off,width=\paperwidth,align={normal,paragraph,verytolerant,stretch}]{\luasetup{libraries}}")
        xysized (PaperWidth,PaperHeight)
        shifted center Page
        withcolor .8white ;

    draw textext.ulft("\definedfont[TitlePageMono]basics")
        xsized .75PaperWidth
        shifted lrcorner Page
        shifted (-1cm,2cm)
        withcolor \MPcolor{maincolor} ;

  % draw textext.ulft("\definedfont[TitlePageMono]in context mkiv")
  %     xsized .6PaperWidth
  %     shifted lrcorner Page
  %     shifted (-1cm,6cm)
  %     withcolor \MPcolor{maincolor} ;

StopPage ;

\stopMPpage

\dontcomplain

\startsubject[title=Contents]

\placelist[section][alternative=a]

\stopsubject

\startsection[title=Introduction]

Since we started developing \LUATEX\ several methods have been explored with
regards to external libraries. Now, before we go into details, it must be said
that in practice this feature is hardly needed. If someone really needs it, it is
likely to be in a setting where one can also write some library interface and
compile \LUATEX\ to suit it. So, what we're talking of here is probably not of
interest for all of you.

In the perspective of \CONTEXT\ a dependency on a library is not what we have in
mind when we advocate long term usage (and stability) of a workflow. If you see
\TEX\ as an independent component but still depend on its use of libraries you
might consider having a backup plan in case a library is no longer maintained, of
when it gets replaced by another favourite, as happens. There are several ways to
use a library, and we mention a few here.

The first one is to use \LUA\ {\em wrapper} libraries that interface to some {\em
specific} library. This is what you do when you use \LUA\ as stand alone program.
In that case you depend on someone cooking up an efficient and reliable
interface. Then you depend on the author or others to provide the binaries. If
there is only one target platform you can wonder if you like that additional
dependency. Another aspect to keep in mind is that such a wrapper library has to
match both the used library and the version of \LUA\ that you use. When we use
this method in \LUATEX, one also has also to make sure that the \LUATEX\ binary
is compiled in a way that permits loading (this has to do with exposing symbols
that need a runtime binding).

Because this didn't work out well in practice, already before version 1.0 showed
up we explored a flexible way to create libraries suited for \LUATEX. This
project was tagged \quotation {\SWIGLIB}. An infrastructure was created and a
couple of example library interfaces were provided. However, in practice this
never catched on and we don't expect distributions to provide libraries in
addition to the main \LUATEX\ binary, but the framework is there for those who
want to play with it.

Not long after we released \LUATEX\ 1.0, we started experimenting a bit more with
so called foreign function interface: \FFI. Originally that interface to external
libraries was only available in \LUAJITTEX, but a good and compatible alternative
is now also available in the normal engine too. For users it is not that relevant
to know how it works, as long as it works. It means that in addition to \SWIGLIB\
we have a method that doesn't demand compilation as it uses normal (public)
libraries.

A last alternative we mention is to just add the libraries to the \LUATEX\
engine. In fact, this happens: the \METAPOST\ binary has been provided this way
for quite a while now. In \LUAMETATEX\ for instance some more math related
libraries were added, simply because the overhead is not that large and because
it is a way to extend \METAPOST\ beyond what it provided out of the box.

Before we proceed, ask yourself this questions: \quotation {Do I really need more
libraries?} For instance does it really make sense to use \CURL, \GHOSTSCRIPT\ or
\GRAPHICSMAGICK\ libraries while the command line version is (normally) just as
efficient and avoids a dependency. This is even more true if you realizes that
for instance a fetch or conversion only needs to happen once per run or in fact
only when there is some change in the resource. On the other hand, when accessing
databases one can avoid the often slower command line calls and save the hassle
of intermediate files. Here efficiency wins. Also, when \CONTEXT\ is used in a
high performance database backend application a distribution and the used
libraries are not updated on a daily basis. Anyway, if the answer to the
question, and it can depend on what variant of \CONTEXT\ you use: \MKIV\ or
\LMTX, is \quotation {Yes!} then read on.

\stopsection

\startsection[title=Usage in \MKIV]

Although there are (still) some examples of \SWIGLIB\ in \MKIV\ we now assume
that the \FFI\ interface is used. Apart from some experiments we currently can
use \FFI\ interfaced libraries in:

\starttabulate[|l|l|l|l|l|]
\FL
\BC module               \BC library        \BC windows     \BC unix           \NC \NR
\ML
\NC util-crl             \NC curl           \NC libcurl     \NC libcurl        \NC \NR % todo: client and ffi
\NC util-sql-imp-ffi     \NC mysql          \NC libmysql    \NC libmysqlclient \NC \NR
\NC util-sql-imp-sqlite  \NC sqlite         \NC sqlite3     \NC sqlite3        \NC \NR
%NC font-phb-imp-library \NC harfbuzz       \NC libharfbuzz \NC libharfbuzz    \NC \NR % old, for testing uniscribe (idris fonts)
%NC                      \NC ghostscript    \NC             \NC                \NC \NR % only a few experiments
%NC                      \NC graphicsmagick \NC             \NC                \NC \NR % only a few experiments
\LL
\stoptabulate

The profiler that we occasionally use to identify bottlenecks in the engine (for
instance when we upgrade \LUA) uses \FFI\ to provide access to the high resolution
timers but this is typically different per platform.

One problem with libraries, especially on \WINDOWS\ is that the library is found on
some system path and it can happen that multiple programs ship the same library but in
different versions. You can try to play safe and put libraries in the \TEX\ tree, for
instance on my system they are in:

\starttyping
c:/data/tex-context/tex/texmf-win64/bin/lib/luatex/foo/libfoo.dll
\stoptyping

You can trace where libraries are looked for with:

\starttyping
\enabletrackers[resolvers.ffilib]
\stoptyping

or in \LUA\ with:

\starttyping
trackers.enable("resolvers.ffilib")
\stoptyping

The library is first located on one of the valid \TDS\ paths (these are sort of
standardized in \TEX\ distributions) and then using the normal \FFI\ loader.

\stopsection

\startsection[title=Usage in \LMTX]

In \CONTEXT\ \LMTX\ we only have so called \quote {optional} libraries. Actually
they are just simple interfaces that register themselves in the \type {optional}
namespace and that can load the real library at runtime. The approach is rather
lightweight in the sense that compilation of the \LUAMETATEX\ binary doesn't
depend on additional resources (there is no need to have the libraries on the
system or their source code to be present) and that at runtime there is no need
to have the libraries present. This sounds similar to delayed loading of \LUA\
wrapper libraries but the difference is that there is no potential \LUA\ version
clash and we also still have one single (relatively small) binary.

This approach works ok because in practice only very few users need a library and
when they do they have to compile the interface anyway. And, as they compile,
they can as well add a few lines that are needed to integrate their library to
the optional mechanism. If you depend on such extensions, you quite certainly
know what you're doing and need to maintain a code base very careful.

There are some optional libraries present, like:

\starttabulate[|l|l|l|l|l|]
\FL
\BC module                  \BC library        \BC windows      \BC unix           \NC \NR
\ML
\NC libs-imp-curl           \NC curl           \NC libcurl      \NC libcurl        \NC \NR
\NC libs-imp-mysql          \NC mysql          \NC libmysql     \NC libmysqlclient \NC \NR
\NC libs-imp-sqlite         \NC sqlite         \NC sqlite3      \NC sqlite3        \NC \NR
%NC libs-imp-postgress      \NC libpg          \NC libpg        \NC libpg          \NC \NR % untested
\NC libs-imp-zint           \NC libzint        \NC libzint      \NC libzint        \NC \NR
\NC libs-imp-mujs           \NC libmujs        \NC libmujs      \NC libmujs        \NC \NR
\NC libs-imp-kpse           \NC kpse           \NC kpathsea*w64 \NC libkpathsea    \NC \NR % left-over for mkii runner
\NC libs-imp-ghostscript    \NC ghostscript    \NC gswin64      \NC libgs          \NC \NR
\NC libs-imp-graphicsmagick \NC graphicsmagick \NC several      \NC unknown        \NC \NR
%NC font-phb-imp-internal   \NC hb             \NC libharfbuzz  \NC libharfbuzz    \NC \NR % maybe, for idris, testing uniscribe
\LL
\stoptabulate

It is still to be decided how these will be used. An example of a library that
can be used out of the box is \ZINT\ for using barcodes: see \type {m-zint.mkxl}
for an example of its usage. The \SQL\ libraries have their own manuals and have
been around for a while. They have a common encapsulating infrastructure written
in \LUA.

It is best to keep libraries in a place where you can keep an eye on them being
updated, like

\starttyping
c:/data/tex-context/tex/texmf-win64/bin/lib/luametatex/foo/libfoo.dll
\stoptyping

You could of course use the ones provided by the system or maybe use symbolic
links so that you still keep en eye on changes. The \CONTEXT\ distribution might
at some point provide the libraries that we support.

\stopsection

\startsection[title=Colofon]

\starttabulate[|B|p|]
\NC author    \NC \documentvariable{author}, \documentvariable{affiliation}, \documentvariable{location} \NC \NR
\NC version   \NC \currentdate \NC \NR
\NC website   \NC \documentvariable{website} \endash\ \documentvariable{support} \NC \NR
\NC comment   \NC many thanks to Luigi Scarso for taking care of \FFI\ support in the engines \NC \NR
\stoptabulate

\stopsection

\stopdocument