Source Browser ?

1% language=us runpath=texruns:manuals/musings
2
3\startcomponent musings-assumptions
4
5\environment musings-style
6
7\setuptolerance[tolerant,stretch]
8
9\startchapter[title={\CONTEXT\ in \TEXLIVE\ 2023}]
10
11Starting with \TEXLIVE\ 2023 the default \CONTEXT\ distribution is \LMTX, a
12follow up on \MKIV, running on top of the \LUAMETATEX\ engine instead of \LUATEX.
13Already for a long time the \MKII\ version used with \PDFTEX, \XETEX\ and \ALEPH\
14has been frozen and most users moved on from \MKIV\ to \LMTX\ (a more distinctive
15tag for what internally is version \MKXL).
16
17In principle one can argue that we now have three versions of \CONTEXT\ and there
18can be the impression that they are very different. However, although \MKXL\ can
19do more than \MKIV\ which can do more than \MKII, the user interface hasn't
20changed that much and old functionality is available in newer versions. Of course
21some old features make no sense in newer variants, like eight|-|bit font
22encodings in an \OPENTYPE\ font realm and input encodings when one uses \UTF,
23although we still support input encodings a.k.a.\ regimes. When we started using
24the \type {Mk*} suffixes the main reason was that we had to distinguish files and
25the official \TEX\ distribution doesn't permit duplicate file names. Using a
26distinctive suffix also makes it possible to treat files differently.
27
28\starttabulate[|T|c|c|c|Tl|]
29\BC suffix \BC engine                           \BC template \BC arguments \BC main file \NC \NR
30\HL
31\NC MkII   \NC \PDFTEX, \XETEX, \ALEPH          \NC          \NC           \NC context.mkii \NC \NR
32\HL
33\NC MkIV   \NC \LUATEX, \LUAJITTEX, \LUAMETATEX \NC          \NC           \NC context.mkiv \NC \NR
34\NC MkVI   \NC idem                             \NC          \NC   yes     \NC \NC \NR
35\NC MkIX   \NC idem                             \NC   yes    \NC           \NC \NC \NR
36\NC MkXI   \NC idem                             \NC   yes    \NC   yes     \NC \NC \NR
37\HL
38\NC MkXL   \NC \LUAMETATEX                      \NC          \NC           \NC context.mkxl \NC \NR
39\NC MkLX   \NC idem                             \NC          \NC   yes     \NC \NC \NR
40\stoptabulate
41
42In this table \quote {template} files are a mix of \TEX\ and \LUA\ and originate
43in the early days of \MKIV; basically, they are a wink to active server pages.
44With \quote {arguments} we refer to files that accept named macro arguments which
45means that they need to be preprocessed. That started as a proof of concept but
46some core files are defined that way. Users will normally just use a \type {.tex}
47file.
48
49The \LUA\ files in the code base have the suffix \type {lua}, or when meant for
50\LUAMETATEX\ that uses a newer \LUA\ engine they can have the suffix \type {lmt}.
51There can also be \type {lfg} (font goodies) and \type {llg} (language goodies)
52plus byte|-|compiled files with various suffixes but these are normally not seen
53by users. We leave it at that.
54
55So, while \TEXLIVE\ 2022 installed \MKII\ and \MKIV, \TEXLIVE\ 2023 installs
56\MKIV\ and \LMTX. Therefore the most significant upgrade is in the engine that is
57used by default: \LUAMETATEX\ instead of \LUATEX. The \MKII\ files are no longer
58installed so we don't need \PDFTEX.
59
60So how did we end up here? Initially the idea was that, because \LUATEX\ is
61basically frozen, \LUAMETATEX\ would be the engine that we conduct experiments
62with and from which occasionally we could backport code to \LUATEX. However it
63soon became clear that this would not work out well so backporting is off the
64table now. Just for the record: the project started years ago so we're not
65talking about something experimental here. There have been articles in \TUGBOAT\
66about what we've been doing over the years.
67
68One of the first decisions I made when starting with \LUAMETATEX\ was to remove
69the built|-|in backend, which then meant also removing the bitmap image inclusion
70code. That made us get rid of dependencies on external libraries. In fact, a
71proof|-|of|-|concept experimental variant didn't use the built|-|in backend at
72all. The font loading code could be removed as well because that was not used in
73\MKIV\ either. In \MKIV\ we also don't use the \KPSE\ library for managing files
74so that code could be dropped from the engine tool; it can be loaded as
75so|-|called optional library if needed but I'll not discuss that here. If you
76look at what happens with the \LUATEX\ code base, you'll notice that updating
77libraries happens frequently and that is not a burden that we want to impose on
78users, especially because it also can involve updating build|-|related files.
79Another advantage of not using them is that the code base remains small.
80
81A direct consequence of all this was that the build process became much more
82efficient and less complex. A fast compilation (seconds instead of minutes) meant
83that more drastic experiments became possible, like most recently an upgrade of
84the math subsystem. All this, combined with an overhaul of the code base, both
85the \TEX\ and \METAPOST\ part, meant that backporting was no longer reasonable.
86Being freed from the constraint that other macro packages might use \LUAMETATEX\
87in turn resulted in more drastic experiments and adding features that had been on
88our wish list for decades. Another side effect was that we could easily compile
89native \MSWINDOWS\ binaries and immediately support transitions to \ARM|-|based
90hardware.
91
92Instead of \quotation {backporting after experimenting}, a leading motive became
93\quotation {fundamentally move forward} while at the same time tightening the
94relation between \CONTEXT\ and the engine: the engine code became part of the
95distribution so that users can compile themselves, which fits perfectly in the
96paradigm (and demands) of distributing all the source code, even that of the
97engine. There is also less danger that patches on behalf of other usage
98interferes with stable support for \CONTEXT. A specific installation is now more
99or less long|-|term stable by design because it no longer depends on binaries
100and|/|or libraries being provided for a specific platform and operating system
101version. Of course installers and \TEXLIVE\ do provide the binaries, so users
102aren't forced to worry about it, but they can move along with a system update by
103recompiling an old, and for their purpose, frozen \CONTEXT\ code base.
104
105An unofficial objective (or challenge) became that the accumulated source stays
106around 12~\MB\ uncompressed, (compressed a bit over 2~\MB) and the binary around
1073~\MB\ so that we could use the engine as an efficient \LUA\ runner as well as a
108launcher stub, thereby removing yet another dependency. That way the official
109\CONTEXT\ distribution didn't grow much in size. A bonus is that we now use the
110same setup for all operating systems. It also opened up the possibility of a
111exceptionally small installation with all bells and whistles included. Another
112nice side effect, combined with automatic compilation on the compile farm, makes
113that we can provide installations that reflect the latest state of affairs: a
114recent binary combined with the latest \CONTEXT. As a result, most users quickly
115went for \LMTX\ instead of \MKIV.
116
117In the code base we avoid dependencies on specific platforms but there are a few
118cases where the code for \MSWINDOWS\ and \UNIX\ differs. However, the
119functionality should be the same. A good test is that for \MSWINDOWS\ we can
120compile with mingw (cross|-|compilation), \MSVC\ (native) and clang (native);
121that order is also the order of runtime performance. The native \MSVC\ binary is
122the smallest but users probably don't care. In any case, it is nice to have a
123fallback plan in place. The code is all in \CCODE; the \METAPOST\ code is
124converted from \CWEB\ into \CCODE\ using a \LUA\ script but we also ship the
125resulting \CCODE\ code. The code base provides a couple of \CMAKE\ files and
126comes with a trivial build script.
127
128When I say that there are no libraries used, I mean external libraries. We do use
129code from elsewhere: adapted \type {avl} as well as \type {decnumber} (for the
130\METAPOST\ library), adapted \type {hjn} (hyphenation), \type {miniz} (zip
131compression), \type {pplib} (for loading \PDF\ files), \type {libcerf} (to
132complement other math library support, but it might be dropped), and \type
133{mimalloc} for memory management. However all the code is in the \LUAMETATEX\
134code base and only updated after checking what changed. The most important
135library originating elsewhere is of course \LUA: we use the latest and greatest
136(currently) 5.4 release. We kept the \type {socket} library but it might be
137dropped or replaced at some point. In addition there is a subsystem for
138dynamically loading libraries; the main reason for that being that I needed \type
139{zint} for barcodes, interfaces to sql databases, a bunch of compression
140libraries, etc. But all that is tagged {\em optional} and \CONTEXT\ will never
141depend on it. There are no consequences for compilation either because we don't
142need the header files. The glue code is very minimalistic and most work gets
143delegated to \LUA.
144
145Initially, because the backend is written in \LUA, there was a drop in
146performance of some 15\percent\ but that was stepwise compensated by gains in
147performance in the engine and additional or improved functionality. The \CONTEXT\
148code base is rather optimized so there was little to gain there, apart from using
149new features. Existing primitive support could also be done a bit more
150efficiently; it helps if one knows where potential bottlenecks are. Therefore, in
151the meantime an \LMTX\ run can be quite a bit faster than a \MKIV\ run and it can
152even outperform a \LUAJITTEX\ run. In practice, the difference between an
153eight|-|bit \MKII\ run using the eight|-|bit \PDFTEX\ engine and a 32|-|bit
154\LUAMETATEX\ run with \LMTX\ can be neglected, definitely on more complex
155documents. I never get complaints about performance from \CONTEXT\ users, so it
156might be a minor concern.
157
158So what are the main differences in the installation? If you really want to
159experience it you should use the standard installation. Currently the small
160installer is the engine that synchronizes the installation over the net and,
161assuming a reasonable internet connection, that takes little time. The
162installation is relatively small, and many of the bytes used are for the
163documentation. Updates are done by transferring only the changed files. The
164\TEXLIVE\ installation is a bit larger because it shares for instance fonts with
165the main installation and these come with resources used by other macro packages.
166Both installations bring \MKIV\ as well as \LMTX\ and therefore provide \LUATEX\
167as well as \LUAMETATEX. However, a \MKIV\ run is now managed by \LUAMETATEX\
168because we use that engine for the runner. The \MKII\ code is no longer in
169\TEXLIVE\ but is in the repositories and used to test and compare with \PDFTEX.
170It just works.
171
172The number of binaries and stubs is reduced to a minimum:
173
174\starttabulate[|T|T||]
175\BC file                           \BC symlink    \BC \NC \NR
176\NC tex/texmf-platform/luametatex  \NC            \NC combined \TEX, \METAPOST\ and \LUA\ engine \NC \NR
177\NC tex/texmf-platform/mtxrun      \NC luametatex \NC script runner, binary \NC \NR
178\NC tex/texmf-platform/context     \NC luametatex \NC CONTEXT\ runner, binary \NC \NR
179\NC tex/texmf-platform/mtxrun.lua  \NC            \NC script runner, lua code \NC \NR
180\NC tex/texmf-platform/context.lua \NC            \NC loader for \CONTEXT\ runner \NC \NR
181\NC tex/texmf-platform/luatex      \NC            \NC the good old ancestor \NC \NR
182\stoptabulate
183
184All of these programs are in the \CONTEXT\ distribution directory \typ
185{tex/texmf-<platform>/}. In addition, \type {context} and \type {mtxrun} are
186symlinks to the \type {luametatex} binary, where possible.
187
188So, the \type {context} command runs \type {luametatex}, but loads the \LUA\ file
189with the same name which in turn will locate the \CONTEXT\ management script
190(\type {mtx-context}) in the \TEX\ tree and run it. The same is true for \type
191{mtxrun}: it is a binary (link) that loads the script in (this time) the same
192path and then can perform numerous tasks. For instance, identifying the installed
193fonts so that they can be accessed by name is done with:
194
195\starttyping
196mtxrun --script font --reload
197\stoptyping
198
199Where in \MKII\ we had stubs for various utility scripts, already in \MKIV\ we
200went for a generic runner and a bit more keying. It's not like these scripts are
201used a lot and by avoiding shortcuts there is also little danger for a mixup with
202the ever|-|growing list of other scripts in \TEXLIVE\ or commands that the
203operating system provides.
204
205The \LUATEX\ binary is optional and only needed if a user also wants to process
206\MKIV\ files. There are no shell scripts used for launching. The two main calls
207used by users are:
208
209\starttyping
210context foo.tex
211context --luatex foo.tex
212\stoptyping
213
214A user has only to make sure that the binaries are in the path specification.
215When you run from an editor, the next command does the work:
216
217\starttyping
218mtxrun --autogenerate --script context <filename>
219\stoptyping
220
221with \type {<filename>} being an editor|-|specific placeholder. Like other
222engines, \LUAMETATEX\ (and \CONTEXT) needs a file database and format file, and
223although it should generate these automatically you can make them with:
224
225\starttyping
226mtxrun --generate
227context --make
228\stoptyping
229
230The rest of the installation is similar to what we always had and is \TDS\
231compliant. The source code of \LUAMETATEX\ is included in the distribution itself
232(which nicely fulfills the requirements) but can also be found at:
233
234\starttyping
235https://github.com/contextgarden/luametatex
236\stoptyping
237
238There are also some optional libraries there but \CONTEXT\ works fine without
239them. The official latest distribution of \CONTEXT\ itself is:
240
241\starttyping
242https://github.com/contextgarden/context
243https://github.com/contextgarden/context-distribution-fonts
244\stoptyping
245
246We see users grab fonts from the Internet and play with them. They can install
247additional fonts in \typ {tex/texmf-fonts/data/<vendor>}. Project|-|specific
248files can be collected in \typ {tex/texmf-project/tex/context/user/<project>}.
249These directories are not touched by installations and can easily be copied or
250shared between different installations. After adding files to the tree \typ
251{mtxrun --generate} will update the file database.
252
253In the distribution there are plenty of documents that describe how \LUAMETATEX\
254with \LMTX\ differs from \MKIV\ with \LUATEX: new primitives, macro extensions,
255more granular math rendering, improved memory management, new (or extended)
256(rendering) concepts, more \METAPOST\ features; most is covered in one way or
257another, and much is already applied in the \CONTEXT\ source code. After all, it
258took a few years before we arrived here so you can expect substantial refactoring
259of the engine as well as the code base, and therefore eventually there is (and
260will be) more than in \MKIV.
261
262When you compare a \CONTEXT\ installation with what is needed for other macro
263packages you will notice a few differences. One concerns the way \TEX\ is
264launched. An engine starts with a blank slate but can be populated with a
265so|-|called format file that is basically a memory dump of a preloaded macro
266package. So, the original way to process a file is to pass a format filename to
267the engine. In order to avoid that a trick is used: when an engine (or
268symlink|/|stub to it) is launched by its format name, the loading happens
269automatically. So, for instance \type {pdflatex} is actually an equivalent for
270starting \PDFTEX\ with the format file \typ {pdflatex.fmt} while \type {latex} is
271\PDFTEX\ with another format file (\typ {latex.fmt}) starting up in \DVI\ mode.
272And, as there are many engines, a specific macro package can have many such
273combinations of its name and engine.
274
275In \CONTEXT\ we don't do it that way. One reason is that we never distinguished
276between backends: \MKII\ uses an abstract backend layer and load driver files at
277runtime (it was one of the reasons why we could support \ACROBAT\ as soon as it
278showed up, because we already supported the now obsolete but quite nice
279\DVIWINDO\ viewer). And that model hasn't changed much as we moved on. Because we
280use a runner, we also don't need to distinguish between engines: all formats have
281the same name but sit on an engine subpath in the \TEX\ tree. Anyway, this
282already removes quite some formats. On the other hand, \CONTEXT\ can be run with
283different language specific user interfaces which means that instead of just
284\type {context.fmt} we have \type {cont-en.fmt} and possibly more, like \type
285{cont-nl.fmt}. So that can increase the number again but by default only the
286English interface is installed. As a side note: where with \MKII\ we needed to
287generate \METAPOST\ mem files, with its descendants having \MPLIB\ we load the
288(actually quite a bit of) \METAPOST\ code at runtime. \footnote {Occasionally I
289do experiments with loading the \TEX\ format code at runtime, but at this moment
290the difference in startup time of about one second (assuming files are cached) is
291too large and running over networks will be less fun, so the format file will
292stay. The time involved in loading \METAPOST\ can be brought down but for now I
293leave it as it is.}
294
295In addition to a format file, for the \LUATEX\ and \LUAMETATEX\ engine we also
296have a (small) \LUA\ loader alongside the format file. All this is handled by the
297runner, also because we provide extensive command line features, and therefore of
298no concern to users and package maintainers. However, it does make integrating
299\CONTEXT\ in for instance \TEXLIVE\ different from other macro packages and
300thereby puts an extra burden on the \TEXLIVE\ team. Here I want to thank the team
301for making it possible to move forward this way, in spite of this rather
302different approach. Hopefully a \LUAMETATEX\ integration is a bit easier in the
303long run because we no longer have different stubs per platform and at least the
304binary part now has no dependencies and only has a handful of files.
305
306For those new to \CONTEXT\ or those who want to try it in \TEXLIVE\ 2023 there is
307not much difference between the versions. However, \MKIV\ is now frozen and new
308functionality only gets added to \LMTX. Of course we could backport some but with
309most users already having moved on, it makes no sense. Just as we keep \MKII\
310around for testing with \PDFTEX, we also keep \MKIV\ alive for testing with
311\LUATEX. Maybe in a couple of years \MKIV\ will go the same route as \MKII:
312ending up in the archives as an optional installation. \footnote {This text
313appeared in \TUGBOAT\ around the 2023 \TEXLIVE\ release. Thanks to Karl Berry for
314his careful reading and fixing of the text and of course for keeping \TEXLIVE\
315alive.}
316
317\stopchapter
318
319\stopcomponent
320