followingup-stubs.tex /size: 10 Kb    last modification: 2021-10-28 13:50
1% language=us runpath=texruns:manuals/followingup
2
3\startcomponent followingup-stubs
4
5\environment followingup-style
6
7\startchapter[title={Stubs}]
8
9\startsection[title={Bare bone}]
10
11The most barebone way to process a \CONTEXT\ file is something like:
12
13\starttyping
14luametatex
15    --fmt="<cache path to>/luametatex/cont-en"
16    --lua="<cache path to>/luametatex/cont-en.lui"
17    --jobname="article"
18    "cont-yes.mkiv"
19\stoptyping
20
21We pas extra options, like:
22
23\starttyping
24    --c:autopdf
25    --c:currentrun=1
26    --c:fulljobname="./article.tex"
27    --c:input="./article.tex"
28    --c:kindofrun=1
29    --c:maxnofruns=9
30    --c:texmfbinpath="c:/data/develop/tex-context/tex/texmf-win64/bin"
31\stoptyping
32
33but for what we are going to discuss here it doesn't really matter. The main point is
34that we use a \LUA\ startup file. That one has a minimal amount of code so that the
35format can be loaded as we like it. For instance we need to start up with initial
36memory settings.
37
38The file \type {cont-yes} sets up the way processing content happens. This can be the
39\type {jobname} file but also something different. It is enough to know that this
40startup is quite controlled.
41
42I will explore a different approach to format loading but for now this is how it
43goes. After al, we need to be compatible with \LUATEX\ and normal \MKIV\ runs, at
44least for now.
45
46\stopsection
47
48\startsection[title={Management (some history)}]
49
50In \CONTEXT\ we always had a script: \type {texexec}, originally a \MODULA2
51program, later a \PERL\ script, then a \RUBY\ script but now we have \type
52{mtxrun}, a \LUA\ script. All take care of making sure that the file is
53processed enough times to get the cross references, tables of contents, indexes,
54multi|-|pass data stable. It also makes it possible to avoid using these special
55binaries (or links) that trick the engine into thinking it is bound to a format:
56we never had \type {pdfcontext} or \type {luacontext}, just one \type {context}.
57Actually, because we have multiple user interfaces, we would have needed many
58stubs instead. Getting this approach accepted was not easy but in the meantime
59I've seen management scripts for other packages being mentioned occasionally.
60
61The same is true for scripts: for a long time \CONTEXT\ came with quite some
62scripts but when an average \TEX\ distribution started growing, including many
63other scripts, we abandoned this approach and stuck to one management script that
64also launched auxiliary scripts. That way we could be sure that there were no
65clashes in names. If you look at a full \TEX\ installation you see many stubs to
66scripts and more keep coming. How that can work out well without unexpected side
67effects (name clashes) is not entirely clear to me, as a modern computer can have
68large bin paths. Just imagine that all large programs (or ecosystems) would
69introduce hundreds of new \quote {binaries}.
70
71Anyway, in the end a \CONTEXT\ installation using \MKIV\ only needs \type {mtxrun}
72and as bonus \type {context}. The above call is triggered by:
73
74\starttyping
75mtxrun --autogenerate --script context --autopdf article.tex
76\stoptyping
77
78from the editor. Here we create formats when none is found, and start or activate
79the \PDF\ viewer afterwards, so more minimal is:
80
81\starttyping
82mtxrun --script context article.tex
83\stoptyping
84
85Normally there is also a \type {context} stub so this also works:
86
87\starttyping
88context article.tex
89\stoptyping
90
91\stopsection
92
93\startsection[title={The launch process (more history)}]
94
95In \MKII, when we use \PDFTEX, the actual launch of these script is somewhat
96complex and a bit different per platform. But, on all platforms \KPSE\ does the
97lookup of the script. Already long ago I found out that this startup overhead
98could amount to seconds on a complete \TEX Live installation (imagine running
99over a network) which is why eventually we came up with the minimals. The reason
100is that the file databases have to be loaded: first for looking up, then for the
101stub that also needs that information and finally by the actual program. There
102were no \SSD's then.
103
104The first hurdle we took was to combine the lookup and the runner. Of course this
105is sort of out of our control because an installer can decide to still use a
106lookup approach but at least on \MSWINDOWS\ this was achieved quite easy. Sort
107of:
108
109\starttyping
110texexex -> [lookup] -->
111    texexec.pl -> [lookup] ->
112        pdftex + formats ->
113            [lookup] -> processing
114\stoptyping
115
116The first lookup can be avoided by some fast relative lookup, but for more
117complex management the second one is always there. Over time this mechanism
118became more sophisticated, for instance we use caching, could work over sockets
119using a \KPSE\ server, etc.
120
121When \LUATEX\ came around, it was already decided early that it also would serve
122as script engine for the \CONTEXT\ runner, this time \type {mtxrun}. The way this
123works differs per platform. On \WINDOWS\ there is a small binary, say \type
124{runner.exe}. It gets two copies: \type {mtxrun.exe} and \type {context.exe}. If
125you find more copies on your system, something might be wrong with your
126installation.
127
128\starttyping
129mtxrun.exe  -> loads mtxrun.lua in same path
130context.exe -> idem but runs with --script=context
131\stoptyping
132
133The \type {mtxrun.lua} script will load its file database which is very efficient
134and fast. It will then load the given script and execute it. In the case of \type
135{context.exe} the \type {mtx-context.lua} script is loaded, which lives in the
136normal place in the \TEX\ tree (alongside other scripts).
137
138So, a minimal amount of programs and scripts is then:
139
140\starttyping
141texmf-win64/bin/luatex.exe
142texmf-win64/bin/mtxrun.exe
143texmf-win64/bin/mtxrun.lua
144texmf-win64/bin/context.exe
145\stoptyping
146
147with (we also need to font manager):
148
149\starttyping
150texmf-context/scripts/context/lua/mtx-context.lua
151texmf-context/scripts/context/lua/mtx-fonts.lua
152\stoptyping
153
154But \unknown\ there is a catch here: \LUATEX\ has to be started in script mode in
155order to process \type {mtxrun}. So, in fact we see this in distributions.
156
157\starttyping
158texmf-win64/bin/luatex.exe
159texmf-win64/bin/texlua.exe
160texmf-win64/bin/mtxrun.exe
161texmf-win64/bin/mtxrun.lua
162texmf-win64/bin/context.exe
163\stoptyping
164
165The \type {texlua} program is just a copy of \type {luatex} that by its name
166knows that is is supposed to run scripts and not process \TEX\ files. The setup
167can be different using dynamic libraries (more files but a shared engine part)
168but the principles are the same. Nowadays the stub doesn't need the \type
169{texlua.exe} binary any more, so this is the real setup:
170
171\starttyping
172texmf-win64/bin/luatex.exe       large program
173texmf-win64/bin/mtxrun.exe       small program
174texmf-win64/bin/mtxrun.lua       large lua file
175texmf-win64/bin/context.exe      small program
176\stoptyping
177
178Just for the record: we cannot really use batch files here because we need to
179know the original command, and when run from a script that is normally not known.
180It works to some extend but for instance when started indirectly from an editor
181it can fail, depending on how that editor is calling programs. Therefore the stub
182is the most robust method.
183
184On a \UNIX\ system the situation differs:
185
186\starttyping
187texmf-linux-64/bin/luatex        large program
188texmf-linux-64/bin/texlua        symlink to luatex
189texmf-linux-64/bin/mtxrun        large lua file
190texmf-linux-64/bin/context       shell script that starts mtxrun
191\stoptyping
192
193Here \type {mtxrun.lua} is renamed to \type {mtxrun} with a shebang line that
194triggers loading by \type {texlua} which is a symlink to \type {luatex} because
195shebang lines don't support the \type {--texlua} argument. As on windows, this
196is not really pretty.
197
198\stopsection
199
200\startsection[title={The \LMTX\ way (the present)}]
201
202Now when we move to \LMTX\ we need to make sure that the method that we choose is
203acceptable for distributions but also nicely consistent over platforms. We only
204have one binary \type {luametatex} with all messy logic removed and no second
205face like \type {metaluatex}. When it is copied to another instance (or linked)
206it will load the script with its own name when it finds one. So on \WINDOWS\ we
207now have:
208
209\starttyping
210texmf-win64/bin/luametatex.exe   medium program
211texmf-win64/bin/mtxrun.exe       copy (or link) of luametatex
212texmf-win64/bin/mtxrun.lua       large lua file
213texmf-win64/bin/context.exe      copy (or link) of luametatex
214texmf-win64/bin/context.lua      small lua file
215\stoptyping
216
217and in \UNIX:
218
219\starttyping
220texmf-linux-64/bin/luametatex    mediumprogram
221texmf-linux-64/bin/mtxrun        copy (or link) of luametatex
222texmf-linux-64/bin/mtxrun.lua    large lua file
223texmf-linux-64/bin/context       copy (or link) of luametatex
224texmf-linux-64/bin/context.lua   small lua file
225\stoptyping
226
227So, \type {luametatex[.exe]}, \type {mtxrun[.exe]} and \type {context[.exe]} are
228all the same. On both platforms there is \type {mtxrun.lua} (with suffix) and on
229both we also use the same runner approach. The \type {context.lua} script is
230really small and just sets the script command line argument before loading \type
231{mtxrun.lua} from the same path. In the case of copied binaries: keep in mind
232that the three copies together are not (much) larger than the \type {luatex} and
233\type {texlua} pair (especially when you take additional libraries into account).
234
235The disadvantage of using copies is that one can forget to copy with an update,
236but the fact that one can use them might be easier for installers. It's up to
237those who create the installers.
238
239One complication is that the \type {mtxrun.lua} script has to deal with the old
240and the new setup. But, when we release we will assume that one used either
241\LUATEX\ or \LUAMETATEX, not some mix. As \type {mtxrun} and \type {context} know
242what got it started they will then trigger the right engine, unless one passes
243\typ {--engine=luatex}. In that case the \LUAMETATEX\ launcher will trigger a
244\LUATEX\ run. But a mixed installation is unlikely to happen.
245
246\stopsection
247
248\startsection[title={Why not \unknown}]
249
250Technically we could use one call for both the runner and \TEX\ processor but
251when multiple runs are needed this would demand an internal engine reset as well
252as macro package reset while keeping some (multi|-|pass) data around. A way
253in|-|between could be to spawn the next run. In the end the gain would be minimal
254(we have now .2 seconds overhead per total run, which can trigger multiple
255passes, due to the management script, to basically we can neglect it. (Triggering
256the viewer takes more time.)
257
258\stopsection
259
260\stopchapter
261
262\stopcomponent
263