followingup-retrospect.tex /size: 11 Kb    last modification: 2021-10-28 13:50
1% language=us runpath=texruns:manuals/followingup
2
3\startcomponent followingup-retrospect
4
5\environment followingup-style
6
7\startchapter[title={Retrospect}]
8
9% \startsection[title={Introduction}]
10% \stopsection
11
12At some point in a new development, and \LUAMETATEX\ feels like that, there comes
13a moment when you need to make a decision. In this case the question is if we
14need to make hybrid \MKIV\ and \LMTX\ files or do the same as with the transition
15from \MKII\ to \MKIV: use two variants. For \TEX\ files a conditional section has
16only overhead in the format generation as skipped code doesn't end up in the
17format. With conditional \LUA\ code it's different: the ignored section is still
18present in byte code. But even for \TEX\ code a conditional section is not
19entirely invisible: encountered control sequences are still creating (bogus) hash
20entries. So the question is: do we go lean and mean and do we omit historic
21non|-|\LMTX\ code?
22
23A comparison with the transition from \MKII\ is actually relevant. For instance
24right from the start \CONTEXT\ had an abstract backend layer, and support for
25engines and output formats was loaded on demand. There was never any specific
26code in the core. With \MKIV\ we changed the model but there is still some
27abstraction.
28
29In \MKII\ we also had to deal with encodings and that has consequences for
30font handling, language support and input encodings. In \MKIV\ all that changed:
31internal all is \UTF, as is normally the input (but we can still use encodings),
32and fonts are always mapped to \UNICODE.
33
34Anyhow, much that made sense for \MKII\ was no longer relevant for \MKIV: code
35could be dropped. But some mechanisms were reimplemented using \LUA: code was
36added. The user interface stayed the same but in \MKIV\ uses a conceptually
37different approach deep down. Therefore the code base was split in \MKII\ and
38\MKIV\ files but this transition was made stepwise.
39
40So should the same happen with \LMTX ? There is not that much that needs to be
41added to \MKIV\ in terms of functionality. In the end, for the \TEX\ code the
42differences are not that substantial, so there we can consider loading different
43files. The files involved are rather stable so there is not much danger of
44functionality between \MKIV\ and \LMTX\ getting out of sync. The same is true for
45the \LUA\ files, although synchronization is probably more an issue there.
46
47Another option is to always assume that \LUAMETATEX\ is used. For testing regular
48\LUATEX\ (patches) we can just use a 2019 stable \CONTEXT. But in order for users
49to benefit from developments we then expect them all to move on to \LMTX. Using a
50frozen 2019 version with upcoming \LUATEX\ is no big deal as we've done the same
51with \MKII\ and that worked out okay.
52
53When we started with \CONTEXT\ development in the previous century we were doing
54pretty weird things. I remember getting comments that what we did made no sense
55because it was not what \TEX\ was meant for and some even suggested that it
56disrupted the picture. Highly structured input, a clear separation (and
57abstraction) of front and backend, inheritance and user defined styling,
58integrated support for \XML, embedded \METAPOST, advanced interactive documents,
59handling of fonts en encodings, the list is long. Occasionally some of the things
60that came with \CONTEXT\ were ridiculed, like the fact that a script was used to
61manage the (multiple) run(s), but in the end, look at how many script are around
62now. Some even wondered why we used \TEX\ at all because \TEX\ was meant for
63typesetting math. And who needs \XML\ let alone \MATHML ? Or interactive \PDF\
64features? Much in \CONTEXT\ and its management got smoother over time and the
65\LUAMETATEX\ engine fits nicely into this evolution. It's hard to keep the
66cutting edge but at least we have the instruments.
67
68During \BACHOTEX\ 2019 (end of April, beginning of May) this project was
69presented the first time outside the \CONTEXT\ community. During that meeting
70Mojca Miklavec, one of the driving forces behind \CONTEXT, upgraded the compile
71farm that already was used to compile (intermediate versions of) \LUATEX\ and
72\TEXLIVE\ to also compile \type {pplib} (handy for development) and \LUAMETATEX.
73This permits us to fine|-|tune the \type {cmake} setup which is still work in
74progress. And, also further improvements take place in the code base itself.
75
76One of the properties of open source is that one can build upon an existing code
77base, so when at \BACHOTEX\ Arthur announced that he was going to make a merge of
78\XETEX\ (which he maintains) and \LUATEX\ no one was surprised. But it could be a
79strong argument for a rather strict code freeze: spin|-|offs need stability. I've
80been told that there are now several projects where more libraries (like
81Harfbuzz) get integrated. Those cases don't influence the parent but here
82stability of the original also is expected, unless of course additional features
83go in these engines, which itself creates instability, but that's another matter.
84One could actually argue that the arrival of variants defeats the argument that
85stability is important: if a macro package uses new features, it needs to adapt,
86and naturally (temporary) issues might show up. Such are the dynamics of todays
87software development. History in general shows that not that much is persistent
88(or even accumulative) and programs are probably the least, so maybe the whole
89stability aspect has lost its relevance. \footnote {In a similar way as that the
90argument \quotation {Publishers want this or that, so we as \TEX\ community need
91to provide it.} is no longer that relevant because publishing is now more a
92business model than vocation.} Of course \LUAMETATEX\ is also a follow up, but
93one of the ideas behind it was that I could use it as platform for (independent)
94experiments that could result in code being put into \LUATEX. Also, the changes
95have a limited impact: only \CONTEXT\ will be affected. \footnote {So maybe, in
96the end, stability boils down to \quotation {The engine behaves the same and the
97\CONTEXT\ that comes with it exploits its features as good as possible}.}
98
99It is not feasible to make \CONTEXT\ work with all kind of engines that in
100practice are not used by its users. For instance, after \XETEX\ showed up it went
101through several iterations or font rendering, so we never really spent time on
102the low level features that it provided (there was no demand anyway). One cannot
103simply claim that one method is better than another that replaces it and expect
104constant adaptation (probably for the sake of a few potential users). There
105simply is no \quote {best} engine and no \quote {perfect} solution. Another
106aspect is that when we would adapt \CONTEXT\ to \LUATEX\ variants the
107dependencies on specific functionality that itself depends on the outside world
108is kind of unavoidable. Especially languages and fonts are fluid and for the
109average user there is not that much difference in that department. Should we
110really complicate matters for a few (potential) users? In \CONTEXT\ support like
111that is added on demand, driven by specific needs of users who use \TEX\ for a
112reason and are willing to test.
113
114There's enough huge and complex software around that demonstrates what happens
115when programs are extended, keep growing, their code base becoming more complex.
116Such a process doesn't really fit in my ideas about for \TEX. We positioned 1.10
117as long term stable, with the option to add a few handy things in the long run.
118For sure there are niches to fill and it is a fact that the \TEX\ community can
119deal with variants of engines: just look at the different \CJK\ engines around,
120with prefixes like \type {p}, \type {up}, \type {ep}, etc. But the question is,
121where does that put further \LUATEX\ development? And, more important, what
122consequences does it have for the \CONTEXT\ code base?
123
124The reason I mention this is that I had in mind to eventually backport features
125that work out well in \LUAMETATEX. I also mentioned that in order to support
126stock \LUATEX\ it made no sense to split the \CONTEXT\ code base. After all, a
127few conditional sections could deal with the difference between \LUATEX\ and
128\LUAMETATEX: some differences could be temporary anyway. But, given recent
129developments it actually made sense to split the code base: why spent time on
130backporting when the engine user base is spread over different spinoffs. I can
131better just assume \CONTEXT\ to exclusively use \LUAMETATEX\ and that other macro
132packages use (one or more) \LUATEX\ variants. I can then keep the generic code up
133to date and maybe occasionally add some proven stable features. It is also no big
134deal to keep the minimum subset needed for (plain) font handling compatible,
135assuming \LUATEX\ compatibility, as in the end that engine is the benchmark,
136especially when I strip it a bit from features not needed outside \CONTEXT.
137
138Thoughts like this show how fragile plans and predictions are: within a year one
139has to adapt ideas and assumptions. But it also proves that \LUAMETATEX\ was a
140good choice for \CONTEXT, especially because it is bound to \CONTEXT\
141development, which keep the users independent and isolated from developments that
142don't mind that much the (side) effects on \CONTEXT.
143
144% \footnote {I mentioned stability a few times, but this aspect is somewhat vague:
145% often I see complaints about \LUATEX, or comparisons with other engines, that
146% have nothing to do with the engine per se, but more with misunderstanding and|/|
147% assumptions, strange usage, maybe or even likely bad user code, comparing apples
148% and pears, etc. The term \type {bug} is very popular and often a preferred
149% qualifications, and it sounds even more impressive when it's qualified as a bug
150% one. I guess that a more tight coupling between specific engines and macro
151% packages at least that aspect becomes cleaner.}
152
153Around the \CONTEXT\ meeting (or maybe a bit later) we hope to have the new
154installation infrastructure stable too (currently it is also experimental). By
155that time it will also be clear how we will proceed with the \LMTX\ project. In
156the meantime I have decided so put \LUAMETATEX\ specific files alongside the
157\MKIV\ files, simply because I always need to be able run stock \LUATEX. In order
158to show the close relationship these files are flagged as \MKXL, so we bump from
159\quote {Mark Four} to \quote {Mark Fourty}. The suffixes \type {mkiv}, \type
160{mkvi} and \type {mpiv} get company from \type {mkxl}, \type {mklx} and \type
161{mpxl}. Depending on backporting features, files can come and go. I'm not yet
162sure about the \LUA\ files but the \type {lmt} suffix is already reserved for
163future use. \footnote {This is because \LUA\ 5.4 introduces some new syntax
164elements and where we can get away with the difference between 5.2 (\LUAJITTEX)
165and 5.3 (\LUATEX) such a syntax change is more drastic.} All this is also driven
166by (user) demand.
167
168Consider this (and these thoughts) a snapshot. There will be the usual reports on
169experiments and developments. And in due time there will also be a manual for
170\LUAMETATEX. \footnote {In fact it already lives on my machine but I'm not in
171ready yet for the usual complaints about manuals, so I'm not in that much of a
172hurry.} And yes, at some point I have to make up my mind with respect to
173backporting features that have proven to be useful.
174
175% \footnote {Actually, it seems to come with the Internet: folks wining on whatever
176% platform about lack of documentation (most of the \CONTEXT\ distribution actually
177% is documentation and quite some articles are, have been, and will be written) or
178% possible bug (always huge, even if no bug at all) without exposing much actual
179% research or knowledge about these matters. Write, post and shout before thinking
180% it through, increase the number hits on your profile. It's for sure a way to make
181% something end up at the bottom of my to do list, if at all. A valid response
182% could be: whatever did you contribute to the community that I myself (or
183% \CONTEXT\ users) can benefit from. Quite likely: nothing (or little)! It looks
184% like even the normally friendly \TEX\ community sometimes gets infected by this.}
185
186\stopchapter
187
188\stopcomponent
189