Source Browser ?

1% language=us runpath=texruns:manuals/musings
2
3\environment musings-style
4
5\startcomponent musings-stability
6
7\startchapter[title={Stability}]
8
9\startsubject[title=Introduction]
10
11How stable is \CONTEXT ? This question is hard to answer. For instance \MKII\
12hasn't changed for years and seems to work quite well: no changes equals
13stability. Those who use it can do with what it offers. The potentially sensitive
14dependencies on for instance fonts are probably absent because there is not much
15development in the 8 bit fonts arena. As long as these are available we're okay,
16in fact, \OPENTYPE\ fonts are more a moving target and therefore less stable.
17
18What do we mean by stable? The fundamental differences between an 8 bit engine
19(and fonts) and an \UNICODE\ aware engine able to handle \OPENTYPE\ fonts is
20substantial which is why we dropped some functionality and added some relevant
21new. One can consider that a problem but in practice using fonts has become
22easier so no one is hurt by it. Here we need to keep in mind that \PDFTEX\ is
23really stable: it uses fonts and technology that doesn't change. On the other
24hand \XETEX\ and \LUATEX\ follow new trends. Thereby \XETEX\ uses libraries,
25which introduces a dependency and instability, while \LUATEX\ assumes solutions
26in \LUA\ which means that users and macro writers can tweak and thereby also
27introduce instability (but at least one can adapt that code).
28
29Due to the way the user interface is set up, it is unlikely that \CONTEXT\ will
30change. But the fact that we now have \LUA\ available means that many commands
31have been touched. Most behave compatible, some have more functionality, and of
32course we have a \LUA\ interface. We include a lot of support code which also
33lessens dependencies.
34
35The user input is normally \TEX\ but when you use \XML\ the move to \MKIV\ meant
36that we dropped the \MKII\ way of dealing with it in favour of a completely new
37mechanism. I get the impression that those using \XML\ don't regret that change.
38Talking of stability the \MKIV\ \XML\ interface is typically a mechanism that is
39stable and might change little. We can add new trickery but the old stays as it
40is.
41
42If we look at the output, there is \DVI\ and \PDF. In \MKII\ the \DVI\ could
43become \POSTSCRIPT. As there are different \DVI\ post|-|processors the backend
44code was using a plug|-|in model. Contrary to other macro packages there was only
45one so called format that could adapt itself to the required (engine specific)
46output. A \CONTEXT\ run has always been managed by a wrapper so users were not
47bothered much by what \TEX\ engine they used and|/|or what backend was triggered.
48This changed with \MKIV\ where we use just \LUATEX, always produce \PDF\ and
49optionally can export \XML. But again the run is managed by a wrapper, which
50incidentally is written in \LUA\ and thereby avoids dependencies on for instance
51\PERL, \RUBY\ or \PYTHON, which are moving targets, use libraries and additional
52user code, and thereby are potentially instable too.
53
54The \PDF\ code that is produced is a mix of what the engine spits out and what
55the macro package injects. The code is normally rather simple. This means that
56it's no big deal to support the so called standards. It also means that we can
57support advanced interactivity and other features but these also depends on the
58viewers used. So, stability here is more fluent, for instance because the \PDF\
59standard evolves and|/|or we need to adapt to viewers. Special demands like
60tagged \PDF\ have been supported right from the start but how that evolves
61depends mostly on input from users who need it. Again, that is less important
62(and crucial) for stability than the rendering capabilities.
63
64The fact that we use \LUA\ creates a dependency on that language but the reason
65that we use it is {\em because} it is so stable. We follow the updates and so far
66that worked out well. Now, say that we had a frozen version of \CONTEXT\ 2010 and
67\LUATEX\ 1.09 that uses \LUA\ 5.3, would that work? First of all, in 2010
68\LUATEX\ itself was evolving so the answer is probably \quotation {no}, unless
69one adds a few compatibility patches. I'm not going to try it. The change from
705.1 to 5.2 to 5.3 was not really a problem I think and the few issues could be
71dealt with easily. If you want long term stability and use a lot of \LUA\ code
72you can take it into account when coding. Avoiding external libraries is a good
73start.
74
75Fonts are more than before moving targets. So, if you want stability there you
76should save them with your document source. The processing of them has evolved
77and has been improved over time. By now it's rather stable. More recent code can
78catch more issues and fixes are relatively easy. But it's an area that you always
79need to check when you update an old distribution. The same is true for language
80related hyphenation patterns and script specific support. The community is no
81longer leading in the math department either (\OPENTYPE\ math is a \MICROSOFT\
82invention). But, the good news is that the \TEX\ ecosystem is always fast to
83adapt and can also often provide more functionality.
84
85Vertical spacing, in fact spacing in general is an aera that can always be
86improved, so there is where you can expect changed. The same is true for side
87floats or mechanisms where content is somehow attached to other moving content,
88for instance marginal notes.
89
90But code dealing with fonts, color, scripts, structure, and specific features
91that once written don't need more, will not change that much. As mentioned for
92fonts, like any resource, we also depend on third parties. Colors can relate to
93standards, but their main properties are unchanged. Support for specific scripts
94can (and will) be improved due to user input and demands so there the users also
95influence stability. Structure doesn't really influence the overall rendering,
96but the way you set it up does, but that's user styling. Of course during the
97transition from \MKII\ to \MKIV\ and the evolution of \LUATEX\ things could be
98broken, but fixing something structural seldom relates to rendering. If for
99instance we improve the interpretation of \BIBTEX\ input , which can be real
100messy, that involves data processing, nor rendering. When we improve support for
101the \APA\ standard, which is complex, it might involve rendering but then that's
102asked for and expected. One cannot do better than the input permits.
103
104\stopsubject
105
106\startsubject[title=Publishers]
107
108When discussing stability and especially stability as requirement we need to look
109at the way \CONTEXT\ is used. So let's look at a few scenarios. Say that a
110publisher gets a camera ready book from an author in \PDF\ format. In that
111case the author can do all tweaks needed. Now say that the publisher also wants
112the source code in a format that makes reuse possible.
113
114But let's face reality. Will that publisher really reformat the document in \PDF\
115again? It's very unlikely. First of all the original \PDF\ can be kept, and
116second, a reformat only makes sense after updating the content or going for a
117completely different layout. It's basically a new book then. In that case literal
118similarity of output is irrelevant. It is a cheap demand without much substance.
119
120When the source is used for a different purpose the tool used to make the \PDF\
121is irrelevant. In that case the coding of the source can matter. If it is in some
122dialect of \TEX, fine, one has to convert it anyway (to suit the other usage). If
123there is an \XML\ export available, fine too as it can be transformed, given that
124the structure is rich enough, something that is unlikely to have been checked
125when the original was archived. Then there could have been the demand for a
126document in some other format and who can guarantee stability of the tools used
127there? Just look at how \MICROSOFT\ Word evolved, or for that matter, its
128competitors. On the average \TEX\ is more stable as one can snapshot a \TEX\ tree
129and run binaries for years, if needed, in a virtual machine.
130
131So, I don't think that a publisher is of any relevance in the discussion about
132stability. Even if we can clearly define what a publisher is, I doubt if
133publishers themselves can be considered long term stable organizations. Not
134today. I'm not sure if (especially the large) publishers really deserve a place
135in the discussion about stability but I'm willing to discuss that when I run into
136one.
137
138The main problem that an author can face when being confronted with the stability
139issue this way is that the times are long gone that publishers have a clue about
140what \TEX\ is, how it evolved and how it always had to and did adapt to changing
141requirements. If you're lucky you will run into someone who does know all this.
142They're normally a bit older and have seen the organization from any angles and
143therefore are fun to work with.
144
145But even then, rendering issues are often not high on their agenda. Outsourcing
146often has become the modus operandi which basically brings us to the second group
147involved in this discussion: suppliers.
148
149\stopsubject
150
151\startsubject[title=Suppliers]
152
153I don't know many suppliers other than the ones we ran into over a few decades.
154At least where I live the departments that are responsible for outsourcing
155typesetting like to deal with only a few large suppliers, interestingly because
156they assume that they are stable. However, in my experience hardly any of those
157seem to have survived. (Of course one can wonder if long term commitment really
158is that important in a world where companies change so fast.) This is somewhat
159obscured by the fact that publishers themselves merge, reorganize, move people
160around, etc. so who can check on the stability of suppliers. It is definitely a
161fact that at least recently hardly any of them played a rol of any relevance in
162the development of stable tools. In the past the membership of \TEX\ user groups
163contained people working at publishers and suppliers but that has changed.
164
165Let's focus on the suppliers that somehow use \TEX\ and let's consider two kind
166of suppliers: small ones, one were only a few people work, and large ones. The
167small ones depend on stable \TEX\ distributions, like \TEX Live where they can
168get the resources from: styles, fonts, patterns, binaries. If they get the
169authors \TEX\ files they need to have that access. They have to rework that input
170into what the customer demands and that likely involves tweaks. So, maybe they
171have developed their own additional code. For that code, stability is their own
172responsibility. Did they tweak core code of a macro package? Fine, but you might
173have it coming when you update. You cannot expect the evolving free meal world to
174stick to your commercial needs. A supplier can play safe and somehow involve the
175developers of macro packages or consult them occasionally, but does that really
176happen often? Interesting is that a few times that I was asked for input it was
177also wrapped in obscurity, as if some holy grail of styling was involved, while
178it's quite likely that the developer of a macro package can write such a style
179(or extra code) easily and probably also better. There really is not that much
180unique code around.
181
182Small suppliers can be on mailing lists where they can contribute, get feedback,
183provide testing, etc. They are part of a process and as such have some influence
184on stability. If they charge by the page, then a change in their tools can be
185reflected in what they charge. Basically redoing a book (or so) after a decade is
186doing a new job. And adapting to some new options in a package, as part of a
187typesetting job is probably no big deal. Is commercial really more stable than
188open source free software? Probably not, except from open source software
189developers whose real objective is to eventually sell their stuff to some company
190(and cash) and even accept it to be ditched. Small suppliers are more flexible.
191
192The large suppliers are a different group. They often guard their secrets and
193stay in the dark. They probably seldom share (fundamental) code and information.
194If they are present in a community it can be for marketing reasons. If at some
195point a large supplier would demand stability, then my first response would be:
196sure I can make you a stable setup and maybe even provide intermediate patches
197but put your money where your mouth is. But that never happened and I've come to
198the conclusion that we can safely ignore that group. The \TEX\ user groups create
199distributions and have for instance funded font development and it are the common
200users who paid for that, not the scale ones. To some extent this is actually good
201because large (software related) organizations often have special agendas that
202can contradict what we aim at in the long term.
203
204From the authors perspective there is a dilemma here. When you submit to a
205publisher who outsources, it can be a demand to deliver in a specific \TEX\
206format. Often a \PDF\ comes with the source then, so that the intended rendering
207is known. Then that source goes to a supplier who then (quite likely) redoes a
208lot of the coding in some stable subset, maybe even in a very old version of the
209macro package. If I were such an author I'd render the document in \quote {as
210stupid as possible mode} because you gain nothing by spending time on the looks.
211So, stability within the package that you use is easy and translation from one to
212another probably also. It's best to check beforehand what will happen with your
213source and let stability, if mentioned, be their problem. After all they get paid
214for it.
215
216Suppliers seldom know \CONTEXT. An interesting question is if they really know
217the alternatives well, apart from the bit they use. A well structured \CONTEXT\
218source (or probably any source) is often easy to convert to another format. You
219can assume that a supplier has tools for that (although we're often surprised
220about the poor quality of tools used). Often the strict demand for some kind of
221format is an excuse for lack of knowledge. Unfortunately you need a large author
222base to change that attitude.
223
224\stopsubject
225
226\startsubject[title=Authors]
227
228Before we move to some variants of the above, first I will look at stability from
229the authors perspective. When a book is being written the typesetting more or
230less happens as part of the process. The way it looks can influence the way you
231write and vise versa. Once the book is done it can go in print and, unless you
232were using beta versions of \CONTEXT\ and updated frequently. Normally you will
233try to work in a stable setup. Of course when a user asks for additional features
234while working on a project, he or she should also accept other beta features
235and side effects.
236
237After a few years an author might decide to update the book. The worst that can
238happen is that the code doesn't run with the latest \CONTEXT. This is not so
239likely because commands are upward compatible. However, the text might come out a
240bit different, for instance because different fonts or patterns are used. But on
241the average paragraphs will come out the same in \TEX. You can encounter
242differences in the vertical spacing and page breaks, because that is where
243improvements are still possible. If you use conceptually and implementation wise
244complex mechanism like side floats, you can also run into compatibility issues.
245But all these don't really matter much because the text will be updated anyway
246and fine|-|tuning of page breaks (if at all) happens at the end. The more you try
247to compete with desk top publishing, and the more tweaks you apply, the greater
248is the risk that you introduce instability. It is okay for a one|-|time job, but
249when you come back to it after a decade, be prepared for surprises.
250
251Even if you stick to the original coding, it makes sense to sacrifice some of that
252stability if new mechanisms have become available. For instance, if you use
253\METAPOST, better ways to solve your problem might have become available. Or if
254you document is 15 years old, a move from \MKII\ to \MKIV\ is a valid option,
255in which case you might also consider using the latest fonts.
256
257Of course, when you made a style where you patched core code, you can expect
258problems, because anything not explicitly mentioned in the interface definition
259files is subjected to change. But you probably see that coming anyway.
260
261So, is an author (or stand alone user) really dependent on stability? Probably
262less than thought. In fact, the operating system, internet and browsers,
263additional tools: all change over time and one adapts. It's something one can
264live with. Just see how people adapt to phones, tablets, social media, electric
265cars, etc. As long as the document processes and reasonable output is generated
266it's fine. And that is always what we aim at! After all we need to be able to use
267it ourselves, don't we?
268
269\stopsubject
270
271\startsubject[title=Projects]
272
273Although it is often overlooked as valid alternative in rendering in large scale
274projects, \CONTEXT\ is perfect as component in a larger whole. Something goes
275in, something comes out. In a long term project one can just install a minimal
276distribution, write styles, and run it for ages. Use a virtual machine and
277we're talking decades without any change. And, when one updates, it's easy to check
278if all still works. Often the demands and styles are simple and predictable. It's
279way more likely that a hard coded solution in some large programming environment has
280stability issues than that the \CONTEXT\ bit has.
281
282If \CONTEXT\ is used in for instance documentation of (say) software, again there
283is no real issue. Such documents are simple, evolve and therefore have no stable
284page flow, and updating \CONTEXT\ is not needed if the once decided upon coding
285is stable. You don't need the latest features. We've written styles and setups for
286such tasks and indeed they run for ages.
287
288It can make me smile to see how much effort sometimes goes in low quality
289rendering where \CONTEXT\ could do a way better job with far less investment in
290time and money but where using some presumed stable toolkit is used instead, one
291that comes with expensive licensing, from companies that come and go but shine in
292marketing. (A valid question is to what extent the quality of and care for
293documentation reflects the core products that a company produces, at least under
294the hood.)
295
296The biggest hurdle in setting up a decent efficient workflow is that it has to be
297seen as a project: proper analysis, proper planning, prototyping and testing,
298etc. You invest first and gain later. When dealing with paper many publishers
299still think in price per page and have problems seeing that a stable mostly
300automated flow in the end can result in a ridiculous low price per page,
301especially in typesetting on demand.
302
303\stopsubsubject
304
305\startsubject[title=Hybrids]
306
307Last I will mention a setup that we sometimes are involved in. An author writes
308books and uses \TEX. The publisher is okay with that and adds some quality
309assurance but in the end the product comes from the author. Maybe images are
310oursourced (not always for the better) but these can be handled easily. It can be that
311a copy|-|editor is involved and that person also then has to use \TEX\ of course,
312or feedback to the author.
313
314Publishers, and this really depends on knowledgeable persons, which as said can
315be fun to work with, can look beyond paper and also decide for additional
316materials, for instance web pages, interactive exercises, etc. In that case
317either \CONTEXT\ input has to be available as \XML\ (an export) or (often better)
318\XML\ is the starting point for multiple output. Contrary to what is believed,
319there are authors out there who have no problem coding in \XML\ directly. They
320think in structured content and reuse! The fact that they can hit a button in the
321editor and see the result in \PDF\ helps a lot. It just works.
322
323Here stability is either achieved by simply not updating during a project. There
324are however cases where an update is needed, for instance because demands
325changed. An example is a project where \ASCIIMATH\ is used which is a moving
326target. Of course one can update just that module, and often that works, but not
327when a module uses some new improved core helpers. Another example is additional
328proofing options.
329
330The budget of such projects seldom permit patching an existing distribution, so
331we then just update to the latest but not after checking if the used style works
332okay. There is no author involvement in this. Depending on the workflow, it can
333even be that the final rendering which involves fine tuning (side) float placement
334or page breaks (often educational documents have special demands) is done by us
335using special directives.
336
337Such hybrid workflows are quite convenient for all parties. The publisher works
338with the author who likes using these tools, the author can do her or his thing
339in the preferred way, and we do what we're best in: supporting this. And it
340scales up pretty well too if needed, without much costs for the publishers.
341
342\stopsubject
343
344\startsubject[title=Conclusion]
345
346So what can we conclude with respect to the demand for stability? First of course
347that it's important that our files keep running well. So, functionality should be
348stable. Freezing a distribution will make sure that during project you don't run
349into issues. Many \CONTEXT\ users update frequently in order to benefit from the
350latest additions. Most will not be harmed by this, but when something really
351breaks it's users like those on the \CONTEXT\ support list (who often also
352contribute in helping out other users) that are listened to first. Publishers
353demands play no role in this, if only because they also play no role in
354typesetting, and if they want to they should also contribute. The same is true
355for large suppliers. We're talking of free software often written without any
356compensation so these parties have no say in the matter unless they pay for it.
357It's small suppliers, authors and general users that matter most. If \CONTEXT\ is
358part of a workflow that we support, of course stability is guaranteed quite well,
359and those paying for that never have an issue with better solutions popping up.
360In fact, \CONTEXT\ is often just a tool then, one that does the job and questions
361about stability don't matter much in practice, as long as it does the job well.
362
363The main engine we use, \LUATEX, will be quite stable from version 1.10 and we'll
364try to make sure that newer versions are capable of running an older \CONTEXT,
365which is easier when no fundamental changes happen in the engine. Maybe a stripped
366down version of \LUATEX\ for \CONTEXT\ can facilitate that objective even more.
367
368Users themselves can try to stick to standard \CONTEXT\ features. The more tricks
369you apply, the less stable your future might be. Most mechanism are not evolving
370but some, like those that deal with columns, might become better over time. But
371typesetting in columns is often a one|-|shot adventure anyway (and who needs
372columns in the future).
373
374Of one thing users can be sure. There will never be a \CONTEXT\ professional or
375\CONTEXT\ enterprise. There is only one variant. All users get the same
376functionality and policies don't change suddenly. There will be no lock in to some
377cloud or web based service either. Of course one can hire us for support of any
378kind but that's independent of the distributed package. There is support by users
379for users on mailing lists and other media. That itself can also guard stability.
380
381But, always keep in mind that stability and progress, either of not driven by the
382environment that we operate in, can be in conflict.
383
384\stopsubject
385
386\stopchapter
387
388\stopcomponent
389