Source Browser ?

1% language=us runpath=texruns:manuals/musings
2
3\startcomponent musings-whytex
4
5\environment musings-style
6
7\startchapter[title={Why use \TEX ?}]
8
9\startlines \setupalign[flushright]
10Hans Hagen
11Hasselt NL
12July 2021 (public 2023)
13\stoplines
14
15\startsection[title={Introduction}]
16
17Let's assume that you know what \TEX\ is: a program that interprets a language
18with the same name that makes it possible to convert (tagged) input into for
19instance \PDF. For many of its users it is a black box: you key in some text, hit
20a button and get some typeset result in return. After a while you start tweaking
21this black box, meet other users (on the web), become more fluent and stick to it
22forever.
23
24But now let's assume that you don't know \TEX\ and are in search of a system that
25helps you create beautiful documents in an efficient way. When your documents
26have a complex structure you are probably willing to spend some time on figuring
27out what the best tool is. Even if a search lets you end up with something called
28\TEX, a three letter word with a dropped E, you still don't know what it is. It
29helps to search for \type {\TeX} which is pronounced as \type {tech}.
30Advertisement for \TEX\ is often pretty weak. It's rather easy to point to the
31numerous documents that can be found on the web. But what exactly does \TEX\ do
32and what are its benefits? In order to answer this we need to know who you are:
33an author, editor, an organization that deals with documents or needs to generate
34readable output, like publishers do.
35
36\stopsection
37
38\startsection[title={Authors}]
39
40We start with authors. Students of sciences that use mathematics don't have much
41of a choice. But most of these documents hardly communicate the message that
42\quotation {Everyone should use \TEX.} or that \quotation {All documents produced
43by \TEX\ look great.} but they do advocate that for rendering math it is a pretty
44good system. The source code of these documents often look rather messy and
45unattractive and for a non|-|math user it can be intimidating. Choosing some
46lightweight click|-|and|-|ping alternative looks attractive.
47
48Making \TEX\ popular is not going to happen by convincing those who have to write
49an occasional letter or report. They should just use whatever suits them. On the
50other hand if you love consistency, long term support, need math, are dealing
51with a rare language or script, like to reuse content, prefer different styling
52from one source, use one source for multiple documents, or maybe love open source
53tools, then you are a candidate. Of course there is a learning curve but normally
54you can master \TEX\ rather fast and once you get the hang of it there's often no
55way back. But you always need to invest a bit beforehand.
56
57So what authors are candidates for \TEX ? It could be that \TEX\ is the only tool
58that does the job. If so, you probably learned that from someone who saw you
59struggle or had the same experience and wrote or talked about it somewhere. In
60that case using \TEX\ for creating just one document (like a thesis) makes sense.
61Otherwise, you should really wonder if you want to invest time in a tool that you
62probably have to ditch later on as most organizations stick to standard
63(commercial) word processing tools.
64
65Talking to customers we are often surprised that people have heard about \TEX, or
66even used it for a few documents in college. Some universities just prescribe the
67use of \TEX\ for reporting, so not much of a choice there. Memories are normally
68rather positive in the sense that they know that it can do the job and that it's
69flexible.
70
71User group journals, presentations at \TEX\ meetings, journals, books and manuals
72that come with \TEX\ macro packages can all be used to determine if this tool
73suits an author. Actually, I started using \TEX\ because the original \TEX book
74had some magic, and reading it was just that: reading it, as I had no running
75implementation. A few years later, when I had to write (evolving) reports, I
76picked up again. But I'm not a typical user.
77
78\stopsection
79
80\startsection[title={Programmers}]
81
82When you are a programmer who has to generate reports, for instance in \PDF, or
83write manuals, then \TEX\ can really be beneficial. Of course \TEX\ is not always
84an obvious choice, but if you're a bit able to use it it's hard to beat in
85quality, flexibility and efficiency. I'm often surprised that companies are
86willing to pay a fortune for functionality that basically comes for free.
87Programmers are accustomed to running commands and working in a code editor with
88syntax highlighting so that helps too. They also recognize when something can be
89done more efficiently.
90
91When you need to go from some kind of input (document source, database,
92generated) to some rendered output there currently are a few endpoints: a
93(dynamic) \HTML\ page, a \PDF\ document, something useable in a word processor,
94or a representation using the desktop user interface. It's the second category
95where \TEX\ is hard to beat but even using \TEX\ and \METAPOST\ for creating a
96chart can make sense.
97
98There are of course special cases where \TEX\ fits in nicely. Say that you have
99to combine \PDF\ documents. There are numerous tools to do that and \TEX\ is one.
100The advantage of \TEX\ over other tools is that it's trivial to add additional
101text, number pages, provide headers and footers. And it will work forever. Why?
102Because \TEX\ has been around for decades and will be around for decades to come.
103It's an independent component. The problem with choosing for \TEX\ is that the
104starting point is important. The question is not \quotation {What tool should I
105use?} but \quotation {What problem do I need to solve?}. An open discussion about
106the objectives and possibilities is needed, not some checklist based on
107assumptions. If you don't know \TEX\ and have never worked with a programmable
108typesetting environment, you probably don't see the possibilities. In fact, you
109might even choose for \TEX\ for the wrong reasons.
110
111The problem with this category of users is that they seldom have the freedom to
112choose their tools. There are not that many jobs where the management is able to
113recognize the clever programmer who can determine that \TEX\ is suitable for a
114lot of jobs and can save money and time. Even the long term availability and
115support is not an argument since not only most tools (or even apis) changes every
116few years but also organizations themselves change ownership, objectives, and
117personnel on a whim. The concept of \quote {long term} is hard to grasp for most
118people (just look at politics) and it's only in retrospect that one can say
119\quote {We used that toolkit for over a decade.}
120
121\stopsection
122
123\startsection[title={Organizations}]
124
125Authors (often) have the advantage that they can choose themselves: they can use
126what they like. In practice any decent programmer is able to find the suitable
127tools but convincing the management to use one of them can be a challenge. Here
128we're also talking of \quote {comfort zones}: you have to like a tool(chain).
129Organizations normally don't look for \TEX. Special departments are responsible
130for choosing and negotiating whatever is used in a company. Unfortunately
131companies don't always start from the open question \quotation {We have this
132problem, we want to go there, what should we do?} and then discuss options with
133for instance those who know \TEX. Instead requirements are formulated and matches
134are found. The question then is \quotation {Are these requirements cut in stone?}
135and if not (read: we just omit some requirements when most alternatives don't
136meet them), were other requirements forgotten? Therefore organizations can end up
137with the wrong choice (using \TEX\ in a situation where it makes no sense) or
138don't see opportunities (not using \TEX\ while it makes most sense). It doesn't
139help that a hybrid solution (use a mix of \TEX\ and other tools) is often not an
140option. Where an author can just stop using a tool after a few days of
141disappointment, and where a programmer can play around a bit before making a
142choice, an organization probably best can start small with a proof of concept.
143
144Let's take a use case. A publisher wants to automatically convert \XML\ files
145into \PDF. One product can come from multiple sources (we have cases where
146thousands of small \XML\ files combine into one final product). Say that we have
147three different layouts: a theory book, a teachers manual and an answer book. In
148addition special proofing documents have to be rendered. The products might be
149produced on demand with different topics in any combination. There is at least
150one image and table per page, but there can be more. There are color and
151backgrounds used, tables of contents generated, there is extensive cross
152referencing and an index. Of course there is math.
153
154Now let's assume an initial setup costs 20K Euro and, what happens often when the
155real products show up, a revision after one year takes the same amount. We also
156assume 10K for the following eight years for support. So, we end up with 120K
157over 10 years. If one goes cheap we can consider half of that, or we can be
158pessimistic and double the amount.
159
160The first year 10K pages are produced, the second year 20K and after that 30K per
161year. So, we're talking of 270K pages. If we include customer specific documents
162and proofing we might as well end up with a multiple of that.
163
164So, we have 120K Euro divided by 270K pages or about half an Euro per page. But
165likely we have more pages so it costs less. If we double the costs then we can
166assume that some major changes took place which means more pages. In fact we had
167projects where the layout changed, all documents were regenerated and the costs
168were included in the revision, so far from double. We also see many more pages
169being generated so in practice the price per page drops below half an Euro. The
170more we process the cheaper it gets and one server can produce a lot of pages!
171
172Now, the interesting bit of such a calculation is that the costs only concern the
173hours spent on a solution. A \TEX\ based system comes for free and there are no
174license costs. Whatever alternative is taken, even if it is as flexible, it will
175involve additional costs. From the perspective of costs it's very hard to beat
176\TEX. Add to that the possibility for custom extensions, long term usage and the
177fact that one can adapt the system. The main question of course is: does it do
178the job. The only way to find out is to either experiment (which is free),
179consult an expert (not free, but then needed anyway for any solution) or ask an
180expert to make a proof of concept (also not free but relatively cheap and
181definitely cheaper than a failure). In fact, before making decisions about what
182solution is best it might be a good idea to check with an expert anyway, because
183more or less than one thinks might be possible. Also, take into account that the
184\TEX\ ecosystem is often one of the first to support new technologies, and
185normally does that within its existing interface. And there is plenty of free
186support and knowledge available once you know how to find it. Instead of wasting
187time and money on advertisement and fancy websites, effort goes into support and
188development. Even if you doubt that the current provider is around in the decade
189to come, you can be sure that there will be others, simply because \TEX\ attracts
190people. Okay, it doesn't help that large companies like to out source to
191far||far||away and expect support around the corner, so in the end they might
192kill their support chain.
193
194When talking of \TEX\ used in organizations we tend to think of publishers. But
195this is only a small subset of organizations where information gets transformed
196into something presentable. For small organizations the choice for \TEX\ can be
197easy: costs, long term stability, knowing some experts are driving forces. For
198large organizations these factors seem (at least to us) hardly relevant. We've
199(had) projects where actually the choice for using a \TEX\ based solution was (in
200retrospect) a negative one: there was no other tool than this relatively unknown
201thing called \TEX. Or, because the normal tools could not be used, one ended up
202with a solution where (behind the scenes) \TEX\ is used, without the organization
203knowing it. Or, it happened that the problem at hand was mostly one that demands
204in|-|depth knowledge of manipulating content, cleaning up messy data, combining
205resources (images or \PDF\ documents), all things that happen to be available in
206the perspective of \TEX. If you can solve a hard to solve problem for them then
207an organization doesn't care what tool you use. What does matter is that the
208solution runs forever, that costs are controllable and above all, that it
209\quotation {Just works.} And if you can make it work fast, that helps too. We
210can safely claim that when \TEX\ is evaluated as being a good option, that in the
211end it always works out quite well.
212
213Among arguments that (large) organizations like to use against a choice for \TEX\
214(or something comparable) are the size of the company that they buy their
215solution from, the expected availability for support, and the wide|-|spread usage
216of the tool at hand. One can wonder if it also matters that many vendors change
217ownership, change products every few years, change license conditions when they
218like, charge a lot for support or just abort a tool chain. Unfortunately when that
219happens those responsible for choosing such a system can have moved on to another
220job, so this is seldom part of an evaluation. For the supplier the other side of
221the table is just as much of a gamble. In that respect, an organization that
222wants to use an open source (and|/|or free) solution should realize that getting
223a return on investment on such a development is pretty hard to achieve. So, who
224really takes the risk for writing open source?
225
226For us, the reason to develop \CONTEXT\ and make it open is that it fits in our
227philosophy and we like the community. It is actually not really giving us an
228advantage commercially: it costs way more to develop, support and keep
229up|-|to|-|date than it will ever return. We can come up with better, faster and
230easier solutions and in the end we pay the price because it takes less time to
231cook up styles. So there is some back slash involved because commercially a
232difficult solution leads to more billable hours. Luckily we tend to avoid wasting
233time so we improve when possible and then it ends up in the distributed code.
234And, once the solution is there, anyone can use it. Basically also for us it's
235just a tool, like the operating system, editor and viewer are. So, what keeps
236development going is mostly the interaction with the community. This also means
237that a customer can't really demand functionality for free: either do it
238yourself, wait for it to show up, or pay for it (which seldom happens). Open
239source is not equivalent with \quotation {You get immediately what you want
240because someone out there writes the code.}. There has to be a valid reason and
241often it's just users and meetings or just some challenge that drives it.
242
243This being said, it is hard to convince a company to use \TEX. It has to come
244from users in the organization. Or, what we sometimes see with publishers, it
245comes with an author team or acquired product line where it's the only option.
246Even then we seldom see transfer to other branches in the organizations. No one
247seems to wonder \quotation {How on earth can that \XML\ to \PDF\ project produce
248whatever output in large quantities in a short period of time} while other (past)
249projects failed. It probably relates to the abstraction of the process. Even
250among \TEX\ users it can be that you demonstrate something with a click on a
251button and that many years afterwards someone present at that moment tells you
252that they just discovered that this or that can be done by hitting a button. I'm
253not claiming that \TEX\ is the magic wand for everything but in some areas it's
254pretty much ahead of the pack. Go to a \TEX\ user meeting and you will be surprised
255about the accumulated diverse knowledge present in the room. It's user demand that
256drives \CONTEXT\ development, not commerce.
257
258\stopsection
259
260\startsection[title={Choosing}]
261
262So, where can one find information about \TEX\ and friends? On the web
263one has to use the right search keys, so adding \type {tex} helps: \typ {context
264tex} or \typ {xml tex pdf} and so on. Can one make a fancy hip website, sure, but
265it being a life|-|long, already old and mature environment, and given that it
266comes for free, or is used low|-|budget, not much effort and money can be spent
267on advertising it. A benefit is that no false promises and hypes are made either.
268If you want to know more, just ask the right folks.
269
270For all kind of topics one can find interesting videos and blogs. One can
271subscribe to channels on YouTube or join forums. Unfortunately not that many
272bloggers or vloggers or podcasters come up with original material every time, and
273often one starts to recognize patterns and will get boring by repetition of wisdom
274and arguments. The same is true for manuals. Is a ten year old manual really
275obsolete? Should we just recompile it to fake an update while in fact there has
276been no need for it? Should we post twenty similar presentations while one can
277do? (If one already wants to present the same topic twenty times in the first
278place?) Maybe one should compare \TEX\ with cars: they became better over time
279and can last for decades. And no new user manual is needed.
280
281As with blogs and vlogs advertising \TEX\ carries the danger for triggering
282political discussions and drawing people into discussions that are not pleasant:
283\TEX\ versus some word processor, open versus closed source, free versus paid
284software, this versus that operating system, editor such or editor so.
285
286To summarize, it's not that trivial to come up with interesting information about
287\TEX, unless one goes into details that are beyond the average user. And those
288who are involved are often involved for a long time so it gets more complex over
289time. User group journals that started with tutorials later on became expert
290platforms. This is a side effect of being an old and long|-|term toolkit. If
291you run into it, and wonder if it can serve your purpose, just ask an expert.
292
293Most \TEX\ solutions are open source and come for free as well. Of course if you
294want a specific solution or want support beyond what is offered on mailing lists
295and forums you should be willing to pay for the hours spent. For a professional
296publisher (of whatever kind) this is not a problem, if only because any other
297solution also will cost something. It is hard to come up with a general estimate.
298A popular measure of typesetting costs is the price per page, which can range
299from a couple of euro's per page to two digit numbers. We've heard of cases where
300initial setup costs were charged. If not much manual intervention is needed a
301\TEX\ solution mostly concerns initial costs.
302
303Let's return to the main question \quotation {Why use \TEX ?} in which you can
304replace \TEX\ by one of the macro packages build on top of it, for instance
305\CONTEXT. If an (somewhat older) organization considers using \TEX\ it should
306also ask itself, why it wasn't considered long ago already? For sure there have
307been developments in \TEX\ engines (in \CONTEXT\ we use \LUATEX) as well as
308possibilities of macro packages but if you look at the documents produced with
309them, there is not that much difference with decades ago. Processing has become
310faster, some things have become easier, but new technologies have always been
311supported as soon at they showed up. Advertising is often just repeating an old
312message.
313
314The \TEX\ ecosystem was among the first in supporting for instance \OPENTYPE, and
315the community even made sure that there were free fonts available. A format like
316\PDF\ was supported as soon as it showed up and \TEX\ was the first to
317demonstrate what advanced features were there and it shows again how it is
318possible to adapt \TEX\ to changes in its environment. Processing \XML\ using
319\TEX\ has never been a big deal and if that is a reason to look at this already
320old and mature technology, then an organization can wonder if years and
321opportunities (for instance for publishing on demand or easy updating of manuals)
322have been lost. Of course there are (and have been) alternative tools but the
323arguments for using \TEX\ or not are not much different now. It can be bad
324marketing of open and free software. It can be that \TEX\ has been around too
325long. It can also be that its message was not understood yet. On the other hand,
326in software development it's quite common to reinvent wheels and present old as
327new. It's never too late to catch on.
328
329\stopsection
330
331\stopchapter
332
333\stopcomponent
334