Source Browser ?

1% language=us runpath=texruns:manuals/lowlevel
2
3% It took some time to get the right balance of using the overload related features
4% but at some point it started feeling right. Of course it will never be as
5% perfectly timed and integrated as Gavin Harrison performance on "Threatening War"
6% (youtube movie) but that doesn't mean I should not aim for perfection. But as
7% with drumming, it takes practising and that is what I did on a subset of sources
8% when writing the engine code.
9%
10% It tooks a few weeks but November 21 2020 the last of the core files was turned
11% \LMTX, not that the work was done (checking to be done (thanks Wolfgang!), imp
12% files to be checked, many \LUA\ files to be updated) but it's a start. This time
13% the musical timestamp is listening to Nikola Cvetkovic (piano) and history
14% podcasts.
15
16\usemodule[system-tokens]
17
18\environment lowlevel-style
19
20\startdocument
21  [title=security,
22   color=middleorange]
23
24\startsectionlevel[title=Preamble]
25
26Here I will discuss a moderate security subsystem of \LUAMETATEX\ and therefore
27\CONTEXT\ \LMTX. This is not about security in the sense of the typesetting
28machinery doing harm to your environment, but more about making sure that a user
29doesn't change the behavior of the macro package in ways that introduce
30interference and thereby unwanted side effect. It's all about protecting macros.
31
32This is all very experimental and we need to adapt the \CONTEXT\ source code to
33this. Actually that will happen a few times because experiments trigger that. It
34might take a few years before the security model is finalized and all files are
35updated accordingly. There are lots of files and macros involved. In the process
36the underlying features in the engine might evolve.
37
38\stopsectionlevel
39
40\startsectionlevel[title=Flags]
41
42Before we go into the security levels we see what flags can be set. The \TEX\
43language has a couple of so called prefixes that can be used when setting values
44and defining macros. Any engine that has traditional \TEX\ with \ETEX\ extensions
45can do this:
46
47\starttyping[option=TEX]
48                 \def\foo{foo}
49\global          \def\foo{foo}
50\global\protected\def\foo{foo}
51\stoptyping
52
53And \LUAMETATEX\ adds another one:
54
55\starttyping[option=TEX]
56       \tolerant          \def\foo{foo}
57\global\tolerant          \def\foo{foo}
58\global\tolerant\protected\def\foo{foo}
59\stoptyping
60
61What these prefixes do is discussed elsewhere. For now is is enough to know that
62the two optional prefixes \type {\protected} and \type {\tolerant} make for four
63distinctive cases of macro calls.
64
65But there are more prefixes:
66
67\starttabulate
68\HL
69\NC \type {frozen}     \NC a macro that has to be redefined in a managed way \NC \NR
70\NC \type {permanent}  \NC a macro that had better not be redefined \NC \NR
71\NC \type {primitive}  \NC a primitive that normally will not be adapted \NC \NR
72\NC \type {immutable}  \NC a macro or quantity that cannot be changed, it is a constant \NC \NR
73\NC \type {mutable}    \NC a macro that can be changed no matter how well protected it is \NC \NR
74\HL
75\NC \type {instance}   \NC a macro marked as (for instance) be generated by an interface \NC \NR
76\HL
77\NC \type {noaligned}  \NC the macro becomes acceptable as \type {\noalign} alias \NC \NR
78\HL
79\NC \type {overloaded} \NC when permitted the flags will be adapted \NC \NR
80\NC \type {enforced}   \NC all is permitted (but only in zero mode or ini mode) \NC \NR
81\NC \type {aliased}    \NC the macro gets the same flags as the original \NC \NR
82\HL
83\stoptabulate
84
85These prefixed set flags to the command at hand which can be a macro but
86basically any control sequence.
87
88To what extent the engine will complain when a property is changed in a way that
89violates the above depends on the parameter \type {\overloadmode}. When this
90parameter is set to zero no checking takes place. More interesting are values
91larger than zero. If that is the case, when a control sequence is flagged as
92mutable, it is always permitted to change. When it is set to immutable one can
93never change it. The other flags determine the kind of checking done. Currently
94the following overload values are used:
95
96\starttabulate[|l|l|c|c|c|c|c|]
97    \NC   \NC         \BC immutable \BC permanent \BC primitive \BC frozen \BC instance \NC \NR
98    \NC 1 \NC warning \NC \star     \NC \star     \NC \star     \NC        \NC          \NC \NR
99    \NC 2 \NC error   \NC \star     \NC \star     \NC \star     \NC        \NC          \NC \NR
100    \NC 3 \NC warning \NC \star     \NC \star     \NC \star     \NC \star  \NC          \NC \NR
101    \NC 4 \NC error   \NC \star     \NC \star     \NC \star     \NC \star  \NC          \NC \NR
102    \NC 5 \NC warning \NC \star     \NC \star     \NC \star     \NC \star  \NC \star    \NC \NR
103    \NC 6 \NC error   \NC \star     \NC \star     \NC \star     \NC \star  \NC \star    \NC \NR
104\stoptabulate
105
106The even values (except zero) will abort the run. In \CONTEXT\ we plug in a
107callback that deals with the messages. A value of 255 will freeze this parameter.
108At level five and above the \type {instance} flag is also checked but no drastic
109action takes place. We use this to signal to the user that a specific instance is
110redefined (of course the definition macros can check for that too).
111
112So, how does it work. The following is okay:
113
114\starttyping[option=TEX]
115\def\MacroA{A}
116\def\MacroB{B}
117\let\MyMacro\MacroA
118\let\MyMacro\MacroB
119\stoptyping
120
121The first two macros are ordinary ones, and the last two lines just create an alias. Such
122an alias shares the definition, but when for instance \type {\MacroA} is redefined, its
123new meaning will not be reflected in the alias.
124
125\starttyping[option=TEX]
126\permanent\protected\def\MacroA{A}
127\permanent\protected\def\MacroB{B}
128\let\MyMacro\MacroA
129\let\MyMacro\MacroB
130\stoptyping
131
132This also works, because the \type {\let} will create an alias with the protected
133property but it will not take the \type {permanent} propery along. For that we need
134to say:
135
136\starttyping[option=TEX]
137\permanent\protected\def\MacroA{A}
138\permanent\protected\def\MacroB{B}
139\permanent\let\MyMacro\MacroA
140\permanent\let\MyMacro\MacroB
141\stoptyping
142
143or, when we want to copy all properties:
144
145\starttyping[option=TEX]
146\permanent\protected\def\MacroA{A}
147\permanent\protected\def\MacroB{B}
148\aliased\let\MyMacro\MacroA
149\aliased\let\MyMacro\MacroB
150\stoptyping
151
152However, in \CONTEXT\ we have commands that we like to protect against
153overloading but at the same time have a different meaning depending on the use
154case. An example is the \type {\NC} (next column) command that has a different
155implementation in each of the table mechanisms.
156
157\starttyping[option=TEX]
158\permanent\protected\def\NC_in_table   {...}
159\permanent\protected\def\NC_in_tabulate{...}
160\aliased\let\NC\NC_in_table
161\aliased\let\NC\NC_in_tabulate
162\stoptyping
163
164Here the second aliasing of \type {\NC} fails (assuming of course that we enabled
165overload checking). One can argue that grouping can be used but often no grouping
166takes place when we redefine on the fly. Because \type {frozen} is less restrictive
167than \type {primitive} or \type {permanent}, and of course \type {immutable}, the
168next variant works:
169
170\starttyping[option=TEX]
171\frozen\protected\def\NC_in_table   {...}
172\frozen\protected\def\NC_in_tabulate{...}
173\overloaded\let\NC\NC_in_table
174\overloaded\let\NC\NC_in_tabulate
175\stoptyping
176
177However, in practice, as we want to keep the overload checking, we have to do:
178
179\starttyping[option=TEX]
180\frozen\protected\def\NC_in_table   {...}
181\frozen\protected\def\NC_in_tabulate{...}
182\overloaded\frozen\let\NC\NC_in_table
183\overloaded\frozen\let\NC\NC_in_tabulate
184\stoptyping
185
186or use \type {\aliased}, but there might be conflicting permissions. This is not
187that nice, so there is a kind of dirty trick possible. Consider this:
188
189\starttyping[option=TEX]
190\frozen\protected\def\NC_in_table   {...}
191\frozen\protected\def\NC_in_tabulate{...}
192\def\setNCintable   {\enforced\let\frozen\let\NC\NC_in_table}
193\def\setNCintabulate{\enforced\let\frozen\let\NC\NC_in_tabulate}
194\stoptyping
195
196When we're in so called \type {initex} mode or when the overload mode is zero,
197the \type {\enforced} prefix is internalized in a way that signals that the
198follow up is not limited by the overload mode and permissions. This definition
199time binding mechanism makes it possible to use \type {permanent} macros that
200users cannot redefine, but existing macros can, unless of course they tweak the
201mode parameter.
202
203Now keep in mind that users can always cheat but that is intentional. If you
204really want to avoid that you can set the overload mode to 255 after which it
205cannot be set any more. However, it can be useful to set the mode to zero (or
206some warning level) when foreign macro packages are used.
207
208\stopsectionlevel
209
210\startsectionlevel[title=Complications]
211
212One side effect of all this is that all those prefixes can lead to more code. On
213the other hand we save some due to the extended macro argument handling features.
214When you take the size of the format file as reference, in the end we get a
215somewhat smaller file. Every token that you add of remove gives a 8~bytes
216difference. The extra overhead that got added to the engine is compensated by the
217fact that some macro implementations can be more efficient. In the end, in spite
218of these new features and the more extensive testing of flags performance is
219about the same. \footnote {And if you wonder about memory, by compacting the used
220(often scattered) token memory before dumping I manages to save some 512K on the
221format file, so often the loss and gain are somewhere else.}
222
223\stopsectionlevel
224
225\startsectionlevel[title=Introspection]
226
227In case you want to get some details about the properties of a macro, you can
228check its meaning. The full variant shows all of them.
229
230\startbuffer
231% a macro with two optional arguments with optional spacing in between:
232
233\permanent\tolerant\protected\def\MyFoo[#1]#*[#2]{(#1)(#2)}
234
235\meaningless\MyFoo\par
236\meaning    \MyFoo\par
237\meaningfull\MyFoo\par
238\stopbuffer
239
240\typebuffer[option=TEX]
241
242\startpacked \getbuffer \stoppacked
243
244\stopsectionlevel
245
246% In \CONTEXT:
247% c! v! s! ??
248% newif newcount ... newconditional etc
249% userinterface (permanent)
250% primitives
251% noaligned
252% frozen is for users
253
254\stopdocument
255