context-2020-luametatex.tex /size: 13 Kb    last modification: 2021-10-28 13:50
 1% language=us 2 3\usemodule[present-boring,abbreviations-logos] 4 5\startdocument 6 [title={LUAMETATEX}, 7 banner={where do we stand}, 8 location={context\enspace {\bf 2020}\enspace meeting}] 9 10\starttitle[title=When it started] 11 12\startitemize 13 14\startitem 15 About three years ago the idea came up to go this route. 16\stopitem 17\startitem 18 At the 2018 meeting it was first mentioned and those present were okay with 19 it. 20\stopitem 21\startitem 22 Early 2019 the first beta release took place. 23\stopitem 24\startitem 25 At the 2019 meeting the first more official version was presented. 26\stopitem 27\startitem 28 Around the 2020 meeting we have more or less arrived at what I had in mind. 29\stopitem 30\startitem 31 At the 2021 meeting I expect the code to be stable and repositories to be set 32 up. 33\stopitem 34\startitem 35 At the 2022 meeting we can make the official transition from \MKIV\ to \LMTX. 36\stopitem 37\startitem 38 Some new options are only enabled in my local \type {cont-exp.tex} file. 39\stopitem 40\startitem 41 Knowing that Wolfgang keeps an eye on all those changes makes me more daring. 42\stopitem 43\startitem 44 We aim to get less (but more efficient) macro code that on the average looks 45 better. 46\stopitem 47 48\stopitemize 49 50\stoptitle 51 52\starttitle[title=Why it started] 53 54\startitemize 55 56\startitem 57 There was an increasing pressure for a stable \LUATEX. 58\stopitem 59\startitem 60 There should be no more changes to the interfaces, no more extensions. 61\stopitem 62\startitem 63 One can run into interesting comments on the web (as usual), like 64 \startitemize[packed] 65 \startitem The \LUATEX\ program has \quote {many bugs}. \stopitem 66 \startitem The \LUATEX\ manual is bad. \stopitem 67 \startitem The \LUATEX\ program is too slow to be useful. \stopitem 68 \startitem The \LUATEX\ program will never end up in distributions. \stopitem 69 \startitem The \LUATEX\ project is funded and developed in a commercial setting. \stopitem 70 \stopitemize 71\stopitem 72\startitem 73 I won't comment on how I read these (demotivating) comments because \unknown 74\stopitem 75\startitem 76 \unknown\ it anyway often says more about the writer (attitudes) than about 77 \LUATEX. 78\stopitem 79\startitem 80 I also looks like (non \CONTEXT) users are charmed by \LUATEX, and the more 81 they code, the more we need to freeze. 82\stopitem 83\startitem 84 So, hopefully, the \LUAMETATEX\ development does not interfere badly with 85 developments outside the \CONTEXT\ community. 86\stopitem 87 88\stopitemize 89 90\stoptitle 91 92\starttitle[title=The development] 93 94The summary on the next pages is partial. More can be found in articles and 95documents that come with the distribution. 96 97\startitemize 98 99\startitem 100 \LUATEX\ started out as \CWEB\ code \unknown\ that eventually became just 101 \CCODE\ \unknown\ which in \LUAMETATEX\ has been detached from the (complex) 102 infrastructure. 103\stopitem 104\startitem 105 The basic idea is to only keep the core of \TEX, but for instance font 106 loading, file handling and the backend are gone. 107\stopitem 108\startitem 109 As a consequence the code has been reorganized (shuffled around). 110\stopitem 111\startitem 112 I experimented a lot without bothering about usage elsewhere and I like the 113 result so far. 114\stopitem 115\startitem 116 The \CONTEXT\ distribution will at some point ship with the source. 117\stopitem 118 119\stopitemize 120 121\starttitle[title=File handling] 122 123\startitemize 124 125\startitem 126 All file handling goes via \LUA, also read and write related primitives. 127\stopitem 128\startitem 129 The same is true for terminal (console) handling. 130\stopitem 131\startitem 132 Part of that (the writing) was actually kind of extension code in \TEX\ and 133 partly a system dependency. 134\stopitem 135\startitem 136 The \ETEX\ pseudo file \type {\scantokens} primitive uses the same mechanism 137 as \LUA\ does. 138\stopitem 139 140\stopitemize 141 142\stoptitle 143 144\starttitle[title=The macro machinery] 145 146\startitemize 147 148\startitem 149 There are extensions to the way macro arguments are handled (less clumsy 150 macros). 151\stopitem 152\startitem 153 There are extra if tests (makes for nicer macros). 154\stopitem 155\startitem 156 Else branches in conditions can be collapsed using \type {\orelse} and \type 157 {\orunless} which gives cleaner low level code. 158\stopitem 159\startitem 160 Tracing gives more detail about node properties and also shows attributes. 161\stopitem 162\startitem 163 Some new data carriers have been added that can be played with from \LUA . 164\stopitem 165\startitem 166 Macros can efficiently be frozen (new) and protected (redone) and the 167 concepts \quote {long} and \type {outer} are gone. \footnote {In \CONTEXT\ 168 macros were always \type {\long} and never \type {\outer}. Most commands were 169 unexpandable (also in \MKII, pre \ETEX). So, users won't notice this.} 170\stopitem 171\startitem 172 Saving and restoring is somewhat more efficient (partly a side effect of 173 wider memory). 174\stopitem 175 176\stopitemize 177 178\stoptitle 179 180\starttitle[title=Language] 181 182\startitemize 183 184\startitem 185 Language control settings now use less parameters but bit sets instead. 186\stopitem 187\startitem 188 Only basic parameters are stored in the format file now. 189\stopitem 190\startitem 191 There are all kind of small improvements. 192\stopitem 193 194\stopitemize 195 196\stoptitle 197 198\starttitle[title=Typesetting] 199 200\startitemize 201 202\startitem 203 Attributes (the lists and states) are implemented more efficiently. 204\stopitem 205\startitem 206 The paragraph state is stored with the paragraph. 207\stopitem 208\startitem 209 Paragraphs can be normalized and options are now set with bit sets. 210\stopitem 211\startitem 212 Boxes carry orientation related information (offsets, rotation, etc). 213\stopitem 214\startitem 215 Some nodes carry more information. 216\stopitem 217\startitem 218 Directions are mostly gone (it's up to the backend). 219\stopitem 220\startitem 221 Migrated content is optionally kept with boxes. 222\stopitem 223 224\stopitemize 225 226\stoptitle 227 228\starttitle[title=Math] 229 230\startitemize 231 232\startitem 233 Some math concepts have been extended (like prescripts and some more 234 control over styles). 235\stopitem 236\startitem 237 There are plenty of new control details. 238\stopitem 239\startitem 240 The math parameter settings obey grouping in a math list. 241\stopitem 242\startitem 243 We can have math in discretionaries in text and more advanced discretionaries 244 in math as well. 245\stopitem 246 247\stopitemize 248 249\stoptitle 250 251\starttitle[title=Fonts] 252 253\startitemize 254 255\startitem 256 Font specification information no longer uses the string pool (which saves a 257 lot). 258\stopitem 259\startitem 260 Of course we still have the basic font handler. 261\stopitem 262\startitem 263 We only store what is needed for traditional \TEX\ font handling. 264\stopitem 265\startitem 266 Virtual fonts are even more virtual (also a backend thing) so we can have 267 more features. 268\stopitem 269 270\stopitemize 271 272\stoptitle 273 274\starttitle[title=The code] 275 276\startitemize 277 278\startitem 279 Artifacts from \PASCAL\ and \CWEB\ have been removed. 280\stopitem 281\startitem 282 Languages, fonts, marks etc are no longer \quote {register} based. 283\stopitem 284\startitem 285 The token interface is more abstract and no longer presents strange numbers. 286\stopitem 287\startitem 288 Some internals have been reconstructed because of cleaner \LUA\ interfacing. 289\stopitem 290\startitem 291 A side effect of this is better abstraction of the equivalent ranges. 292\stopitem 293\startitem 294 The code has been made more abstract (and looks easier in e.g. Visual Studio). 295\stopitem 296\startitem 297 The compile farm is used to check if compilation works out of the box. 298\stopitem 299\startitem 300 Compilation is fast and easy, otherwise this project was not possible. 301\stopitem 302\startitem 303 Readability of the code is constantly improved (the usual: has to look okay 304 in my editor). 305\stopitem 306\startitem 307 The code has been made mostly independent of specific operating system needs. 308\stopitem 309\startitem 310 Wide characters are dealt with in Windows interfaces. 311\stopitem 312 313\stopitemize 314 315\stoptitle 316 317\starttitle[title=Libraries] 318 319\startitemize 320 321\startitem 322 We really want to stay lean and mean: the engine is also a \LUA\ engine. 323\stopitem 324\startitem 325 All code is included, a few libraries are used, but these are small, old and 326 stable. 327\stopitem 328\startitem 329 In addition some helper libraries are made (including pplib by Pawel). 330\stopitem 331\startitem 332 What we ship is what you get: \CONTEXT\ will not depend on more than that. 333\stopitem 334\startitem 335 If something is updated (at all) the differences are checked first. 336\stopitem 337 338\stopitemize 339 340\stoptitle 341 342\starttitle[title=The \LUA\ engine] 343 344\startitemize 345 346\startitem 347 We use the latest (even alpha) \LUA\ (5.4) because \LUAMETATEX\ is a good 348 test. 349\stopitem 350\startitem 351 There is no support for \LUAJIT\ and the \FFI\ interface is gone. 352\stopitem 353\startitem 354 There is a limited set of libraries that we support but no code is (and will 355 be) included. 356\stopitem 357\startitem 358 There are less callbacks (because we only have a frontend). 359\stopitem 360\startitem 361 There are more token scanners and some options have been added. 362\stopitem 363 364\stopitemize 365 366\stoptitle 367 368\starttitle[title=Efficiency] 369 370\startitemize 371 372\startitem 373 We benefit some more from the wider memory words (some constructs could go). 374\stopitem 375\startitem 376 The format file is smaller and not longer compressed. 377\stopitem 378\startitem 379 Memory management is now mostly dynamic and usage is much lower. 380\stopitem 381\startitem 382 There are more statistics (also as side effect of memory management). 383\stopitem 384\startitem 385 Dumping the format has been made a bit more robust and is faster. 386\stopitem 387\startitem 388 The core engine performs a bit better (machines don't get that much faster). 389\stopitem 390\startitem 391 We want to be prepared for future architectures. 392\stopitem 393\startitem 394 We manage to keep the binary way below 3 MB. 395\stopitem 396\startitem 397 The lot runs quite well on e.g.\ a Raspberry Pi 4. 398\stopitem 399 400\stopitemize 401 402\starttitle[title=Upgraded \METAPOST] 403 404\startitemize 405 406\startitem 407 All (eight bit) font stuff has been stripped from the \METAPOST\ library. 408\stopitem 409\startitem 410 The library no longer has a \POSTSCRIPT\ backend. 411\stopitem 412\startitem 413 The library provides scanners that make extensions possible. 414\stopitem 415\startitem 416 All file \IO\ goes via \LUA. 417\stopitem 418\startitem 419 There are a few additions like pre|/|postscripts for clip and bounding boxes. 420\stopitem 421\stopitemize 422 423\stoptitle 424 425\starttitle[title=Praise for the users] 426 427\startitemize 428 429\startitem 430 Much has been done and I probably forget to mention a lot. 431\stopitem 432\startitem 433 The number of bugs is relative small compared to what gets changed and added. 434\stopitem 435\startitem 436 The test suite gets ran very often, also to check if performance is okay. 437\stopitem 438\startitem 439 I could only do this because the \CONTEXT\ users are so tolerant. 440\stopitem 441\startitem 442 Some seem to constantly check for updates so they help with fast testing. 443\stopitem 444\startitem 445 The \CONTEXT\ code base gets stepwise adapted (split files) which again 446 forces users to test. 447\stopitem 448\startitem 449 It takes a lot of time because we take small steps in order not to mess up. 450\stopitem 451\startitem 452 I would not do it without the positive attribute of the \CONTEXT\ users. 453\stopitem 454\startitem 455 It's all about motivation and I thank the \CONTEXT\ users for providing this 456 friendly and non|-|competitive bubble! 457\stopitem 458 459\stopitemize 460 461\starttitle[title=Todo] 462 463\startitemize 464 465\startitem 466 Maybe add some more sanity checks in order to catch errors intruded by 467 callbacks. Maybe add some more tracing too. 468\stopitem 469\startitem 470 Explore variants, like having registers in dedicated eqtb tables so that we 471 can allocate them dynamically (mostly for the fun of doing it). 472\stopitem 473\startitem 474 Add some more documentation (read: addition cq.\ remarks about where the 475 original documentation no longer applies, but we have years for doing that). 476\stopitem 477\startitem 478 Update the manual (which is done occasionally in batch based on print|-|outs; 479 there is no real need to hurry because we still experiment). 480\stopitem 481\startitem 482 Apply some of the new stuff in \LMTX. Take up some challenges. 483\stopitem 484\startitem 485 Wrap up new functionality (once it's stable) in articles and other documents. 486\stopitem 487 488\stopitemize 489 490\stoptitle 491 492\starttitle[title=And \LUATEX ?] 493 494\startitemize 495 496\startitem 497 Of course \LUATEX\ will be maintained! After all, \MKIV\ needs it and it 498 serves as reference for the front|-|end rendering and back|-|end generation 499 when we're messing with \LUAMETATEX. 500\stopitem 501\startitem 502 It is used by \LATEX\ and there are now also plain inspired packages. Because 503 there are spin|-|offs (\LATEX\ has settled on a version with built|-|in font 504 processing) we cannot change much. 505\stopitem 506\startitem 507 And \LUATEX\ being nicely integrated into \TEXLIVE\ is another argument for 508 not touching it too much. 509\stopitem 510\startitem 511 I have no clue of \LUATEX\ usage but that fact alone already makes an 512 argument for being even more careful. It's bad advertisement for \TEX\ when 513 users who use the low level interfaces get confronted with conceptual 514 changes. 515\stopitem 516\startitem 517 So in the end not much will be back ported to \LUATEX: at some point the code 518 base became too different and it's the price paid for the stability demand. 519 That way we cannot introduce new bugs either. It also doesn't pay of. 520\stopitem 521\startitem 522 But, a few non|-|intrusive things might actually trickle into it in due time, 523 also out of self interest: it might help to share code between \MKIV\ and 524 \LMTX. 525\stopitem 526 527\stopitemize 528 529\stoptitle 530 531\stopdocument 532