1/* 2 See license.txt in the root of this project. 3*/ 4 5# ifndef LMT_LUAMETATEX_H 6# define LMT_LUAMETATEX_H 7 8/*tex 9 10 The \LUATEX\ project started in 2005 with an experiments by Hartmut and me: adding the \LUA\ 11 Scripting language (that I knew from the \SCITE\ editor) to \PDFTEX. When we came to the 12 conclusion that a more tight integration made sense Taco did the impressive conversion from 13 \PASCAL\ |WEB\ to \CWEB. This happened in the perspective of the Oriental \TEX\ project, that 14 has as objective high quality Arabic typesetting. The way to achieve that was opening up the 15 font machinery and access to the paragraph building. It was an intense development period, 16 with Taco doing the coding, Hans exploring possibilities and extending \CONTEXT, and Idris 17 making fonts and testing. Taco and I discussed, compiled, accepted and rejected ideas. These 18 were interesting times! Over the years that we had used \TEX\ we could finally explore what we 19 had been talking about for years (long trips to user group meetings are good for that). We 20 ame to the first version(s) of \LUATEX\ with \CONTEXT\ \MKIV\ providing a testbed and as we 21 progressed we ended up with something we liked a lot. 22 23 After half a decade, where in the meantime Taco also had turned MetaPost into a library, we 24 had a version that had proved itself well. The following years, with Taco having less time 25 available, I started loking at the code. Some more got added to the Lua interfaces. Math got 26 split code paths and some new primitives were introduced. Luigi started taking care of managing 27 the code base so that I could cross compile for \MSWINDOWS. He also deals with the libraries 28 that were used and integration in \TEXLIVE\ and maintains the (by now stable) \METAPOST\ code 29 base. 30 31 After a while it became clear that users other than \CONTEXT\ wanted the program to stay as it 32 was and not introduce features or improve interfaces in ways that demanded a change in used 33 \LUA\ code. So, after a decade of development the official stable release took place. We already 34 had a split between stable (normally the \TEXLIVE\ release) and experimental (that we used for 35 development). However, in practice experimental versions were seen as real releases and we got 36 complaints that something could be broken (which actually is natural for an experimental 37 version). So, this split model didn't work out well in practice: you cannot explore and 38 experiment when you cannot play with yet unfinished code. 39 40 So at some point I decided that the best approach to a follow up, one not interfering with 41 usage of a stable \LUATEX, would be a more drastic split: the idea of \LUAMETATEX\ took shape. 42 This code base is the result of that. For whatever bad was introduced in \LUAMETATEX, and maybe 43 already before that in \LUATEX), you can blame me (Hans) and not Taco: Luigi consistently added 44 (hh) to the \LUATEX\ svn entries when that was feasible, so one can check where I messed up. 45 In the end all this work can be considered a co-product and the \CONTEXT\ (dev) community was 46 instrumental in this as well. 47 48 There are some fundamental changes: there is no backend but maybe I'll introduce a framework 49 for that at some point because the impact on performance has been quite noticeable (although 50 it has been compensated in the meantime). There is no support for \LUAJIT, because it doesn't 51 keep up with \LUA. Also, there is no support for \FFI, because that project is orphaned, but 52 there are other ways. Some more is delegated to \LUA, but also some more has been added to \TEX. 53 54 Over the 15 years that it took to go from the first version of \LUATEX\ in 2005 to the first 55 release of \LUAMETATEX\ in 2020 (although intermediate versions have always been good enough 56 to be used in production with \CONTEXT) I've written numerous articles in user group journals 57 as well as several presentations each year on progress and features. There are also wrapups 58 available in the \CONTEXT\ distribution that shed some light on how the developments 59 progress(ed). In the end it's all a work of many. There are no commercial interrests and 60 everything is done out of love for TeX and in free time, so take that into account when you 61 bark about code or documentation. 62 63 The \LUAMETATEX\ code base is maintained by Hans Hagen and Wolfgang Schuster (code, programming, 64 etc) with help from Mojca Miklavec (distribution, compile farm, etc) and Alan Braslau (testing, 65 feedback, etc). Of course with get help from all those \CONTEXT\ users who are always very 66 willing to test. 67 68 We start with the version numbers. While \LUATEX\ operates in the 100 range, the \LUAMETATEX\ 69 engine takes the 200 range. Revisions range from 00 upto 99 and the dates \unknown\ depend on 70 the mood. The |2.05.00| version with the development id |20200229| was more or less the first 71 official version, in the sense that most of the things on my initial todo list were done. It's 72 a kind of virtual date as it happens to be a leapyear. As with LuaTeX the .10 version will be 73 the first 'stable' one, released somewhere around the ConTeXt 2021 meeting. 74 75 2.08.18 : around TeXLive 2021 code freeze (so a bit of a reference version) 76 2.09.35 : near the end of 2021 (so close to the 2.10 release date) 77 2.09.55 : in July 2022 (the official release of the new math engine) 78 2.10.00 : a few days before the ctx 2022 meeting (starting September 19) 79 2.10.01 : mid October 2022 (some more upgrades of the math engine) 80 2.10.05 : late January 2023 81 2.10.07 : around TeXLive 2023 82 2.10.08 : close to BachoTeX 2023 83 2.10.09 : evolved in May/June 2023 (math & text linebreak experiments) 84 2.10.10 : around the ctx 2023 meeting 85 2.10.11 : till the end of 2024 86 87 At some point the \CONTEXT\ group will be responsible for guaranteeing that the official version 88 is what comes with \CONTEXT\ and that long term support and stabilty is guaranteed and that no 89 feature creep or messing up happens. We'll see. 90 91 Hans Hagen 92 93*/ 94 95# include "tex/textypes.h" 96 97# define luametatex_version 210 98# define luametatex_revision 10 99# define luametatex_release 11 100# define luametatex_version_string "2.10.11" 101# define luametatex_development_id 20231231 102 103# define luametatex_name_camelcase "LuaMetaTeX" 104# define luametatex_name_lowercase "luametatex" 105# define luametatex_copyright_holder "Taco Hoekwater, Hans Hagen & Wolfgang Schuster" 106# define luametatex_bug_address "dev-context@ntg.nl" 107# define luametatex_support_address "context@ntg.nl" 108 109/*tex 110 111 One difference with \LUATEX\ is that we keep global variables that kind of belong together in 112 structures. This also has the advantage that we have more specific access (via a namespace) and 113 don't use that many macros (that can conflict later on). 114 115 We deliberately don't rewrite everything, if only because the original documentation suit apply 116 where possible and also because \quotation {What works, works}. 117 118*/ 119 120typedef struct version_state_info { 121 int version; 122 int revision; 123 int release; 124 int developmentid; 125 const char *verbose; 126 const char *banner; 127 const char *compiler; 128 const char *copyright; 129 int formatid; 130 int luaversionmajor; 131 int luaversionminor; 132 int luaversionrelease; 133 double luatexversion; 134 double luaversion; 135} version_state_info; 136 137extern version_state_info lmt_version_state; 138 139/*tex 140 141 This is actually the main headere file. Of course we could split it up and be more explicit in 142 other files but this is simple and just works. There is of course some overhead in loading 143 headers that are not used, but because compilation is simple and fast I don't care. 144 145*/ 146 147# include <stdarg.h> 148# include <string.h> 149# include <math.h> 150# include <stdlib.h> 151# include <errno.h> 152# include <float.h> 153# include <locale.h> 154# include <ctype.h> 155# include <stdint.h> 156# include <stdio.h> 157# include <time.h> 158# include <signal.h> 159# include <sys/stat.h> 160# include <stdbool.h> 161 162# ifdef _WIN32 163 # include <windows.h> 164 # include <winerror.h> 165 # include <fcntl.h> 166 # include <io.h> 167# else 168 # include <unistd.h> 169 # include <sys/time.h> 170# endif 171 172/*tex 173 174 We use stock \LUA\ where we only adapt the bytecode format flag so that we can use intermediate 175 \LUA\ versions without crashes due to different bytecode. Here are some constants that have to 176 be set: 177 178 \starttyping 179 # define LUAI_HASHLIMIT 6 180 # define LUA_USE_JUMPTABLE 0 181 # define LUA_BUILD_AS_DLL 0 182 # define LUA_CORE 0 183 \stoptyping 184 185 Earlier versions of \LUA\ an definitely \LUAJIT\ needed the |LUAI_HASHLIMIT| setting to be 186 adapted in order not to loose performance. This flag is no longer in \LUA\ version 5.4+. 187 188*/ 189 190# include "lua.h" 191# include "lauxlib.h" 192 193# define LUA_VERSION_STRING ("Lua " LUA_VERSION_MAJOR "." LUA_VERSION_MINOR "." LUA_VERSION_RELEASE) 194 195/*tex 196 197 The code in \LUAMETATEX\ is a follow up on \LUATEX\ which is itself a follow up on \PDFTEX\ 198 (and parts of \ALEPH). The original \PASCAL\ code has been converted \CCODE. Substantial amounts 199 of code were added over a decade. Stepwise artifacts have been removed (for instance originating 200 in the transations from \PASCAL, or from integration in the infrastructure), parts of code has 201 been rewritten. As much as possible we keep the old naming intact (so that most of the \TEX\ 202 documentation applies. However, as we now assume \CCODE, some things have changed. Among the 203 changes are handling datatypes and certain checks. For instance, when |null| is used this is 204 now always assumed to be |0|, so a zero test is also valid. Old side effects of zero nodes for 205 zero gluespecs are gone because these have been reimplemented. Of course we keep |NULL| as 206 abstraction for unset pointers. This way it's clear when we have a \CCODE\ pointer or a \TEX\ 207 managed one (where |null| or |0| means no node or token). 208 209 As with all \TEX\ engines, \LUATEX\ started out with the \PASCAL\ version of \TEX\ and as 210 mentioned we started with \PDFTEX. The first thing that was done (by Taco) was to create a 211 permanent \CCODE\ base instead of \PASCAL. In the process, some macros and library interfacing 212 wrappers were moved to the \LUATEX\ code base. Sometimes \PASCAL\ and \CCODE\ don't map well 213 end intermediate functions were used for that. Over time some artifacts that resulted from 214 automatic conversions from one to the other has been removed. 215 216 In the next stage of \LUATEX\ development, we went a but further and tried to get rid of more 217 dependencies. Among the rationales for this is that we depend on \LUA, and whatever works for 218 the \LUA\ codebase (which is quite portable) should also work for \LUATEX. But there are always 219 some overloads because (especially in \LUATEX\ where one can use \KPSE) the integration in a 220 \TEX\ ecosystem expects some behaviour with respect to files and running subprocesses and such. 221 In \LUAMETATEX\ there is less of that because \CONTEXT\ does more of that itself. 222 223 So, one of the biggest complications was the dependency on the \WEBC\ helpers and file system 224 interface. However, because that was already kind of isolated, it could be removed. If needed 225 we can always bring back \KPSE\ as an external library. In the process there can be some side 226 effects but in the end it gives a cleaner codebase and less depedencies. We suddenly don't need 227 all kind of tweaks to get the program compiled. 228 229 The \TEX\ memory model is based on packing data in memory words, but that concept is somewhat 230 fluid as in the past we had 16 byte processors too. However, we now mostly think in 32 bit and 231 internally \LUATEX\ will pack most of its node data in a multiples of 64 bits (called words). On 232 the one hand there is more memory involved but on the other hand it suits the architectures 233 well. In \LUAMETATEX\ we target 64 bit machines, but still provide binaries for 32 bit 234 architectures. The endianness related code has been dropped, simply because already for decades, 235 format files are not shared between platforms either. 236 237 Because \TEX\ efficiently implements its own memory management of nodes, the address of a node 238 is actually a number. Numbers like are sometimes indicates as |pointer|, but can also be called 239 |halfword|. Dimensions also fit into half a word and are called |scaled| but again we see them 240 being called |halfword|. What term is used depends a bit on the location and also on the 241 original code. For now we keep this mix but maybe some day we will normalize this. I did look 242 into more dynamic loading (only using the main memory numeric address pointers because that is 243 fast and efficient) but it makes the code more complex and probably hit performance badly. But 244 I keep an eye on it. 245 246 When we have halfwords representing pointers (into the main memory array) we indicate an unset 247 pointer as |null| (lowercase). But, because the usage of |null| and |0| was kind of mixed and 248 inconstent the |null| is only used to indicate zeroing a halfword encoded pointer. It will 249 always remain |0|. 250 251 We could reshuffle a lot more and normalize defines and enums but for now we stick to the way 252 it's done in order to divert not too much from the ancestors. However, in due time it can 253 evolve. Some constants used in \TEX\ the program now have a prefix |namespace_| or suffix 254 |_code| or |_cmd| in order not to clash with other usage. Some of these are in files like 255 |texcommands.h| and |texequivalents.h| but others end up in other |.h| files. This might change 256 but in the end it's not that important. Consider the spread a side effect of the still present 257 ideas of literate programming. 258 259 Some of the modules put data into the structures that could have been kept private but for now 260 I decided to be a bit consistent. However, of course there are still quite some private 261 variables left. 262 263*/ 264 265/*tex This is not used (yet) as I don't expect much from it, but \LUA\ has some of it. */ 266 267# if defined(__GNUC__) 268# define lmt_likely(x) (__builtin_expect(((x) != 0), 1)) 269# define lmt_unlikely(x) (__builtin_expect(((x) != 0), 0)) 270# else 271# define lmt_likely(x) (x) 272# define lmt_unlikely(x) (x) 273# endif 274 275# include "utilities/auxarithmetic.h" 276# include "utilities/auxmemory.h" 277# include "utilities/auxposit.h" 278# include "utilities/auxzlib.h" 279 280# include "tex/texmainbody.h" 281 282# include "lua/lmtinterface.h" 283# include "lua/lmtlibrary.h" 284# include "lua/lmttexiolib.h" 285 286# include "utilities/auxsystem.h" 287# include "utilities/auxsparsearray.h" 288# include "utilities/auxunistring.h" 289# include "utilities/auxfile.h" 290 291# include "libraries/hnj/hnjhyphen.h" 292 293# include "tex/texexpand.h" 294# include "tex/texmarks.h" 295# include "tex/texconditional.h" 296# include "tex/textextcodes.h" 297# include "tex/texmathcodes.h" 298# include "tex/texalign.h" 299# include "tex/texrules.h" 300/* "tex/texdirections.h" */ 301# include "tex/texerrors.h" 302# include "tex/texinputstack.h" 303# include "tex/texstringpool.h" 304# include "tex/textoken.h" 305# include "tex/texprinting.h" 306# include "tex/texfileio.h" 307# include "tex/texarithmetic.h" 308# include "tex/texnesting.h" 309# include "tex/texadjust.h" 310# include "tex/texinserts.h" 311# include "tex/texlocalboxes.h" 312# include "tex/texpackaging.h" 313# include "tex/texscanning.h" 314# include "tex/texbuildpage.h" 315# include "tex/texmaincontrol.h" 316# include "tex/texdumpdata.h" 317# include "tex/texmainbody.h" 318# include "tex/texnodes.h" 319# include "tex/texdirections.h" 320# include "tex/texlinebreak.h" 321# include "tex/texmath.h" 322# include "tex/texmlist.h" 323# include "tex/texcommands.h" 324# include "tex/texprimitive.h" 325# include "tex/texequivalents.h" 326# include "tex/texfont.h" 327# include "tex/texlanguage.h" 328 329# include "lua/lmtcallbacklib.h" 330# include "lua/lmttokenlib.h" 331# include "lua/lmtnodelib.h" 332# include "lua/lmtlanguagelib.h" 333# include "lua/lmtfontlib.h" 334# include "lua/lmtlualib.h" 335# include "lua/lmtluaclib.h" 336# include "lua/lmttexlib.h" 337# include "lua/lmtenginelib.h" 338 339/*tex 340 341 We use proper warnings, error messages, and confusion reporting instead of: 342 343 \starttyping 344 # ifdef HAVE_ASSERT_H 345 # include <assert.h> 346 # else 347 # define assert(expr) 348 # endif 349 \stoptyping 350 351 In fact, we don't use assert at all in \LUAMETATEX\ because if we need it we should do a decent 352 test and report an issue. In the \TEXLIVE\ eco system there can be assignments and function 353 calls in asserts which can disappear in case of e.g. compiling with msvc, so the above define 354 is even wrong! 355 356*/ 357 358// # ifndef _WIN32 359// 360// /* We don't want these use |foo_s| instead of |foo| messages. This will move. */ 361// 362// # define _CRT_SECURE_NO_WARNINGS 363// 364// # endif 365 366# endif 367 |