1/* 2 See license.txt in the root of this project. 3*/ 4 5# ifndef LMT_LUAMETATEX_H 6# define LMT_LUAMETATEX_H 7 8/*tex 9 10 The \LUATEX\ project started in 2005 with an experiments by Hartmut and me: adding the \LUA\ 11 Scripting language (that I knew from the \SCITE\ editor) to \PDFTEX. When we came to the 12 conclusion that a more tight integration made sense Taco did the impressive conversion from 13 \PASCAL\ |WEB\ to \CWEB. This happened in the perspective of the Oriental \TEX\ project, that 14 has as objective high quality Arabic typesetting. The way to achieve that was opening up the 15 font machinery and access to the paragraph building. It was an intense development period, 16 with Taco doing the coding, Hans exploring possibilities and extending \CONTEXT, and Idris 17 making fonts and testing. Taco and I discussed, compiled, accepted and rejected ideas. These 18 were interesting times! Over the years that we had used \TEX\ we could finally explore what we 19 had been talking about for years (long trips to user group meetings are good for that). We 20 ame to the first version(s) of \LUATEX\ with \CONTEXT\ \MKIV\ providing a testbed and as we 21 progressed we ended up with something we liked a lot. 22 23 After half a decade, where in the meantime Taco also had turned MetaPost into a library, we 24 had a version that had proved itself well. The following years, with Taco having less time 25 available, I started loking at the code. Some more got added to the Lua interfaces. Math got 26 split code paths and some new primitives were introduced. Luigi started taking care of managing 27 the code base so that I could cross compile for \MSWINDOWS. He also deals with the libraries 28 that were used and integration in \TEXLIVE\ and maintains the (by now stable) \METAPOST\ code 29 base. 30 31 After a while it became clear that users other than \CONTEXT\ wanted the program to stay as it 32 was and not introduce features or improve interfaces in ways that demanded a change in used 33 \LUA\ code. So, after a decade of development the official stable release took place. We already 34 had a split between stable (normally the \TEXLIVE\ release) and experimental (that we used for 35 development). However, in practice experimental versions were seen as real releases and we got 36 complaints that something could be broken (which actually is natural for an experimental 37 version). So, this split model didn't work out well in practice: you cannot explore and 38 experiment when you cannot play with yet unfinished code. 39 40 So at some point I decided that the best approach to a follow up, one not interfering with 41 usage of a stable \LUATEX, would be a more drastic split: the idea of \LUAMETATEX\ took shape. 42 This code base is the result of that. For whatever bad was introduced in \LUAMETATEX, and maybe 43 already before that in \LUATEX), you can blame me (Hans) and not Taco: Luigi consistently added 44 (hh) to the \LUATEX\ svn entries when that was feasible, so one can check where I messed up. 45 In the end all this work can be considered a co-product and the \CONTEXT\ (dev) community was 46 instrumental in this as well. 47 48 There are some fundamental changes: there is no backend but maybe I'll introduce a framework 49 for that at some point because the impact on performance has been quite noticeable (although 50 it has been compensated in the meantime). There is no support for \LUAJIT, because it doesn't 51 keep up with \LUA. Also, there is no support for \FFI, because that project is orphaned, but 52 there are other ways. Some more is delegated to \LUA, but also some more has been added to \TEX. 53 54 Over the 15 years that it took to go from the first version of \LUATEX\ in 2005 to the first 55 release of \LUAMETATEX\ in 2020 (although intermediate versions have always been good enough 56 to be used in production with \CONTEXT) I've written numerous articles in user group journals 57 as well as several presentations each year on progress and features. There are also wrapups 58 available in the \CONTEXT\ distribution that shed some light on how the developments 59 progress(ed). In the end it's all a work of many. There are no commercial interrests and 60 everything is done out of love for TeX and in free time, so take that into account when you 61 bark about code or documentation. 62 63 The \LUAMETATEX\ code base is maintained by Hans Hagen and Wolfgang Schuster (code, programming, 64 etc) with help from Mojca Miklavec (distribution, compile farm, etc) and Alan Braslau (testing, 65 feedback, etc). Of course with get help from all those \CONTEXT\ users who are always very 66 willing to test. 67 68 We start with the version numbers. While \LUATEX\ operates in the 100 range, the \LUAMETATEX\ 69 engine takes the 200 range. Revisions range from 00 upto 99 and the dates \unknown\ depend on 70 the mood. The |2.05.00| version with the development id |20200229| was more or less the first 71 official version, in the sense that most of the things on my initial todo list were done. It's 72 a kind of virtual date as it happens to be a leapyear. As with LuaTeX the .10 version will be 73 the first 'stable' one, released somewhere around the ConTeXt 2021 meeting. 74 75 2.08.18 : around TeXLive 2021 code freeze (so a bit of a reference version) 76 2.09.35 : near the end of 2021 (so close to the 2.10 release date) 77 2.09.55 : in July 2022 (the official release of the new math engine) 78 2.10.00 : a few days before the ctx 2022 meeting (starting September 19) 79 2.10.01 : mid October 2022 (some more upgrades of the math engine) 80 2.10.05 : late January 2023 81 2.10.07 : around TeXLive 2023 82 2.10.08 : close to BachoTeX 2023 83 2.10.09 : evolved in May/June 2023 (math & text linebreak experiments) 84 2.10.10 : around the ctx 2023 meeting 85 2.10.11 : first half of 2024 86 2.10.12 : around the ctx 2024 meeting 87 88 At some point the \CONTEXT\ group will be responsible for guaranteeing that the official version 89 is what comes with \CONTEXT\ and that long term support and stabilty is guaranteed and that no 90 feature creep or messing up happens. We'll see. 91 92 Hans Hagen 93 94 PS. Does anyone use (lib)cerf? If so, should I strip it a bit? 95 PS. Maybe use the Lua randomizer. 96 97*/ 98 99# include "tex/textypes.h" 100 101/*tex Currently LUAC_FORMAT is set to 2 awaiting an official version bump. */ 102 103# define luametatex_majorversion 2 104# define luametatex_minorversion 11 105# define luametatex_version 211 106# define luametatex_revision 0 107# define luametatex_release 06 108# define luametatex_version_string "2.11.06" 109# define luametatex_version_number 211.6 110# define luametatex_development_id 20250218 111# define luametatex_name_camelcase "LuaMetaTeX" 112# define luametatex_name_lowercase "luametatex" 113# define luametatex_copyright_holder "Taco Hoekwater, Hans Hagen, Wolfgang Schuster & Mikael Sundqvist" 114# define luametatex_bug_address "dev-context@ntg.nl" 115# define luametatex_support_address "context@ntg.nl" 116 117/*tex 118 119 One difference with \LUATEX\ is that we keep global variables that kind of belong together in 120 structures. This also has the advantage that we have more specific access (via a namespace) and 121 don't use that many macros (that can conflict later on). 122 123 We deliberately don't rewrite everything, if only because the original documentation suit apply 124 where possible and also because \quotation {What works, works}. 125 126*/ 127 128typedef struct version_state_info { 129 int majorversion; 130 int minorversion; 131 int version; 132 int revision; 133 int release; 134 int developmentid; 135 const char *verbose; 136 const char *banner; 137 const char *compiler; 138 const char *copyright; 139 int formatid; 140 int luaversionmajor; 141 int luaversionminor; 142 int luaversionrelease; 143 int luaformat; 144 double luatexversion; 145 double luaversion; 146} version_state_info; 147 148extern version_state_info lmt_version_state; 149 150/*tex 151 152 This is actually the main header file. Of course we could split it up and be more explicit in 153 other files but this is simple and just works. There is of course some overhead in loading 154 headers that are not used, but because compilation is simple and fast I don't care. 155 156*/ 157 158# include <stdarg.h> 159# include <string.h> 160# include <math.h> 161# include <stdlib.h> 162# include <errno.h> 163# include <float.h> 164# include <locale.h> 165# include <ctype.h> 166# include <stdint.h> 167# include <stdio.h> 168# include <time.h> 169# include <signal.h> 170# include <sys/stat.h> 171# include <stdbool.h> 172 173# ifdef _WIN32 174 # include <windows.h> 175 # include <winerror.h> 176 # include <fcntl.h> 177 # include <io.h> 178# else 179 # include <unistd.h> 180 # include <sys/time.h> 181# endif 182 183/*tex 184 185 We use stock \LUA\ where we only adapt the bytecode format flag so that we can use intermediate 186 \LUA\ versions without crashes due to different bytecode. Here are some constants that have to 187 be set: 188 189 \starttyping 190 # define LUAI_HASHLIMIT 6 191 # define LUA_USE_JUMPTABLE 0 192 # define LUA_BUILD_AS_DLL 0 193 # define LUA_CORE 0 194 \stoptyping 195 196 Earlier versions of \LUA\ an definitely \LUAJIT\ needed the |LUAI_HASHLIMIT| setting to be 197 adapted in order not to loose performance. This flag is no longer in \LUA\ version 5.4+. 198 199*/ 200 201# include "lua.h" 202# include "lauxlib.h" 203# include "lundump.h" 204 205# define LUA_VERSION_STRING ("Lua " LUA_VERSION_MAJOR "." LUA_VERSION_MINOR "." LUA_VERSION_RELEASE) 206 207/*tex 208 209 The code in \LUAMETATEX\ is a follow up on \LUATEX\ which is itself a follow up on \PDFTEX\ 210 (and parts of \ALEPH). The original \PASCAL\ code has been converted \CCODE. Substantial amounts 211 of code were added over a decade. Stepwise artifacts have been removed (for instance originating 212 in the transations from \PASCAL, or from integration in the infrastructure), parts of code has 213 been rewritten. As much as possible we keep the old naming intact (so that most of the \TEX\ 214 documentation applies. However, as we now assume \CCODE, some things have changed. Among the 215 changes are handling datatypes and certain checks. For instance, when |null| is used this is 216 now always assumed to be |0|, so a zero test is also valid. Old side effects of zero nodes for 217 zero gluespecs are gone because these have been reimplemented. Of course we keep |NULL| as 218 abstraction for unset pointers. This way it's clear when we have a \CCODE\ pointer or a \TEX\ 219 managed one (where |null| or |0| means no node or token). 220 221 As with all \TEX\ engines, \LUATEX\ started out with the \PASCAL\ version of \TEX\ and as 222 mentioned we started with \PDFTEX. The first thing that was done (by Taco) was to create a 223 permanent \CCODE\ base instead of \PASCAL. In the process, some macros and library interfacing 224 wrappers were moved to the \LUATEX\ code base. Sometimes \PASCAL\ and \CCODE\ don't map well 225 end intermediate functions were used for that. Over time some artifacts that resulted from 226 automatic conversions from one to the other has been removed. 227 228 In the next stage of \LUATEX\ development, we went a but further and tried to get rid of more 229 dependencies. Among the rationales for this is that we depend on \LUA, and whatever works for 230 the \LUA\ codebase (which is quite portable) should also work for \LUATEX. But there are always 231 some overloads because (especially in \LUATEX\ where one can use \KPSE) the integration in a 232 \TEX\ ecosystem expects some behaviour with respect to files and running subprocesses and such. 233 In \LUAMETATEX\ there is less of that because \CONTEXT\ does more of that itself. 234 235 So, one of the biggest complications was the dependency on the \WEBC\ helpers and file system 236 interface. However, because that was already kind of isolated, it could be removed. If needed 237 we can always bring back \KPSE\ as an external library. In the process there can be some side 238 effects but in the end it gives a cleaner codebase and less depedencies. We suddenly don't need 239 all kind of tweaks to get the program compiled. 240 241 The \TEX\ memory model is based on packing data in memory words, but that concept is somewhat 242 fluid as in the past we had 16 byte processors too. However, we now mostly think in 32 bit and 243 internally \LUATEX\ will pack most of its node data in a multiples of 64 bits (called words). On 244 the one hand there is more memory involved but on the other hand it suits the architectures 245 well. In \LUAMETATEX\ we target 64 bit machines, but still provide binaries for 32 bit 246 architectures. The endianness related code has been dropped, simply because already for decades, 247 format files are not shared between platforms either. 248 249 Because \TEX\ efficiently implements its own memory management of nodes, the address of a node 250 is actually a number. Numbers like are sometimes indicates as |pointer|, but can also be called 251 |halfword|. Dimensions also fit into half a word and are called |scaled| but again we see them 252 being called |halfword|. What term is used depends a bit on the location and also on the 253 original code. For now we keep this mix but maybe some day we will normalize this. I did look 254 into more dynamic loading (only using the main memory numeric address pointers because that is 255 fast and efficient) but it makes the code more complex and probably hit performance badly. But 256 I keep an eye on it. 257 258 When we have halfwords representing pointers (into the main memory array) we indicate an unset 259 pointer as |null| (lowercase). But, because the usage of |null| and |0| was kind of mixed and 260 inconstent the |null| is only used to indicate zeroing a halfword encoded pointer. It will 261 always remain |0|. 262 263 We could reshuffle a lot more and normalize defines and enums but for now we stick to the way 264 it's done in order to divert not too much from the ancestors. However, in due time it can 265 evolve. Some constants used in \TEX\ the program now have a prefix |namespace_| or suffix 266 |_code| or |_cmd| in order not to clash with other usage. Some of these are in files like 267 |texcommands.h| and |texequivalents.h| but others end up in other |.h| files. This might change 268 but in the end it's not that important. Consider the spread a side effect of the still present 269 ideas of literate programming. 270 271 Some of the modules put data into the structures that could have been kept private but for now 272 I decided to be a bit consistent. However, of course there are still quite some private 273 variables left. 274 275*/ 276 277/*tex This is not used (yet) as I don't expect much from it, but \LUA\ has some of it. */ 278 279# if defined(__GNUC__) 280 // Lua: 281 // # define lmt_likely(x) (__builtin_expect(((x) != 0), 1)) 282 // # define lmt_unlikely(x) (__builtin_expect(((x) != 0), 0)) 283 // Kernel: 284 # define lmt_likely(x) (__builtin_expect(!!(x), 1)) 285 # define lmt_unlikely(x) (__builtin_expect(!!(x), 0)) 286# else 287# define lmt_likely(x) (x) 288# define lmt_unlikely(x) (x) 289# endif 290 291# include "utilities/auxarithmetic.h" 292# include "utilities/auxmemory.h" 293# include "utilities/auxposit.h" 294# include "utilities/auxzlib.h" 295 296# include "tex/texmainbody.h" 297 298# include "lua/lmtinterface.h" 299# include "lua/lmtlibrary.h" 300# include "lua/lmttexiolib.h" 301 302# include "utilities/auxsystem.h" 303# include "utilities/auxsparsearray.h" 304# include "utilities/auxunistring.h" 305# include "utilities/auxfile.h" 306 307# include "libraries/hnj/hnjhyphen.h" 308 309# include "tex/texexpand.h" 310# include "tex/texmarks.h" 311# include "tex/texconditional.h" 312# include "tex/textextcodes.h" 313# include "tex/texmathcodes.h" 314# include "tex/texalign.h" 315# include "tex/texrules.h" 316/* "tex/texdirections.h" */ 317# include "tex/texerrors.h" 318# include "tex/texinputstack.h" 319# include "tex/texstringpool.h" 320# include "tex/textoken.h" 321# include "tex/texprinting.h" 322# include "tex/texfileio.h" 323# include "tex/texarithmetic.h" 324# include "tex/texnesting.h" 325# include "tex/texadjust.h" 326# include "tex/texinserts.h" 327# include "tex/texlocalboxes.h" 328# include "tex/texpackaging.h" 329# include "tex/texscanning.h" 330# include "tex/texbuildpage.h" 331# include "tex/texmaincontrol.h" 332# include "tex/texdumpdata.h" 333# include "tex/texmainbody.h" 334# include "tex/texnodes.h" 335# include "tex/texspecifications.h" 336# include "tex/texdirections.h" 337# include "tex/texlinebreak.h" 338# include "tex/texmath.h" 339# include "tex/texmlist.h" 340# include "tex/texcommands.h" 341# include "tex/texprimitive.h" 342# include "tex/texequivalents.h" 343# include "tex/texfont.h" 344# include "tex/texbalance.h" 345# include "tex/texlanguage.h" 346# include "lua/lmtcallbacklib.h" 347# include "lua/lmttokenlib.h" 348# include "lua/lmtnodelib.h" 349# include "lua/lmtlanguagelib.h" 350# include "lua/lmtfontlib.h" 351# include "lua/lmtlualib.h" 352# include "lua/lmtluaclib.h" 353# include "lua/lmttexlib.h" 354# include "lua/lmtenginelib.h" 355 356/*tex 357 358 We use proper warnings, error messages, and confusion reporting instead of: 359 360 \starttyping 361 # ifdef HAVE_ASSERT_H 362 # include <assert.h> 363 # else 364 # define assert(expr) 365 # endif 366 \stoptyping 367 368 In fact, we don't use assert at all in \LUAMETATEX\ because if we need it we should do a decent 369 test and report an issue. In the \TEXLIVE\ eco system there can be assignments and function 370 calls in asserts which can disappear in case of e.g. compiling with msvc, so the above define 371 is even wrong! 372 373*/ 374 375// # ifndef _WIN32 376// 377// /* We don't want these use |foo_s| instead of |foo| messages. This will move. */ 378// 379// # define _CRT_SECURE_NO_WARNINGS 380// 381// # endif 382 383# endif 384 |