% language=us runpath=texruns:manuals/luametatex \environment luametatex-style \startcomponent luametatex-lua \startchapter[reference=lua,title={Using \LUAMETATEX}] \startsection[title={Initialization},reference=init] \startsubsection[title={A bare bone engine}] Although the \LUAMETATEX\ engine will start up when you call the program it will not do much useful. You can compare it to computer hardware without (high level) operating system with a \TEX\ kernel being the bios. It can interpret \TEX\ code but for typesetting you need a reasonable setup. You also need to load fonts, and for output you need a backend, and both can be implemented in \LUA. If you don't like that and want to get up and running immediately, you will be more happy with \LUATEX, \PDFTEX\ or \XETEX, combined with your favorite macro package. If you just want to play around you can install the \CONTEXT\ distribution which (including manuals and some fonts) is tiny compared to a full \TEXLIVE\ installation and can be run alongside it without problems. If there are issues you can go to the usual \CONTEXT\ support platforms and seek help where you can find the people who made \LUATEX\ and \LUAMETATEX. \stopsubsection \startsubsection[title={\LUAMETATEX\ as a \LUA\ interpreter}] \topicindex {initialization} \topicindex {\LUA+interpreter} Although \LUAMETATEX\ is primarily meant as a \TEX\ engine, it can also serve as a stand alone \LUA\ interpreter. There are two ways to make \LUAMETATEX\ behave like a standalone \LUA\ interpreter. The first method uses the command line option \type {--luaonly} followed by a filename. The second is more automatic: if the only non|-|option argument (file) on the commandline has the extension \type {lmt} or \type {lua}. The \type {luc} extension has been dropped because bytecode compiled files are not portable and one can always load indirect. The \type {lmt} suffix is more \CONTEXT\ specific and makes it possible to have files for \LUATEX\ and \LUAMETATEX\ alongside. In this mode, it will set \LUA's \type {arg[0]} to the found script name, pushing preceding options in negative values and the rest of the command line in the positive values, just like the \LUA\ interpreter does. \LUAMETATEX\ will exit immediately after executing the specified \LUA\ script and is, in effect, a somewhat bulky stand alone \LUA\ interpreter with a bunch of extra preloaded libraries. But we really want to keep the binary small, if possible below the 3MB which is okay for a script engine. When no argument is given, \LUAMETATEX\ will look for a \LUA\ file with the same name as the binary and run that one when present. This makes it possible to use the engine as a stub. For instance, in \CONTEXT\ a symlink from \type {mtxrun} to type {luametatex} will run the \type {mtxrun.lmt} or \type {mtxrun.lua} script when present in the same path as the binary itself. As mentioned before first checking for (\CONTEXT) \type {lmt} files permits different files for different engines in the same path. \stopsubsection \startsubsection[title={Other commandline processing}] \topicindex {command line} When the \LUAMETATEX\ executable starts, it looks for the \type {--lua} command line option. If there is no \type {--lua} option, the command line is interpreted in a similar fashion as the other \TEX\ engines. All options are accepted but only some are understood by \LUAMETATEX\ itself: \starttabulate[|l|p|] \DB commandline argument \BC explanation \NC \NR \TB \NC \type{--credits} \NC display credits and exit \NC \NR \NC \type{--fmt=FORMAT} \NC load the format file \type {FORMAT} \NC\NR \NC \type{--help} \NC display help and exit \NC\NR \NC \type{--ini} \NC be \type {iniluatex}, for dumping formats \NC\NR \NC \type{--jobname=STRING} \NC set the job name to \type {STRING} \NC \NR \NC \type{--lua=FILE} \NC load and execute a \LUA\ initialization script \NC\NR \NC \type{--version} \NC display version and exit \NC \NR \LL \stoptabulate There are less options than with \LUATEX, because one has to deal with them in \LUA\ anyway. So for instance there are no options to enter a safer mode or control executing programs because this can easily be achieved with a startup \LUA\ script. Next the initialization script is loaded and executed. From within the script, the entire command line is available in the \LUA\ table \type {arg}, beginning with \type {arg[0]}, containing the name of the executable. As consequence warnings about unrecognized options are suppressed. Command line processing happens very early on. So early, in fact, that none of \TEX's initializations have taken place yet. The \LUA\ libraries that don't deal with \TEX\ are initialized rather soon so you have these available. \LUAMETATEX\ allows some of the command line options to be overridden by reading values from the \type {texconfig} table at the end of script execution (see the description of the \type {texconfig} table later on in this document for more details on which ones exactly). The value to use for \prm {jobname} is decided as follows: \startitemize \startitem If \type {--jobname} is given on the command line, its argument will be the value for \prm {jobname}, without any changes. The argument will not be used for actual input so it need not exist. The \type {--jobname} switch only controls the \prm {jobname} setting. \stopitem \startitem Otherwise, \prm {jobname} will be the name of the first file that is read from the file system, with any path components and the last extension (the part following the last \type {.}) stripped off. \stopitem \startitem There is an exception to the previous point: if the command line goes into interactive mode (by starting with a command) and there are no files input via \prm {everyjob} either, then the \prm {jobname} is set to \type {texput} as a last resort. \stopitem \stopitemize So let's summarize this. The handling of what is called jobname is a bit complex. There can be explicit names set on the command line but when not set they can be taken from the \type {texconfig} table. \starttabulate[|l|T|T|T|] \NC startup filename \NC --lua \NC a \LUA\ file \NC \NC \NR \NC startup jobname \NC --jobname \NC a \TEX\ tex \NC texconfig.jobname \NC \NR \NC startup dumpname \NC --fmt \NC a format file \NC texconfig.formatname \NC \NR \stoptabulate These names are initialized according to \type {--luaonly} or the first filename seen in the list of options. Special treatment of \type {&} and \type {*} as well as interactive startup is gone but we still enter \TEX\ via an forced \type {\input} into the input buffer. \footnote {This might change at some point into an explicit loading triggered via \LUA.} When we are in \TEX\ mode at some point the engine needs a filename, for instance for opening a log file. At that moment the set jobname becomes the internal one and when it has not been set which internalized to jobname but when not set becomes \type {texput}. When you see a \type {texput.log} file someplace on your system it normally indicates a bad run. When running on \MSWINDOWS\ the command line, filenames, environment variable access etc.\ internally uses the current code page but to the user is exposed as \UTF8. Normally users won't notice this. % fileio_state .jobname : a tex string (set when a (log) file is opened) % engine_state .startup_jobname : handles by option parser % environment_state.input_name : temporary interceptor There is an extra options \type{--permitloadlib} that needs to be given when you load external libraries via \LUA. Although you could manage this via \LUA\ itself in a startup script, the reason for having this as option is the wish for security (at some point that became a demand for \LUATEX), so this might give an extra feeling of protection. \stopsubsection \stopsection \startsection[title={\LUA\ behaviour}] \startsubsection[title={The \LUA\ version}] \topicindex {\LUA+libraries} \topicindex {\LUA+extensions} We currently use \LUA\ 5.4 and will follow developments of the language but normally with some delay. Therefore the user needs to keep an eye on (subtle) differences in successive versions of the language. Here is an example of one aspect. \LUA s \type {tostring} function (and \type {string.format}) may return values in scientific notation, thereby confusing the \TEX\ end of things when it is used as the right|-|hand side of an assignment to a \prm {dimen} or \prm {count}. The output of these serializers also depend on the \LUA\ version, so in \LUA\ 5.3 you can get different output than from 5.2. It is best not to depend the automatic cast from string to number and vise versa as this can change in future versions. \stopsubsection \startsubsection[title={Locales}] \index {locales} In stock \LUA, many things depend on the current locale. In \LUAMETATEX, we can't do that, because it makes documents non|-|portable. While \LUAMETATEX\ is running if forces the following locale settings: \starttyping LC_CTYPE=C LC_COLLATE=C LC_NUMERIC=C \stoptyping There is no way to change that as it would interfere badly with the often language specific conversions needed at the \TEX\ end. \stopsubsection \stopsection \startsection[title={\LUA\ modules}] \topicindex {\LUA+libraries} \topicindex {\LUA+modules} Of course the regular \LUA\ modules are present. In addition we provide the \type {lpeg} library by Roberto Ierusalimschy, This library is not \UNICODE|-|aware, but interprets strings on a byte|-|per|-|byte basis. This mainly means that \type {lpeg.S} cannot be used with \UTF8 characters that need more than one byte, and thus \type {lpeg.S} will look for one of those two bytes when matching, not the combination of the two. The same is true for \type {lpeg.R}, although the latter will display an error message if used with multibyte characters. Therefore \type {lpeg.R('aä')} results in the message \type {bad argument #1 to 'R' (range must have two characters)}, since to \type {lpeg}, \type {ä} is two 'characters' (bytes), so \type {aä} totals three. In practice this is no real issue and with some care you can deal with \UNICODE\ just fine. There are some more libraries present. These are discussed on a later chapter. For instance we embed \type {luasocket} but contrary to \LUATEX\ don't embed the related \LUA\ code. The \type {luafilesystem} module has been replaced by a more efficient one that also deals with the \MSWINDOWS\ file and environment properties better (\UNICODE\ support in \MSWINDOWS\ dates from before \UTF8 became dominant so we need to deal with wide \UNICODE16). There are more extensive math libraries and there are libraries that deal with encryption and compression. There are also some optional libraries that we do interface but that are loaded on demand. The interfaces are as minimal as can be because we so much in \LUA, which also means that one can tune behaviour to usage better. \stopsection \startsection[title={Files}] \startsubsection[title={File syntax}] \topicindex {files+names} \LUAMETATEX\ will accept a braced argument as a file name: \starttyping \input {plain} \openin 0 {plain} \stoptyping This allows for embedded spaces, without the need for double quotes. Macro expansion takes place inside the argument. Keep in mind that as side effect of delegating \IO\ to \LUA\ the \tex {openin} primitive is not provided by the engine and has to be implemented by the macro package. This also means that the limit on the number of open files is not enforced by the engine. \stopsubsection \startsubsection[title={Writing to file}] \topicindex {files+writing} Writing to a file in \TEX\ has two forms: delayed and immediate. Delayed writing means that the to be written text is anchored in the node list and flushed by the backend. As all \IO\ is delegated to \LUA, this also means that it has to deal with distinction. In \LUATEX\ the number of open files was already bumped to 127, but in \LUAMETATEX\ it depends on the macro package. The special meaning of channel 18 was already dropped in \LUATEX\ because we have \type {os.execute}. \stopsubsection \stopsection \startsection[title={Testing}] \topicindex {testing} For development reasons you can influence the used startup date and time. By setting the \type {start_time} variable in the \type {texconfig} table; as with other variables we use the internal name there. When Universal Time is needed, set the entry \type {use_utc_time} in the \type {texconfig} table. In \CONTEXT\ we provide the command line argument \type {--nodates} that does a bit more than disabling dates; it avoids time dependent information in the output file for instance. \stopsection \startsection[reference=internals,title={The internals}] \topicindex{nodes} \topicindex{boxes} \topicindex{\LUA} This is a reference manual and not a tutorial. This means that we discuss changes relative to traditional \TEX\ and also present new (or extended) functionality. As a consequence we will refer to concepts that we assume to be known or that might be explained later. Because the \LUATEX\ and \LUAMETATEX\ engines open up \TEX\ there's suddenly quite some more to explain, especially about the way a (to be) typeset stream moves through the machinery. However, discussing all that in detail makes not much sense, because deep knowledge is only relevant for those who write code not possible with regular \TEX\ and who are already familiar with these internals (or willing to spend time on figuring it out). So, the average user doesn't need to know much about what is in this manual. For instance fonts and languages are normally dealt with in the macro package that you use. Messing around with node lists is also often not really needed at the user level. If you do mess around, you'd better know what you're dealing with. Reading \quotation {The \TEX\ Book} by Donald Knuth is a good investment of time then also because it's good to know where it all started. A more summarizing overview is given by \quotation {\TEX\ by Topic} by Victor Eijkhout. You might want to peek in \quotation {The \ETEX\ manual} too. But \unknown\ if you're here because of \LUA, then all you need to know is that you can call it from within a run. If you want to learn the language, just read the well written \LUA\ book. The macro package that you use probably will provide a few wrapper mechanisms but the basic \prm {directlua} command that does the job is: \starttyping \directlua{tex.print("Hi there")} \stoptyping You can put code between curly braces but if it's a lot you can also put it in a file and load that file with the usual \LUA\ commands. If you don't know what this means, you definitely need to have a look at the \LUA\ book first. If you still decide to read on, then it's good to know what nodes are, so we do a quick introduction here. If you input this text: \starttyping Hi There ... \stoptyping eventually we will get a linked lists of nodes, which in \ASCII\ art looks like: \starttyping H <=> i <=> [glue] <=> T <=> h <=> e <=> r <=> e ... \stoptyping When we have a paragraph, we actually get something like this, where a \type {par} node stores some metadata and is followed by a \type {hlist} flagged as indent box: \starttyping [par] <=> [hlist] <=> H <=> i <=> [glue] <=> T <=> h <=> e <=> r <=> e ... \stoptyping Each character becomes a so called glyph node, a record with properties like the current font, the character code and the current language. Spaces become glue nodes. There are many node types and nodes can have many properties but that will be discussed later. Each node points back to a previous node or next node, given that these exist. Sometimes multiple characters are represented by one glyph (shape), so one can also get: \starttyping [par] <=> [hlist] <=> H <=> i <=> [glue] <=> Th <=> e <=> r <=> e ... \stoptyping And maybe some characters get positioned relative to each other, so we might see: \starttyping [par] <=> [hlist] <=> H <=> [kern] <=> i <=> [glue] <=> Th <=> e <=> r <=> e ... \stoptyping Actually, the above representation is one view, because in \LUAMETATEX\ we can choose for this: \starttyping [par] <=> [glue] <=> H <=> [kern] <=> i <=> [glue] <=> Th <=> e <=> r <=> e ... \stoptyping where glue (currently fixed) is used instead of an empty hlist (think of a \type {\hbox}). Options like this are available because want a certain view on these lists from the \LUA\ end and the result being predicable is part of that. It's also good to know beforehand that \TEX\ is basically centered around creating paragraphs and pages. The par builder takes a list and breaks it into lines. At some point horizontal blobs are wrapped into vertical ones. Lines are so called boxes and can be separated by glue, penalties and more. The page builder accumulates lines and when feasible triggers an output routine that will take the list so far. Constructing the actual page is not part of \TEX\ but done using primitives that permit manipulation of boxes. The result is handled back to \TEX\ and flushed to a (often \PDF) file. \starttyping \setbox\scratchbox\vbox\bgroup line 1\par line 2 \egroup \showbox\scratchbox \stoptyping The above code produces the next log lines that reveal how the engines sees a paragraph (wrapped in a \type {\vbox}): \starttyping[style=small] 1:4: > \box257= 1:4: \vbox[normal][16=1,17=1,47=1], width 483.69687, height 27.58083, depth 0.1416, direction l2r 1:4: .\list 1:4: ..\hbox[line][16=1,17=1,47=1], width 483.69687, height 7.59766, depth 0.1416, glue 455.40097fil, direction l2r 1:4: ...\list 1:4: ....\glue[left hang][16=1,17=1,47=1] 0.0pt 1:4: ....\glue[left][16=1,17=1,47=1] 0.0pt 1:4: ....\glue[parfillleft][16=1,17=1,47=1] 0.0pt 1:4: ....\par[newgraf][16=1,17=1,47=1], hangafter 1, hsize 483.69687, pretolerance 100, tolerance 3000, adjdemerits 10000, linepenalty 10, doublehyphendemerits 10000, finalhyphendemerits 5000, clubpenalty 2000, widowpenalty 2000, brokenpenalty 100, emergencystretch 12.0, parfillskip 0.0pt plus 1.0fil, hyphenationmode 499519 1:4: ....\glue[indent][16=1,17=1,47=1] 0.0pt 1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+00006C l 1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+000069 i 1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+00006E n 1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+000065 e 1:4: ....\glue[space][16=1,17=1,47=1] 3.17871pt plus 1.58936pt minus 1.05957pt, font 30 1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+000031 1 1:4: ....\penalty[line][16=1,17=1,47=1] 10000 1:4: ....\glue[parfill][16=1,17=1,47=1] 0.0pt plus 1.0fil 1:4: ....\glue[right][16=1,17=1,47=1] 0.0pt 1:4: ....\glue[right hang][16=1,17=1,47=1] 0.0pt 1:4: ..\glue[par][16=1,17=1,47=1] 5.44995pt plus 1.81665pt minus 1.81665pt 1:4: ..\glue[baseline][16=1,17=1,47=1] 6.79396pt 1:4: ..\hbox[line][16=1,17=1,47=1], width 483.69687, height 7.59766, depth 0.1416, glue 455.40097fil, direction l2r 1:4: ...\list 1:4: ....\glue[left hang][16=1,17=1,47=1] 0.0pt 1:4: ....\glue[left][16=1,17=1,47=1] 0.0pt 1:4: ....\glue[parfillleft][16=1,17=1,47=1] 0.0pt 1:4: ....\par[newgraf][16=1,17=1,47=1], hangafter 1, hsize 483.69687, pretolerance 100, tolerance 3000, adjdemerits 10000, linepenalty 10, doublehyphendemerits 10000, finalhyphendemerits 5000, clubpenalty 2000, widowpenalty 2000, brokenpenalty 100, emergencystretch 12.0, parfillskip 0.0pt plus 1.0fil, hyphenationmode 499519 1:4: ....\glue[indent][16=1,17=1,47=1] 0.0pt 1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+00006C l 1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+000069 i 1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+00006E n 1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+000065 e 1:4: ....\glue[space][16=1,17=1,47=1] 3.17871pt plus 1.58936pt minus 1.05957pt, font 30 1:4: ....\glyph[32768][16=1,17=1,47=1], language (n=1,l=2,r=3), hyphenationmode 499519, options 128 , font <30: DejaVuSerif @ 10.0pt>, glyph U+000032 2 1:4: ....\penalty[line][16=1,17=1,47=1] 10000 1:4: ....\glue[parfill][16=1,17=1,47=1] 0.0pt plus 1.0fil 1:4: ....\glue[right][16=1,17=1,47=1] 0.0pt 1:4: ....\glue[right hang][16=1,17=1,47=1] 0.0pt \stoptyping The \LUATEX\ engine provides hooks for \LUA\ code at nearly every reasonable point in the process: collecting content, hyphenating, applying font features, breaking into lines, etc. This means that you can overload \TEX's natural behaviour, which still is the benchmark. When we refer to \quote {callbacks} we means these hooks. The \TEX\ engine itself is pretty well optimized but when you kick in much \LUA\ code, you will notices that performance drops. Don't blame and bother the authors with performance issues. In \CONTEXT\ over 50\% of the time can be spent in \LUA, but so far we didn't get many complaints about efficiency. Adding more callbacks makes no sense, also because at some point the performance hit gets too large. There are plenty of ways to achieve goals. For that reason: take remarks about \LUATEX, features, potential, performance etc.\ with a natural grain of salt. Where plain \TEX\ is basically a basic framework for writing a specific style, macro packages like \CONTEXT\ and \LATEX\ provide the user a whole lot of additional tools to make documents look good. They hide the dirty details of font management, language support, turning structure into typeset results, wrapping pages, including images, and so on. You should be aware of the fact that when you hook in your own code to manipulate lists, this can interfere with the macro package that you use. Each successive step expects a certain result and if you mess around to much, the engine eventually might bark and quit. It can even crash, because testing everywhere for what users can do wrong is no real option. When you read about nodes in the following chapters it's good to keep in mind what commands relate to them. Here are a few: \starttabulate[|l|l|p|] \DB command \BC node \BC explanation \NC \NR \TB \NC \prm {hbox} \NC \nod {hlist} \NC horizontal box \NC \NR \NC \prm {vbox} \NC \nod {vlist} \NC vertical box with the baseline at the bottom \NC \NR \NC \prm {vtop} \NC \nod {vlist} \NC vertical box with the baseline at the top \NC \NR \NC \prm {hskip} \NC \nod {glue} \NC horizontal skip with optional stretch and shrink \NC \NR \NC \prm {vskip} \NC \nod {glue} \NC vertical skip with optional stretch and shrink \NC \NR \NC \prm {kern} \NC \nod {kern} \NC horizontal or vertical fixed skip \NC \NR \NC \prm {discretionary} \NC \nod {disc} \NC hyphenation point (pre, post, replace) \NC \NR \NC \prm {char} \NC \nod {glyph} \NC a character \NC \NR \NC \prm {hrule} \NC \nod {rule} \NC a horizontal rule \NC \NR \NC \prm {vrule} \NC \nod {rule} \NC a vertical rule \NC \NR \NC \prm {textdirection} \NC \nod {dir} \NC a change in text direction \NC \NR \LL \stoptabulate Whatever we feed into \TEX\ at some point becomes a token which is either interpreted directly or stored in a linked list. A token is just a number that encodes a specific command (operator) and some value (operand) that further specifies what that command is supposed to do. In addition to an interface to nodes, there is an interface to tokens, as later chapters will demonstrate. Text (interspersed with macros) comes from an input medium. This can be a file, token list, macro body cq.\ arguments, some internal quantity (like a number), \LUA, etc. Macros get expanded. In the process \TEX\ can enter a group. Inside the group, changes to registers get saved on a stack, and restored after leaving the group. When conditionals are encountered, another kind of nesting happens, and again there is a stack involved. Tokens, expansion, stacks, input levels are all terms used in the next chapters. Don't worry, they loose their magic once you use \TEX\ a lot. You have access to most of the internals and when not, at least it is possible to query some state we're in or level we're at. When we talk about pack(ag)ing it can mean two things. When \TEX\ has consumed some tokens that represent text they are added to the current list. When the text is put into a so called \prm {hbox} (for instance a line in a paragraph) it (normally) first gets hyphenated, next ligatures are build, and finally kerns are added. Each of these stages can be overloaded using \LUA\ code. When these three stages are finished, the dimension of the content is calculated and the box gets its width, height and depth. What happens with the box depends on what macros do with it. The other thing that can happen is that the text starts a new paragraph. In that case some information is stored in a leading \type {par} node. Then indentation is appended and the paragraph ends with some glue. Again the three stages are applied but this time afterwards, the long line is broken into lines and the result is either added to the content of a box or to the main vertical list (the running text so to say). This is called par building. At some point \TEX\ decides that enough is enough and it will trigger the page builder. So, building is another concept we will encounter. Another example of a builder is the one that turns an intermediate math list into something typeset. Wrapping something in a box is called packing. Adding something to a list is described in terms of contributing. The more complicated processes are wrapped into builders. For now this should be enough to enable you to understand the next chapters. The text is not as enlightening and entertaining as Don Knuths books, sorry. \stopsection \stopchapter \stopcomponent