font-ots.lmt /size: 173 Kb    last modification: 2024-01-16 09:02
1if not modules then modules = { } end modules ['font-ots'] = { -- sequences
2    version   = 1.001,
3    optimize  = true,
4    comment   = "companion to font-ini.mkiv",
5    author    = "Hans Hagen, PRAGMA-ADE, Hasselt NL",
6    copyright = "PRAGMA ADE / ConTeXt Development Team",
7    license   = "see context related readme files",
8}
9
10-- I need to check the description at the microsoft site ... it has been improved so
11-- maybe there are some interesting details there. Most below is based on old and
12-- incomplete documentation and involved quite a bit of guesswork (checking with the
13-- abstract uniscribe of those days. But changing things is tricky!
14--
15-- This module is a bit more split up that I'd like but since we also want to test
16-- with plain TeX it has to be so. This module is part of ConTeXt and discussion
17-- about improvements and functionality mostly happens on the ConTeXt mailing list.
18--
19-- The specification of OpenType is (or at least decades ago was) kind of vague.
20-- Apart from a lack of a proper free specifications there's also the problem that
21-- Microsoft and Adobe may have their own interpretation of how and in what order to
22-- apply features. In general the Microsoft website has more detailed specifications
23-- and is a better reference. There is also some information in the FontForge help
24-- files. In the end we rely most on the Microsoft specification.
25--
26-- Because there is so much possible, fonts might contain bugs and/or be made to
27-- work with certain rederers. These may evolve over time which may have the side
28-- effect that suddenly fonts behave differently. We don't want to catch all font
29-- issues.
30--
31-- After a lot of experiments (mostly by Taco, me and Idris) the first
32-- implementation was already quite useful. When it did most of what we wanted, a
33-- more optimized version evolved. Of course all errors are mine and of course the
34-- code can be improved. There are quite some optimizations going on here and
35-- processing speed is currently quite acceptable and has been improved over time.
36-- Many complex scripts are not yet supported yet, but I will look into them as soon
37-- as ConTeXt users ask for it.
38--
39-- The specification leaves room for interpretation. In case of doubt the Microsoft
40-- implementation is the reference as it is the most complete one. As they deal with
41-- lots of scripts and fonts, Kai and Ivo did a lot of testing of the generic code
42-- and their suggestions help improve the code. I'm aware that not all border cases
43-- can be taken care of, unless we accept excessive runtime, and even then the
44-- interference with other mechanisms (like hyphenation) are not trivial.
45--
46-- Especially discretionary handling has been improved much by Kai Eigner who uses
47-- complex (latin) fonts. The current implementation is a compromis between his
48-- patches and my code and in the meantime performance is quite ok. We cannot check
49-- all border cases without compromising speed but so far we're okay. Given good
50-- test cases we can probably improve it here and there. Especially chain lookups
51-- are non trivial with discretionaries but things got much better over time thanks
52-- to Kai.
53--
54-- Glyphs are indexed not by unicode but in their own way. This is because there is
55-- no relationship with unicode at all, apart from the fact that a font might cover
56-- certain ranges of characters. One character can have multiple shapes. However, at
57-- the TeX end we use unicode so and all extra glyphs are mapped into a private
58-- space. This is needed because we need to access them and TeX has to include then
59-- in the output eventually.
60--
61-- The initial data table is rather close to the open type specification and also
62-- not that different from the one produced by Fontforge but we uses hashes instead.
63-- In ConTeXt that table is packed (similar tables are shared) and cached on disk so
64-- that successive runs can use the optimized table (after loading the table is
65-- unpacked).
66--
67-- This module is sparsely documented because it is has been a moving target. The
68-- table format of the reader changed a bit over time and we experiment a lot with
69-- different methods for supporting features. By now the structures are quite stable
70--
71-- Incrementing the version number will force a re-cache. We jump the number by one
72-- when there's a fix in the reader or processing code that can result in different
73-- results.
74--
75-- This code is also used outside ConTeXt but in ConTeXt it has to work with other
76-- mechanisms. Both put some constraints on the code here.
77--
78-- Remark: We assume that cursives don't cross discretionaries which is okay because
79-- it is only used in semitic scripts.
80--
81-- Remark: We assume that marks precede base characters.
82--
83-- Remark: When complex ligatures extend into discs nodes we can get side effects.
84-- Normally this doesn't happen; ff\d{l}{l}{l} in lm works but ff\d{f}{f}{f}.
85--
86-- Todo: check if we copy attributes to disc nodes if needed.
87--
88-- Todo: it would be nice if we could get rid of components. In other places we can
89-- use the unicode properties. We can just keep a lua table.
90--
91-- Remark: We do some disc juggling where we need to keep in mind that the pre, post
92-- and replace fields can have prev pointers to a nesting node ... I wonder if that
93-- is still needed.
94--
95-- Remark: This is not possible:
96--
97-- \discretionary {alpha-} {betagammadelta}
98--   {\discretionary {alphabeta-} {gammadelta}
99--      {\discretionary {alphabetagamma-} {delta}
100--         {alphabetagammadelta}}}
101--
102-- Remark: Something is messed up: we have two mark / ligature indices, one at the
103-- injection end and one here ... this is based on KE's patches but there is something
104-- fishy there as I'm pretty sure that for husayni we need some connection (as it's much
105-- more complex than an average font) but I need proper examples of all cases, not of
106-- only some.
107--
108-- Remark: I wonder if indexed would be faster than unicoded. It would be a major
109-- rewrite to have char being unicode + an index field in glyph nodes. Also more
110-- assignments have to be made in order to keep things in sync. So, it's a no-go.
111--
112-- Remark: We can provide a fast loop when there are no disc nodes (tests show a 1%
113-- gain). Smaller functions might perform better cache-wise. But ... memory becomes
114-- faster anyway, so ...
115--
116-- Remark: Some optimizations made sense for 5.2 but seem less important for 5.3 but
117-- anyway served their purpose.
118--
119-- Todo: just (0=l2r and 1=r2l) or maybe (r2l = true)
120
121-- Experiments with returning the data with the ischar are positive for lmtx but
122-- have a performance hit on mkiv because there we need to wrap ischardata (pending
123-- extensions to luatex which is unlikely to happen for such an experiment because
124-- we then can't remove it). Actually it might make generic slightly faster. Also,
125-- there are some corner cases where a data check comes before a char fetch and
126-- we're talking of millions of calls there. At some point I might make a version
127-- for lmtx that does it slightly different anyway.
128
129local type, next, tonumber = type, next, tonumber
130local random = math.random
131local formatters = string.formatters
132local insert = table.insert
133
134local registertracker      = trackers.register
135
136local logs                 = logs
137local trackers             = trackers
138local nodes                = nodes
139local fonts                = fonts
140
141local otf                  = fonts.handlers.otf
142local tracers              = nodes.tracers
143
144local trace_singles        = false  registertracker("otf.singles",      function(v) trace_singles      = v end)
145local trace_multiples      = false  registertracker("otf.multiples",    function(v) trace_multiples    = v end)
146local trace_alternatives   = false  registertracker("otf.alternatives", function(v) trace_alternatives = v end)
147local trace_ligatures      = false  registertracker("otf.ligatures",    function(v) trace_ligatures    = v end)
148local trace_contexts       = false  registertracker("otf.contexts",     function(v) trace_contexts     = v end)
149local trace_marks          = false  registertracker("otf.marks",        function(v) trace_marks        = v end)
150local trace_kerns          = false  registertracker("otf.kerns",        function(v) trace_kerns        = v end)
151local trace_cursive        = false  registertracker("otf.cursive",      function(v) trace_cursive      = v end)
152local trace_preparing      = false  registertracker("otf.preparing",    function(v) trace_preparing    = v end)
153local trace_bugs           = false  registertracker("otf.bugs",         function(v) trace_bugs         = v end)
154local trace_details        = false  registertracker("otf.details",      function(v) trace_details      = v end)
155local trace_steps          = false  registertracker("otf.steps",        function(v) trace_steps        = v end)
156local trace_skips          = false  registertracker("otf.skips",        function(v) trace_skips        = v end)
157local trace_plugins        = false  registertracker("otf.plugins",      function(v) trace_plugins      = v end)
158local trace_chains         = false  registertracker("otf.chains",       function(v) trace_chains       = v end)
159
160local trace_kernruns       = false  registertracker("otf.kernruns",     function(v) trace_kernruns     = v end)
161----- trace_discruns       = false  registertracker("otf.discruns",     function(v) trace_discruns     = v end)
162local trace_compruns       = false  registertracker("otf.compruns",     function(v) trace_compruns     = v end)
163local trace_testruns       = false  registertracker("otf.testruns",     function(v) trace_testruns     = v end)
164
165local forcediscretionaries = false
166local forcepairadvance     = false -- for testing
167
168local repeatlastmultiple   = context and true
169
170directives.register("otf.forcediscretionaries", function(v) forcediscretionaries = v end)
171directives.register("otf.forcepairadvance",     function(v) forcepairadvance     = v end)
172
173local report_direct      = logs.reporter("fonts","otf direct")
174local report_subchain    = logs.reporter("fonts","otf subchain")
175local report_chain       = logs.reporter("fonts","otf chain")
176local report_process     = logs.reporter("fonts","otf process")
177local report_warning     = logs.reporter("fonts","otf warning")
178local report_run         = logs.reporter("fonts","otf run")
179
180registertracker("otf.substitutions", "otf.singles","otf.multiples","otf.alternatives","otf.ligatures")
181registertracker("otf.positions",     "otf.marks","otf.kerns","otf.cursive")
182registertracker("otf.actions",       "otf.substitutions","otf.positions")
183registertracker("otf.sample",        "otf.steps","otf.substitutions","otf.positions","otf.analyzing")
184registertracker("otf.sample.silent", "otf.steps=silent","otf.substitutions","otf.positions","otf.analyzing")
185
186local nuts               = nodes.nuts
187
188local getnext            = nuts.getnext
189local setnext            = nuts.setnext
190local getprev            = nuts.getprev
191local setprev            = nuts.setprev
192local getboth            = nuts.getboth
193local setboth            = nuts.setboth
194local getid              = nuts.getid
195local getstate           = nuts.getstate
196local getchar            = nuts.getchar
197local setchar            = nuts.setchar
198local getdisc            = nuts.getdisc
199local setdisc            = nuts.setdisc
200local getreplace         = nuts.getreplace
201local setlink            = nuts.setlink
202local getwidth           = nuts.getwidth
203local getoptions         = nuts.getoptions
204local setoptions         = nuts.setoptions
205
206local getglyphdata       = nuts.getglyphdata
207local getscales          = nuts.getscales
208
209---------------------------------------------------------------------------------------
210
211-- Beware: In ConTeXt components no longer are real components. We only keep track of
212-- their positions because some complex ligatures might need that.
213
214local components         = nuts.components
215local copynocomponents   = components.copynocomponents
216local copyonlyglyphs     = components.copyonlyglyphs
217local countcomponents    = components.count
218local setcomponents      = components.set
219local getcomponents      = components.get
220local flushcomponents    = components.flush
221
222---------------------------------------------------------------------------------------
223
224local ischar             = nuts.ischar
225local isnextchar         = nuts.isnextchar -- only makes sense when we know we need next
226local isprevchar         = nuts.isprevchar -- only makes sense when we know we need prev
227local usesfont           = nuts.usesfont
228
229local insertnodeafter    = nuts.insertafter
230local copynode           = nuts.copy
231local copynodelist       = nuts.copylist
232local removenode         = nuts.remove
233local findnodetail       = nuts.tail
234local flushnodelist      = nuts.flushlist
235local flushnode          = nuts.flushnode
236local endofmath          = nuts.endofmath
237
238local startofpar         = nuts.startofpar
239
240local setmetatable       = setmetatable
241local setmetatableindex  = table.setmetatableindex
242
243local nextnode           = nuts.traversers.node
244
245local nodecodes          = nodes.nodecodes
246
247local glyph_code         = nodecodes.glyph
248local glue_code          = nodecodes.glue
249local disc_code          = nodecodes.disc
250local math_code          = nodecodes.math
251local dir_code           = nodecodes.dir
252local par_code           = nodecodes.par
253
254local lefttoright_code   = tex.directioncodes.lefttoright
255local righttoleft_code   = tex.directioncodes.righttoleft
256
257local discretionarydisc_code = nodes.disccodes.discretionary
258
259local injections         = nodes.injections
260local setmark            = injections.setmark
261local setcursive         = injections.setcursive
262local setkern            = injections.setkern
263local setmove            = injections.setmove
264local setposition        = injections.setposition
265local resetinjection     = injections.reset
266local copyinjection      = injections.copy
267local setligaindex       = injections.setligaindex
268local getligaindex       = injections.getligaindex
269
270local fontdata           = fonts.hashes.identifiers
271local fontfeatures       = fonts.hashes.features
272
273local otffeatures        = fonts.constructors.features.otf
274local registerotffeature = otffeatures.register
275
276local onetimemessage     = fonts.loggers.onetimemessage or function() end
277
278local getrandom          = utilities and utilities.randomizer and utilities.randomizer.get
279
280otf.defaultnodealternate = "none" -- first last
281
282-- We use a few semi-global variables. The handler can be called nested but this assumes
283-- that the same font is used.
284
285local tfmdata            = false
286local characters         = false
287local descriptions       = false
288local marks              = false
289local classes            = false
290local currentfont        = false
291local currentdynamic     = false
292local currentscale       = 1000
293local currentxscale      = 1000
294local currentyscale      = 1000
295local factor             = 0
296local threshold          = 0
297local checkmarks         = false
298
299local discs              = false
300local spaces             = false
301
302local sweepnode          = nil
303local sweephead          = { } -- we don't nil entries but false them (no collection and such)
304
305local notmatchpre        = { } -- to be checked: can we use false instead of nil / what if a == b tests
306local notmatchpost       = { } -- to be checked: can we use false instead of nil / what if a == b tests
307local notmatchreplace    = { } -- to be checked: can we use false instead of nil / what if a == b tests
308
309local handlers           = { }
310
311local isspace            = injections.isspace
312local getthreshold       = injections.getthreshold
313
314local checkstep          = (tracers and tracers.steppers.check)    or function() end
315local registerstep       = (tracers and tracers.steppers.register) or function() end
316local registermessage    = (tracers and tracers.steppers.message)  or function() end
317
318do
319
320    local setchr = setchar
321    local direct = nil
322
323    setchar = function(n,c)
324        direct = fonts.collections.direct
325        setchar = function(n,c)
326            setchr(n,c)
327            direct(n)
328        end
329        setchar(n,c)
330    end
331
332end
333
334local function logprocess(...)
335    if trace_steps then
336        registermessage(...)
337        if trace_steps == "silent" then
338            return
339        end
340    end
341    report_direct(...)
342end
343
344local function logwarning(...)
345    report_direct(...)
346end
347
348local gref  do
349
350    local f_unicode = formatters["U+%X"]      -- was ["%U"]
351    local f_uniname = formatters["U+%X (%s)"] -- was ["%U (%s)"]
352    local f_unilist = formatters["% t"]
353
354    gref = function(n) -- currently the same as in font-otb
355        if type(n) == "number" then
356            local description = descriptions[n]
357            local name = description and description.name
358            if name then
359                return f_uniname(n,name)
360            else
361                return f_unicode(n)
362            end
363        elseif n then
364            local t = { }
365            for i=1,#n do
366                local ni = n[i]
367                if tonumber(ni) then -- later we will start at 2
368                    local di = descriptions[ni]
369                    local nn = di and di.name
370                    if nn then
371                        t[#t+1] = f_uniname(ni,nn)
372                    else
373                        t[#t+1] = f_unicode(ni)
374                    end
375                end
376            end
377            return f_unilist(t)
378        else
379            return "<error in node mode tracing>"
380        end
381    end
382
383end
384
385local function cref(dataset,sequence,index)
386    if not dataset then
387        return "no valid dataset"
388    end
389    local merged = sequence.merged and "merged " or ""
390    if index and index > 1 then
391        return formatters["feature %a, type %a, %schain lookup %a, index %a"](
392            dataset[4],sequence.type,merged,sequence.name,index)
393    else
394        return formatters["feature %a, type %a, %schain lookup %a"](
395            dataset[4],sequence.type,merged,sequence.name)
396    end
397end
398
399local function pref(dataset,sequence)
400    return formatters["feature %a, type %a, %slookup %a"](
401        dataset[4],sequence.type,sequence.merged and "merged " or "",sequence.name)
402end
403
404local function mref(rlmode)
405    if not rlmode or rlmode >= 0 then
406        return "l2r"
407    else
408        return "r2l"
409    end
410end
411
412-- The next code is somewhat complicated by the fact that some fonts can have ligatures made
413-- from ligatures that themselves have marks. This was identified by Kai in for instance
414-- arabtype:  KAF LAM SHADDA ALEF FATHA (0x0643 0x0644 0x0651 0x0627 0x064E). This becomes
415-- KAF LAM-ALEF with a SHADDA on the first and a FATHA op de second component. In a next
416-- iteration this becomes a KAF-LAM-ALEF with a SHADDA on the second and a FATHA on the
417-- third component.
418
419-- We can assume that languages that use marks are not hyphenated. We can also assume
420-- that at most one discretionary is present.
421
422-- We do need components in funny kerning mode but maybe I can better reconstruct then
423-- as we do have the font components info available; removing components makes the
424-- previous code much simpler. Also, later on copying and freeing becomes easier.
425-- However, for arabic we need to keep them around for the sake of mark placement
426-- and indices.
427
428local function flattendisk(head,disc) -- happen seldom, otherwise can be a helper
429    local pre, post, replace, pretail, posttail, replacetail = getdisc(disc,true)
430    local prev, next = getboth(disc)
431    local ishead = head == disc
432    setdisc(disc)
433    flushnode(disc)
434    if pre then
435        flushnodelist(pre)
436    end
437    if post then
438        flushnodelist(post)
439    end
440    if ishead then
441        if replace then
442            if next then
443                setlink(replacetail,next)
444            end
445            return replace, replace
446        elseif next then
447            return next, next
448        else
449         -- return -- maybe warning
450        end
451    else
452        if replace then
453            if next then
454                setlink(replacetail,next)
455            end
456            setlink(prev,replace)
457            return head, replace
458        else
459            setlink(prev,next) -- checks for next anyway
460            return head, next
461        end
462    end
463end
464
465local function appenddisc(disc,list) -- happen seldom, otherwise can be a helper
466    local pre, post, replace, pretail, posttail, replacetail = getdisc(disc,true)
467    local posthead    = list
468    local replacehead = copynodelist(list)
469    if post then
470        setlink(posttail,posthead)
471    else
472        post = posthead
473    end
474    if replace then
475        setlink(replacetail,replacehead)
476    else
477        replace = replacehead
478    end
479    setdisc(disc,pre,post,replace)
480end
481
482local function markstoligature(head,start,stop,char)
483    if start == stop and getchar(start) == char then
484        return head, start
485    else
486        local prev = getprev(start)
487        local next = getnext(stop)
488        setprev(start)
489        setnext(stop)
490        local base = copynocomponents(start,copyinjection)
491        if head == start then
492            head = base
493        end
494        resetinjection(base)
495        setchar(base,char)
496        setcomponents(base,start)
497        setlink(prev,base,next)
498        flushcomponents(start)
499        return head, base
500    end
501end
502
503-- Remark for Kai: (some arabic fonts do mark + mark = other mark and such)
504--
505-- The hasmarks is needed for ligatures of marks that are part of a ligature in
506-- which case we assume that we can delete the marks anyway (we can always become
507-- more clever if needed) .. in fact the whole logic here should be redone. We're
508-- in the not discfound branch then. We now have skiphash too so we can be more
509-- selective if needed (todo).
510
511-- we can have more granularity here but for now we only do a simple check
512
513local glyphoptioncodes       = tex.glyphoptioncodes
514
515local no_left_ligature_code  = glyphoptioncodes.noleftligature
516local no_right_ligature_code = glyphoptioncodes.norightligature
517----- no_left_kern_code      = glyphoptioncodes.noleftkern
518local no_right_kern_code     = glyphoptioncodes.norightkern
519
520local hasglyphoption         = nuts.hasglyphoption
521
522-- in lmtx we need to check the components and can be slightly more clever
523
524local function inhibited(start,stop)
525    for n in nextnode, start do
526        -- we asume glyph nodes
527        if n == start then
528            if hasglyphoption(n,no_right_ligature_code) then
529                return true
530            end
531        elseif n == stop then
532            if hasglyphoption(n,no_left_ligature_code) then
533                return true
534            else
535                return false
536            end
537        elseif hasglyphoption(n,no_left_ligature_code) then
538            return true
539        elseif hasglyphoption(n,no_right_ligature_code) then
540            return true
541        end
542    end
543    return false
544end
545
546local function toligature(head,start,stop,char,dataset,sequence,skiphash,discfound,hasmarks) -- brr head
547    if start == stop and getchar(start) == char and not hasmarks then
548        resetinjection(start)
549        setchar(start,char)
550     -- fonts.collections.direct(start)
551        return head, start
552    end
553    if inhibited(start,stop) then
554        return head, start
555    end
556    local prev = getprev(start)
557    local next = getnext(stop)
558    local comp = start
559    setprev(start)
560    setnext(stop)
561    local base = copynocomponents(start,copyinjection)
562    if start == head then
563        head = base
564    end
565    resetinjection(base)
566    setchar(base,char)
567 -- fonts.collections.direct(base)
568    setoptions(base,getoptions(start) | getoptions(stop)) -- maybe only lig options
569    setcomponents(base,comp)
570    setlink(prev,base,next)
571    if not discfound then
572        local deletemarks = not skiphash or hasmarks
573        local components = start -- not used
574        local baseindex = 0
575        local componentindex = 0
576        local head = base
577        local current = base
578        -- first we loop over the glyphs in start ... stop
579        while start do
580            local char = getchar(start)
581            if not marks[char] then
582                baseindex = baseindex + componentindex
583                componentindex = countcomponents(start,marks)
584             -- we can be more clever here: "not deletemarks or (skiphash and not skiphash[char])"
585             -- and such:
586            elseif not deletemarks then
587                -- we can get a loop when the font expects otherwise (i.e. unexpected deletemarks)
588                setligaindex(start,baseindex + getligaindex(start,componentindex))
589                if trace_marks then
590                    logwarning("%s: keep ligature mark %s, gets index %s",pref(dataset,sequence),gref(char),getligaindex(start))
591                end
592                local n = copynode(start)
593                copyinjection(n,start) -- is this ok ? we position later anyway
594                head, current = insertnodeafter(head,current,n) -- unlikely that mark has components
595            elseif trace_marks then
596                logwarning("%s: delete ligature mark %s",pref(dataset,sequence),gref(char))
597            end
598            start = getnext(start)
599        end
600        -- we can have one accent as part of a lookup and another following
601        local start = getnext(current)
602        while start do
603            local nxt, char = isnextchar(start,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
604            if char then
605                -- also something skiphash here?
606                if marks[char] then
607                    setligaindex(start,baseindex + getligaindex(start,componentindex))
608                    if trace_marks then
609                        logwarning("%s: set ligature mark %s, gets index %s",pref(dataset,sequence),gref(char),getligaindex(start))
610                    end
611                    start = nxt
612                else
613                    break
614                end
615            else
616                break
617            end
618        end
619        flushcomponents(components)
620    else
621        -- discfound ... forget about marks .. probably no scripts that hyphenate and have marks
622        local discprev, discnext = getboth(discfound)
623        if discprev and discnext then
624            -- we assume normalization in context, and don't care about generic ... especially
625            -- \- can give problems as there we can have a negative char but that won't match
626            -- anyway
627            local pre, post, replace, pretail, posttail, replacetail = getdisc(discfound,true)
628            if not replace then
629                -- looks like we never come here as it's not okay
630                local prev = getprev(base)
631             -- local comp = getcomponents(base) -- already set
632                local copied = copyonlyglyphs(comp)
633                if pre then
634                    setlink(discprev,pre)
635                else
636                    setnext(discprev) -- also blocks funny assignments
637                end
638                pre = comp -- is start
639                if post then
640                    setlink(posttail,discnext)
641                    setprev(post) -- nil anyway
642                else
643                    post = discnext
644                    setprev(discnext) -- also blocks funny assignments
645                end
646                setlink(prev,discfound,next)
647                setboth(base)
648                -- here components have a pointer so we can't free it!
649                setcomponents(base,copied)
650                replace = base
651                if forcediscretionaries then
652                    setdisc(discfound,pre,post,replace,discretionarydisc_code)
653                else
654                    setdisc(discfound,pre,post,replace)
655                end
656                base = prev
657            end
658        end
659    end
660    return head, base
661end
662
663local function multiple_glyphs(head,start,multiple,skiphash,what,stop) -- what to do with skiphash matches here
664    local nofmultiples = #multiple
665    if nofmultiples > 0 then
666        local first = start
667        resetinjection(start)
668        setchar(start,multiple[1])
669        if nofmultiples > 1 then
670         -- local sn = getnext(start)
671            for i=2,nofmultiples do
672             -- untested:
673             --
674             -- while ignoremarks and marks[getchar(sn)] then
675             --     local sn = getnext(sn)
676             -- end
677                local n = copynode(start) -- ignore components
678                resetinjection(n)
679                setchar(n,multiple[i])
680                insertnodeafter(head,start,n)
681                start = n
682            end
683        end
684        if what ~= true then
685            -- This is just some experimental code; we might introduce gsub_extensible
686            -- some day instead. Beware: when we have a feature that mixes alternates and
687            -- multiples we need to make sure we don't handle the alternate string values
688            -- here.
689            local kind = type(what)
690            local m, f, l
691            if kind == "string" then
692                local what, n = string.match(what,"^repeat(.-)[:=](%d+)$")
693                if what == "middle" then
694                    m = tonumber(n)
695                elseif what == "first" then
696                    f = tonumber(n)
697                elseif what == "last" then
698                    l = tonumber(n)
699                end
700            elseif kind == "table" then
701                -- won't happen because currently we don't split these values
702               m = what.middle
703               f = what.first
704               l = what.last
705            end
706            if f or m or l then
707                if m and m > 1 and nofmultiples == 3 then
708                    local middle = getnext(first)
709                    for i=2,m do
710                        local n = copynode(middle) -- ignore components
711                        resetinjection(n)
712                        insertnodeafter(head,first,n)
713                    end
714                end
715                if f and f > 1 then
716                    for i=2,f do
717                        local n = copynode(first) -- ignore components
718                        resetinjection(n)
719                        insertnodeafter(head,first,n)
720                    end
721                end
722                if l and l > 1 then
723                    for i=2,l do
724                        local n = copynode(start) -- ignore components
725                        resetinjection(n)
726                        insertnodeafter(head,start,n)
727                        start = n
728                    end
729                end
730            end
731        end
732        return head, start, true
733    else
734        if trace_multiples then
735            logprocess("no multiple for %s",gref(getchar(start)))
736        end
737        return head, start, false
738    end
739end
740
741local function get_alternative_glyph(start,alternatives,value)
742    local n = #alternatives
743    if n == 1 then
744        -- we could actually change that into a gsub and save some memory in the
745        -- font loader but it makes tracing more messy
746        return alternatives[1], trace_alternatives and "1 (only one present)"
747    elseif value == "random" then
748        local r = getrandom and getrandom("glyph",1,n) or random(1,n)
749        return alternatives[r], trace_alternatives and formatters["value %a, taking %a"](value,r)
750    elseif value == "first" then
751        return alternatives[1], trace_alternatives and formatters["value %a, taking %a"](value,1)
752    elseif value == "last" then
753        return alternatives[n], trace_alternatives and formatters["value %a, taking %a"](value,n)
754    end
755    value = value == true and 1 or tonumber(value)
756    if type(value) ~= "number" then
757        return alternatives[1], trace_alternatives and formatters["invalid value %s, taking %a"](value,1)
758    end
759 -- local a = alternatives[value]
760 -- if a then
761 --     -- some kind of hash
762 --     return a, trace_alternatives and formatters["value %a, taking %a"](value,a)
763 -- end
764    if value > n then
765        local defaultalt = otf.defaultnodealternate
766        if defaultalt == "first" then
767            return alternatives[n], trace_alternatives and formatters["invalid value %s, taking %a"](value,1)
768        elseif defaultalt == "last" then
769            return alternatives[1], trace_alternatives and formatters["invalid value %s, taking %a"](value,n)
770        else
771            return false, trace_alternatives and formatters["invalid value %a, %s"](value,"out of range")
772        end
773    elseif value == 0 then
774        return getchar(start), trace_alternatives and formatters["invalid value %a, %s"](value,"no change")
775    elseif value < 1 then
776        return alternatives[1], trace_alternatives and formatters["invalid value %a, taking %a"](value,1)
777    else
778        return alternatives[value], trace_alternatives and formatters["value %a, taking %a"](value,value)
779    end
780end
781
782-- handlers
783
784function handlers.gsub_single(head,start,dataset,sequence,replacement)
785    if trace_singles then
786        logprocess("%s: replacing %s by single %s",pref(dataset,sequence),gref(getchar(start)),gref(replacement))
787    end
788    resetinjection(start)
789    setchar(start,replacement)
790    return head, start, true
791end
792
793function handlers.gsub_alternate(head,start,dataset,sequence,alternative)
794    local kind  = dataset[4]
795    local what  = dataset[1]
796    local value = what == true and tfmdata.shared.features[kind] or what
797    local choice, comment = get_alternative_glyph(start,alternative,value)
798    if choice then
799        if trace_alternatives then
800            logprocess("%s: replacing %s by alternative %a to %s, %s",pref(dataset,sequence),gref(getchar(start)),gref(choice),comment)
801        end
802        resetinjection(start)
803        setchar(start,choice)
804    else
805        if trace_alternatives then
806            logwarning("%s: no variant %a for %s, %s",pref(dataset,sequence),value,gref(getchar(start)),comment)
807        end
808    end
809    return head, start, true
810end
811
812function handlers.gsub_multiple(head,start,dataset,sequence,multiple,rlmode,skiphash)
813    if trace_multiples then
814        logprocess("%s: replacing %s by multiple %s",pref(dataset,sequence),gref(getchar(start)),gref(multiple))
815    end
816    return multiple_glyphs(head,start,multiple,skiphash,dataset[1])
817end
818
819-- Don't we deal with disc otherwise now? I need to check if the next one can be
820-- simplified. Anyway, it can be way messier: marks that get removed as well as
821-- marks that are kept.
822
823-- Blocking is nasty: (ff\zwj l) vs (ff\zwj l) vs (ffl) vs (f\zwj fl}
824
825function handlers.gsub_ligature(head,start,dataset,sequence,ligature,rlmode,skiphash)
826    if start and hasglyphoption(start,no_right_ligature_code) then
827        return head, start, false, nil
828    end
829    local current = getnext(start)
830    if not current then
831        return head, start, false, nil
832    end
833    local stop      = nil
834    local startchar = getchar(start)
835    if skiphash and skiphash[startchar] then
836        while current do
837            local nxt, char = isnextchar(current,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
838            if char and not hasglyphoption(current,no_left_ligature_code) then
839                local lg = not tonumber(ligature) and ligature[char]
840                if lg then
841                    stop     = current
842                    ligature = lg
843                    current  = nxt
844                else
845                    break
846                end
847            else
848                break
849            end
850        end
851        if stop then
852            local ligature = tonumber(ligature) or ligature.ligature
853            if ligature then
854                if trace_ligatures then
855                    local stopchar = getchar(stop)
856                    head, start = markstoligature(head,start,stop,ligature)
857                    logprocess("%s: replacing %s upto %s by ligature %s case 1",pref(dataset,sequence),gref(startchar),gref(stopchar),gref(getchar(start)))
858                else
859                    head, start = markstoligature(head,start,stop,ligature)
860                end
861                return head, start, true, false
862            else
863                -- ok, goto next lookup
864            end
865        end
866    else
867        local discfound = false
868        local hasmarks  = marks[startchar]
869        while current do
870            local nxt, char, id = isnextchar(current,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
871            if char then
872                if skiphash and skiphash[char] then
873                    current = nxt
874                elseif hasglyphoption(current,no_left_ligature_code) then
875                    break
876                else
877                    local lg = not tonumber(ligature) and ligature[char]
878                    if lg then
879                        if marks[char] then
880                            hasmarks = true
881                        end
882                        stop     = current -- needed for fake so outside then
883                        ligature = lg
884                        current  = nxt
885                    else
886                        break
887                    end
888                end
889            elseif char == false then
890                -- kind of weird
891                break
892            elseif id == disc_code then
893                discfound = current
894                break
895            else
896                break
897            end
898        end
899        -- of{f-}{}{f}e  o{f-}{}{f}fe  o{-}{}{ff}e (oe and ff ligature)
900        -- we can end up here when we have a start run .. testruns start at a disc but
901        -- so here we have the other case: char + disc
902        --
903        -- Challenge for Kai (latinmodern):  \hyphenation{fii-f-f-iif} fiiffiif
904        --
905        if tonumber(ligature) then
906            -- we're already done (endpoint reached)
907        elseif discfound then
908            -- todo: check for no right ligature
909            -- don't assume marks in a disc and we don't run over a disc (for now)
910            local pre, post, replace = getdisc(discfound)
911            local match
912            if replace then
913                local nxt, char = isnextchar(replace,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
914                if char and ligature[char] and not hasglyphoption(replace,no_left_ligature_code) then
915                    match = true
916                end
917            end
918            if not match and pre then
919                local nxt, char = isnextchar(pre,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
920                if char and ligature[char] and not hasglyphoption(pre,no_left_ligature_code) then
921                    match = true
922                end
923            end
924            -- is this test ok:
925            if not match and not pre or not replace then
926                local ndf = getnext(discfound)
927                local nxt, char = isnextchar(ndf,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
928                if char and ligature[char] and not hasglyphoption(ndf,no_left_ligature_code) then
929                    match = true
930                end
931            end
932            if match then
933                -- we force a restart
934                local ishead = head == start
935                local prev   = getprev(start)
936                if stop then
937                    setnext(stop)
938                    local copy = copynodelist(start)
939                    local tail = stop -- was: getprev(stop) -- Kai: needs checking on your samples
940                    local liat = findnodetail(copy)
941                    if pre then
942                        setlink(liat,pre)
943                    end
944                    if replace then
945                        setlink(tail,replace)
946                    end
947                    pre     = copy
948                    replace = start
949                else
950                    setnext(start)
951                    local copy = copynode(start)
952                    if pre then
953                        setlink(copy,pre)
954                    end
955                    if replace then
956                        setlink(start,replace)
957                    end
958                    pre     = copy
959                    replace = start
960                end
961                setdisc(discfound,pre,post,replace)
962                if prev then
963                    setlink(prev,discfound)
964                else
965                    setprev(discfound)
966                    head  = discfound
967                end
968                start = discfound
969                return head, start, true, true
970            end
971            ligature = ligature.ligature
972        else
973            ligature = ligature.ligature
974        end
975     -- local ligature = tonumber(ligature) or ligature.ligature
976        if ligature then
977            if stop then
978                if trace_ligatures then
979                    local stopchar = getchar(stop)
980                 -- head, start = toligature(head,start,stop,ligature,dataset,sequence,skiphash,discfound,hasmarks)
981                    head, start = toligature(head,start,stop,ligature,dataset,sequence,skiphash,false,hasmarks)
982                    logprocess("%s: replacing %s upto %s by ligature %s case 2",pref(dataset,sequence),gref(startchar),gref(stopchar),gref(ligature))
983                 -- we can have a rare case of multiple disc in a lig but that makes no sense language wise but if really
984                 -- needed we could backtrack if we're in a disc node
985                else
986                 -- head, start = toligature(head,start,stop,ligature,dataset,sequence,skiphash,discfound,hasmarks)
987                    head, start = toligature(head,start,stop,ligature,dataset,sequence,skiphash,false,hasmarks)
988                end
989            else
990                -- weird but happens (in some arabic font)
991                resetinjection(start)
992                setchar(start,ligature)
993                if trace_ligatures then
994                    logprocess("%s: replacing %s by (no real) ligature %s case 3",pref(dataset,sequence),gref(startchar),gref(ligature))
995                end
996            end
997            return head, start, true, false
998        else
999            -- weird but happens, pseudo ligatures ... just the components
1000        end
1001    end
1002    return head, start, false, false
1003end
1004
1005function handlers.gpos_single(head,start,dataset,sequence,kerns,rlmode,skiphash,step,injection)
1006    if hasglyphoption(start,no_right_kern_code) then
1007        return head, start, false
1008    else
1009        local startchar = getchar(start)
1010        local format    = step.format
1011        if format == "single" or type(kerns) == "table" then -- the table check can go
1012            local dx, dy, w, h = setposition(0,start,factor,rlmode,kerns,injection)
1013            if trace_kerns then
1014                logprocess("%s: shifting single %s by %s xy (%p,%p) and wh (%p,%p)",pref(dataset,sequence),gref(startchar),format,dx,dy,w,h)
1015            end
1016        else
1017            local k = (format == "move" and setmove or setkern)(start,factor,rlmode,kerns,injection)
1018            if trace_kerns then
1019                logprocess("%s: shifting single %s by %s %p",pref(dataset,sequence),gref(startchar),format,k)
1020            end
1021        end
1022        return head, start, true
1023    end
1024end
1025
1026function handlers.gpos_pair(head,start,dataset,sequence,kerns,rlmode,skiphash,step,injection)
1027    if hasglyphoption(start,no_right_kern_code) then
1028        return head, start, false
1029    else
1030        local snext = getnext(start)
1031        if not snext then
1032            return head, start, false
1033        else
1034            local prev = start
1035            while snext do
1036                local nxt, nextchar = isnextchar(snext,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
1037                if nextchar then
1038                    if skiphash and skiphash[nextchar] then -- includes marks too when flag
1039                        prev  = snext
1040                        snext = nxt
1041                    else
1042                        local krn = kerns[nextchar]
1043                        if not krn then
1044                            break
1045                        end
1046                        local format = step.format
1047                        if format == "pair" then
1048                            local a = krn[1]
1049                            local b = krn[2]
1050                            if a == true then
1051                                -- zero
1052                            elseif a then -- #a > 0
1053                                local x, y, w, h = setposition(1,start,factor,rlmode,a,injection)
1054                                if trace_kerns then
1055                                    local startchar = getchar(start)
1056                                    logprocess("%s: shifting first of pair %s and %s by xy (%p,%p) and wh (%p,%p) as %s",pref(dataset,sequence),gref(startchar),gref(nextchar),x,y,w,h,injection or "injections")
1057                                end
1058                            end
1059                            if b == true then
1060                                -- zero
1061                                start = snext -- cf spec
1062                            elseif b then -- #b > 0
1063                                local x, y, w, h = setposition(2,snext,factor,rlmode,b,injection)
1064                                if trace_kerns then
1065                                    local startchar = getchar(start)
1066                                    logprocess("%s: shifting second of pair %s and %s by xy (%p,%p) and wh (%p,%p) as %s",pref(dataset,sequence),gref(startchar),gref(nextchar),x,y,w,h,injection or "injections")
1067                                end
1068                                start = snext -- cf spec
1069                            elseif forcepairadvance then
1070                                start = snext -- for testing, not cf spec
1071                            end
1072                            return head, start, true
1073                        elseif krn ~= 0 then
1074                            local k = (format == "move" and setmove or setkern)(snext,factor,rlmode,krn,injection)
1075                            if trace_kerns then
1076                                logprocess("%s: inserting %s %p between %s and %s as %s",pref(dataset,sequence),format,k,gref(getchar(prev)),gref(nextchar),injection or "injections")
1077                            end
1078                            return head, start, true
1079                        else -- can't happen
1080                            break
1081                        end
1082                    end
1083                else
1084                    break
1085                end
1086            end
1087            return head, start, false
1088        end
1089    end
1090end
1091
1092-- We get hits on a mark, but we're not sure if the it has to be applied so we need
1093-- to explicitly test for basechar, baselig and basemark entries.
1094
1095function handlers.gpos_mark2base(head,start,dataset,sequence,markanchors,rlmode,skiphash)
1096    local markchar = getchar(start) -- combine with getprev
1097    if marks[markchar] then
1098        local base = getprev(start) -- [glyph] [start=mark]
1099        if base then
1100            local prv, basechar = isprevchar(base,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
1101            if basechar then
1102                if marks[basechar] then
1103                    -- todo: use prv
1104                    while base do
1105                        base = getprev(base)
1106                        if base then
1107                            prv, basechar = isprevchar(base,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
1108                            if basechar then
1109                                if not marks[basechar] then
1110                                    break
1111                                end
1112                            else
1113                                if trace_bugs then
1114                                    logwarning("%s: no base for mark %s, case %i",pref(dataset,sequence),gref(markchar),1)
1115                                end
1116                                return head, start, false
1117                            end
1118                        else
1119                            if trace_bugs then
1120                                logwarning("%s: no base for mark %s, case %i",pref(dataset,sequence),gref(markchar),2)
1121                            end
1122                            return head, start, false
1123                        end
1124                    end
1125                end
1126                local ba = markanchors[1][basechar]
1127                if ba then
1128                    local ma = markanchors[2]
1129                    local dx, dy, bound = setmark(start,base,factor,rlmode,ba,ma,characters[basechar],false,checkmarks)
1130                    if trace_marks then
1131                        logprocess("%s, bound %s, anchoring mark %s to basechar %s => (%p,%p)",
1132                            pref(dataset,sequence),bound,gref(markchar),gref(basechar),dx,dy)
1133                    end
1134                    return head, start, true
1135                elseif trace_bugs then
1136                 -- onetimemessage(currentfont,basechar,"no base anchors")
1137                    logwarning("%s: mark %s is not anchored to %s",pref(dataset,sequence),gref(markchar),gref(basechar))
1138                end
1139            elseif trace_bugs then
1140                logwarning("%s: nothing preceding, case %i",pref(dataset,sequence),1)
1141            end
1142        elseif trace_bugs then
1143            logwarning("%s: nothing preceding, case %i",pref(dataset,sequence),2)
1144        end
1145    elseif trace_bugs then
1146        logwarning("%s: mark %s is no mark",pref(dataset,sequence),gref(markchar))
1147    end
1148    return head, start, false
1149end
1150
1151function handlers.gpos_mark2ligature(head,start,dataset,sequence,markanchors,rlmode,skiphash)
1152    local markchar = getchar(start)
1153    if marks[markchar] then
1154        local base = getprev(start) -- [glyph] [optional marks] [start=mark]
1155        if base then
1156            local prv, basechar = isprevchar(base,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
1157            if basechar then
1158                if marks[basechar] then
1159                    -- todo: use prv
1160                    while base do
1161                        base = getprev(base)
1162                        if base then
1163                            prv, basechar = isprevchar(base,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
1164                            if basechar then
1165                                if not marks[basechar] then
1166                                    break
1167                                end
1168                            else
1169                                if trace_bugs then
1170                                    logwarning("%s: no base for mark %s, case %i",pref(dataset,sequence),gref(markchar),1)
1171                                end
1172                                return head, start, false
1173                            end
1174                        else
1175                            if trace_bugs then
1176                                logwarning("%s: no base for mark %s, case %i",pref(dataset,sequence),gref(markchar),2)
1177                            end
1178                            return head, start, false
1179                        end
1180                    end
1181                end
1182                local ba = markanchors[1][basechar]
1183                if ba then
1184                    local ma = markanchors[2]
1185                    if ma then
1186                        local index = getligaindex(start)
1187                        ba = ba[index]
1188                        if ba then
1189                            local dx, dy, bound = setmark(start,base,factor,rlmode,ba,ma,characters[basechar],false,checkmarks)
1190                            if trace_marks then
1191                                logprocess("%s, index %s, bound %s, anchoring mark %s to baselig %s at index %s => (%p,%p)",
1192                                    pref(dataset,sequence),index,bound,gref(markchar),gref(basechar),index,dx,dy)
1193                            end
1194                            return head, start, true
1195                        else
1196                            if trace_bugs then
1197                                logwarning("%s: no matching anchors for mark %s and baselig %s with index %a",pref(dataset,sequence),gref(markchar),gref(basechar),index)
1198                            end
1199                        end
1200                    end
1201                elseif trace_bugs then
1202                --  logwarning("%s: char %s is missing in font",pref(dataset,sequence),gref(basechar))
1203                    onetimemessage(currentfont,basechar,"no base anchors")
1204                end
1205            elseif trace_bugs then
1206                logwarning("%s: prev node is no char, case %i",pref(dataset,sequence),1)
1207            end
1208        elseif trace_bugs then
1209            logwarning("%s: prev node is no char, case %i",pref(dataset,sequence),2)
1210        end
1211    elseif trace_bugs then
1212        logwarning("%s: mark %s is no mark",pref(dataset,sequence),gref(markchar))
1213    end
1214    return head, start, false
1215end
1216
1217function handlers.gpos_mark2mark(head,start,dataset,sequence,markanchors,rlmode,skiphash)
1218    local markchar = getchar(start)
1219    if marks[markchar] then
1220        local base = getprev(start) -- [glyph] [basemark] [start=mark]
1221        local slc = getligaindex(start)
1222        if slc then -- a rather messy loop ... needs checking with husayni
1223            while base do
1224                local blc = getligaindex(base)
1225                if blc and blc ~= slc then
1226                    base = getprev(base)
1227                else
1228                    break
1229                end
1230            end
1231        end
1232        if base then
1233            local nxt, basechar = isnextchar(base,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
1234            if basechar then -- subtype test can go
1235                local ba = markanchors[1][basechar] -- slot 1 has been made copy of the class hash
1236                if ba then
1237                    local ma = markanchors[2]
1238                    local dx, dy, bound = setmark(start,base,factor,rlmode,ba,ma,characters[basechar],true,checkmarks)
1239                    if trace_marks then
1240                        logprocess("%s, bound %s, anchoring mark %s to basemark %s => (%p,%p)",
1241                            pref(dataset,sequence),bound,gref(markchar),gref(basechar),dx,dy)
1242                    end
1243                    return head, start, true
1244                end
1245            end
1246        end
1247    elseif trace_bugs then
1248        logwarning("%s: mark %s is no mark",pref(dataset,sequence),gref(markchar))
1249    end
1250    return head, start, false
1251end
1252
1253function handlers.gpos_cursive(head,start,dataset,sequence,exitanchors,rlmode,skiphash,step) -- to be checked
1254    local startchar = getchar(start)
1255    if marks[startchar] then
1256        if trace_cursive then
1257            logprocess("%s: ignoring cursive for mark %s",pref(dataset,sequence),gref(startchar))
1258        end
1259    else
1260        local nxt = getnext(start)
1261        while nxt do
1262            local n, nextchar = isnextchar(nxt,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
1263            if not nextchar then
1264                break
1265            elseif marks[nextchar] then -- always sequence.flags[1]
1266                nxt = n
1267            else
1268                local exit = exitanchors[3]
1269                if exit then
1270                    local entry = exitanchors[1][nextchar]
1271                    if entry then
1272                        entry = entry[2]
1273                        if entry then
1274                            local r2lflag = sequence.flags[4] -- mentioned in the standard
1275                            local dx, dy, bound = setcursive(start,nxt,factor,rlmode,exit,entry,characters[startchar],characters[nextchar],r2lflag)
1276                            if trace_cursive then
1277                                logprocess("%s: moving %s to %s cursive (%p,%p) using bound %s in %s mode",pref(dataset,sequence),gref(startchar),gref(nextchar),dx,dy,bound,mref(rlmode))
1278                            end
1279                            return head, start, true
1280                        end
1281                    end
1282                end
1283                break
1284            end
1285        end
1286    end
1287    return head, start, false
1288end
1289
1290-- I will implement multiple chain replacements once I run into a font that uses it.
1291-- It's not that complex to handle.
1292
1293local chainprocs = { }
1294
1295local function logprocess(...)
1296    if trace_steps then
1297        registermessage(...)
1298        if trace_steps == "silent" then
1299            return
1300        end
1301    end
1302    report_subchain(...)
1303end
1304
1305local logwarning = report_subchain
1306
1307local function logprocess(...)
1308    if trace_steps then
1309        registermessage(...)
1310        if trace_steps == "silent" then
1311            return
1312        end
1313    end
1314    report_chain(...)
1315end
1316
1317local logwarning = report_chain
1318
1319-- We could share functions but that would lead to extra function calls with many
1320-- arguments, redundant tests and confusing messages.
1321
1322-- The reversesub is a special case, which is why we need to store the replacements
1323-- in a bit weird way. There is no lookup and the replacement comes from the lookup
1324-- itself. It is meant mostly for dealing with Urdu.
1325
1326local function reversesub(head,start,stop,dataset,sequence,replacements,rlmode,skiphash)
1327    local char        = getchar(start)
1328    local replacement = replacements[char]
1329    if replacement then
1330        if trace_singles then
1331            logprocess("%s: single reverse replacement of %s by %s",cref(dataset,sequence),gref(char),gref(replacement))
1332        end
1333        resetinjection(start)
1334        setchar(start,replacement)
1335        return head, start, true
1336    else
1337        return head, start, false
1338    end
1339end
1340
1341
1342chainprocs.reversesub = reversesub
1343
1344-- This chain stuff is somewhat tricky since we can have a sequence of actions to be
1345-- applied: single, alternate, multiple or ligature where ligature can be an invalid
1346-- one in the sense that it will replace multiple by one but not neccessary one that
1347-- looks like the combination (i.e. it is the counterpart of multiple then). For
1348-- example, the following is valid:
1349--
1350--   xxxabcdexxx [single a->A][multiple b->BCD][ligature cde->E] xxxABCDExxx
1351--
1352-- Therefore we we don't really do the replacement here already unless we have the
1353-- single lookup case. The efficiency of the replacements can be improved by
1354-- deleting as less as needed but that would also make the code even more messy.
1355--
1356-- Here we replace start by a single variant.
1357--
1358-- To be done   : what if > 1 steps (example needed)
1359-- This is messy: do we need this disc checking also in alternates?
1360
1361local function reportzerosteps(dataset,sequence)
1362    logwarning("%s: no steps",cref(dataset,sequence))
1363end
1364
1365local function reportmoresteps(dataset,sequence)
1366    logwarning("%s: more than 1 step",cref(dataset,sequence))
1367end
1368
1369-- local function reportbadsteps(dataset,sequence)
1370--     logwarning("%s: bad step, no proper return values",cref(dataset,sequence))
1371-- end
1372
1373local function getmapping(dataset,sequence,currentlookup)
1374    local steps    = currentlookup.steps
1375    local nofsteps = currentlookup.nofsteps
1376    if nofsteps == 0 then
1377        reportzerosteps(dataset,sequence)
1378        currentlookup.mapping = false
1379        return false
1380    else
1381        if nofsteps > 1 then
1382            reportmoresteps(dataset,sequence)
1383        end
1384        local mapping = steps[1].coverage
1385        currentlookup.mapping = mapping
1386        currentlookup.format  = steps[1].format
1387        return mapping
1388    end
1389end
1390
1391function chainprocs.gsub_remove(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash,chainindex)
1392    if trace_chains then
1393        logprocess("%s: removing character %s",cref(dataset,sequence,chainindex),gref(getchar(start)))
1394    end
1395    head, start = removenode(head,start,true)
1396    return head, getprev(start), true
1397end
1398
1399function chainprocs.gsub_single(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash,chainindex)
1400    local mapping = currentlookup.mapping
1401    if mapping == nil then
1402        mapping = getmapping(dataset,sequence,currentlookup)
1403    end
1404    if mapping then
1405        local current = start
1406        while current do
1407            local nxt, currentchar = isnextchar(current,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
1408            if currentchar then
1409                local replacement = mapping[currentchar]
1410                if not replacement or replacement == "" then
1411                    if trace_bugs then
1412                        logwarning("%s: no single for %s",cref(dataset,sequence,chainindex),gref(currentchar))
1413                    end
1414                else
1415                    if trace_singles then
1416                        logprocess("%s: replacing single %s by %s",cref(dataset,sequence,chainindex),gref(currentchar),gref(replacement))
1417                    end
1418                    resetinjection(current)
1419                    setchar(current,replacement)
1420                end
1421                return head, start, true
1422            elseif currentchar == false then
1423                -- can't happen
1424                break
1425            elseif current == stop then
1426                break
1427            else
1428                current = nxt
1429            end
1430        end
1431    end
1432    return head, start, false
1433end
1434
1435-- Here we replace start by new glyph. First we delete the rest of the match.
1436
1437-- char_1 mark_1 -> char_x mark_1 (ignore marks)
1438-- char_1 mark_1 -> char_x
1439
1440-- to be checked: do we always have just one glyph?
1441-- we can also have alternates for marks
1442-- marks come last anyway
1443-- are there cases where we need to delete the mark
1444
1445function chainprocs.gsub_alternate(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash,chainindex)
1446    local mapping = currentlookup.mapping
1447    if mapping == nil then
1448        mapping = getmapping(dataset,sequence,currentlookup)
1449    end
1450    if mapping then
1451        local kind    = dataset[4]
1452        local what    = dataset[1]
1453        local value   = what == true and tfmdata.shared.features[kind] or what -- todo: optimize in ctx
1454        local current = start
1455        while current do
1456            local nxt, currentchar = isnextchar(current,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
1457            if currentchar then
1458                local alternatives = mapping[currentchar]
1459                if alternatives then
1460                    local choice, comment = get_alternative_glyph(current,alternatives,value)
1461                    if choice then
1462                        if trace_alternatives then
1463                            logprocess("%s: replacing %s by alternative %a to %s, %s",cref(dataset,sequence),gref(currentchar),choice,gref(choice),comment)
1464                        end
1465                        resetinjection(start)
1466                        setchar(start,choice)
1467                    else
1468                        if trace_alternatives then
1469                            logwarning("%s: no variant %a for %s, %s",cref(dataset,sequence),value,gref(currentchar),comment)
1470                        end
1471                    end
1472                end
1473                return head, start, true
1474            elseif currentchar == false then
1475                -- can't happen
1476                break
1477            elseif current == stop then
1478                break
1479            else
1480                current = nxt
1481            end
1482        end
1483    end
1484    return head, start, false
1485end
1486
1487-- Here we replace start by a sequence of new glyphs.
1488
1489function chainprocs.gsub_multiple(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash,chainindex)
1490    local mapping = currentlookup.mapping
1491    if mapping == nil then
1492        mapping = getmapping(dataset,sequence,currentlookup)
1493    end
1494    if mapping then
1495        local startchar   = getchar(start)
1496        local replacement = mapping[startchar]
1497        if not replacement or replacement == "" then
1498            if trace_bugs then
1499                logwarning("%s: no multiple for %s",cref(dataset,sequence),gref(startchar))
1500            end
1501        else
1502            if trace_multiples then
1503                logprocess("%s: replacing %s by multiple characters %s",cref(dataset,sequence),gref(startchar),gref(replacement))
1504            end
1505            return multiple_glyphs(head,start,replacement,skiphash,dataset[1],stop)
1506        end
1507    end
1508    return head, start, false
1509end
1510
1511-- When we replace ligatures we use a helper that handles the marks. I might change
1512-- this function (move code inline and handle the marks by a separate function). We
1513-- assume rather stupid ligatures (no complex disc nodes).
1514
1515-- compare to handlers.gsub_ligature which is more complex ... why
1516
1517function chainprocs.gsub_ligature(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash,chainindex)
1518    if start and hasglyphoption(start,no_right_ligature_code) then
1519        return head, start, false, 0, false
1520    end
1521    local mapping = currentlookup.mapping
1522    if mapping == nil then
1523        mapping = getmapping(dataset,sequence,currentlookup)
1524    end
1525    if mapping then
1526        local startchar = getchar(start)
1527        local ligatures = mapping[startchar]
1528        if not ligatures then
1529            if trace_bugs then
1530                logwarning("%s: no ligatures starting with %s",cref(dataset,sequence,chainindex),gref(startchar))
1531            end
1532        else
1533            local hasmarks        = marks[startchar]
1534            local current         = getnext(start)
1535            local discfound       = false
1536            local last            = stop
1537            local nofreplacements = 1
1538            while current do
1539                local nxt, char, id = isnextchar(current,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
1540                if char then
1541                    if current and hasglyphoption(current,no_left_ligature_code) then
1542                        break
1543                    end
1544                    if skiphash and skiphash[char] then -- marks
1545                        -- if current == stop then -- maybe add this
1546                        --     break
1547                        -- else
1548                            current = nxt
1549                        -- end
1550                    else
1551                        local lg = not tonumber(ligatures) and ligatures[char]
1552                        if lg then
1553                            ligatures       = lg
1554                            last            = current
1555                            nofreplacements = nofreplacements + 1
1556                            if marks[char] then
1557                                hasmarks = true
1558                            end
1559                            if current == stop then
1560                                break
1561                            else
1562                                current = nxt
1563                            end
1564                        else
1565                            break
1566                        end
1567                    end
1568                elseif id == disc_code then
1569                    if not discfound then
1570                        discfound = current
1571                    end
1572                    if current == stop then
1573                        break -- okay? or before the disc
1574                    else
1575                        current = nxt
1576                    end
1577                else
1578                    break
1579                end
1580            end
1581            -- we can avoid the tonumber as already done but messy
1582            local ligature = tonumber(ligatures) or ligatures.ligature
1583            if ligature then
1584                if chainindex then
1585                    stop = last
1586                end
1587                if trace_ligatures then
1588                    if start == stop then
1589                        logprocess("%s: replacing character %s by ligature %s case 3",cref(dataset,sequence,chainindex),gref(startchar),gref(ligature))
1590                    else
1591                        logprocess("%s: replacing character %s upto %s by ligature %s case 4",cref(dataset,sequence,chainindex),gref(startchar),gref(getchar(stop)),gref(ligature))
1592                    end
1593                end
1594                head, start = toligature(head,start,stop,ligature,dataset,sequence,skiphash,discfound,hasmarks)
1595                return head, start, true, nofreplacements, discfound
1596            elseif trace_bugs then
1597                if start == stop then
1598                    logwarning("%s: replacing character %s by ligature fails",cref(dataset,sequence,chainindex),gref(startchar))
1599                else
1600                    logwarning("%s: replacing character %s upto %s by ligature fails",cref(dataset,sequence,chainindex),gref(startchar),gref(getchar(stop)))
1601                end
1602            end
1603        end
1604    end
1605    return head, start, false, 0, false
1606end
1607
1608function chainprocs.gpos_single(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash,chainindex)
1609    -- we actually should check no_left_kern_code with next
1610    if not hasglyphoption(start,no_right_kern_code) then
1611        local mapping = currentlookup.mapping
1612        if mapping == nil then
1613            mapping = getmapping(dataset,sequence,currentlookup)
1614        end
1615        if mapping then
1616            local startchar = getchar(start)
1617            local kerns     = mapping[startchar]
1618            if kerns then
1619                local format = currentlookup.format
1620                if format == "single" then
1621                    local dx, dy, w, h = setposition(0,start,factor,rlmode,kerns) -- currentlookup.flags ?
1622                    if trace_kerns then
1623                        logprocess("%s: shifting single %s by %s (%p,%p) and correction (%p,%p)",cref(dataset,sequence),gref(startchar),format,dx,dy,w,h)
1624                    end
1625                else -- needs checking .. maybe no kerns format for single
1626                    local k = (format == "move" and setmove or setkern)(start,factor,rlmode,kerns,injection)
1627                    if trace_kerns then
1628                        logprocess("%s: shifting single %s by %s %p",cref(dataset,sequence),gref(startchar),format,k)
1629                    end
1630                end
1631                return head, start, true
1632            end
1633        end
1634    end
1635    return head, start, false
1636end
1637
1638function chainprocs.gpos_pair(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash,chainindex) -- todo: injections ?
1639    -- we actually should check no_left_kern_code with next
1640    if not hasglyphoption(start,no_right_kern_code) then
1641        local mapping = currentlookup.mapping
1642        if mapping == nil then
1643            mapping = getmapping(dataset,sequence,currentlookup)
1644        end
1645        if mapping then
1646            local snext = getnext(start)
1647            if snext then
1648                local startchar = getchar(start)
1649                local kerns     = mapping[startchar] -- always 1 step
1650                if kerns then
1651                    local prev = start
1652                    while snext do
1653                        local nxt, nextchar = isnextchar(snext,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
1654                        if not nextchar then
1655                            break
1656                        elseif skiphash and skiphash[nextchar] then
1657                            prev  = snext
1658                            snext = nxt
1659                        else
1660                            local krn = kerns[nextchar]
1661                            if not krn then
1662                                break
1663                            end
1664                            local format = currentlookup.format
1665                            if format == "pair" then
1666                                local a = krn[1]
1667                                local b = krn[2]
1668                                if a == true then
1669                                    -- zero
1670                                elseif a then
1671                                    local x, y, w, h = setposition(1,start,factor,rlmode,a,"injections") -- currentlookups flags?
1672                                    if trace_kerns then
1673                                        local startchar = getchar(start)
1674                                        logprocess("%s: shifting first of pair %s and %s by (%p,%p) and correction (%p,%p)",cref(dataset,sequence),gref(startchar),gref(nextchar),x,y,w,h)
1675                                    end
1676                                end
1677                                if b == true then
1678                                    -- zero
1679                                    start = snext -- cf spec
1680                                elseif b then -- #b > 0
1681                                    local x, y, w, h = setposition(2,snext,factor,rlmode,b,"injections")
1682                                    if trace_kerns then
1683                                        local startchar = getchar(start)
1684                                        logprocess("%s: shifting second of pair %s and %s by (%p,%p) and correction (%p,%p)",cref(dataset,sequence),gref(startchar),gref(nextchar),x,y,w,h)
1685                                    end
1686                                    start = snext -- cf spec
1687                                elseif forcepairadvance then
1688                                    start = snext -- for testing, not cf spec
1689                                end
1690                                return head, start, true
1691                            elseif krn ~= 0 then
1692                                local k = (format == "move" and setmove or setkern)(snext,factor,rlmode,krn)
1693                                if trace_kerns then
1694                                    logprocess("%s: inserting %s %p between %s and %s",cref(dataset,sequence),format,k,gref(getchar(prev)),gref(nextchar))
1695                                end
1696                                return head, start, true
1697                            else
1698                                break
1699                            end
1700                        end
1701                    end
1702                end
1703            end
1704        end
1705    end
1706    return head, start, false
1707end
1708
1709function chainprocs.gpos_mark2base(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash,chainindex)
1710    local mapping = currentlookup.mapping
1711    if mapping == nil then
1712        mapping = getmapping(dataset,sequence,currentlookup)
1713    end
1714    if mapping then
1715        local markchar = getchar(start)
1716        if marks[markchar] then
1717            local markanchors = mapping[markchar] -- always 1 step
1718            if markanchors then
1719                local base = getprev(start) -- [glyph] [start=mark]
1720                if base then
1721                    local prv, basechar = isprevchar(base,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
1722                    if basechar then
1723                        if marks[basechar] then
1724                            -- todo: use prv
1725                            while base do
1726                                base = getprev(base)
1727                                if base then
1728                                    local prv, basechar = isprevchar(base,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
1729                                    if basechar then
1730                                        if not marks[basechar] then
1731                                            break
1732                                        end
1733                                    else
1734                                        if trace_bugs then
1735                                            logwarning("%s: no base for mark %s, case %i",pref(dataset,sequence),gref(markchar),1)
1736                                        end
1737                                        return head, start, false
1738                                    end
1739                                else
1740                                    if trace_bugs then
1741                                        logwarning("%s: no base for mark %s, case %i",pref(dataset,sequence),gref(markchar),2)
1742                                    end
1743                                    return head, start, false
1744                                end
1745                            end
1746                        end
1747                        local ba = markanchors[1][basechar]
1748                        if ba then
1749                            local ma = markanchors[2]
1750                            if ma then
1751                                local dx, dy, bound = setmark(start,base,factor,rlmode,ba,ma,characters[basechar],false,checkmarks)
1752                                if trace_marks then
1753                                    logprocess("%s, bound %s, anchoring mark %s to basechar %s => (%p,%p)",
1754                                        cref(dataset,sequence),bound,gref(markchar),gref(basechar),dx,dy)
1755                                end
1756                                return head, start, true
1757                            end
1758                        end
1759                    elseif trace_bugs then
1760                        logwarning("%s: prev node is no char, case %i",cref(dataset,sequence),1)
1761                    end
1762                elseif trace_bugs then
1763                    logwarning("%s: prev node is no char, case %i",cref(dataset,sequence),2)
1764                end
1765            elseif trace_bugs then
1766                logwarning("%s: mark %s has no anchors",cref(dataset,sequence),gref(markchar))
1767            end
1768        elseif trace_bugs then
1769            logwarning("%s: mark %s is no mark",cref(dataset,sequence),gref(markchar))
1770        end
1771    end
1772    return head, start, false
1773end
1774
1775function chainprocs.gpos_mark2ligature(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash,chainindex)
1776    local mapping = currentlookup.mapping
1777    if mapping == nil then
1778        mapping = getmapping(dataset,sequence,currentlookup)
1779    end
1780    if mapping then
1781        local markchar = getchar(start)
1782        if marks[markchar] then
1783            local markanchors = mapping[markchar] -- always 1 step
1784            if markanchors then
1785                local base = getprev(start) -- [glyph] [optional marks] [start=mark]
1786                if base then
1787                    local prv, basechar = isprevchar(base,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
1788                    if basechar then
1789                        if marks[basechar] then
1790                            -- todo: use prv
1791                            while base do
1792                                base = getprev(base)
1793                                if base then
1794                                    local prv, basechar = isprevchar(base,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
1795                                    if basechar then
1796                                        if not marks[basechar] then
1797                                            break
1798                                        end
1799                                    else
1800                                        if trace_bugs then
1801                                            logwarning("%s: no base for mark %s, case %i",cref(dataset,sequence),markchar,1)
1802                                        end
1803                                        return head, start, false
1804                                    end
1805                                else
1806                                    if trace_bugs then
1807                                        logwarning("%s: no base for mark %s, case %i",cref(dataset,sequence),markchar,2)
1808                                    end
1809                                    return head, start, false
1810                                end
1811                            end
1812                        end
1813                        local ba = markanchors[1][basechar]
1814                        if ba then
1815                            local ma = markanchors[2]
1816                            if ma then
1817                                local index = getligaindex(start)
1818                                ba = ba[index]
1819                                if ba then
1820                                    local dx, dy, bound = setmark(start,base,factor,rlmode,ba,ma,characters[basechar],false,checkmarks)
1821                                    if trace_marks then
1822                                        logprocess("%s, bound %s, anchoring mark %s to baselig %s at index %s => (%p,%p)",
1823                                            cref(dataset,sequence),a or bound,gref(markchar),gref(basechar),index,dx,dy)
1824                                    end
1825                                    return head, start, true
1826                                end
1827                            end
1828                        end
1829                    elseif trace_bugs then
1830                        logwarning("%s, prev node is no char, case %i",cref(dataset,sequence),1)
1831                    end
1832                elseif trace_bugs then
1833                    logwarning("%s, prev node is no char, case %i",cref(dataset,sequence),2)
1834                end
1835            elseif trace_bugs then
1836                logwarning("%s, mark %s has no anchors",cref(dataset,sequence),gref(markchar))
1837            end
1838        elseif trace_bugs then
1839            logwarning("%s, mark %s is no mark",cref(dataset,sequence),gref(markchar))
1840        end
1841    end
1842    return head, start, false
1843end
1844
1845function chainprocs.gpos_mark2mark(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash,chainindex)
1846    local mapping = currentlookup.mapping
1847    if mapping == nil then
1848        mapping = getmapping(dataset,sequence,currentlookup)
1849    end
1850    if mapping then
1851        local markchar = getchar(start)
1852        if marks[markchar] then
1853            local markanchors = mapping[markchar] -- always 1 step
1854            if markanchors then
1855                local base = getprev(start) -- [glyph] [basemark] [start=mark]
1856                local slc = getligaindex(start)
1857                if slc then -- a rather messy loop ... needs checking with husayni
1858                    while base do
1859                        local blc = getligaindex(base)
1860                        if blc and blc ~= slc then
1861                            base = getprev(base)
1862                        else
1863                            break
1864                        end
1865                    end
1866                end
1867                if base then -- subtype test can go
1868                    local nxt, basechar = isnextchar(base,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
1869                    if basechar then
1870                        local ba = markanchors[1][basechar]
1871                        if ba then
1872                            local ma = markanchors[2]
1873                            if ma then
1874                                local dx, dy, bound = setmark(start,base,factor,rlmode,ba,ma,characters[basechar],true,checkmarks)
1875                                if trace_marks then
1876                                    logprocess("%s, bound %s, anchoring mark %s to basemark %s => (%p,%p)",
1877                                        cref(dataset,sequence),bound,gref(markchar),gref(basechar),dx,dy)
1878                                end
1879                                return head, start, true
1880                            end
1881                        end
1882                    elseif trace_bugs then
1883                        logwarning("%s: prev node is no mark, case %i",cref(dataset,sequence),1)
1884                    end
1885                elseif trace_bugs then
1886                    logwarning("%s: prev node is no mark, case %i",cref(dataset,sequence),2)
1887                end
1888            elseif trace_bugs then
1889                logwarning("%s: mark %s has no anchors",cref(dataset,sequence),gref(markchar))
1890            end
1891        elseif trace_bugs then
1892            logwarning("%s: mark %s is no mark",cref(dataset,sequence),gref(markchar))
1893        end
1894    end
1895    return head, start, false
1896end
1897
1898function chainprocs.gpos_cursive(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash,chainindex)
1899    local mapping = currentlookup.mapping
1900    if mapping == nil then
1901        mapping = getmapping(dataset,sequence,currentlookup)
1902    end
1903    if mapping then
1904        local startchar   = getchar(start)
1905        local exitanchors = mapping[startchar] -- always 1 step
1906        if exitanchors then
1907            if marks[startchar] then
1908                if trace_cursive then
1909                    logprocess("%s: ignoring cursive for mark %s",pref(dataset,sequence),gref(startchar))
1910                end
1911            else
1912                local nxt = getnext(start)
1913                while nxt do
1914                    local n, nextchar = isnextchar(nxt,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
1915                    if not nextchar then
1916                        break
1917                    elseif marks[nextchar] then
1918                        -- should not happen (maybe warning)
1919                        nxt = n
1920                    else
1921                        local exit = exitanchors[3]
1922                        if exit then
1923                            local entry = exitanchors[1][nextchar]
1924                            if entry then
1925                                entry = entry[2]
1926                                if entry then
1927                                    local r2lflag = sequence.flags[4] -- mentioned in the standard
1928                                    local dx, dy, bound = setcursive(start,nxt,factor,rlmode,exit,entry,characters[startchar],characters[nextchar],r2lflag)
1929                                    if trace_cursive then
1930                                        logprocess("%s: moving %s to %s cursive (%p,%p) using bound %s in %s mode",pref(dataset,sequence),gref(startchar),gref(nextchar),dx,dy,bound,mref(rlmode))
1931                                    end
1932                                    return head, start, true
1933                                end
1934                            end
1935                        elseif trace_bugs then
1936                            onetimemessage(currentfont,startchar,"no entry anchors")
1937                        end
1938                        break
1939                    end
1940                end
1941            end
1942        elseif trace_cursive and trace_details then
1943            logprocess("%s, cursive %s is already done",pref(dataset,sequence),gref(getchar(start)),alreadydone)
1944        end
1945    end
1946    return head, start, false
1947end
1948
1949-- what pointer to return, spec says stop
1950-- to be discussed ... is bidi changer a space?
1951-- elseif char == zwnj and sequence[n][32] then -- brrr
1952
1953local function show_skip(dataset,sequence,char,ck,class)
1954    logwarning("%s: skipping char %s, class %a, rule %a, lookuptype %a",cref(dataset,sequence),gref(char),class,ck[1],ck[8] or ck[2])
1955end
1956
1957-- A previous version had disc collapsing code in the (single sub) handler plus some
1958-- checking in the main loop, but that left the pre/post sequences undone. The best
1959-- solution is to add some checking there and backtrack when a replace/post matches
1960-- but it takes a bit of work to figure out an efficient way (this is what the
1961-- sweep* names refer to). I might look into that variant one day again as it can
1962-- replace some other code too. In that approach we can have a special version for
1963-- gub and pos which gains some speed. This method does the test and passes info to
1964-- the handlers. Here collapsing is handled in the main loop which also makes code
1965-- elsewhere simpler (i.e. no need for the other special runners and disc code in
1966-- ligature building). I also experimented with pushing preceding glyphs sequences
1967-- in the replace/pre fields beforehand which saves checking afterwards but at the
1968-- cost of duplicate glyphs (memory) but it's too much overhead (runtime).
1969--
1970-- In the meantime Kai had moved the code from the single chain into a more general
1971-- handler and this one (renamed to chaindisk) is used now. I optimized the code a
1972-- bit and brought it in sycn with the other code. Hopefully I didn't introduce
1973-- errors. Note: this somewhat complex approach is meant for fonts that implement
1974-- (for instance) ligatures by character replacement which to some extend is not
1975-- that suitable for hyphenation. I also use some helpers. This method passes some
1976-- states but reparses the list. There is room for a bit of speed up but that will
1977-- be done in the context version. (In fact a partial rewrite of all code can bring
1978-- some more efficiency.)
1979--
1980-- I didn't test it with extremes but successive disc nodes still can give issues
1981-- but in order to handle that we need more complex code which also slows down even
1982-- more. The main loop variant could deal with that: test, collapse, backtrack.
1983
1984local newuserkern = nuts.pool.newkern
1985local newboundary = nuts.pool.boundary
1986
1987local function checked(head)
1988    local current = head
1989    while current do
1990        if getid(current) == glue_code then
1991            local kern = newuserkern(getwidth(current))
1992            if head == current then
1993                local next = getnext(current)
1994                if next then
1995                    setlink(kern,next)
1996                end
1997                flushnode(current)
1998                head    = kern
1999                current = next
2000            else
2001                local prev, next = getboth(current)
2002                setlink(prev,kern,next)
2003                flushnode(current)
2004                current = next
2005            end
2006        else
2007            current = getnext(current)
2008        end
2009    end
2010    return head
2011end
2012
2013local function setdiscchecked(d,pre,post,replace)
2014    if pre     then pre     = checked(pre)     end
2015    if post    then post    = checked(post)    end
2016    if replace then replace = checked(replace) end
2017    setdisc(d,pre,post,replace)
2018end
2019
2020local function chainrun(head,start,last,dataset,sequence,rlmode,skiphash,ck,where)
2021
2022    local size         = ck[5] - ck[4] + 1
2023    local chainlookups = ck[6]
2024    local done         = false
2025    -- current match
2026    if chainlookups then
2027        -- Lookups can be like { 1, false, 3 } or { false, 2 } or basically anything and
2028        -- #lookups can be less than #current
2029
2030        if size == 1 then
2031
2032         -- if nofchainlookups > size then
2033         --     -- bad rules
2034         -- end
2035            local chainlookup = chainlookups[1]
2036            if chainlookup then
2037                for j=1,#chainlookup do
2038                    local chainstep = chainlookup[j]
2039                    if chainstep then
2040                        local chainkind = chainstep.type
2041                        local chainproc = chainprocs[chainkind]
2042                        if chainproc then
2043                            local ok
2044                         -- HH: chainindex 1 added here (for KAI to check too), there are weird ligatures e.g.
2045                         -- char + mark -> char where mark has to disappear
2046                         -- head, start, ok = chainproc(head,start,last,dataset,sequence,chainstep,rlmode,skiphash,1)
2047                            head, start, ok = chainproc(head,start,last,dataset,sequence,chainstep,rlmode,skiphash)
2048                            if ok then
2049                                done = true
2050                            end
2051                        else
2052                            logprocess("%s: %s is not yet supported (1)",cref(dataset,sequence),chainkind)
2053                        end
2054                    else
2055                        logprocess("%s: has an issue (1)",cref(dataset,sequence))
2056                    end
2057                end
2058            else
2059                -- whatever
2060            end
2061
2062         else
2063
2064            -- See LookupType 5: Contextual Substitution Subtable. Now it becomes messy. The
2065            -- easiest case is where #current maps on #lookups i.e. one-to-one. But what if
2066            -- we have a ligature. Cf the spec we then need to advance one character but we
2067            -- really need to test it as there are fonts out there that are fuzzy and have
2068            -- too many lookups:
2069            --
2070            -- U+1105 U+119E U+1105 U+119E : sourcehansansklight: script=hang ccmp=yes
2071            --
2072            -- Even worse are these family emoji shapes as they can have multiple lookups
2073            -- per slot (probably only for gpos).
2074
2075            -- It's very unlikely that we will have skip classes here but still ... we seldom
2076            -- enter this branch anyway.
2077
2078            local i = 1
2079            local laststart = start
2080            local nofchainlookups = #chainlookups -- useful?
2081            while start do
2082                if skiphash then -- hm, so we know we skip some
2083                    while start do
2084                        local nxt, char = isnextchar(start,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
2085                        if char then
2086                            if skiphash and skiphash[char] then
2087                                start = nxt
2088                            else
2089                                break
2090                            end
2091                        else
2092                            break
2093                        end
2094                    end
2095                end
2096                local chainlookup = chainlookups[i]
2097                if chainlookup then
2098                    for j=1,#chainlookup do
2099                        local chainstep = chainlookup[j]
2100                        if chainstep then
2101                            local chainkind = chainstep.type
2102                            local chainproc = chainprocs[chainkind]
2103                            if chainproc then
2104                                local ok, n
2105                                head, start, ok, n = chainproc(head,start,last,dataset,sequence,chainstep,rlmode,skiphash,i)
2106                                -- messy since last can be changed !
2107                                if ok then
2108                                    done = true
2109                                    if n and n > 1 and i + n > nofchainlookups then
2110                                        -- this is a safeguard, we just ignore the rest of the lookups
2111                                        i = size -- prevents an advance
2112                                        break
2113                                    end
2114                                end
2115                            else
2116                                -- actually an error
2117                                logprocess("%s: %s is not yet supported (2)",cref(dataset,sequence),chainkind)
2118                            end
2119                        else
2120                            -- actually an error
2121                            logprocess("%s: has an issue (2)",cref(dataset,sequence))
2122                        end
2123                    end
2124                else
2125                    -- we skip but we could also delete as option .. what does an empty lookup actually mean
2126                    -- in opentype ... anyway, we could map it onto gsub_remove if needed
2127                end
2128                i = i + 1
2129                if i > size or not start then
2130                    break
2131                elseif start then
2132                    laststart = start
2133                    start = getnext(start)
2134                end
2135            end
2136            if not start then
2137                start = laststart
2138            end
2139
2140        end
2141    else
2142        -- todo: needs checking for holes in the replacements
2143        local replacements = ck[7]
2144        if replacements then
2145            head, start, done = reversesub(head,start,last,dataset,sequence,replacements,rlmode,skiphash)
2146        else
2147            done = true
2148            if trace_contexts then
2149                logprocess("%s: skipping match @ %i",cref(dataset,sequence),where)
2150            end
2151        end
2152    end
2153    return head, start, done
2154end
2155
2156local function chaindisk(head,start,dataset,sequence,rlmode,skiphash,ck)
2157
2158    if not start then
2159        return head, start, false
2160    end
2161
2162    local startishead   = start == head
2163    local seq           = ck[3]
2164    local f             = ck[4]
2165    local l             = ck[5]
2166    local s             = #seq
2167    local done          = false
2168    local sweepnode     = sweepnode
2169    local sweeptype     = sweeptype
2170    local sweepoverflow = false
2171    local checkdisc     = getprev(head)
2172    local keepdisc      = not sweepnode
2173    local lookaheaddisc = nil
2174    local backtrackdisc = nil
2175    local current       = start
2176    local last          = start
2177    local prev          = getprev(start)
2178    local hasglue       = false
2179    local useddisc      = nil   -- new 2022-09-25
2180    local usedstart     = start -- new 2022-09-25
2181
2182    -- fishy: so we can overflow and then go on in the sweep?
2183    -- todo : id can also be glue_code as we checked spaces
2184
2185    local i = f
2186    while i <= l do
2187        local id = getid(current)
2188        if id == glyph_code then
2189            i       = i + 1
2190            last    = current
2191            current = getnext(current)
2192        elseif id == glue_code then
2193            i       = i + 1
2194            last    = current
2195            current = getnext(current)
2196            hasglue = true
2197        elseif id == disc_code then
2198            if keepdisc then
2199                keepdisc      = false
2200                lookaheaddisc = current
2201                local replace = getreplace(current)
2202                if not replace then
2203                    sweepoverflow = true
2204                    sweepnode     = current
2205                    current       = getnext(current)
2206                else
2207                    -- we can use an iterator
2208                    while replace and i <= l do
2209                        if getid(replace) == glyph_code then
2210                            i = i + 1
2211                        end
2212                        replace = getnext(replace)
2213                    end
2214                    current = getnext(replace)
2215                end
2216                last = current
2217            else
2218                head, current = flattendisk(head,current)
2219            end
2220        else
2221            last    = current
2222            current = getnext(current)
2223        end
2224        if current then
2225            -- go on
2226        elseif sweepoverflow then
2227            -- we already are following up on sweepnode
2228            break
2229        elseif sweeptype == "post" or sweeptype == "replace" then
2230            current = getnext(sweepnode)
2231            if current then
2232                sweeptype     = nil
2233                sweepoverflow = true
2234            else
2235                break
2236            end
2237        else
2238            break -- added
2239        end
2240    end
2241
2242    if sweepoverflow then
2243        local prev = current and getprev(current)
2244        if not current or prev ~= sweepnode then
2245            local head = getnext(sweepnode)
2246            local tail = nil
2247            if prev then
2248                tail = prev
2249                setprev(current,sweepnode)
2250            else
2251                tail = findnodetail(head)
2252            end
2253            setnext(sweepnode,current)
2254            setprev(head)
2255            setnext(tail)
2256            appenddisc(sweepnode,head)
2257        end
2258    end
2259
2260    if l < s then
2261        local i = l
2262        local t = sweeptype == "post" or sweeptype == "replace"
2263        while current and i < s do
2264            local id = getid(current)
2265            if id == glyph_code then
2266                i       = i + 1
2267                current = getnext(current)
2268            elseif id == glue_code then
2269                i       = i + 1
2270                current = getnext(current)
2271                hasglue = true
2272            elseif id == disc_code then
2273                if keepdisc then
2274                    keepdisc = false
2275                    if notmatchpre[current] ~= notmatchreplace[current] then
2276                        lookaheaddisc = current
2277                    end
2278                    -- we assume a simple text only replace (we could use nuts.count)
2279                    local replace = getreplace(current)
2280                    while replace and i < s do
2281                        if getid(replace) == glyph_code then
2282                            i = i + 1
2283                        end
2284                        replace = getnext(replace)
2285                    end
2286                    current = getnext(current)
2287                elseif notmatchpre[current] ~= notmatchreplace[current] then
2288                    head, current = flattendisk(head,current)
2289                else
2290                    current = getnext(current) -- HH
2291                end
2292            else
2293                current = getnext(current)
2294            end
2295            if not current and t then
2296                current = getnext(sweepnode)
2297                if current then
2298                    sweeptype = nil
2299                end
2300            end
2301        end
2302    end
2303
2304    if f > 1 then
2305        local current = prev
2306        local i       = f
2307        local t       = sweeptype == "pre" or sweeptype == "replace"
2308        if not current and t and current == checkdisc then
2309            current = getprev(sweepnode)
2310        end
2311        while current and i > 1 do -- missing getprev added / moved outside
2312            local id = getid(current)
2313            if id == glyph_code then
2314                i = i - 1
2315            elseif id == glue_code then
2316                i       = i - 1
2317                hasglue = true
2318            elseif id == disc_code then
2319                if keepdisc then
2320                    keepdisc = false
2321                    if notmatchpost[current] ~= notmatchreplace[current] then
2322-- print(notmatchpost[current],notmatchreplace[current])
2323                        backtrackdisc = current
2324                    end
2325                    -- we assume a simple text only replace (we could use nuts.count)
2326                    local replace = getreplace(current)
2327                    while replace and i > 1 do
2328                        if getid(replace) == glyph_code then
2329                            i = i - 1
2330                        end
2331                        replace = getnext(replace)
2332                    end
2333                elseif notmatchpost[current] ~= notmatchreplace[current] then
2334                    head, current = flattendisk(head,current)
2335                end
2336            end
2337            current = getprev(current)
2338            if t and current == checkdisc then
2339                current = getprev(sweepnode)
2340            end
2341        end
2342    end
2343
2344    local done = false
2345
2346    if lookaheaddisc then
2347        local cf            = start
2348        local cl            = getprev(lookaheaddisc)
2349        local cprev         = getprev(start)
2350        local insertedmarks = 0
2351        while cprev do
2352            local nxt, char = isnextchar(cf,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
2353            if char and marks[char] then
2354                insertedmarks = insertedmarks + 1
2355                cf            = cprev
2356                startishead   = cf == head
2357                cprev         = getprev(cprev)
2358            else
2359                break
2360            end
2361        end
2362        setlink(cprev,lookaheaddisc)
2363        setprev(cf)
2364        setnext(cl)
2365        if startishead then
2366            head = lookaheaddisc
2367        end
2368        local pre, post, replace = getdisc(lookaheaddisc)
2369        local new  = copynodelist(cf) -- br, how often does that happen
2370        local cnew = new
2371        if pre then
2372            setlink(findnodetail(cf),pre)
2373        end
2374        if replace then
2375            local tail = findnodetail(new)
2376            setlink(tail,replace)
2377        end
2378        for i=1,insertedmarks do
2379            cnew = getnext(cnew)
2380        end
2381        cl = start
2382        local clast = cnew
2383        for i=f,l do
2384            cl    = getnext(cl)
2385            clast = getnext(clast)
2386        end
2387        if not notmatchpre[lookaheaddisc] then
2388            local ok = false
2389            cf, start, ok = chainrun(cf,start,cl,dataset,sequence,rlmode,skiphash,ck,1)
2390            if ok then
2391                done = true
2392            end
2393        end
2394        if not notmatchreplace[lookaheaddisc] then
2395            local ok = false
2396            new, cnew, ok = chainrun(new,cnew,clast,dataset,sequence,rlmode,skiphash,ck,2)
2397            if ok then
2398                done = true
2399            end
2400        end
2401        if hasglue then
2402            setdiscchecked(lookaheaddisc,cf,post,new)
2403        else
2404            setdisc(lookaheaddisc,cf,post,new)
2405        end
2406        start          = getprev(lookaheaddisc)
2407if done then -- new 2022-12-22 : discretionary-001.tex
2408        useddisc       = lookaheaddisc -- new 2022-09-25
2409end
2410        sweephead[cf]  = getnext(clast) or false
2411        sweephead[new] = getnext(cl) or false
2412    elseif backtrackdisc then
2413        local cf            = getnext(backtrackdisc)
2414        local cl            = start
2415        local cnext         = getnext(start)
2416        local insertedmarks = 0
2417        while cnext do
2418            local nxt, char = isnextchar(cnext,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
2419            if char and marks[char] then
2420                insertedmarks = insertedmarks + 1
2421                cl            = cnext
2422                cnext         = nxt
2423            else
2424                break
2425            end
2426        end
2427        setlink(backtrackdisc,cnext)
2428        setprev(cf)
2429        setnext(cl)
2430        local pre, post, replace, pretail, posttail, replacetail = getdisc(backtrackdisc,true)
2431        local new  = copynodelist(cf)
2432        local cnew = findnodetail(new)
2433        for i=1,insertedmarks do
2434            cnew = getprev(cnew)
2435        end
2436        local clast = cnew
2437        for i=f,l do
2438            clast = getnext(clast)
2439        end
2440        if not notmatchpost[backtrackdisc] then
2441            local ok = false
2442            cf, start, ok = chainrun(cf,start,last,dataset,sequence,rlmode,skiphash,ck,3)
2443            if ok then
2444                done = true
2445            end
2446        end
2447        if not notmatchreplace[backtrackdisc] then
2448            local ok = false
2449            new, cnew, ok = chainrun(new,cnew,clast,dataset,sequence,rlmode,skiphash,ck,4)
2450            if ok then
2451                done = true
2452            end
2453        end
2454        if post then
2455            setlink(posttail,cf)
2456        else
2457            post = cf
2458        end
2459        if replace then
2460            setlink(replacetail,new)
2461        else
2462            replace = new
2463        end
2464        if hasglue then
2465            setdiscchecked(backtrackdisc,pre,post,replace)
2466        else
2467            setdisc(backtrackdisc,pre,post,replace)
2468        end
2469        start              = getprev(backtrackdisc)
2470if done then -- new 2022-12-22 : discretionary-001.tex
2471        useddisc           = backtrackdisc -- new 2022-09-25
2472end
2473        sweephead[post]    = getnext(clast) or false
2474        sweephead[replace] = getnext(last) or false
2475    else
2476
2477        local ok = false
2478        head, start, ok = chainrun(head,start,last,dataset,sequence,rlmode,skiphash,ck,5)
2479        if ok then
2480            done = true
2481        end
2482
2483    end
2484-- test case ?
2485    if useddisc and start ~= usedstart then -- make this option per font -- new 2022-09-25
2486       start = getnext(start)                                            -- new 2022-09-25
2487    end                                                                  -- new 2022-09-25
2488    return head, start, done, useddisc                                   -- new 2022-09-25
2489end
2490
2491local chaintrac do
2492
2493    local level = 0
2494    local last  = { }
2495
2496    chaintrac = function(head,start,dataset,sequence,rlmode,skiphash,ck,match,discseen,sweepnode)
2497        if dataset then
2498            level = level + 1
2499            last[level] = start
2500            local rule       = ck[1]
2501            local lookuptype = ck[8] or ck[2]
2502            local nofseq     = #ck[3] -- ck[3].n
2503            local first      = ck[4]
2504            local last       = ck[5]
2505            local char       = getchar(start)
2506            logwarning("+ %i : %s: rule %s %s at char %s for (%s,%s,%s) chars, lookuptype %a, %sdisc seen, %ssweeping",
2507                level,cref(dataset,sequence),rule,match and "matches" or "nomatch",
2508                gref(char),first-1,last-first+1,nofseq-last,lookuptype,
2509                discseen and "" or "no ", sweepnode and "" or "not ")
2510        else
2511            -- (start,done)
2512            local what  = start and "done" or "continue"
2513            local where = head == last[level] and "same" or "different"
2514            local char  = getchar(head)
2515            if char then
2516                logwarning("- %i : %s at char %s, %s node",level,what,gref(char),where)
2517            else
2518                logwarning("- %i : %s, %s node",level,what,where)
2519            end
2520            level = level - 1
2521        end
2522    end
2523
2524end
2525
2526-- The next one is quite optimized but still somewhat slow, fonts like ebgaramond
2527-- are real torture tests because they have many steps with one context (having
2528-- multiple contexts makes more sense) also because we (can) reduce them. Instead of
2529-- a match boolean variable and check for that I decided to use a goto with labels
2530-- instead. This is one of the cases where it makes the code more readable and we
2531-- might even gain a bit performance.
2532
2533-- when we have less replacements (lookups) then current matches we can push too much into
2534-- the previous disc .. such be it (<before><disc><current=fl><after> with only f done)
2535
2536local function handle_contextchain(head,start,dataset,sequence,contexts,rlmode,skiphash)
2537    if not contexts then
2538        return head, start, false
2539    end
2540    -- optimizing for rlmode gains nothing
2541    local sweepnode    = sweepnode
2542    local sweeptype    = sweeptype
2543    local postreplace
2544    local prereplace
2545    local checkdisc
2546    local discseen  -- = false
2547    if sweeptype then
2548        if sweeptype == "replace" then
2549            postreplace = true
2550            prereplace  = true
2551        else
2552            postreplace = sweeptype == "post"
2553            prereplace  = sweeptype == "pre"
2554        end
2555        checkdisc = getprev(head)
2556    end
2557 -- local currentfont   = currentfont   -- really ?
2558 -- local currentscale  = currentscale  -- really ?
2559 -- local currentxscale = currentxscale -- really ?
2560 -- local currentyscale = currentyscale -- really ?
2561
2562    local skipped   -- = false
2563
2564    local startprev,
2565          startnext    = getboth(start)
2566    local done      -- = false
2567
2568    -- we can have multiple hits and as we scan (currently) all we need to check
2569    -- if we have a match ... contextchains have no real coverage table (with
2570    -- unique entries)
2571
2572    -- fonts can have many steps (each doing one check) or many contexts
2573
2574    -- todo: make a per-char cache so that we have small contexts (when we have a context
2575    -- n == 1 and otherwise it can be more so we can even distinguish n == 1 or more)
2576
2577    local nofcontexts = contexts.n -- #contexts
2578
2579    local startchar = nofcontext == 1 or ischar(start,currentfont) -- already checked
2580    for k=1,nofcontexts do -- does this disc mess work well with n > 1
2581
2582        local ck   = contexts[k]
2583        local seq  = ck[3]
2584        local f    = ck[4] -- first current
2585        local last = start
2586        if not startchar or not seq[f][startchar] then
2587            -- report("no hit in %a at %i of %i contexts",sequence.type,k,nofcontexts)
2588            goto next
2589        end
2590        local s = seq.n -- or #seq
2591        if s == 1 then
2592            -- bit weird case: why use a chain, but it is a hit
2593        else
2594            local l       = ck[5] -- last current
2595            local current = start
2596         -- local last    = start
2597
2598            -- current match
2599
2600            if l > f then
2601                -- before/current/after | before/current | current/after
2602                local discfound -- = nil
2603                local n = f + 1
2604                last = startnext -- the second in current (first already matched)
2605                while n <= l do
2606                    if postreplace and not last then
2607                        last      = getnext(sweepnode)
2608                        sweeptype = nil
2609                    end
2610                    if last then
2611                        local nxt, char, id = isnextchar(last,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
2612                        if char then
2613                            if seq[n][char] then
2614                                if n < l then
2615                                    last = nxt
2616                                end
2617                                n = n + 1
2618                            elseif skiphash and skiphash[char] then
2619                                skipped = true
2620                                if trace_skips then
2621                                    show_skip(dataset,sequence,char,ck,classes[char])
2622                                end
2623                                last = nxt
2624                            elseif discfound then
2625                                notmatchreplace[discfound] = true
2626                                if notmatchpre[discfound] then
2627                                    goto next
2628                                else
2629                                    break
2630                                end
2631                            else
2632                                goto next
2633                            end
2634                        elseif char == false then
2635                            if discfound then
2636                                notmatchreplace[discfound] = true
2637                                if notmatchpre[discfound] then
2638                                    goto next
2639                                else
2640                                    break
2641                                end
2642                            else
2643                                goto next
2644                            end
2645                        elseif id == disc_code then
2646                     -- elseif id == disc_code and (not discs or discs[last]) then
2647                            discseen              = true
2648                            discfound             = last
2649                            notmatchpre[last]     = nil
2650                            notmatchpost[last]    = true
2651                            notmatchreplace[last] = nil
2652                            local pre, post, replace = getdisc(last)
2653                            if pre then
2654                                local n = n
2655                                while pre do
2656                                    if seq[n][getchar(pre)] then
2657                                        n = n + 1
2658                                        if n > l then
2659                                            break
2660                                        end
2661                                        pre = getnext(pre)
2662                                    else
2663                                        notmatchpre[last] = true
2664                                        break
2665                                    end
2666                                end
2667                                if n <= l then
2668                                    notmatchpre[last] = true
2669                                end
2670                            else
2671                                notmatchpre[last] = true
2672                            end
2673                            if replace then
2674                                -- so far we never entered this branch
2675                                while replace do
2676                                    if seq[n][getchar(replace)] then
2677                                        n = n + 1
2678                                        if n > l then
2679                                            break
2680                                        end
2681                                        replace = getnext(replace)
2682                                    else
2683                                        notmatchreplace[last] = true
2684                                        if notmatchpre[last] then
2685                                            goto next
2686                                        else
2687                                            break
2688                                        end
2689                                    end
2690                                end
2691                                -- why here again
2692                                if notmatchpre[last] then
2693                                    goto next
2694                                end
2695                            end
2696                            -- maybe only if match
2697                            last = nxt
2698                        else
2699                            goto next
2700                        end
2701                    else
2702                        goto next
2703                    end
2704                end
2705            end
2706
2707            -- before
2708
2709            if f > 1 then
2710              -- if startprev then -- new 2022-09-25
2711                    local prev = startprev
2712                    if prereplace and prev == checkdisc then
2713                        prev = getprev(sweepnode)
2714                    end
2715                    if prev then
2716                        local discfound -- = nil
2717                        local n = f - 1
2718                        while n >= 1 do
2719                            if prev then
2720                                local prv, char, id = isprevchar(prev,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
2721                                if char then
2722                                   if seq[n][char] then
2723                                        if n > 1 then
2724                                            prev = prv
2725                                        end
2726                                        n = n - 1
2727                                    elseif skiphash and skiphash[char] then
2728                                        skipped = true
2729                                        if trace_skips then
2730                                            show_skip(dataset,sequence,char,ck,classes[char])
2731                                        end
2732                                        prev = prv
2733                                    elseif discfound then
2734                                        notmatchreplace[discfound] = true
2735                                        if notmatchpost[discfound] then
2736                                            goto next
2737                                        else
2738                                            break
2739                                        end
2740                                    else
2741                                        goto next
2742                                    end
2743                                elseif char == false then
2744                                    if discfound then
2745                                        notmatchreplace[discfound] = true
2746                                        if notmatchpost[discfound] then
2747                                            goto next
2748                                        end
2749                                    else
2750                                        goto next
2751                                    end
2752                                    break
2753                                elseif id == disc_code then
2754                             -- elseif id == disc_code and (not discs or discs[prev]) then
2755                                    -- the special case: f i where i becomes dottless i ..
2756                                    discseen              = true
2757                                    discfound             = prev
2758                                    notmatchpre[prev]     = true
2759                                    notmatchpost[prev]    = nil
2760                                    notmatchreplace[prev] = nil
2761                                    local pre, post, replace, pretail, posttail, replacetail = getdisc(prev,true)
2762                                    -- weird test: needs checking
2763                                    if pre ~= start and post ~= start and replace ~= start then
2764                                        if post then
2765                                            local n = n
2766                                            while posttail do
2767                                                if seq[n][getchar(posttail)] then
2768                                                    n = n - 1
2769                                                    if posttail == post or n < 1 then
2770                                                        break
2771                                                    else
2772                                                        posttail = getprev(posttail)
2773                                                    end
2774                                                else
2775                                                    notmatchpost[prev] = true
2776                                                    break
2777                                                end
2778                                            end
2779                                            if n >= 1 then
2780                                                notmatchpost[prev] = true
2781                                            end
2782                                        else
2783                                            notmatchpost[prev] = true
2784                                        end
2785                                        if replace then
2786                                            -- we seldom enter this branch (e.g. on brill efficient)
2787                                            while replacetail do
2788                                                if seq[n][getchar(replacetail)] then
2789                                                    n = n - 1
2790                                                    if replacetail == replace or n < 1 then
2791                                                        break
2792                                                    else
2793                                                        replacetail = getprev(replacetail)
2794                                                    end
2795                                                else
2796                                                    notmatchreplace[prev] = true
2797                                                    if notmatchpost[prev] then
2798                                                        goto next
2799                                                    else
2800                                                        break
2801                                                    end
2802                                                end
2803                                            end
2804                                        else
2805                                         -- notmatchreplace[prev] = true -- not according to Kai
2806                                        end
2807                                    end
2808                                 -- prev = getprev(prev)
2809                                    prev = prv
2810                             -- elseif id == glue_code and seq[n][32] and isspace(prev,threshold,id) then
2811                             -- elseif seq[n][32] and spaces[prev] then
2812                             --     n = n - 1
2813                             --     prev = prv
2814                                elseif id == glue_code then
2815                                    local sn = seq[n]
2816                                    if (sn[32] and spaces[prev]) or sn[0xFFFC] then
2817                                        n = n - 1
2818                                        prev = prv
2819                                    else
2820                                        goto next
2821                                    end
2822                                elseif seq[n][0xFFFC] then
2823                                    n = n - 1
2824                                    prev = prv
2825                                else
2826                                    goto next
2827                                end
2828                            else
2829                                goto next
2830                            end
2831                        end
2832                    else
2833                        goto next
2834                    end
2835             -- else          -- new 2022-09-25
2836             --     goto next -- new 2022-09-25
2837             -- end           -- new 2022-09-25
2838            end
2839
2840            -- after
2841
2842            if s > l then
2843                local current = last and getnext(last)
2844                if not current and postreplace then
2845                    current = getnext(sweepnode)
2846                end
2847                if current then
2848                    local discfound -- = nil
2849                    local n = l + 1
2850                    while n <= s do
2851                        if current then
2852                            local nxt, char, id = isnextchar(current,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
2853                            if char then
2854                                if seq[n][char] then
2855                                    if n < s then
2856                                        current = nxt
2857                                    end
2858                                    n = n + 1
2859                                elseif skiphash and skiphash[char] then
2860                                    skipped = true
2861                                    if trace_skips then
2862                                        show_skip(dataset,sequence,char,ck,classes[char])
2863                                    end
2864                                    current = nxt
2865                                elseif discfound then
2866                                    notmatchreplace[discfound] = true
2867                                    if notmatchpre[discfound] then
2868                                        goto next
2869                                    else
2870                                        break
2871                                    end
2872                                else
2873                                    goto next
2874                                end
2875                            elseif char == false then
2876                                if discfound then
2877                                    notmatchreplace[discfound] = true
2878                                    if notmatchpre[discfound] then
2879                                        goto next
2880                                    else
2881                                        break
2882                                    end
2883                                else
2884                                    goto next
2885                                end
2886                            elseif id == disc_code then
2887                         -- elseif id == disc_code and (not discs or discs[current]) then
2888                                discseen                 = true
2889                                discfound                = current
2890                                notmatchpre[current]     = nil
2891                                notmatchpost[current]    = true
2892                                notmatchreplace[current] = nil
2893                                local pre, post, replace = getdisc(current)
2894                                -- no detailed checking here, todo ?
2895                                if pre then
2896                                    local n = n
2897                                    while pre do
2898                                        if seq[n][getchar(pre)] then
2899                                            n = n + 1
2900                                            if n > s then
2901                                                break
2902                                            else
2903                                                pre = getnext(pre)
2904                                            end
2905                                        else
2906                                            notmatchpre[current] = true
2907                                            break
2908                                        end
2909                                    end
2910                                    if n <= s then
2911                                        notmatchpre[current] = true
2912                                    end
2913                                else
2914                                    notmatchpre[current] = true
2915                                end
2916                                if replace then
2917                                    -- so far we never entered this branch
2918                                    while replace do
2919                                        if seq[n][getchar(replace)] then
2920                                            n = n + 1
2921                                            if n > s then
2922                                                break
2923                                            else
2924                                                replace = getnext(replace)
2925                                            end
2926                                        else
2927                                            notmatchreplace[current] = true
2928                                            if notmatchpre[current] then
2929                                                goto next
2930                                            else
2931                                                break
2932                                            end
2933                                        end
2934                                    end
2935                                else
2936                                 -- notmatchreplace[current] = true -- not according to Kai
2937                                end
2938                                current = getnext(current)
2939                            elseif id == glue_code then
2940                                local sn = seq[n]
2941                                if (sn[32] and spaces[current]) or sn[0xFFFC] then
2942                                    n = n + 1
2943                                    current = nxt
2944                                else
2945                                    goto next
2946                                end
2947                            elseif seq[n][0xFFFC] then
2948                                n = n + 1
2949                                current = nxt
2950                            else
2951                                goto next
2952                            end
2953                        else
2954                            goto next
2955                        end
2956                    end
2957                else
2958                    goto next
2959                end
2960            end
2961        end
2962        if trace_contexts then
2963            chaintrac(head,start,dataset,sequence,rlmode,skipped and skiphash,ck,true,discseen,sweepnode)
2964        end
2965        if discseen or sweepnode then
2966            -- When we process a disc we can collapse and therefore we backtrack one node (start) and
2967            -- reprocess. This is needed because there might be more in the collapsed list.
2968            head, start, done = chaindisk(head,start,dataset,sequence,rlmode,skipped and skiphash,ck)
2969        else
2970            head, start, done = chainrun(head,start,last,dataset,sequence,rlmode,skipped and skiphash,ck,6)
2971        end
2972        if trace_contexts then
2973            chaintrac(start,done)
2974        end
2975        if done then
2976            break
2977     -- else
2978            -- next context
2979        end
2980      ::next::
2981    end
2982    if discseen then
2983        notmatchpre     = { }
2984        notmatchpost    = { }
2985        notmatchreplace = { }
2986     -- notmatchpre     = { a = 1, b = 1 }  notmatchpre    .a = nil notmatchpre    .b = nil
2987     -- notmatchpost    = { a = 1, b = 1 }  notmatchpost   .a = nil notmatchpost   .b = nil
2988     -- notmatchreplace = { a = 1, b = 1 }  notmatchreplace.a = nil notmatchreplace.b = nil
2989    end
2990    return head, start, done
2991end
2992
2993handlers.gsub_context             = handle_contextchain
2994handlers.gsub_contextchain        = handle_contextchain
2995handlers.gsub_reversecontextchain = handle_contextchain
2996handlers.gpos_contextchain        = handle_contextchain
2997handlers.gpos_context             = handle_contextchain
2998
2999-- local function chained_contextchain(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash)
3000--     local steps    = currentlookup.steps
3001--     local nofsteps = currentlookup.nofsteps
3002--     if nofsteps > 1 then
3003--         reportmoresteps(dataset,sequence)
3004--     end
3005--     -- probably wrong
3006--     local l = steps[1].coverage[getchar(start)]
3007--     if l then
3008--         return handle_contextchain(head,start,dataset,sequence,l,rlmode,skiphash)
3009--     else
3010--         return head, start, false
3011--     end
3012-- end
3013
3014-- new 2022-09-25
3015
3016local function chained_contextchain(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash)
3017    local steps    = currentlookup.steps
3018    local nofsteps = currentlookup.nofsteps
3019    local char     = getchar(start)
3020    if nofsteps == 1 then
3021        local s = steps[1]
3022        local l = s.coverage[char]
3023        if l then
3024            return handle_contextchain(head,start,dataset,sequence,l,rlmode,skiphash)
3025        end
3026    else
3027        for i=1,nofsteps do
3028            local s = steps[i]
3029            local l = s.coverage[char]
3030            if l then
3031                local h, s, d = handle_contextchain(head,start,dataset,sequence,l,rlmode,skiphash)
3032                if d then
3033                    return h, s, d
3034                end
3035            end
3036        end
3037    end
3038    return head, start, false
3039end
3040
3041chainprocs.gsub_context             = chained_contextchain
3042chainprocs.gsub_contextchain        = chained_contextchain
3043chainprocs.gsub_reversecontextchain = chained_contextchain
3044chainprocs.gpos_contextchain        = chained_contextchain
3045chainprocs.gpos_context             = chained_contextchain
3046
3047------------------------------
3048
3049-- experiment (needs no handler in font-otc so not now):
3050--
3051-- function otf.registerchainproc(name,f)
3052--  -- chainprocs[name] = f
3053--     chainprocs[name] = function(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash)
3054--         local done = currentlookup.nofsteps > 0
3055--         if not done then
3056--             reportzerosteps(dataset,sequence)
3057--         else
3058--             head, start, done = f(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash)
3059--             if not head or not start then
3060--                 reportbadsteps(dataset,sequence)
3061--             end
3062--         end
3063--         return head, start, done
3064--     end
3065-- end
3066
3067local missing    = setmetatableindex("table")
3068local logwarning = report_process
3069local resolved   = { } -- we only resolve a font,script,language pair once
3070
3071local function logprocess(...)
3072    if trace_steps then
3073        registermessage(...)
3074        if trace_steps == "silent" then
3075            return
3076        end
3077    end
3078    report_process(...)
3079end
3080
3081-- todo: pass all these 'locals' in a table
3082
3083local sequencelists = setmetatableindex(function(t,font)
3084    local sequences = fontdata[font].resources.sequences
3085    if not sequences or not next(sequences) then
3086        sequences = false
3087    end
3088    t[font] = sequences
3089    return sequences
3090end)
3091
3092-- fonts.hashes.sequences = sequencelists
3093
3094do -- overcome local limit
3095
3096    local autofeatures    = fonts.analyzers.features
3097    local featuretypes    = otf.tables.featuretypes
3098    local defaultscript   = otf.features.checkeddefaultscript
3099    local defaultlanguage = otf.features.checkeddefaultlanguage
3100
3101    local wildcard        = "*"
3102    local default         = "dflt"
3103
3104    local function initialize(sequence,script,language,enabled,autoscript,autolanguage)
3105        local features = sequence.features
3106        if features then
3107            local order = sequence.order
3108            if order then
3109                local featuretype = featuretypes[sequence.type or "unknown"]
3110                for i=1,#order do
3111                    local kind  = order[i]
3112                    local valid = enabled[kind]
3113                    if valid then
3114                        local scripts   = features[kind]
3115                        local languages = scripts and (
3116                            scripts[script] or
3117                            scripts[wildcard] or
3118                            (autoscript and defaultscript(featuretype,autoscript,scripts))
3119                        )
3120                        local enabled = languages and (
3121                            languages[language] or
3122                            languages[wildcard] or
3123                            (autolanguage and defaultlanguage(featuretype,autolanguage,languages))
3124                        )
3125                        if enabled then
3126                            return { valid, autofeatures[kind] or false, sequence, kind }
3127                        end
3128                    end
3129                end
3130            else
3131                -- can't happen
3132            end
3133        end
3134        return false
3135    end
3136
3137    function otf.dataset(tfmdata,font) -- generic variant, overloaded in context
3138        local shared       = tfmdata.shared
3139        local properties   = tfmdata.properties
3140        local language     = properties.language or "dflt"
3141        local script       = properties.script   or "dflt"
3142        local enabled      = shared.features
3143        local autoscript   = enabled and enabled.autoscript
3144        local autolanguage = enabled and enabled.autolanguage
3145        local res = resolved[font]
3146        if not res then
3147            res = { }
3148            resolved[font] = res
3149        end
3150        local rs = res[script]
3151        if not rs then
3152            rs = { }
3153            res[script] = rs
3154        end
3155        local rl = rs[language]
3156        if not rl then
3157            rl = {
3158                -- indexed but we can also add specific data by key
3159            }
3160            rs[language] = rl
3161            local sequences = tfmdata.resources.sequences
3162            if sequences then
3163                for s=1,#sequences do
3164                    local v = enabled and initialize(sequences[s],script,language,enabled,autoscript,autolanguage)
3165                    if v then
3166                        rl[#rl+1] = v
3167                    end
3168                end
3169            end
3170        end
3171        return rl
3172    end
3173
3174end
3175
3176-- Functions like kernrun, comprun etc evolved over time and in the end look rather
3177-- complex. It's a bit of a compromise between extensive copying and creating subruns.
3178-- The logic has been improved a lot by Kai and Ivo who use complex fonts which
3179-- really helped to identify border cases on the one hand and get insight in the diverse
3180-- ways fonts implement features (not always that consistent and efficient). At the same
3181-- time I tried to keep the code relatively efficient so that the overhead in runtime
3182-- stays acceptable.
3183
3184local function report_disc(what,n)
3185    report_run("%s: %s > %s",what,n,languages.serializediscretionary(n))
3186end
3187
3188local function kernrun(disc,k_run,...)
3189    --
3190    -- we catch <font 1><disc font 2>
3191    --
3192    if trace_kernruns then
3193        report_disc("kern",disc)
3194    end
3195    --
3196    local prev, next = getboth(disc)
3197    --
3198    local nextstart = next
3199    local done      = false
3200    --
3201    local pre, post, replace, pretail, posttail, replacetail = getdisc(disc,true)
3202    --
3203    local prevmarks = prev
3204    --
3205    -- can be optional, because why on earth do we get a disc after a mark (okay, maybe when a ccmp
3206    -- has happened but then it should be in the disc so basically this test indicates an error)
3207    --
3208    while prevmarks do
3209        local prv, char = isprevchar(prevmarks,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
3210        if char and marks[char] then
3211            prevmarks = prv
3212        else
3213            break
3214        end
3215    end
3216    --
3217    if prev and not ischar(prev,currentfont) then  -- and (pre or replace) -- weak test
3218        prev = false
3219    end
3220    if next and not ischar(next,currentfont) then  -- and (post or replace) -- weak test
3221        next = false
3222    end
3223    --
3224    -- we need to get rid of this nest mess some day .. has to be done otherwise
3225    --
3226    if pre then
3227        if k_run(pre,"injections",nil,...) then
3228            done = true
3229        end
3230        if prev then
3231            setlink(prev,pre)
3232            if k_run(prevmarks,"preinjections",pre,...) then -- or prev?
3233                done = true
3234            end
3235            setprev(pre)
3236            setlink(prev,disc)
3237        end
3238    end
3239    --
3240    if post then
3241        if k_run(post,"injections",nil,...) then
3242            done = true
3243        end
3244        if next then
3245            setlink(posttail,next)
3246            if k_run(posttail,"postinjections",next,...) then
3247                done = true
3248            end
3249            setnext(posttail)
3250            setlink(disc,next)
3251        end
3252    end
3253    --
3254    if replace then
3255        if k_run(replace,"injections",nil,...) then
3256            done = true
3257        end
3258        if prev then
3259            setlink(prev,replace)
3260            if k_run(prevmarks,"replaceinjections",replace,...) then -- getnext(replace))
3261                done = true
3262            end
3263            setprev(replace)
3264            setlink(prev,disc)
3265        end
3266        if next then
3267            setlink(replacetail,next)
3268            if k_run(replacetail,"replaceinjections",next,...) then
3269                done = true
3270            end
3271            setnext(replacetail)
3272            setlink(disc,next)
3273        end
3274    elseif prev and next then
3275        setlink(prev,next)
3276        if k_run(prevmarks,"emptyinjections",next,...) then
3277            done = true
3278        end
3279        setlink(prev,disc,next)
3280    end
3281    if done and trace_testruns then
3282        report_disc("done",disc)
3283    end
3284 -- return nextstart, done
3285    return nextstart
3286end
3287
3288-- fonts like ebgaramond do ligatures this way (less efficient than e.g. dejavu which
3289-- will do the testrun variant)
3290
3291local function comprun(disc,c_run,...) -- vararg faster than the whole list
3292    if trace_compruns then
3293        report_disc("comp",disc)
3294    end
3295    --
3296    local pre, post, replace = getdisc(disc)
3297    local renewed = false
3298    --
3299    if pre then
3300        sweepnode = disc
3301        sweeptype = "pre" -- in alternative code preinjections is used (also used then for properties, saves a variable)
3302        local new, done = c_run(pre,...)
3303        if done then
3304            pre     = new
3305            renewed = true
3306        end
3307    end
3308    --
3309    if post then
3310        sweepnode = disc
3311        sweeptype = "post"
3312        local new, done = c_run(post,...)
3313        if done then
3314            post    = new
3315            renewed = true
3316        end
3317    end
3318    --
3319    if replace then
3320        sweepnode = disc
3321        sweeptype = "replace"
3322        local new, done = c_run(replace,...)
3323        if done then
3324            replace = new
3325            renewed = true
3326        end
3327    end
3328    --
3329    sweepnode = nil
3330    sweeptype = nil
3331    if renewed then
3332        if trace_testruns then
3333            report_disc("done",disc)
3334        end
3335        setdisc(disc,pre,post,replace)
3336    end
3337    --
3338 -- return getnext(disc), renewed
3339    return getnext(disc)
3340end
3341
3342-- if we can hyphenate in a lig then unlikely a lig so we
3343-- could have a option here to ignore lig
3344
3345local test_flatten_start = 2 -- must start at 2 according to Kai
3346
3347directives.register("otf.testrun.forceflatten", function(v)
3348    test_flatten_start = v and 1 or 2
3349end)
3350
3351local function testrun(disc,t_run,c_run,...)
3352    if trace_testruns then
3353        report_disc("test",disc)
3354    end
3355    local prev, next = getboth(disc)
3356    if not next then
3357        -- weird discretionary
3358     -- return
3359        -- better, removing it is tricky
3360        next = newboundary()
3361        setlink(disc,next)
3362    end
3363    local pre, post, replace, pretail, posttail, replacetail = getdisc(disc,true)
3364    local renewed = false
3365    if post or replace then -- and prev then -- hm, we can start with a disc
3366        if post then
3367            setlink(posttail,next)
3368        else
3369            post = next
3370        end
3371        if replace then
3372            setlink(replacetail,next)
3373        else
3374            replace = next
3375        end
3376        local d_post    = t_run(post,next,...)
3377        local d_replace = t_run(replace,next,...)
3378        if d_post > 0 or d_replace > 0 then
3379            local d = d_replace > d_post and d_replace or d_post
3380            local head = getnext(disc) -- is: next
3381            local tail = head
3382            for i=test_flatten_start,d do
3383                local nx = getnext(tail)
3384                local id = getid(nx)
3385                if id == disc_code then
3386                    head, tail = flattendisk(head,nx)
3387                elseif id == glyph_code then
3388                    tail = nx
3389                else
3390                    -- we can have overrun into a glue
3391                    break
3392                end
3393            end
3394            next = getnext(tail)
3395            setnext(tail)
3396            setprev(head)
3397            local new  = copynodelist(head)
3398            if posttail then
3399                setlink(posttail,head)
3400            else
3401                post = head
3402            end
3403            if replacetail then
3404                setlink(replacetail,new)
3405            else
3406                replace = new
3407            end
3408        else
3409            -- we stay inside the disc
3410            if posttail then
3411                setnext(posttail)
3412            else
3413                post = nil
3414            end
3415            if replacetail then
3416                setnext(replacetail)
3417            else
3418                replace = nil
3419            end
3420        end
3421        setlink(disc,next)
3422     -- pre, post, replace, pretail, posttail, replacetail = getdisc(disc,true)
3423    end
3424    --
3425    -- like comprun
3426    --
3427    if trace_testruns then
3428        report_disc("more",disc)
3429    end
3430    --
3431    if pre then
3432        sweepnode = disc
3433        sweeptype = "pre"
3434        local new, ok = c_run(pre,...)
3435        if ok then
3436            pre     = new
3437            renewed = true
3438        end
3439    end
3440    --
3441    if post then
3442        sweepnode = disc
3443        sweeptype = "post"
3444        local new, ok = c_run(post,...)
3445        if ok then
3446            post    = new
3447            renewed = true
3448        end
3449    end
3450    --
3451    if replace then
3452        sweepnode = disc
3453        sweeptype = "replace"
3454        local new, ok = c_run(replace,...)
3455        if ok then
3456            replace = new
3457            renewed = true
3458        end
3459    end
3460    --
3461    sweepnode = nil
3462    sweeptype = nil
3463    if renewed then
3464        setdisc(disc,pre,post,replace)
3465        if trace_testruns then
3466            report_disc("done",disc)
3467        end
3468    end
3469    -- next can have changed (copied list)
3470 -- return getnext(disc), renewed
3471    return getnext(disc)
3472end
3473
3474--  1{2{\oldstyle\discretionary{3}{4}{5}}6}7\par
3475--  1{2\discretionary{3{\oldstyle3}}{{\oldstyle4}4}{5{\oldstyle5}5}6}7\par
3476
3477local nesting = 0
3478
3479local function c_run_single(head,lookupcache,step,dataset,sequence,rlmode,skiphash,handler)
3480    local done  = false
3481    local sweep = sweephead[head]
3482    local start
3483    if sweep then
3484        start = sweep
3485     -- sweephead[head] = nil
3486        sweephead[head] = false
3487    else
3488        start = head
3489    end
3490    while start do
3491        local nxt, char, id = isnextchar(start,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
3492        if char then
3493            local lookupmatch = lookupcache[char]
3494            if lookupmatch then
3495                local ok
3496                head, start, ok = handler(head,start,dataset,sequence,lookupmatch,rlmode,skiphash,step)
3497                if ok then
3498                    done = true
3499                end
3500                if start then
3501                    start = getnext(start)
3502                end
3503            else
3504                start = nxt
3505            end
3506        elseif char == false then
3507            return head, done
3508        elseif sweep then
3509            -- else we loose the rest
3510            return head, done
3511        else
3512            -- in disc component ... really?
3513            start = nxt
3514        end
3515    end
3516    return head, done
3517end
3518
3519local function t_run_single(start,stop,lookupcache)
3520    local lastd = nil
3521    while start ~= stop do
3522        local nxt, char = isnextchar(start,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
3523        if char then
3524            local lookupmatch = lookupcache[char]
3525            if lookupmatch then -- hm, hyphens can match (tlig) so we need to really check
3526                -- if we need more than ligatures we can outline the code and use functions
3527                local s     = nxt
3528                local ss    = nil
3529                local sstop = s == stop
3530                if not s then
3531                    s = ss
3532                    ss = nil
3533                end
3534                -- a bit weird: why multiple ... anyway we can't have a disc in a disc
3535                -- how about post ... we can probably merge this into the while
3536                while getid(s) == disc_code do
3537                    ss = getnext(s)
3538                    s  = getreplace(s)
3539                    if not s then
3540                        s = ss
3541                        ss = nil
3542                    end
3543                end
3544                local l = nil
3545                local d = 0
3546                while s do
3547                    local nxt, char = isnextchar(s,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
3548                    if char then
3549                        local lg = not tonumber(lookupmatch) and lookupmatch[char]
3550                        if lg then
3551                            if sstop then
3552                                d = 1
3553                            elseif d > 0 then
3554                                d = d + 1
3555                            end
3556                            l = lg
3557                            s = nxt
3558                            sstop = s == stop
3559                            if not s then
3560                                s  = ss
3561                                ss = nil
3562                            end
3563                            while getid(s) == disc_code do
3564                                ss = getnext(s)
3565                                s  = getreplace(s)
3566                                if not s then
3567                                    s  = ss
3568                                    ss = nil
3569                                end
3570                            end
3571                            lookupmatch = lg
3572                        else
3573                            break
3574                        end
3575                    else
3576                        break
3577                    end
3578                end
3579                if l and (tonumber(l) or l.ligature) then -- so we test for ligature
3580                    lastd = d
3581                end
3582            end
3583            if lastd then
3584                return lastd
3585            else
3586                start = nxt
3587            end
3588        else
3589            break
3590        end
3591    end
3592    return 0
3593end
3594
3595local function k_run_single(sub,injection,last,lookupcache,step,dataset,sequence,rlmode,skiphash,handler)
3596    for n in nextnode, sub do -- only gpos
3597        if n == last then
3598            break
3599        else
3600            local nxt, char = isnextchar(n,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
3601            if char then
3602                local lookupmatch = lookupcache[char]
3603                if lookupmatch then
3604                    local h, d, ok = handler(sub,n,dataset,sequence,lookupmatch,rlmode,skiphash,step,injection)
3605                    if ok then
3606                        return true
3607                    end
3608                end
3609            end
3610        end
3611    end
3612end
3613
3614local function c_run_multiple(head,steps,nofsteps,dataset,sequence,rlmode,skiphash,handler)
3615    local done  = false
3616    local sweep = sweephead[head]
3617    local start
3618    if sweep then
3619        start = sweep
3620     -- sweephead[head] = nil
3621        sweephead[head] = false
3622    else
3623        start = head
3624    end
3625    while start do
3626        local nxt, char = isnextchar(start,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
3627        if char then
3628            for i=1,nofsteps do
3629                local step        = steps[i]
3630                local lookupcache = step.coverage
3631                local lookupmatch = lookupcache[char]
3632                if lookupmatch then
3633                    -- we could move all code inline but that makes things even more unreadable
3634                    local ok
3635                    head, start, ok = handler(head,start,dataset,sequence,lookupmatch,rlmode,skiphash,step)
3636                    if ok then
3637                        done = true
3638                        break
3639                    elseif not start then
3640                        -- don't ask why ... shouldn't happen
3641                        break
3642                    end
3643                end
3644            end
3645            if start then
3646                start = getnext(start)
3647            end
3648        elseif char == false then
3649            -- whatever glyph
3650            return head, done
3651        elseif sweep then
3652            -- else we loose the rest
3653            return head, done
3654        else
3655            -- in disc component
3656            start = nxt
3657        end
3658    end
3659    return head, done
3660end
3661
3662local function t_run_multiple(start,stop,steps,nofsteps)
3663    local lastd = nil
3664    while start ~= stop do
3665        local startnext, char = isnextchar(start,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
3666        if char then
3667            for i=1,nofsteps do
3668                local step = steps[i]
3669                local lookupcache = step.coverage
3670                local lookupmatch = lookupcache[char]
3671                if lookupmatch then
3672                    -- if we need more than ligatures we can outline the code and use functions
3673                    local s     = startnext
3674                    local ss    = nil
3675                    local sstop = s == stop
3676                    if not s then
3677                        s  = ss
3678                        ss = nil
3679                    end
3680                    while getid(s) == disc_code do
3681                        ss = getnext(s)
3682                        s  = getreplace(s)
3683                        if not s then
3684                            s  = ss
3685                            ss = nil
3686                        end
3687                    end
3688                    local l = nil
3689                    local d = 0
3690                    while s do
3691                        local nxt, char = isnextchar(s,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
3692                        if char then
3693                            local lg = not tonumber(lookupmatch) and lookupmatch[char]
3694                            if lg then
3695                                if sstop then
3696                                    d = 1
3697                                elseif d > 0 then
3698                                    d = d + 1
3699                                end
3700                                l = lg
3701                                s = nxt
3702                                sstop = s == stop
3703                                if not s then
3704                                    s  = ss
3705                                    ss = nil
3706                                end
3707                                while getid(s) == disc_code do
3708                                    ss = getnext(s)
3709                                    s  = getreplace(s)
3710                                    if not s then
3711                                        s  = ss
3712                                        ss = nil
3713                                    end
3714                                end
3715                                lookupmatch = lg
3716                            else
3717                                break
3718                            end
3719                        else
3720                            break
3721                        end
3722                    end
3723                    if l and (tonumber(l) or l.ligature) then
3724                        lastd = d
3725                    end
3726                end
3727            end
3728            if lastd then
3729                return lastd
3730            else
3731                start = startnext
3732            end
3733        else
3734            break
3735        end
3736    end
3737    return 0
3738end
3739
3740local function k_run_multiple(sub,injection,last,steps,nofsteps,dataset,sequence,rlmode,skiphash,handler)
3741    for n in nextnode, sub do -- only gpos
3742        if n == last then
3743            break
3744        else
3745            local nxt, char = isnextchar(n,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
3746            if char then
3747                for i=1,nofsteps do
3748                    local step        = steps[i]
3749                    local lookupcache = step.coverage
3750                    local lookupmatch = lookupcache[char]
3751                    if lookupmatch then
3752                        local h, d, ok = handler(sub,n,dataset,sequence,lookupmatch,rlmode,skiphash,step,injection) -- sub was head
3753                        if ok then
3754                            return true
3755                        end
3756                    end
3757                end
3758            end
3759        end
3760    end
3761end
3762
3763local txtdirstate, pardirstate  do -- this might change (no need for nxt in pardirstate)
3764
3765    local getdirection = nuts.getdirection
3766
3767    txtdirstate = function(start,stack,top,rlparmode)
3768        local dir, pop = getdirection(start)
3769        if pop then
3770            if top == 1 then
3771                return 0, rlparmode
3772            else
3773                top = top - 1
3774                if stack[top] == righttoleft_code then
3775                    return top, -1
3776                else
3777                    return top, 1
3778                end
3779            end
3780        elseif dir == lefttoright_code then
3781            top = top + 1
3782            stack[top] = lefttoright_code
3783            return top, 1
3784        elseif dir == righttoleft_code then
3785            top = top + 1
3786            stack[top] = righttoleft_code
3787            return top, -1
3788        else
3789            return top, rlparmode
3790        end
3791    end
3792
3793    pardirstate = function(start)
3794        local dir = getdirection(start)
3795        if dir == lefttoright_code then
3796            return 1, 1
3797        elseif dir == righttoleft_code then
3798            return -1, -1
3799        else
3800            return 0, 0
3801        end
3802    end
3803
3804end
3805
3806-- These are non public helpers that can change without notice!
3807
3808otf.helpers             = otf.helpers or { }
3809otf.helpers.txtdirstate = txtdirstate
3810otf.helpers.pardirstate = pardirstate
3811
3812-- This is the main loop. We run over the node list dealing with a specific font. The
3813-- dynamic is a context specific thing. We could work on sub start-stop ranges instead
3814-- but I wonder if there is that much speed gain (experiments showed that it made not
3815-- much sense) and we need to keep track of directions anyway. Also at some point I
3816-- want to play with font interactions and then we do need the full sweeps. Apart from
3817-- optimizations the principles of processing the features hasn't changed much since
3818-- the beginning.
3819
3820do
3821
3822    -- This is a measurable experimental speedup (only with hyphenated text and multiple
3823    -- fonts per processor call), especially for fonts with lots of contextual lookups.
3824
3825    local fastdisc = true
3826    local testdics = false
3827
3828    directives.register("otf.fastdisc",function(v) fastdisc = v end) -- normally enabled
3829
3830    -- using a merged combined hash as first test saves some 30% on ebgaramond and
3831    -- about 15% on arabtype .. then moving the a test also saves a bit (even when
3832    -- often a is not set at all so that one is a bit debatable
3833
3834 -- local otfdataset = nil
3835    local otfdataset
3836
3837    otfdataset = function(...)
3838        -- delayed alias
3839        otfdataset = otf.dataset
3840        return otfdataset(...)
3841    end
3842
3843    local getfastdisc = { __index = function(t,k)
3844        local v = usesfont(k,currentfont)
3845        t[k] = v
3846        return v
3847    end }
3848
3849    local getfastspace = { __index = function(t,k)
3850        -- we don't pass the id so that one can overload isspace
3851        local v = isspace(k,threshold) or false
3852        t[k] = v
3853        return v
3854    end }
3855
3856    function otf.featuresprocessor(head,font,dynamic,direction,n)
3857
3858        local sequences = sequencelists[font] -- temp hack
3859
3860        nesting = nesting + 1
3861
3862        if nesting == 1 then
3863            currentfont    = font
3864            currentdynamic = dynamic
3865            currentscale   = 1000
3866            currentxscale  = 1000
3867            currentyscale  = 1000
3868            tfmdata        = fontdata[font]
3869            descriptions   = tfmdata.descriptions -- only needed in gref so we could pass node there instead
3870            characters     = tfmdata.characters   -- but this branch is not entered that often anyway
3871      local resources      = tfmdata.resources
3872            marks          = resources.marks
3873            classes        = resources.classes
3874            threshold,
3875            factor         = getthreshold(font)
3876            checkmarks     = tfmdata.properties.checkmarks
3877
3878         -- if not otfdataset then
3879         --     otfdataset = otf.dataset
3880         -- end
3881
3882            discs  = fastdisc and n and n > 1 and setmetatable({},getfastdisc)
3883            spaces = setmetatable({},getfastspace)
3884
3885        elseif currentfont ~= font then
3886
3887            report_warning("nested call with a different font, level %s, quitting",nesting)
3888            nesting = nesting - 1
3889            return head, false
3890
3891        end
3892
3893        if trace_steps then
3894            checkstep(head)
3895        end
3896
3897        local initialrl = 0
3898
3899        if getid(head) == par_code and startofpar(head) then
3900            initialrl = pardirstate(head)
3901        elseif direction == righttoleft_code then
3902            initialrl = -1
3903        end
3904
3905        local datasets  = otfdataset(tfmdata,font,dynamic)
3906        local dirstack  = { nil } -- could move outside function but we can have local runs
3907        sweephead       = { }
3908     -- sweephead  = { a = 1, b = 1 } sweephead.a = nil sweephead.b = nil
3909
3910        -- Keeping track of the headnode is needed for devanagari. (I generalized it a bit
3911        -- so that multiple cases are also covered.) We could prepend a temp node.
3912
3913        -- We don't goto the next node when a disc node is created so that we can then treat
3914        -- the pre, post and replace. It's a bit of a hack but works out ok for most cases.
3915
3916        for s=1,#datasets do -- todo: datasets.n
3917            local dataset      = datasets[s]
3918            local state        = dataset[2]
3919            local sequence     = dataset[3] -- sequences[s] -- also dataset[5]
3920            local rlparmode    = initialrl
3921            local topstack     = 0
3922            local typ          = sequence.type
3923            local gpossing     = typ == "gpos_single" or typ == "gpos_pair" -- store in dataset
3924            local forcetestrun = typ == "gsub_ligature" -- testrun is only for ligatures
3925            local handler      = handlers[typ] -- store in dataset
3926            local steps        = sequence.steps
3927            local nofsteps     = sequence.nofsteps
3928            local skiphash     = sequence.skiphash
3929            if not steps then
3930                -- This permits injection, watch the different arguments. Watch out, the arguments passed
3931                -- are not frozen as we might extend or change this. Is this used at all apart from some
3932                -- experiments?
3933                local h = handler(head,dataset,sequence,initialrl,font,dynamic) -- less arguments now
3934                if h and h ~= head then
3935                    head = h
3936                end
3937            elseif typ == "gsub_reversecontextchain" then
3938                --
3939                -- This might need a check: if we have #before or #after > 0 then we might need to reverse
3940                -- the before and after lists in the loader. But first I need to see a font that uses multiple
3941                -- matches.
3942                --
3943                local start  = findnodetail(head)
3944                local rlmode = 0 -- how important is this .. do we need to check for dir?
3945                local merged = steps.merged
3946                while start do
3947                    local prv, char, id = isprevchar(start,font,dynamic,state)
3948                    if char then
3949                        local m = merged[char]
3950                        if m then
3951                            currentscale, currentxscale, currentyscale = getscales(start)
3952                            for i=m[1],m[2] do
3953                                local step = steps[i]
3954                                local lookupcache = step.coverage
3955                                local lookupmatch = lookupcache[char]
3956                                if lookupmatch then
3957                                    local ok
3958                                    head, start, ok = handler(head,start,dataset,sequence,lookupmatch,rlmode,skiphash,step)
3959                                    if ok then
3960                                     -- done = true
3961                                        break
3962                                    end
3963                                end
3964                            end
3965                            if start then
3966                                start = getprev(start)
3967                            end
3968                        else
3969                            start = prv
3970                        end
3971                    else
3972                        start = prv
3973                    end
3974                end
3975            else
3976                local start  = head
3977                local rlmode = initialrl
3978                if nofsteps == 1 then -- happens often
3979                    local step = steps[1]
3980                    local lookupcache = step.coverage
3981                    while start do
3982                        local nxt, char, id = isnextchar(start,font,dynamic,state) -- we can move the data/state checks here
3983                        if char then
3984                            -- no skiphash check here
3985                            local lookupmatch = lookupcache[char]
3986                            if lookupmatch then
3987                                currentscale, currentxscale, currentyscale = getscales(start)
3988                                local ok, df
3989                                head, start, ok, df = handler(head,start,dataset,sequence,lookupmatch,rlmode,skiphash,step)
3990                                if df then
3991                                 -- print("restart 1",typ)
3992                                elseif start then
3993                                    start = getnext(start) -- can be a new start
3994                                end
3995                            else
3996                                start = nxt
3997                            end
3998                        elseif char == false or id == glue_code then
3999                            -- a different font|state or glue (happens often)
4000                            start = nxt
4001                        elseif id == disc_code then
4002                            if not discs or discs[start] == true then
4003                                if gpossing then
4004                                    start = kernrun(start,k_run_single,             lookupcache,step,dataset,sequence,rlmode,skiphash,handler)
4005                                elseif forcetestrun then
4006                                    start = testrun(start,t_run_single,c_run_single,lookupcache,step,dataset,sequence,rlmode,skiphash,handler)
4007                                else
4008                                    start = comprun(start,c_run_single,             lookupcache,step,dataset,sequence,rlmode,skiphash,handler)
4009                                end
4010                            else
4011                                start = nxt
4012                            end
4013                        elseif id == math_code then
4014                            start = getnext(endofmath(start))
4015                        elseif id == dir_code then
4016                            topstack, rlmode = txtdirstate(start,dirstack,topstack,rlparmode)
4017                            start = nxt
4018                     -- elseif id == par_code and startofpar(start) then
4019                     --     rlparmode, rlmode = pardirstate(start)
4020                     --     start = nxt
4021                        else
4022                            start = nxt
4023                        end
4024                    end
4025                else
4026                    local merged = steps.merged
4027                    while start do
4028                        local nxt, char, id = isnextchar(start,font,dynamic,state)
4029                        if char then
4030                            -- no skiphash check here
4031                            local m = merged[char]
4032                            if m then
4033                                currentscale, currentxscale, currentyscale = getscales(start)
4034                                local ok, df
4035                                for i=m[1],m[2] do
4036                                    local step = steps[i]
4037                                    local lookupcache = step.coverage
4038                                    local lookupmatch = lookupcache[char]
4039                                    if lookupmatch then
4040                                        -- we could move all code inline but that makes things even more unreadable
4041--                                                 local ok, df
4042                                        head, start, ok, df = handler(head,start,dataset,sequence,lookupmatch,rlmode,skiphash,step)
4043                                        if df then
4044                                            break
4045                                        elseif ok then
4046                                            break
4047                                        elseif not start then
4048                                            -- don't ask why ... shouldn't happen
4049                                            break
4050                                        end
4051                                    end
4052                                end
4053                                if df then
4054                                 -- print("restart 2",typ)
4055                                elseif start then
4056                                    start = getnext(start) -- can be a new next
4057                                end
4058                            else
4059                                start = nxt
4060                            end
4061                        elseif char == false or id == glue_code then
4062                            -- a different font|state or glue (happens often)
4063                            start = nxt
4064                        elseif id == disc_code then
4065                            if not discs or discs[start] == true then
4066                                if gpossing then
4067                                    start = kernrun(start,k_run_multiple,               steps,nofsteps,dataset,sequence,rlmode,skiphash,handler)
4068                                elseif forcetestrun then
4069                                    start = testrun(start,t_run_multiple,c_run_multiple,steps,nofsteps,dataset,sequence,rlmode,skiphash,handler)
4070                                else
4071                                    start = comprun(start,c_run_multiple,               steps,nofsteps,dataset,sequence,rlmode,skiphash,handler)
4072                                end
4073                            else
4074                                start = nxt
4075                            end
4076                        elseif id == math_code then
4077                            start = getnext(endofmath(start))
4078                        elseif id == dir_code then
4079                            topstack, rlmode = txtdirstate(start,dirstack,topstack,rlparmode)
4080                            start = nxt
4081                     -- elseif id == par_code and startofpar(start) then
4082                     --     rlparmode, rlmode = pardirstate(start)
4083                     --     start = nxt
4084                        else
4085                            start = nxt
4086                        end
4087                    end
4088                end
4089            end
4090
4091            if trace_steps then -- ?
4092                registerstep(head)
4093            end
4094
4095        end
4096
4097        nesting = nesting - 1
4098
4099        return head
4100    end
4101
4102    -- This is not an official helper and used for tracing experiments. It can be changed as I like
4103    -- at any moment. At some point it might be used in a module that can help font development.
4104
4105    function otf.datasetpositionprocessor(head,font,direction,dataset)
4106
4107        currentfont     = font
4108        currentdynamic  = false
4109        currentscale    = 1000
4110        currentxscale   = 1000
4111        currentyscale   = 1000
4112        tfmdata         = fontdata[font]
4113        descriptions    = tfmdata.descriptions -- only needed in gref so we could pass node there instead
4114        characters      = tfmdata.characters   -- but this branch is not entered that often anyway
4115  local resources       = tfmdata.resources
4116        marks           = resources.marks
4117        classes         = resources.classes
4118        threshold,
4119        factor          = getthreshold(font)
4120        checkmarks      = tfmdata.properties.checkmarks
4121
4122        if type(dataset) == "number" then
4123            dataset = otfdataset(tfmdata,font,0)[dataset]
4124        end
4125
4126        local sequence  = dataset[3] -- sequences[s] -- also dataset[5]
4127        local typ       = sequence.type
4128     -- local gpossing  = typ == "gpos_single" or typ == "gpos_pair" -- store in dataset
4129
4130     -- gpos_contextchain gpos_context
4131
4132     -- if not gpossing then
4133     --     return head, false
4134     -- end
4135
4136        local handler   = handlers[typ] -- store in dataset
4137        local steps     = sequence.steps
4138        local nofsteps  = sequence.nofsteps
4139
4140        local done      = false
4141        local dirstack  = { nil } -- could move outside function but we can have local runs (maybe a few more nils)
4142        local start     = head
4143        local initialrl = (direction == righttoleft_code) and -1 or 0
4144        local rlmode    = initialrl
4145        local rlparmode = initialrl
4146        local topstack  = 0
4147        local merged    = steps.merged
4148
4149     -- local matches   = false
4150        local position  = 0
4151
4152        while start do
4153            local nxt, char, id = isnextchar(start,currentfont,currentdynamic)
4154            if char then
4155                position = position + 1
4156                local m = merged[char]
4157                if m then
4158                    -- no skiphash check here
4159                    for i=m[1],m[2] do
4160                        local step = steps[i]
4161                        local lookupcache = step.coverage
4162                        local lookupmatch = lookupcache[char]
4163                        if lookupmatch then
4164                            currentscale, currentxscale, currentyscale = getscales(start)
4165                            local ok
4166                            head, start, ok = handler(head,start,dataset,sequence,lookupmatch,rlmode,skiphash,step)
4167                            if ok then
4168                             -- if matches then
4169                             --     matches[position] = i
4170                             -- else
4171                             --     matches = { [position] = i }
4172                             -- end
4173                                break
4174                            elseif not start then
4175                                break
4176                            end
4177                        end
4178                    end
4179                    if start then
4180                        start = getnext(start) -- start can be new
4181                    end
4182                else
4183                 -- start = getnext(start)
4184                    start = nxt
4185                end
4186            elseif char == false or id == glue_code then
4187                -- a different font|state or glue (happens often)
4188                start = nxt
4189            elseif id == math_code then
4190                start = getnext(endofmath(start))
4191            elseif id == dir_code then
4192                topstack, rlmode = txtdirstate(start,dirstack,topstack,rlparmode)
4193                start = nxt
4194         -- elseif id == par_code and startofpar(start) then
4195         --     rlparmode, rlmode = pardirstate(start)
4196         --     start = nxt
4197            else
4198                start = nxt
4199            end
4200        end
4201
4202        return head
4203    end
4204
4205    -- end of experiment
4206
4207end
4208
4209-- so far
4210
4211do
4212
4213    local plugins = { }
4214    otf.plugins   = plugins
4215
4216    local report  = logs.reporter("fonts")
4217    local warned  = false
4218    local okay    = { text = true }
4219
4220    function otf.registerplugin(name,f)
4221        if type(name) == "string" and type(f) == "function" then
4222            plugins[name] = { name, f }
4223            if okay[name] then
4224                -- no warning (e.g. the diagnostic text plugin)
4225            else
4226                report("plugin %a has been loaded, please be aware of possible side effects",name)
4227                if not warned then
4228                    if logs.pushtarget then
4229                        logs.pushtarget("log")
4230                    end
4231                    report("Plugins are not officially supported unless stated otherwise. This is because")
4232                    report("they bypass the regular font handling and therefore some features in ConTeXt")
4233                    report("(especially those related to fonts) might not work as expected or might not work")
4234                    report("at all. Some plugins are for testing and development only and might change")
4235                    report("whenever we feel the need for it.")
4236                    report()
4237                    if logs.poptarget then
4238                        logs.poptarget()
4239                    end
4240                    warned = true
4241                end
4242            end
4243        end
4244    end
4245
4246    function otf.plugininitializer(tfmdata,value)
4247        if type(value) == "string" then
4248            tfmdata.shared.plugin = plugins[value]
4249        end
4250    end
4251
4252    function otf.pluginprocessor(head,font,dynamic,direction) -- n
4253        local s = fontdata[font].shared
4254        local p = s and s.plugin
4255        if p then
4256            if trace_plugins then
4257                report_process("applying plugin %a",p[1])
4258            end
4259            return p[2](head,font,dynamic,direction)
4260        else
4261            return head, false
4262        end
4263    end
4264
4265end
4266
4267function otf.featuresinitializer(tfmdata,value)
4268    -- nothing done here any more
4269end
4270
4271registerotffeature {
4272    name         = "features",
4273    description  = "features",
4274    default      = true,
4275    initializers = {
4276        position = 1,
4277        node     = otf.featuresinitializer,
4278        plug     = otf.plugininitializer,
4279    },
4280    processors   = {
4281        node     = otf.featuresprocessor,
4282        plug     = otf.pluginprocessor,
4283    }
4284}
4285
4286-- Moved here (up) a bit. This doesn't really belong in generic so it will move to a
4287-- context module some day.
4288
4289local function markinitializer(tfmdata,value)
4290    local properties = tfmdata.properties
4291    properties.checkmarks = value
4292end
4293
4294registerotffeature {
4295    name         = "checkmarks",
4296    description  = "check mark widths",
4297    default      = true,
4298    initializers = {
4299        node     = markinitializer,
4300    },
4301}
4302
4303-- This can be used for extra handlers, but should be used with care! We implement one
4304-- here but some more can be found in the osd (script devanagary) file. Now watch out:
4305-- when a handler has steps, it is called as the other ones, but when we have no steps,
4306-- we use a different call:
4307--
4308--   function(head,dataset,sequence,initialrl,font,dynamic)
4309--       return head
4310--   end
4311--
4312-- Also see (!!).
4313
4314otf.handlers = handlers
4315