Source Browser ?

1if not modules then modules = { } end modules ['font-ots'] = { -- sequences
2    version   = 1.001,
3    optimize  = true,
4    comment   = "companion to font-ini.mkiv",
5    author    = "Hans Hagen, PRAGMA-ADE, Hasselt NL",
6    copyright = "PRAGMA ADE / ConTeXt Development Team",
7    license   = "see context related readme files",
8}
9
10-- I need to check the description at the microsoft site ... it has been improved so
11-- maybe there are some interesting details there. Most below is based on old and
12-- incomplete documentation and involved quite a bit of guesswork (checking with the
13-- abstract uniscribe of those days. But changing things is tricky!
14--
15-- This module is a bit more split up that I'd like but since we also want to test
16-- with plain TeX it has to be so. This module is part of ConTeXt and discussion
17-- about improvements and functionality mostly happens on the ConTeXt mailing list.
18--
19-- The specification of OpenType is (or at least decades ago was) kind of vague.
20-- Apart from a lack of a proper free specifications there's also the problem that
21-- Microsoft and Adobe may have their own interpretation of how and in what order to
22-- apply features. In general the Microsoft website has more detailed specifications
23-- and is a better reference. There is also some information in the FontForge help
24-- files. In the end we rely most on the Microsoft specification.
25--
26-- Because there is so much possible, fonts might contain bugs and/or be made to
27-- work with certain rederers. These may evolve over time which may have the side
28-- effect that suddenly fonts behave differently. We don't want to catch all font
29-- issues.
30--
31-- After a lot of experiments (mostly by Taco, me and Idris) the first
32-- implementation was already quite useful. When it did most of what we wanted, a
33-- more optimized version evolved. Of course all errors are mine and of course the
34-- code can be improved. There are quite some optimizations going on here and
35-- processing speed is currently quite acceptable and has been improved over time.
36-- Many complex scripts are not yet supported yet, but I will look into them as soon
37-- as ConTeXt users ask for it.
38--
39-- The specification leaves room for interpretation. In case of doubt the Microsoft
40-- implementation is the reference as it is the most complete one. As they deal with
41-- lots of scripts and fonts, Kai and Ivo did a lot of testing of the generic code
42-- and their suggestions help improve the code. I'm aware that not all border cases
43-- can be taken care of, unless we accept excessive runtime, and even then the
44-- interference with other mechanisms (like hyphenation) are not trivial.
45--
46-- Especially discretionary handling has been improved much by Kai Eigner who uses
47-- complex (latin) fonts. The current implementation is a compromis between his
48-- patches and my code and in the meantime performance is quite ok. We cannot check
49-- all border cases without compromising speed but so far we're okay. Given good
50-- test cases we can probably improve it here and there. Especially chain lookups
51-- are non trivial with discretionaries but things got much better over time thanks
52-- to Kai.
53--
54-- Glyphs are indexed not by unicode but in their own way. This is because there is
55-- no relationship with unicode at all, apart from the fact that a font might cover
56-- certain ranges of characters. One character can have multiple shapes. However, at
57-- the TeX end we use unicode so and all extra glyphs are mapped into a private
58-- space. This is needed because we need to access them and TeX has to include then
59-- in the output eventually.
60--
61-- The initial data table is rather close to the open type specification and also
62-- not that different from the one produced by Fontforge but we uses hashes instead.
63-- In ConTeXt that table is packed (similar tables are shared) and cached on disk so
64-- that successive runs can use the optimized table (after loading the table is
65-- unpacked).
66--
67-- This module is sparsely documented because it is has been a moving target. The
68-- table format of the reader changed a bit over time and we experiment a lot with
69-- different methods for supporting features. By now the structures are quite stable
70--
71-- Incrementing the version number will force a re-cache. We jump the number by one
72-- when there's a fix in the reader or processing code that can result in different
73-- results.
74--
75-- This code is also used outside ConTeXt but in ConTeXt it has to work with other
76-- mechanisms. Both put some constraints on the code here.
77--
78-- Remark: We assume that cursives don't cross discretionaries which is okay because
79-- it is only used in semitic scripts.
80--
81-- Remark: We assume that marks precede base characters.
82--
83-- Remark: When complex ligatures extend into discs nodes we can get side effects.
84-- Normally this doesn't happen; ff\d{l}{l}{l} in lm works but ff\d{f}{f}{f}.
85--
86-- Todo: check if we copy attributes to disc nodes if needed.
87--
88-- Todo: it would be nice if we could get rid of components. In other places we can
89-- use the unicode properties. We can just keep a lua table.
90--
91-- Remark: We do some disc juggling where we need to keep in mind that the pre, post
92-- and replace fields can have prev pointers to a nesting node ... I wonder if that
93-- is still needed.
94--
95-- Remark: This is not possible:
96--
97-- \discretionary {alpha-} {betagammadelta}
98--   {\discretionary {alphabeta-} {gammadelta}
99--      {\discretionary {alphabetagamma-} {delta}
100--         {alphabetagammadelta}}}
101--
102-- Remark: Something is messed up: we have two mark / ligature indices, one at the
103-- injection end and one here ... this is based on KE's patches but there is something
104-- fishy there as I'm pretty sure that for husayni we need some connection (as it's much
105-- more complex than an average font) but I need proper examples of all cases, not of
106-- only some.
107--
108-- Remark: I wonder if indexed would be faster than unicoded. It would be a major
109-- rewrite to have char being unicode + an index field in glyph nodes. Also more
110-- assignments have to be made in order to keep things in sync. So, it's a no-go.
111--
112-- Remark: We can provide a fast loop when there are no disc nodes (tests show a 1%
113-- gain). Smaller functions might perform better cache-wise. But ... memory becomes
114-- faster anyway, so ...
115--
116-- Remark: Some optimizations made sense for 5.2 but seem less important for 5.3 but
117-- anyway served their purpose.
118--
119-- Todo: just (0=l2r and 1=r2l) or maybe (r2l = true)
120
121-- Experiments with returning the data with the ischar are positive for lmtx but
122-- have a performance hit on mkiv because there we need to wrap ischardata (pending
123-- extensions to luatex which is unlikely to happen for such an experiment because
124-- we then can't remove it). Actually it might make generic slightly faster. Also,
125-- there are some corner cases where a data check comes before a char fetch and
126-- we're talking of millions of calls there. At some point I might make a version
127-- for lmtx that does it slightly different anyway.
128
129local type, next, tonumber = type, next, tonumber
130local random = math.random
131local formatters = string.formatters
132local insert = table.insert
133
134local registertracker      = trackers.register
135
136local logs                 = logs
137local trackers             = trackers
138local nodes                = nodes
139local attributes           = attributes
140local fonts                = fonts
141
142local otf                  = fonts.handlers.otf
143local tracers              = nodes.tracers
144
145local trace_singles        = false  registertracker("otf.singles",      function(v) trace_singles      = v end)
146local trace_multiples      = false  registertracker("otf.multiples",    function(v) trace_multiples    = v end)
147local trace_alternatives   = false  registertracker("otf.alternatives", function(v) trace_alternatives = v end)
148local trace_ligatures      = false  registertracker("otf.ligatures",    function(v) trace_ligatures    = v end)
149local trace_contexts       = false  registertracker("otf.contexts",     function(v) trace_contexts     = v end)
150local trace_marks          = false  registertracker("otf.marks",        function(v) trace_marks        = v end)
151local trace_kerns          = false  registertracker("otf.kerns",        function(v) trace_kerns        = v end)
152local trace_cursive        = false  registertracker("otf.cursive",      function(v) trace_cursive      = v end)
153local trace_preparing      = false  registertracker("otf.preparing",    function(v) trace_preparing    = v end)
154local trace_bugs           = false  registertracker("otf.bugs",         function(v) trace_bugs         = v end)
155local trace_details        = false  registertracker("otf.details",      function(v) trace_details      = v end)
156local trace_steps          = false  registertracker("otf.steps",        function(v) trace_steps        = v end)
157local trace_skips          = false  registertracker("otf.skips",        function(v) trace_skips        = v end)
158local trace_plugins        = false  registertracker("otf.plugins",      function(v) trace_plugins      = v end)
159local trace_chains         = false  registertracker("otf.chains",       function(v) trace_chains       = v end)
160
161local trace_kernruns       = false  registertracker("otf.kernruns",     function(v) trace_kernruns     = v end)
162----- trace_discruns       = false  registertracker("otf.discruns",     function(v) trace_discruns     = v end)
163local trace_compruns       = false  registertracker("otf.compruns",     function(v) trace_compruns     = v end)
164local trace_testruns       = false  registertracker("otf.testruns",     function(v) trace_testruns     = v end)
165
166local forcediscretionaries = false
167local forcepairadvance     = false -- for testing
168
169local repeatablemultiples  = context or false
170
171directives.register("otf.forcediscretionaries", function(v) forcediscretionaries = v end)
172directives.register("otf.forcepairadvance",     function(v) forcepairadvance     = v end)
173
174local report_direct      = logs.reporter("fonts","otf direct")
175local report_subchain    = logs.reporter("fonts","otf subchain")
176local report_chain       = logs.reporter("fonts","otf chain")
177local report_process     = logs.reporter("fonts","otf process")
178local report_warning     = logs.reporter("fonts","otf warning")
179local report_run         = logs.reporter("fonts","otf run")
180
181registertracker("otf.substitutions", "otf.singles","otf.multiples","otf.alternatives","otf.ligatures")
182registertracker("otf.positions",     "otf.marks","otf.kerns","otf.cursive")
183registertracker("otf.actions",       "otf.substitutions","otf.positions")
184registertracker("otf.sample",        "otf.steps","otf.substitutions","otf.positions","otf.analyzing")
185registertracker("otf.sample.silent", "otf.steps=silent","otf.substitutions","otf.positions","otf.analyzing")
186
187local nuts               = nodes.nuts
188
189local getnext            = nuts.getnext
190local setnext            = nuts.setnext
191local getprev            = nuts.getprev
192local setprev            = nuts.setprev
193local getboth            = nuts.getboth
194local setboth            = nuts.setboth
195local getid              = nuts.getid
196local getstate           = nuts.getstate
197local getsubtype         = nuts.getsubtype
198local getchar            = nuts.getchar
199local setchar            = nuts.setchar
200local getdisc            = nuts.getdisc
201local setdisc            = nuts.setdisc
202local getreplace         = nuts.getreplace
203local setlink            = nuts.setlink
204local getwidth           = nuts.getwidth
205local getattr            = nuts.getattr
206
207local getglyphdata       = nuts.getglyphdata
208
209---------------------------------------------------------------------------------------
210
211-- Beware: In ConTeXt components no longer are real components. We only keep track of
212-- their positions because some complex ligatures might need that. For the moment we
213-- use an x_ prefix because for now generic follows the other approach.
214
215local components         = nuts.components
216local copynocomponents   = components.copynocomponents
217local copyonlyglyphs     = components.copyonlyglyphs
218local countcomponents    = components.count
219local setcomponents      = components.set
220local getcomponents      = components.get
221local flushcomponents    = components.flush
222
223---------------------------------------------------------------------------------------
224
225local ischar             = nuts.ischar
226local usesfont           = nuts.usesfont
227
228local insertnodeafter    = nuts.insertafter
229local copynode           = nuts.copy
230local copynodelist       = nuts.copylist
231local removenode         = nuts.remove
232local findnodetail       = nuts.tail
233local flushnodelist      = nuts.flushlist
234local flushnode          = nuts.flushnode
235local endofmath          = nuts.endofmath
236
237local startofpar         = nuts.startofpar
238
239local setmetatable       = setmetatable
240local setmetatableindex  = table.setmetatableindex
241
242local nextnode           = nuts.traversers.node
243
244local nodecodes          = nodes.nodecodes
245local glyphcodes         = nodes.glyphcodes
246
247local glyph_code         = nodecodes.glyph
248local glue_code          = nodecodes.glue
249local disc_code          = nodecodes.disc
250local math_code          = nodecodes.math
251local dir_code           = nodecodes.dir
252local par_code           = nodecodes.par
253
254local lefttoright_code   = nodes.dirvalues.lefttoright
255local righttoleft_code   = nodes.dirvalues.righttoleft
256
257local discretionarydisc_code = nodes.disccodes.discretionary
258
259local a_noligature       = attributes.private("noligature")
260
261local injections         = nodes.injections
262local setmark            = injections.setmark
263local setcursive         = injections.setcursive
264local setkern            = injections.setkern
265local setmove            = injections.setmove
266local setposition        = injections.setposition
267local resetinjection     = injections.reset
268local copyinjection      = injections.copy
269local setligaindex       = injections.setligaindex
270local getligaindex       = injections.getligaindex
271
272local fontdata           = fonts.hashes.identifiers
273local fontfeatures       = fonts.hashes.features
274
275local otffeatures        = fonts.constructors.features.otf
276local registerotffeature = otffeatures.register
277
278local onetimemessage     = fonts.loggers.onetimemessage or function() end
279
280local getrandom          = utilities and utilities.randomizer and utilities.randomizer.get
281
282otf.defaultnodealternate = "none" -- first last
283
284-- We use a few semi-global variables. The handler can be called nested but this assumes
285-- that the same font is used.
286
287local tfmdata            = false
288local characters         = false
289local descriptions       = false
290local marks              = false
291local classes            = false
292local currentfont        = false
293local factor             = 0
294local threshold          = 0
295local checkmarks         = false
296
297local discs              = false
298local spaces             = false
299
300local sweepnode          = nil
301local sweephead          = { } -- we don't nil entries but false them (no collection and such)
302
303local notmatchpre        = { } -- to be checked: can we use false instead of nil / what if a == b tests
304local notmatchpost       = { } -- to be checked: can we use false instead of nil / what if a == b tests
305local notmatchreplace    = { } -- to be checked: can we use false instead of nil / what if a == b tests
306
307local handlers           = { }
308
309local isspace            = injections.isspace
310local getthreshold       = injections.getthreshold
311
312local checkstep          = (tracers and tracers.steppers.check)    or function() end
313local registerstep       = (tracers and tracers.steppers.register) or function() end
314local registermessage    = (tracers and tracers.steppers.message)  or function() end
315
316local function logprocess(...)
317    if trace_steps then
318        registermessage(...)
319        if trace_steps == "silent" then
320            return
321        end
322    end
323    report_direct(...)
324end
325
326local function logwarning(...)
327    report_direct(...)
328end
329
330local gref  do
331
332    local f_unicode = formatters["U+%X"]      -- was ["%U"]
333    local f_uniname = formatters["U+%X (%s)"] -- was ["%U (%s)"]
334    local f_unilist = formatters["% t"]
335
336    gref = function(n) -- currently the same as in font-otb
337        if type(n) == "number" then
338            local description = descriptions[n]
339            local name = description and description.name
340            if name then
341                return f_uniname(n,name)
342            else
343                return f_unicode(n)
344            end
345        elseif n then
346            local t = { }
347            for i=1,#n do
348                local ni = n[i]
349                if tonumber(ni) then -- later we will start at 2
350                    local di = descriptions[ni]
351                    local nn = di and di.name
352                    if nn then
353                        t[#t+1] = f_uniname(ni,nn)
354                    else
355                        t[#t+1] = f_unicode(ni)
356                    end
357                end
358            end
359            return f_unilist(t)
360        else
361            return "<error in node mode tracing>"
362        end
363    end
364
365end
366
367local function cref(dataset,sequence,index)
368    if not dataset then
369        return "no valid dataset"
370    end
371    local merged = sequence.merged and "merged " or ""
372    if index and index > 1 then
373        return formatters["feature %a, type %a, %schain lookup %a, index %a"](
374            dataset[4],sequence.type,merged,sequence.name,index)
375    else
376        return formatters["feature %a, type %a, %schain lookup %a"](
377            dataset[4],sequence.type,merged,sequence.name)
378    end
379end
380
381local function pref(dataset,sequence)
382    return formatters["feature %a, type %a, %slookup %a"](
383        dataset[4],sequence.type,sequence.merged and "merged " or "",sequence.name)
384end
385
386local function mref(rlmode)
387    if not rlmode or rlmode >= 0 then
388        return "l2r"
389    else
390        return "r2l"
391    end
392end
393
394-- The next code is somewhat complicated by the fact that some fonts can have ligatures made
395-- from ligatures that themselves have marks. This was identified by Kai in for instance
396-- arabtype:  KAF LAM SHADDA ALEF FATHA (0x0643 0x0644 0x0651 0x0627 0x064E). This becomes
397-- KAF LAM-ALEF with a SHADDA on the first and a FATHA op de second component. In a next
398-- iteration this becomes a KAF-LAM-ALEF with a SHADDA on the second and a FATHA on the
399-- third component.
400
401-- We can assume that languages that use marks are not hyphenated. We can also assume
402-- that at most one discretionary is present.
403
404-- We do need components in funny kerning mode but maybe I can better reconstruct then
405-- as we do have the font components info available; removing components makes the
406-- previous code much simpler. Also, later on copying and freeing becomes easier.
407-- However, for arabic we need to keep them around for the sake of mark placement
408-- and indices.
409
410local function flattendisk(head,disc)
411    local pre, post, replace, pretail, posttail, replacetail = getdisc(disc,true)
412    local prev, next = getboth(disc)
413    local ishead = head == disc
414    setdisc(disc)
415    flushnode(disc)
416    if pre then
417        flushnodelist(pre)
418    end
419    if post then
420        flushnodelist(post)
421    end
422    if ishead then
423        if replace then
424            if next then
425                setlink(replacetail,next)
426            end
427            return replace, replace
428        elseif next then
429            return next, next
430        else
431         -- return -- maybe warning
432        end
433    else
434        if replace then
435            if next then
436                setlink(replacetail,next)
437            end
438            setlink(prev,replace)
439            return head, replace
440        else
441            setlink(prev,next) -- checks for next anyway
442            return head, next
443        end
444    end
445end
446
447local function appenddisc(disc,list)
448    local pre, post, replace, pretail, posttail, replacetail = getdisc(disc,true)
449    local posthead    = list
450    local replacehead = copynodelist(list)
451    if post then
452        setlink(posttail,posthead)
453    else
454        post = posthead
455    end
456    if replace then
457        setlink(replacetail,replacehead)
458    else
459        replace = replacehead
460    end
461    setdisc(disc,pre,post,replace)
462end
463
464local function markstoligature(head,start,stop,char)
465    if start == stop and getchar(start) == char then
466        return head, start
467    else
468        local prev = getprev(start)
469        local next = getnext(stop)
470        setprev(start)
471        setnext(stop)
472        local base = copynocomponents(start,copyinjection)
473        if head == start then
474            head = base
475        end
476        resetinjection(base)
477        setchar(base,char)
478        setcomponents(base,start)
479        setlink(prev,base,next)
480        flushcomponents(start)
481        return head, base
482    end
483end
484
485-- Remark for Kai: (some arabic fonts do mark + mark = other mark and such)
486--
487-- The hasmarks is needed for ligatures of marks that are part of a ligature in
488-- which case we assume that we can delete the marks anyway (we can always become
489-- more clever if needed) .. in fact the whole logic here should be redone. We're
490-- in the not discfound branch then. We now have skiphash too so we can be more
491-- selective if needed (todo).
492
493-- we can have more granularity here but for now we only do a simple check
494
495local no_left_ligature_code  = 1
496local no_right_ligature_code = 2
497local no_left_kern_code      = 4
498local no_right_kern_code     = 8
499
500local hasglyphoption = function(n,c)
501    if c == no_left_ligature_code or c == no_right_ligature_code then
502        return getattr(n,a_noligature) == 1
503    else
504        return false
505    end
506end
507
508-- in lmtx we need to check the components and can be slightly more clever
509
510local function toligature(head,start,stop,char,dataset,sequence,skiphash,discfound,hasmarks) -- brr head
511    if hasglyphoption(start,no_right_ligature_code) then
512        return head, start
513    end
514    if start == stop and getchar(start) == char then
515        resetinjection(start)
516        setchar(start,char)
517        return head, start
518    end
519    local prev = getprev(start)
520    local next = getnext(stop)
521    local comp = start
522    setprev(start)
523    setnext(stop)
524    local base = copynocomponents(start,copyinjection)
525    if start == head then
526        head = base
527    end
528    resetinjection(base)
529    setchar(base,char)
530    setcomponents(base,comp)
531    setlink(prev,base,next)
532    if not discfound then
533        local deletemarks = not skiphash or hasmarks
534        local components = start -- not used
535        local baseindex = 0
536        local componentindex = 0
537        local head = base
538        local current = base
539        -- first we loop over the glyphs in start ... stop
540        while start do
541            local char = getchar(start)
542            if not marks[char] then
543                baseindex = baseindex + componentindex
544                componentindex = countcomponents(start,marks)
545             -- we can be more clever here: "not deletemarks or (skiphash and not skiphash[char])"
546             -- and such:
547            elseif not deletemarks then
548                -- we can get a loop when the font expects otherwise (i.e. unexpected deletemarks)
549                setligaindex(start,baseindex + getligaindex(start,componentindex))
550                if trace_marks then
551                    logwarning("%s: keep ligature mark %s, gets index %s",pref(dataset,sequence),gref(char),getligaindex(start))
552                end
553                local n = copynode(start)
554                copyinjection(n,start) -- is this ok ? we position later anyway
555                head, current = insertnodeafter(head,current,n) -- unlikely that mark has components
556            elseif trace_marks then
557                logwarning("%s: delete ligature mark %s",pref(dataset,sequence),gref(char))
558            end
559            start = getnext(start)
560        end
561        -- we can have one accent as part of a lookup and another following
562        local start = getnext(current)
563        while start do
564            local char = ischar(start)
565            if char then
566                -- also something skiphash here?
567                if marks[char] then
568                    setligaindex(start,baseindex + getligaindex(start,componentindex))
569                    if trace_marks then
570                        logwarning("%s: set ligature mark %s, gets index %s",pref(dataset,sequence),gref(char),getligaindex(start))
571                    end
572                    start = getnext(start)
573                else
574                    break
575                end
576            else
577                break
578            end
579        end
580        flushcomponents(components)
581    else
582        -- discfound ... forget about marks .. probably no scripts that hyphenate and have marks
583        local discprev, discnext = getboth(discfound)
584        if discprev and discnext then
585            -- we assume normalization in context, and don't care about generic ... especially
586            -- \- can give problems as there we can have a negative char but that won't match
587            -- anyway
588            local pre, post, replace, pretail, posttail, replacetail = getdisc(discfound,true)
589            if not replace then
590                -- looks like we never come here as it's not okay
591                local prev = getprev(base)
592             -- local comp = getcomponents(base) -- already set
593                local copied = copyonlyglyphs(comp)
594                if pre then
595                    setlink(discprev,pre)
596                else
597                    setnext(discprev) -- also blocks funny assignments
598                end
599                pre = comp -- is start
600                if post then
601                    setlink(posttail,discnext)
602                    setprev(post) -- nil anyway
603                else
604                    post = discnext
605                    setprev(discnext) -- also blocks funny assignments
606                end
607                setlink(prev,discfound,next)
608                setboth(base)
609                -- here components have a pointer so we can't free it!
610                setcomponents(base,copied)
611                replace = base
612                if forcediscretionaries then
613                    setdisc(discfound,pre,post,replace,discretionarydisc_code)
614                else
615                    setdisc(discfound,pre,post,replace)
616                end
617                base = prev
618            end
619        end
620    end
621    return head, base
622end
623
624local function multiple_glyphs(head,start,multiple,skiphash,what,stop) -- what to do with skiphash matches here
625    local nofmultiples = #multiple
626    if nofmultiples > 0 then
627        local first = start
628        resetinjection(start)
629        setchar(start,multiple[1])
630        if nofmultiples > 1 then
631         -- local sn = getnext(start)
632            for i=2,nofmultiples do
633             -- untested:
634             --
635             -- while ignoremarks and marks[getchar(sn)] then
636             --     local sn = getnext(sn)
637             -- end
638                local n = copynode(start) -- ignore components
639                resetinjection(n)
640                setchar(n,multiple[i])
641                insertnodeafter(head,start,n)
642                start = n
643            end
644        end
645        if what ~= true and repeatablemultiples then
646            -- This is just some experimental code; we might introduce gsub_extensible
647            -- some day instead. Beware: when we have a feature that mixes alternates and
648            -- multiples we need to make sure we don't handle the alternate string values
649            -- here.
650            local kind = type(what)
651            local m, f, l
652            if kind == "string" then
653                local what, n = string.match(what,"^repeat(.-)[:=](%d+)$")
654                if what == "middle" then
655                    m = tonumber(n)
656                elseif what == "first" then
657                    f = tonumber(n)
658                elseif what == "last" then
659                    l = tonumber(n)
660                end
661            elseif kind == "table" then
662                -- won't happen because currently we don't split these values
663               m = what.middle
664               f = what.first
665               l = what.last
666            end
667            if f or m or l then
668                if m and m > 1 and nofmultiples == 3 then
669                    local middle = getnext(first)
670                    for i=2,m do
671                        local n = copynode(middle) -- ignore components
672                        resetinjection(n)
673                        insertnodeafter(head,first,n)
674                    end
675                end
676                if f and f > 1 then
677                    for i=2,f do
678                        local n = copynode(first) -- ignore components
679                        resetinjection(n)
680                        insertnodeafter(head,first,n)
681                    end
682                end
683                if l and l > 1 then
684                    for i=2,l do
685                        local n = copynode(start) -- ignore components
686                        resetinjection(n)
687                        insertnodeafter(head,start,n)
688                        start = n
689                    end
690                end
691            end
692        end
693        return head, start, true
694    else
695        if trace_multiples then
696            logprocess("no multiple for %s",gref(getchar(start)))
697        end
698        return head, start, false
699    end
700end
701
702local function get_alternative_glyph(start,alternatives,value)
703    local n = #alternatives
704    if n == 1 then
705        -- we could actually change that into a gsub and save some memory in the
706        -- font loader but it makes tracing more messy
707        return alternatives[1], trace_alternatives and "1 (only one present)"
708    elseif value == "random" then
709        local r = getrandom and getrandom("glyph",1,n) or random(1,n)
710        return alternatives[r], trace_alternatives and formatters["value %a, taking %a"](value,r)
711    elseif value == "first" then
712        return alternatives[1], trace_alternatives and formatters["value %a, taking %a"](value,1)
713    elseif value == "last" then
714        return alternatives[n], trace_alternatives and formatters["value %a, taking %a"](value,n)
715    end
716    value = value == true and 1 or tonumber(value)
717    if type(value) ~= "number" then
718        return alternatives[1], trace_alternatives and formatters["invalid value %s, taking %a"](value,1)
719    end
720 -- local a = alternatives[value]
721 -- if a then
722 --     -- some kind of hash
723 --     return a, trace_alternatives and formatters["value %a, taking %a"](value,a)
724 -- end
725    if value > n then
726        local defaultalt = otf.defaultnodealternate
727        if defaultalt == "first" then
728            return alternatives[n], trace_alternatives and formatters["invalid value %s, taking %a"](value,1)
729        elseif defaultalt == "last" then
730            return alternatives[1], trace_alternatives and formatters["invalid value %s, taking %a"](value,n)
731        else
732            return false, trace_alternatives and formatters["invalid value %a, %s"](value,"out of range")
733        end
734    elseif value == 0 then
735        return getchar(start), trace_alternatives and formatters["invalid value %a, %s"](value,"no change")
736    elseif value < 1 then
737        return alternatives[1], trace_alternatives and formatters["invalid value %a, taking %a"](value,1)
738    else
739        return alternatives[value], trace_alternatives and formatters["value %a, taking %a"](value,value)
740    end
741end
742
743-- handlers
744
745function handlers.gsub_single(head,start,dataset,sequence,replacement)
746    if trace_singles then
747        logprocess("%s: replacing %s by single %s",pref(dataset,sequence),gref(getchar(start)),gref(replacement))
748    end
749    resetinjection(start)
750    setchar(start,replacement)
751    return head, start, true
752end
753
754function handlers.gsub_alternate(head,start,dataset,sequence,alternative)
755    local kind  = dataset[4]
756    local what  = dataset[1]
757    local value = what == true and tfmdata.shared.features[kind] or what
758    local choice, comment = get_alternative_glyph(start,alternative,value)
759    if choice then
760        if trace_alternatives then
761            logprocess("%s: replacing %s by alternative %a to %s, %s",pref(dataset,sequence),gref(getchar(start)),gref(choice),comment)
762        end
763        resetinjection(start)
764        setchar(start,choice)
765    else
766        if trace_alternatives then
767            logwarning("%s: no variant %a for %s, %s",pref(dataset,sequence),value,gref(getchar(start)),comment)
768        end
769    end
770    return head, start, true
771end
772
773function handlers.gsub_multiple(head,start,dataset,sequence,multiple,rlmode,skiphash)
774    if trace_multiples then
775        logprocess("%s: replacing %s by multiple %s",pref(dataset,sequence),gref(getchar(start)),gref(multiple))
776    end
777    return multiple_glyphs(head,start,multiple,skiphash,dataset[1])
778end
779
780-- Don't we deal with disc otherwise now? I need to check if the next one can be
781-- simplified. Anyway, it can be way messier: marks that get removed as well as
782-- marks that are kept.
783
784function handlers.gsub_ligature(head,start,dataset,sequence,ligature,rlmode,skiphash)
785    local current   = getnext(start)
786    if not current then
787        return head, start, false, nil
788    end
789    local stop      = nil
790    local startchar = getchar(start)
791    if skiphash and skiphash[startchar] then
792        while current do
793            local char = ischar(current,currentfont)
794            if char then
795                local lg = not tonumber(ligature) and ligature[char]
796                if lg then
797                    stop     = current
798                    ligature = lg
799                    current  = getnext(current)
800                else
801                    break
802                end
803            else
804                break
805            end
806        end
807        if stop then
808            local ligature = tonumber(ligature) or ligature.ligature
809            if ligature then
810                if trace_ligatures then
811                    local stopchar = getchar(stop)
812                    head, start = markstoligature(head,start,stop,ligature)
813                    logprocess("%s: replacing %s upto %s by ligature %s case 1",pref(dataset,sequence),gref(startchar),gref(stopchar),gref(getchar(start)))
814                else
815                    head, start = markstoligature(head,start,stop,ligature)
816                end
817                return head, start, true, false
818            else
819                -- ok, goto next lookup
820            end
821        end
822    else
823        local discfound = false
824        local hasmarks  = marks[startchar]
825        while current do
826            local char, id = ischar(current,currentfont)
827            if char then
828                if skiphash and skiphash[char] then
829                    current = getnext(current)
830                else
831                    local lg = not tonumber(ligature) and ligature[char]
832                    if lg then
833                        if marks[char] then
834                            hasmarks = true
835                        end
836                        stop     = current -- needed for fake so outside then
837                        ligature = lg
838                        current  = getnext(current)
839                    else
840                        break
841                    end
842                end
843            elseif char == false then
844                -- kind of weird
845                break
846            elseif id == disc_code then
847                discfound = current
848                break
849            else
850                break
851            end
852        end
853        -- of{f-}{}{f}e  o{f-}{}{f}fe  o{-}{}{ff}e (oe and ff ligature)
854        -- we can end up here when we have a start run .. testruns start at a disc but
855        -- so here we have the other case: char + disc
856        --
857        -- Challenge for Kai (latinmodern):  \hyphenation{fii-f-f-iif} fiiffiif
858        --
859        if discfound then
860            -- don't assume marks in a disc and we don't run over a disc (for now)
861            local pre, post, replace = getdisc(discfound)
862            local match
863            if replace then
864                local char = ischar(replace,currentfont)
865                if char and (not tonumber(ligature) and ligature[char]) then
866                    match = true
867                end
868            end
869            if not match and pre then
870                local char = ischar(pre,currentfont)
871                if char and (not tonumber(ligature) and ligature[char]) then
872                    match = true
873                end
874            end
875            if not match and not pre or not replace then
876                local n    = getnext(discfound)
877                local char = ischar(n,currentfont)
878                if char and (not tonumber(ligature) and ligature[char]) then
879                    match = true
880                end
881            end
882            if match then
883                -- we force a restart
884                local ishead = head == start
885                local prev   = getprev(start)
886                if stop then
887                    setnext(stop)
888                    local copy = copynodelist(start)
889                    local tail = stop -- was: getprev(stop) -- Kai: needs checking on your samples
890                    local liat = findnodetail(copy)
891                    if pre then
892                        setlink(liat,pre)
893                    end
894                    if replace then
895                        setlink(tail,replace)
896                    end
897                    pre     = copy
898                    replace = start
899                else
900                    setnext(start)
901                    local copy = copynode(start)
902                    if pre then
903                        setlink(copy,pre)
904                    end
905                    if replace then
906                        setlink(start,replace)
907                    end
908                    pre     = copy
909                    replace = start
910                end
911                setdisc(discfound,pre,post,replace)
912                if prev then
913                    setlink(prev,discfound)
914                else
915                    setprev(discfound)
916                    head  = discfound
917                end
918                start = discfound
919                return head, start, true, true
920            end
921        end
922        local ligature = tonumber(ligature) or ligature.ligature
923        if ligature then
924            if stop then
925                if trace_ligatures then
926                    local stopchar = getchar(stop)
927                 -- head, start = toligature(head,start,stop,ligature,dataset,sequence,skiphash,discfound,hasmarks)
928                    head, start = toligature(head,start,stop,ligature,dataset,sequence,skiphash,false,hasmarks)
929                    logprocess("%s: replacing %s upto %s by ligature %s case 2",pref(dataset,sequence),gref(startchar),gref(stopchar),gref(ligature))
930                 -- we can have a rare case of multiple disc in a lig but that makes no sense language wise but if really
931                 -- needed we could backtrack if we're in a disc node
932                else
933                 -- head, start = toligature(head,start,stop,ligature,dataset,sequence,skiphash,discfound,hasmarks)
934                    head, start = toligature(head,start,stop,ligature,dataset,sequence,skiphash,false,hasmarks)
935                end
936            else
937                -- weird but happens (in some arabic font)
938                resetinjection(start)
939                setchar(start,ligature)
940                if trace_ligatures then
941                    logprocess("%s: replacing %s by (no real) ligature %s case 3",pref(dataset,sequence),gref(startchar),gref(ligature))
942                end
943            end
944            return head, start, true, false
945        else
946            -- weird but happens, pseudo ligatures ... just the components
947        end
948    end
949    return head, start, false, false
950end
951
952function handlers.gpos_single(head,start,dataset,sequence,kerns,rlmode,skiphash,step,injection)
953    if hasglyphoption(start,no_right_kern_code) then
954        return head, start, false
955    else
956        local startchar = getchar(start)
957        local format    = step.format
958        if format == "single" or type(kerns) == "table" then -- the table check can go
959            local dx, dy, w, h = setposition(0,start,factor,rlmode,kerns,injection)
960            if trace_kerns then
961                logprocess("%s: shifting single %s by %s xy (%p,%p) and wh (%p,%p)",pref(dataset,sequence),gref(startchar),format,dx,dy,w,h)
962            end
963        else
964            local k = (format == "move" and setmove or setkern)(start,factor,rlmode,kerns,injection)
965            if trace_kerns then
966                logprocess("%s: shifting single %s by %s %p",pref(dataset,sequence),gref(startchar),format,k)
967            end
968        end
969        return head, start, true
970    end
971end
972
973function handlers.gpos_pair(head,start,dataset,sequence,kerns,rlmode,skiphash,step,injection)
974    if hasglyphoption(start,no_right_kern_code) then
975        return head, start, false
976    else
977        local snext = getnext(start)
978        if not snext then
979            return head, start, false
980        else
981            local prev = start
982            while snext do
983                local nextchar = ischar(snext,currentfont)
984                if nextchar then
985                    if skiphash and skiphash[nextchar] then -- includes marks too when flag
986                        prev  = snext
987                        snext = getnext(snext)
988                    else
989                        local krn = kerns[nextchar]
990                        if not krn then
991                            break
992                        end
993                        local format = step.format
994                        if format == "pair" then
995                            local a = krn[1]
996                            local b = krn[2]
997                            if a == true then
998                                -- zero
999                            elseif a then -- #a > 0
1000                                local x, y, w, h = setposition(1,start,factor,rlmode,a,injection)
1001                                if trace_kerns then
1002                                    local startchar = getchar(start)
1003                                    logprocess("%s: shifting first of pair %s and %s by xy (%p,%p) and wh (%p,%p) as %s",pref(dataset,sequence),gref(startchar),gref(nextchar),x,y,w,h,injection or "injections")
1004                                end
1005                            end
1006                            if b == true then
1007                                -- zero
1008                                start = snext -- cf spec
1009                            elseif b then -- #b > 0
1010                                local x, y, w, h = setposition(2,snext,factor,rlmode,b,injection)
1011                                if trace_kerns then
1012                                    local startchar = getchar(start)
1013                                    logprocess("%s: shifting second of pair %s and %s by xy (%p,%p) and wh (%p,%p) as %s",pref(dataset,sequence),gref(startchar),gref(nextchar),x,y,w,h,injection or "injections")
1014                                end
1015                                start = snext -- cf spec
1016                            elseif forcepairadvance then
1017                                start = snext -- for testing, not cf spec
1018                            end
1019                            return head, start, true
1020                        elseif krn ~= 0 then
1021                            local k = (format == "move" and setmove or setkern)(snext,factor,rlmode,krn,injection)
1022                            if trace_kerns then
1023                                logprocess("%s: inserting %s %p between %s and %s as %s",pref(dataset,sequence),format,k,gref(getchar(prev)),gref(nextchar),injection or "injections")
1024                            end
1025                            return head, start, true
1026                        else -- can't happen
1027                            break
1028                        end
1029                    end
1030                else
1031                    break
1032                end
1033            end
1034            return head, start, false
1035        end
1036    end
1037end
1038
1039-- We get hits on a mark, but we're not sure if the it has to be applied so we need
1040-- to explicitly test for basechar, baselig and basemark entries.
1041
1042function handlers.gpos_mark2base(head,start,dataset,sequence,markanchors,rlmode,skiphash)
1043    local markchar = getchar(start)
1044    if marks[markchar] then
1045        local base = getprev(start) -- [glyph] [start=mark]
1046        if base then
1047            local basechar = ischar(base,currentfont)
1048            if basechar then
1049                if marks[basechar] then
1050                    while base do
1051                        base = getprev(base)
1052                        if base then
1053                            basechar = ischar(base,currentfont)
1054                            if basechar then
1055                                if not marks[basechar] then
1056                                    break
1057                                end
1058                            else
1059                                if trace_bugs then
1060                                    logwarning("%s: no base for mark %s, case %i",pref(dataset,sequence),gref(markchar),1)
1061                                end
1062                                return head, start, false
1063                            end
1064                        else
1065                            if trace_bugs then
1066                                logwarning("%s: no base for mark %s, case %i",pref(dataset,sequence),gref(markchar),2)
1067                            end
1068                            return head, start, false
1069                        end
1070                    end
1071                end
1072                local ba = markanchors[1][basechar]
1073                if ba then
1074                    local ma = markanchors[2]
1075                    local dx, dy, bound = setmark(start,base,factor,rlmode,ba,ma,characters[basechar],false,checkmarks)
1076                    if trace_marks then
1077                        logprocess("%s, bound %s, anchoring mark %s to basechar %s => (%p,%p)",
1078                            pref(dataset,sequence),bound,gref(markchar),gref(basechar),dx,dy)
1079                    end
1080                    return head, start, true
1081                elseif trace_bugs then
1082                 -- onetimemessage(currentfont,basechar,"no base anchors")
1083                    logwarning("%s: mark %s is not anchored to %s",pref(dataset,sequence),gref(markchar),gref(basechar))
1084                end
1085            elseif trace_bugs then
1086                logwarning("%s: nothing preceding, case %i",pref(dataset,sequence),1)
1087            end
1088        elseif trace_bugs then
1089            logwarning("%s: nothing preceding, case %i",pref(dataset,sequence),2)
1090        end
1091    elseif trace_bugs then
1092        logwarning("%s: mark %s is no mark",pref(dataset,sequence),gref(markchar))
1093    end
1094    return head, start, false
1095end
1096
1097function handlers.gpos_mark2ligature(head,start,dataset,sequence,markanchors,rlmode,skiphash)
1098    local markchar = getchar(start)
1099    if marks[markchar] then
1100        local base = getprev(start) -- [glyph] [optional marks] [start=mark]
1101        if base then
1102            local basechar = ischar(base,currentfont)
1103            if basechar then
1104                if marks[basechar] then
1105                    while base do
1106                        base = getprev(base)
1107                        if base then
1108                            basechar = ischar(base,currentfont)
1109                            if basechar then
1110                                if not marks[basechar] then
1111                                    break
1112                                end
1113                            else
1114                                if trace_bugs then
1115                                    logwarning("%s: no base for mark %s, case %i",pref(dataset,sequence),gref(markchar),1)
1116                                end
1117                                return head, start, false
1118                            end
1119                        else
1120                            if trace_bugs then
1121                                logwarning("%s: no base for mark %s, case %i",pref(dataset,sequence),gref(markchar),2)
1122                            end
1123                            return head, start, false
1124                        end
1125                    end
1126                end
1127                local ba = markanchors[1][basechar]
1128                if ba then
1129                    local ma = markanchors[2]
1130                    if ma then
1131                        local index = getligaindex(start)
1132                        ba = ba[index]
1133                        if ba then
1134                            local dx, dy, bound = setmark(start,base,factor,rlmode,ba,ma,characters[basechar],false,checkmarks)
1135                            if trace_marks then
1136                                logprocess("%s, index %s, bound %s, anchoring mark %s to baselig %s at index %s => (%p,%p)",
1137                                    pref(dataset,sequence),index,bound,gref(markchar),gref(basechar),index,dx,dy)
1138                            end
1139                            return head, start, true
1140                        else
1141                            if trace_bugs then
1142                                logwarning("%s: no matching anchors for mark %s and baselig %s with index %a",pref(dataset,sequence),gref(markchar),gref(basechar),index)
1143                            end
1144                        end
1145                    end
1146                elseif trace_bugs then
1147                --  logwarning("%s: char %s is missing in font",pref(dataset,sequence),gref(basechar))
1148                    onetimemessage(currentfont,basechar,"no base anchors")
1149                end
1150            elseif trace_bugs then
1151                logwarning("%s: prev node is no char, case %i",pref(dataset,sequence),1)
1152            end
1153        elseif trace_bugs then
1154            logwarning("%s: prev node is no char, case %i",pref(dataset,sequence),2)
1155        end
1156    elseif trace_bugs then
1157        logwarning("%s: mark %s is no mark",pref(dataset,sequence),gref(markchar))
1158    end
1159    return head, start, false
1160end
1161
1162function handlers.gpos_mark2mark(head,start,dataset,sequence,markanchors,rlmode,skiphash)
1163    local markchar = getchar(start)
1164    if marks[markchar] then
1165        local base = getprev(start) -- [glyph] [basemark] [start=mark]
1166        local slc = getligaindex(start)
1167        if slc then -- a rather messy loop ... needs checking with husayni
1168            while base do
1169                local blc = getligaindex(base)
1170                if blc and blc ~= slc then
1171                    base = getprev(base)
1172                else
1173                    break
1174                end
1175            end
1176        end
1177        if base then
1178            local basechar = ischar(base,currentfont)
1179            if basechar then -- subtype test can go
1180                local ba = markanchors[1][basechar] -- slot 1 has been made copy of the class hash
1181                if ba then
1182                    local ma = markanchors[2]
1183                    local dx, dy, bound = setmark(start,base,factor,rlmode,ba,ma,characters[basechar],true,checkmarks)
1184                    if trace_marks then
1185                        logprocess("%s, bound %s, anchoring mark %s to basemark %s => (%p,%p)",
1186                            pref(dataset,sequence),bound,gref(markchar),gref(basechar),dx,dy)
1187                    end
1188                    return head, start, true
1189                end
1190            end
1191        end
1192    elseif trace_bugs then
1193        logwarning("%s: mark %s is no mark",pref(dataset,sequence),gref(markchar))
1194    end
1195    return head, start, false
1196end
1197
1198function handlers.gpos_cursive(head,start,dataset,sequence,exitanchors,rlmode,skiphash,step) -- to be checked
1199    local startchar = getchar(start)
1200    if marks[startchar] then
1201        if trace_cursive then
1202            logprocess("%s: ignoring cursive for mark %s",pref(dataset,sequence),gref(startchar))
1203        end
1204    else
1205        local nxt = getnext(start)
1206        while nxt do
1207            local nextchar = ischar(nxt,currentfont)
1208            if not nextchar then
1209                break
1210            elseif marks[nextchar] then -- always sequence.flags[1]
1211                nxt = getnext(nxt)
1212            else
1213                local exit = exitanchors[3]
1214                if exit then
1215                    local entry = exitanchors[1][nextchar]
1216                    if entry then
1217                        entry = entry[2]
1218                        if entry then
1219                            local r2lflag = sequence.flags[4] -- mentioned in the standard
1220                            local dx, dy, bound = setcursive(start,nxt,factor,rlmode,exit,entry,characters[startchar],characters[nextchar],r2lflag)
1221                            if trace_cursive then
1222                                logprocess("%s: moving %s to %s cursive (%p,%p) using bound %s in %s mode",pref(dataset,sequence),gref(startchar),gref(nextchar),dx,dy,bound,mref(rlmode))
1223                            end
1224                            return head, start, true
1225                        end
1226                    end
1227                end
1228                break
1229            end
1230        end
1231    end
1232    return head, start, false
1233end
1234
1235-- I will implement multiple chain replacements once I run into a font that uses it.
1236-- It's not that complex to handle.
1237
1238local chainprocs = { }
1239
1240local function logprocess(...)
1241    if trace_steps then
1242        registermessage(...)
1243        if trace_steps == "silent" then
1244            return
1245        end
1246    end
1247    report_subchain(...)
1248end
1249
1250local logwarning = report_subchain
1251
1252local function logprocess(...)
1253    if trace_steps then
1254        registermessage(...)
1255        if trace_steps == "silent" then
1256            return
1257        end
1258    end
1259    report_chain(...)
1260end
1261
1262local logwarning = report_chain
1263
1264-- We could share functions but that would lead to extra function calls with many
1265-- arguments, redundant tests and confusing messages.
1266
1267-- The reversesub is a special case, which is why we need to store the replacements
1268-- in a bit weird way. There is no lookup and the replacement comes from the lookup
1269-- itself. It is meant mostly for dealing with Urdu.
1270
1271local function reversesub(head,start,stop,dataset,sequence,replacements,rlmode,skiphash)
1272    local char        = getchar(start)
1273    local replacement = replacements[char]
1274    if replacement then
1275        if trace_singles then
1276            logprocess("%s: single reverse replacement of %s by %s",cref(dataset,sequence),gref(char),gref(replacement))
1277        end
1278        resetinjection(start)
1279        setchar(start,replacement)
1280        return head, start, true
1281    else
1282        return head, start, false
1283    end
1284end
1285
1286
1287chainprocs.reversesub = reversesub
1288
1289-- This chain stuff is somewhat tricky since we can have a sequence of actions to be
1290-- applied: single, alternate, multiple or ligature where ligature can be an invalid
1291-- one in the sense that it will replace multiple by one but not neccessary one that
1292-- looks like the combination (i.e. it is the counterpart of multiple then). For
1293-- example, the following is valid:
1294--
1295--   xxxabcdexxx [single a->A][multiple b->BCD][ligature cde->E] xxxABCDExxx
1296--
1297-- Therefore we we don't really do the replacement here already unless we have the
1298-- single lookup case. The efficiency of the replacements can be improved by
1299-- deleting as less as needed but that would also make the code even more messy.
1300--
1301-- Here we replace start by a single variant.
1302--
1303-- To be done   : what if > 1 steps (example needed)
1304-- This is messy: do we need this disc checking also in alternates?
1305
1306local function reportzerosteps(dataset,sequence)
1307    logwarning("%s: no steps",cref(dataset,sequence))
1308end
1309
1310local function reportmoresteps(dataset,sequence)
1311    logwarning("%s: more than 1 step",cref(dataset,sequence))
1312end
1313
1314-- local function reportbadsteps(dataset,sequence)
1315--     logwarning("%s: bad step, no proper return values",cref(dataset,sequence))
1316-- end
1317
1318local function getmapping(dataset,sequence,currentlookup)
1319    local steps    = currentlookup.steps
1320    local nofsteps = currentlookup.nofsteps
1321    if nofsteps == 0 then
1322        reportzerosteps(dataset,sequence)
1323        currentlookup.mapping = false
1324        return false
1325    else
1326        if nofsteps > 1 then
1327            reportmoresteps(dataset,sequence)
1328        end
1329        local mapping = steps[1].coverage
1330        currentlookup.mapping = mapping
1331        currentlookup.format  = steps[1].format
1332        return mapping
1333    end
1334end
1335
1336function chainprocs.gsub_remove(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash,chainindex)
1337    if trace_chains then
1338        logprocess("%s: removing character %s",cref(dataset,sequence,chainindex),gref(getchar(start)))
1339    end
1340    head, start = removenode(head,start,true)
1341    return head, getprev(start), true
1342end
1343
1344function chainprocs.gsub_single(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash,chainindex)
1345    local mapping = currentlookup.mapping
1346    if mapping == nil then
1347        mapping = getmapping(dataset,sequence,currentlookup)
1348    end
1349    if mapping then
1350        local current = start
1351        while current do
1352            local currentchar = ischar(current)
1353            if currentchar then
1354                local replacement = mapping[currentchar]
1355                if not replacement or replacement == "" then
1356                    if trace_bugs then
1357                        logwarning("%s: no single for %s",cref(dataset,sequence,chainindex),gref(currentchar))
1358                    end
1359                else
1360                    if trace_singles then
1361                        logprocess("%s: replacing single %s by %s",cref(dataset,sequence,chainindex),gref(currentchar),gref(replacement))
1362                    end
1363                    resetinjection(current)
1364                    setchar(current,replacement)
1365                end
1366                return head, start, true
1367            elseif currentchar == false then
1368                -- can't happen
1369                break
1370            elseif current == stop then
1371                break
1372            else
1373                current = getnext(current)
1374            end
1375        end
1376    end
1377    return head, start, false
1378end
1379
1380-- Here we replace start by new glyph. First we delete the rest of the match.
1381
1382-- char_1 mark_1 -> char_x mark_1 (ignore marks)
1383-- char_1 mark_1 -> char_x
1384
1385-- to be checked: do we always have just one glyph?
1386-- we can also have alternates for marks
1387-- marks come last anyway
1388-- are there cases where we need to delete the mark
1389
1390function chainprocs.gsub_alternate(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash,chainindex)
1391    local mapping = currentlookup.mapping
1392    if mapping == nil then
1393        mapping = getmapping(dataset,sequence,currentlookup)
1394    end
1395    if mapping then
1396        local kind    = dataset[4]
1397        local what    = dataset[1]
1398        local value   = what == true and tfmdata.shared.features[kind] or what -- todo: optimize in ctx
1399        local current = start
1400        while current do
1401            local currentchar = ischar(current)
1402            if currentchar then
1403                local alternatives = mapping[currentchar]
1404                if alternatives then
1405                    local choice, comment = get_alternative_glyph(current,alternatives,value)
1406                    if choice then
1407                        if trace_alternatives then
1408                            logprocess("%s: replacing %s by alternative %a to %s, %s",cref(dataset,sequence),gref(currentchar),choice,gref(choice),comment)
1409                        end
1410                        resetinjection(start)
1411                        setchar(start,choice)
1412                    else
1413                        if trace_alternatives then
1414                            logwarning("%s: no variant %a for %s, %s",cref(dataset,sequence),value,gref(currentchar),comment)
1415                        end
1416                    end
1417                end
1418                return head, start, true
1419            elseif currentchar == false then
1420                -- can't happen
1421                break
1422            elseif current == stop then
1423                break
1424            else
1425                current = getnext(current)
1426            end
1427        end
1428    end
1429    return head, start, false
1430end
1431
1432-- Here we replace start by a sequence of new glyphs.
1433
1434function chainprocs.gsub_multiple(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash,chainindex)
1435    local mapping = currentlookup.mapping
1436    if mapping == nil then
1437        mapping = getmapping(dataset,sequence,currentlookup)
1438    end
1439    if mapping then
1440        local startchar   = getchar(start)
1441        local replacement = mapping[startchar]
1442        if not replacement or replacement == "" then
1443            if trace_bugs then
1444                logwarning("%s: no multiple for %s",cref(dataset,sequence),gref(startchar))
1445            end
1446        else
1447            if trace_multiples then
1448                logprocess("%s: replacing %s by multiple characters %s",cref(dataset,sequence),gref(startchar),gref(replacement))
1449            end
1450            return multiple_glyphs(head,start,replacement,skiphash,dataset[1],stop)
1451        end
1452    end
1453    return head, start, false
1454end
1455
1456-- When we replace ligatures we use a helper that handles the marks. I might change
1457-- this function (move code inline and handle the marks by a separate function). We
1458-- assume rather stupid ligatures (no complex disc nodes).
1459
1460-- compare to handlers.gsub_ligature which is more complex ... why
1461
1462function chainprocs.gsub_ligature(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash,chainindex)
1463    local mapping = currentlookup.mapping
1464    if mapping == nil then
1465        mapping = getmapping(dataset,sequence,currentlookup)
1466    end
1467    if mapping then
1468        local startchar = getchar(start)
1469        local ligatures = mapping[startchar]
1470        if not ligatures then
1471            if trace_bugs then
1472                logwarning("%s: no ligatures starting with %s",cref(dataset,sequence,chainindex),gref(startchar))
1473            end
1474        else
1475            local hasmarks        = marks[startchar]
1476            local current         = getnext(start)
1477            local discfound       = false
1478            local last            = stop
1479            local nofreplacements = 1
1480            while current do
1481                -- todo: ischar ... can there really be disc nodes here?
1482                local id = getid(current)
1483                if id == disc_code then
1484                    if not discfound then
1485                        discfound = current
1486                    end
1487                    if current == stop then
1488                        break -- okay? or before the disc
1489                    else
1490                        current = getnext(current)
1491                    end
1492                else
1493                    local schar = getchar(current)
1494                    if skiphash and skiphash[schar] then -- marks
1495                        -- if current == stop then -- maybe add this
1496                        --     break
1497                        -- else
1498                            current = getnext(current)
1499                        -- end
1500                    else
1501                        local lg = not tonumber(ligatures) and ligatures[schar]
1502                        if lg then
1503                            ligatures       = lg
1504                            last            = current
1505                            nofreplacements = nofreplacements + 1
1506                            if marks[char] then
1507                                hasmarks = true
1508                            end
1509                            if current == stop then
1510                                break
1511                            else
1512                                current = getnext(current)
1513                            end
1514                        else
1515                            break
1516                        end
1517                    end
1518                end
1519            end
1520            local ligature = tonumber(ligatures) or ligatures.ligature
1521            if ligature then
1522                if chainindex then
1523                    stop = last
1524                end
1525                if trace_ligatures then
1526                    if start == stop then
1527                        logprocess("%s: replacing character %s by ligature %s case 3",cref(dataset,sequence,chainindex),gref(startchar),gref(ligature))
1528                    else
1529                        logprocess("%s: replacing character %s upto %s by ligature %s case 4",cref(dataset,sequence,chainindex),gref(startchar),gref(getchar(stop)),gref(ligature))
1530                    end
1531                end
1532                head, start = toligature(head,start,stop,ligature,dataset,sequence,skiphash,discfound,hasmarks)
1533                return head, start, true, nofreplacements, discfound
1534            elseif trace_bugs then
1535                if start == stop then
1536                    logwarning("%s: replacing character %s by ligature fails",cref(dataset,sequence,chainindex),gref(startchar))
1537                else
1538                    logwarning("%s: replacing character %s upto %s by ligature fails",cref(dataset,sequence,chainindex),gref(startchar),gref(getchar(stop)))
1539                end
1540            end
1541        end
1542    end
1543    return head, start, false, 0, false
1544end
1545
1546function chainprocs.gpos_single(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash,chainindex)
1547    -- we actually should check no_left_kern_code with next
1548    if not hasglyphoption(start,no_right_kern_code) then
1549        local mapping = currentlookup.mapping
1550        if mapping == nil then
1551            mapping = getmapping(dataset,sequence,currentlookup)
1552        end
1553        if mapping then
1554            local startchar = getchar(start)
1555            local kerns     = mapping[startchar]
1556            if kerns then
1557                local format = currentlookup.format
1558                if format == "single" then
1559                    local dx, dy, w, h = setposition(0,start,factor,rlmode,kerns) -- currentlookup.flags ?
1560                    if trace_kerns then
1561                        logprocess("%s: shifting single %s by %s (%p,%p) and correction (%p,%p)",cref(dataset,sequence),gref(startchar),format,dx,dy,w,h)
1562                    end
1563                else -- needs checking .. maybe no kerns format for single
1564                    local k = (format == "move" and setmove or setkern)(start,factor,rlmode,kerns,injection)
1565                    if trace_kerns then
1566                        logprocess("%s: shifting single %s by %s %p",cref(dataset,sequence),gref(startchar),format,k)
1567                    end
1568                end
1569                return head, start, true
1570            end
1571        end
1572    end
1573    return head, start, false
1574end
1575
1576function chainprocs.gpos_pair(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash,chainindex) -- todo: injections ?
1577    -- we actually should check no_left_kern_code with next
1578    if not hasglyphoption(start,no_right_kern_code) then
1579        local mapping = currentlookup.mapping
1580        if mapping == nil then
1581            mapping = getmapping(dataset,sequence,currentlookup)
1582        end
1583        if mapping then
1584            local snext = getnext(start)
1585            if snext then
1586                local startchar = getchar(start)
1587                local kerns     = mapping[startchar] -- always 1 step
1588                if kerns then
1589                    local prev = start
1590                    while snext do
1591                        local nextchar = ischar(snext,currentfont)
1592                        if not nextchar then
1593                            break
1594                        end
1595                        if skiphash and skiphash[nextchar] then
1596                            prev  = snext
1597                            snext = getnext(snext)
1598                        else
1599                            local krn = kerns[nextchar]
1600                            if not krn then
1601                                break
1602                            end
1603                            local format = currentlookup.format
1604                            if format == "pair" then
1605                                local a = krn[1]
1606                                local b = krn[2]
1607                                if a == true then
1608                                    -- zero
1609                                elseif a then
1610                                    local x, y, w, h = setposition(1,start,factor,rlmode,a,"injections") -- currentlookups flags?
1611                                    if trace_kerns then
1612                                        local startchar = getchar(start)
1613                                        logprocess("%s: shifting first of pair %s and %s by (%p,%p) and correction (%p,%p)",cref(dataset,sequence),gref(startchar),gref(nextchar),x,y,w,h)
1614                                    end
1615                                end
1616                                if b == true then
1617                                    -- zero
1618                                    start = snext -- cf spec
1619                                elseif b then -- #b > 0
1620                                    local x, y, w, h = setposition(2,snext,factor,rlmode,b,"injections")
1621                                    if trace_kerns then
1622                                        local startchar = getchar(start)
1623                                        logprocess("%s: shifting second of pair %s and %s by (%p,%p) and correction (%p,%p)",cref(dataset,sequence),gref(startchar),gref(nextchar),x,y,w,h)
1624                                    end
1625                                    start = snext -- cf spec
1626                                elseif forcepairadvance then
1627                                    start = snext -- for testing, not cf spec
1628                                end
1629                                return head, start, true
1630                            elseif krn ~= 0 then
1631                                local k = (format == "move" and setmove or setkern)(snext,factor,rlmode,krn)
1632                                if trace_kerns then
1633                                    logprocess("%s: inserting %s %p between %s and %s",cref(dataset,sequence),format,k,gref(getchar(prev)),gref(nextchar))
1634                                end
1635                                return head, start, true
1636                            else
1637                                break
1638                            end
1639                        end
1640                    end
1641                end
1642            end
1643        end
1644    end
1645    return head, start, false
1646end
1647
1648function chainprocs.gpos_mark2base(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash,chainindex)
1649    local mapping = currentlookup.mapping
1650    if mapping == nil then
1651        mapping = getmapping(dataset,sequence,currentlookup)
1652    end
1653    if mapping then
1654        local markchar = getchar(start)
1655        if marks[markchar] then
1656            local markanchors = mapping[markchar] -- always 1 step
1657            if markanchors then
1658                local base = getprev(start) -- [glyph] [start=mark]
1659                if base then
1660                    local basechar = ischar(base,currentfont)
1661                    if basechar then
1662                        if marks[basechar] then
1663                            while base do
1664                                base = getprev(base)
1665                                if base then
1666                                    local basechar = ischar(base,currentfont)
1667                                    if basechar then
1668                                        if not marks[basechar] then
1669                                            break
1670                                        end
1671                                    else
1672                                        if trace_bugs then
1673                                            logwarning("%s: no base for mark %s, case %i",pref(dataset,sequence),gref(markchar),1)
1674                                        end
1675                                        return head, start, false
1676                                    end
1677                                else
1678                                    if trace_bugs then
1679                                        logwarning("%s: no base for mark %s, case %i",pref(dataset,sequence),gref(markchar),2)
1680                                    end
1681                                    return head, start, false
1682                                end
1683                            end
1684                        end
1685                        local ba = markanchors[1][basechar]
1686                        if ba then
1687                            local ma = markanchors[2]
1688                            if ma then
1689                                local dx, dy, bound = setmark(start,base,factor,rlmode,ba,ma,characters[basechar],false,checkmarks)
1690                                if trace_marks then
1691                                    logprocess("%s, bound %s, anchoring mark %s to basechar %s => (%p,%p)",
1692                                        cref(dataset,sequence),bound,gref(markchar),gref(basechar),dx,dy)
1693                                end
1694                                return head, start, true
1695                            end
1696                        end
1697                    elseif trace_bugs then
1698                        logwarning("%s: prev node is no char, case %i",cref(dataset,sequence),1)
1699                    end
1700                elseif trace_bugs then
1701                    logwarning("%s: prev node is no char, case %i",cref(dataset,sequence),2)
1702                end
1703            elseif trace_bugs then
1704                logwarning("%s: mark %s has no anchors",cref(dataset,sequence),gref(markchar))
1705            end
1706        elseif trace_bugs then
1707            logwarning("%s: mark %s is no mark",cref(dataset,sequence),gref(markchar))
1708        end
1709    end
1710    return head, start, false
1711end
1712
1713function chainprocs.gpos_mark2ligature(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash,chainindex)
1714    local mapping = currentlookup.mapping
1715    if mapping == nil then
1716        mapping = getmapping(dataset,sequence,currentlookup)
1717    end
1718    if mapping then
1719        local markchar = getchar(start)
1720        if marks[markchar] then
1721            local markanchors = mapping[markchar] -- always 1 step
1722            if markanchors then
1723                local base = getprev(start) -- [glyph] [optional marks] [start=mark]
1724                if base then
1725                    local basechar = ischar(base,currentfont)
1726                    if basechar then
1727                        if marks[basechar] then
1728                            while base do
1729                                base = getprev(base)
1730                                if base then
1731                                    local basechar = ischar(base,currentfont)
1732                                    if basechar then
1733                                        if not marks[basechar] then
1734                                            break
1735                                        end
1736                                    else
1737                                        if trace_bugs then
1738                                            logwarning("%s: no base for mark %s, case %i",cref(dataset,sequence),markchar,1)
1739                                        end
1740                                        return head, start, false
1741                                    end
1742                                else
1743                                    if trace_bugs then
1744                                        logwarning("%s: no base for mark %s, case %i",cref(dataset,sequence),markchar,2)
1745                                    end
1746                                    return head, start, false
1747                                end
1748                            end
1749                        end
1750                        local ba = markanchors[1][basechar]
1751                        if ba then
1752                            local ma = markanchors[2]
1753                            if ma then
1754                                local index = getligaindex(start)
1755                                ba = ba[index]
1756                                if ba then
1757                                    local dx, dy, bound = setmark(start,base,factor,rlmode,ba,ma,characters[basechar],false,checkmarks)
1758                                    if trace_marks then
1759                                        logprocess("%s, bound %s, anchoring mark %s to baselig %s at index %s => (%p,%p)",
1760                                            cref(dataset,sequence),a or bound,gref(markchar),gref(basechar),index,dx,dy)
1761                                    end
1762                                    return head, start, true
1763                                end
1764                            end
1765                        end
1766                    elseif trace_bugs then
1767                        logwarning("%s, prev node is no char, case %i",cref(dataset,sequence),1)
1768                    end
1769                elseif trace_bugs then
1770                    logwarning("%s, prev node is no char, case %i",cref(dataset,sequence),2)
1771                end
1772            elseif trace_bugs then
1773                logwarning("%s, mark %s has no anchors",cref(dataset,sequence),gref(markchar))
1774            end
1775        elseif trace_bugs then
1776            logwarning("%s, mark %s is no mark",cref(dataset,sequence),gref(markchar))
1777        end
1778    end
1779    return head, start, false
1780end
1781
1782function chainprocs.gpos_mark2mark(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash,chainindex)
1783    local mapping = currentlookup.mapping
1784    if mapping == nil then
1785        mapping = getmapping(dataset,sequence,currentlookup)
1786    end
1787    if mapping then
1788        local markchar = getchar(start)
1789        if marks[markchar] then
1790            local markanchors = mapping[markchar] -- always 1 step
1791            if markanchors then
1792                local base = getprev(start) -- [glyph] [basemark] [start=mark]
1793                local slc = getligaindex(start)
1794                if slc then -- a rather messy loop ... needs checking with husayni
1795                    while base do
1796                        local blc = getligaindex(base)
1797                        if blc and blc ~= slc then
1798                            base = getprev(base)
1799                        else
1800                            break
1801                        end
1802                    end
1803                end
1804                if base then -- subtype test can go
1805                    local basechar = ischar(base,currentfont)
1806                    if basechar then
1807                        local ba = markanchors[1][basechar]
1808                        if ba then
1809                            local ma = markanchors[2]
1810                            if ma then
1811                                local dx, dy, bound = setmark(start,base,factor,rlmode,ba,ma,characters[basechar],true,checkmarks)
1812                                if trace_marks then
1813                                    logprocess("%s, bound %s, anchoring mark %s to basemark %s => (%p,%p)",
1814                                        cref(dataset,sequence),bound,gref(markchar),gref(basechar),dx,dy)
1815                                end
1816                                return head, start, true
1817                            end
1818                        end
1819                    elseif trace_bugs then
1820                        logwarning("%s: prev node is no mark, case %i",cref(dataset,sequence),1)
1821                    end
1822                elseif trace_bugs then
1823                    logwarning("%s: prev node is no mark, case %i",cref(dataset,sequence),2)
1824                end
1825            elseif trace_bugs then
1826                logwarning("%s: mark %s has no anchors",cref(dataset,sequence),gref(markchar))
1827            end
1828        elseif trace_bugs then
1829            logwarning("%s: mark %s is no mark",cref(dataset,sequence),gref(markchar))
1830        end
1831    end
1832    return head, start, false
1833end
1834
1835function chainprocs.gpos_cursive(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash,chainindex)
1836    local mapping = currentlookup.mapping
1837    if mapping == nil then
1838        mapping = getmapping(dataset,sequence,currentlookup)
1839    end
1840    if mapping then
1841        local startchar   = getchar(start)
1842        local exitanchors = mapping[startchar] -- always 1 step
1843        if exitanchors then
1844            if marks[startchar] then
1845                if trace_cursive then
1846                    logprocess("%s: ignoring cursive for mark %s",pref(dataset,sequence),gref(startchar))
1847                end
1848            else
1849                local nxt = getnext(start)
1850                while nxt do
1851                    local nextchar = ischar(nxt,currentfont)
1852                    if not nextchar then
1853                        break
1854                    elseif marks[nextchar] then
1855                        -- should not happen (maybe warning)
1856                        nxt = getnext(nxt)
1857                    else
1858                        local exit = exitanchors[3]
1859                        if exit then
1860                            local entry = exitanchors[1][nextchar]
1861                            if entry then
1862                                entry = entry[2]
1863                                if entry then
1864                                    local r2lflag = sequence.flags[4] -- mentioned in the standard
1865                                    local dx, dy, bound = setcursive(start,nxt,factor,rlmode,exit,entry,characters[startchar],characters[nextchar],r2lflag)
1866                                    if trace_cursive then
1867                                        logprocess("%s: moving %s to %s cursive (%p,%p) using bound %s in %s mode",pref(dataset,sequence),gref(startchar),gref(nextchar),dx,dy,bound,mref(rlmode))
1868                                    end
1869                                    return head, start, true
1870                                end
1871                            end
1872                        elseif trace_bugs then
1873                            onetimemessage(currentfont,startchar,"no entry anchors")
1874                        end
1875                        break
1876                    end
1877                end
1878            end
1879        elseif trace_cursive and trace_details then
1880            logprocess("%s, cursive %s is already done",pref(dataset,sequence),gref(getchar(start)),alreadydone)
1881        end
1882    end
1883    return head, start, false
1884end
1885
1886-- what pointer to return, spec says stop
1887-- to be discussed ... is bidi changer a space?
1888-- elseif char == zwnj and sequence[n][32] then -- brrr
1889
1890local function show_skip(dataset,sequence,char,ck,class)
1891    logwarning("%s: skipping char %s, class %a, rule %a, lookuptype %a",cref(dataset,sequence),gref(char),class,ck[1],ck[8] or ck[2])
1892end
1893
1894-- A previous version had disc collapsing code in the (single sub) handler plus some
1895-- checking in the main loop, but that left the pre/post sequences undone. The best
1896-- solution is to add some checking there and backtrack when a replace/post matches
1897-- but it takes a bit of work to figure out an efficient way (this is what the
1898-- sweep* names refer to). I might look into that variant one day again as it can
1899-- replace some other code too. In that approach we can have a special version for
1900-- gub and pos which gains some speed. This method does the test and passes info to
1901-- the handlers. Here collapsing is handled in the main loop which also makes code
1902-- elsewhere simpler (i.e. no need for the other special runners and disc code in
1903-- ligature building). I also experimented with pushing preceding glyphs sequences
1904-- in the replace/pre fields beforehand which saves checking afterwards but at the
1905-- cost of duplicate glyphs (memory) but it's too much overhead (runtime).
1906--
1907-- In the meantime Kai had moved the code from the single chain into a more general
1908-- handler and this one (renamed to chaindisk) is used now. I optimized the code a
1909-- bit and brought it in sycn with the other code. Hopefully I didn't introduce
1910-- errors. Note: this somewhat complex approach is meant for fonts that implement
1911-- (for instance) ligatures by character replacement which to some extend is not
1912-- that suitable for hyphenation. I also use some helpers. This method passes some
1913-- states but reparses the list. There is room for a bit of speed up but that will
1914-- be done in the context version. (In fact a partial rewrite of all code can bring
1915-- some more efficiency.)
1916--
1917-- I didn't test it with extremes but successive disc nodes still can give issues
1918-- but in order to handle that we need more complex code which also slows down even
1919-- more. The main loop variant could deal with that: test, collapse, backtrack.
1920
1921local userkern = nuts.pool and nuts.pool.newkern -- context
1922
1923do if not userkern then -- generic
1924
1925    local thekern = nuts.new("kern",1) -- userkern
1926    local setkern = nuts.setkern       -- not injections.setkern
1927
1928    userkern = function(k)
1929        local n = copynode(thekern)
1930        setkern(n,k)
1931        return n
1932    end
1933
1934end end
1935
1936local function checked(head)
1937    local current = head
1938    while current do
1939        if getid(current) == glue_code then
1940            local kern = userkern(getwidth(current))
1941            if head == current then
1942                local next = getnext(current)
1943                if next then
1944                    setlink(kern,next)
1945                end
1946                flushnode(current)
1947                head    = kern
1948                current = next
1949            else
1950                local prev, next = getboth(current)
1951                setlink(prev,kern,next)
1952                flushnode(current)
1953                current = next
1954            end
1955        else
1956            current = getnext(current)
1957        end
1958    end
1959    return head
1960end
1961
1962local function setdiscchecked(d,pre,post,replace)
1963    if pre     then pre     = checked(pre)     end
1964    if post    then post    = checked(post)    end
1965    if replace then replace = checked(replace) end
1966    setdisc(d,pre,post,replace)
1967end
1968
1969local noflags = { false, false, false, false }
1970
1971local function chainrun(head,start,last,dataset,sequence,rlmode,skiphash,ck,where)
1972
1973    local size         = ck[5] - ck[4] + 1
1974    local chainlookups = ck[6]
1975    local done         = false
1976    -- current match
1977    if chainlookups then
1978        -- Lookups can be like { 1, false, 3 } or { false, 2 } or basically anything and
1979        -- #lookups can be less than #current
1980
1981        if size == 1 then
1982
1983         -- if nofchainlookups > size then
1984         --     -- bad rules
1985         -- end
1986            local chainlookup = chainlookups[1]
1987            if chainlookup then
1988                for j=1,#chainlookup do
1989                    local chainstep = chainlookup[j]
1990                    if chainstep then
1991                        local chainkind = chainstep.type
1992                        local chainproc = chainprocs[chainkind]
1993                        if chainproc then
1994                            local ok
1995                         -- HH: chainindex 1 added here (for KAI to check too), there are weird ligatures e.g.
1996                         -- char + mark -> char where mark has to disappear
1997                            head, start, ok = chainproc(head,start,last,dataset,sequence,chainstep,rlmode,skiphash,1)
1998                            if ok then
1999                                done = true
2000                            end
2001                        else
2002                            logprocess("%s: %s is not yet supported (1)",cref(dataset,sequence),chainkind)
2003                        end
2004                    else
2005                        logprocess("%s: has an issue (1)",cref(dataset,sequence))
2006                    end
2007                end
2008            else
2009                -- whatever
2010            end
2011
2012         else
2013
2014            -- See LookupType 5: Contextual Substitution Subtable. Now it becomes messy. The
2015            -- easiest case is where #current maps on #lookups i.e. one-to-one. But what if
2016            -- we have a ligature. Cf the spec we then need to advance one character but we
2017            -- really need to test it as there are fonts out there that are fuzzy and have
2018            -- too many lookups:
2019            --
2020            -- U+1105 U+119E U+1105 U+119E : sourcehansansklight: script=hang ccmp=yes
2021            --
2022            -- Even worse are these family emoji shapes as they can have multiple lookups
2023            -- per slot (probably only for gpos).
2024
2025            -- It's very unlikely that we will have skip classes here but still ... we seldom
2026            -- enter this branch anyway.
2027
2028            local i = 1
2029            local laststart = start
2030            local nofchainlookups = #chainlookups -- useful?
2031            while start do
2032                if skiphash then -- hm, so we know we skip some
2033                    while start do
2034                        local char = ischar(start,currentfont)
2035                        if char then
2036                            if skiphash and skiphash[char] then
2037                                start = getnext(start)
2038                            else
2039                                break
2040                            end
2041                        else
2042                            break
2043                        end
2044                    end
2045                end
2046                local chainlookup = chainlookups[i]
2047                if chainlookup then
2048                    for j=1,#chainlookup do
2049                        local chainstep = chainlookup[j]
2050                        if chainstep then
2051                            local chainkind = chainstep.type
2052                            local chainproc = chainprocs[chainkind]
2053                            if chainproc then
2054                                local ok, n
2055                                head, start, ok, n = chainproc(head,start,last,dataset,sequence,chainstep,rlmode,skiphash,i)
2056                                -- messy since last can be changed !
2057                                if ok then
2058                                    done = true
2059                                    if n and n > 1 and i + n > nofchainlookups then
2060                                        -- this is a safeguard, we just ignore the rest of the lookups
2061                                        i = size -- prevents an advance
2062                                        break
2063                                    end
2064                                end
2065                            else
2066                                -- actually an error
2067                                logprocess("%s: %s is not yet supported (2)",cref(dataset,sequence),chainkind)
2068                            end
2069                        else
2070                            -- actually an error
2071                            logprocess("%s: has an issue (2)",cref(dataset,sequence))
2072                        end
2073                    end
2074                else
2075                    -- we skip but we could also delete as option .. what does an empty lookup actually mean
2076                    -- in opentype ... anyway, we could map it onto gsub_remove if needed
2077                end
2078                i = i + 1
2079                if i > size or not start then
2080                    break
2081                elseif start then
2082                    laststart = start
2083                    start = getnext(start)
2084                end
2085            end
2086            if not start then
2087                start = laststart
2088            end
2089
2090        end
2091    else
2092        -- todo: needs checking for holes in the replacements
2093        local replacements = ck[7]
2094        if replacements then
2095            head, start, done = reversesub(head,start,last,dataset,sequence,replacements,rlmode,skiphash)
2096        else
2097            done = true
2098            if trace_contexts then
2099                logprocess("%s: skipping match @ %i",cref(dataset,sequence),where)
2100            end
2101        end
2102    end
2103    return head, start, done
2104end
2105
2106local function chaindisk(head,start,dataset,sequence,rlmode,skiphash,ck)
2107
2108    if not start then
2109        return head, start, false
2110    end
2111
2112    local startishead   = start == head
2113    local seq           = ck[3]
2114    local f             = ck[4]
2115    local l             = ck[5]
2116    local s             = #seq
2117    local done          = false
2118    local sweepnode     = sweepnode
2119    local sweeptype     = sweeptype
2120    local sweepoverflow = false
2121    local checkdisc     = getprev(head)
2122    local keepdisc      = not sweepnode
2123    local lookaheaddisc = nil
2124    local backtrackdisc = nil
2125    local current       = start
2126    local last          = start
2127    local prev          = getprev(start)
2128    local hasglue       = false
2129    local useddisc      = nil   -- new 2022-09-25
2130    local usedstart     = start -- new 2022-09-25
2131
2132    -- fishy: so we can overflow and then go on in the sweep?
2133    -- todo : id can also be glue_code as we checked spaces
2134
2135    local i = f
2136    while i <= l do
2137        local id = getid(current)
2138        if id == glyph_code then
2139            i       = i + 1
2140            last    = current
2141            current = getnext(current)
2142        elseif id == glue_code then
2143            i       = i + 1
2144            last    = current
2145            current = getnext(current)
2146            hasglue = true
2147        elseif id == disc_code then
2148            if keepdisc then
2149                keepdisc      = false
2150                lookaheaddisc = current
2151                local replace = getreplace(current)
2152                if not replace then
2153                    sweepoverflow = true
2154                    sweepnode     = current
2155                    current       = getnext(current)
2156                else
2157                    -- we can use an iterator
2158                    while replace and i <= l do
2159                        if getid(replace) == glyph_code then
2160                            i = i + 1
2161                        end
2162                        replace = getnext(replace)
2163                    end
2164                    current = getnext(replace)
2165                end
2166                last = current
2167            else
2168                head, current = flattendisk(head,current)
2169            end
2170        else
2171            last    = current
2172            current = getnext(current)
2173        end
2174        if current then
2175            -- go on
2176        elseif sweepoverflow then
2177            -- we already are following up on sweepnode
2178            break
2179        elseif sweeptype == "post" or sweeptype == "replace" then
2180            current = getnext(sweepnode)
2181            if current then
2182                sweeptype     = nil
2183                sweepoverflow = true
2184            else
2185                break
2186            end
2187        else
2188            break -- added
2189        end
2190    end
2191
2192    if sweepoverflow then
2193        local prev = current and getprev(current)
2194        if not current or prev ~= sweepnode then
2195            local head = getnext(sweepnode)
2196            local tail = nil
2197            if prev then
2198                tail = prev
2199                setprev(current,sweepnode)
2200            else
2201                tail = findnodetail(head)
2202            end
2203            setnext(sweepnode,current)
2204            setprev(head)
2205            setnext(tail)
2206            appenddisc(sweepnode,head)
2207        end
2208    end
2209
2210    if l < s then
2211        local i = l
2212        local t = sweeptype == "post" or sweeptype == "replace"
2213        while current and i < s do
2214            local id = getid(current)
2215            if id == glyph_code then
2216                i       = i + 1
2217                current = getnext(current)
2218            elseif id == glue_code then
2219                i       = i + 1
2220                current = getnext(current)
2221                hasglue = true
2222            elseif id == disc_code then
2223                if keepdisc then
2224                    keepdisc = false
2225                    if notmatchpre[current] ~= notmatchreplace[current] then
2226                        lookaheaddisc = current
2227                    end
2228                    -- we assume a simple text only replace (we could use nuts.count)
2229                    local replace = getreplace(current)
2230                    while replace and i < s do
2231                        if getid(replace) == glyph_code then
2232                            i = i + 1
2233                        end
2234                        replace = getnext(replace)
2235                    end
2236                    current = getnext(current)
2237                elseif notmatchpre[current] ~= notmatchreplace[current] then
2238                    head, current = flattendisk(head,current)
2239                else
2240                    current = getnext(current) -- HH
2241                end
2242            else
2243                current = getnext(current)
2244            end
2245            if not current and t then
2246                current = getnext(sweepnode)
2247                if current then
2248                    sweeptype = nil
2249                end
2250            end
2251        end
2252    end
2253
2254    if f > 1 then
2255        local current = prev
2256        local i       = f
2257        local t       = sweeptype == "pre" or sweeptype == "replace"
2258        if not current and t and current == checkdisc then
2259            current = getprev(sweepnode)
2260        end
2261        while current and i > 1 do -- missing getprev added / moved outside
2262            local id = getid(current)
2263            if id == glyph_code then
2264                i = i - 1
2265            elseif id == glue_code then
2266                i       = i - 1
2267                hasglue = true
2268            elseif id == disc_code then
2269                if keepdisc then
2270                    keepdisc = false
2271                    if notmatchpost[current] ~= notmatchreplace[current] then
2272                        backtrackdisc = current
2273                    end
2274                    -- we assume a simple text only replace (we could use nuts.count)
2275                    local replace = getreplace(current)
2276                    while replace and i > 1 do
2277                        if getid(replace) == glyph_code then
2278                            i = i - 1
2279                        end
2280                        replace = getnext(replace)
2281                    end
2282                elseif notmatchpost[current] ~= notmatchreplace[current] then
2283                    head, current = flattendisk(head,current)
2284                end
2285            end
2286            current = getprev(current)
2287            if t and current == checkdisc then
2288                current = getprev(sweepnode)
2289            end
2290        end
2291    end
2292
2293    local done = false
2294
2295    if lookaheaddisc then
2296        local cf            = start
2297        local cl            = getprev(lookaheaddisc)
2298        local cprev         = getprev(start)
2299        local insertedmarks = 0
2300        while cprev do
2301            local char = ischar(cf,currentfont)
2302            if char and marks[char] then
2303                insertedmarks = insertedmarks + 1
2304                cf            = cprev
2305                startishead   = cf == head
2306                cprev         = getprev(cprev)
2307            else
2308                break
2309            end
2310        end
2311        setlink(cprev,lookaheaddisc)
2312        setprev(cf)
2313        setnext(cl)
2314        if startishead then
2315            head = lookaheaddisc
2316        end
2317        local pre, post, replace = getdisc(lookaheaddisc)
2318        local new  = copynodelist(cf) -- br, how often does that happen
2319        local cnew = new
2320        if pre then
2321            setlink(findnodetail(cf),pre)
2322        end
2323        if replace then
2324            local tail = findnodetail(new)
2325            setlink(tail,replace)
2326        end
2327        for i=1,insertedmarks do
2328            cnew = getnext(cnew)
2329        end
2330        cl = start
2331        local clast = cnew
2332        for i=f,l do
2333            cl    = getnext(cl)
2334            clast = getnext(clast)
2335        end
2336        if not notmatchpre[lookaheaddisc] then
2337            local ok = false
2338            cf, start, ok = chainrun(cf,start,cl,dataset,sequence,rlmode,skiphash,ck,1)
2339            if ok then
2340                done = true
2341            end
2342        end
2343        if not notmatchreplace[lookaheaddisc] then
2344            local ok = false
2345            new, cnew, ok = chainrun(new,cnew,clast,dataset,sequence,rlmode,skiphash,ck,2)
2346            if ok then
2347                done = true
2348            end
2349        end
2350        if hasglue then
2351            setdiscchecked(lookaheaddisc,cf,post,new)
2352        else
2353            setdisc(lookaheaddisc,cf,post,new)
2354        end
2355        start          = getprev(lookaheaddisc)
2356        useddisc       = lookaheaddisc -- new 2022-09-25
2357        sweephead[cf]  = getnext(clast) or false
2358        sweephead[new] = getnext(cl) or false
2359    elseif backtrackdisc then
2360        local cf            = getnext(backtrackdisc)
2361        local cl            = start
2362        local cnext         = getnext(start)
2363        local insertedmarks = 0
2364        while cnext do
2365            local char = ischar(cnext,currentfont)
2366            if char and marks[char] then
2367                insertedmarks = insertedmarks + 1
2368                cl            = cnext
2369                cnext         = getnext(cnext)
2370            else
2371                break
2372            end
2373        end
2374        setlink(backtrackdisc,cnext)
2375        setprev(cf)
2376        setnext(cl)
2377        local pre, post, replace, pretail, posttail, replacetail = getdisc(backtrackdisc,true)
2378        local new  = copynodelist(cf)
2379        local cnew = findnodetail(new)
2380        for i=1,insertedmarks do
2381            cnew = getprev(cnew)
2382        end
2383        local clast = cnew
2384        for i=f,l do
2385            clast = getnext(clast)
2386        end
2387        if not notmatchpost[backtrackdisc] then
2388            local ok = false
2389            cf, start, ok = chainrun(cf,start,last,dataset,sequence,rlmode,skiphash,ck,3)
2390            if ok then
2391                done = true
2392            end
2393        end
2394        if not notmatchreplace[backtrackdisc] then
2395            local ok = false
2396            new, cnew, ok = chainrun(new,cnew,clast,dataset,sequence,rlmode,skiphash,ck,4)
2397            if ok then
2398                done = true
2399            end
2400        end
2401        if post then
2402            setlink(posttail,cf)
2403        else
2404            post = cf
2405        end
2406        if replace then
2407            setlink(replacetail,new)
2408        else
2409            replace = new
2410        end
2411        if hasglue then
2412            setdiscchecked(backtrackdisc,pre,post,replace)
2413        else
2414            setdisc(backtrackdisc,pre,post,replace)
2415        end
2416        start              = getprev(backtrackdisc)
2417        useddisc           = backtrackdisc -- new 2022-09-25
2418        sweephead[post]    = getnext(clast) or false
2419        sweephead[replace] = getnext(last) or false
2420    else
2421
2422        local ok = false
2423        head, start, ok = chainrun(head,start,last,dataset,sequence,rlmode,skiphash,ck,5)
2424        if ok then
2425            done = true
2426        end
2427
2428    end
2429    if useddisc and start ~= usedstart then -- make this option per font -- new 2022-09-25
2430       start = getnext(start)                                            -- new 2022-09-25
2431    end                                                                  -- new 2022-09-25
2432    return head, start, done, useddisc                                   -- new 2022-09-25
2433end
2434
2435local chaintrac do
2436
2437    local level = 0
2438    local last  = { }
2439
2440    chaintrac = function(head,start,dataset,sequence,rlmode,skiphash,ck,match,discseen,sweepnode)
2441        if dataset then
2442            level = level + 1
2443            last[level] = start
2444            local rule       = ck[1]
2445            local lookuptype = ck[8] or ck[2]
2446            local nofseq     = #ck[3] -- ck[3].n
2447            local first      = ck[4]
2448            local last       = ck[5]
2449            local char       = getchar(start)
2450            logwarning("+ %i : %s: rule %s %s at char %s for (%s,%s,%s) chars, lookuptype %a, %sdisc seen, %ssweeping",
2451                level,cref(dataset,sequence),rule,match and "matches" or "nomatch",
2452                gref(char),first-1,last-first+1,nofseq-last,lookuptype,
2453                discseen and "" or "no ", sweepnode and "" or "not ")
2454        else
2455            -- (start,done)
2456            local what  = start and "done" or "continue"
2457            local where = head == last[level] and "same" or "different"
2458            local char  = getchar(head)
2459            if char then
2460                logwarning("- %i : %s at char %s, %s node",level,what,gref(char),where)
2461            else
2462                logwarning("- %i : %s, %s node",level,what,where)
2463            end
2464            level = level - 1
2465        end
2466    end
2467
2468end
2469
2470-- The next one is quite optimized but still somewhat slow, fonts like ebgaramond
2471-- are real torture tests because they have many steps with one context (having
2472-- multiple contexts makes more sense) also because we (can) reduce them. Instead of
2473-- a match boolean variable and check for that I decided to use a goto with labels
2474-- instead. This is one of the cases where it makes the code more readable and we
2475-- might even gain a bit performance.
2476
2477-- when we have less replacements (lookups) then current matches we can push too much into
2478-- the previous disc .. such be it (<before><disc><current=fl><after> with only f done)
2479
2480local function handle_contextchain(head,start,dataset,sequence,contexts,rlmode,skiphash)
2481    if not contexts then
2482        return head, start, false
2483    end
2484    -- optimizing for rlmode gains nothing
2485    local sweepnode    = sweepnode
2486    local sweeptype    = sweeptype
2487    local postreplace
2488    local prereplace
2489    local checkdisc
2490    local discseen  -- = false
2491    if sweeptype then
2492        if sweeptype == "replace" then
2493            postreplace = true
2494            prereplace  = true
2495        else
2496            postreplace = sweeptype == "post"
2497            prereplace  = sweeptype == "pre"
2498        end
2499        checkdisc = getprev(head)
2500    end
2501    local currentfont  = currentfont
2502
2503    local skipped   -- = false
2504
2505    local startprev,
2506          startnext    = getboth(start)
2507    local done      -- = false
2508
2509    -- we can have multiple hits and as we scan (currently) all we need to check
2510    -- if we have a match ... contextchains have no real coverage table (with
2511    -- unique entries)
2512
2513    -- fonts can have many steps (each doing one check) or many contexts
2514
2515    -- todo: make a per-char cache so that we have small contexts (when we have a context
2516    -- n == 1 and otherwise it can be more so we can even distinguish n == 1 or more)
2517
2518    local nofcontexts = contexts.n -- #contexts
2519
2520    local startchar = nofcontext == 1 or ischar(start,currentfont) -- only needed in a chain
2521
2522    for k=1,nofcontexts do -- does this disc mess work well with n > 1
2523
2524        local ck   = contexts[k]
2525        local seq  = ck[3]
2526        local f    = ck[4] -- first current
2527        local last = start
2528        if not startchar or not seq[f][startchar] then
2529            -- report("no hit in %a at %i of %i contexts",sequence.type,k,nofcontexts)
2530            goto next
2531        end
2532        local s = seq.n -- or #seq
2533        if s == 1 then
2534            -- bit weird case: why use a chain, but it is a hit
2535        else
2536            local l       = ck[5] -- last current
2537            local current = start
2538         -- local last    = start
2539
2540            -- current match
2541
2542            if l > f then
2543                -- before/current/after | before/current | current/after
2544                local discfound -- = nil
2545                local n = f + 1
2546                last = startnext -- the second in current (first already matched)
2547                while n <= l do
2548                    if postreplace and not last then
2549                        last      = getnext(sweepnode)
2550                        sweeptype = nil
2551                    end
2552                    if last then
2553                        local char, id = ischar(last,currentfont)
2554                        if char then
2555                            if seq[n][char] then
2556                                if n < l then
2557                                    last = getnext(last)
2558                                end
2559                                n = n + 1
2560                            elseif skiphash and skiphash[char] then
2561                                skipped = true
2562                                if trace_skips then
2563                                    show_skip(dataset,sequence,char,ck,classes[char])
2564                                end
2565                                last = getnext(last)
2566                            elseif discfound then
2567                                notmatchreplace[discfound] = true
2568                                if notmatchpre[discfound] then
2569                                    goto next
2570                                else
2571                                    break
2572                                end
2573                            else
2574                                goto next
2575                            end
2576                        elseif char == false then
2577                            if discfound then
2578                                notmatchreplace[discfound] = true
2579                                if notmatchpre[discfound] then
2580                                    goto next
2581                                else
2582                                    break
2583                                end
2584                            else
2585                                goto next
2586                            end
2587                        elseif id == disc_code then
2588                     -- elseif id == disc_code and (not discs or discs[last]) then
2589                            discseen              = true
2590                            discfound             = last
2591                            notmatchpre[last]     = nil
2592                            notmatchpost[last]    = true
2593                            notmatchreplace[last] = nil
2594                            local pre, post, replace = getdisc(last)
2595                            if pre then
2596                                local n = n
2597                                while pre do
2598                                    if seq[n][getchar(pre)] then
2599                                        n = n + 1
2600                                        if n > l then
2601                                            break
2602                                        end
2603                                        pre = getnext(pre)
2604                                    else
2605                                        notmatchpre[last] = true
2606                                        break
2607                                    end
2608                                end
2609                                if n <= l then
2610                                    notmatchpre[last] = true
2611                                end
2612                            else
2613                                notmatchpre[last] = true
2614                            end
2615                            if replace then
2616                                -- so far we never entered this branch
2617                                while replace do
2618                                    if seq[n][getchar(replace)] then
2619                                        n = n + 1
2620                                        if n > l then
2621                                            break
2622                                        end
2623                                        replace = getnext(replace)
2624                                    else
2625                                        notmatchreplace[last] = true
2626                                        if notmatchpre[last] then
2627                                            goto next
2628                                        else
2629                                            break
2630                                        end
2631                                    end
2632                                end
2633                                -- why here again
2634                                if notmatchpre[last] then
2635                                    goto next
2636                                end
2637                            end
2638                            -- maybe only if match
2639                            last = getnext(last)
2640                        else
2641                            goto next
2642                        end
2643                    else
2644                        goto next
2645                    end
2646                end
2647            end
2648
2649            -- before
2650
2651            if f > 1 then
2652              -- if startprev then -- new 2022-09-25
2653                    local prev = startprev
2654                    if prereplace and prev == checkdisc then
2655                        prev = getprev(sweepnode)
2656                    end
2657                    if prev then
2658                        local discfound -- = nil
2659                        local n = f - 1
2660                        while n >= 1 do
2661                            if prev then
2662                                local char, id = ischar(prev,currentfont)
2663                                if char then
2664                                    if seq[n][char] then
2665                                        if n > 1 then
2666                                            prev = getprev(prev)
2667                                        end
2668                                        n = n - 1
2669                                    elseif skiphash and skiphash[char] then
2670                                        skipped = true
2671                                        if trace_skips then
2672                                            show_skip(dataset,sequence,char,ck,classes[char])
2673                                        end
2674                                        prev = getprev(prev)
2675                                    elseif discfound then
2676                                        notmatchreplace[discfound] = true
2677                                        if notmatchpost[discfound] then
2678                                            goto next
2679                                        else
2680                                            break
2681                                        end
2682                                    else
2683                                        goto next
2684                                    end
2685                                elseif char == false then
2686                                    if discfound then
2687                                        notmatchreplace[discfound] = true
2688                                        if notmatchpost[discfound] then
2689                                            goto next
2690                                        end
2691                                    else
2692                                        goto next
2693                                    end
2694                                    break
2695                                elseif id == disc_code then
2696                             -- elseif id == disc_code and (not discs or discs[prev]) then
2697                                    -- the special case: f i where i becomes dottless i ..
2698                                    discseen              = true
2699                                    discfound             = prev
2700                                    notmatchpre[prev]     = true
2701                                    notmatchpost[prev]    = nil
2702                                    notmatchreplace[prev] = nil
2703                                    local pre, post, replace, pretail, posttail, replacetail = getdisc(prev,true)
2704                                    -- weird test: needs checking
2705                                    if pre ~= start and post ~= start and replace ~= start then
2706                                        if post then
2707                                            local n = n
2708                                            while posttail do
2709                                                if seq[n][getchar(posttail)] then
2710                                                    n = n - 1
2711                                                    if posttail == post or n < 1 then
2712                                                        break
2713                                                    else
2714                                                        posttail = getprev(posttail)
2715                                                    end
2716                                                else
2717                                                    notmatchpost[prev] = true
2718                                                    break
2719                                                end
2720                                            end
2721                                            if n >= 1 then
2722                                                notmatchpost[prev] = true
2723                                            end
2724                                        else
2725                                            notmatchpost[prev] = true
2726                                        end
2727                                        if replace then
2728                                            -- we seldom enter this branch (e.g. on brill efficient)
2729                                            while replacetail do
2730                                                if seq[n][getchar(replacetail)] then
2731                                                    n = n - 1
2732                                                    if replacetail == replace or n < 1 then
2733                                                        break
2734                                                    else
2735                                                        replacetail = getprev(replacetail)
2736                                                    end
2737                                                else
2738                                                    notmatchreplace[prev] = true
2739                                                    if notmatchpost[prev] then
2740                                                        goto next
2741                                                    else
2742                                                        break
2743                                                    end
2744                                                end
2745                                            end
2746                                        else
2747                                         -- notmatchreplace[prev] = true -- not according to Kai
2748                                        end
2749                                    end
2750                                    prev = getprev(prev)
2751                             -- elseif id == glue_code and seq[n][32] and isspace(prev,threshold,id) then
2752                             -- elseif seq[n][32] and spaces[prev] then
2753                             --     n = n - 1
2754                             --     prev = getprev(prev)
2755                                elseif id == glue_code then
2756                                    local sn = seq[n]
2757                                    if (sn[32] and spaces[prev]) or sn[0xFFFC] then
2758                                        n = n - 1
2759                                        prev = getprev(prev)
2760                                    else
2761                                        goto next
2762                                    end
2763                                elseif seq[n][0xFFFC] then
2764                                    n = n - 1
2765                                    prev = getprev(prev)
2766                                else
2767                                    goto next
2768                                end
2769                            else
2770                                goto next
2771                            end
2772                        end
2773                    else
2774                        goto next
2775                    end
2776             -- else          -- new 2022-09-25
2777             --     goto next -- new 2022-09-25
2778             -- end           -- new 2022-09-25
2779            end
2780
2781            -- after
2782
2783            if s > l then
2784                local current = last and getnext(last)
2785                if not current and postreplace then
2786                    current = getnext(sweepnode)
2787                end
2788                if current then
2789                    local discfound -- = nil
2790                    local n = l + 1
2791                    while n <= s do
2792                        if current then
2793                            local char, id = ischar(current,currentfont)
2794                            if char then
2795                                if seq[n][char] then
2796                                    if n < s then
2797                                        current = getnext(current)
2798                                    end
2799                                    n = n + 1
2800                                elseif skiphash and skiphash[char] then
2801                                    skipped = true
2802                                    if trace_skips then
2803                                        show_skip(dataset,sequence,char,ck,classes[char])
2804                                    end
2805                                    current = getnext(current)
2806                                elseif discfound then
2807                                    notmatchreplace[discfound] = true
2808                                    if notmatchpre[discfound] then
2809                                        goto next
2810                                    else
2811                                        break
2812                                    end
2813                                else
2814                                    goto next
2815                                end
2816                            elseif char == false then
2817                                if discfound then
2818                                    notmatchreplace[discfound] = true
2819                                    if notmatchpre[discfound] then
2820                                        goto next
2821                                    else
2822                                        break
2823                                    end
2824                                else
2825                                    goto next
2826                                end
2827                            elseif id == disc_code then
2828                         -- elseif id == disc_code and (not discs or discs[current]) then
2829                                discseen                 = true
2830                                discfound                = current
2831                                notmatchpre[current]     = nil
2832                                notmatchpost[current]    = true
2833                                notmatchreplace[current] = nil
2834                                local pre, post, replace = getdisc(current)
2835                                if pre then
2836                                    local n = n
2837                                    while pre do
2838                                        if seq[n][getchar(pre)] then
2839                                            n = n + 1
2840                                            if n > s then
2841                                                break
2842                                            else
2843                                                pre = getnext(pre)
2844                                            end
2845                                        else
2846                                            notmatchpre[current] = true
2847                                            break
2848                                        end
2849                                    end
2850                                    if n <= s then
2851                                        notmatchpre[current] = true
2852                                    end
2853                                else
2854                                    notmatchpre[current] = true
2855                                end
2856                                if replace then
2857                                    -- so far we never entered this branch
2858                                    while replace do
2859                                        if seq[n][getchar(replace)] then
2860                                            n = n + 1
2861                                            if n > s then
2862                                                break
2863                                            else
2864                                                replace = getnext(replace)
2865                                            end
2866                                        else
2867                                            notmatchreplace[current] = true
2868                                            if notmatchpre[current] then
2869                                                goto next
2870                                            else
2871                                                break
2872                                            end
2873                                        end
2874                                    end
2875                                else
2876                                 -- notmatchreplace[current] = true -- not according to Kai
2877                                end
2878                                current = getnext(current)
2879                            elseif id == glue_code then
2880                                local sn = seq[n]
2881                                if (sn[32] and spaces[current]) or sn[0xFFFC] then
2882                                    n = n + 1
2883                                    current = getnext(current)
2884                                else
2885                                    goto next
2886                                end
2887                            elseif seq[n][0xFFFC] then
2888                                n = n + 1
2889                                current = getnext(current)
2890                            else
2891                                goto next
2892                            end
2893                        else
2894                            goto next
2895                        end
2896                    end
2897                else
2898                    goto next
2899                end
2900            end
2901        end
2902        if trace_contexts then
2903            chaintrac(head,start,dataset,sequence,rlmode,skipped and skiphash,ck,true,discseen,sweepnode)
2904        end
2905        if discseen or sweepnode then
2906            -- When we process a disc we can collapse and therefore we backtrack one node (start) and
2907            -- reprocess. This is needed because there might be more in the collapsed list.
2908            head, start, done = chaindisk(head,start,dataset,sequence,rlmode,skipped and skiphash,ck)
2909        else
2910            head, start, done = chainrun(head,start,last,dataset,sequence,rlmode,skipped and skiphash,ck,6)
2911        end
2912        if trace_contexts then
2913            chaintrac(start,done)
2914        end
2915        if done then
2916            break
2917     -- else
2918            -- next context
2919        end
2920      ::next::
2921    end
2922    if discseen then
2923        notmatchpre     = { }
2924        notmatchpost    = { }
2925        notmatchreplace = { }
2926     -- notmatchpre     = { a = 1, b = 1 }  notmatchpre    .a = nil notmatchpre    .b = nil
2927     -- notmatchpost    = { a = 1, b = 1 }  notmatchpost   .a = nil notmatchpost   .b = nil
2928     -- notmatchreplace = { a = 1, b = 1 }  notmatchreplace.a = nil notmatchreplace.b = nil
2929    end
2930    return head, start, done
2931end
2932
2933handlers.gsub_context             = handle_contextchain
2934handlers.gsub_contextchain        = handle_contextchain
2935handlers.gsub_reversecontextchain = handle_contextchain
2936handlers.gpos_contextchain        = handle_contextchain
2937handlers.gpos_context             = handle_contextchain
2938
2939-- local function chained_contextchain(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash)
2940--     local steps    = currentlookup.steps
2941--     local nofsteps = currentlookup.nofsteps
2942--     if nofsteps > 1 then
2943--         reportmoresteps(dataset,sequence)
2944--     end
2945--     -- probably wrong
2946--     local l = steps[1].coverage[getchar(start)]
2947--     if l then
2948--         return handle_contextchain(head,start,dataset,sequence,l,rlmode,skiphash)
2949--     else
2950--         return head, start, false
2951--     end
2952-- end
2953
2954-- new 2022-09-25
2955
2956local function chained_contextchain(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash)
2957    local steps    = currentlookup.steps
2958    local nofsteps = currentlookup.nofsteps
2959    local char     = getchar(start)
2960    if nofsteps == 1 then
2961        local s = steps[1]
2962        local l = s.coverage[char]
2963        if l then
2964            return handle_contextchain(head,start,dataset,sequence,l,rlmode,skiphash)
2965        end
2966    else
2967        for i=1,nofsteps do
2968            local s = steps[i]
2969            local l = s.coverage[char]
2970            if l then
2971                local h, s, d = handle_contextchain(head,start,dataset,sequence,l,rlmode,skiphash)
2972                if d then
2973                    return h, s, d
2974                end
2975            end
2976        end
2977    end
2978    return head, start, false
2979end
2980
2981chainprocs.gsub_context             = chained_contextchain
2982chainprocs.gsub_contextchain        = chained_contextchain
2983chainprocs.gsub_reversecontextchain = chained_contextchain
2984chainprocs.gpos_contextchain        = chained_contextchain
2985chainprocs.gpos_context             = chained_contextchain
2986
2987------------------------------
2988
2989-- experiment (needs no handler in font-otc so not now):
2990--
2991-- function otf.registerchainproc(name,f)
2992--  -- chainprocs[name] = f
2993--     chainprocs[name] = function(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash)
2994--         local done = currentlookup.nofsteps > 0
2995--         if not done then
2996--             reportzerosteps(dataset,sequence)
2997--         else
2998--             head, start, done = f(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash)
2999--             if not head or not start then
3000--                 reportbadsteps(dataset,sequence)
3001--             end
3002--         end
3003--         return head, start, done
3004--     end
3005-- end
3006
3007local missing    = setmetatableindex("table")
3008local logwarning = report_process
3009local resolved   = { } -- we only resolve a font,script,language pair once
3010
3011local function logprocess(...)
3012    if trace_steps then
3013        registermessage(...)
3014        if trace_steps == "silent" then
3015            return
3016        end
3017    end
3018    report_process(...)
3019end
3020
3021-- todo: pass all these 'locals' in a table
3022
3023local sequencelists = setmetatableindex(function(t,font)
3024    local sequences = fontdata[font].resources.sequences
3025    if not sequences or not next(sequences) then
3026        sequences = false
3027    end
3028    t[font] = sequences
3029    return sequences
3030end)
3031
3032-- fonts.hashes.sequences = sequencelists
3033
3034do -- overcome local limit
3035
3036    local autofeatures    = fonts.analyzers.features
3037    local featuretypes    = otf.tables.featuretypes
3038    local defaultscript   = otf.features.checkeddefaultscript
3039    local defaultlanguage = otf.features.checkeddefaultlanguage
3040
3041    local wildcard        = "*"
3042    local default         = "dflt"
3043
3044    local function initialize(sequence,script,language,enabled,autoscript,autolanguage)
3045        local features = sequence.features
3046        if features then
3047            local order = sequence.order
3048            if order then
3049                local featuretype = featuretypes[sequence.type or "unknown"]
3050                for i=1,#order do
3051                    local kind  = order[i]
3052                    local valid = enabled[kind]
3053                    if valid then
3054                        local scripts   = features[kind]
3055                        local languages = scripts and (
3056                            scripts[script] or
3057                            scripts[wildcard] or
3058                            (autoscript and defaultscript(featuretype,autoscript,scripts))
3059                        )
3060                        local enabled = languages and (
3061                            languages[language] or
3062                            languages[wildcard] or
3063                            (autolanguage and defaultlanguage(featuretype,autolanguage,languages))
3064                        )
3065                        if enabled then
3066                            return { valid, autofeatures[kind] or false, sequence, kind }
3067                        end
3068                    end
3069                end
3070            else
3071                -- can't happen
3072            end
3073        end
3074        return false
3075    end
3076
3077    function otf.dataset(tfmdata,font) -- generic variant, overloaded in context
3078        local shared       = tfmdata.shared
3079        local properties   = tfmdata.properties
3080        local language     = properties.language or "dflt"
3081        local script       = properties.script   or "dflt"
3082        local enabled      = shared.features
3083        local autoscript   = enabled and enabled.autoscript
3084        local autolanguage = enabled and enabled.autolanguage
3085        local res = resolved[font]
3086        if not res then
3087            res = { }
3088            resolved[font] = res
3089        end
3090        local rs = res[script]
3091        if not rs then
3092            rs = { }
3093            res[script] = rs
3094        end
3095        local rl = rs[language]
3096        if not rl then
3097            rl = {
3098                -- indexed but we can also add specific data by key
3099            }
3100            rs[language] = rl
3101            local sequences = tfmdata.resources.sequences
3102            if sequences then
3103                for s=1,#sequences do
3104                    local v = enabled and initialize(sequences[s],script,language,enabled,autoscript,autolanguage)
3105                    if v then
3106                        rl[#rl+1] = v
3107                    end
3108                end
3109            end
3110        end
3111        return rl
3112    end
3113
3114end
3115
3116-- Functions like kernrun, comprun etc evolved over time and in the end look rather
3117-- complex. It's a bit of a compromise between extensive copying and creating subruns.
3118-- The logic has been improved a lot by Kai and Ivo who use complex fonts which
3119-- really helped to identify border cases on the one hand and get insight in the diverse
3120-- ways fonts implement features (not always that consistent and efficient). At the same
3121-- time I tried to keep the code relatively efficient so that the overhead in runtime
3122-- stays acceptable.
3123
3124local function report_disc(what,n)
3125    report_run("%s: %s > %s",what,n,languages.serializediscretionary(n))
3126end
3127
3128local function kernrun(disc,k_run,font,attr,...)
3129    --
3130    -- we catch <font 1><disc font 2>
3131    --
3132    if trace_kernruns then
3133        report_disc("kern",disc)
3134    end
3135    --
3136    local prev, next = getboth(disc)
3137    --
3138    local nextstart = next
3139    local done      = false
3140    --
3141    local pre, post, replace, pretail, posttail, replacetail = getdisc(disc,true)
3142    --
3143    local prevmarks = prev
3144    --
3145    -- can be optional, because why on earth do we get a disc after a mark (okay, maybe when a ccmp
3146    -- has happened but then it should be in the disc so basically this test indicates an error)
3147    --
3148    while prevmarks do
3149        local char = ischar(prevmarks,font)
3150        if char and marks[char] then
3151            prevmarks = getprev(prevmarks)
3152        else
3153            break
3154        end
3155    end
3156    --
3157    if prev and not ischar(prev,font) then  -- and (pre or replace)
3158        prev = false
3159    end
3160    if next and not ischar(next,font) then  -- and (post or replace)
3161        next = false
3162    end
3163    --
3164    -- we need to get rid of this nest mess some day .. has to be done otherwise
3165    --
3166    if pre then
3167        if k_run(pre,"injections",nil,font,attr,...) then
3168            done = true
3169        end
3170        if prev then
3171            setlink(prev,pre)
3172            if k_run(prevmarks,"preinjections",pre,font,attr,...) then -- or prev?
3173                done = true
3174            end
3175            setprev(pre)
3176            setlink(prev,disc)
3177        end
3178    end
3179    --
3180    if post then
3181        if k_run(post,"injections",nil,font,attr,...) then
3182            done = true
3183        end
3184        if next then
3185            setlink(posttail,next)
3186            if k_run(posttail,"postinjections",next,font,attr,...) then
3187                done = true
3188            end
3189            setnext(posttail)
3190            setlink(disc,next)
3191        end
3192    end
3193    --
3194    if replace then
3195        if k_run(replace,"injections",nil,font,attr,...) then
3196            done = true
3197        end
3198        if prev then
3199            setlink(prev,replace)
3200            if k_run(prevmarks,"replaceinjections",replace,font,attr,...) then -- getnext(replace))
3201                done = true
3202            end
3203            setprev(replace)
3204            setlink(prev,disc)
3205        end
3206        if next then
3207            setlink(replacetail,next)
3208            if k_run(replacetail,"replaceinjections",next,font,attr,...) then
3209                done = true
3210            end
3211            setnext(replacetail)
3212            setlink(disc,next)
3213        end
3214    elseif prev and next then
3215        setlink(prev,next)
3216        if k_run(prevmarks,"emptyinjections",next,font,attr,...) then
3217            done = true
3218        end
3219        setlink(prev,disc,next)
3220    end
3221    if done and trace_testruns then
3222        report_disc("done",disc)
3223    end
3224 -- return nextstart, done
3225    return nextstart
3226end
3227
3228-- fonts like ebgaramond do ligatures this way (less efficient than e.g. dejavu which
3229-- will do the testrun variant)
3230
3231local function comprun(disc,c_run,...) -- vararg faster than the whole list
3232    if trace_compruns then
3233        report_disc("comp",disc)
3234    end
3235    --
3236    local pre, post, replace = getdisc(disc)
3237    local renewed = false
3238    --
3239    if pre then
3240        sweepnode = disc
3241        sweeptype = "pre" -- in alternative code preinjections is used (also used then for properties, saves a variable)
3242        local new, done = c_run(pre,...)
3243        if done then
3244            pre     = new
3245            renewed = true
3246        end
3247    end
3248    --
3249    if post then
3250        sweepnode = disc
3251        sweeptype = "post"
3252        local new, done = c_run(post,...)
3253        if done then
3254            post    = new
3255            renewed = true
3256        end
3257    end
3258    --
3259    if replace then
3260        sweepnode = disc
3261        sweeptype = "replace"
3262        local new, done = c_run(replace,...)
3263        if done then
3264            replace = new
3265            renewed = true
3266        end
3267    end
3268    --
3269    sweepnode = nil
3270    sweeptype = nil
3271    if renewed then
3272        if trace_testruns then
3273            report_disc("done",disc)
3274        end
3275        setdisc(disc,pre,post,replace)
3276    end
3277    --
3278 -- return getnext(disc), renewed
3279    return getnext(disc)
3280end
3281
3282-- if we can hyphenate in a lig then unlikely a lig so we
3283-- could have a option here to ignore lig
3284
3285local test_flatten_start = 2 -- must start at 2 according to Kai
3286
3287directives.register("otf.testrun.forceflatten", function(v)
3288    test_flatten_start = v and 1 or 2
3289end)
3290
3291local function testrun(disc,t_run,c_run,...)
3292    if trace_testruns then
3293        report_disc("test",disc)
3294    end
3295    local prev, next = getboth(disc)
3296    if not next then
3297        -- weird discretionary
3298        return
3299    end
3300    local pre, post, replace, pretail, posttail, replacetail = getdisc(disc,true)
3301    local renewed = false
3302    if post or replace then -- and prev then -- hm, we can start with a disc
3303        if post then
3304            setlink(posttail,next)
3305        else
3306            post = next
3307        end
3308        if replace then
3309            setlink(replacetail,next)
3310        else
3311            replace = next
3312        end
3313        local d_post    = t_run(post,next,...)
3314        local d_replace = t_run(replace,next,...)
3315        if d_post > 0 or d_replace > 0 then
3316            local d = d_replace > d_post and d_replace or d_post
3317            local head = getnext(disc) -- is: next
3318            local tail = head
3319            for i=test_flatten_start,d do
3320                local nx = getnext(tail)
3321                local id = getid(nx)
3322                if id == disc_code then
3323                    head, tail = flattendisk(head,nx)
3324                elseif id == glyph_code then
3325                    tail = nx
3326                else
3327                    -- we can have overrun into a glue
3328                    break
3329                end
3330            end
3331            next = getnext(tail)
3332            setnext(tail)
3333            setprev(head)
3334            local new  = copynodelist(head)
3335            if posttail then
3336                setlink(posttail,head)
3337            else
3338                post = head
3339            end
3340            if replacetail then
3341                setlink(replacetail,new)
3342            else
3343                replace = new
3344            end
3345        else
3346            -- we stay inside the disc
3347            if posttail then
3348                setnext(posttail)
3349            else
3350                post = nil
3351            end
3352            if replacetail then
3353                setnext(replacetail)
3354            else
3355                replace = nil
3356            end
3357        end
3358        setlink(disc,next)
3359     -- pre, post, replace, pretail, posttail, replacetail = getdisc(disc,true)
3360    end
3361    --
3362    -- like comprun
3363    --
3364    if trace_testruns then
3365        report_disc("more",disc)
3366    end
3367    --
3368    if pre then
3369        sweepnode = disc
3370        sweeptype = "pre"
3371        local new, ok = c_run(pre,...)
3372        if ok then
3373            pre     = new
3374            renewed = true
3375        end
3376    end
3377    --
3378    if post then
3379        sweepnode = disc
3380        sweeptype = "post"
3381        local new, ok = c_run(post,...)
3382        if ok then
3383            post    = new
3384            renewed = true
3385        end
3386    end
3387    --
3388    if replace then
3389        sweepnode = disc
3390        sweeptype = "replace"
3391        local new, ok = c_run(replace,...)
3392        if ok then
3393            replace = new
3394            renewed = true
3395        end
3396    end
3397    --
3398    sweepnode = nil
3399    sweeptype = nil
3400    if renewed then
3401        setdisc(disc,pre,post,replace)
3402        if trace_testruns then
3403            report_disc("done",disc)
3404        end
3405    end
3406    -- next can have changed (copied list)
3407 -- return getnext(disc), renewed
3408    return getnext(disc)
3409end
3410
3411--  1{2{\oldstyle\discretionary{3}{4}{5}}6}7\par
3412--  1{2\discretionary{3{\oldstyle3}}{{\oldstyle4}4}{5{\oldstyle5}5}6}7\par
3413
3414local nesting = 0
3415
3416local function c_run_single(head,font,attr,lookupcache,step,dataset,sequence,rlmode,skiphash,handler)
3417    local done  = false
3418    local sweep = sweephead[head]
3419    local start
3420    if sweep then
3421        start = sweep
3422     -- sweephead[head] = nil
3423        sweephead[head] = false
3424    else
3425        start = head
3426    end
3427    while start do
3428        local char, id = ischar(start,font)
3429        if char then
3430            local a -- happens often so no assignment is faster
3431            if attr then
3432                a = getglyphdata(start)
3433            end
3434            if not a or (a == attr) then
3435                local lookupmatch = lookupcache[char]
3436                if lookupmatch then
3437                    local ok
3438                    head, start, ok = handler(head,start,dataset,sequence,lookupmatch,rlmode,skiphash,step)
3439                    if ok then
3440                        done = true
3441                    end
3442                end
3443                if start then
3444                    start = getnext(start)
3445                end
3446            else
3447                -- go on can be a mixed one
3448                start = getnext(start)
3449            end
3450        elseif char == false then
3451            return head, done
3452        elseif sweep then
3453            -- else we loose the rest
3454            return head, done
3455        else
3456            -- in disc component
3457            start = getnext(start)
3458        end
3459    end
3460    return head, done
3461end
3462
3463-- only replace?
3464
3465local function t_run_single(start,stop,font,attr,lookupcache)
3466    local lastd = nil
3467    while start ~= stop do
3468        local char = ischar(start,font)
3469        if char then
3470            local a -- happens often so no assignment is faster
3471            if attr then
3472                a = getglyphdata(start)
3473            end
3474            local startnext = getnext(start)
3475            if not a or (a == attr) then
3476                local lookupmatch = lookupcache[char]
3477                if lookupmatch then -- hm, hyphens can match (tlig) so we need to really check
3478                    -- if we need more than ligatures we can outline the code and use functions
3479                    local s     = startnext
3480                    local ss    = nil
3481                    local sstop = s == stop
3482                    if not s then
3483                        s = ss
3484                        ss = nil
3485                    end
3486                    -- a bit weird: why multiple ... anyway we can't have a disc in a disc
3487                    -- how about post ... we can probably merge this into the while
3488                    while getid(s) == disc_code do
3489                        ss = getnext(s)
3490                        s  = getreplace(s)
3491                        if not s then
3492                            s = ss
3493                            ss = nil
3494                        end
3495                    end
3496                    local l = nil
3497                    local d = 0
3498                    while s do
3499                        local char = ischar(s,font)
3500                        if char then
3501                            local lg = not tonumber(lookupmatch) and lookupmatch[char]
3502                            if lg then
3503                                if sstop then
3504                                    d = 1
3505                                elseif d > 0 then
3506                                    d = d + 1
3507                                end
3508                                l = lg
3509                                s = getnext(s)
3510                                sstop = s == stop
3511                                if not s then
3512                                    s  = ss
3513                                    ss = nil
3514                                end
3515                                while getid(s) == disc_code do
3516                                    ss = getnext(s)
3517                                    s  = getreplace(s)
3518                                    if not s then
3519                                        s  = ss
3520                                        ss = nil
3521                                    end
3522                                end
3523                                lookupmatch = lg
3524                            else
3525                                break
3526                            end
3527                        else
3528                            break
3529                        end
3530                    end
3531                    if l and (tonumber(l) or l.ligature) then -- so we test for ligature
3532                        lastd = d
3533                    end
3534                    -- why not: if not l then break elseif l.ligature then return d end
3535                else
3536                    -- why not: break
3537                    -- no match (yet)
3538                end
3539            else
3540                -- go on can be a mixed one
3541                -- why not: break
3542            end
3543            if lastd then
3544                return lastd
3545            end
3546            start = startnext
3547        else
3548            break
3549        end
3550    end
3551    return 0
3552end
3553
3554local function k_run_single(sub,injection,last,font,attr,lookupcache,step,dataset,sequence,rlmode,skiphash,handler)
3555    local a -- happens often so no assignment is faster
3556    if attr then
3557        a = getglyphdata(sub)
3558    end
3559    if not a or (a == attr) then
3560        for n in nextnode, sub do -- only gpos
3561            if n == last then
3562                break
3563            end
3564            local char = ischar(n,font)
3565            if char then
3566                local lookupmatch = lookupcache[char]
3567                if lookupmatch then
3568                    local h, d, ok = handler(sub,n,dataset,sequence,lookupmatch,rlmode,skiphash,step,injection)
3569                    if ok then
3570                        return true
3571                    end
3572                end
3573            end
3574        end
3575    end
3576end
3577
3578local function c_run_multiple(head,font,attr,steps,nofsteps,dataset,sequence,rlmode,skiphash,handler)
3579    local done  = false
3580    local sweep = sweephead[head]
3581    local start
3582    if sweep then
3583        start = sweep
3584     -- sweephead[head] = nil
3585        sweephead[head] = false
3586    else
3587        start = head
3588    end
3589    while start do
3590        local char = ischar(start,font)
3591        if char then
3592            local a -- happens often so no assignment is faster
3593            if attr then
3594                a = getglyphdata(start)
3595            end
3596            if not a or (a == attr) then
3597                for i=1,nofsteps do
3598                    local step        = steps[i]
3599                    local lookupcache = step.coverage
3600                    local lookupmatch = lookupcache[char]
3601                    if lookupmatch then
3602                        -- we could move all code inline but that makes things even more unreadable
3603                        local ok
3604                        head, start, ok = handler(head,start,dataset,sequence,lookupmatch,rlmode,skiphash,step)
3605                        if ok then
3606                            done = true
3607                            break
3608                        elseif not start then
3609                            -- don't ask why ... shouldn't happen
3610                            break
3611                        end
3612                    end
3613                end
3614                if start then
3615                    start = getnext(start)
3616                end
3617            else
3618                -- go on can be a mixed one
3619                start = getnext(start)
3620            end
3621        elseif char == false then
3622            -- whatever glyph
3623            return head, done
3624        elseif sweep then
3625            -- else we loose the rest
3626            return head, done
3627        else
3628            -- in disc component
3629            start = getnext(start)
3630        end
3631    end
3632    return head, done
3633end
3634
3635local function t_run_multiple(start,stop,font,attr,steps,nofsteps)
3636    local lastd = nil
3637    while start ~= stop do
3638        local char = ischar(start,font)
3639        if char then
3640            local a -- happens often so no assignment is faster
3641            if attr then
3642                a = getglyphdata(start)
3643            end
3644            local startnext = getnext(start)
3645            if not a or (a == attr) then
3646                for i=1,nofsteps do
3647                    local step = steps[i]
3648                    local lookupcache = step.coverage
3649                    local lookupmatch = lookupcache[char]
3650                    if lookupmatch then
3651                        -- if we need more than ligatures we can outline the code and use functions
3652                        local s     = startnext
3653                        local ss    = nil
3654                        local sstop = s == stop
3655                        if not s then
3656                            s  = ss
3657                            ss = nil
3658                        end
3659                        while getid(s) == disc_code do
3660                            ss = getnext(s)
3661                            s  = getreplace(s)
3662                            if not s then
3663                                s  = ss
3664                                ss = nil
3665                            end
3666                        end
3667                        local l = nil
3668                        local d = 0
3669                        while s do
3670                            local char = ischar(s)
3671                            if char then
3672                                local lg = not tonumber(lookupmatch) and lookupmatch[char]
3673                                if lg then
3674                                    if sstop then
3675                                        d = 1
3676                                    elseif d > 0 then
3677                                        d = d + 1
3678                                    end
3679                                    l = lg
3680                                    s = getnext(s)
3681                                    sstop = s == stop
3682                                    if not s then
3683                                        s  = ss
3684                                        ss = nil
3685                                    end
3686                                    while getid(s) == disc_code do
3687                                        ss = getnext(s)
3688                                        s  = getreplace(s)
3689                                        if not s then
3690                                            s  = ss
3691                                            ss = nil
3692                                        end
3693                                    end
3694                                    lookupmatch = lg
3695                                else
3696                                    break
3697                                end
3698                            else
3699                                break
3700                            end
3701                        end
3702                        if l and (tonumber(l) or l.ligature) then
3703                            lastd = d
3704                        end
3705                    end
3706                end
3707            else
3708                -- go on can be a mixed one
3709            end
3710            if lastd then
3711                return lastd
3712            end
3713            start = startnext
3714        else
3715            break
3716        end
3717    end
3718    return 0
3719end
3720
3721local function k_run_multiple(sub,injection,last,font,attr,steps,nofsteps,dataset,sequence,rlmode,skiphash,handler)
3722    local a -- happens often so no assignment is faster
3723    if attr then
3724        a = getglyphdata(sub)
3725    end
3726    if not a or (a == attr) then
3727        for n in nextnode, sub do -- only gpos
3728            if n == last then
3729                break
3730            end
3731            local char = ischar(n)
3732            if char then
3733                for i=1,nofsteps do
3734                    local step        = steps[i]
3735                    local lookupcache = step.coverage
3736                    local lookupmatch = lookupcache[char]
3737                    if lookupmatch then
3738                        local h, d, ok = handler(sub,n,dataset,sequence,lookupmatch,rlmode,skiphash,step,injection) -- sub was head
3739                        if ok then
3740                            return true
3741                        end
3742                    end
3743                end
3744            end
3745        end
3746    end
3747end
3748
3749local txtdirstate, pardirstate  do -- this might change (no need for nxt in pardirstate)
3750
3751    local getdirection = nuts.getdirection
3752
3753    txtdirstate = function(start,stack,top,rlparmode)
3754        local dir, pop = getdirection(start)
3755        if pop then
3756            if top == 1 then
3757                return 0, rlparmode
3758            else
3759                top = top - 1
3760                if stack[top] == righttoleft_code then
3761                    return top, -1
3762                else
3763                    return top, 1
3764                end
3765            end
3766        elseif dir == lefttoright_code then
3767            top = top + 1
3768            stack[top] = lefttoright_code
3769            return top, 1
3770        elseif dir == righttoleft_code then
3771            top = top + 1
3772            stack[top] = righttoleft_code
3773            return top, -1
3774        else
3775            return top, rlparmode
3776        end
3777    end
3778
3779    pardirstate = function(start)
3780        local dir = getdirection(start)
3781        if dir == lefttoright_code then
3782            return 1, 1
3783        elseif dir == righttoleft_code then
3784            return -1, -1
3785        else
3786            return 0, 0
3787        end
3788    end
3789
3790end
3791
3792-- These are non public helpers that can change without notice!
3793
3794otf.helpers             = otf.helpers or { }
3795otf.helpers.txtdirstate = txtdirstate
3796otf.helpers.pardirstate = pardirstate
3797
3798-- This is the main loop. We run over the node list dealing with a specific font. The
3799-- attribute is a context specific thing. We could work on sub start-stop ranges instead
3800-- but I wonder if there is that much speed gain (experiments showed that it made not
3801-- much sense) and we need to keep track of directions anyway. Also at some point I
3802-- want to play with font interactions and then we do need the full sweeps. Apart from
3803-- optimizations the principles of processing the features hasn't changed much since
3804-- the beginning.
3805
3806do
3807
3808    -- This is a measurable experimental speedup (only with hyphenated text and multiple
3809    -- fonts per processor call), especially for fonts with lots of contextual lookups.
3810
3811    local fastdisc = true
3812    local testdics = false
3813
3814    directives.register("otf.fastdisc",function(v) fastdisc = v end) -- normally enabled
3815
3816    -- using a merged combined hash as first test saves some 30% on ebgaramond and
3817    -- about 15% on arabtype .. then moving the a test also saves a bit (even when
3818    -- often a is not set at all so that one is a bit debatable
3819
3820    local otfdataset = nil -- todo: make an installer
3821
3822    local getfastdisc = { __index = function(t,k)
3823        local v = usesfont(k,currentfont)
3824        t[k] = v
3825        return v
3826    end }
3827
3828    local getfastspace = { __index = function(t,k)
3829        -- we don't pass the id so that one can overload isspace
3830        local v = isspace(k,threshold) or false
3831        t[k] = v
3832        return v
3833    end }
3834
3835    function otf.featuresprocessor(head,font,attr,direction,n)
3836
3837        local sequences = sequencelists[font] -- temp hack
3838
3839        nesting = nesting + 1
3840
3841        if nesting == 1 then
3842            currentfont  = font
3843            tfmdata      = fontdata[font]
3844            descriptions = tfmdata.descriptions -- only needed in gref so we could pass node there instead
3845            characters   = tfmdata.characters   -- but this branch is not entered that often anyway
3846      local resources    = tfmdata.resources
3847            marks        = resources.marks
3848            classes      = resources.classes
3849            threshold,
3850            factor       = getthreshold(font)
3851            checkmarks   = tfmdata.properties.checkmarks
3852
3853            if not otfdataset then
3854                otfdataset = otf.dataset
3855            end
3856
3857            discs  = fastdisc and n and n > 1 and setmetatable({},getfastdisc) -- maybe inline
3858            spaces = setmetatable({},getfastspace)
3859
3860        elseif currentfont ~= font then
3861
3862            report_warning("nested call with a different font, level %s, quitting",nesting)
3863            nesting = nesting - 1
3864            return head, false
3865
3866        end
3867
3868        -- some 10% faster when no dynamics but hardly measureable on real runs .. but: it only
3869        -- works when we have no other dynamics as otherwise the zero run will be applied to the
3870        -- whole stream for which we then need to pass another variable which we won't
3871
3872        -- if attr == 0 then
3873        --     attr = false
3874        -- end
3875
3876        if trace_steps then
3877            checkstep(head)
3878        end
3879
3880        local initialrl = 0
3881
3882        if getid(head) == par_code and startofpar(head) then
3883            initialrl = pardirstate(head)
3884        elseif direction == righttoleft_code then
3885            initialrl = -1
3886        end
3887
3888     -- local done      = false
3889        local datasets  = otfdataset(tfmdata,font,attr)
3890        local dirstack  = { nil } -- could move outside function but we can have local runs
3891        sweephead       = { }
3892     -- sweephead  = { a = 1, b = 1 } sweephead.a = nil sweephead.b = nil
3893
3894        -- Keeping track of the headnode is needed for devanagari. (I generalized it a bit
3895        -- so that multiple cases are also covered.) We could prepend a temp node.
3896
3897        -- We don't goto the next node when a disc node is created so that we can then treat
3898        -- the pre, post and replace. It's a bit of a hack but works out ok for most cases.
3899
3900        for s=1,#datasets do
3901            local dataset      = datasets[s]
3902            local attribute    = dataset[2]
3903            local sequence     = dataset[3] -- sequences[s] -- also dataset[5]
3904            local rlparmode    = initialrl
3905            local topstack     = 0
3906            local typ          = sequence.type
3907            local gpossing     = typ == "gpos_single" or typ == "gpos_pair" -- store in dataset
3908            local forcetestrun = typ == "gsub_ligature" -- testrun is only for ligatures
3909            local handler      = handlers[typ] -- store in dataset
3910            local steps        = sequence.steps
3911            local nofsteps     = sequence.nofsteps
3912            local skiphash     = sequence.skiphash
3913
3914            if not steps then
3915                -- This permits injection, watch the different arguments. Watch out, the arguments passed
3916                -- are not frozen as we might extend or change this. Is this used at all apart from some
3917                -- experiments?
3918                local h, ok = handler(head,dataset,sequence,initialrl,font,attr) -- less arguments now
3919             -- if ok then
3920             --     done = true
3921             -- end
3922                if h and h ~= head then
3923                    head = h
3924                end
3925            elseif typ == "gsub_reversecontextchain" then
3926                --
3927                -- This might need a check: if we have #before or #after > 0 then we might need to reverse
3928                -- the before and after lists in the loader. But first I need to see a font that uses multiple
3929                -- matches.
3930                --
3931                local start  = findnodetail(head)
3932                local rlmode = 0 -- how important is this .. do we need to check for dir?
3933                local merged = steps.merged
3934                while start do
3935                    local char = ischar(start,font)
3936                    if char then
3937                        local m = merged[char]
3938                        if m then
3939                            local a -- happens often so no assignment is faster
3940                            if attr then
3941                                a = getglyphdata(start)
3942                            end
3943                            if not a or (a == attr) then
3944                                for i=m[1],m[2] do
3945                                    local step = steps[i]
3946                             -- for i=1,#m do
3947                             --     local step = m[i]
3948                                    local lookupcache = step.coverage
3949                                    local lookupmatch = lookupcache[char]
3950                                    if lookupmatch then
3951                                        local ok
3952                                        head, start, ok = handler(head,start,dataset,sequence,lookupmatch,rlmode,skiphash,step)
3953                                        if ok then
3954                                         -- done = true
3955                                            break
3956                                        end
3957                                    end
3958                                end
3959                                if start then
3960                                    start = getprev(start)
3961                                end
3962                            else
3963                                start = getprev(start)
3964                            end
3965                        else
3966                            start = getprev(start)
3967                        end
3968                    else
3969                        start = getprev(start)
3970                    end
3971                end
3972            else
3973                local start  = head
3974                local rlmode = initialrl
3975                if nofsteps == 1 then -- happens often
3976                    local step = steps[1]
3977                    local lookupcache = step.coverage
3978                    while start do
3979                        local char, id = ischar(start,font)
3980                        if char then
3981                            local lookupmatch = lookupcache[char]
3982                            if lookupmatch then
3983                                local a -- happens often so no assignment is faster
3984                                if attr then
3985                                    if getglyphdata(start) == attr and (not attribute or getstate(start,attribute)) then
3986                                        a = true
3987                                    end
3988                                elseif not attribute or getstate(start,attribute) then
3989                                    a = true
3990                                end
3991                                if a then
3992                                    local ok, df
3993                                    head, start, ok, df = handler(head,start,dataset,sequence,lookupmatch,rlmode,skiphash,step)
3994                                 -- if ok then
3995                                 --     done = true
3996                                 -- end
3997                                    if df then
3998-- print("restart 1",typ)
3999                                    elseif start then
4000                                        start = getnext(start)
4001                                    end
4002                                else
4003                                    start = getnext(start)
4004                                end
4005                            else
4006                               start = getnext(start)
4007                            end
4008                        elseif char == false or id == glue_code then
4009                            -- a different font|state or glue (happens often)
4010                            start = getnext(start)
4011                        elseif id == disc_code then
4012                            if not discs or discs[start] == true then
4013                                if gpossing then
4014                                    start = kernrun(start,k_run_single,             font,attr,lookupcache,step,dataset,sequence,rlmode,skiphash,handler)
4015                                elseif forcetestrun then
4016                                    start = testrun(start,t_run_single,c_run_single,font,attr,lookupcache,step,dataset,sequence,rlmode,skiphash,handler)
4017                                else
4018                                    start = comprun(start,c_run_single,             font,attr,lookupcache,step,dataset,sequence,rlmode,skiphash,handler)
4019                                end
4020                            else
4021                                start = getnext(start)
4022                            end
4023                        elseif id == math_code then
4024                            start = getnext(endofmath(start))
4025                        elseif id == dir_code then
4026                            topstack, rlmode = txtdirstate(start,dirstack,topstack,rlparmode)
4027                            start = getnext(start)
4028                     -- elseif id == par_code and startofpar(start) then
4029                     --     rlparmode, rlmode = pardirstate(start)
4030                     --     start = getnext(start)
4031                        else
4032                            start = getnext(start)
4033                        end
4034                    end
4035                else
4036                    local merged = steps.merged
4037                    while start do
4038                        local char, id = ischar(start,font)
4039                        if char then
4040                            local m = merged[char]
4041                            if m then
4042                                local a -- happens often so no assignment is faster
4043                                if attr then
4044                                    if getglyphdata(start) == attr and (not attribute or getstate(start,attribute)) then
4045                                        a = true
4046                                    end
4047                                elseif not attribute or getstate(start,attribute) then
4048                                    a = true
4049                                end
4050                                if a then
4051                                    local ok, df
4052                                    for i=m[1],m[2] do
4053                                        local step = steps[i]
4054                                 -- for i=1,#m do
4055                                 --     local step = m[i]
4056                                        local lookupcache = step.coverage
4057                                        local lookupmatch = lookupcache[char]
4058                                        if lookupmatch then
4059                                            -- we could move all code inline but that makes things even more unreadable
4060--                                                 local ok, df
4061                                            head, start, ok, df = handler(head,start,dataset,sequence,lookupmatch,rlmode,skiphash,step)
4062                                            if df then
4063                                                break
4064                                            elseif ok then
4065                                             -- done = true
4066                                                break
4067                                            elseif not start then
4068                                                -- don't ask why ... shouldn't happen
4069                                                break
4070                                            end
4071                                        end
4072                                    end
4073                                    if df then
4074-- print("restart 2",typ)
4075                                    elseif start then
4076                                        start = getnext(start)
4077                                    end
4078                                else
4079                                    start = getnext(start)
4080                                end
4081                            else
4082                                start = getnext(start)
4083                            end
4084                        elseif char == false or id == glue_code then
4085                            -- a different font|state or glue (happens often)
4086                            start = getnext(start)
4087                        elseif id == disc_code then
4088                            if not discs or discs[start] == true then
4089                                if gpossing then
4090                                    start = kernrun(start,k_run_multiple,               font,attr,steps,nofsteps,dataset,sequence,rlmode,skiphash,handler)
4091                                elseif forcetestrun then
4092                                    start = testrun(start,t_run_multiple,c_run_multiple,font,attr,steps,nofsteps,dataset,sequence,rlmode,skiphash,handler)
4093                                else
4094                                    start = comprun(start,c_run_multiple,               font,attr,steps,nofsteps,dataset,sequence,rlmode,skiphash,handler)
4095                                end
4096                            else
4097                                start = getnext(start)
4098                            end
4099                        elseif id == math_code then
4100                            start = getnext(endofmath(start))
4101                        elseif id == dir_code then
4102                            topstack, rlmode = txtdirstate(start,dirstack,topstack,rlparmode)
4103                            start = getnext(start)
4104                     -- elseif id == par_code and startofpar(start) then
4105                     --     rlparmode, rlmode = pardirstate(start)
4106                     --     start = getnext(start)
4107                        else
4108                            start = getnext(start)
4109                        end
4110                    end
4111                end
4112            end
4113
4114            if trace_steps then -- ?
4115                registerstep(head)
4116            end
4117
4118        end
4119
4120        nesting = nesting - 1
4121
4122     -- return head, done
4123        return head
4124    end
4125
4126    -- This is not an official helper and used for tracing experiments. It can be changed as I like
4127    -- at any moment. At some point it might be used in a module that can help font development.
4128
4129    function otf.datasetpositionprocessor(head,font,direction,dataset)
4130
4131        currentfont     = font
4132        tfmdata         = fontdata[font]
4133        descriptions    = tfmdata.descriptions -- only needed in gref so we could pass node there instead
4134        characters      = tfmdata.characters   -- but this branch is not entered that often anyway
4135  local resources       = tfmdata.resources
4136        marks           = resources.marks
4137        classes         = resources.classes
4138        threshold,
4139        factor          = getthreshold(font)
4140        checkmarks      = tfmdata.properties.checkmarks
4141
4142        if type(dataset) == "number" then
4143            dataset = otfdataset(tfmdata,font,0)[dataset]
4144        end
4145
4146        local sequence  = dataset[3] -- sequences[s] -- also dataset[5]
4147        local typ       = sequence.type
4148     -- local gpossing  = typ == "gpos_single" or typ == "gpos_pair" -- store in dataset
4149
4150     -- gpos_contextchain gpos_context
4151
4152     -- if not gpossing then
4153     --     return head, false
4154     -- end
4155
4156        local handler   = handlers[typ] -- store in dataset
4157        local steps     = sequence.steps
4158        local nofsteps  = sequence.nofsteps
4159
4160        local done      = false
4161        local dirstack  = { nil } -- could move outside function but we can have local runs (maybe a few more nils)
4162        local start     = head
4163        local initialrl = (direction == righttoleft_code) and -1 or 0
4164        local rlmode    = initialrl
4165        local rlparmode = initialrl
4166        local topstack  = 0
4167        local merged    = steps.merged
4168
4169     -- local matches   = false
4170        local position  = 0
4171
4172        while start do
4173            local char, id = ischar(start,font)
4174            if char then
4175                position = position + 1
4176                local m = merged[char]
4177                if m then
4178                    for i=m[1],m[2] do
4179                        local step = steps[i]
4180                        local lookupcache = step.coverage
4181                        local lookupmatch = lookupcache[char]
4182                        if lookupmatch then
4183                            local ok
4184                            head, start, ok = handler(head,start,dataset,sequence,lookupmatch,rlmode,skiphash,step)
4185                            if ok then
4186                             -- if matches then
4187                             --     matches[position] = i
4188                             -- else
4189                             --     matches = { [position] = i }
4190                             -- end
4191                                break
4192                            elseif not start then
4193                                break
4194                            end
4195                        end
4196                    end
4197                    if start then
4198                        start = getnext(start)
4199                    end
4200                else
4201                    start = getnext(start)
4202                end
4203            elseif char == false or id == glue_code then
4204                -- a different font|state or glue (happens often)
4205                start = getnext(start)
4206            elseif id == math_code then
4207                start = getnext(endofmath(start))
4208            elseif id == dir_code then
4209                topstack, rlmode = txtdirstate(start,dirstack,topstack,rlparmode)
4210                start = getnext(start)
4211         -- elseif id == par_code and startofpar(start) then
4212         --     rlparmode, rlmode = pardirstate(start)
4213         --     start = nxt
4214            else
4215                start = getnext(start)
4216            end
4217        end
4218
4219        return head
4220    end
4221
4222    -- end of experiment
4223
4224end
4225
4226-- so far
4227
4228do
4229
4230    local plugins = { }
4231    otf.plugins   = plugins
4232
4233    local report  = logs.reporter("fonts")
4234    local warned  = false
4235    local okay    = { text = true }
4236
4237    function otf.registerplugin(name,f)
4238        if type(name) == "string" and type(f) == "function" then
4239            plugins[name] = { name, f }
4240            if okay[name] then
4241                -- no warning (e.g. the diagnostic text plugin)
4242            else
4243                report("plugin %a has been loaded, please be aware of possible side effects",name)
4244                if not warned then
4245                    if logs.pushtarget then
4246                        logs.pushtarget("log")
4247                    end
4248                    report("Plugins are not officially supported unless stated otherwise. This is because")
4249                    report("they bypass the regular font handling and therefore some features in ConTeXt")
4250                    report("(especially those related to fonts) might not work as expected or might not work")
4251                    report("at all. Some plugins are for testing and development only and might change")
4252                    report("whenever we feel the need for it.")
4253                    report()
4254                    if logs.poptarget then
4255                        logs.poptarget()
4256                    end
4257                    warned = true
4258                end
4259            end
4260        end
4261    end
4262
4263    function otf.plugininitializer(tfmdata,value)
4264        if type(value) == "string" then
4265            tfmdata.shared.plugin = plugins[value]
4266        end
4267    end
4268
4269    function otf.pluginprocessor(head,font,dynamic,direction) -- n
4270        local s = fontdata[font].shared
4271        local p = s and s.plugin
4272        if p then
4273            if trace_plugins then
4274                report_process("applying plugin %a",p[1])
4275            end
4276            return p[2](head,font,dynamic,direction)
4277        else
4278            return head, false
4279        end
4280    end
4281
4282end
4283
4284function otf.featuresinitializer(tfmdata,value)
4285    -- nothing done here any more
4286end
4287
4288registerotffeature {
4289    name         = "features",
4290    description  = "features",
4291    default      = true,
4292    initializers = {
4293        position = 1,
4294        node     = otf.featuresinitializer,
4295        plug     = otf.plugininitializer,
4296    },
4297    processors   = {
4298        node     = otf.featuresprocessor,
4299        plug     = otf.pluginprocessor,
4300    }
4301}
4302
4303-- Moved here (up) a bit. This doesn't really belong in generic so it will move to a
4304-- context module some day.
4305
4306local function markinitializer(tfmdata,value)
4307    local properties = tfmdata.properties
4308    properties.checkmarks = value
4309end
4310
4311registerotffeature {
4312    name         = "checkmarks",
4313    description  = "check mark widths",
4314    default      = true,
4315    initializers = {
4316        node     = markinitializer,
4317    },
4318}
4319
4320-- This can be used for extra handlers, but should be used with care! We implement one
4321-- here but some more can be found in the osd (script devanagary) file. Now watch out:
4322-- when a handler has steps, it is called as the other ones, but when we have no steps,
4323-- we use a different call:
4324--
4325--   function(head,dataset,sequence,initialrl,font,attr)
4326--       return head, done
4327--   end
4328--
4329-- Also see (!!).
4330
4331otf.handlers = handlers
4332
4333if context then
4334    return
4335else
4336    -- todo: move the following code someplace else
4337end
4338
4339local setspacekerns = nodes.injections.setspacekerns if not setspacekerns then os.exit() end
4340
4341local tag = "kern"
4342
4343-- if fontfeatures then
4344
4345--     function handlers.trigger_space_kerns(head,dataset,sequence,initialrl,font,attr)
4346--         local features = fontfeatures[font]
4347--         local enabled  = features and features.spacekern and features[tag]
4348--         if enabled then
4349--             setspacekerns(font,sequence)
4350--         end
4351--         return head, enabled
4352--     end
4353
4354-- else -- generic (no hashes)
4355
4356    function handlers.trigger_space_kerns(head,dataset,sequence,initialrl,font,attr)
4357        local shared   = fontdata[font].shared
4358        local features = shared and shared.features
4359        local enabled  = features and features.spacekern and features[tag]
4360        if enabled then
4361            setspacekerns(font,sequence)
4362        end
4363        return head, enabled
4364    end
4365
4366-- end
4367
4368-- There are fonts out there that change the space but we don't do that kind of
4369-- things in TeX.
4370
4371local function hasspacekerns(data)
4372    local resources = data.resources
4373    local sequences = resources.sequences
4374    local validgpos = resources.features.gpos
4375    if validgpos and sequences then
4376        for i=1,#sequences do
4377            local sequence = sequences[i]
4378            local steps    = sequence.steps
4379            if steps and sequence.features[tag] then
4380                local kind = sequence.type
4381                if kind == "gpos_pair" or kind == "gpos_single" then
4382                    for i=1,#steps do
4383                        local step     = steps[i]
4384                        local coverage = step.coverage
4385                        local rules    = step.rules
4386                        if rules then
4387                            -- not now: analyze (simple) rules
4388                        elseif not coverage then
4389                            -- nothing to do
4390                        elseif kind == "gpos_single" then
4391                            -- maybe a message that we ignore
4392                        elseif kind == "gpos_pair" then
4393                            local format = step.format
4394                            if format == "move" or format == "kern" then
4395                                local kerns  = coverage[32]
4396                                if kerns then
4397                                    return true
4398                                end
4399                                for k, v in next, coverage do
4400                                    if v[32] then
4401                                        return true
4402                                    end
4403                                end
4404                            elseif format == "pair" then
4405                                local kerns  = coverage[32]
4406                                if kerns then
4407                                    for k, v in next, kerns do
4408                                        local one = v[1]
4409                                        if one and one ~= true then
4410                                            return true
4411                                        end
4412                                    end
4413                                end
4414                                for k, v in next, coverage do
4415                                    local kern = v[32]
4416                                    if kern then
4417                                        local one = kern[1]
4418                                        if one and one ~= true then
4419                                            return true
4420                                        end
4421                                    end
4422                                end
4423                            end
4424                        end
4425                    end
4426                end
4427            end
4428        end
4429    end
4430    return false
4431end
4432
4433otf.readers.registerextender {
4434    name   = "spacekerns",
4435    action = function(data)
4436        data.properties.hasspacekerns = hasspacekerns(data)
4437    end
4438}
4439
4440local function spaceinitializer(tfmdata,value) -- attr
4441    local resources  = tfmdata.resources
4442    local spacekerns = resources and resources.spacekerns
4443    if value and spacekerns == nil then
4444        local rawdata    = tfmdata.shared and tfmdata.shared.rawdata
4445        local properties = rawdata.properties
4446        if properties and properties.hasspacekerns then
4447            local sequences = resources.sequences
4448            local validgpos = resources.features.gpos
4449            if validgpos and sequences then
4450                local left  = { }
4451                local right = { }
4452                local last  = 0
4453                local feat  = nil
4454                for i=1,#sequences do
4455                    local sequence = sequences[i]
4456                    local steps    = sequence.steps
4457                    if steps then
4458                        -- we don't support space kerns in other features
4459                        local kern = sequence.features[tag]
4460                        if kern then
4461                            local kind = sequence.type
4462                            if kind == "gpos_pair" or kind == "gpos_single" then
4463                                if feat then
4464                                    for script, languages in next, kern do
4465                                        local f = feat[script]
4466                                        if f then
4467                                            for l in next, languages do
4468                                                f[l] = true
4469                                            end
4470                                        else
4471                                            feat[script] = languages
4472                                        end
4473                                    end
4474                                else
4475                                    feat = kern
4476                                end
4477                                for i=1,#steps do
4478                                    local step     = steps[i]
4479                                    local coverage = step.coverage
4480                                    local rules    = step.rules
4481                                    if rules then
4482                                        -- not now: analyze (simple) rules
4483                                    elseif not coverage then
4484                                        -- nothing to do
4485                                    elseif kind == "gpos_single" then
4486                                        -- makes no sense in TeX