SourceBrowser

font-ots.lmt /size: 174 Kb last modification: 2025-02-21 11:03
1if not modules then modules = { } end modules ['font-ots'] = { -- sequences
2    version   = 1.001,
3    optimize  = true,
4    comment   = "companion to font-ini.mkiv",
5    author    = "Hans Hagen, PRAGMA-ADE, Hasselt NL",
6    copyright = "PRAGMA ADE / ConTeXt Development Team",
7    license   = "see context related readme files",
8}
9
10-- I need to check the description at the microsoft site ... it has been improved so
11-- maybe there are some interesting details there. Most below is based on old and
12-- incomplete documentation and involved quite a bit of guesswork (checking with the
13-- abstract uniscribe of those days. But changing things is tricky!
14--
15-- This module is a bit more split up that I'd like but since we also want to test
16-- with plain TeX it has to be so. This module is part of ConTeXt and discussion
17-- about improvements and functionality mostly happens on the ConTeXt mailing list.
18--
19-- The specification of OpenType is (or at least decades ago was) kind of vague.
20-- Apart from a lack of a proper free specifications there's also the problem that
21-- Microsoft and Adobe may have their own interpretation of how and in what order to
22-- apply features. In general the Microsoft website has more detailed specifications
23-- and is a better reference. There is also some information in the FontForge help
24-- files. In the end we rely most on the Microsoft specification.
25--
26-- Because there is so much possible, fonts might contain bugs and/or be made to
27-- work with certain rederers. These may evolve over time which may have the side
28-- effect that suddenly fonts behave differently. We don't want to catch all font
29-- issues.
30--
31-- After a lot of experiments (mostly by Taco, me and Idris) the first
32-- implementation was already quite useful. When it did most of what we wanted, a
33-- more optimized version evolved. Of course all errors are mine and of course the
34-- code can be improved. There are quite some optimizations going on here and
35-- processing speed is currently quite acceptable and has been improved over time.
36-- Many complex scripts are not yet supported yet, but I will look into them as soon
37-- as ConTeXt users ask for it.
38--
39-- The specification leaves room for interpretation. In case of doubt the Microsoft
40-- implementation is the reference as it is the most complete one. As they deal with
41-- lots of scripts and fonts, Kai and Ivo did a lot of testing of the generic code
42-- and their suggestions help improve the code. I'm aware that not all border cases
43-- can be taken care of, unless we accept excessive runtime, and even then the
44-- interference with other mechanisms (like hyphenation) are not trivial.
45--
46-- Especially discretionary handling has been improved much by Kai Eigner who uses
47-- complex (latin) fonts. The current implementation is a compromis between his
48-- patches and my code and in the meantime performance is quite ok. We cannot check
49-- all border cases without compromising speed but so far we're okay. Given good
50-- test cases we can probably improve it here and there. Especially chain lookups
51-- are non trivial with discretionaries but things got much better over time thanks
52-- to Kai.
53--
54-- Glyphs are indexed not by unicode but in their own way. This is because there is
55-- no relationship with unicode at all, apart from the fact that a font might cover
56-- certain ranges of characters. One character can have multiple shapes. However, at
57-- the TeX end we use unicode so and all extra glyphs are mapped into a private
58-- space. This is needed because we need to access them and TeX has to include then
59-- in the output eventually.
60--
61-- The initial data table is rather close to the open type specification and also
62-- not that different from the one produced by Fontforge but we uses hashes instead.
63-- In ConTeXt that table is packed (similar tables are shared) and cached on disk so
64-- that successive runs can use the optimized table (after loading the table is
65-- unpacked).
66--
67-- This module is sparsely documented because it is has been a moving target. The
68-- table format of the reader changed a bit over time and we experiment a lot with
69-- different methods for supporting features. By now the structures are quite stable
70--
71-- Incrementing the version number will force a re-cache. We jump the number by one
72-- when there's a fix in the reader or processing code that can result in different
73-- results.
74--
75-- This code is also used outside ConTeXt but in ConTeXt it has to work with other
76-- mechanisms. Both put some constraints on the code here.
77--
78-- Remark: We assume that cursives don't cross discretionaries which is okay because
79-- it is only used in semitic scripts.
80--
81-- Remark: We assume that marks precede base characters.
82--
83-- Remark: When complex ligatures extend into discs nodes we can get side effects.
84-- Normally this doesn't happen; ff\d{l}{l}{l} in lm works but ff\d{f}{f}{f}.
85--
86-- Todo: check if we copy attributes to disc nodes if needed.
87--
88-- Todo: it would be nice if we could get rid of components. In other places we can
89-- use the unicode properties. We can just keep a lua table.
90--
91-- Remark: We do some disc juggling where we need to keep in mind that the pre, post
92-- and replace fields can have prev pointers to a nesting node ... I wonder if that
93-- is still needed.
94--
95-- Remark: This is not possible:
96--
97-- \discretionary {alpha-} {betagammadelta}
98--   {\discretionary {alphabeta-} {gammadelta}
99--      {\discretionary {alphabetagamma-} {delta}
100--         {alphabetagammadelta}}}
101--
102-- Remark: Something is messed up: we have two mark / ligature indices, one at the
103-- injection end and one here ... this is based on KE's patches but there is something
104-- fishy there as I'm pretty sure that for husayni we need some connection (as it's much
105-- more complex than an average font) but I need proper examples of all cases, not of
106-- only some.
107--
108-- Remark: I wonder if indexed would be faster than unicoded. It would be a major
109-- rewrite to have char being unicode + an index field in glyph nodes. Also more
110-- assignments have to be made in order to keep things in sync. So, it's a no-go.
111--
112-- Remark: We can provide a fast loop when there are no disc nodes (tests show a 1%
113-- gain). Smaller functions might perform better cache-wise. But ... memory becomes
114-- faster anyway, so ...
115--
116-- Remark: Some optimizations made sense for 5.2 but seem less important for 5.3 but
117-- anyway served their purpose.
118--
119-- Todo: just (0=l2r and 1=r2l) or maybe (r2l = true)
120
121-- Experiments with returning the data with the ischar are positive for lmtx but
122-- have a performance hit on mkiv because there we need to wrap ischardata (pending
123-- extensions to luatex which is unlikely to happen for such an experiment because
124-- we then can't remove it). Actually it might make generic slightly faster. Also,
125-- there are some corner cases where a data check comes before a char fetch and
126-- we're talking of millions of calls there. At some point I might make a version
127-- for lmtx that does it slightly different anyway.
128
129local type, next, tonumber = type, next, tonumber
130local random = math.random
131local formatters = string.formatters
132local insert = table.insert
133
134local registertracker      = trackers.register
135
136local logs                 = logs
137local trackers             = trackers
138local nodes                = nodes
139local fonts                = fonts
140
141local otf                  = fonts.handlers.otf
142local tracers              = nodes.tracers
143
144local trace_singles        = false  registertracker("otf.singles",      function(v) trace_singles      = v end)
145local trace_multiples      = false  registertracker("otf.multiples",    function(v) trace_multiples    = v end)
146local trace_alternatives   = false  registertracker("otf.alternatives", function(v) trace_alternatives = v end)
147local trace_ligatures      = false  registertracker("otf.ligatures",    function(v) trace_ligatures    = v end)
148local trace_contexts       = false  registertracker("otf.contexts",     function(v) trace_contexts     = v end)
149local trace_marks          = false  registertracker("otf.marks",        function(v) trace_marks        = v end)
150local trace_kerns          = false  registertracker("otf.kerns",        function(v) trace_kerns        = v end)
151local trace_cursive        = false  registertracker("otf.cursive",      function(v) trace_cursive      = v end)
152local trace_preparing      = false  registertracker("otf.preparing",    function(v) trace_preparing    = v end)
153local trace_bugs           = false  registertracker("otf.bugs",         function(v) trace_bugs         = v end)
154local trace_details        = false  registertracker("otf.details",      function(v) trace_details      = v end)
155local trace_steps          = false  registertracker("otf.steps",        function(v) trace_steps        = v end)
156local trace_skips          = false  registertracker("otf.skips",        function(v) trace_skips        = v end)
157local trace_plugins        = false  registertracker("otf.plugins",      function(v) trace_plugins      = v end)
158local trace_chains         = false  registertracker("otf.chains",       function(v) trace_chains       = v end)
159
160local trace_kernruns       = false  registertracker("otf.kernruns",     function(v) trace_kernruns     = v end)
161----- trace_discruns       = false  registertracker("otf.discruns",     function(v) trace_discruns     = v end)
162local trace_compruns       = false  registertracker("otf.compruns",     function(v) trace_compruns     = v end)
163local trace_testruns       = false  registertracker("otf.testruns",     function(v) trace_testruns     = v end)
164
165local forcediscretionaries = false
166local forcepairadvance     = false -- for testing
167
168local repeatlastmultiple   = context and true
169
170directives.register("otf.forcediscretionaries", function(v) forcediscretionaries = v end)
171directives.register("otf.forcepairadvance",     function(v) forcepairadvance     = v end)
172
173local report_direct      = logs.reporter("fonts","otf direct")
174local report_subchain    = logs.reporter("fonts","otf subchain")
175local report_chain       = logs.reporter("fonts","otf chain")
176local report_process     = logs.reporter("fonts","otf process")
177local report_warning     = logs.reporter("fonts","otf warning")
178local report_run         = logs.reporter("fonts","otf run")
179
180registertracker("otf.substitutions", "otf.singles","otf.multiples","otf.alternatives","otf.ligatures")
181registertracker("otf.positions",     "otf.marks","otf.kerns","otf.cursive")
182registertracker("otf.actions",       "otf.substitutions","otf.positions")
183registertracker("otf.sample",        "otf.steps","otf.substitutions","otf.positions","otf.analyzing")
184registertracker("otf.sample.silent", "otf.steps=silent","otf.substitutions","otf.positions","otf.analyzing")
185
186local nuts               = nodes.nuts
187
188local getnext            = nuts.getnext
189local setnext            = nuts.setnext
190local getprev            = nuts.getprev
191local setprev            = nuts.setprev
192local getboth            = nuts.getboth
193local setboth            = nuts.setboth
194local getid              = nuts.getid
195local getstate           = nuts.getstate
196local getchar            = nuts.getchar
197local setchar            = nuts.setchar
198local getdisc            = nuts.getdisc
199local setdisc            = nuts.setdisc
200local getreplace         = nuts.getreplace
201local setlink            = nuts.setlink
202local getwidth           = nuts.getwidth
203local getoptions         = nuts.getoptions
204local setoptions         = nuts.setoptions
205
206local getglyphdata       = nuts.getglyphdata
207local getscales          = nuts.getscales
208
209---------------------------------------------------------------------------------------
210
211-- Beware: In ConTeXt components no longer are real components. We only keep track of
212-- their positions because some complex ligatures might need that.
213
214local components         = nuts.components
215local copynocomponents   = components.copynocomponents
216local copyonlyglyphs     = components.copyonlyglyphs
217local countcomponents    = components.count
218local setcomponents      = components.set
219local getcomponents      = components.get
220local flushcomponents    = components.flush
221
222---------------------------------------------------------------------------------------
223
224local ischar             = nuts.ischar
225local isnextchar         = nuts.isnextchar -- only makes sense when we know we need next
226local isprevchar         = nuts.isprevchar -- only makes sense when we know we need prev
227local usesfont           = nuts.usesfont
228
229local insertnodeafter    = nuts.insertafter
230local copynode           = nuts.copy
231local copynodelist       = nuts.copylist
232local removenode         = nuts.remove
233local findnodetail       = nuts.tail
234local flushnodelist      = nuts.flushlist
235local flushnode          = nuts.flushnode
236local endofmath          = nuts.endofmath
237
238local startofpar         = nuts.startofpar
239
240local setmetatable       = setmetatable
241local setmetatableindex  = table.setmetatableindex
242
243local nextnode           = nuts.traversers.node
244
245local nodecodes          = nodes.nodecodes
246
247local glyph_code         <const> = nodecodes.glyph
248local glue_code          <const> = nodecodes.glue
249local disc_code          <const> = nodecodes.disc
250local math_code          <const> = nodecodes.math
251local dir_code           <const> = nodecodes.dir
252local par_code           <const> = nodecodes.par
253
254local lefttoright_code   <const> = tex.directioncodes.lefttoright
255local righttoleft_code   <const> = tex.directioncodes.righttoleft
256
257local discretionarydisc_code <const> = nodes.disccodes.discretionary
258
259local injections         = nodes.injections
260local setmark            = injections.setmark
261local setcursive         = injections.setcursive
262local setkern            = injections.setkern
263local setmove            = injections.setmove
264local setposition        = injections.setposition
265local resetinjection     = injections.reset
266local copyinjection      = injections.copy
267local setligaindex       = injections.setligaindex
268local getligaindex       = injections.getligaindex
269
270local fontdata           = fonts.hashes.identifiers
271local fontfeatures       = fonts.hashes.features
272
273local otffeatures        = fonts.constructors.features.otf
274local registerotffeature = otffeatures.register
275
276local onetimemessage     = fonts.loggers.onetimemessage or function() end
277
278local getrandom          = utilities and utilities.randomizer and utilities.randomizer.get
279
280otf.defaultnodealternate = "none" -- first last
281
282-- We use a few semi-global variables. The handler can be called nested but this assumes
283-- that the same font is used.
284
285local tfmdata            = false
286local characters         = false
287local descriptions       = false
288local marks              = false
289local classes            = false
290local currentfont        = false
291local currentdynamic     = false
292local currentscale       = 1000
293local currentxscale      = 1000
294local currentyscale      = 1000
295local factor             = 0
296local threshold          = 0
297local checkmarks         = false
298
299local discs              = false
300local spaces             = false
301
302local sweepnode          = nil
303local sweephead          = { } -- we don't nil entries but false them (no collection and such)
304
305local notmatchpre        = { } -- to be checked: can we use false instead of nil / what if a == b tests
306local notmatchpost       = { } -- to be checked: can we use false instead of nil / what if a == b tests
307local notmatchreplace    = { } -- to be checked: can we use false instead of nil / what if a == b tests
308
309local handlers           = { }
310
311local isspace            = injections.isspace
312local getthreshold       = injections.getthreshold
313
314local checkstep          = (tracers and tracers.steppers.check)    or function() end
315local registerstep       = (tracers and tracers.steppers.register) or function() end
316local registermessage    = (tracers and tracers.steppers.message)  or function() end
317
318do
319
320    local setchr = setchar
321    local direct = nil
322
323    setchar = function(n,c)
324        direct = fonts.collections.direct
325        setchar = function(n,c)
326            setchr(n,c)
327            direct(n)
328        end
329        setchar(n,c)
330    end
331
332end
333
334local function logprocess(...)
335    if trace_steps then
336        registermessage(...)
337        if trace_steps == "silent" then
338            return
339        end
340    end
341    report_direct(...)
342end
343
344local function logwarning(...)
345    report_direct(...)
346end
347
348local gref  do
349
350    local f_unicode = formatters["U+%X"]      -- was ["%U"]
351    local f_uniname = formatters["U+%X (%s)"] -- was ["%U (%s)"]
352    local f_unilist = formatters["% t"]
353
354    gref = function(n) -- currently the same as in font-otb
355        if type(n) == "number" then
356            local description = descriptions[n]
357            local name = description and description.name
358            if name then
359                return f_uniname(n,name)
360            else
361                return f_unicode(n)
362            end
363        elseif n then
364            local t = { }
365            for i=1,#n do
366                local ni = n[i]
367                if tonumber(ni) then -- later we will start at 2
368                    local di = descriptions[ni]
369                    local nn = di and di.name
370                    if nn then
371                        t[#t+1] = f_uniname(ni,nn)
372                    else
373                        t[#t+1] = f_unicode(ni)
374                    end
375                end
376            end
377            return f_unilist(t)
378        else
379            return "<error in node mode tracing>"
380        end
381    end
382
383end
384
385local function cref(dataset,sequence,index)
386    if not dataset then
387        return "no valid dataset"
388    end
389    local merged = sequence.merged and "merged " or ""
390    if index and index > 1 then
391        return formatters["feature %a, type %a, %schain lookup %a, index %a"](
392            dataset[4],sequence.type,merged,sequence.name,index)
393    else
394        return formatters["feature %a, type %a, %schain lookup %a"](
395            dataset[4],sequence.type,merged,sequence.name)
396    end
397end
398
399local function pref(dataset,sequence)
400    return formatters["feature %a, type %a, %slookup %a"](
401        dataset[4],sequence.type,sequence.merged and "merged " or "",sequence.name)
402end
403
404local function mref(rlmode)
405    if not rlmode or rlmode >= 0 then
406        return "l2r"
407    else
408        return "r2l"
409    end
410end
411
412-- The next code is somewhat complicated by the fact that some fonts can have ligatures made
413-- from ligatures that themselves have marks. This was identified by Kai in for instance
414-- arabtype:  KAF LAM SHADDA ALEF FATHA (0x0643 0x0644 0x0651 0x0627 0x064E). This becomes
415-- KAF LAM-ALEF with a SHADDA on the first and a FATHA op de second component. In a next
416-- iteration this becomes a KAF-LAM-ALEF with a SHADDA on the second and a FATHA on the
417-- third component.
418
419-- We can assume that languages that use marks are not hyphenated. We can also assume
420-- that at most one discretionary is present.
421
422-- We do need components in funny kerning mode but maybe I can better reconstruct then
423-- as we do have the font components info available; removing components makes the
424-- previous code much simpler. Also, later on copying and freeing becomes easier.
425-- However, for arabic we need to keep them around for the sake of mark placement
426-- and indices.
427
428local function flattendisk(head,disc) -- happen seldom, otherwise can be a helper
429    local pre, post, replace, pretail, posttail, replacetail = getdisc(disc,true)
430    local prev, next = getboth(disc)
431    local ishead = head == disc
432    setdisc(disc)
433    flushnode(disc)
434    if pre then
435        flushnodelist(pre)
436    end
437    if post then
438        flushnodelist(post)
439    end
440    if ishead then
441        if replace then
442            if next then
443                setlink(replacetail,next)
444            end
445            return replace, replace
446        elseif next then
447            return next, next
448        else
449         -- return -- maybe warning
450        end
451    else
452        if replace then
453            if next then
454                setlink(replacetail,next)
455            end
456            setlink(prev,replace)
457            return head, replace
458        else
459            setlink(prev,next) -- checks for next anyway
460            return head, next
461        end
462    end
463end
464
465local function appenddisc(disc,list) -- happen seldom, otherwise can be a helper
466    local pre, post, replace, pretail, posttail, replacetail = getdisc(disc,true)
467    local posthead    = list
468    local replacehead = copynodelist(list)
469    if post then
470        setlink(posttail,posthead)
471    else
472        post = posthead
473    end
474    if replace then
475        setlink(replacetail,replacehead)
476    else
477        replace = replacehead
478    end
479    setdisc(disc,pre,post,replace)
480end
481
482local function markstoligature(head,start,stop,char)
483    if start == stop and getchar(start) == char then
484        return head, start
485    else
486        local prev = getprev(start)
487        local next = getnext(stop)
488        setprev(start)
489        setnext(stop)
490        local base = copynocomponents(start,copyinjection)
491        if head == start then
492            head = base
493        end
494        resetinjection(base)
495        setchar(base,char)
496        setcomponents(base,start)
497        setlink(prev,base,next)
498        flushcomponents(start)
499        return head, base
500    end
501end
502
503-- Remark for Kai: (some arabic fonts do mark + mark = other mark and such)
504--
505-- The hasmarks is needed for ligatures of marks that are part of a ligature in
506-- which case we assume that we can delete the marks anyway (we can always become
507-- more clever if needed) .. in fact the whole logic here should be redone. We're
508-- in the not discfound branch then. We now have skiphash too so we can be more
509-- selective if needed (todo).
510
511-- we can have more granularity here but for now we only do a simple check
512
513local glyphoptioncodes       = tex.glyphoptioncodes
514
515local no_left_ligature_code  <const> = glyphoptioncodes.noleftligature
516local no_right_ligature_code <const> = glyphoptioncodes.norightligature
517----- no_left_kern_code      <const> = glyphoptioncodes.noleftkern
518local no_right_kern_code     <const> = glyphoptioncodes.norightkern
519
520local hasglyphoption         = nuts.hasglyphoption
521
522-- in lmtx we need to check the components and can be slightly more clever
523
524local function inhibited(start,stop)
525    for n in nextnode, start do
526        -- we asume glyph nodes
527        if n == start then
528            if hasglyphoption(n,no_right_ligature_code) then
529                return true
530            end
531        elseif n == stop then
532            if hasglyphoption(n,no_left_ligature_code) then
533                return true
534            else
535                return false
536            end
537        elseif hasglyphoption(n,no_left_ligature_code) then
538            return true
539        elseif hasglyphoption(n,no_right_ligature_code) then
540            return true
541        end
542    end
543    return false
544end
545
546local function toligature(head,start,stop,char,dataset,sequence,skiphash,discfound,hasmarks) -- brr head
547    if start == stop and getchar(start) == char and not hasmarks then
548        resetinjection(start)
549        setchar(start,char)
550     -- fonts.collections.direct(start)
551        return head, start
552    end
553    if inhibited(start,stop) then
554        return head, start
555    end
556    local prev = getprev(start)
557    local next = getnext(stop)
558    local comp = start
559    setprev(start)
560    setnext(stop)
561    local base = copynocomponents(start,copyinjection)
562    if start == head then
563        head = base
564    end
565    resetinjection(base)
566    setchar(base,char)
567 -- fonts.collections.direct(base)
568    setoptions(base,getoptions(start) | getoptions(stop)) -- maybe only lig options
569    setcomponents(base,comp)
570    setlink(prev,base,next)
571    if not discfound then
572        local deletemarks = not skiphash or hasmarks
573        local components = start -- not used
574        local baseindex = 0
575        local componentindex = 0
576        local head = base
577        local current = base
578        -- first we loop over the glyphs in start ... stop
579        while start do
580            local char = getchar(start)
581            if not marks[char] then
582                baseindex = baseindex + componentindex
583                componentindex = countcomponents(start,marks)
584             -- we can be more clever here: "not deletemarks or (skiphash and not skiphash[char])"
585             -- and such:
586            elseif not deletemarks or (skiphash and skiphash[char]) then
587                -- we can get a loop when the font expects otherwise (i.e. unexpected deletemarks)
588                setligaindex(start,baseindex + getligaindex(start,componentindex))
589                if trace_marks then
590                    logwarning("%s: keep ligature mark %s, gets index %s",pref(dataset,sequence),gref(char),getligaindex(start))
591                end
592                local n = copynode(start)
593                copyinjection(n,start) -- is this ok ? we position later anyway
594                head, current = insertnodeafter(head,current,n) -- unlikely that mark has components
595            elseif trace_marks then
596                logwarning("%s: delete ligature mark %s",pref(dataset,sequence),gref(char))
597            end
598            start = getnext(start)
599        end
600        -- we can have one accent as part of a lookup and another following
601        local start = getnext(current)
602        while start do
603            local nxt, char = isnextchar(start,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
604            if char then
605                -- also something skiphash here?
606                if marks[char] then
607                    setligaindex(start,baseindex + getligaindex(start,componentindex))
608                    if trace_marks then
609                        logwarning("%s: set ligature mark %s, gets index %s",pref(dataset,sequence),gref(char),getligaindex(start))
610                    end
611                    start = nxt
612                else
613                    break
614                end
615            else
616                break
617            end
618        end
619        flushcomponents(components)
620    else
621        -- discfound ... forget about marks .. probably no scripts that hyphenate and have marks
622        local discprev, discnext = getboth(discfound)
623        if discprev and discnext then
624            -- we assume normalization in context, and don't care about generic ... especially
625            -- \- can give problems as there we can have a negative char but that won't match
626            -- anyway
627            local pre, post, replace, pretail, posttail, replacetail = getdisc(discfound,true)
628            if not replace then
629                -- looks like we never come here as it's not okay
630                local prev = getprev(base)
631             -- local comp = getcomponents(base) -- already set
632                local copied = copyonlyglyphs(comp)
633                if pre then
634                    setlink(discprev,pre)
635                else
636                    setnext(discprev) -- also blocks funny assignments
637                end
638                pre = comp -- is start
639                if post then
640                    setlink(posttail,discnext)
641                    setprev(post) -- nil anyway
642                else
643                    post = discnext
644                    setprev(discnext) -- also blocks funny assignments
645                end
646                setlink(prev,discfound,next)
647                setboth(base)
648                -- here components have a pointer so we can't free it!
649                setcomponents(base,copied)
650                replace = base
651                if forcediscretionaries then
652                    setdisc(discfound,pre,post,replace,discretionarydisc_code)
653                else
654                    setdisc(discfound,pre,post,replace)
655                end
656                base = prev
657            end
658        end
659    end
660    return head, base
661end
662
663local function multiple_glyphs(head,start,multiple,skiphash,what,stop) -- what to do with skiphash matches here
664    local nofmultiples = #multiple
665    if nofmultiples > 0 then
666        local first = start
667        resetinjection(start)
668        setchar(start,multiple[1])
669        if nofmultiples > 1 then
670         -- local sn = getnext(start)
671            for i=2,nofmultiples do
672             -- untested:
673             --
674             -- while ignoremarks and marks[getchar(sn)] then
675             --     local sn = getnext(sn)
676             -- end
677                local n = copynode(start) -- ignore components
678                resetinjection(n)
679                setchar(n,multiple[i])
680                insertnodeafter(head,start,n)
681                start = n
682            end
683        end
684        if what ~= true then
685            -- This is just some experimental code; we might introduce gsub_extensible
686            -- some day instead. Beware: when we have a feature that mixes alternates and
687            -- multiples we need to make sure we don't handle the alternate string values
688            -- here.
689            local kind = type(what)
690            local m, f, l
691            if kind == "string" then
692                local what, n = string.match(what,"^repeat(.-)[:=](%d+)$")
693                if what == "middle" then
694                    m = tonumber(n)
695                elseif what == "first" then
696                    f = tonumber(n)
697                elseif what == "last" then
698                    l = tonumber(n)
699                end
700            elseif kind == "table" then
701                -- won't happen because currently we don't split these values
702               m = what.middle
703               f = what.first
704               l = what.last
705            end
706            if f or m or l then
707                if m and m > 1 and nofmultiples == 3 then
708                    local middle = getnext(first)
709                    for i=2,m do
710                        local n = copynode(middle) -- ignore components
711                        resetinjection(n)
712                        insertnodeafter(head,first,n)
713                    end
714                end
715                if f and f > 1 then
716                    for i=2,f do
717                        local n = copynode(first) -- ignore components
718                        resetinjection(n)
719                        insertnodeafter(head,first,n)
720                    end
721                end
722                if l and l > 1 then
723                    for i=2,l do
724                        local n = copynode(start) -- ignore components
725                        resetinjection(n)
726                        insertnodeafter(head,start,n)
727                        start = n
728                    end
729                end
730            end
731        end
732        return head, start, true
733    else
734        if trace_multiples then
735            logprocess("no multiple for %s",gref(getchar(start)))
736        end
737        return head, start, false
738    end
739end
740
741local function get_alternative_glyph(start,alternatives,value)
742    local n = #alternatives
743    if n == 0 then
744        return false, trace_alternatives and "0 (not okay)"
745    elseif n == 1 then
746        -- we could actually change that into a gsub and save some memory in the
747        -- font loader but it makes tracing more messy
748        return alternatives[1], trace_alternatives and "1 (only one present)"
749    elseif value == "random" then
750        local r = getrandom and getrandom("glyph",1,n) or random(1,n)
751        return alternatives[r], trace_alternatives and formatters["value %a, taking %a"](value,r)
752    elseif value == "first" then
753        return alternatives[1], trace_alternatives and formatters["value %a, taking %a"](value,1)
754    elseif value == "last" then
755        return alternatives[n], trace_alternatives and formatters["value %a, taking %a"](value,n)
756    end
757    value = value == true and 1 or tonumber(value)
758    if type(value) ~= "number" then
759        return alternatives[1], trace_alternatives and formatters["invalid value %s, taking %a"](value,1)
760    end
761 -- local a = alternatives[value]
762 -- if a then
763 --     -- some kind of hash
764 --     return a, trace_alternatives and formatters["value %a, taking %a"](value,a)
765 -- end
766    if value > n then
767        local defaultalt = otf.defaultnodealternate
768        if defaultalt == "first" then
769            return alternatives[n], trace_alternatives and formatters["invalid value %s, taking %a"](value,1)
770        elseif defaultalt == "last" then
771            return alternatives[1], trace_alternatives and formatters["invalid value %s, taking %a"](value,n)
772        else
773            return false, trace_alternatives and formatters["invalid value %a, %s"](value,"out of range")
774        end
775    elseif value == 0 then
776        return getchar(start), trace_alternatives and formatters["invalid value %a, %s"](value,"no change")
777    elseif value < 1 then
778        return alternatives[1], trace_alternatives and formatters["invalid value %a, taking %a"](value,1)
779    else
780        return alternatives[value], trace_alternatives and formatters["value %a, taking %a"](value,value)
781    end
782end
783
784-- handlers
785
786function handlers.gsub_single(head,start,dataset,sequence,replacement)
787    if trace_singles then
788        logprocess("%s: replacing %s by single %s",pref(dataset,sequence),gref(getchar(start)),gref(replacement))
789    end
790    resetinjection(start)
791    setchar(start,replacement)
792    return head, start, true
793end
794
795function handlers.gsub_alternate(head,start,dataset,sequence,alternative)
796    local kind  = dataset[4]
797    local what  = dataset[1]
798    local value = what == true and tfmdata.shared.features[kind] or what
799    local choice, comment = get_alternative_glyph(start,alternative,value)
800    if choice then
801        if trace_alternatives then
802            logprocess("%s: replacing %s by alternative %a to %s, %s",pref(dataset,sequence),gref(getchar(start)),gref(choice),comment)
803        end
804        resetinjection(start)
805        setchar(start,choice)
806    else
807        if trace_alternatives then
808            logwarning("%s: no variant %a for %s, %s",pref(dataset,sequence),value,gref(getchar(start)),comment)
809        end
810    end
811    return head, start, true
812end
813
814function handlers.gsub_multiple(head,start,dataset,sequence,multiple,rlmode,skiphash)
815    if trace_multiples then
816        logprocess("%s: replacing %s by multiple %s",pref(dataset,sequence),gref(getchar(start)),gref(multiple))
817    end
818    return multiple_glyphs(head,start,multiple,skiphash,dataset[1])
819end
820
821-- Don't we deal with disc otherwise now? I need to check if the next one can be
822-- simplified. Anyway, it can be way messier: marks that get removed as well as
823-- marks that are kept.
824
825-- Blocking is nasty: (ff\zwj l) vs (ff\zwj l) vs (ffl) vs (f\zwj fl}
826
827function handlers.gsub_ligature(head,start,dataset,sequence,ligature,rlmode,skiphash)
828    if start and hasglyphoption(start,no_right_ligature_code) then
829        return head, start, false, nil
830    end
831    local current = getnext(start)
832    if not current then
833        return head, start, false, nil
834    end
835    local stop      = nil
836    local startchar = getchar(start)
837    if skiphash and skiphash[startchar] then
838        while current do
839            local nxt, char = isnextchar(current,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
840            if char and not hasglyphoption(current,no_left_ligature_code) then
841                local lg = not tonumber(ligature) and ligature[char]
842                if lg then
843                    stop     = current
844                    ligature = lg
845                    current  = nxt
846                else
847                    break
848                end
849            else
850                break
851            end
852        end
853        if stop then
854            local ligature = tonumber(ligature) or ligature.ligature
855            if ligature then
856                if trace_ligatures then
857                    local stopchar = getchar(stop)
858                    head, start = markstoligature(head,start,stop,ligature)
859                    logprocess("%s: replacing %s upto %s by ligature %s case 1",pref(dataset,sequence),gref(startchar),gref(stopchar),gref(getchar(start)))
860                else
861                    head, start = markstoligature(head,start,stop,ligature)
862                end
863                return head, start, true, false
864            else
865                -- ok, goto next lookup
866            end
867        end
868    else
869        local discfound = false
870        local hasmarks  = marks[startchar]
871        while current do
872            local nxt, char, id = isnextchar(current,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
873            if char then
874                if skiphash and skiphash[char] then
875                    current = nxt
876                elseif hasglyphoption(current,no_left_ligature_code) then
877                    break
878                else
879                    local lg = not tonumber(ligature) and ligature[char]
880                    if lg then
881                        if marks[char] then
882                            hasmarks = true
883                        end
884                        stop     = current -- needed for fake so outside then
885                        ligature = lg
886                        current  = nxt
887                    else
888                        break
889                    end
890                end
891            elseif char == false then
892                -- kind of weird
893                break
894            elseif id == disc_code then
895                discfound = current
896                break
897            else
898                break
899            end
900        end
901        -- of{f-}{}{f}e  o{f-}{}{f}fe  o{-}{}{ff}e (oe and ff ligature)
902        -- we can end up here when we have a start run .. testruns start at a disc but
903        -- so here we have the other case: char + disc
904        --
905        -- Challenge for Kai (latinmodern):  \hyphenation{fii-f-f-iif} fiiffiif
906        --
907        if tonumber(ligature) then
908            -- we're already done (endpoint reached)
909        elseif discfound then
910            -- todo: check for no right ligature
911            -- don't assume marks in a disc and we don't run over a disc (for now)
912            local pre, post, replace = getdisc(discfound)
913            local match
914            if replace then
915                local nxt, char = isnextchar(replace,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
916                if char and ligature[char] and not hasglyphoption(replace,no_left_ligature_code) then
917                    match = true
918                end
919            end
920            if not match and pre then
921                local nxt, char = isnextchar(pre,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
922                if char and ligature[char] and not hasglyphoption(pre,no_left_ligature_code) then
923                    match = true
924                end
925            end
926            -- is this test ok:
927            if not match and not pre or not replace then
928                local ndf = getnext(discfound)
929                local nxt, char = isnextchar(ndf,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
930                if char and ligature[char] and not hasglyphoption(ndf,no_left_ligature_code) then
931                    match = true
932                end
933            end
934            if match then
935                -- we force a restart
936                local ishead = head == start
937                local prev   = getprev(start)
938                if stop then
939                    setnext(stop)
940                    local copy = copynodelist(start)
941                    local tail = stop -- was: getprev(stop) -- Kai: needs checking on your samples
942                    local liat = findnodetail(copy)
943                    if pre then
944                        setlink(liat,pre)
945                    end
946                    if replace then
947                        setlink(tail,replace)
948                    end
949                    pre     = copy
950                    replace = start
951                else
952                    setnext(start)
953                    local copy = copynode(start)
954                    if pre then
955                        setlink(copy,pre)
956                    end
957                    if replace then
958                        setlink(start,replace)
959                    end
960                    pre     = copy
961                    replace = start
962                end
963                setdisc(discfound,pre,post,replace)
964                if prev then
965                    setlink(prev,discfound)
966                else
967                    setprev(discfound)
968                    head  = discfound
969                end
970                start = discfound
971                return head, start, true, true
972            end
973            ligature = ligature.ligature
974        else
975            ligature = ligature.ligature
976        end
977     -- local ligature = tonumber(ligature) or ligature.ligature
978        if ligature then
979            if stop then
980                if trace_ligatures then
981                    local stopchar = getchar(stop)
982                 -- head, start = toligature(head,start,stop,ligature,dataset,sequence,skiphash,discfound,hasmarks)
983                    head, start = toligature(head,start,stop,ligature,dataset,sequence,skiphash,false,hasmarks)
984                    logprocess("%s: replacing %s upto %s by ligature %s case 2",pref(dataset,sequence),gref(startchar),gref(stopchar),gref(ligature))
985                 -- we can have a rare case of multiple disc in a lig but that makes no sense language wise but if really
986                 -- needed we could backtrack if we're in a disc node
987                else
988                 -- head, start = toligature(head,start,stop,ligature,dataset,sequence,skiphash,discfound,hasmarks)
989                    head, start = toligature(head,start,stop,ligature,dataset,sequence,skiphash,false,hasmarks)
990                end
991            else
992                -- weird but happens (in some arabic font)
993                resetinjection(start)
994                setchar(start,ligature)
995                if trace_ligatures then
996                    logprocess("%s: replacing %s by (no real) ligature %s case 3",pref(dataset,sequence),gref(startchar),gref(ligature))
997                end
998            end
999            return head, start, true, false
1000        else
1001            -- weird but happens, pseudo ligatures ... just the components
1002        end
1003    end
1004    return head, start, false, false
1005end
1006
1007function handlers.gpos_single(head,start,dataset,sequence,kerns,rlmode,skiphash,step,injection)
1008    if hasglyphoption(start,no_right_kern_code) then
1009        return head, start, false
1010    else
1011        local startchar = getchar(start)
1012        local format    = step.format
1013        if format == "single" or type(kerns) == "table" then -- the table check can go
1014            local dx, dy, w, h = setposition(0,start,factor,rlmode,kerns,injection)
1015            if trace_kerns then
1016                logprocess("%s: shifting single %s by %s xy (%p,%p) and wh (%p,%p)",pref(dataset,sequence),gref(startchar),format,dx,dy,w,h)
1017            end
1018        else
1019            local k = (format == "move" and setmove or setkern)(start,factor,rlmode,kerns,injection)
1020            if trace_kerns then
1021                logprocess("%s: shifting single %s by %s %p",pref(dataset,sequence),gref(startchar),format,k)
1022            end
1023        end
1024        return head, start, true
1025    end
1026end
1027
1028function handlers.gpos_pair(head,start,dataset,sequence,kerns,rlmode,skiphash,step,injection)
1029    if hasglyphoption(start,no_right_kern_code) then
1030        return head, start, false
1031    else
1032        local snext = getnext(start)
1033        if not snext then
1034            return head, start, false
1035        else
1036            local prev = start
1037            while snext do
1038                local nxt, nextchar = isnextchar(snext,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
1039                if nextchar then
1040                    if skiphash and skiphash[nextchar] then -- includes marks too when flag
1041                        prev  = snext
1042                        snext = nxt
1043                    else
1044                        local krn = kerns[nextchar]
1045                        if not krn then
1046                            break
1047                        end
1048                        local format = step.format
1049                        if format == "pair" then
1050                            local a = krn[1]
1051                            local b = krn[2]
1052                            if a == true then
1053                                -- zero
1054                            elseif a then -- #a > 0
1055                                local x, y, w, h = setposition(1,start,factor,rlmode,a,injection)
1056                                if trace_kerns then
1057                                    local startchar = getchar(start)
1058                                    logprocess("%s: shifting first of pair %s and %s by xy (%p,%p) and wh (%p,%p) as %s",pref(dataset,sequence),gref(startchar),gref(nextchar),x,y,w,h,injection or "injections")
1059                                end
1060                            end
1061                            if b == true then
1062                                -- zero
1063                                start = snext -- cf spec
1064                            elseif b then -- #b > 0
1065                                local x, y, w, h = setposition(2,snext,factor,rlmode,b,injection)
1066                                if trace_kerns then
1067                                    local startchar = getchar(start)
1068                                    logprocess("%s: shifting second of pair %s and %s by xy (%p,%p) and wh (%p,%p) as %s",pref(dataset,sequence),gref(startchar),gref(nextchar),x,y,w,h,injection or "injections")
1069                                end
1070                                start = snext -- cf spec
1071                            elseif forcepairadvance then
1072                                start = snext -- for testing, not cf spec
1073                            end
1074                            return head, start, true
1075                        elseif krn ~= 0 then
1076                            local k = (format == "move" and setmove or setkern)(snext,factor,rlmode,krn,injection)
1077                            if trace_kerns then
1078                                logprocess("%s: inserting %s %p between %s and %s as %s",pref(dataset,sequence),format,k,gref(getchar(prev)),gref(nextchar),injection or "injections")
1079                            end
1080                            return head, start, true
1081                        else -- can't happen
1082                            break
1083                        end
1084                    end
1085                else
1086                    break
1087                end
1088            end
1089            return head, start, false
1090        end
1091    end
1092end
1093
1094-- We get hits on a mark, but we're not sure if the it has to be applied so we need
1095-- to explicitly test for basechar, baselig and basemark entries.
1096
1097function handlers.gpos_mark2base(head,start,dataset,sequence,markanchors,rlmode,skiphash)
1098    local markchar = getchar(start) -- combine with getprev
1099    if marks[markchar] then
1100        local base = getprev(start) -- [glyph] [start=mark]
1101        if base then
1102            local prv, basechar = isprevchar(base,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
1103            if basechar then
1104                if marks[basechar] then
1105                    -- todo: use prv
1106                    while base do
1107                        base = getprev(base)
1108                        if base then
1109                            prv, basechar = isprevchar(base,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
1110                            if basechar then
1111                                if not marks[basechar] then
1112                                    break
1113                                end
1114                            else
1115                                if trace_bugs then
1116                                    logwarning("%s: no base for mark %s, case %i",pref(dataset,sequence),gref(markchar),1)
1117                                end
1118                                return head, start, false
1119                            end
1120                        else
1121                            if trace_bugs then
1122                                logwarning("%s: no base for mark %s, case %i",pref(dataset,sequence),gref(markchar),2)
1123                            end
1124                            return head, start, false
1125                        end
1126                    end
1127                end
1128                local ba = markanchors[1][basechar]
1129                if ba then
1130                    local ma = markanchors[2]
1131                    local dx, dy, bound = setmark(start,base,factor,rlmode,ba,ma,characters[basechar],false,checkmarks)
1132                    if trace_marks then
1133                        logprocess("%s, bound %s, anchoring mark %s to basechar %s => (%p,%p)",
1134                            pref(dataset,sequence),bound,gref(markchar),gref(basechar),dx,dy)
1135                    end
1136                    return head, start, true
1137                elseif trace_bugs then
1138                 -- onetimemessage(currentfont,basechar,"no base anchors")
1139                    logwarning("%s: mark %s is not anchored to %s",pref(dataset,sequence),gref(markchar),gref(basechar))
1140                end
1141            elseif trace_bugs then
1142                logwarning("%s: nothing preceding, case %i",pref(dataset,sequence),1)
1143            end
1144        elseif trace_bugs then
1145            logwarning("%s: nothing preceding, case %i",pref(dataset,sequence),2)
1146        end
1147    elseif trace_bugs then
1148        logwarning("%s: mark %s is no mark",pref(dataset,sequence),gref(markchar))
1149    end
1150    return head, start, false
1151end
1152
1153function handlers.gpos_mark2ligature(head,start,dataset,sequence,markanchors,rlmode,skiphash)
1154    local markchar = getchar(start)
1155    if marks[markchar] then
1156        local base = getprev(start) -- [glyph] [optional marks] [start=mark]
1157        if base then
1158            local prv, basechar = isprevchar(base,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
1159            if basechar then
1160                if marks[basechar] then
1161                    -- todo: use prv
1162                    while base do
1163                        base = getprev(base)
1164                        if base then
1165                            prv, basechar = isprevchar(base,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
1166                            if basechar then
1167                                if not marks[basechar] then
1168                                    break
1169                                end
1170                            else
1171                                if trace_bugs then
1172                                    logwarning("%s: no base for mark %s, case %i",pref(dataset,sequence),gref(markchar),1)
1173                                end
1174                                return head, start, false
1175                            end
1176                        else
1177                            if trace_bugs then
1178                                logwarning("%s: no base for mark %s, case %i",pref(dataset,sequence),gref(markchar),2)
1179                            end
1180                            return head, start, false
1181                        end
1182                    end
1183                end
1184                local ba = markanchors[1][basechar]
1185                if ba then
1186                    local ma = markanchors[2]
1187                    if ma then
1188                        local index = getligaindex(start)
1189                        ba = ba[index]
1190                        if ba then
1191                            local dx, dy, bound = setmark(start,base,factor,rlmode,ba,ma,characters[basechar],false,checkmarks)
1192                            if trace_marks then
1193                                logprocess("%s, index %s, bound %s, anchoring mark %s to baselig %s at index %s => (%p,%p)",
1194                                    pref(dataset,sequence),index,bound,gref(markchar),gref(basechar),index,dx,dy)
1195                            end
1196                            return head, start, true
1197                        else
1198                            if trace_bugs then
1199                                logwarning("%s: no matching anchors for mark %s and baselig %s with index %a",pref(dataset,sequence),gref(markchar),gref(basechar),index)
1200                            end
1201                        end
1202                    end
1203                elseif trace_bugs then
1204                --  logwarning("%s: char %s is missing in font",pref(dataset,sequence),gref(basechar))
1205                    onetimemessage(currentfont,basechar,"no base anchors")
1206                end
1207            elseif trace_bugs then
1208                logwarning("%s: prev node is no char, case %i",pref(dataset,sequence),1)
1209            end
1210        elseif trace_bugs then
1211            logwarning("%s: prev node is no char, case %i",pref(dataset,sequence),2)
1212        end
1213    elseif trace_bugs then
1214        logwarning("%s: mark %s is no mark",pref(dataset,sequence),gref(markchar))
1215    end
1216    return head, start, false
1217end
1218
1219function handlers.gpos_mark2mark(head,start,dataset,sequence,markanchors,rlmode,skiphash)
1220    local markchar = getchar(start)
1221    if marks[markchar] then
1222        local base = getprev(start) -- [glyph] [basemark] [start=mark]
1223        local slc = getligaindex(start)
1224        if slc then -- a rather messy loop ... needs checking with husayni
1225            while base do
1226                local blc = getligaindex(base)
1227                if blc and blc ~= slc then
1228                    base = getprev(base)
1229                else
1230                    break
1231                end
1232            end
1233        end
1234        if base then
1235            local nxt, basechar = isnextchar(base,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
1236            if basechar then -- subtype test can go
1237                local ba = markanchors[1][basechar] -- slot 1 has been made copy of the class hash
1238                if ba then
1239                    local ma = markanchors[2]
1240                    local dx, dy, bound = setmark(start,base,factor,rlmode,ba,ma,characters[basechar],true,checkmarks)
1241                    if trace_marks then
1242                        logprocess("%s, bound %s, anchoring mark %s to basemark %s => (%p,%p)",
1243                            pref(dataset,sequence),bound,gref(markchar),gref(basechar),dx,dy)
1244                    end
1245                    return head, start, true
1246                end
1247            end
1248        end
1249    elseif trace_bugs then
1250        logwarning("%s: mark %s is no mark",pref(dataset,sequence),gref(markchar))
1251    end
1252    return head, start, false
1253end
1254
1255function handlers.gpos_cursive(head,start,dataset,sequence,exitanchors,rlmode,skiphash,step) -- to be checked
1256    local startchar = getchar(start)
1257    if marks[startchar] then
1258        if trace_cursive then
1259            logprocess("%s: ignoring cursive for mark %s",pref(dataset,sequence),gref(startchar))
1260        end
1261    else
1262        local nxt = getnext(start)
1263        while nxt do
1264            local n, nextchar = isnextchar(nxt,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
1265            if not nextchar then
1266                break
1267            elseif marks[nextchar] then -- always sequence.flags[1]
1268                nxt = n
1269            else
1270                local exit = exitanchors[3]
1271                if exit then
1272                    local entry = exitanchors[1][nextchar]
1273                    if entry then
1274                        entry = entry[2]
1275                        if entry then
1276                            local r2lflag = sequence.flags[4] -- mentioned in the standard
1277                            local dx, dy, bound = setcursive(start,nxt,factor,rlmode,exit,entry,characters[startchar],characters[nextchar],r2lflag)
1278                            if trace_cursive then
1279                                logprocess("%s: moving %s to %s cursive (%p,%p) using bound %s in %s mode",pref(dataset,sequence),gref(startchar),gref(nextchar),dx,dy,bound,mref(rlmode))
1280                            end
1281                            return head, start, true
1282                        end
1283                    end
1284                end
1285                break
1286            end
1287        end
1288    end
1289    return head, start, false
1290end
1291
1292-- I will implement multiple chain replacements once I run into a font that uses it.
1293-- It's not that complex to handle.
1294
1295local chainprocs = { }
1296
1297local function logprocess(...)
1298    if trace_steps then
1299        registermessage(...)
1300        if trace_steps == "silent" then
1301            return
1302        end
1303    end
1304    report_subchain(...)
1305end
1306
1307local logwarning = report_subchain
1308
1309local function logprocess(...)
1310    if trace_steps then
1311        registermessage(...)
1312        if trace_steps == "silent" then
1313            return
1314        end
1315    end
1316    report_chain(...)
1317end
1318
1319local logwarning = report_chain
1320
1321-- We could share functions but that would lead to extra function calls with many
1322-- arguments, redundant tests and confusing messages.
1323
1324-- The reversesub is a special case, which is why we need to store the replacements
1325-- in a bit weird way. There is no lookup and the replacement comes from the lookup
1326-- itself. It is meant mostly for dealing with Urdu.
1327
1328local function reversesub(head,start,stop,dataset,sequence,replacements,rlmode,skiphash)
1329    local char        = getchar(start)
1330    local replacement = replacements[char]
1331    if replacement then
1332        if trace_singles then
1333            logprocess("%s: single reverse replacement of %s by %s",cref(dataset,sequence),gref(char),gref(replacement))
1334        end
1335        resetinjection(start)
1336        setchar(start,replacement)
1337        return head, start, true
1338    else
1339        return head, start, false
1340    end
1341end
1342
1343
1344chainprocs.reversesub = reversesub
1345
1346-- This chain stuff is somewhat tricky since we can have a sequence of actions to be
1347-- applied: single, alternate, multiple or ligature where ligature can be an invalid
1348-- one in the sense that it will replace multiple by one but not neccessary one that
1349-- looks like the combination (i.e. it is the counterpart of multiple then). For
1350-- example, the following is valid:
1351--
1352--   xxxabcdexxx [single a->A][multiple b->BCD][ligature cde->E] xxxABCDExxx
1353--
1354-- Therefore we we don't really do the replacement here already unless we have the
1355-- single lookup case. The efficiency of the replacements can be improved by
1356-- deleting as less as needed but that would also make the code even more messy.
1357--
1358-- Here we replace start by a single variant.
1359--
1360-- To be done   : what if > 1 steps (example needed)
1361-- This is messy: do we need this disc checking also in alternates?
1362
1363local function reportzerosteps(dataset,sequence)
1364    logwarning("%s: no steps",cref(dataset,sequence))
1365end
1366
1367local function reportmoresteps(dataset,sequence)
1368    logwarning("%s: more than 1 step",cref(dataset,sequence))
1369end
1370
1371-- local function reportbadsteps(dataset,sequence)
1372--     logwarning("%s: bad step, no proper return values",cref(dataset,sequence))
1373-- end
1374
1375local function getmapping(dataset,sequence,currentlookup)
1376    local steps    = currentlookup.steps
1377    local nofsteps = currentlookup.nofsteps
1378    if nofsteps == 0 then
1379        reportzerosteps(dataset,sequence)
1380        currentlookup.mapping = false
1381        return false
1382    else
1383        if nofsteps > 1 then
1384            reportmoresteps(dataset,sequence)
1385        end
1386        local mapping = steps[1].coverage
1387        currentlookup.mapping = mapping
1388        currentlookup.format  = steps[1].format
1389        return mapping
1390    end
1391end
1392
1393function chainprocs.gsub_remove(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash,chainindex)
1394    if trace_chains then
1395        logprocess("%s: removing character %s",cref(dataset,sequence,chainindex),gref(getchar(start)))
1396    end
1397    head, start = removenode(head,start,true)
1398    return head, getprev(start), true
1399end
1400
1401function chainprocs.gsub_single(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash,chainindex)
1402    local mapping = currentlookup.mapping
1403    if mapping == nil then
1404        mapping = getmapping(dataset,sequence,currentlookup)
1405    end
1406    if mapping then
1407        local current = start
1408        while current do
1409            local nxt, currentchar = isnextchar(current,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
1410            if currentchar then
1411                local replacement = mapping[currentchar]
1412                if not replacement or replacement == "" then
1413                    if trace_bugs then
1414                        logwarning("%s: no single for %s",cref(dataset,sequence,chainindex),gref(currentchar))
1415                    end
1416                else
1417                    if trace_singles then
1418                        logprocess("%s: replacing single %s by %s",cref(dataset,sequence,chainindex),gref(currentchar),gref(replacement))
1419                    end
1420                    resetinjection(current)
1421                    setchar(current,replacement)
1422                end
1423                return head, start, true
1424            elseif currentchar == false then
1425                -- can't happen
1426                break
1427            elseif current == stop then
1428                break
1429            else
1430                current = nxt
1431            end
1432        end
1433    end
1434    return head, start, false
1435end
1436
1437-- Here we replace start by new glyph. First we delete the rest of the match.
1438
1439-- char_1 mark_1 -> char_x mark_1 (ignore marks)
1440-- char_1 mark_1 -> char_x
1441
1442-- to be checked: do we always have just one glyph?
1443-- we can also have alternates for marks
1444-- marks come last anyway
1445-- are there cases where we need to delete the mark
1446
1447function chainprocs.gsub_alternate(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash,chainindex)
1448    local mapping = currentlookup.mapping
1449    if mapping == nil then
1450        mapping = getmapping(dataset,sequence,currentlookup)
1451    end
1452    if mapping then
1453        local kind    = dataset[4]
1454        local what    = dataset[1]
1455        local value   = what == true and tfmdata.shared.features[kind] or what -- todo: optimize in ctx
1456        local current = start
1457        while current do
1458            local nxt, currentchar = isnextchar(current,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
1459            if currentchar then
1460                local alternatives = mapping[currentchar]
1461                if alternatives then
1462                    local choice, comment = get_alternative_glyph(current,alternatives,value)
1463                    if choice then
1464                        if trace_alternatives then
1465                            logprocess("%s: replacing %s by alternative %a to %s, %s",cref(dataset,sequence),gref(currentchar),choice,gref(choice),comment)
1466                        end
1467                        resetinjection(start)
1468                        setchar(start,choice)
1469                    else
1470                        if trace_alternatives then
1471                            logwarning("%s: no variant %a for %s, %s",cref(dataset,sequence),value,gref(currentchar),comment)
1472                        end
1473                    end
1474                end
1475                return head, start, true
1476            elseif currentchar == false then
1477                -- can't happen
1478                break
1479            elseif current == stop then
1480                break
1481            else
1482                current = nxt
1483            end
1484        end
1485    end
1486    return head, start, false
1487end
1488
1489-- Here we replace start by a sequence of new glyphs.
1490
1491function chainprocs.gsub_multiple(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash,chainindex)
1492    local mapping = currentlookup.mapping
1493    if mapping == nil then
1494        mapping = getmapping(dataset,sequence,currentlookup)
1495    end
1496    if mapping then
1497        local startchar   = getchar(start)
1498        local replacement = mapping[startchar]
1499        if not replacement or replacement == "" then
1500            if trace_bugs then
1501                logwarning("%s: no multiple for %s",cref(dataset,sequence),gref(startchar))
1502            end
1503        else
1504            if trace_multiples then
1505                logprocess("%s: replacing %s by multiple characters %s",cref(dataset,sequence),gref(startchar),gref(replacement))
1506            end
1507            return multiple_glyphs(head,start,replacement,skiphash,dataset[1],stop)
1508        end
1509    end
1510    return head, start, false
1511end
1512
1513-- When we replace ligatures we use a helper that handles the marks. I might change
1514-- this function (move code inline and handle the marks by a separate function). We
1515-- assume rather stupid ligatures (no complex disc nodes).
1516
1517-- compare to handlers.gsub_ligature which is more complex ... why
1518
1519function chainprocs.gsub_ligature(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash,chainindex)
1520    if start and hasglyphoption(start,no_right_ligature_code) then
1521        return head, start, false, 0, false
1522    end
1523    local mapping = currentlookup.mapping
1524    if mapping == nil then
1525        mapping = getmapping(dataset,sequence,currentlookup)
1526    end
1527    if mapping then
1528        local startchar = getchar(start)
1529        local ligatures = mapping[startchar]
1530        if not ligatures then
1531            if trace_bugs then
1532                logwarning("%s: no ligatures starting with %s",cref(dataset,sequence,chainindex),gref(startchar))
1533            end
1534        else
1535            local hasmarks        = marks[startchar]
1536            local current         = getnext(start)
1537            local discfound       = false
1538            local last            = stop
1539            local nofreplacements = 1
1540            while current do
1541                local nxt, char, id = isnextchar(current,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
1542                if char then
1543                    if current and hasglyphoption(current,no_left_ligature_code) then
1544                        break
1545                    end
1546                    if skiphash and skiphash[char] then -- marks
1547                        -- if current == stop then -- maybe add this
1548                        --     break
1549                        -- else
1550                            current = nxt
1551                        -- end
1552                    else
1553                        local lg = not tonumber(ligatures) and ligatures[char]
1554                        if lg then
1555                            ligatures       = lg
1556                            last            = current
1557                            nofreplacements = nofreplacements + 1
1558                            if marks[char] then
1559                                hasmarks = true
1560                            end
1561                            if current == stop then
1562                                break
1563                            else
1564                                current = nxt
1565                            end
1566                        else
1567                            break
1568                        end
1569                    end
1570                elseif id == disc_code then
1571                    if not discfound then
1572                        discfound = current
1573                    end
1574                    if current == stop then
1575                        break -- okay? or before the disc
1576                    else
1577                        current = nxt
1578                    end
1579                else
1580                    break
1581                end
1582            end
1583            -- we can avoid the tonumber as already done but messy
1584            local ligature = tonumber(ligatures) or ligatures.ligature
1585            if ligature then
1586                if chainindex then
1587                    stop = last
1588                end
1589                if trace_ligatures then
1590                    if start == stop then
1591                        logprocess("%s: replacing character %s by ligature %s case 3",cref(dataset,sequence,chainindex),gref(startchar),gref(ligature))
1592                    else
1593                        logprocess("%s: replacing character %s upto %s by ligature %s case 4",cref(dataset,sequence,chainindex),gref(startchar),gref(getchar(stop)),gref(ligature))
1594                    end
1595                end
1596                head, start = toligature(head,start,stop,ligature,dataset,sequence,skiphash,discfound,hasmarks)
1597                return head, start, true, nofreplacements, discfound
1598            elseif trace_bugs then
1599                if start == stop then
1600                    logwarning("%s: replacing character %s by ligature fails",cref(dataset,sequence,chainindex),gref(startchar))
1601                else
1602                    logwarning("%s: replacing character %s upto %s by ligature fails",cref(dataset,sequence,chainindex),gref(startchar),gref(getchar(stop)))
1603                end
1604            end
1605        end
1606    end
1607    return head, start, false, 0, false
1608end
1609
1610function chainprocs.gpos_single(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash,chainindex)
1611    -- we actually should check no_left_kern_code with next
1612    if not hasglyphoption(start,no_right_kern_code) then
1613        local mapping = currentlookup.mapping
1614        if mapping == nil then
1615            mapping = getmapping(dataset,sequence,currentlookup)
1616        end
1617        if mapping then
1618            local startchar = getchar(start)
1619            local kerns     = mapping[startchar]
1620            if kerns then
1621                local format = currentlookup.format
1622                if format == "single" then
1623                    local dx, dy, w, h = setposition(0,start,factor,rlmode,kerns) -- currentlookup.flags ?
1624                    if trace_kerns then
1625                        logprocess("%s: shifting single %s by %s (%p,%p) and correction (%p,%p)",cref(dataset,sequence),gref(startchar),format,dx,dy,w,h)
1626                    end
1627                else -- needs checking .. maybe no kerns format for single
1628                    local k = (format == "move" and setmove or setkern)(start,factor,rlmode,kerns,injection)
1629                    if trace_kerns then
1630                        logprocess("%s: shifting single %s by %s %p",cref(dataset,sequence),gref(startchar),format,k)
1631                    end
1632                end
1633                return head, start, true
1634            end
1635        end
1636    end
1637    return head, start, false
1638end
1639
1640function chainprocs.gpos_pair(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash,chainindex) -- todo: injections ?
1641    -- we actually should check no_left_kern_code with next
1642    if not hasglyphoption(start,no_right_kern_code) then
1643        local mapping = currentlookup.mapping
1644        if mapping == nil then
1645            mapping = getmapping(dataset,sequence,currentlookup)
1646        end
1647        if mapping then
1648            local snext = getnext(start)
1649            if snext then
1650                local startchar = getchar(start)
1651                local kerns     = mapping[startchar] -- always 1 step
1652                if kerns then
1653                    local prev = start
1654                    while snext do
1655                        local nxt, nextchar = isnextchar(snext,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
1656                        if not nextchar then
1657                            break
1658                        elseif skiphash and skiphash[nextchar] then
1659                            prev  = snext
1660                            snext = nxt
1661                        else
1662                            local krn = kerns[nextchar]
1663                            if not krn then
1664                                break
1665                            end
1666                            local format = currentlookup.format
1667                            if format == "pair" then
1668                                local a = krn[1]
1669                                local b = krn[2]
1670                                if a == true then
1671                                    -- zero
1672                                elseif a then
1673                                    local x, y, w, h = setposition(1,start,factor,rlmode,a,"injections") -- currentlookups flags?
1674                                    if trace_kerns then
1675                                        local startchar = getchar(start)
1676                                        logprocess("%s: shifting first of pair %s and %s by (%p,%p) and correction (%p,%p)",cref(dataset,sequence),gref(startchar),gref(nextchar),x,y,w,h)
1677                                    end
1678                                end
1679                                if b == true then
1680                                    -- zero
1681                                    start = snext -- cf spec
1682                                elseif b then -- #b > 0
1683                                    local x, y, w, h = setposition(2,snext,factor,rlmode,b,"injections")
1684                                    if trace_kerns then
1685                                        local startchar = getchar(start)
1686                                        logprocess("%s: shifting second of pair %s and %s by (%p,%p) and correction (%p,%p)",cref(dataset,sequence),gref(startchar),gref(nextchar),x,y,w,h)
1687                                    end
1688                                    start = snext -- cf spec
1689                                elseif forcepairadvance then
1690                                    start = snext -- for testing, not cf spec
1691                                end
1692                                return head, start, true
1693                            elseif krn ~= 0 then
1694                                local k = (format == "move" and setmove or setkern)(snext,factor,rlmode,krn)
1695                                if trace_kerns then
1696                                    logprocess("%s: inserting %s %p between %s and %s",cref(dataset,sequence),format,k,gref(getchar(prev)),gref(nextchar))
1697                                end
1698                                return head, start, true
1699                            else
1700                                break
1701                            end
1702                        end
1703                    end
1704                end
1705            end
1706        end
1707    end
1708    return head, start, false
1709end
1710
1711function chainprocs.gpos_mark2base(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash,chainindex)
1712    local mapping = currentlookup.mapping
1713    if mapping == nil then
1714        mapping = getmapping(dataset,sequence,currentlookup)
1715    end
1716    if mapping then
1717        local markchar = getchar(start)
1718        if marks[markchar] then
1719            local markanchors = mapping[markchar] -- always 1 step
1720            if markanchors then
1721                local base = getprev(start) -- [glyph] [start=mark]
1722                if base then
1723                    local prv, basechar = isprevchar(base,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
1724                    if basechar then
1725                        if marks[basechar] then
1726                            -- todo: use prv
1727                            while base do
1728                                base = getprev(base)
1729                                if base then
1730                                    local prv, basechar = isprevchar(base,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
1731                                    if basechar then
1732                                        if not marks[basechar] then
1733                                            break
1734                                        end
1735                                    else
1736                                        if trace_bugs then
1737                                            logwarning("%s: no base for mark %s, case %i",pref(dataset,sequence),gref(markchar),1)
1738                                        end
1739                                        return head, start, false
1740                                    end
1741                                else
1742                                    if trace_bugs then
1743                                        logwarning("%s: no base for mark %s, case %i",pref(dataset,sequence),gref(markchar),2)
1744                                    end
1745                                    return head, start, false
1746                                end
1747                            end
1748                        end
1749                        local ba = markanchors[1][basechar]
1750                        if ba then
1751                            local ma = markanchors[2]
1752                            if ma then
1753                                local dx, dy, bound = setmark(start,base,factor,rlmode,ba,ma,characters[basechar],false,checkmarks)
1754                                if trace_marks then
1755                                    logprocess("%s, bound %s, anchoring mark %s to basechar %s => (%p,%p)",
1756                                        cref(dataset,sequence),bound,gref(markchar),gref(basechar),dx,dy)
1757                                end
1758                                return head, start, true
1759                            end
1760                        end
1761                    elseif trace_bugs then
1762                        logwarning("%s: prev node is no char, case %i",cref(dataset,sequence),1)
1763                    end
1764                elseif trace_bugs then
1765                    logwarning("%s: prev node is no char, case %i",cref(dataset,sequence),2)
1766                end
1767            elseif trace_bugs then
1768                logwarning("%s: mark %s has no anchors",cref(dataset,sequence),gref(markchar))
1769            end
1770        elseif trace_bugs then
1771            logwarning("%s: mark %s is no mark",cref(dataset,sequence),gref(markchar))
1772        end
1773    end
1774    return head, start, false
1775end
1776
1777function chainprocs.gpos_mark2ligature(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash,chainindex)
1778    local mapping = currentlookup.mapping
1779    if mapping == nil then
1780        mapping = getmapping(dataset,sequence,currentlookup)
1781    end
1782    if mapping then
1783        local markchar = getchar(start)
1784        if marks[markchar] then
1785            local markanchors = mapping[markchar] -- always 1 step
1786            if markanchors then
1787                local base = getprev(start) -- [glyph] [optional marks] [start=mark]
1788                if base then
1789                    local prv, basechar = isprevchar(base,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
1790                    if basechar then
1791                        if marks[basechar] then
1792                            -- todo: use prv
1793                            while base do
1794                                base = getprev(base)
1795                                if base then
1796                                    local prv, basechar = isprevchar(base,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
1797                                    if basechar then
1798                                        if not marks[basechar] then
1799                                            break
1800                                        end
1801                                    else
1802                                        if trace_bugs then
1803                                            logwarning("%s: no base for mark %s, case %i",cref(dataset,sequence),markchar,1)
1804                                        end
1805                                        return head, start, false
1806                                    end
1807                                else
1808                                    if trace_bugs then
1809                                        logwarning("%s: no base for mark %s, case %i",cref(dataset,sequence),markchar,2)
1810                                    end
1811                                    return head, start, false
1812                                end
1813                            end
1814                        end
1815                        local ba = markanchors[1][basechar]
1816                        if ba then
1817                            local ma = markanchors[2]
1818                            if ma then
1819                                local index = getligaindex(start)
1820                                ba = ba[index]
1821                                if ba then
1822                                    local dx, dy, bound = setmark(start,base,factor,rlmode,ba,ma,characters[basechar],false,checkmarks)
1823                                    if trace_marks then
1824                                        logprocess("%s, bound %s, anchoring mark %s to baselig %s at index %s => (%p,%p)",
1825                                            cref(dataset,sequence),a or bound,gref(markchar),gref(basechar),index,dx,dy)
1826                                    end
1827                                    return head, start, true
1828                                end
1829                            end
1830                        end
1831                    elseif trace_bugs then
1832                        logwarning("%s, prev node is no char, case %i",cref(dataset,sequence),1)
1833                    end
1834                elseif trace_bugs then
1835                    logwarning("%s, prev node is no char, case %i",cref(dataset,sequence),2)
1836                end
1837            elseif trace_bugs then
1838                logwarning("%s, mark %s has no anchors",cref(dataset,sequence),gref(markchar))
1839            end
1840        elseif trace_bugs then
1841            logwarning("%s, mark %s is no mark",cref(dataset,sequence),gref(markchar))
1842        end
1843    end
1844    return head, start, false
1845end
1846
1847function chainprocs.gpos_mark2mark(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash,chainindex)
1848    local mapping = currentlookup.mapping
1849    if mapping == nil then
1850        mapping = getmapping(dataset,sequence,currentlookup)
1851    end
1852    if mapping then
1853        local markchar = getchar(start)
1854        if marks[markchar] then
1855            local markanchors = mapping[markchar] -- always 1 step
1856            if markanchors then
1857                local base = getprev(start) -- [glyph] [basemark] [start=mark]
1858                local slc = getligaindex(start)
1859                if slc then -- a rather messy loop ... needs checking with husayni
1860                    while base do
1861                        local blc = getligaindex(base)
1862                        if blc and blc ~= slc then
1863                            base = getprev(base)
1864                        else
1865                            break
1866                        end
1867                    end
1868                end
1869                if base then -- subtype test can go
1870                    local nxt, basechar = isnextchar(base,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
1871                    if basechar then
1872                        local ba = markanchors[1][basechar]
1873                        if ba then
1874                            local ma = markanchors[2]
1875                            if ma then
1876                                local dx, dy, bound = setmark(start,base,factor,rlmode,ba,ma,characters[basechar],true,checkmarks)
1877                                if trace_marks then
1878                                    logprocess("%s, bound %s, anchoring mark %s to basemark %s => (%p,%p)",
1879                                        cref(dataset,sequence),bound,gref(markchar),gref(basechar),dx,dy)
1880                                end
1881                                return head, start, true
1882                            end
1883                        end
1884                    elseif trace_bugs then
1885                        logwarning("%s: prev node is no mark, case %i",cref(dataset,sequence),1)
1886                    end
1887                elseif trace_bugs then
1888                    logwarning("%s: prev node is no mark, case %i",cref(dataset,sequence),2)
1889                end
1890            elseif trace_bugs then
1891                logwarning("%s: mark %s has no anchors",cref(dataset,sequence),gref(markchar))
1892            end
1893        elseif trace_bugs then
1894            logwarning("%s: mark %s is no mark",cref(dataset,sequence),gref(markchar))
1895        end
1896    end
1897    return head, start, false
1898end
1899
1900function chainprocs.gpos_cursive(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash,chainindex)
1901    local mapping = currentlookup.mapping
1902    if mapping == nil then
1903        mapping = getmapping(dataset,sequence,currentlookup)
1904    end
1905    if mapping then
1906        local startchar   = getchar(start)
1907        local exitanchors = mapping[startchar] -- always 1 step
1908        if exitanchors then
1909            if marks[startchar] then
1910                if trace_cursive then
1911                    logprocess("%s: ignoring cursive for mark %s",pref(dataset,sequence),gref(startchar))
1912                end
1913            else
1914                local nxt = getnext(start)
1915                while nxt do
1916                    local n, nextchar = isnextchar(nxt,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
1917                    if not nextchar then
1918                        break
1919                    elseif marks[nextchar] then
1920                        -- should not happen (maybe warning)
1921                        nxt = n
1922                    else
1923                        local exit = exitanchors[3]
1924                        if exit then
1925                            local entry = exitanchors[1][nextchar]
1926                            if entry then
1927                                entry = entry[2]
1928                                if entry then
1929                                    local r2lflag = sequence.flags[4] -- mentioned in the standard
1930                                    local dx, dy, bound = setcursive(start,nxt,factor,rlmode,exit,entry,characters[startchar],characters[nextchar],r2lflag)
1931                                    if trace_cursive then
1932                                        logprocess("%s: moving %s to %s cursive (%p,%p) using bound %s in %s mode",pref(dataset,sequence),gref(startchar),gref(nextchar),dx,dy,bound,mref(rlmode))
1933                                    end
1934                                    return head, start, true
1935                                end
1936                            end
1937                        elseif trace_bugs then
1938                            onetimemessage(currentfont,startchar,"no entry anchors")
1939                        end
1940                        break
1941                    end
1942                end
1943            end
1944        elseif trace_cursive and trace_details then
1945            logprocess("%s, cursive %s is already done",pref(dataset,sequence),gref(getchar(start)),alreadydone)
1946        end
1947    end
1948    return head, start, false
1949end
1950
1951-- what pointer to return, spec says stop
1952-- to be discussed ... is bidi changer a space?
1953-- elseif char == zwnj and sequence[n][32] then -- brrr
1954
1955local function show_skip(dataset,sequence,char,ck,class)
1956    logwarning("%s: skipping char %s, class %a, rule %a, lookuptype %a",cref(dataset,sequence),gref(char),class,ck[1],ck[8] or ck[2])
1957end
1958
1959-- A previous version had disc collapsing code in the (single sub) handler plus some
1960-- checking in the main loop, but that left the pre/post sequences undone. The best
1961-- solution is to add some checking there and backtrack when a replace/post matches
1962-- but it takes a bit of work to figure out an efficient way (this is what the
1963-- sweep* names refer to). I might look into that variant one day again as it can
1964-- replace some other code too. In that approach we can have a special version for
1965-- gub and pos which gains some speed. This method does the test and passes info to
1966-- the handlers. Here collapsing is handled in the main loop which also makes code
1967-- elsewhere simpler (i.e. no need for the other special runners and disc code in
1968-- ligature building). I also experimented with pushing preceding glyphs sequences
1969-- in the replace/pre fields beforehand which saves checking afterwards but at the
1970-- cost of duplicate glyphs (memory) but it's too much overhead (runtime).
1971--
1972-- In the meantime Kai had moved the code from the single chain into a more general
1973-- handler and this one (renamed to chaindisk) is used now. I optimized the code a
1974-- bit and brought it in sycn with the other code. Hopefully I didn't introduce
1975-- errors. Note: this somewhat complex approach is meant for fonts that implement
1976-- (for instance) ligatures by character replacement which to some extend is not
1977-- that suitable for hyphenation. I also use some helpers. This method passes some
1978-- states but reparses the list. There is room for a bit of speed up but that will
1979-- be done in the context version. (In fact a partial rewrite of all code can bring
1980-- some more efficiency.)
1981--
1982-- I didn't test it with extremes but successive disc nodes still can give issues
1983-- but in order to handle that we need more complex code which also slows down even
1984-- more. The main loop variant could deal with that: test, collapse, backtrack.
1985
1986local newuserkern = nuts.pool.newkern
1987local newboundary = nuts.pool.boundary
1988
1989local function checked(head)
1990    local current = head
1991    while current do
1992        if getid(current) == glue_code then
1993            local kern = newuserkern(getwidth(current))
1994            if head == current then
1995                local next = getnext(current)
1996                if next then
1997                    setlink(kern,next)
1998                end
1999                flushnode(current)
2000                head    = kern
2001                current = next
2002            else
2003                local prev, next = getboth(current)
2004                setlink(prev,kern,next)
2005                flushnode(current)
2006                current = next
2007            end
2008        else
2009            current = getnext(current)
2010        end
2011    end
2012    return head
2013end
2014
2015local function setdiscchecked(d,pre,post,replace)
2016    if pre     then pre     = checked(pre)     end
2017    if post    then post    = checked(post)    end
2018    if replace then replace = checked(replace) end
2019    setdisc(d,pre,post,replace)
2020end
2021
2022local function chainrun(head,start,last,dataset,sequence,rlmode,skiphash,ck,where)
2023
2024    local size         = ck[5] - ck[4] + 1
2025    local chainlookups = ck[6]
2026    local done         = false
2027    -- current match
2028    if chainlookups then
2029        -- Lookups can be like { 1, false, 3 } or { false, 2 } or basically anything and
2030        -- #lookups can be less than #current
2031
2032        if size == 1 then
2033
2034         -- if nofchainlookups > size then
2035         --     -- bad rules
2036         -- end
2037            local chainlookup = chainlookups[1]
2038            if chainlookup then
2039                for j=1,#chainlookup do
2040                    local chainstep = chainlookup[j]
2041                    if chainstep then
2042                        local chainkind = chainstep.type
2043                        local chainproc = chainprocs[chainkind]
2044                        if chainproc then
2045                            local ok
2046                         -- HH: chainindex 1 added here (for KAI to check too), there are weird ligatures e.g.
2047                         -- char + mark -> char where mark has to disappear
2048                         -- head, start, ok = chainproc(head,start,last,dataset,sequence,chainstep,rlmode,skiphash,1)
2049                            head, start, ok = chainproc(head,start,last,dataset,sequence,chainstep,rlmode,skiphash)
2050                            if ok then
2051                                done = true
2052                            end
2053                        else
2054                            logprocess("%s: %s is not yet supported (1)",cref(dataset,sequence),chainkind)
2055                        end
2056                    else
2057                        logprocess("%s: has an issue (1)",cref(dataset,sequence))
2058                    end
2059                end
2060            else
2061                -- whatever
2062            end
2063
2064         else
2065
2066            -- See LookupType 5: Contextual Substitution Subtable. Now it becomes messy. The
2067            -- easiest case is where #current maps on #lookups i.e. one-to-one. But what if
2068            -- we have a ligature. Cf the spec we then need to advance one character but we
2069            -- really need to test it as there are fonts out there that are fuzzy and have
2070            -- too many lookups:
2071            --
2072            -- U+1105 U+119E U+1105 U+119E : sourcehansansklight: script=hang ccmp=yes
2073            --
2074            -- Even worse are these family emoji shapes as they can have multiple lookups
2075            -- per slot (probably only for gpos).
2076
2077            -- It's very unlikely that we will have skip classes here but still ... we seldom
2078            -- enter this branch anyway.
2079
2080            local i = 1
2081            local laststart = start
2082            local nofchainlookups = #chainlookups -- useful?
2083            while start do
2084                if skiphash then -- hm, so we know we skip some
2085                    while start do
2086                        local nxt, char = isnextchar(start,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
2087                        if char then
2088                            if skiphash and skiphash[char] then
2089                                start = nxt
2090                            else
2091                                break
2092                            end
2093                        else
2094                            break
2095                        end
2096                    end
2097                end
2098                local chainlookup = chainlookups[i]
2099                if chainlookup then
2100                    for j=1,#chainlookup do
2101                        local chainstep = chainlookup[j]
2102                        if chainstep then
2103                            local chainkind = chainstep.type
2104                            local chainproc = chainprocs[chainkind]
2105                            if chainproc then
2106                                local ok, n
2107                                head, start, ok, n = chainproc(head,start,last,dataset,sequence,chainstep,rlmode,skiphash,i)
2108                                -- messy since last can be changed !
2109                                if ok then
2110                                    done = true
2111                                    if n and n > 1 and i + n > nofchainlookups then
2112                                        -- this is a safeguard, we just ignore the rest of the lookups
2113                                        i = size -- prevents an advance
2114                                        break
2115                                    end
2116                                end
2117                            else
2118                                -- actually an error
2119                                logprocess("%s: %s is not yet supported (2)",cref(dataset,sequence),chainkind)
2120                            end
2121                        else
2122                            -- actually an error
2123                            logprocess("%s: has an issue (2)",cref(dataset,sequence))
2124                        end
2125                    end
2126                else
2127                    -- we skip but we could also delete as option .. what does an empty lookup actually mean
2128                    -- in opentype ... anyway, we could map it onto gsub_remove if needed
2129                end
2130                i = i + 1
2131                if i > size or not start then
2132                    break
2133                elseif start then
2134                    laststart = start
2135                    start = getnext(start)
2136                end
2137            end
2138            if not start then
2139                start = laststart
2140            end
2141
2142        end
2143    else
2144        -- todo: needs checking for holes in the replacements
2145        local replacements = ck[7]
2146        if replacements then
2147            head, start, done = reversesub(head,start,last,dataset,sequence,replacements,rlmode,skiphash)
2148        else
2149            done = true
2150            if trace_contexts then
2151                logprocess("%s: skipping match @ %i",cref(dataset,sequence),where)
2152            end
2153        end
2154    end
2155    return head, start, done
2156end
2157
2158local function chaindisk(head,start,dataset,sequence,rlmode,skiphash,ck)
2159
2160    if not start then
2161        return head, start, false
2162    end
2163
2164    local startishead   = start == head
2165    local seq           = ck[3]
2166    local f             = ck[4]
2167    local l             = ck[5]
2168    local s             = #seq
2169    local done          = false
2170    local sweepnode     = sweepnode
2171    local sweeptype     = sweeptype
2172    local sweepoverflow = false
2173    local checkdisc     = getprev(head)
2174    local keepdisc      = not sweepnode
2175    local lookaheaddisc = nil
2176    local backtrackdisc = nil
2177    local current       = start
2178    local last          = start
2179    local prev          = getprev(start)
2180    local hasglue       = false
2181    local useddisc      = nil   -- new 2022-09-25
2182    local usedstart     = start -- new 2022-09-25
2183
2184    -- fishy: so we can overflow and then go on in the sweep?
2185    -- todo : id can also be glue_code as we checked spaces
2186
2187    local i = f
2188    while i <= l do
2189        local id = getid(current)
2190        if id == glyph_code then
2191            i       = i + 1
2192            last    = current
2193            current = getnext(current)
2194        elseif id == glue_code then
2195            i       = i + 1
2196            last    = current
2197            current = getnext(current)
2198            hasglue = true
2199        elseif id == disc_code then
2200            if keepdisc then
2201                keepdisc      = false
2202                lookaheaddisc = current
2203                local replace = getreplace(current)
2204                if not replace then
2205                    sweepoverflow = true
2206                    sweepnode     = current
2207                    current       = getnext(current)
2208                else
2209                    -- we can use an iterator
2210                    while replace and i <= l do
2211                        if getid(replace) == glyph_code then
2212                            i = i + 1
2213                        end
2214                        replace = getnext(replace)
2215                    end
2216                    current = getnext(replace)
2217                end
2218                last = current
2219            else
2220                head, current = flattendisk(head,current)
2221            end
2222        else
2223            last    = current
2224            current = getnext(current)
2225        end
2226        if current then
2227            -- go on
2228        elseif sweepoverflow then
2229            -- we already are following up on sweepnode
2230            break
2231        elseif sweeptype == "post" or sweeptype == "replace" then
2232            current = getnext(sweepnode)
2233            if current then
2234                sweeptype     = nil
2235                sweepoverflow = true
2236            else
2237                break
2238            end
2239        else
2240            break -- added
2241        end
2242    end
2243
2244    if sweepoverflow then
2245        local prev = current and getprev(current)
2246        if not current or prev ~= sweepnode then
2247            local head = getnext(sweepnode)
2248            local tail = nil
2249            if prev then
2250                tail = prev
2251                setprev(current,sweepnode)
2252            else
2253                tail = findnodetail(head)
2254            end
2255            setnext(sweepnode,current)
2256            setprev(head)
2257            setnext(tail)
2258            appenddisc(sweepnode,head)
2259        end
2260    end
2261
2262    if l < s then
2263        local i = l
2264        local t = sweeptype == "post" or sweeptype == "replace"
2265        while current and i < s do
2266            local id = getid(current)
2267            if id == glyph_code then
2268                i       = i + 1
2269                current = getnext(current)
2270            elseif id == glue_code then
2271                i       = i + 1
2272                current = getnext(current)
2273                hasglue = true
2274            elseif id == disc_code then
2275                if keepdisc then
2276                    keepdisc = false
2277                    if notmatchpre[current] ~= notmatchreplace[current] then
2278                        lookaheaddisc = current
2279                    end
2280                    -- we assume a simple text only replace (we could use nuts.count)
2281                    local replace = getreplace(current)
2282                    while replace and i < s do
2283                        if getid(replace) == glyph_code then
2284                            i = i + 1
2285                        end
2286                        replace = getnext(replace)
2287                    end
2288                    current = getnext(current)
2289                elseif notmatchpre[current] ~= notmatchreplace[current] then
2290                    head, current = flattendisk(head,current)
2291                else
2292                    current = getnext(current) -- HH
2293                end
2294            else
2295                current = getnext(current)
2296            end
2297            if not current and t then
2298                current = getnext(sweepnode)
2299                if current then
2300                    sweeptype = nil
2301                end
2302            end
2303        end
2304    end
2305
2306    if f > 1 then
2307        local current = prev
2308        local i       = f
2309        local t       = sweeptype == "pre" or sweeptype == "replace"
2310        if not current and t and current == checkdisc then
2311            current = getprev(sweepnode)
2312        end
2313        while current and i > 1 do -- missing getprev added / moved outside
2314            local id = getid(current)
2315            if id == glyph_code then
2316                i = i - 1
2317            elseif id == glue_code then
2318                i       = i - 1
2319                hasglue = true
2320            elseif id == disc_code then
2321                if keepdisc then
2322                    keepdisc = false
2323                    if notmatchpost[current] ~= notmatchreplace[current] then
2324-- print(notmatchpost[current],notmatchreplace[current])
2325                        backtrackdisc = current
2326                    end
2327                    -- we assume a simple text only replace (we could use nuts.count)
2328                    local replace = getreplace(current)
2329                    while replace and i > 1 do
2330                        if getid(replace) == glyph_code then
2331                            i = i - 1
2332                        end
2333                        replace = getnext(replace)
2334                    end
2335                elseif notmatchpost[current] ~= notmatchreplace[current] then
2336                    head, current = flattendisk(head,current)
2337                end
2338            end
2339            current = getprev(current)
2340            if t and current == checkdisc then
2341                current = getprev(sweepnode)
2342            end
2343        end
2344    end
2345
2346    local done = false
2347
2348    if lookaheaddisc then
2349        local cf            = start
2350        local cl            = getprev(lookaheaddisc)
2351        local cprev         = getprev(start)
2352        local insertedmarks = 0
2353        while cprev do
2354            local nxt, char = isnextchar(cf,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
2355            if char and marks[char] then
2356                insertedmarks = insertedmarks + 1
2357                cf            = cprev
2358                startishead   = cf == head
2359                cprev         = getprev(cprev)
2360            else
2361                break
2362            end
2363        end
2364        setlink(cprev,lookaheaddisc)
2365        setprev(cf)
2366        setnext(cl)
2367        if startishead then
2368            head = lookaheaddisc
2369        end
2370        local pre, post, replace = getdisc(lookaheaddisc)
2371        local new  = copynodelist(cf) -- br, how often does that happen
2372        local cnew = new
2373        if pre then
2374            setlink(findnodetail(cf),pre)
2375        end
2376        if replace then
2377            local tail = findnodetail(new)
2378            setlink(tail,replace)
2379        end
2380        for i=1,insertedmarks do
2381            cnew = getnext(cnew)
2382        end
2383        cl = start
2384        local clast = cnew
2385        for i=f,l do
2386            cl    = getnext(cl)
2387            clast = getnext(clast)
2388        end
2389        if not notmatchpre[lookaheaddisc] then
2390            local ok = false
2391            cf, start, ok = chainrun(cf,start,cl,dataset,sequence,rlmode,skiphash,ck,1)
2392            if ok then
2393                done = true
2394            end
2395        end
2396        if not notmatchreplace[lookaheaddisc] then
2397            local ok = false
2398            new, cnew, ok = chainrun(new,cnew,clast,dataset,sequence,rlmode,skiphash,ck,2)
2399            if ok then
2400                done = true
2401            end
2402        end
2403        if hasglue then
2404            setdiscchecked(lookaheaddisc,cf,post,new)
2405        else
2406            setdisc(lookaheaddisc,cf,post,new)
2407        end
2408        start          = getprev(lookaheaddisc)
2409if done then -- new 2022-12-22 : discretionary-001.tex
2410        useddisc       = lookaheaddisc -- new 2022-09-25
2411end
2412        sweephead[cf]  = getnext(clast) or false
2413        sweephead[new] = getnext(cl) or false
2414    elseif backtrackdisc then
2415        local cf            = getnext(backtrackdisc)
2416        local cl            = start
2417        local cnext         = getnext(start)
2418        local insertedmarks = 0
2419        while cnext do
2420            local nxt, char = isnextchar(cnext,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
2421            if char and marks[char] then
2422                insertedmarks = insertedmarks + 1
2423                cl            = cnext
2424                cnext         = nxt
2425            else
2426                break
2427            end
2428        end
2429        setlink(backtrackdisc,cnext)
2430        setprev(cf)
2431        setnext(cl)
2432        local pre, post, replace, pretail, posttail, replacetail = getdisc(backtrackdisc,true)
2433        local new  = copynodelist(cf)
2434        local cnew = findnodetail(new)
2435        for i=1,insertedmarks do
2436            cnew = getprev(cnew)
2437        end
2438        local clast = cnew
2439        for i=f,l do
2440            clast = getnext(clast)
2441        end
2442        if not notmatchpost[backtrackdisc] then
2443            local ok = false
2444            cf, start, ok = chainrun(cf,start,last,dataset,sequence,rlmode,skiphash,ck,3)
2445            if ok then
2446                done = true
2447            end
2448        end
2449        if not notmatchreplace[backtrackdisc] then
2450            local ok = false
2451            new, cnew, ok = chainrun(new,cnew,clast,dataset,sequence,rlmode,skiphash,ck,4)
2452            if ok then
2453                done = true
2454            end
2455        end
2456        if post then
2457            setlink(posttail,cf)
2458        else
2459            post = cf
2460        end
2461        if replace then
2462            setlink(replacetail,new)
2463        else
2464            replace = new
2465        end
2466        if hasglue then
2467            setdiscchecked(backtrackdisc,pre,post,replace)
2468        else
2469            setdisc(backtrackdisc,pre,post,replace)
2470        end
2471        start              = getprev(backtrackdisc)
2472if done then -- new 2022-12-22 : discretionary-001.tex
2473        useddisc           = backtrackdisc -- new 2022-09-25
2474end
2475        sweephead[post]    = getnext(clast) or false
2476        sweephead[replace] = getnext(last) or false
2477    else
2478
2479        local ok = false
2480        head, start, ok = chainrun(head,start,last,dataset,sequence,rlmode,skiphash,ck,5)
2481        if ok then
2482            done = true
2483        end
2484
2485    end
2486-- test case ?
2487    if useddisc and start ~= usedstart then -- make this option per font -- new 2022-09-25
2488       start = getnext(start)                                            -- new 2022-09-25
2489    end                                                                  -- new 2022-09-25
2490    return head, start, done, useddisc                                   -- new 2022-09-25
2491end
2492
2493local chaintrac do
2494
2495    local level = 0
2496    local last  = { }
2497
2498    chaintrac = function(head,start,dataset,sequence,rlmode,skiphash,ck,match,discseen,sweepnode)
2499        if dataset then
2500            level = level + 1
2501            last[level] = start
2502            local rule       = ck[1]
2503            local lookuptype = ck[8] or ck[2]
2504            local nofseq     = #ck[3] -- ck[3].n
2505            local first      = ck[4]
2506            local last       = ck[5]
2507            local char       = getchar(start)
2508            logwarning("+ %i : %s: rule %s %s at char %s for (%s,%s,%s) chars, lookuptype %a, %sdisc seen, %ssweeping",
2509                level,cref(dataset,sequence),rule,match and "matches" or "nomatch",
2510                gref(char),first-1,last-first+1,nofseq-last,lookuptype,
2511                discseen and "" or "no ", sweepnode and "" or "not ")
2512        else
2513            -- (start,done)
2514            local what  = start and "done" or "continue"
2515            local where = head == last[level] and "same" or "different"
2516            local char  = getchar(head)
2517            if char then
2518                logwarning("- %i : %s at char %s, %s node",level,what,gref(char),where)
2519            else
2520                logwarning("- %i : %s, %s node",level,what,where)
2521            end
2522            level = level - 1
2523        end
2524    end
2525
2526end
2527
2528-- The next one is quite optimized but still somewhat slow, fonts like ebgaramond
2529-- are real torture tests because they have many steps with one context (having
2530-- multiple contexts makes more sense) also because we (can) reduce them. Instead of
2531-- a match boolean variable and check for that I decided to use a goto with labels
2532-- instead. This is one of the cases where it makes the code more readable and we
2533-- might even gain a bit performance.
2534
2535-- when we have less replacements (lookups) then current matches we can push too much into
2536-- the previous disc .. such be it (<before><disc><current=fl><after> with only f done)
2537
2538local function handle_contextchain(head,start,dataset,sequence,contexts,rlmode,skiphash)
2539    if not contexts then
2540        return head, start, false
2541    end
2542    -- optimizing for rlmode gains nothing
2543    local sweepnode    = sweepnode
2544    local sweeptype    = sweeptype
2545    local postreplace
2546    local prereplace
2547    local checkdisc
2548    local discseen  -- = false
2549    if sweeptype then
2550        if sweeptype == "replace" then
2551            postreplace = true
2552            prereplace  = true
2553        else
2554            postreplace = sweeptype == "post"
2555            prereplace  = sweeptype == "pre"
2556        end
2557        checkdisc = getprev(head)
2558    end
2559 -- local currentfont   = currentfont   -- really ?
2560 -- local currentscale  = currentscale  -- really ?
2561 -- local currentxscale = currentxscale -- really ?
2562 -- local currentyscale = currentyscale -- really ?
2563
2564    local skipped   -- = false
2565
2566    local startprev,
2567          startnext    = getboth(start)
2568    local done      -- = false
2569
2570    -- we can have multiple hits and as we scan (currently) all we need to check
2571    -- if we have a match ... contextchains have no real coverage table (with
2572    -- unique entries)
2573
2574    -- fonts can have many steps (each doing one check) or many contexts
2575
2576    -- todo: make a per-char cache so that we have small contexts (when we have a context
2577    -- n == 1 and otherwise it can be more so we can even distinguish n == 1 or more)
2578
2579    local nofcontexts = contexts.n -- #contexts
2580
2581    local startchar = nofcontext == 1 or ischar(start,currentfont) -- already checked
2582    for k=1,nofcontexts do -- does this disc mess work well with n > 1
2583
2584        local ck   = contexts[k]
2585        local seq  = ck[3]
2586        local f    = ck[4] -- first current
2587        local last = start
2588        if not startchar or not seq[f][startchar] then
2589            -- report("no hit in %a at %i of %i contexts",sequence.type,k,nofcontexts)
2590            goto next
2591        end
2592        local s = seq.n -- or #seq
2593        if s == 1 then
2594            -- bit weird case: why use a chain, but it is a hit
2595        else
2596            local l       = ck[5] -- last current
2597            local current = start
2598         -- local last    = start
2599
2600            -- current match
2601
2602            if l > f then
2603                -- before/current/after | before/current | current/after
2604                local discfound -- = nil
2605                local n = f + 1
2606                last = startnext -- the second in current (first already matched)
2607                while n <= l do
2608                    if postreplace and not last then
2609                        last      = getnext(sweepnode)
2610                        sweeptype = nil
2611                    end
2612                    if last then
2613                        local nxt, char, id = isnextchar(last,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
2614                        if char then
2615                            if seq[n][char] then
2616                                if n < l then
2617                                    last = nxt
2618                                end
2619                                n = n + 1
2620                            elseif skiphash and skiphash[char] then
2621                                skipped = true
2622                                if trace_skips then
2623                                    show_skip(dataset,sequence,char,ck,classes[char])
2624                                end
2625                                last = nxt
2626                            elseif discfound then
2627                                notmatchreplace[discfound] = true
2628                                if notmatchpre[discfound] then
2629                                    goto next
2630                                else
2631                                    break
2632                                end
2633                            else
2634                                goto next
2635                            end
2636                        elseif char == false then
2637                            if discfound then
2638                                notmatchreplace[discfound] = true
2639                                if notmatchpre[discfound] then
2640                                    goto next
2641                                else
2642                                    break
2643                                end
2644                            else
2645                                goto next
2646                            end
2647                        elseif id == disc_code then
2648                     -- elseif id == disc_code and (not discs or discs[last]) then
2649                            discseen              = true
2650                            discfound             = last
2651                            notmatchpre[last]     = nil
2652                            notmatchpost[last]    = true
2653                            notmatchreplace[last] = nil
2654                            local pre, post, replace = getdisc(last)
2655                            if pre then
2656                                local n = n
2657                                while pre do
2658                                    if seq[n][getchar(pre)] then
2659                                        n = n + 1
2660                                        if n > l then
2661                                            break
2662                                        end
2663                                        pre = getnext(pre)
2664                                    else
2665                                        notmatchpre[last] = true
2666                                        break
2667                                    end
2668                                end
2669                                if n <= l then
2670                                    notmatchpre[last] = true
2671                                end
2672                            else
2673                                notmatchpre[last] = true
2674                            end
2675                            if replace then
2676                                -- so far we never entered this branch
2677                                while replace do
2678                                    if seq[n][getchar(replace)] then
2679                                        n = n + 1
2680                                        if n > l then
2681                                            break
2682                                        end
2683                                        replace = getnext(replace)
2684                                    else
2685                                        notmatchreplace[last] = true
2686                                        if notmatchpre[last] then
2687                                            goto next
2688                                        else
2689                                            break
2690                                        end
2691                                    end
2692                                end
2693                                -- why here again
2694                                if notmatchpre[last] then
2695                                    goto next
2696                                end
2697                            end
2698                            -- maybe only if match
2699                            last = nxt
2700                        else
2701                            goto next
2702                        end
2703                    else
2704                        goto next
2705                    end
2706                end
2707            end
2708
2709            -- before
2710
2711            if f > 1 then
2712              -- if startprev then -- new 2022-09-25
2713                    local prev = startprev
2714                    if prereplace and prev == checkdisc then
2715                        prev = getprev(sweepnode)
2716                    end
2717                    if prev then
2718                        local discfound -- = nil
2719                        local n = f - 1
2720                        while n >= 1 do
2721                            if prev then
2722                                local prv, char, id = isprevchar(prev,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
2723                                if char then
2724                                   if seq[n][char] then
2725                                        if n > 1 then
2726                                            prev = prv
2727                                        end
2728                                        n = n - 1
2729                                    elseif skiphash and skiphash[char] then
2730                                        skipped = true
2731                                        if trace_skips then
2732                                            show_skip(dataset,sequence,char,ck,classes[char])
2733                                        end
2734                                        prev = prv
2735                                    elseif discfound then
2736                                        notmatchreplace[discfound] = true
2737                                        if notmatchpost[discfound] then
2738                                            goto next
2739                                        else
2740                                            break
2741                                        end
2742                                    else
2743                                        goto next
2744                                    end
2745                                elseif char == false then
2746                                    if discfound then
2747                                        notmatchreplace[discfound] = true
2748                                        if notmatchpost[discfound] then
2749                                            goto next
2750                                        end
2751                                    else
2752                                        goto next
2753                                    end
2754                                    break
2755                                elseif id == disc_code then
2756                             -- elseif id == disc_code and (not discs or discs[prev]) then
2757                                    -- the special case: f i where i becomes dottless i ..
2758                                    discseen              = true
2759                                    discfound             = prev
2760                                    notmatchpre[prev]     = true
2761                                    notmatchpost[prev]    = nil
2762                                    notmatchreplace[prev] = nil
2763                                    local pre, post, replace, pretail, posttail, replacetail = getdisc(prev,true)
2764                                    -- weird test: needs checking
2765                                    if pre ~= start and post ~= start and replace ~= start then
2766                                        if post then
2767                                            local n = n
2768                                            while posttail do
2769                                                if seq[n][getchar(posttail)] then
2770                                                    n = n - 1
2771                                                    if posttail == post or n < 1 then
2772                                                        break
2773                                                    else
2774                                                        posttail = getprev(posttail)
2775                                                    end
2776                                                else
2777                                                    notmatchpost[prev] = true
2778                                                    break
2779                                                end
2780                                            end
2781                                            if n >= 1 then
2782                                                notmatchpost[prev] = true
2783                                            end
2784                                        else
2785                                            notmatchpost[prev] = true
2786                                        end
2787                                        if replace then
2788                                            -- we seldom enter this branch (e.g. on brill efficient)
2789                                            while replacetail do
2790                                                if seq[n][getchar(replacetail)] then
2791                                                    n = n - 1
2792                                                    if replacetail == replace or n < 1 then
2793                                                        break
2794                                                    else
2795                                                        replacetail = getprev(replacetail)
2796                                                    end
2797                                                else
2798                                                    notmatchreplace[prev] = true
2799                                                    if notmatchpost[prev] then
2800                                                        goto next
2801                                                    else
2802                                                        break
2803                                                    end
2804                                                end
2805                                            end
2806                                        else
2807                                         -- notmatchreplace[prev] = true -- not according to Kai
2808                                        end
2809                                    end
2810                                 -- prev = getprev(prev)
2811                                    prev = prv
2812                             -- elseif id == glue_code and seq[n][32] and isspace(prev,threshold,id) then
2813                             -- elseif seq[n][32] and spaces[prev] then
2814                             --     n = n - 1
2815                             --     prev = prv
2816                                elseif id == glue_code then
2817                                    local sn = seq[n]
2818                                    if (sn[32] and spaces[prev]) or sn[0xFFFC] then
2819                                        n = n - 1
2820                                        prev = prv
2821                                    else
2822                                        goto next
2823                                    end
2824                                elseif seq[n][0xFFFC] then
2825                                    n = n - 1
2826                                    prev = prv
2827                                else
2828                                    goto next
2829                                end
2830                            else
2831                                goto next
2832                            end
2833                        end
2834                    else
2835                        goto next
2836                    end
2837             -- else          -- new 2022-09-25
2838             --     goto next -- new 2022-09-25
2839             -- end           -- new 2022-09-25
2840            end
2841
2842            -- after
2843
2844            if s > l then
2845                local current = last and getnext(last)
2846                if not current and postreplace then
2847                    current = getnext(sweepnode)
2848                end
2849                if current then
2850                    local discfound -- = nil
2851                    local n = l + 1
2852                    while n <= s do
2853                        if current then
2854                            local nxt, char, id = isnextchar(current,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
2855                            if char then
2856                                if seq[n][char] then
2857                                    if n < s then
2858                                        current = nxt
2859                                    end
2860                                    n = n + 1
2861                                elseif skiphash and skiphash[char] then
2862                                    skipped = true
2863                                    if trace_skips then
2864                                        show_skip(dataset,sequence,char,ck,classes[char])
2865                                    end
2866                                    current = nxt
2867                                elseif discfound then
2868                                    notmatchreplace[discfound] = true
2869                                    if notmatchpre[discfound] then
2870                                        goto next
2871                                    else
2872                                        break
2873                                    end
2874                                else
2875                                    goto next
2876                                end
2877                            elseif char == false then
2878                                if discfound then
2879                                    notmatchreplace[discfound] = true
2880                                    if notmatchpre[discfound] then
2881                                        goto next
2882                                    else
2883                                        break
2884                                    end
2885                                else
2886                                    goto next
2887                                end
2888                            elseif id == disc_code then
2889                         -- elseif id == disc_code and (not discs or discs[current]) then
2890                                discseen                 = true
2891                                discfound                = current
2892                                notmatchpre[current]     = nil
2893                                notmatchpost[current]    = true
2894                                notmatchreplace[current] = nil
2895                                local pre, post, replace = getdisc(current)
2896                                -- no detailed checking here, todo ?
2897                                if pre then
2898                                    local n = n
2899                                    while pre do
2900                                        if seq[n][getchar(pre)] then
2901                                            n = n + 1
2902                                            if n > s then
2903                                                break
2904                                            else
2905                                                pre = getnext(pre)
2906                                            end
2907                                        else
2908                                            notmatchpre[current] = true
2909                                            break
2910                                        end
2911                                    end
2912                                    if n <= s then
2913                                        notmatchpre[current] = true
2914                                    end
2915                                else
2916                                    notmatchpre[current] = true
2917                                end
2918                                if replace then
2919                                    -- so far we never entered this branch
2920                                    while replace do
2921                                        if seq[n][getchar(replace)] then
2922                                            n = n + 1
2923                                            if n > s then
2924                                                break
2925                                            else
2926                                                replace = getnext(replace)
2927                                            end
2928                                        else
2929                                            notmatchreplace[current] = true
2930                                            if notmatchpre[current] then
2931                                                goto next
2932                                            else
2933                                                break
2934                                            end
2935                                        end
2936                                    end
2937                                else
2938                                 -- notmatchreplace[current] = true -- not according to Kai
2939                                end
2940                                current = getnext(current)
2941                            elseif id == glue_code then
2942                                local sn = seq[n]
2943                                if (sn[32] and spaces[current]) or sn[0xFFFC] then
2944                                    n = n + 1
2945                                    current = nxt
2946                                else
2947                                    goto next
2948                                end
2949                            elseif seq[n][0xFFFC] then
2950                                n = n + 1
2951                                current = nxt
2952                            else
2953                                goto next
2954                            end
2955                        else
2956                            goto next
2957                        end
2958                    end
2959                else
2960                    goto next
2961                end
2962            end
2963        end
2964        if trace_contexts then
2965            chaintrac(head,start,dataset,sequence,rlmode,skipped and skiphash,ck,true,discseen,sweepnode)
2966        end
2967        if discseen or sweepnode then
2968            -- When we process a disc we can collapse and therefore we backtrack one node (start) and
2969            -- reprocess. This is needed because there might be more in the collapsed list.
2970            head, start, done = chaindisk(head,start,dataset,sequence,rlmode,skipped and skiphash,ck)
2971        else
2972            head, start, done = chainrun(head,start,last,dataset,sequence,rlmode,skipped and skiphash,ck,6)
2973        end
2974        if trace_contexts then
2975            chaintrac(start,done)
2976        end
2977        if done then
2978            break
2979     -- else
2980            -- next context
2981        end
2982      ::next::
2983    end
2984    if discseen then
2985        notmatchpre     = { }
2986        notmatchpost    = { }
2987        notmatchreplace = { }
2988     -- notmatchpre     = { a = 1, b = 1 }  notmatchpre    .a = nil notmatchpre    .b = nil
2989     -- notmatchpost    = { a = 1, b = 1 }  notmatchpost   .a = nil notmatchpost   .b = nil
2990     -- notmatchreplace = { a = 1, b = 1 }  notmatchreplace.a = nil notmatchreplace.b = nil
2991    end
2992    return head, start, done
2993end
2994
2995handlers.gsub_context             = handle_contextchain
2996handlers.gsub_contextchain        = handle_contextchain
2997handlers.gsub_reversecontextchain = handle_contextchain
2998handlers.gpos_contextchain        = handle_contextchain
2999handlers.gpos_context             = handle_contextchain
3000
3001-- local function chained_contextchain(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash)
3002--     local steps    = currentlookup.steps
3003--     local nofsteps = currentlookup.nofsteps
3004--     if nofsteps > 1 then
3005--         reportmoresteps(dataset,sequence)
3006--     end
3007--     -- probably wrong
3008--     local l = steps[1].coverage[getchar(start)]
3009--     if l then
3010--         return handle_contextchain(head,start,dataset,sequence,l,rlmode,skiphash)
3011--     else
3012--         return head, start, false
3013--     end
3014-- end
3015
3016-- new 2022-09-25
3017
3018local function chained_contextchain(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash)
3019    local steps    = currentlookup.steps
3020    local nofsteps = currentlookup.nofsteps
3021    local char     = getchar(start)
3022    if nofsteps == 1 then
3023        local s = steps[1]
3024        local l = s.coverage[char]
3025        if l then
3026            return handle_contextchain(head,start,dataset,sequence,l,rlmode,skiphash)
3027        end
3028    else
3029        for i=1,nofsteps do
3030            local s = steps[i]
3031            local l = s.coverage[char]
3032            if l then
3033                local h, s, d = handle_contextchain(head,start,dataset,sequence,l,rlmode,skiphash)
3034                if d then
3035                    return h, s, d
3036                end
3037            end
3038        end
3039    end
3040    return head, start, false
3041end
3042
3043chainprocs.gsub_context             = chained_contextchain
3044chainprocs.gsub_contextchain        = chained_contextchain
3045chainprocs.gsub_reversecontextchain = chained_contextchain
3046chainprocs.gpos_contextchain        = chained_contextchain
3047chainprocs.gpos_context             = chained_contextchain
3048
3049------------------------------
3050
3051-- experiment (needs no handler in font-otc so not now):
3052--
3053-- function otf.registerchainproc(name,f)
3054--  -- chainprocs[name] = f
3055--     chainprocs[name] = function(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash)
3056--         local done = currentlookup.nofsteps > 0
3057--         if not done then
3058--             reportzerosteps(dataset,sequence)
3059--         else
3060--             head, start, done = f(head,start,stop,dataset,sequence,currentlookup,rlmode,skiphash)
3061--             if not head or not start then
3062--                 reportbadsteps(dataset,sequence)
3063--             end
3064--         end
3065--         return head, start, done
3066--     end
3067-- end
3068
3069local missing    = setmetatableindex("table")
3070local logwarning = report_process
3071local resolved   = { } -- we only resolve a font,script,language pair once
3072
3073local function logprocess(...)
3074    if trace_steps then
3075        registermessage(...)
3076        if trace_steps == "silent" then
3077            return
3078        end
3079    end
3080    report_process(...)
3081end
3082
3083-- todo: pass all these 'locals' in a table
3084
3085local sequencelists = setmetatableindex(function(t,font)
3086    local sequences = fontdata[font].resources.sequences
3087    if not sequences or not next(sequences) then
3088        sequences = false
3089    end
3090    t[font] = sequences
3091    return sequences
3092end)
3093
3094-- fonts.hashes.sequences = sequencelists
3095
3096do -- overcome local limit
3097
3098    local autofeatures    = fonts.analyzers.features
3099    local featuretypes    = otf.tables.featuretypes
3100    local defaultscript   = otf.features.checkeddefaultscript
3101    local defaultlanguage = otf.features.checkeddefaultlanguage
3102
3103    local wildcard        <const> = "*"
3104    local default         <const> = "dflt"
3105
3106    local function initialize(sequence,script,language,enabled,autoscript,autolanguage)
3107        local features = sequence.features
3108        if features then
3109            local order = sequence.order
3110            if order then
3111                local featuretype = featuretypes[sequence.type or "unknown"]
3112                for i=1,#order do
3113                    local kind  = order[i]
3114                    local valid = enabled[kind]
3115                    if valid then
3116                        local scripts   = features[kind]
3117                        local languages = scripts and (
3118                            scripts[script] or
3119                            scripts[wildcard] or
3120                            (autoscript and defaultscript(featuretype,autoscript,scripts))
3121                        )
3122                        local enabled = languages and (
3123                            languages[language] or
3124                            languages[wildcard] or
3125                            (autolanguage and defaultlanguage(featuretype,autolanguage,languages))
3126                        )
3127                        if enabled then
3128                            return { valid, autofeatures[kind] or false, sequence, kind }
3129                        end
3130                    end
3131                end
3132            else
3133                -- can't happen
3134            end
3135        end
3136        return false
3137    end
3138
3139    function otf.dataset(tfmdata,font) -- generic variant, overloaded in context
3140        local shared       = tfmdata.shared
3141        local properties   = tfmdata.properties
3142        local language     = properties.language or "dflt"
3143        local script       = properties.script   or "dflt"
3144        local enabled      = shared.features
3145        local autoscript   = enabled and enabled.autoscript
3146        local autolanguage = enabled and enabled.autolanguage
3147        local res = resolved[font]
3148        if not res then
3149            res = { }
3150            resolved[font] = res
3151        end
3152        local rs = res[script]
3153        if not rs then
3154            rs = { }
3155            res[script] = rs
3156        end
3157        local rl = rs[language]
3158        if not rl then
3159            rl = {
3160                -- indexed but we can also add specific data by key
3161            }
3162            rs[language] = rl
3163            local sequences = tfmdata.resources.sequences
3164            if sequences then
3165                for s=1,#sequences do
3166                    local v = enabled and initialize(sequences[s],script,language,enabled,autoscript,autolanguage)
3167                    if v then
3168                        rl[#rl+1] = v
3169                    end
3170                end
3171            end
3172        end
3173        return rl
3174    end
3175
3176end
3177
3178-- Functions like kernrun, comprun etc evolved over time and in the end look rather
3179-- complex. It's a bit of a compromise between extensive copying and creating subruns.
3180-- The logic has been improved a lot by Kai and Ivo who use complex fonts which
3181-- really helped to identify border cases on the one hand and get insight in the diverse
3182-- ways fonts implement features (not always that consistent and efficient). At the same
3183-- time I tried to keep the code relatively efficient so that the overhead in runtime
3184-- stays acceptable.
3185
3186local function report_disc(what,n)
3187    report_run("%s: %s > %s",what,n,languages.serializediscretionary(n))
3188end
3189
3190local function kernrun(disc,k_run,...)
3191    --
3192    -- we catch <font 1><disc font 2>
3193    --
3194    if trace_kernruns then
3195        report_disc("kern",disc)
3196    end
3197    --
3198    local prev, next = getboth(disc)
3199    --
3200    local nextstart = next
3201    local done      = false
3202    --
3203    local pre, post, replace, pretail, posttail, replacetail = getdisc(disc,true)
3204    --
3205    local prevmarks = prev
3206    --
3207    -- can be optional, because why on earth do we get a disc after a mark (okay, maybe when a ccmp
3208    -- has happened but then it should be in the disc so basically this test indicates an error)
3209    --
3210    while prevmarks do
3211        local prv, char = isprevchar(prevmarks,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
3212        if char and marks[char] then
3213            prevmarks = prv
3214        else
3215            break
3216        end
3217    end
3218    --
3219    if prev and not ischar(prev,currentfont) then  -- and (pre or replace) -- weak test
3220        prev = false
3221    end
3222    if next and not ischar(next,currentfont) then  -- and (post or replace) -- weak test
3223        next = false
3224    end
3225    --
3226    -- we need to get rid of this nest mess some day .. has to be done otherwise
3227    --
3228    if pre then
3229        if k_run(pre,"injections",nil,...) then
3230            done = true
3231        end
3232        if prev then
3233            setlink(prev,pre)
3234            if k_run(prevmarks,"preinjections",pre,...) then -- or prev?
3235                done = true
3236            end
3237            setprev(pre)
3238            setlink(prev,disc)
3239        end
3240    end
3241    --
3242    if post then
3243        if k_run(post,"injections",nil,...) then
3244            done = true
3245        end
3246        if next then
3247            setlink(posttail,next)
3248            if k_run(posttail,"postinjections",next,...) then
3249                done = true
3250            end
3251            setnext(posttail)
3252            setlink(disc,next)
3253        end
3254    end
3255    --
3256    if replace then
3257        if k_run(replace,"injections",nil,...) then
3258            done = true
3259        end
3260        if prev then
3261            setlink(prev,replace)
3262            if k_run(prevmarks,"replaceinjections",replace,...) then -- getnext(replace))
3263                done = true
3264            end
3265            setprev(replace)
3266            setlink(prev,disc)
3267        end
3268        if next then
3269            setlink(replacetail,next)
3270            if k_run(replacetail,"replaceinjections",next,...) then
3271                done = true
3272            end
3273            setnext(replacetail)
3274            setlink(disc,next)
3275        end
3276    elseif prev and next then
3277        setlink(prev,next)
3278        if k_run(prevmarks,"emptyinjections",next,...) then
3279            done = true
3280        end
3281        setlink(prev,disc,next)
3282    end
3283    if done and trace_testruns then
3284        report_disc("done",disc)
3285    end
3286 -- return nextstart, done
3287    return nextstart
3288end
3289
3290-- fonts like ebgaramond do ligatures this way (less efficient than e.g. dejavu which
3291-- will do the testrun variant)
3292
3293local function comprun(disc,c_run,...) -- vararg faster than the whole list
3294    if trace_compruns then
3295        report_disc("comp",disc)
3296    end
3297    --
3298    local pre, post, replace = getdisc(disc)
3299    local renewed = false
3300    --
3301    if pre then
3302        sweepnode = disc
3303        sweeptype = "pre" -- in alternative code preinjections is used (also used then for properties, saves a variable)
3304        local new, done = c_run(pre,...)
3305        if done then
3306            pre     = new
3307            renewed = true
3308        end
3309    end
3310    --
3311    if post then
3312        sweepnode = disc
3313        sweeptype = "post"
3314        local new, done = c_run(post,...)
3315        if done then
3316            post    = new
3317            renewed = true
3318        end
3319    end
3320    --
3321    if replace then
3322        sweepnode = disc
3323        sweeptype = "replace"
3324        local new, done = c_run(replace,...)
3325        if done then
3326            replace = new
3327            renewed = true
3328        end
3329    end
3330    --
3331    sweepnode = nil
3332    sweeptype = nil
3333    if renewed then
3334        if trace_testruns then
3335            report_disc("done",disc)
3336        end
3337        setdisc(disc,pre,post,replace)
3338    end
3339    --
3340 -- return getnext(disc), renewed
3341    return getnext(disc)
3342end
3343
3344-- if we can hyphenate in a lig then unlikely a lig so we
3345-- could have a option here to ignore lig
3346
3347local test_flatten_start = 2 -- must start at 2 according to Kai
3348
3349directives.register("otf.testrun.forceflatten", function(v)
3350    test_flatten_start = v and 1 or 2
3351end)
3352
3353local function testrun(disc,t_run,c_run,...)
3354    if trace_testruns then
3355        report_disc("test",disc)
3356    end
3357    local prev, next = getboth(disc)
3358    if not next then
3359        -- weird discretionary
3360     -- return
3361        -- better, removing it is tricky
3362        next = newboundary()
3363        setlink(disc,next)
3364    end
3365    local pre, post, replace, pretail, posttail, replacetail = getdisc(disc,true)
3366    local renewed = false
3367    if post or replace then -- and prev then -- hm, we can start with a disc
3368        if post then
3369            setlink(posttail,next)
3370        else
3371            post = next
3372        end
3373        if replace then
3374            setlink(replacetail,next)
3375        else
3376            replace = next
3377        end
3378        local d_post    = t_run(post,next,...)
3379        local d_replace = t_run(replace,next,...)
3380        if d_post > 0 or d_replace > 0 then
3381            local d = d_replace > d_post and d_replace or d_post
3382            local head = getnext(disc) -- is: next
3383            local tail = head
3384            for i=test_flatten_start,d do
3385                local nx = getnext(tail)
3386                local id = getid(nx)
3387                if id == disc_code then
3388                    head, tail = flattendisk(head,nx)
3389                elseif id == glyph_code then
3390                    tail = nx
3391                else
3392                    -- we can have overrun into a glue
3393                    break
3394                end
3395            end
3396            next = getnext(tail)
3397            setnext(tail)
3398            setprev(head)
3399            local new  = copynodelist(head)
3400            if posttail then
3401                setlink(posttail,head)
3402            else
3403                post = head
3404            end
3405            if replacetail then
3406                setlink(replacetail,new)
3407            else
3408                replace = new
3409            end
3410        else
3411            -- we stay inside the disc
3412            if posttail then
3413                setnext(posttail)
3414            else
3415                post = nil
3416            end
3417            if replacetail then
3418                setnext(replacetail)
3419            else
3420                replace = nil
3421            end
3422        end
3423        setlink(disc,next)
3424     -- pre, post, replace, pretail, posttail, replacetail = getdisc(disc,true)
3425    end
3426    --
3427    -- like comprun
3428    --
3429    if trace_testruns then
3430        report_disc("more",disc)
3431    end
3432    --
3433    if pre then
3434        sweepnode = disc
3435        sweeptype = "pre"
3436        local new, ok = c_run(pre,...)
3437        if ok then
3438            pre     = new
3439            renewed = true
3440        end
3441    end
3442    --
3443    if post then
3444        sweepnode = disc
3445        sweeptype = "post"
3446        local new, ok = c_run(post,...)
3447        if ok then
3448            post    = new
3449            renewed = true
3450        end
3451    end
3452    --
3453    if replace then
3454        sweepnode = disc
3455        sweeptype = "replace"
3456        local new, ok = c_run(replace,...)
3457        if ok then
3458            replace = new
3459            renewed = true
3460        end
3461    end
3462    --
3463    sweepnode = nil
3464    sweeptype = nil
3465    if renewed then
3466        setdisc(disc,pre,post,replace)
3467        if trace_testruns then
3468            report_disc("done",disc)
3469        end
3470    end
3471    -- next can have changed (copied list)
3472 -- return getnext(disc), renewed
3473    return getnext(disc)
3474end
3475
3476--  1{2{\oldstyle\discretionary{3}{4}{5}}6}7\par
3477--  1{2\discretionary{3{\oldstyle3}}{{\oldstyle4}4}{5{\oldstyle5}5}6}7\par
3478
3479local nesting = 0
3480
3481local function c_run_single(head,lookupcache,step,dataset,sequence,rlmode,skiphash,handler)
3482    local done  = false
3483    local sweep = sweephead[head]
3484    local start
3485    if sweep then
3486        start = sweep
3487     -- sweephead[head] = nil
3488        sweephead[head] = false
3489    else
3490        start = head
3491    end
3492    while start do
3493        local nxt, char, id = isnextchar(start,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
3494        if char then
3495            local lookupmatch = lookupcache[char]
3496            if lookupmatch then
3497                local ok
3498                head, start, ok = handler(head,start,dataset,sequence,lookupmatch,rlmode,skiphash,step)
3499                if ok then
3500                    done = true
3501                end
3502                if start then
3503                    start = getnext(start)
3504                end
3505            else
3506                start = nxt
3507            end
3508        elseif char == false then
3509            return head, done
3510        elseif sweep then
3511            -- else we loose the rest
3512            return head, done
3513        else
3514            -- in disc component ... really?
3515            start = nxt
3516        end
3517    end
3518    return head, done
3519end
3520
3521local function t_run_single(start,stop,lookupcache)
3522    local lastd = nil
3523    while start ~= stop do
3524        local nxt, char = isnextchar(start,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
3525        if char then
3526            local lookupmatch = lookupcache[char]
3527            if lookupmatch then -- hm, hyphens can match (tlig) so we need to really check
3528                -- if we need more than ligatures we can outline the code and use functions
3529                local s     = nxt
3530                local ss    = nil
3531                local sstop = s == stop
3532                if not s then
3533                    s = ss
3534                    ss = nil
3535                end
3536                -- a bit weird: why multiple ... anyway we can't have a disc in a disc
3537                -- how about post ... we can probably merge this into the while
3538                while getid(s) == disc_code do
3539                    ss = getnext(s)
3540                    s  = getreplace(s)
3541                    if not s then
3542                        s = ss
3543                        ss = nil
3544                    end
3545                end
3546                local l = nil
3547                local d = 0
3548                while s do
3549                    local nxt, char = isnextchar(s,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
3550                    if char then
3551                        local lg = not tonumber(lookupmatch) and lookupmatch[char]
3552                        if lg then
3553                            if sstop then
3554                                d = 1
3555                            elseif d > 0 then
3556                                d = d + 1
3557                            end
3558                            l = lg
3559                            s = nxt
3560                            sstop = s == stop
3561                            if not s then
3562                                s  = ss
3563                                ss = nil
3564                            end
3565                            while getid(s) == disc_code do
3566                                ss = getnext(s)
3567                                s  = getreplace(s)
3568                                if not s then
3569                                    s  = ss
3570                                    ss = nil
3571                                end
3572                            end
3573                            lookupmatch = lg
3574                        else
3575                            break
3576                        end
3577                    else
3578                        break
3579                    end
3580                end
3581                if l and (tonumber(l) or l.ligature) then -- so we test for ligature
3582                    lastd = d
3583                end
3584            end
3585            if lastd then
3586                return lastd
3587            else
3588                start = nxt
3589            end
3590        else
3591            break
3592        end
3593    end
3594    return 0
3595end
3596
3597local function k_run_single(sub,injection,last,lookupcache,step,dataset,sequence,rlmode,skiphash,handler)
3598    for n in nextnode, sub do -- only gpos
3599        if n == last then
3600            break
3601        else
3602            local nxt, char = isnextchar(n,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
3603            if char then
3604                local lookupmatch = lookupcache[char]
3605                if lookupmatch then
3606                    local h, d, ok = handler(sub,n,dataset,sequence,lookupmatch,rlmode,skiphash,step,injection)
3607                    if ok then
3608                        return true
3609                    end
3610                end
3611            end
3612        end
3613    end
3614end
3615
3616local function c_run_multiple(head,steps,nofsteps,dataset,sequence,rlmode,skiphash,handler)
3617    local done  = false
3618    local sweep = sweephead[head]
3619    local start
3620    if sweep then
3621        start = sweep
3622     -- sweephead[head] = nil
3623        sweephead[head] = false
3624    else
3625        start = head
3626    end
3627    while start do
3628        local nxt, char = isnextchar(start,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
3629        if char then
3630            for i=1,nofsteps do
3631                local step        = steps[i]
3632                local lookupcache = step.coverage
3633                local lookupmatch = lookupcache[char]
3634                if lookupmatch then
3635                    -- we could move all code inline but that makes things even more unreadable
3636                    local ok
3637                    head, start, ok = handler(head,start,dataset,sequence,lookupmatch,rlmode,skiphash,step)
3638                    if ok then
3639                        done = true
3640                        break
3641                    elseif not start then
3642                        -- don't ask why ... shouldn't happen
3643                        break
3644                    end
3645                end
3646            end
3647            if start then
3648                start = getnext(start)
3649            end
3650        elseif char == false then
3651            -- whatever glyph
3652            return head, done
3653        elseif sweep then
3654            -- else we loose the rest
3655            return head, done
3656        else
3657            -- in disc component
3658            start = nxt
3659        end
3660    end
3661    return head, done
3662end
3663
3664local function t_run_multiple(start,stop,steps,nofsteps)
3665    local lastd = nil
3666    while start ~= stop do
3667        local startnext, char = isnextchar(start,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
3668        if char then
3669            for i=1,nofsteps do
3670                local step = steps[i]
3671                local lookupcache = step.coverage
3672                local lookupmatch = lookupcache[char]
3673                if lookupmatch then
3674                    -- if we need more than ligatures we can outline the code and use functions
3675                    local s     = startnext
3676                    local ss    = nil
3677                    local sstop = s == stop
3678                    if not s then
3679                        s  = ss
3680                        ss = nil
3681                    end
3682                    while getid(s) == disc_code do
3683                        ss = getnext(s)
3684                        s  = getreplace(s)
3685                        if not s then
3686                            s  = ss
3687                            ss = nil
3688                        end
3689                    end
3690                    local l = nil
3691                    local d = 0
3692                    while s do
3693                        local nxt, char = isnextchar(s,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
3694                        if char then
3695                            local lg = not tonumber(lookupmatch) and lookupmatch[char]
3696                            if lg then
3697                                if sstop then
3698                                    d = 1
3699                                elseif d > 0 then
3700                                    d = d + 1
3701                                end
3702                                l = lg
3703                                s = nxt
3704                                sstop = s == stop
3705                                if not s then
3706                                    s  = ss
3707                                    ss = nil
3708                                end
3709                                while getid(s) == disc_code do
3710                                    ss = getnext(s)
3711                                    s  = getreplace(s)
3712                                    if not s then
3713                                        s  = ss
3714                                        ss = nil
3715                                    end
3716                                end
3717                                lookupmatch = lg
3718                            else
3719                                break
3720                            end
3721                        else
3722                            break
3723                        end
3724                    end
3725                    if l and (tonumber(l) or l.ligature) then
3726                        lastd = d
3727                    end
3728                end
3729            end
3730            if lastd then
3731                return lastd
3732            else
3733                start = startnext
3734            end
3735        else
3736            break
3737        end
3738    end
3739    return 0
3740end
3741
3742local function k_run_multiple(sub,injection,last,steps,nofsteps,dataset,sequence,rlmode,skiphash,handler)
3743    for n in nextnode, sub do -- only gpos
3744        if n == last then
3745            break
3746        else
3747            local nxt, char = isnextchar(n,currentfont,currentdynamic,currentscale,currentxscale,currentyscale)
3748            if char then
3749                for i=1,nofsteps do
3750                    local step        = steps[i]
3751                    local lookupcache = step.coverage
3752                    local lookupmatch = lookupcache[char]
3753                    if lookupmatch then
3754                        local h, d, ok = handler(sub,n,dataset,sequence,lookupmatch,rlmode,skiphash,step,injection) -- sub was head
3755                        if ok then
3756                            return true
3757                        end
3758                    end
3759                end
3760            end
3761        end
3762    end
3763end
3764
3765local txtdirstate, pardirstate  do -- this might change (no need for nxt in pardirstate)
3766
3767    local getdirection = nuts.getdirection
3768
3769    txtdirstate = function(start,stack,top,rlparmode)
3770        local dir, pop = getdirection(start)
3771        if pop then
3772            if top == 1 then
3773                return 0, rlparmode
3774            else
3775                top = top - 1
3776                if stack[top] == righttoleft_code then
3777                    return top, -1
3778                else
3779                    return top, 1
3780                end
3781            end
3782        elseif dir == lefttoright_code then
3783            top = top + 1
3784            stack[top] = lefttoright_code
3785            return top, 1
3786        elseif dir == righttoleft_code then
3787            top = top + 1
3788            stack[top] = righttoleft_code
3789            return top, -1
3790        else
3791            return top, rlparmode
3792        end
3793    end
3794
3795    pardirstate = function(start)
3796        local dir = getdirection(start)
3797        if dir == lefttoright_code then
3798            return 1, 1
3799        elseif dir == righttoleft_code then
3800            return -1, -1
3801        else
3802            return 0, 0
3803        end
3804    end
3805
3806end
3807
3808-- These are non public helpers that can change without notice!
3809
3810otf.helpers             = otf.helpers or { }
3811otf.helpers.txtdirstate = txtdirstate
3812otf.helpers.pardirstate = pardirstate
3813
3814-- This is the main loop. We run over the node list dealing with a specific font. The
3815-- dynamic is a context specific thing. We could work on sub start-stop ranges instead
3816-- but I wonder if there is that much speed gain (experiments showed that it made not
3817-- much sense) and we need to keep track of directions anyway. Also at some point I
3818-- want to play with font interactions and then we do need the full sweeps. Apart from
3819-- optimizations the principles of processing the features hasn't changed much since
3820-- the beginning.
3821
3822do
3823
3824    -- This is a measurable experimental speedup (only with hyphenated text and multiple
3825    -- fonts per processor call), especially for fonts with lots of contextual lookups.
3826
3827    local fastdisc = true
3828    local testdics = false
3829
3830    directives.register("otf.fastdisc",function(v) fastdisc = v end) -- normally enabled
3831
3832    -- using a merged combined hash as first test saves some 30% on ebgaramond and
3833    -- about 15% on arabtype .. then moving the a test also saves a bit (even when
3834    -- often a is not set at all so that one is a bit debatable
3835
3836 -- local otfdataset = nil
3837    local otfdataset
3838
3839    otfdataset = function(...)
3840        -- delayed alias
3841        otfdataset = otf.dataset
3842        return otfdataset(...)
3843    end
3844
3845    local getfastdisc = { __index = function(t,k)
3846        local v = usesfont(k,currentfont)
3847        t[k] = v
3848        return v
3849    end }
3850
3851    local getfastspace = { __index = function(t,k)
3852        -- we don't pass the id so that one can overload isspace
3853        local v = isspace(k,threshold) or false
3854        t[k] = v
3855        return v
3856    end }
3857
3858    function otf.featuresprocessor(head,font,dynamic,direction,n)
3859
3860        local sequences = sequencelists[font] -- temp hack
3861
3862        nesting = nesting + 1
3863
3864        if nesting == 1 then
3865            currentfont    = font
3866            currentdynamic = dynamic
3867            currentscale   = 1000
3868            currentxscale  = 1000
3869            currentyscale  = 1000
3870            tfmdata        = fontdata[font]
3871            descriptions   = tfmdata.descriptions -- only needed in gref so we could pass node there instead
3872            characters     = tfmdata.characters   -- but this branch is not entered that often anyway
3873      local resources      = tfmdata.resources
3874            marks          = resources.marks
3875            classes        = resources.classes
3876            threshold,
3877            factor         = getthreshold(font)
3878            checkmarks     = tfmdata.properties.checkmarks
3879
3880         -- if not otfdataset then
3881         --     otfdataset = otf.dataset
3882         -- end
3883
3884            discs  = fastdisc and n and n > 1 and setmetatable({},getfastdisc)
3885            spaces = setmetatable({},getfastspace)
3886
3887        elseif currentfont ~= font then
3888
3889            report_warning("nested call with a different font, level %s, quitting",nesting)
3890            nesting = nesting - 1
3891            return head, false
3892
3893        end
3894
3895        if trace_steps then
3896            checkstep(head)
3897        end
3898
3899        local initialrl = 0
3900
3901        if getid(head) == par_code and startofpar(head) then
3902            initialrl = pardirstate(head)
3903        elseif direction == righttoleft_code then
3904            initialrl = -1
3905        end
3906
3907        local datasets  = otfdataset(tfmdata,font,dynamic)
3908        local dirstack  = { nil } -- could move outside function but we can have local runs
3909        sweephead       = { }
3910     -- sweephead  = { a = 1, b = 1 } sweephead.a = nil sweephead.b = nil
3911
3912        -- Keeping track of the headnode is needed for devanagari. (I generalized it a bit
3913        -- so that multiple cases are also covered.) We could prepend a temp node.
3914
3915        -- We don't goto the next node when a disc node is created so that we can then treat
3916        -- the pre, post and replace. It's a bit of a hack but works out ok for most cases.
3917
3918        for s=1,#datasets do -- todo: datasets.n
3919            local dataset      = datasets[s]
3920            local state        = dataset[2]
3921            local sequence     = dataset[3] -- sequences[s] -- also dataset[5]
3922            local rlparmode    = initialrl
3923            local topstack     = 0
3924            local typ          = sequence.type
3925            local gpossing     = typ == "gpos_single" or typ == "gpos_pair" -- store in dataset
3926            local forcetestrun = typ == "gsub_ligature" -- testrun is only for ligatures
3927            local handler      = handlers[typ] -- store in dataset
3928            local steps        = sequence.steps
3929            local nofsteps     = sequence.nofsteps
3930            local skiphash     = sequence.skiphash
3931            if not steps then
3932                -- This permits injection, watch the different arguments. Watch out, the arguments passed
3933                -- are not frozen as we might extend or change this. Is this used at all apart from some
3934                -- experiments?
3935                local h = handler(head,dataset,sequence,initialrl,font,dynamic) -- less arguments now
3936                if h and h ~= head then
3937                    head = h
3938                end
3939            elseif typ == "gsub_reversecontextchain" then
3940                --
3941                -- This might need a check: if we have #before or #after > 0 then we might need to reverse
3942                -- the before and after lists in the loader. But first I need to see a font that uses multiple
3943                -- matches.
3944                --
3945                local start  = findnodetail(head)
3946                local rlmode = 0 -- how important is this .. do we need to check for dir?
3947                local merged = steps.merged
3948                while start do
3949                    local prv, char, id = isprevchar(start,font,dynamic,state)
3950                    if char then
3951                        local m = merged[char]
3952currentscale, currentxscale, currentyscale = getscales(start)
3953                        if m then
3954--                             currentscale, currentxscale, currentyscale = getscales(start)
3955                            for i=m[1],m[2] do
3956                                local step = steps[i]
3957                                local lookupcache = step.coverage
3958                                local lookupmatch = lookupcache[char]
3959                                if lookupmatch then
3960                                    local ok
3961                                    head, start, ok = handler(head,start,dataset,sequence,lookupmatch,rlmode,skiphash,step)
3962                                    if ok then
3963                                     -- done = true
3964                                        break
3965                                    end
3966                                end
3967                            end
3968                            if start then
3969                                start = getprev(start)
3970                            end
3971                        else
3972                            start = prv
3973                        end
3974                    else
3975                        start = prv
3976                    end
3977                end
3978            else
3979                local start  = head
3980                local rlmode = initialrl
3981                if nofsteps == 1 then -- happens often
3982                    local step = steps[1]
3983                    local lookupcache = step.coverage
3984                    while start do
3985                        local nxt, char, id = isnextchar(start,font,dynamic,state) -- we can move the data/state checks here
3986                        if char then
3987                            -- no skiphash check here
3988                            local lookupmatch = lookupcache[char]
3989currentscale, currentxscale, currentyscale = getscales(start)
3990                            if lookupmatch then
3991--                                 currentscale, currentxscale, currentyscale = getscales(start)
3992                                local ok, df
3993                                head, start, ok, df = handler(head,start,dataset,sequence,lookupmatch,rlmode,skiphash,step)
3994                                if df then
3995                                 -- print("restart 1",typ)
3996                                elseif start then
3997                                    start = getnext(start) -- can be a new start
3998                                end
3999                            else
4000                                start = nxt
4001                            end
4002                        elseif char == false or id == glue_code then
4003                            -- a different font|state or glue (happens often)
4004                            start = nxt
4005                        elseif id == disc_code then
4006                            if not discs or discs[start] == true then
4007                                if gpossing then
4008                                    start = kernrun(start,k_run_single,             lookupcache,step,dataset,sequence,rlmode,skiphash,handler)
4009                                elseif forcetestrun then
4010                                    start = testrun(start,t_run_single,c_run_single,lookupcache,step,dataset,sequence,rlmode,skiphash,handler)
4011                                else
4012                                    start = comprun(start,c_run_single,             lookupcache,step,dataset,sequence,rlmode,skiphash,handler)
4013                                end
4014                            else
4015                                start = nxt
4016                            end
4017                        elseif id == math_code then
4018                            start = getnext(endofmath(start))
4019                        elseif id == dir_code then
4020                            topstack, rlmode = txtdirstate(start,dirstack,topstack,rlparmode)
4021                            start = nxt
4022                     -- elseif id == par_code and startofpar(start) then
4023                     --     rlparmode, rlmode = pardirstate(start)
4024                     --     start = nxt
4025                        else
4026                            start = nxt
4027                        end
4028                    end
4029                else
4030                    local merged = steps.merged
4031                    while start do
4032                        local nxt, char, id = isnextchar(start,font,dynamic,state)
4033                        if char then
4034                            -- no skiphash check here
4035                            local m = merged[char]
4036currentscale, currentxscale, currentyscale = getscales(start)
4037                            if m then
4038--                                 currentscale, currentxscale, currentyscale = getscales(start)
4039                                local ok, df
4040                                for i=m[1],m[2] do
4041                                    local step = steps[i]
4042                                    local lookupcache = step.coverage
4043                                    local lookupmatch = lookupcache[char]
4044                                    if lookupmatch then
4045                                        -- we could move all code inline but that makes things even more unreadable
4046--                                                 local ok, df
4047                                        head, start, ok, df = handler(head,start,dataset,sequence,lookupmatch,rlmode,skiphash,step)
4048                                        if df then
4049                                            break
4050                                        elseif ok then
4051                                            break
4052                                        elseif not start then
4053                                            -- don't ask why ... shouldn't happen
4054                                            break
4055                                        end
4056                                    end
4057                                end
4058                                if df then
4059                                 -- print("restart 2",typ)
4060                                elseif start then
4061                                    start = getnext(start) -- can be a new next
4062                                end
4063                            else
4064                                start = nxt
4065                            end
4066                        elseif char == false or id == glue_code then
4067                            -- a different font|state or glue (happens often)
4068                            start = nxt
4069                        elseif id == disc_code then
4070                            if not discs or discs[start] == true then
4071                                if gpossing then
4072                                    start = kernrun(start,k_run_multiple,               steps,nofsteps,dataset,sequence,rlmode,skiphash,handler)
4073                                elseif forcetestrun then
4074                                    start = testrun(start,t_run_multiple,c_run_multiple,steps,nofsteps,dataset,sequence,rlmode,skiphash,handler)
4075                                else
4076                                    start = comprun(start,c_run_multiple,               steps,nofsteps,dataset,sequence,rlmode,skiphash,handler)
4077                                end
4078                            else
4079                                start = nxt
4080                            end
4081                        elseif id == math_code then
4082                            start = getnext(endofmath(start))
4083                        elseif id == dir_code then
4084                            topstack, rlmode = txtdirstate(start,dirstack,topstack,rlparmode)
4085                            start = nxt
4086                     -- elseif id == par_code and startofpar(start) then
4087                     --     rlparmode, rlmode = pardirstate(start)
4088                     --     start = nxt
4089                        else
4090                            start = nxt
4091                        end
4092                    end
4093                end
4094            end
4095
4096            if trace_steps then -- ?
4097                registerstep(head)
4098            end
4099
4100        end
4101
4102        nesting = nesting - 1
4103
4104        return head
4105    end
4106
4107    -- This is not an official helper and used for tracing experiments. It can be changed as I like
4108    -- at any moment. At some point it might be used in a module that can help font development.
4109
4110    function otf.datasetpositionprocessor(head,font,direction,dataset)
4111
4112        currentfont     = font
4113        currentdynamic  = false
4114        currentscale    = 1000
4115        currentxscale   = 1000
4116        currentyscale   = 1000
4117        tfmdata         = fontdata[font]
4118        descriptions    = tfmdata.descriptions -- only needed in gref so we could pass node there instead
4119        characters      = tfmdata.characters   -- but this branch is not entered that often anyway
4120  local resources       = tfmdata.resources
4121        marks           = resources.marks
4122        classes         = resources.classes
4123        threshold,
4124        factor          = getthreshold(font)
4125        checkmarks      = tfmdata.properties.checkmarks
4126
4127        if type(dataset) == "number" then
4128            dataset = otfdataset(tfmdata,font,0)[dataset]
4129        end
4130
4131        local sequence  = dataset[3] -- sequences[s] -- also dataset[5]
4132        local typ       = sequence.type
4133     -- local gpossing  = typ == "gpos_single" or typ == "gpos_pair" -- store in dataset
4134
4135     -- gpos_contextchain gpos_context
4136
4137     -- if not gpossing then
4138     --     return head, false
4139     -- end
4140
4141        local handler   = handlers[typ] -- store in dataset
4142        local steps     = sequence.steps
4143        local nofsteps  = sequence.nofsteps
4144
4145        local done      = false
4146        local dirstack  = { nil } -- could move outside function but we can have local runs (maybe a few more nils)
4147        local start     = head
4148        local initialrl = (direction == righttoleft_code) and -1 or 0
4149        local rlmode    = initialrl
4150        local rlparmode = initialrl
4151        local topstack  = 0
4152        local merged    = steps.merged
4153
4154     -- local matches   = false
4155        local position  = 0
4156
4157        while start do
4158            local nxt, char, id = isnextchar(start,currentfont,currentdynamic)
4159            if char then
4160                position = position + 1
4161                local m = merged[char]
4162                if m then
4163                    -- no skiphash check here
4164                    for i=m[1],m[2] do
4165                        local step = steps[i]
4166                        local lookupcache = step.coverage
4167                        local lookupmatch = lookupcache[char]
4168currentscale, currentxscale, currentyscale = getscales(start)
4169                        if lookupmatch then
4170--                             currentscale, currentxscale, currentyscale = getscales(start)
4171                            local ok
4172                            head, start, ok = handler(head,start,dataset,sequence,lookupmatch,rlmode,skiphash,step)
4173                            if ok then
4174                             -- if matches then
4175                             --     matches[position] = i
4176                             -- else
4177                             --     matches = { [position] = i }
4178                             -- end
4179                                break
4180                            elseif not start then
4181                                break
4182                            end
4183                        end
4184                    end
4185                    if start then
4186                        start = getnext(start) -- start can be new
4187                    end
4188                else
4189                 -- start = getnext(start)
4190                    start = nxt
4191                end
4192            elseif char == false or id == glue_code then
4193                -- a different font|state or glue (happens often)
4194                start = nxt
4195            elseif id == math_code then
4196                start = getnext(endofmath(start))
4197            elseif id == dir_code then
4198                topstack, rlmode = txtdirstate(start,dirstack,topstack,rlparmode)
4199                start = nxt
4200         -- elseif id == par_code and startofpar(start) then
4201         --     rlparmode, rlmode = pardirstate(start)
4202         --     start = nxt
4203            else
4204                start = nxt
4205            end
4206        end
4207
4208        return head
4209    end
4210
4211    -- end of experiment
4212
4213end
4214
4215-- so far
4216
4217do
4218
4219    local plugins = { }
4220    otf.plugins   = plugins
4221
4222    local report  = logs.reporter("fonts")
4223    local warned  = false
4224    local okay    = { text = true }
4225
4226    function otf.registerplugin(name,f)
4227        if type(name) == "string" and type(f) == "function" then
4228            plugins[name] = { name, f }
4229            if okay[name] then
4230                -- no warning (e.g. the diagnostic text plugin)
4231            else
4232                report("plugin %a has been loaded, please be aware of possible side effects",name)
4233                if not warned then
4234                    if logs.pushtarget then
4235                        logs.pushtarget("log")
4236                    end
4237                    report("Plugins are not officially supported unless stated otherwise. This is because")
4238                    report("they bypass the regular font handling and therefore some features in ConTeXt")
4239                    report("(especially those related to fonts) might not work as expected or might not work")
4240                    report("at all. Some plugins are for testing and development only and might change")
4241                    report("whenever we feel the need for it.")
4242                    report()
4243                    if logs.poptarget then
4244                        logs.poptarget()
4245                    end
4246                    warned = true
4247                end
4248            end
4249        end
4250    end
4251
4252    function otf.plugininitializer(tfmdata,value)
4253        if type(value) == "string" then
4254            tfmdata.shared.plugin = plugins[value]
4255        end
4256    end
4257
4258    function otf.pluginprocessor(head,font,dynamic,direction) -- n
4259        local s = fontdata[font].shared
4260        local p = s and s.plugin
4261        if p then
4262            if trace_plugins then
4263                report_process("applying plugin %a",p[1])
4264            end
4265            return p[2](head,font,dynamic,direction)
4266        else
4267            return head, false
4268        end
4269    end
4270
4271end
4272
4273function otf.featuresinitializer(tfmdata,value)
4274    -- nothing done here any more
4275end
4276
4277registerotffeature {
4278    name         = "features",
4279    description  = "features",
4280    default      = true,
4281    initializers = {
4282        position = 1,
4283        node     = otf.featuresinitializer,
4284        plug     = otf.plugininitializer,
4285    },
4286    processors   = {
4287        node     = otf.featuresprocessor,
4288        plug     = otf.pluginprocessor,
4289    }
4290}
4291
4292-- Moved here (up) a bit. This doesn't really belong in generic so it will move to a
4293-- context module some day. It is kind of bsolete anyway.
4294
4295local function markinitializer(tfmdata,value)
4296    local properties = tfmdata.properties
4297    properties.checkmarks = value
4298end
4299
4300registerotffeature {
4301    name         = "checkmarks",
4302    description  = "check mark widths",
4303    default      = true,
4304    initializers = {
4305        node     = markinitializer,
4306    },
4307}
4308
4309-- This can be used for extra handlers, but should be used with care! We implement one
4310-- here but some more can be found in the osd (script devanagary) file. Now watch out:
4311-- when a handler has steps, it is called as the other ones, but when we have no steps,
4312-- we use a different call:
4313--
4314--   function(head,dataset,sequence,initialrl,font,dynamic)
4315--       return head
4316--   end
4317--
4318-- Also see (!!).
4319
4320otf.handlers = handlers
4321
Source Browser ?
