Source Browser ?

1if not modules then modules = { } end modules ['node-syn'] = {
2    version   = 1.001,
3    comment   = "companion to node-ini.mkiv",
4    author    = "Hans Hagen, PRAGMA-ADE, Hasselt NL",
5    copyright = "PRAGMA ADE / ConTeXt Development Team",
6    license   = "see context related readme files"
7}
8
9-- Because we have these fields in some nodes that are used by synctex, and because
10-- some users seem to like that feature, I decided to implement a variant that might
11-- work out better for ConTeXt. This is experimental code. I don't use it myself so
12-- it will take a while to mature. There will be some helpers that one can use in
13-- more complex situations like included xml files. Currently (somewhere else) we
14-- take care of valid files, that is: we prohibit access to files in the tree
15-- because we don't want users to mess up styles.
16--
17-- It is unclear how the output gets interpreted but by reverse engineering (and
18-- stripping) the file generated by generic synctex, I got there eventually. For
19-- instance, we only need to be able to go back to a place where text is entered,
20-- but still we need all that redundant box wrapping. Anyway, I was able to get a
21-- minimal output and cross my fingers that the parser used in editors is not
22-- changed in fundamental ways.
23--
24-- I only tested SumatraPDF with SciTE, for which one needs to configure in the
25-- viewer:
26--
27--   InverseSearchCmdLine = c:\data\system\scite\wscite\scite.exe "%f" "-goto:%l" $
28--
29-- In fact, a way more powerful implementation would have been not to add a library
30-- to a viewer, but letthe viewer call an external program:
31--
32--   InverseSearchCmdLine = mtxrun.exe --script synctex --edit --name="%f" --line="%l" $
33--
34-- which would (re)launch the editor in the right spot. That way we can really
35-- tune well to the macro package used and also avoid the fuzzy heuristics of
36-- the library.
37--
38-- Unfortunately syntex always removes the files at the end and not at the start
39-- (this happens in synctexterminate) so we need to work around that by using an
40-- intermediate file. This is no big deal in context (which has a runner) but
41-- definitely not nice. This is no longer an issue in LMTX because there we don't
42-- have the library interfering with files.
43--
44-- The visualizer code is only needed for testing so we don't use fancy colors or
45-- provide more detail. After all we're only interested in rendered source text
46-- anyway. We try to play safe which sometimes means that we'd better no go
47-- somewhere than go someplace wrong.
48--
49-- A previous version had a mode for exporting boxes and such but I removed that
50-- as it made no sense. Also, collecting output in a table was not faster than
51-- directly piping to the file, probably because the amount is not that large. We
52-- keep some left-overs commented.
53--
54-- A significant reduction in file size can be realized when reusing the same
55-- values. Actually that triggered the current approach in ConTeXt. In the latest
56-- synctex parser vertical positions can be repeated by an "=" sign but for some
57-- reason only for that field. It's probably trivial to do that for all of "w h d v
58-- h" but it currently not the case so I'll delay that till all are supported. (We
59-- could benefit a lot from such a repetition scheme but not much from a "v" alone
60-- which -alas- indicates that synctex is still mostly a latex targeted story.)
61--
62-- It's kind of hard to fight the parser because it really wants to go to some file
63-- but maybe some day I can figure it out. Some untagged text (in the pdf) somehow
64-- gets seen as part of the last box. Anonymous content is simply not part of the
65-- concept. Using a dummy name doesn't help either as the editor gets a signal to
66-- open that dummy. Even an empty filename doesn't work.
67--
68-- We output really simple and compact code, like:
69--
70-- SyncTeX Version:1
71-- Input:1:e:/tmp/oeps.tex
72-- Input:2:c:/data/develop/context/sources/klein.tex
73-- Output:pdf
74-- Magnification:1000
75-- Unit:1
76-- X Offset:0
77-- Y Offset:0
78-- Content:
79-- !160
80-- {1
81-- h0,0:0,0,0,0,0
82-- v0,0:0,55380990:39158276,55380990,0
83-- h2,1:4661756,9176901:27969941,655360,327680
84-- h2,2:4661756,10125967:26048041,655360,327680
85-- h2,3:30962888,10125967:1668809,655360,327680
86-- h2,3:4661756,11075033:23142527,655360,327680
87-- h2,4:28046650,11075033:4585047,655360,327680
88-- h2,4:4661756,12024099:22913954,655360,327680
89-- h2,5:27908377,12024099:4723320,655360,327680
90-- h2,5:4661756,12973165:22918783,655360,327680
91-- h2,6:27884864,12973165:4746833,655360,327680
92-- h2,6:4661756,13922231:18320732,655360,327680
93-- ]
94-- !533
95-- }1
96-- Input:3:c:/data/develop/context/sources/ward.tex
97-- !57
98-- {2
99-- h0,0:0,0,0,0,0
100-- v0,0:0,55380990:39158276,55380990,0
101-- h3,1:4661756,9176901:18813145,655360,327680
102-- h3,2:23713999,9176901:8917698,655360,327680
103-- h3,2:4661756,10125967:10512978,655360,327680
104-- h3,3:15457206,10125967:17174491,655360,327680
105-- h3,3:4661756,11075033:3571223,655360,327680
106-- h3,4:8459505,11075033:19885281,655360,327680
107-- h3,5:28571312,11075033:4060385,655360,327680
108-- h3,5:4661756,12024099:15344870,655360,327680
109-- ]
110-- !441
111-- }2
112-- !8
113-- Postamble:
114-- Count:22
115-- !23
116-- Post scriptum:
117--
118-- But for some reason, when the pdf file has some extra content (like page numbers)
119-- the main document is consulted. Bah. It would be nice to have a mode for *only*
120-- looking at marked areas. It somehow works not but maybe depends on the parser.
121--
122-- Supporting reuseable objects makes not much sense as these are often graphics or
123-- ornamental. They can not have hyperlinks etc (at least not without some hackery
124-- which I'm not willing to do) so basically they are sort of useless for text.
125--
126-- Some generic (more clever code) has been removed as I don't see things change
127-- that much. However, we do have a problem. Files that work okay in sumatra, and
128-- on linux in zathura pdf, don't work well in for instance okular. When Mikael
129-- Sundqvist installed SciTE we tried to get it working with Okular and failed to
130-- do so. It took us a while to figure out the reason:
131--
132-- - There has been a change in library. In Sumatra there is built in code for
133--   an older format and it uses an old version of the library for the new format.
134-- - In Zathura an old version of the library is used.
135-- - Okular seems to use a newer version.
136--
137-- So, then we tested how a simple latex file performed ane eventually we figured
138-- out a work-around.
139--
140-- The parser hooked into the viewer is way too complex for what context needs. We
141-- just need areas but there are heuristics, probably geared for how latex outputs
142-- pages. It also looks like the newer library ignores (what it calls) void boxes
143-- which is what we use. However, when we put a fake rule in there we can get it
144-- working; it's an ugly hack but hopefully one that will work from now on. What we
145-- did notice is that an empty box with dimensions didn't work in latex either. Even
146-- stranger was that the file produced by lualatex did have the right line numbers
147-- while the file made by pdflatex didn't, so actually we assume support for latex
148-- is also fragile.
149--
150-- We still hope that some day editors will delegate resolving the coordinates and
151-- launching the editor to an external program (script). Some viewers communicate
152-- via a so called dbus already so wel'll see. If that (ever) happens we can drop
153-- the current synctex output for a more efficient one.
154
155-- This version has some details stripped because they are irrelevant in the
156-- perspective of \LUAMETATEX.
157--
158-- Compression doesn't work out well because there are old and new synctex
159-- versions in viewers and it makes no sense to accomodate this and guessing if
160-- it works. It's also only related to vertical coordinates so the savings are
161-- little.
162
163local type, rawset = type, rawset
164local concat = table.concat
165local formatters = string.formatters
166local replacesuffix, suffixonly, nameonly, collapsepath = file.replacesuffix, file.suffix, file.nameonly, file.collapsepath
167local openfile, removefile = io.open, os.remove
168
169local report_system = logs.reporter("system")
170
171local tex                 = tex
172local texget              = tex.get
173
174local nuts                = nodes.nuts
175
176local getid               = nuts.getid
177local getlist             = nuts.getlist
178local setlist             = nuts.setlist
179local getnext             = nuts.getnext
180local getwhd              = nuts.getwhd
181local getsubtype          = nuts.getsubtype
182
183local nodecodes           = nodes.nodecodes
184local kerncodes           = nodes.kerncodes
185
186local glyph_code          <const> = nodecodes.glyph
187local disc_code           <const> = nodecodes.disc
188local glue_code           <const> = nodecodes.glue
189local penalty_code        <const> = nodecodes.penalty
190local kern_code           <const> = nodecodes.kern
191local hlist_code          <const> = nodecodes.hlist
192local vlist_code          <const> = nodecodes.vlist
193local fontkern_code       <const> = kerncodes.fontkern
194
195local insertbefore        = nuts.insertbefore
196local insertafter         = nuts.insertafter
197
198local nextnode            = nuts.traversers.node
199
200local nodepool            = nuts.pool
201local new_latelua         = nodepool.latelua
202local new_rule            = nodepool.rule
203local new_outline_rule    = nodepool.outlinerule
204local new_kern            = nodepool.kern
205
206local getdimensions       = nuts.dimensions
207local getrangedimensions  = nuts.rangedimensions
208
209local getinputfields      = nuts.getinputfields
210local forceinputstatefile = tex.forceinputstatefile
211local forceinputstateline = tex.forceinputstateline
212local getinputstateline   = tex.getinputstateline
213local setinputstatemode   = tex.setinputstatemode
214
215local foundintree         = resolvers.foundintree
216
217local getpagedimensions   = layouts.getpagedimensions
218
219local eol        <const>  = "\010"
220local z_hlist    <const>  = "[0,0:0,0:0,0,0\010"
221local s_hlist    <const>  = "]\010"
222local f_hvoid             = formatters["h%i,%i:%i,%i:%i,%i,%i\010"]
223local f_hlist             = formatters["(%i,%i:%i,%i:%i,%i,%i\010"]
224local s_hlist    <const>  = ")\010"
225local f_rule              = formatters["r%i,%i:%i,%s:%i,%i,%i\010"]
226local f_plist             = formatters["[%i,%i:%i,%i:%i,%i,%i\010"]
227local s_plist    <const>  = "]\010"
228
229local synctex             = luatex.synctex or { }
230luatex.synctex            = synctex
231
232local getpos ; getpos = function() getpos = job.positions.getpos return getpos() end
233
234-- status stuff
235
236local enabled = false
237local userule = false
238local paused  = 0
239local used    = false
240local never   = false
241
242-- the file name stuff (called tags in synctex)
243
244local noftags            = 0
245local stnums             = { }
246local nofblocked         = 0
247local blockedfilenames   = { }
248local blockedsuffixes    = {
249    mkii = true,
250    mkiv = true,
251    mkvi = true,
252    mkxl = true,
253    mklx = true,
254    mkix = true,
255    mkxi = true,
256 -- lfg  = true,
257}
258
259local sttags = table.setmetatableindex(function(t,fullname)
260    local name = collapsepath(fullname)
261    if blockedsuffixes[suffixonly(name)] then
262        -- Just so that I don't get the ones on my development tree.
263        nofblocked = nofblocked + 1
264        return 0
265    elseif blockedfilenames[nameonly(name)] then
266        -- So we can block specific files.
267        nofblocked = nofblocked + 1
268        return 0
269    elseif foundintree(name) then
270        -- One shouldn't edit styles etc this way.
271        nofblocked = nofblocked + 1
272        return 0
273    else
274        noftags = noftags + 1
275        t[name] = noftags
276        if name ~= fullname then
277            t[fullname] = noftags
278        end
279        stnums[noftags] = name
280        return noftags
281    end
282end)
283
284function synctex.blockfilename(name)
285    blockedfilenames[nameonly(name)] = name
286end
287
288function synctex.setfilename(name,line)
289    if paused == 0 and name then
290        forceinputstatefile(sttags[name])
291        if line then
292            forceinputstateline(line)
293        end
294    end
295end
296
297function synctex.resetfilename()
298    if paused == 0 then
299        forceinputstatefile(0)
300        forceinputstateline(0)
301    end
302end
303
304do
305
306    local nesting = 0
307    local ignored = false
308
309    function synctex.pushline()
310        nesting = nesting + 1
311        if nesting == 1 then
312            local l = getinputstateline()
313            ignored = l and l > 0
314            if not ignored then
315                forceinputstateline(texget("inputlineno"))
316            end
317        end
318    end
319
320    function synctex.popline()
321        if nesting == 1 then
322            if not ignored then
323                forceinputstateline()
324                ignored = false
325            end
326        end
327        nesting = nesting - 1
328    end
329
330end
331
332-- the node stuff
333
334local filehandle = nil
335local nofsheets  = 0
336local nofobjects = 0
337local last       = 0
338local filesdone  = 0
339local sncfile    = false
340
341local function writeanchor()
342    local size = filehandle:seek("end")
343    filehandle:write("!",size-last,eol)
344    last = size
345end
346
347local function writefiles()
348    local total = #stnums
349    if filesdone < total then
350        for i=filesdone+1,total do
351            filehandle:write("Input:",i,":",stnums[i],eol)
352        end
353        filesdone = total
354    end
355end
356
357local function makenames()
358    sncfile = replacesuffix(tex.jobname,"synctex")
359end
360
361local function flushpreamble()
362    makenames()
363    filehandle = openfile(sncfile,"wb")
364    if filehandle then
365        filehandle:setvbuf("full",64*1024)
366        filehandle:write("SyncTeX Version:1",eol)
367        writefiles()
368        filehandle:write("Output:pdf",eol)
369        filehandle:write("Magnification:1000",eol)
370        filehandle:write("Unit:1",eol)
371        filehandle:write("X Offset:0",eol)
372        filehandle:write("Y Offset:0",eol)
373        filehandle:write("Content:",eol)
374        flushpreamble = function()
375            writefiles()
376            return filehandle
377        end
378    else
379        enabled = false
380    end
381    return filehandle
382end
383
384function synctex.wrapup()
385    sncfile = nil
386end
387
388local function flushpostamble()
389    if not filehandle then
390        return
391    end
392    writeanchor()
393    filehandle:write("Postamble:",eol)
394    filehandle:write("Count:",nofobjects,eol)
395    writeanchor()
396    filehandle:write("Post scriptum:",eol)
397    filehandle:close()
398    enabled = false
399end
400
401local x_hlist  do
402
403    local function doaction(data)
404        local pagewidth, pageheight = getpagedimensions() -- we could save some by setting it per page
405        local x, y = getpos()
406        local t = data[1]
407        local l = data[2]
408        local w = data[3]
409        local h = data[4]
410        local d = data[5]
411        y = pageheight - y
412        if userule then
413            -- This cheat works in viewers that use the newer library:
414            filehandle:write(f_hlist(t,l,x,y,0,0,0)) -- w,h,d))
415            filehandle:write(f_rule(t,l,x,y,w,h,d))
416            filehandle:write(s_hlist)
417        else
418            -- This works in viewers that use the older library:
419            filehandle:write(f_hvoid(t,l,x,y,w,h,d))
420        end
421        nofobjects = nofobjects + 1
422    end
423
424    x_hlist = function(head,current,t,l,w,h,d)
425        if filehandle then
426            return insertbefore(head,current,new_latelua { action = doaction, t, l, w, h, d })
427        else
428            return head
429        end
430    end
431
432end
433
434-- color is already handled so no colors
435
436local collect      = nil
437local fulltrace    = false
438local trace        = false
439local height       = 10 * 65536
440local depth        =  5 * 65536
441local traceheight  =      32768
442local tracedepth   =      32768
443local outlinewidth = 65536 / 4
444
445trackers.register("system.synctex.visualize", function(v)
446    trace     = v
447    fulltrace = v == "real"
448end)
449
450local collect_min  do
451
452    local function inject(head,first,last,tag,line)
453        local w, h, d = getdimensions(first,getnext(last))
454        if h < height then
455            h = height
456        end
457        if d < depth then
458            d = depth
459        end
460        if trace then
461            if fulltrace then
462                head = insertbefore(head,first,new_outline_rule(w,h,d,outlinewidth))
463            else
464                head = insertbefore(head,first,new_rule(w,traceheight,tracedepth))
465            end
466            head = insertbefore(head,first,new_kern(-w))
467        end
468        head = x_hlist(head,first,tag,line,w,h,d)
469        return head
470    end
471
472    collect_min = function(head)
473        local current = head
474        while current do
475            local id = getid(current)
476            if id == glyph_code then
477                local first = current
478                local last  = current
479                local tag   = 0
480                local line  = 0
481                while true do
482                    if id == glyph_code then
483                        local tc, lc = getinputfields(current)
484                        if tc and tc > 0 then
485                            tag  = tc
486                            line = lc
487                        end
488                        last = current
489                    elseif id == disc_code or (id == kern_code and getsubtype(current) == fontkern_code) then
490                        last = current
491                    else
492                        if tag > 0 then
493                            head = inject(head,first,last,tag,line)
494                        end
495                        break
496                    end
497                    current = getnext(current)
498                    if current then
499                        id = getid(current)
500                    else
501                        if tag > 0 then
502                            head = inject(head,first,last,tag,line)
503                        end
504                        return head
505                    end
506                end
507            end
508            -- pick up (as id can have changed)
509            if id == hlist_code or id == vlist_code then
510                local list = getlist(current)
511                if list then
512                    local l = collect(list)
513                    if l ~= list then
514                        setlist(current,l)
515                    end
516                end
517            end
518            current = getnext(current)
519        end
520        return head
521    end
522
523end
524
525local collect_max  do
526
527    local function inject(parent,head,first,last,tag,line)
528        local w, h, d = getrangedimensions(parent,first,getnext(last))
529        if h < height then
530            h = height
531        end
532        if d < depth then
533            d = depth
534        end
535        if trace then
536            if fulltrace then
537                head = insertbefore(head,first,new_outline_rule(w,h,d,outlinewidth))
538            else
539                head = insertbefore(head,first,new_rule(w,traceheight,tracedepth))
540            end
541            head = insertbefore(head,first,new_kern(-w))
542        end
543        head = x_hlist(head,first,tag,line,w,h,d)
544        return head
545    end
546
547    collect_max = function(head,parent)
548        local current = head
549        while current do
550            local id = getid(current)
551            if id == glyph_code then
552                local first = current
553                local last  = current
554                local tag   = 0
555                local line  = 0
556                while true do
557                    if id == glyph_code then
558                        local tc, lc = getinputfields(current)
559                        if tc and tc > 0 then
560                            if tag > 0 and (tag ~= tc or line ~= lc) then
561                                head  = inject(parent,head,first,last,tag,line)
562                                first = current
563                            end
564                            tag  = tc
565                            line = lc
566                            last = current
567                        else
568                            if tag > 0 then
569                                head = inject(parent,head,first,last,tag,line)
570                                tag  = 0
571                            end
572                            first = nil
573                            last  = nil
574                        end
575                    elseif id == disc_code then
576                        if not first then
577                            first = current
578                        end
579                        last = current
580                    elseif id == kern_code and getsubtype(current) == fontkern_code then
581                        if first then
582                            last = current
583                        end
584                    elseif id == glue_code then
585                    elseif id == penalty_code then
586                        -- go on (and be nice for math)
587                    else
588                        if tag > 0 then
589                            head = inject(parent,head,first,last,tag,line)
590                            tag  = 0
591                        end
592                        break
593                    end
594                    current = getnext(current)
595                    if current then
596                        id = getid(current)
597                    else
598                        if tag > 0 then
599                            head = inject(parent,head,first,last,tag,line)
600                        end
601                        return head
602                    end
603                end
604            end
605            -- pick up (as id can have changed)
606            if id == hlist_code or id == vlist_code then
607                local list = getlist(current)
608                if list then
609                    local l = collect(list,current)
610                    if l and l ~= list then
611                        setlist(current,l)
612                    end
613                end
614            end
615            current = getnext(current)
616        end
617        return head
618    end
619
620end
621
622collect = collect_max
623
624function synctex.collect(head,where)
625    if enabled and where ~= "object" then
626        return collect(head,head)
627    else
628        return head
629    end
630end
631
632-- also no solution for bad first file resolving in sumatra
633
634function synctex.start()
635    if enabled then
636        nofsheets = nofsheets + 1 -- could be realpageno
637        if flushpreamble() then
638            writeanchor()
639            filehandle:write("{",nofsheets,eol)
640            -- this seems to work:
641         -- local pagewidth, pageheight = getpagedimensions()
642            filehandle:write(f_plist(1,0,0,0,0,0,0))
643        end
644    end
645end
646
647function synctex.stop()
648    if enabled then
649        filehandle:write(s_plist)
650        writeanchor()
651        filehandle:write("}",nofsheets,eol)
652        nofobjects = nofobjects + 2
653    end
654end
655
656local enablers  = { }
657local disablers = { }
658
659function synctex.registerenabler(f)
660    enablers[#enablers+1] = f
661end
662
663function synctex.registerdisabler(f)
664    disablers[#disablers+1] = f
665end
666
667function synctex.enable(use_rule)
668    if tex.getinteraction() ~= tex.interactioncodes.errorstop then
669        report_system("synctex functionality is only enabled in error stop mode")
670    elseif not never and not enabled then
671        enabled = true
672        userule = use_rule
673        setinputstatemode(3) -- we want details
674        if not used then
675            nodes.tasks.enableaction("shipouts","luatex.synctex.collect")
676            report_system("synctex functionality is enabled, expect 5-10 pct runtime overhead!")
677            used = true
678        end
679        for i=1,#enablers do
680            enablers[i](true)
681        end
682        -- we have a different trigger moment in lmtx
683        flushpreamble()
684    end
685end
686
687function synctex.disable()
688    if enabled then
689        setinputstatemode(0)
690        report_system("synctex functionality is disabled!")
691        enabled = false
692        for i=1,#disablers do
693            disablers[i](false)
694        end
695    end
696end
697
698function synctex.finish()
699    if enabled then
700        flushpostamble()
701    else
702        makenames()
703        removefile(sncfile)
704    end
705end
706
707local filename = nil
708
709function synctex.pause()
710    paused = paused + 1
711    if enabled and paused == 1 then
712        setinputstatemode(0)
713    end
714end
715
716function synctex.resume()
717    if enabled and paused == 1 then
718        setinputstatemode(3)
719    end
720    paused = paused - 1
721end
722
723-- not the best place
724
725luatex.registerstopactions(synctex.finish)
726
727statistics.register("synctex tracing",function()
728    if used then
729        return string.format("%i referenced files, %i files ignored, %i objects flushed, logfile: %s",
730            noftags,nofblocked,nofobjects,sncfile)
731    end
732end)
733
734local implement = interfaces.implement
735local variables = interfaces.variables
736
737function synctex.setup(t)
738    if t.state == variables.never then
739        synctex.disable() -- just in case
740        never = true
741        return
742    end
743    if t.method == variables.max then
744        collect = collect_max
745    else
746        collect = collect_min
747    end
748    if t.state == variables.start then
749        synctex.enable(false)
750    elseif t.state == variables["repeat"] then
751        synctex.enable(true)
752    else
753        synctex.disable()
754    end
755end
756
757implement {
758    name      = "synctexblockfilename",
759    arguments = "string",
760    protected = true,
761    public    = true,
762    actions   = synctex.blockfilename,
763}
764
765implement {
766    name      = "synctexsetfilename",
767    arguments = "string",
768    protected = true,
769    public    = true,
770    actions   = synctex.setfilename,
771}
772
773implement {
774    name      = "synctexresetfilename",
775    protected = true,
776    public    = true,
777    actions   = synctex.resetfilename,
778}
779
780implement {
781    name      = "setupsynctex",
782    actions   = synctex.setup,
783    arguments = {
784        {
785            { "state" },
786            { "method" },
787        },
788    },
789}
790
791implement {
792    name    = "synctexpause",
793    actions = synctex.pause,
794}
795
796implement {
797    name    = "synctexresume",
798    actions = synctex.resume,
799}
800
801implement {
802    name    = "synctexpushline",
803    actions = synctex.pushline,
804}
805
806implement {
807    name    = "synctexpopline",
808    actions = synctex.popline,
809}
810
811implement {
812    name    = "synctexdisable",
813    actions = synctex.disable,
814}
815