2

We want to provide version control to a small sized Virtuoso SPARQL endpoint but the workflow we use is quite cumbersome and always generates huge diffs for blank nodes because they get new IDs every time.

My question: Is there a more efficient way than the following that also minimizes diffs for blank nodes?

  1. Go to the http://myendpoint/conductor -> Database -> Interactive SQL, and execute dump_one_graph() for all graphs
  2. Get the new dump from the server via scp
  3. Unpack the dump .ttl.gz files.
  4. Convert the dump files to ntriples.
  5. Sort the dump files so that the diffs are minimized.
  6. Replace the files in the repository with the dump files.
  7. View the diff and perform partial commits with appropriate messages.

For steps 3-6, we use the following Python script:

import glob
import gzip
import rdflib
import sys
import os
import re

if(len(sys.argv)<3):
    print("Usage: virtuoso2git infolder outfolder prefix")
    sys.exit(1)

infolder = sys.argv[1]
outfolder = sys.argv[2]
prefix = sys.argv[3]
suffix = "000001"

ttl = infolder+"/"+prefix+"*.ttl"
files = [f for f in [glob.glob(p) for p in [ttl,ttl+".gz"]] for f in f]

if(len(files)<1):
    print("No files found in folder",infolder)
    sys.exit(1)

if not os.path.exists(outfolder):
    os.mkdir(outfolder)

for f in files:
    print("Read",f)
    outname = re.search(prefix+r"(.*)"+suffix,f).group(1) + ".nt"
    outpath = outfolder + "/" + outname

    if(f.endswith(".gz")):
        stream = gzip.open(f,"rt")
    else:
        stream = open(f,"rt")
    turtle = stream.read()
    g = rdflib.Graph()
    try:
        g.parse(publicID="/" ,format="n3",data=turtle)
        print("Write",outpath)
        g.serialize(destination=outpath,format="nt", encoding=None)
        os.system("sort "+outpath+" -o "+outpath)
    except rdflib.plugins.parsers.notation3.BadSyntax:
        print("Cannot parse file ",f)
        continue

The script is also hosted at https://github.com/KonradHoeffner/virtuoso2git but with small adjustments for our project.

1
  • 1
    As well as I remember, Virtuoso supports the<nodeId://b1> syntax (similar to <_:b1> in Jena. These blank node labels are more or less persistent. May 20 '20 at 22:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.