We want to provide version control to a small sized Virtuoso SPARQL endpoint but the workflow we use is quite cumbersome and always generates huge diffs for blank nodes because they get new IDs every time.
My question: Is there a more efficient way than the following that also minimizes diffs for blank nodes?
- Go to the http://myendpoint/conductor -> Database -> Interactive SQL, and execute
dump_one_graph()for all graphs
- Get the new dump from the server via
- Unpack the dump .ttl.gz files.
- Convert the dump files to ntriples.
- Sort the dump files so that the diffs are minimized.
- Replace the files in the repository with the dump files.
- View the diff and perform partial commits with appropriate messages.
For steps 3-6, we use the following Python script:
import glob import gzip import rdflib import sys import os import re if(len(sys.argv)<3): print("Usage: virtuoso2git infolder outfolder prefix") sys.exit(1) infolder = sys.argv outfolder = sys.argv prefix = sys.argv suffix = "000001" ttl = infolder+"/"+prefix+"*.ttl" files = [f for f in [glob.glob(p) for p in [ttl,ttl+".gz"]] for f in f] if(len(files)<1): print("No files found in folder",infolder) sys.exit(1) if not os.path.exists(outfolder): os.mkdir(outfolder) for f in files: print("Read",f) outname = re.search(prefix+r"(.*)"+suffix,f).group(1) + ".nt" outpath = outfolder + "/" + outname if(f.endswith(".gz")): stream = gzip.open(f,"rt") else: stream = open(f,"rt") turtle = stream.read() g = rdflib.Graph() try: g.parse(publicID="/" ,format="n3",data=turtle) print("Write",outpath) g.serialize(destination=outpath,format="nt", encoding=None) os.system("sort "+outpath+" -o "+outpath) except rdflib.plugins.parsers.notation3.BadSyntax: print("Cannot parse file ",f) continue
The script is also hosted at https://github.com/KonradHoeffner/virtuoso2git but with small adjustments for our project.