3

I've been trying to set up a CKAN installation for use as a Caribbean open data portal for a few weeks now but I've been running into challenges when I try to set up the CKAN Harvester and import some data sets. More specifically, I'm running into two specific errors.

1) The Harvester does not work unless I start the gather and fetch queues manually in the command line. If I don't do that, I get a simple error when I try to run the harvester on a data source from the front-end.

An error occurred: [2.0]

2) Like I said, I've been able to work around this error by starting those queues via the terminal but I'm guessing that isn't how it's supposed to work (I shouldn't need to do that right?). The second issue occurs when I resolve the first one and has to do with me trying to set the "default_groups" parameter.

If I set it to an empty set like so --> "default_groups" : []

There are no issues.

But if I try to include groups like so ---> "default_groups" : ["trinidad"]

I get this error :

The form contains invalid entries:

Config: Error parsing the configuration options: default_groups must be a list of group names/ids (i.e. strings)

I've done my best to be thorough here but please feel free to ask me any additional questions for clarification. I'd be very grateful for any insight/help with figuring this out.

Lastly, I have a couple of hunches that I'm still exploring. One is that there might be a version compatibility issue with the more recent version of CKAN (2.5) and the current Harvester version. I suspect this because the other versions of CKAN where the Harvester is actually working are running 2.2.

Not sure if that will mean anything to someone more familiar with the platform.

I'm open to all ideas, solutions, hunches, or whatever else might help me resolve this!

Cheers!

Improve this question

2 Answers 2

Reset to default
2

  1. The harvester processes need to be running in order for the harvesting to work. As you mention you can fire up these manually for debugging and development, but in production you will want to use something like Supervisor to keep them running. The documentation provides an example configuration.

  2. There was a recently merged fix involving default_groups. Could you pull the latest version of ckanext-harvest, restart the server and all harvester processes and try again?

  3. The current ckanext-harvest should work fine with all versions of CKAN including 2.5

Hope this helps

Improve this answer
2
  • Regarding 2 'default_groups' - it looks like you have that fix installed, judging by the error message. This could be caused if you have an old version of simplejson (less than 2.0) - that the case? I suggest you open an issue on ckanext-harvest and we try to debug there.
    – D Read
    Jun 22, 2016 at 9:42
  • @DRead Thanks so much for this tip but upgrading the simplejson version had no discernible effect. The error still persists. I've opened an issue for this as you suggested.
    – Nick Jaden Williams
    Jun 23, 2016 at 1:09
1

A second fix has been required to get default_groups working for Nick, thanks to his feedback and proposed PR as well.

Note that that PR is still waiting review, so hopefully someone will confirm it works and it can be merged.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.