15

I'd want to publish some "real-time" data, i.e. my FV panel energy production, on a server. Which is the best way of cataloguing the data and make it available?

Energy is just an example, say that I add weather info or anything else.

I looked into CKAN, but it seems that it's tuned more for finalized datasets, than real-time "streaming" data, which continuously grow over time.

The interface should be easy and accessible both from humans and automatic tools.

Improve this question
3
  • If you've got long series, netCDF format is a possible match for your needs. Please also have a look at OPeNDAP for serving data over the Net.
    – Deer Hunter
    Commented May 25, 2013 at 12:03
  • You can use Socrata : socrata.com
    – Julien
    Commented Jun 10, 2013 at 16:35
  • Take a look at geomesa geomesa.org
    – risail
    Commented Nov 2, 2016 at 20:28

4 Answers 4

Reset to default
11

You have a few options for real time (or "near real time", which is when you have a delay between the collection & time to serve it, or for those that sample at a lower cadence)

There are a lot of considerations when dealing with 'real time' data:

  1. Who is the intended audience? (and do they already have standards for serving this type of data?)
  2. Is the data of value over the long term, or only ephemerally?
  3. If the data is of value long term, at what cadence does it need to be at to be of value? Does that cadence change as the data ages?
  4. If the data is ephemeral, what is the maximum age for which it's of value?
  5. What is the maximum latency acceptable for the data?
  6. How large is each record / observation?
  7. How would the intended audience expect to work with the data?
  8. How many people are going to be calling this API?
  9. Is the data useful individually, or only as part of a larger sensor network?

... I'm probably missing a few things ... but without knowing the answers, I can't say if it makes sense to :

  • have an API that queries the sensors in real time
  • have the sensor report in every few minutes and serve the latest value
  • have some sort of data logger and serve the last 10 or 100 values
  • log to a database and have a protocol for requesting data from a given period of time
  • append each entry to a log, and roll the log every day (or hour, or some other time period).

If you're just looking to put up a few graphs on a website to show how much sunlight you get ... then just use RRDTool or something similar.

If you're trying to contribute back to a citizen science project ... you could look into the National Weather Service's Cooperative Observer Program. I don't know if they deal with solar irradiance though. DOE does, as they have their Solar Energy Potential map, but I don't know how they get the input for it. You can also try to find if there are any local mesonets in your area that need a node near you.

Also be aware that for some types of real time and NRT (near real time) data, there are APIs for reporting significant events, and interested parties connect and subscribe to the types of alerts they're interested in. Or individual sensors report back to some central clearing house, and people can then query that for data of interest. For instance, in astronomy there's the Gamma-ray Coordinates Network, for US earth science data, there's the NOAA Observing System Architecture.

Improve this answer
6
  • Thank you Joe. I may have asked a slightly wrong question, and maybe you could help me improve it. I know RRDTool, which as you say is very nice for putting up some graphs of some ephemereal data, showing also averages or so. I'd like, however, to put data from my sensors on a catalogue, and maybe they are of no value, but they could be interesting for someone, at least for comparison. I'd like to give them away as with CKAN, helping people direct linking them and reusing. Same could be with say blood pressure or quantified self data.
    – Metiu
    Commented May 25, 2013 at 21:43
  • It's quite likely that the data's not of much use to others without knowing all of the details about how it was collected. At a minimum, GPS coordinates and the pointing of the panels, possibly information about things in the area that might cause shadows. Then there's panel temperature, if you're using a peak power tracker, etc. ... my thought on the whole matter is that if someone out there wants this type of data, they'd have both a system to collect it and standards for how to collect & document it. Your collecting up the data in trying to find someone to use it isn't that useful.
    – Joe
    Commented May 26, 2013 at 1:19
  • Thank you for you comment, but that goes back to my original question: assume (why not?) I have all the details you mentioned (I might also be an institution or willing to share the open data provided by an institution currently in a horrible format) and I want to share that data, how do I do it in a useful, open data, standard way? The lack of answers in this sense makes me thinks that there's no standard or at least common practice as there is for city budgets. Is pachube/cosm/xively the only one working on this?
    – Metiu
    Commented May 26, 2013 at 2:54
  • Joe, I see from your profile that for example you publish very interesting data on a website. However, it's not easily mash-uppable with tools like google fusion tables, isn't there another way?
    – Metiu
    Commented May 26, 2013 at 10:35
  • Metiu : the intended audience for the data that I distribute primarily use SolarSoft, so I've also written tools there to work with the data. I've also worked with the SunPy group to build a new client to interact with the API that I maintain. You need to identify an audience, and know what they need of the data. Otherwise, it's like being a packrat. Situations like : "well, this data would have been useful to us, but without the panel temperature and an initial calibration from a controlled test we can't derive the irradiance".
    – Joe
    Commented May 26, 2013 at 10:56
7

Sounds like you could be interested in the Sensor Observation Service (SOS) standard published by the OGC. It's an open standard which describes a service to publish sensor readings and meta data.

I haven't been using it for a while now but I had an SOS server from 52North running successfully for a while.

Improve this answer
2

I can help. OpenSensors is a real-time service for publishing Open Data, partnered with the Open Data Institute, we are trying to make real-time Open Data access and publishing easy as possible enabling people to build cool stuff based upon it. All open data projects on OpenSensors.io are free to use, we only charge if you keep the data private.

Besides publishing open data, you can also search for, subscribe to and access historical data and real time data through a streaming firehose. We have help and getting started docs here.

I hope that helps, email me if you need a hand: [email protected]

Source: I work at OpenSensors.io

Improve this answer
1

Simpler than using a big framework for a single file is just to create a simple CSV file format (datetime, value) and then set up a GitHub repository to host the file.

You can then use a tool like cron, inotifywait or Gitwatch to periodically and automatically commit and push the local CSV file to the Github repository.

inotifywait -q -m -e CLOSE_WRITE --format="git commit -m 'autocommit on change' %w" data.csv | sh

Then on top of the Github-hosted file, you can create and host a simple visualization, perhaps with D3.js, that reads the "live" CSV file as input.

enter image description here

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.