Historical hourly solar generation data in California

Question

In 2013 the California Independent System Operator (CAISO, the bulk electric grid operator) published this image, dubbed it "the duck curve", and took the internet by storm (or, at least the part of the internet that tracks trends in renewable electricity generation):

Here's an explanation of the duck curve from Wikipedia:

[The] curve comes from the Net Load ("the difference between expected load and anticipated electricity production from the range of renewable energy sources"). In certain times of the year (namely Spring and Summer), the curves create a “belly” appearance in the midday that then drastically increases portraying an “arch” similar to the neck of a duck, consequently the name “The Duck Chart.”

Essentially, solar electricity production causes a reduction in net energy demand in the mid-day (the belly). Later, sunset coincides with increased demand, causing a sharp ramp (the neck). This effect is shown in the graph with 24 hours of solar and demand data on March 31.

This image generated lots of interest and people are still talking about it today -- lots of search results in the past year are still showing the same image.

But the most recent actual data on the chart is from 2013, and the duck shape was created by the projected net load curves from 2014 through 2020.

I tried to find some recent data from CAISO to see where things stand. To do this, I'll need hourly historical demand and solar production data, in order to calculate net load.

CAISO's OASIS tool includes historical load data, but as far as I could tell solar data is only included lumped with wind, and only goes back to 2016.

Is there another data source where I could find the historical hourly solar generation for the state of California?

---

I posted this question previously on electronics.se, but it was closed. Folks on meta there advised I re-post here.

Answer 1

Exact hourly solar/PV production will be with the grid operator, and if you have an academic/NGO credentials, maybe they can make you a 1-time export. I don't know California, but I've worked in the energy sector and because there is energy trading between stakeholders, no one is really eager to make open data.

The grid operator may not even have total exact PV production, because of how balancing areas are calculated, it would be up to each producers to report their production (and label it as PV).

---

If you don't need exact numbers, here's a potential hack. You can fit the grey curve "Solar output" from the Wikipedia article "Duck curve", which shouldn't be too hard because it's so smooth.

Then you'd want to download PV installations from a site like NREL, which has the production capacity and installation date: https://openpv.nrel.gov/search?state=CA

With some clever SQL/Pandas/R, you can get the running sum for all calendar dates (try a date scaffold, and then join to the installation date, and then get a running sum per calendar date). Now you have the total CA capacity on every calendar date, which is the max of the grey fit curve.

The last step would be to get the hourly solar radiation for a point or set of points in California. If you've made it this far, let me know and I'll dig something up.

---

The demand data is available from https://www.eia.gov/realtime_grid, click on California and then download from this URL

https://www.eia.gov/realtime_grid/#/data/graphs?end=20190401T13&start=20190325T17&regions=g

If you can't get all the data in one URL request, a few-liner script can iterate over dates.

Answer 2

I'd read the Wikipedia article, but seeing the chart blown up in philshem's answer sent me looking for the data source, which luckily is linked in the image attribution.

Renewables and emissions reports on CAISO's market operations site includes a *.txt file with hourly generation data by day for all renewable energy sources in the state, going back to 4/20/2010.

As mentioned, the hourly historical demand data can be found on CAISO's OASIS tool (under "System Demand" > "CAISO Demand Forecast" and filtering for "Market/Process" > "ACTUAL").

Using these two data sources I was able to recreate the duck curve showing actual data for 2012 through 2019:

The interesting thing to note from this chart is that while the "belly" of the duck has been getting fatter as predicted, the "head" is not getting higher -- net load in the late evenings has been trending down over the last three years.