Data on ROC embedded generator is reported through https://renewablesandchp.ofgem.gov.uk/ albeit with quite a lag to the public anyhow so theoretically possible to do a back check. Mind you how much of embedded wind is covered by ROCs im not sure as there also FiT wind generators as well of course.
I also looked at the metering requirements for ROCs and doesn't appear that automated data collection was mandated for the subsidy payment although I imagine the majority do do that for contracts with suppliers or traders. You would have also thought that for the system operator as well as the DNOs better visibility of generation on the system now was essential and rather than pissing money away on consumer SMART meters they would have dealt with embedded generators first.
Anyhow thanks for providing thought provoking posts and seasons greeting to yourself and fellow readers.
The Ofgem database is strictly monthly data and that is filed by the generators who want to receive either ROCs or REGOs. The method I described of using Elexon's Open Settlement Data - series ABV - is better because it provides data by settlement period. The snag is that the datasets are very large and require considerable processing. Further there are repeated revisions to the data so it is only final after about 4 weeks.
Recently I reviewed the EV charging regulations 2021 which include:
A relevant charge point must be configured so that it is able—
(a)on each occasion it is used, to measure or calculate every one second the electrical power it has imported or exported (as the case may be), such measurement or calculation to be in watts or kilowatts; and
(b)to provide the information referred to in sub-paragraph (a) via a communications network.
Yet we have large numbers of generators that provide no remote metering at all. True, EV meters are only required to be accurate to +/-10% which is a large tolerance. Control of solar will become a big issue if we add significantly to the present installations. To really understand what batteries are up to you need high resolution data as they often alter output/charging on very short timescales, particularly when providing ancillary frequency services. I've seen some 4 second data from AEMO which shows that even the 5 minute data can hide important detail: in any event it is important to separate out charging from discharging so that at least the round trip efficiency can be monitored.
Reviewing some of my previous downloads I see that Elexon have changed what is packaged together. Interconnector data are for the most part in separate data downloads. Downloads for individual BMUs include any imports for battery/pumped storage pumping and to meet parasitic demand of windfarms for keeping the blades turning during Dunkeflaute etc. The coverage of wind has long been a mystery, but I think it is slowly being sorted out, yet is still a mess. I have found a number of cases of fairly obvious data glitches, also affecting CCGT. If you look at this chart (which blows up to allow examination of detail)
you will see there is a significant glitch on 24th Jan 2023 with an hour where there is no report for Biomass, and some of the other figures for CCGT, nuclear and probably wind are suspect. Reviewing my analysis of 2023 which I did at hourly resolution so that charts per month are legible I see similar glitches occur roughly monthly: some may last slightly longer.
I think the AGPT/B1620 data include embedded wind at least as an estimate whereas the prompt data only includes transmission connected BMUs. This may change as NESO start to move towards their Open Balancing Platform: they have recently been forced to put in new facilities to allow for bids and dispatch for distribution connected batteries, so they are starting to acquire some visibility of life beyond the transmission network. At least the per BMU B1610 data includes any use of electricity for charging batteries, pumping hydro and keeping wind turbines turning during Dunkelflaute periods as negative generation. The treatment of parasitic loads probably differs among the sources: those that rely on accredited metering will of course tend to reflect data adjusted for revised loss estimates etc. as each update run occurs.
I was shortly going to be attempting to assemble a detailed picture for 2024 (waiting for the data that only comes through in arrears like CFD payments). It may be useful to tackle the insights team as a wider project for those of us who are interested in research based on longer runs of historical data to try to get them interested in repairing the data glitches and providing convenient download options. It is enormously frustrating that unless there is a "stream" version of data the time range of downloads can be very short (a severe handicap of BMRS it has to be said), and yet the stream version only allows JSON output which is unnecessary verbose and involves extra conversion steps via csv to make it usable in most forms of analysis. They also need to pay attention to precise definitions of what is and isn't included in the data and trying to make some of it more user friendly: we could do with data on BMUs that gives more readable names, geographic coordinates, capacity, and probably any linkage to CFD and ROC payments.
I did try to suggest some ideas when they were switching to the new system, and got told it may be a long time before anything happens...
Thank you Gordon. Oh dear. I quite frequently take a peek at the instantaneous grid demand on https://gridwatch.co.uk/demand Is that suspect too? (I note it doesn't separate off- and onshore wind.) You report a curious situation since the grid operator has to balance load and demand continuously of course, so it appears something is amiss with data collection and reporting rather than (so far) grid management.
One wouldn't want Miliband and co to pilot one's plane.
In the past Gridwatch used the FUELHH dataset and I think that it still does but with the addition of data on embedded generation for solar. If I am correct about that, then its data on total generation is roughly correct though as the first figure shows the numbers are 100 to 500 MW too high.
Actually NESO doesn't much care how the electricity is generated - just whether there is too little, enough or too much - but it does worry about flows on congested transmission lines. But, as I said, the blind leading the blind.
Pretty sure it does but the likes of https://grid.iamkate.com/ also add in the embedded generation forecasts for each sp which NESO publish everyday. However that is only a forecast and not sure its even updated during the day unlike the BM connected forecast which gets six updates a day. That in itself is also interesting to see how the forecast evolves between each update and then what the final generation is thats declared at FPN. As an aside that evolution isn't unsurprising as we are dealing with the weather!! The outturn reported via Elexon takes account of units constrained off and there has been a lot of that going on recently.
So im surmising there will of course be in day evolution of output from the embedded generators as well. Thus when NESO have boasted of new wind peaks a couple days after the event they aren't simply adding BM outturn to embedded forecast but have actually been able to source the actual output of the embedded windmills to come up with an accurate number?
Reasonable question. I have tried asking questions of Elexon in the past, as they publish the data. I received nothing more than boiler plate answers. I will try again with both Elexon and NESO. If I get any response, I will post it as an addendum to the article.
Data on ROC embedded generator is reported through https://renewablesandchp.ofgem.gov.uk/ albeit with quite a lag to the public anyhow so theoretically possible to do a back check. Mind you how much of embedded wind is covered by ROCs im not sure as there also FiT wind generators as well of course.
I also looked at the metering requirements for ROCs and doesn't appear that automated data collection was mandated for the subsidy payment although I imagine the majority do do that for contracts with suppliers or traders. You would have also thought that for the system operator as well as the DNOs better visibility of generation on the system now was essential and rather than pissing money away on consumer SMART meters they would have dealt with embedded generators first.
Anyhow thanks for providing thought provoking posts and seasons greeting to yourself and fellow readers.
The Ofgem database is strictly monthly data and that is filed by the generators who want to receive either ROCs or REGOs. The method I described of using Elexon's Open Settlement Data - series ABV - is better because it provides data by settlement period. The snag is that the datasets are very large and require considerable processing. Further there are repeated revisions to the data so it is only final after about 4 weeks.
Recently I reviewed the EV charging regulations 2021 which include:
A relevant charge point must be configured so that it is able—
(a)on each occasion it is used, to measure or calculate every one second the electrical power it has imported or exported (as the case may be), such measurement or calculation to be in watts or kilowatts; and
(b)to provide the information referred to in sub-paragraph (a) via a communications network.
Yet we have large numbers of generators that provide no remote metering at all. True, EV meters are only required to be accurate to +/-10% which is a large tolerance. Control of solar will become a big issue if we add significantly to the present installations. To really understand what batteries are up to you need high resolution data as they often alter output/charging on very short timescales, particularly when providing ancillary frequency services. I've seen some 4 second data from AEMO which shows that even the 5 minute data can hide important detail: in any event it is important to separate out charging from discharging so that at least the round trip efficiency can be monitored.
Reviewing some of my previous downloads I see that Elexon have changed what is packaged together. Interconnector data are for the most part in separate data downloads. Downloads for individual BMUs include any imports for battery/pumped storage pumping and to meet parasitic demand of windfarms for keeping the blades turning during Dunkeflaute etc. The coverage of wind has long been a mystery, but I think it is slowly being sorted out, yet is still a mess. I have found a number of cases of fairly obvious data glitches, also affecting CCGT. If you look at this chart (which blows up to allow examination of detail)
https://i0.wp.com/wattsupwiththat.com/wp-content/uploads/2024/08/Generation-jan-2023-1722888879.3157.png
you will see there is a significant glitch on 24th Jan 2023 with an hour where there is no report for Biomass, and some of the other figures for CCGT, nuclear and probably wind are suspect. Reviewing my analysis of 2023 which I did at hourly resolution so that charts per month are legible I see similar glitches occur roughly monthly: some may last slightly longer.
I think the AGPT/B1620 data include embedded wind at least as an estimate whereas the prompt data only includes transmission connected BMUs. This may change as NESO start to move towards their Open Balancing Platform: they have recently been forced to put in new facilities to allow for bids and dispatch for distribution connected batteries, so they are starting to acquire some visibility of life beyond the transmission network. At least the per BMU B1610 data includes any use of electricity for charging batteries, pumping hydro and keeping wind turbines turning during Dunkelflaute periods as negative generation. The treatment of parasitic loads probably differs among the sources: those that rely on accredited metering will of course tend to reflect data adjusted for revised loss estimates etc. as each update run occurs.
I was shortly going to be attempting to assemble a detailed picture for 2024 (waiting for the data that only comes through in arrears like CFD payments). It may be useful to tackle the insights team as a wider project for those of us who are interested in research based on longer runs of historical data to try to get them interested in repairing the data glitches and providing convenient download options. It is enormously frustrating that unless there is a "stream" version of data the time range of downloads can be very short (a severe handicap of BMRS it has to be said), and yet the stream version only allows JSON output which is unnecessary verbose and involves extra conversion steps via csv to make it usable in most forms of analysis. They also need to pay attention to precise definitions of what is and isn't included in the data and trying to make some of it more user friendly: we could do with data on BMUs that gives more readable names, geographic coordinates, capacity, and probably any linkage to CFD and ROC payments.
I did try to suggest some ideas when they were switching to the new system, and got told it may be a long time before anything happens...
Thank you Gordon. Oh dear. I quite frequently take a peek at the instantaneous grid demand on https://gridwatch.co.uk/demand Is that suspect too? (I note it doesn't separate off- and onshore wind.) You report a curious situation since the grid operator has to balance load and demand continuously of course, so it appears something is amiss with data collection and reporting rather than (so far) grid management.
One wouldn't want Miliband and co to pilot one's plane.
In the past Gridwatch used the FUELHH dataset and I think that it still does but with the addition of data on embedded generation for solar. If I am correct about that, then its data on total generation is roughly correct though as the first figure shows the numbers are 100 to 500 MW too high.
Actually NESO doesn't much care how the electricity is generated - just whether there is too little, enough or too much - but it does worry about flows on congested transmission lines. But, as I said, the blind leading the blind.
Pretty sure it does but the likes of https://grid.iamkate.com/ also add in the embedded generation forecasts for each sp which NESO publish everyday. However that is only a forecast and not sure its even updated during the day unlike the BM connected forecast which gets six updates a day. That in itself is also interesting to see how the forecast evolves between each update and then what the final generation is thats declared at FPN. As an aside that evolution isn't unsurprising as we are dealing with the weather!! The outturn reported via Elexon takes account of units constrained off and there has been a lot of that going on recently.
So im surmising there will of course be in day evolution of output from the embedded generators as well. Thus when NESO have boasted of new wind peaks a couple days after the event they aren't simply adding BM outturn to embedded forecast but have actually been able to source the actual output of the embedded windmills to come up with an accurate number?
Have you asked NESO if they can explain the discrepancies?
Reasonable question. I have tried asking questions of Elexon in the past, as they publish the data. I received nothing more than boiler plate answers. I will try again with both Elexon and NESO. If I get any response, I will post it as an addendum to the article.
Boiler plate replies are probably the single most frustrating thing going!