According to the International Energy Agency (IEA), electricity has grown at twice the rate of overall energy demand over the past decade. This pace of electricity growth could accelerate to as much as six times the rate of overall energy demand in the next decade, driven by rising electricity demand from data centres, air conditioning, and the electrification of transport, industry and heating.    

Although there are examples of new large data centres preferring firm generation, current IEA analysis shows that electricity generated from solar and wind is expected to account for much of the supply growth. Even in markets without clean power targets, the favourable economics and short development cycles of wind and solar have led to significant capacity expansion. For example, Texas has the largest installed wind and solar capacity installed in the US (in 2024, approximately 164 TWh of electricity was generated from wind and solar in ERCOT, compared to 62 TWh in California).      

Many electricity markets are already capable of managing instantaneous wind and solar penetrations of over 80%, and even up to 100%. This reflects significant progress by electricity system operators, where operators have adapted to manage changes in system inertia and voltage associated with more distributed and variable generation.      

However, when the share of electricity from wind and solar grows, the cost of ensuring resource adequacy increases. Forecasting variable wind and solar production and estimating backup capacity is increasingly challenging due to new loads and consumption profiles, as well as more frequent and extreme climate events.     

System outages force electricity operators to disconnect industrial, commercial and residential consumers who rely on firm access. The recent 10-hour outage in Spain on 28 April 2025 resulted in multiple casualties, and Reuters reported that Spain’s main business lobby CEOE (Confederación Española de Organizaciones Empresariales) estimated that the outage would reduce gross domestic product by €1.6bn ($1.82bn).  

In August 2024, the Texas Public Utilities Commission approved a value of load loss of $35,000/MWh for ERCOT.   

In the UK’s Clean Power Action Plan, 35 GW of unabated natural gas capacity (roughly the current installed base) is retained to support energy security with the expectation that these plants will account for 5% or less of electricity generation in 2030 (compared to about 26% in 2024).   

What are the challenges faced by electricity system operators? 

Given the scale and potential of problems in grid operations worldwide during the integration of renewable energy systems in the energy transition, a Rice University MBA team compared two leading system operators, ERCOT in the US and the UK’s NESO. From a market structure and policy context, these two electricity systems could be viewed as representative bookends of grids around the world. ERCOT has a nodal, energy-only market while NESO operates national energy and capacity markets and supports the UK’s Clean Power 2030 Action Plan.  

Both electricity system operators prioritise reliability and resiliency and aim to keep electricity costs as low as possible within their respective policy contexts. They also face common challenges seen globally: increasing wind and solar penetration, growing co-location and behind-the-meter capacity, the emergence of new types of demand loads and consumption profiles, and the increasing sensitivity to extreme weather events as people are more dependent on electricity.    

Both electricity system operators prioritise reliability and resiliency and aim to keep electricity costs as low as possible within their respective policy contexts.

What action to take? 

Every market is different, but there are five ‘no regret’ actions that grids around the world can take, irrespective of policy contexts, market structures and installed capacities.   

Improve forecasting and planning of intermittent generation and future loads. Both independent system operators (ISOs) agree that forecasting accuracy for wind and solar is mission-critical for managing the future grid. As predictions deviate from reality, errors in grid projections can significantly impact the level of grid services needed and have a direct impact on affordability. Similarly, forecasting load, through an improved information exchange with large consumers and transmission operators, is also critical due to new types of loads and changes in consumption patterns (such as, commuting, remote work, industrial decarbonisation, electrification of transport), potential under/over reporting by transmission providers and flexibility in behind-the-meter self-consumption.  

Expand demand-side flexibility programmes. Incentivising consumption to vary with the generation profile and smoothing peaks is an important lever in managing resource adequacy and capacity costs. Although still in the early stages, the UK’s demand flexibility service has (DFS) demonstrated how to unlock large-scale load-side response through third-party aggregators, flexible pricing and dynamic contracts.  

Treat behind-the-meter (BTM) generation and new large loads as a flexibility lever. With new large BTM loads being developed to avoid grid charges and/or connection bottlenecks, the primary concern is that large uncoordinated BTM systems can introduce sudden load swings at the transmission level, potentially causing voltage instability or frequency events. Related to demand response, the emergence of new loads and load profiles could also present an opportunity.   

For example, ERCOT is exploring the classification of large BTM loads as ‘non-firm load’, meaning they could be curtailed in times of system scarcity. With the recent passing of SB6 in Texas, ERCOT may issue reasonable notice that large load customers with on-site backup generating facilities may be directed to either curtail load or deploy the customer’s on-site backup generation.  

This policy could help protect the grid from unexpected strain by signalling that some BTM-connected assets, particularly highly interruptible or non-critical loads, should not expect guaranteed service during high-demand events. Although not BTM, an example of how this could work has been demonstrated with bitcoin miners in ERCOT, lowering power grid draw to assist ERCOT in managing grid resources during heatwaves in the summer of 2024.    

Invest in digital, AI and transparent data. Both operators stressed the importance of data transparency to enable market participants to make informed decisions that help maintain system balance. They are implementing AI and machine learning across forecasting and balancing services to reduce human error and accelerate decision-making with tools like anomaly detection, predictive analytics and embedding intelligence at every level of control. NESO also developing a shared data platform that will provide near real-time data to market participants. Fig 1 (headline image) is an example of one of NESO’s forecasting tools.  

Recognise continued importance of dispatchable generation and optimise its use. Today, existing dispatchable generation – primarily natural gas-fired and hydropower – plays a key role in integrating wind and solar by following load and mitigating resource adequacy issues. As one NESO interviewee stated: ‘We’re operating a system that’s now much less reliant on fossil generation, but we still need the services that came with them. 

Complex challenges 

Some of the most complex challenges in electricity system operations are tied to policy contexts and market structures. These include the role of battery storage, approaches to system planning and modelling, strategies for managing security-of-supply risk, and the interface with the distribution network.    

Batteries can act as both generators and consumers, and operate both BTM to add flexibility to consumption and in front-of-the-meter to support the grid. Batteries and other forms of energy storage can provide many of the same capabilities as fossil generation, interconnection, flexible demand, and digital- and AI- based solutions, but they require new capabilities and regulatory structures.    

For example, should there be requirements and monitoring of the battery ‘state of charge’. Additionally, batteries are limited duration, a critical attribute not shared with thermal resources, and adapting energy control systems that have never dealt with limited duration (energy) is a major effort for the modern grid.   

The appropriate level of system planning and modelling-for load growth, generation mix, infrastructure development and integration, with other energy vectors such as natural gas and liquid fuels, also depends on the policy decisions. Similarly, whether to ensure supply through market-based mechanisms like scarcity pricing or through forward contracting by capacity markets is also a matter of policy design. 

Perhaps the most complex issue is the interface between the system operators and distribution network operators as more generation and storage assets are deployed at the distribution level. In some markets, distribution network operators are evolving into distribution system operators (DSO), taking on responsibilities such as managing local flexibility markets and ensuring local resiliency.  

Electricity system operators are executors of policy. Without clear regulatory and policy guidance, they may not be in a position to resolve these complex issues. However, they can make progress on the universally beneficial ‘no regret’ actions: improving wind, solar and load forecasting; expanding demand-side flexibility programmes; treating new large BTM generation and loads as a new lever; investing in digital tools, AI, and transparent data; and strategically using dispatchable generation.

• Further reading: ‘Why Great Britain is going towards a ‘whole system’ network transmission approach to energy operations’. To deliver a ‘net zero carbon’ economy, Great Britain has to remake its energy system, both physically with expanded networks and economically with new markets. A new National Energy System Operator (NESO) will have responsibility for delivering the change across both electricity and gas, reports Janet Wood FEI. 
• Find out how Texas is steering towards renewables despite many grid challenges.