Harish Suryanarayana, Aniket Joshi, James Stoupis ABB Research Center, Raleigh, NC, United States
harish.suryanarayana@us.abb.com, aniket.m.joshi@ us.abb.com, james.stoupis@ us.abb.com
Parashar Parikh ABB, e-Mobility, Raleigh, NC, United States, parashar.parikh@ us.abb.com
The electric grid of the future must handle increased loads from the accelerated electrification of transportation as well as variable generation from distributed energy resources (DERs). Balancing load and generation requires a combination of location-specific grid-responsive solutions. Recognizing this key challenge, ABB proposes a free market approach to influence EV charging decisions by varying the price of EV charging in response to grid conditions. The resultant solution will lower the cost of delivering power to the end customer and help utilities maintain grid stability.
Shifting to electric vehicles
The electric grid is in the process of undergoing a major transformation. The world has largely recognized the need to shift away from fossil fuel-based energy, especially for transportation. The resulting rapid increase in EVs is projected to lead to a heightened demand for power and energy. This increase, along with the expansion of distributed energy sources connected to the grid, leads to increased variability in power generation. Two ways to handle the subsequent balancing act between power generation and load would be to add more energy storage and, or, to reinforce the grid. However, this comes with significant capital expenditure. While such infrastructure-based improvements are welcome and have already begun [1], the scale of the problem is potentially alarming. For example, in September 2022, residents of twelve California counties were requested to reduce power consumption or face rolling blackouts due to higher power demand than usual resulting from an extreme heat wave [2]. Such scenarios indicate the need to implement a broader combination of approaches.
One obvious consideration is to evaluate the future of EV charging, which has been growing at an ever higher rate due to increased customer demand. To help meet this demand, ABB launched the world’s fastest EV charger, the Terra 360, in 2021. With a maximum output of 360 kW, an electric car can be charged in less than 15 minutes, delivering 100 km of range in less than three minutes [3]. And once the capability to charge rapidly is developed further and installed, it would be more than unfortunate if the speed of charging needed to be artificially and forcibly curtailed. Such actions might be required to maintain grid stability and ensure the security of power supply.
ABB, the world leader in EV infrastructure, offering the full range of charging and electrification solutions for EVs of all kinds, is collaborating with partners in the utilities, and academia, to provide technical solutions to address these future challenges in the real-world. ABB’s Electrification Mosaic Platform for Grid-Informed Smart Charging Management (eMosaic), is one such project, that was initiated in 2020 to provide a combined view of multiple charging sites, levels, and types of EV charging for utility-informed smart charging management.
Time-of-use rates – a locked-in structural solution
A traditional free market approach to handle the load to generation mismatch, especially in areas where solar energy generation is strong, has been the use of time-of-use rates. Electric utility companies, such as PG&E in California, USA, already have EV time-of-use rates in their rate structure [4] →01 which means that the price that end consumers pay better reflects the time varying costs of generating electricity. Beneficially, time-of-use rates have a daily and seasonal aspect, in which different rates are applied to night and day as well as to summer and winter to account for various factors, eg, variability in solar generation. Moreover, there are different rates depending on whether the power is for the entire home, without separate metering for EV charging (EV2-A rates), or only for EV charging (EV-B rates). Such pricing is expected to incentivize customers to charge during specific hours of the day. Even so, EV customers are free to charge their car during peak hours, albeit at more than double the cost of the off-peak rate. Despite this seemingly logical solution, time-of-use rate structures are fixed and therefore unresponsive to changing grid conditions, such as unexpected peak use due to extreme temperature conditions, thereby reducing the effectiveness of this rate structure. While the certainty of such a pricing structure is positive, changing grid conditions and even weather could reduce its effectiveness.
Dynamic pricing
The next extension of time-of-use rates is to introduce grid-informed dynamic pricing, wherein the price of EV charging is responsive to what has happened historically and what the grid is currently experiencing. While this approach lacks the certainty of time-of-use rates, it provides the flexibility to respond to changing grid conditions through a free market approach. With the ever-increasing demand for power and energy, rapidly changing loads, and expansion of distributed energy resources, the future grid will face more variability. Dynamic pricing is another tool that could be used to help balance generation and load. According to the dynamic pricing approach, the EV user is rewarded with a lower price of charging during times when it is favorable to the grid (and in turn to the EV site owner) yet has to pay a higher price during high-demand situations. Nevertheless, there are many technical challenges to resolve before dynamic pricing can become a reality.
Setting the best price
Capturing the complex interaction between dynamic EV charging price, EV charging user behavior, electric load, and distribution grid dynamics is a formidable challenge. The goal is to provide benefits to the grid while simultaneously reducing the cost of delivering power to the EV or EV charging site owner. For a positive impact, the pricing approach must not only be flexible, it must be tailored to the location-based dynamics of the EV charging site. And, herein lies the rub: the rapidly increasing and changing EV penetration complicates the issue. Clearly, the dynamic price setting mechanism must be automated, but how? Smart grids can help solve the supply issues originating from the burgeoning number of rooftop solar and EV battery power sources operating in real-time by balancing demand from customer devices (air conditioners, water heaters, batteries, EVs). Dynamic pricing in real time would remove the pressure from the load, but how is this possible? The solution lies in the use of artificial intelligence (AI), specifically the field of reinforcement learning.
RL for dynamic pricing
The RL technique is based on the ability to learn the optimal behavior in a certain environment for maximum reward. Heavy research ensued after groundbreaking results were achieved in 2016 when AlphaGo, an RL-based computer program, beat the world Go 1) champion, Lee Sedol. Since then, there has been deepening research into RL and its use in industrial applications eg, for data center cooling, robotics, etc., with good success [5]. Recently at ABB, RL has been used to capture complicated interactions between EVs and the grid to dynamically set the charging price of electricity, as a follow-up step to the time-of-use electricity pricing in current use [6].
In this case, the RL agent can be thought of as a controller that takes the grid status as input and generates a charging price or charging price factor as output. This output is communicated to the EV charging station, which in turn transmits this information to the EV end-user.
In framing and developing an RL-based solution for any problem, the RL components must be defined →03. For the distribution grid, ABB used an IEEE 34-bus distribution system model with multiple EV charging stations at different nodes.
The next step involved training the RL agent to perform this price setting function. During the training process, the RL agent learns by interacting with the environment and trying different actions, which in this case are different price points, and determining how it affects the reward (a combination of grid health constraints and cost of total power delivery).
Ideally, to learn about its environment, the RL agent should train in the real world where it receives reality-based feedback for its actions. The drawback is the well-known exploration-exploitation trade-off. To learn, the RL agent needs to be able to explore the impact of different actions; during this time, its performance may be severely suboptimal. Moreover, the amount of time required to learn could be unreasonable for many real-world applications. To circumvent these drawbacks, a synthetic environment, which is similar to, yet distinct from a simulation twin, is created to represent the real world. In this way, the RL agent explores the effects of its actions in simulation before it is deployed and fine-tuned, using feedback, in the real world.
Synthetic environment for training
Allowing the RL agent to explore a virtual synthetic environment prior to deployment, makes it possible to use RL to solve complex problems. Ultimately, the closer the simulated environment is to the real world, the better the agent will perform once it is deployed. To this end, ABB developed and employed various tools and routines to simulate different aspects of EV charging, electrical grid dynamics, and load environment as well as end user behavior. The key tools and routines employed are:
• Caldera, an infrastructure simulation platform, developed by Idaho National Laboratory, which simulates the EV charging sessions and EV site electrical dynamics [7].
• OpenDSS, from the Electric Power Research Institute, which is used to simulate the IEEE 34 bus distribution system.
• Other routines developed in-house in Python, eg, EV site load forecaster, which predicts the day-ahead EV charging load for a charging site using multiple time horizons to capture the usage pattern and EV penetration dynamics using only past EV metering data [6]; EV user session selector, which models user response to price signals; and charging session generator, which stochastically generates charging sessions based on a day-ahead prediction of EV behavior.
In ABB’s case, the synthetic environment interacts with the RL agent →04, wherein it obtains the grid health metrics from the synthetic environment every hour and modifies the changing price accordingly.
Communication architecture
Dynamic pricing requires the underlying communication infrastructure to assure a flawless exchange of information. ABB determined that this infrastructure needed to be updated. This was accomplished as part of ABB’s eMosaic project. Here, ABB developed and established secure communications between the EV site, the eMosaic cloud and different users →05.
Training and testing in the synthetic environment
The RL agent was trained in the developed synthetic environment for over 900 episodes (each episode is equivalent to 24 hours of charging). The training process takes about 5 days on a medium duty desktop computer server. To complete this training process in the real world, it would take about two and a half years. To generate the necessary performance metrics, the stochastic simulation was run to collect 30 days’ worth of data: Voltages and currents were captured from distribution grid simulations →06. Here, it is to be noted that the grid has been deliberately loaded to reproduce the anticipated stress on the grid caused by EV charging. To compare performance, a constant pricing use case (baseline) with a similar price for energy delivery as the average dynamic price was simulated. The dynamic pricing use case demonstrated a nearly 50 percent reduction in the time spent in the restricted voltage region (less than 0.9 per unit). These results are extremely promising.
Future steps
Having defined the required algorithms, models and completed simulations to train and test this dynamic price model, ABB’s next step is to implement this pricing strategy in a demonstration site with project partners. This will include a utility company and a university in the United States. Thus, the impact of dynamic pricing with real EV users can be rigorously tested, so that dynamic pricing will be ready to serve EV charging end customers and the utilities. After all it is only through balancing the needs of energy producers and consumers alike that the electrical grid can maintain a secure supply of energy as the electrification of transportation expands.
Acknowledgement
This material is based upon work supported by the U.S. Department of Energy’s Office of Energy Efficiency and Renewable Energy (EERE) under the Advanced Vehicle Technologies Research Program Award Number DE-EE0009194. The views expressed herein do not necessarily represent the views of the U.S. Department of Energy or the United States Government.
Footnote
1) Go is a board game, invented in China, with over 2.1×10170 legal positions on the board.
References
[1] P. Perani, “In grid we trust”, ABB Review 03/2023, pp. 172 – 179.
[2] D. Smith, “California sent a scary text message urging residents to cut their power use and it worked”, The Sacramento BEE, Sept, 2022 [Online] Available: https://www.sacbee.com/news/local/article265446471.html [Accessed May 17, 2023.]
[3] ABB Website, “ABB launches the world’s fastest electric car charger”, Available: https://global.abb/group/en/calendar/2021/terra-360 [Accessed May 17, 2023.]
[4] PG&E Website, “Electric Vehicle (EV) Rate Plans”, Available: https://www.pge.com/en_US/residential/rate-plans/rate-plan-options/electric-vehicle-base-plan/electric-vehicle-base-plan.page [Accessed May 17, 2023.]
[5] J. Luo, “Controlling Commercial Cooling Systems Using Reinforcement Learning” in arXiv, November 2022, [Online] Available: https://arxiv.org/abs/2211.07357 [Accessed May 17, 2023.]
[6] H. Suryanarayana and A. Brissette, “EV charging site day-ahead load prediction in a synthetic environment for RL based grid-informed charging”, 2023 IEEE Transportation Electrification Conference & Expo ITEC, June, 2023, pp. 1 – 5,
[7] Idaho National Laboratory Website, “Caldera: EV infra-structure simulation platform”, Available: https://cet.inl.gov/caldera/SitePages/Caldera.aspx [Accessed May 17, 2023.]
