Anna Kulmala ABB Distribution -Solutions Tampere, Finland anna.kulmala@fi.abb.com
Sandro Schönborn, Thanikesavan Sivanthi ABB Corporate Research Center Baden-Dättwil, Switzerland sandro.schoenborn@ch.abb.com thanikesavan.sivanthi@ch.abb.com
Henry Niveri ABB Distribution Solutions Tampere, Finland henry.x.niveri@fi.abb.co
Robert Birke Former ABB employee
Although they generally go unnoticed by most, MV substations are found everywhere there is any appreciable concentration of people or industry →01. Responsible for stepping down the voltage delivered by high-voltage (HV) transmission systems, substations are vital for the supply of electricity to industrial and residential customers.
Modern substations typically feature devices from multiple vendors. These devices run applications, on their proprietary hardware, that are essential to the functioning of the substation. However, maintaining and updating a myriad of proprietary devices is an expensive and challenging business. And substation operators face other challenges, too – for example, the rising numbers of distributed energy resources (DERs) on the distribution grid. This increase in DERs brings with it the need for further investment in intelligence and resilience across the grid. Moreover, the distribution system operators (DSOs) are under constant pressure to reduce capital expenditure (CAPEX) and operating expenses (OPEX) generally.
One effective way to address these challenges is to introduce a flexible and rapid deployment of applications in substations. The DSOs can then not only deploy applications from different vendors onto the same hardware but also add functionalities or upgrade existing functionalities on demand. The key to accomplishing flexible and rapid application deployment is virtualization.
Virtualization decouples software from hardware
The concept of decoupling software from the underlying hardware – ie, virtualization – has been very successful in the IT landscape over the past few decades. Indeed, virtualization technology is the workhorse that enables modern cloud infrastructure as it reduces hardware costs and simplifies maintenance. With virtualization, applications run in a “virtual” environment, abstracted from the actual underlying platform and isolated from the other applications running on that platform. Virtualization enables software to be deployed, executed, exchanged and migrated almost independently of the underlying platform →02. Virtualization also helps to achieve high availability and resiliency at a reasonable cost. These benefits of virtualization technology can be exploited to assist DSOs in addressing the challenges and limitations they face in substation automation while also making it easier to deploy and scale new applications.
Customizing virtualization for substation automation
Virtualization in computing environments can take place at different levels:
- In hardware, in virtual machines (VMs) that provide the functionality of a physical computer without the developer having to deal with actual physical hardware.
- In the operating system (OS), in so-called containers. In contrast to a VM, which virtualizes an entire hardware machine, a container virtualizes the software levels above the level of the OS and can isolate processes and control the access of processes to CPUs, memory, etc.
- At the function level, as in serverless computing, where there are still servers involved, but these are abstracted away from the application, so the developers do not have to worry about provisioning or managing server infrastructure as this is handled by the cloud provider.
When applied to substation automation, virtualization technology faces challenges not seen in cloud computing. While the cloud has ample and scalable computing, storage and networking resources, the resources available in substation devices are more modest. Furthermore, the applications running on the devices are of mixed criticality (safety-critical and non-safety-critical) and some applications may require specific hardware configurations – for example, support for the Precision Time Protocol in a network card. Moreover, many substation applications have special timing and high availability requirements that must be met during operation.
These needs sharply differentiate virtualization when it is applied to distribution or substation automation. Consequently, careful resource assignment and provisioning are required when consolidating several VMs or containers onto devices. Otherwise, resource contentions can result in non-deterministic delays that could compromise the real-time guarantees for the applications running in the VMs or containers.
ABB investigated the applicability of virtualization technology in MV substations, specifically the use of real-time protection and control applications in Linux containers that provide virtualization at the OS level. Multiple instances of centralized substation protection and control on the same host were run and effective real-time performance of the control and protection applications using tailored resource provisioning and virtual networking solutions was demonstrated. The discussion below shows just how critical effective resource provisioning and management are for a successful implementation of virtualization in substation automation.
Implementation and resource management
ABB researched the applicability of running demanding real-time workloads in Linux, using not only OS-level virtualization with containers but also VMs. The basis of the test systems is a Linux kernel with a PREEMPT-RT patch for real-time operation. In essence, PREEMPT-RT is a Linux kernel patch that adds real-time capabilities. Although PREEMPT-RT may not be suitable for very fast real-time loads with latency requirements in the order of a few microseconds, in reality only the more realistically achievable millisecond latencies are needed for substation automation applications.
To achieve the real-time performance required in substation automation, the protection application must:
- Receive the incoming voltage and current measurements promptly.
- Have guaranteed access to the computing hardware when needed.
- Complete calculations within a limited period and be able to send out a response action in time.
These conditions create different challenges for a virtualized setup with multiple applications sharing the same host system.
Latent latency challenges
The first obstacle is retrieving process data from the merging units (MUs) in a timely manner. MUs act as interfaces that receive process data such as measurements from instrument transformers and switchgear alarms and indications. The focus here was placed on digital substations where process and control data are exchanged over an Ethernet network, as specified in the IEC 61850 standard. In such situations, multiple applications need to share the physical network access with virtualized networking technologies, such as a virtual switch or macvlan (a network driver that presents software containers to the physical network as if they were physical devices). Therefore, various ways of attaching a virtual environment to the host physical network were studied. This investigation was performed using the virtual product known as ABB Ability™ Smart Substation Control and Protection for electrical systems SSC600 SW.
Since mainstream network virtualization approaches do not optimize for latency, substantial pitfalls for real-time operation (due to how network drivers defer work for later) were discovered. The final setup was optimized to avoid these pitfalls. Modern networking hardware can also help to improve the situation through explicit virtualization features.
Accessing the CPU as needed
With data arriving on time, the next step toward achieving real-time performance is to gain access to the CPU when needed. Scheduling process execution is a core task of any OS, and Linux with PREEMPT-RT offers multiple real-time policies and enables the protection software to preempt other applications.
It was found that scheduling performance – ie, getting the CPU exactly when requested – is not significantly impacted by running the software inside containers on modern server hardware and is not the main bottleneck on a properly configured system, at least for latencies above a few dozen microseconds. However, the protection process or a whole VM might still be blocked by another application with a similar priority or by one of the many OS housekeeping tasks. Different CPU core allocation policies were studied and potentially dangerous contentions were resolved by exclusive allocation of CPU cores to applications, including maximum isolation of the OS to dedicated CPU cores. Exclusive allocations further ensure proper sizing of host hardware to guarantee the availability of sufficient computational resources for all applications.
The policy chosen is also a good match to the high computational demand of protection algorithms and the mostly horizontal nature of scaling in modern hardware with its consistently increasing numbers of CPU cores.
Not forgetting memory considerations
Even with calculations started at the right time and dedicated CPU cores available, interference can still delay computations unacceptably. For dataintensive applications, such as protection and control, execution timings depend strongly on the transfer times of data between memory and CPU. A modern system employs multiple levels of fast intermediate memory (cache) to reduce retrieval times for frequently used data. Unfortunately, the cache size is limited and holds further potential for interference between applications with high data loads. Therefore, policies of cache partitioning and exclusive reservation of memory bandwidth were implemented.
With careful consideration of resource reservations, it becomes possible to fully isolate virtual real-time applications from each other and ensure steady operation with timely responses for all of them on the same platform.
Pilot installation and performance
The project work moved from early proof-of-concept studies to lab tests of virtual protection and control to validation of real-world deployments in the field.
In the laboratory, the team validated protection reaction times of the whole system running in different configurations and virtual environments, and with varying loads, against actual signal injection in hardware-in-the-loop tests and real-time network simulations. In all tests, the virtual setup succeeded and lived up to the timing and functional requirements of the equivalent dedicated embedded protection and control device.
Following the positive lab results, ABB moved to validation in the field in collaboration with one of Finland’s leading DSOs. One IEC 61850 certified server was deployed in a research cabinet at a substation in Western Finland province →01. Using careful resource partitioning, this server can run two fully configured virtual SSC600 instances in parallel.
Since the deployment in September 2021, the instances complied with all functional and timing requirements of the physical SSC600. Some (of the very few) events observed at the substation were captured and, by comparing event logs, it could be seen that the behavior of the virtual instances perfectly aligns with the behavior of the physical counterpart →03. →04 highlights a critical performance index: compliance with the internal cycle time. →05 shows how the execution times of the internal functions met the required deadline (red threshold). The carefully designed virtual network layer allowed the arrival of all the incoming sampled measurement value traffic in time with no late samples or other losses during normal operation. The only observed packet losses happened during external data disconnection events, unrelated to the virtual setup.
03 Screenshot of one of the rare substation events showing how the multifrequency admittance-based earth fault protection operated at the same time (to the millisecond) across all three substation instances. 
04 Execution times show good compliance with requirements. 
05 Compliance with the internal cycle time: Execution times of the internal functions met the required deadline (red line) throughout a whole year of operation.
The field test in Finland successfully validates that the approach chosen is fit for purpose for virtual protection and control in substation automation in the real world. Virtualization provides substation operators with a way to reduce costs and make equipment maintenance easier as well as successfully take on board the rapidly rising numbers of renewable power sources appearing on the grid →06.
