Anna Mazzoleni, Callisto Gatti, ABB S.p.A. Dalmine, Italy, anna.mazzoleni@
it.abb.com, callisto.gatti@it.abb.com
Tomas Kozel, ABB s.r.o.Brno, Czech Republic, tomas.kozel@
cz.abb.com
In addition to leveraging cognitive services such as speech-to-text, text-to-speech, and natural language understanding to find needed content, the app uses augmented reality (AR) to superimpose visual and audio information on existing systems, thus creating an interactive virtual assistant.
When it comes to technology, what was new yesterday can be history tomorrow. This is particularly true for energy distribution networks, which are becoming increasingly complex as renewable sources become a growing slice of the pie. This complexity poses a challenge for service technicians and engineers in the field who typically must diagnose and solve problems associated with a variety of elements in a customer’s infrastructure, ranging from motors and valves to switchgear and transformers.
And that is just the beginning of the challenge, because sometimes troubleshooting is not just about the equipment itself. Hidden behind physical systems are hurdles in the form of product names and local slang expressions that can significantly slow a technician’s ability to diagnose and repair a problem. How, for instance, can a novice field operator make sense of a request to repair “the VD4 breaker in outgoing to T11 in TS4 UniGear ZS1 panel”? And things become even more complicated when components from different manufacturers are involved.
Furthermore, according to ABB’s Energy Insights Survey [1] of 2,300 leaders from small and large businesses, 92 percent of respondents believe the continuing instability in energy prices is having a significant impact on the workforce in terms of decreased investment in employees – and therefore in training. So, as older workers retire, and training budgets are squeezed, the need for support grows.
A dedicated assistant for each operator
In view of these trends, in 2023 ABB introduced its Service Assist mobile application →01-02. Specifically designed for field operators who would benefit from an assistant in their day-to-day jobs, the app provides access to documents for electrical equipment, including augmented reality immersive guides. All content is tailored to the customer’s installed base and presented with the customer nomenclature for the equipment.Service Assist can also be used for submitting training requests and booking appointments for either on-site or remote services.
One of the app’s special features is ABB-e, a virtual assistant that helps users find guides and book appointments through voice control. This virtual assistant leverages cognitive services such as speech-to-text, text-to-speech, and intent recognition – to guide the user quickly and efficiently to the app’s most useful content. It is also particularly versed in electrification network slang.
Technicians can request manuals, videos, and perhaps most important, augmented reality (AR) guides, which make it possible to troubleshoot problems even if the technician’s experience level is not advanced. Furthermore, the system is not limited to ABB documentation or equipment; it can host other brands’ documentation, as well as customer-specific information such as, for instance, a safety checklist to access a certain part of an electrical infrastructure
Situation-specific data and images
Thanks to the app’s AR technology, operational information is presented in a completely new way, such that the user’s view of a real environment is augmented by situation-specific data and images that function as a digital assistant →03a, 03b. The result is fast support and easy-to-access operation and troubleshooting guides via an immersive augmented reality experience.
03a Situation-specific data and images function as a virtual digital assistant 
03b Easy access to guides results in accelerated problem resolution.
03 Service Assist helps a technician troubleshoot an ABB medium voltage switchgear system.
Crucially, this AR modality makes digital assistance interactive, more practical to absorb, as well as easier to understand and act upon. In other words, its core capabilities are visualization instruction and interaction.
In case the user is unclear about the information offered by the system’s AR guidance, he or she can request remote assistance for electrical systems (RAISE) →04, which provides real-time guided remote support. Here, an ABB service specialist can place augmented reality instructions in the operator’s field of view, allowing visual information to enhance voice instructions covering subjects such as repairs and troubleshooting, maintenance support, technical information on equipment operation, spare parts identification, monitoring and diagnostic analysis, or installation and commissioning. This reduces repair and maintenance time on electrical equipment, as ABB experts can remotely guide field operators on their Android or iOS smartphone, tablet, or via smart glasses or other supported wearables.
Accelerated problem resolution
All of this adds up to accelerated problem resolution, which is vital in minimizing potentially disruptive and costly downtime, especially when critical systems such as data centers or hospital power systems are involved. For instance, one of the world’s largest marine shipping operators needed remote maintenance to support problem solving for its global fleet and reduce the impact of issues while at sea. Here, service support delivered through AR significantly extended the ability of onboard technicians to address failures they would have otherwise lacked the experience to diagnose and repair.
Furthermore, ABB’s Service Assist remote support can significantly reduce service-related energy demand. According to company figures, the technology has cut ABB’s CO₂ emissions by approximately 332 tons per year by reducing customer site visits by company service engineers by up to a third.
In addition to using live on-screen annotations and digital overlays in the technician’s field of vision, remote assistance also makes it possible to take pictures and share audio and video in the context of a live text chat.
A new AR experience
Looking ahead, the introduction of large language models (LLM) such as ChatGPT has given impetus to the world of chatbots and assistants, and ABB’s Service Assist is no exception. ABB is preparing a proof of concept →05 to include an LLM as an engine designed to prepare comprehensive answers to written or spoken questions rather than pre-designed responses. This new area of development may eventually be enhanced through the use of Apple’s Vision Pro and Meta’s Quest 3 headsets – systems that hold the potential of providing a new dimension to the handsfree AR experience.
Reference
[1] ABB. ABB Energy Insights Survey Report 2023. Available: https://campaign.abb.com/energy-insights-report-2023 Accessed October 2, 2023
