Diamond Dong, JiaFan Zhang ABB Robotics and Discrete Automation Research and Development Shanghai, China, diamond-daimeng.dong@cn.abb.com, jiafan.zhang@cn.abb.com; Xiaolong Feng ABB Robotics and Discrete Automation Research and Development Västerås, Sweden, xiaolong.feng@se.abb.com; Qi Lu, York Qian ABB Robotics and Discrete Automation Shanghai, China, qi.lu@cn.abb.com, york-yingjie.qian@cn.abb.com
Sales of smartphones, and other high-tech devices are growing, paralleling recent advances in electronics. As such, devices are becoming smaller and more powerful. Consequently, new automation solutions are required to quickly and accurately pick and place components in electronic production lines [1]. But this is more easily said than done. There is a well-known process trade-off between accuracy and speed if high-level performance is to be achieved.
ABB’s articulated 6-axis robots, for example, are designed for such automation solutions. Flexible and able to handle complex tasks in tight spaces, such robots can perform a variety of meticulous placement tasks. ABB experts wondered if it was possible to push alignment accuracy and speed beyond the current limits in such cases. The result is ABB’s High Speed Alignment1) software with visual servoing technology – designed and developed to explore new limits for alignment accuracy and speed, while also achieving reduced cycle times →01.
High Speed Alignment: past, present and future
Traditionally, robots were pre-programmed – ready to execute orders on command and work in determinate environments. A high degree of technical competence was required for both calibration and commissioning. Using an iterative look-then-move approach for robotic alignment, the robot had to first stop before the vision system could take snapshots, process the data and then determine and execute the necessary action. The robot then repeated this course of action, over and over again as needed →02a.
02a The traditional alignment method is shown. Here, processing of data occurs before the robot moves; and before checks are made about whether the robot has been aligned in the correct position. If not, another snapshot is taken and the robot is rerun, once proper alignment is completed, the robot jumps to the next task. 
02b ABB’s new visual servoing method creates a continuous loop of data processing and servoing, which includes checks for alignment performance, before the robot proceeds with future tasks.
02 Schematics that illustrate the traditional alignment method and the visual servoing method.
In recent years, shifts in paradigms have led to a new approach to robot alignment. Nowadays, systems are generally smarter, able to sense uncertainties and the dynamics of the environments in which the robots operate; this is due to the aid of various sensors, significantly, vision sensors. As a result, a more adaptive approach to alignment, one that is much less deterministic, is possible. Sensor feedback is now tied to the operation of the robot →02b. This means that while sensor input is collected, data is processed simultaneously; feedback is directed at the same time to the robot system, thereby allowing commands to be executed almost instantaneously. By combining this real-time sensor data capability with built-in smart algorithms, simultaneous prediction is now possible. Every forthcoming robotic movement can be predicted; instructions can be provided while, or even before, the next feedback loop is being processed. Use of this approach results in greatly shortened repetitive cycle times for: collecting input data, processing the data and executing tasks, as well as increasing the robustness of the robotic system. Thus, users can reach the desired high accuracy and speed they need – high speed alignment during operation is now possible.
Thanks to this paradigm shift, it is estimated that by 2025 more than 30 percent of new robots will be equipped with vision sensors. These sensors will allow robots not only to accomplish more advanced tasks, the new visual capabilities will boost robot performance. By combining the fields of robotics with vision and artificial intelligence (AI) the way in which robots work and interact with their environment will change rapidly.
Technology and business-driven product development
Previous research on high-speed alignment focused primarily on improving the controllers’ range of stability and robustness to system errors. However, in industrial applications →03, besides system accuracy and stability, cost and efficiency must be considered whenever new products are developed. For these reasons, ABB relied on real customer cases from the electronics industry and immediately began their investigation of the production line challenges that customers face on a day-to-day basis; the aim has been to develop a system that satisfies industry’s needs.
Giving demonstrations at robotics fairs, and performing in-house demonstrations based on real-world customer cases has allowed ABB to enhance the technology development phase. Observed holistically, an interactive process between ongoing research and business-case studies has formed a sound foundation for development. This ultimately ended with a high-level classified technical readiness prototype. Thus, the technical effort needed for the research and development function during the productization phase could be greatly reduced. And, as an added bonus, customer pilot tests, which ran successful validation studies in parallel to ABB’s development project, helped ABB to secure orders for the software even before the official launch of the product.
Speeding up productivity while reaching high accuracy
Developed for ultra-high precision assembly tasks and successfully tested using customer setups, ABB’s High Speed Alignment not only achieves increased system reliability, but this unmatched technology also reduces cycle times by 70 percent, from ~5.6 s to ~1.7 s, while increasing accuracy by 50 percent, from 20 μ to 10 μ.
How does High Speed Alignment achieve such phenomenal results? A comparison of the traditional look-then-move approach with ABB’s new technology provides answers →05-06: In the traditional approach, cycle times are dependent on the number of loops that are required for the sensor system to recognize the designated objects, with a single loop lasting approximately 2 seconds →05. Here, there is a certain amount of luck involved in the determination of just how many cycles will be required. In contrast, with High Speed Alignment, this recognition process occurs continuously, whether the previous robotic motion has finished or not. And, by fusing the data acquired by the sensor together with the feedback data of the robot at the same time, the target motion of the robot can be updated – in as little as 4 ms. Significantly, this means that a much larger spread of values is created, and the system does not need to wait on the completition of full cycles →06.
System complexity
It is noteworthy that tests for both the look-then-move and High Speed Alignment approaches were conducted in a simple system set-up with one or two vision sensors. Would the cycle times be different if the system complexity is increased? To test this notion, two additional sensors were added to the set-up, thereby increasing complexity. In the case of the look-then-move approach, any misalignment requires the system to rerun the entire alignment cycle, thereby greatly increasing cycle times. Further, this would increase the possibility of system failure before proper alignment is achieved – a clear disadvantage. In contrast, adding increased complexity to visual servoing technology did not impact alignment cycles since all areas are rectified simultaneously. In other words, the cycling times remain unchanged. Thus, ABB’s High Speed Alignment system with visual servoing technology – using a simple vision sensor system – enables precision of robotic movements to achieve 10 μ levels while greatly improving speed.
Ease-of-use is key
ABB designed the ease-of-use software and interface with the knowledge that by sacrificing either accuracy or speed, it is possible to push the other entity to an even greater level. By simply tuning either accuracy or speed, the converging time of the robot alignment, or the cycle time can be modified, respectively →04. As a result of the easy-to-use software, in combination with the intuitive user interface, only minimal expertise is required to operate the system.
High Speed Alignment has also been designed for effortless commissioning as well as ease of system alignment so that inexperienced users can operate the system. By using vision-guided motion, a combination of machine vision and precision motion control, which is the core technology behind High Speed Alignment, the flexibility of the alignment process is improved. Previously, to implement a high-performance controller, users required tremendous effort (expertise) to tune the control parameters for the servoing, generating greater costs. Thanks to High Speed Alignment this is no longer the case.
For these aforementioned reasons, ABB included an auto-tuning feature. In contrast to traditional systems, in which users had to consider the time delay of sensors, filter buffer length, control gain, delay of robot and dead zones etc., ABB’s solution allows all of these parameters to be calculated automatically. Users need only drag the tuning sliders and observe the alignment performance. This removes some of the skill barriers to interacting with the system to achieve desired alignment results. By including an auto-tuning feature, ABB effectively reduces commissioning time, reducing deployment time from an entire shift to just an hour.
Additionally, ABB’s system allows users to select previously recorded servoing data for comparison whenever they tune new servoing parameters. Such functionality supports users, experts and novices alike, in all their efforts and reduces costs.
Expanding compatibility
Currently, High Speed Alignment is compatible with a wide range of cameras, IRC5 and OmniCore™ robot controllers for high flexibility. Although High Speed Alignment is independent of hardware and the image processing tool used, the type of communication protocol used is important. The solution supports any sensor if the data is transmitted in the correct format via a User Datagram Protocol (UDP) channel to the client →07. Nonetheless, there are plans to support Transmission Control Protocol (TCP) communication in the near future.
The entire framework consists of a sensor (a camera in this case), a robot controller, an image processor and visual servoing technology →07. The latter technology and the image processor are included within the industrial PC. The camera is attached to the end effector of a robot to capture visual information in the task space. The resultant image stream is received by the image process, which computes the position of the target object relative to the camera. The robot controller, namely an IRC5 (or OmniCore™), controls the robot and provides an External Guided Motion (EGM) interface to communicate with visual servoing through the UDP datagram. Visual servoing then acquires data from both the EGM and the image processor and generates motion commands that are sent to the IRC5 through UDP. It is the feedback of the robot’s information from the EGM interface that allows the performance of the visual servoing controller to be maintained at an optimal level [2].
New opportunities
ABB’s High Speed Alignment software was originally developed for ultra-high precision tasks in the computer, communications, and consumer electronics industry, such as assembly in electronics manufacturing, alignment of components, picking or placing a work piece in a tool, and placing a part in a fixture. Other opportunities for potential applications include:
• For bottle-picking, eg, in breweries that rely on Causal State Splitting Reconstruction algorithms
• For gluing, sealing and tire assembly, and friction stir welding in the auto industry
• For seat assembly in Tier 1 industries; suppliers of parts or services to an original equipment manufacturer (OEM).
With High Speed Alignment, ABB has released the first software package in the market to provide visual servoing technology for 4- and 6-axis robots for ultra-high precision assembly tasks →08. Using one or more visual sensors and a computer vision system to control the position of the robot’s device or tool relative to the workpiece in real-time, this technology satisfies industry’s demand for higher alignment accuracy and speed →08-09. High Speed Alignment enables robotic movements to achieve 10 μ levels for precise electronics assembly while meeting industry’s cost and efficiency demands.
Footnote
1) Visual servoing for ultra-accurate alignment (also known as High Speed Alignment, High Accuracy Assembly, in text referred to as High Speed Alignment)
References
[1] ABB press release, “ABB launches breakthrough robot alignment software increasing electronics manufacturing speed and accuracy”, 12 April 2022, Available: https://new.abb.com/news/detail/89415/prsrl-abb-launches-breakthrough-robot-alignment-software-increasing-electronics-manufacturing-speed-and-accuracy [Accessed September 17, 2022.]
[2] Y. Mao et al., “Visual-Servoing Control Based on EGM Interface of an ABB Robot”, IEEE Publication, Chinese Automation Conference, 2018, Available: https://ieeexplore.ieee.org/document/8623638 [Accessed March 14, 2023.]
