Translating robotic ­vision into increasingly precise depalletization

Subscribe to ABB Review

Toni Roda Robotics and Discrete Automation Barcelona, Spain toni.roda@es.abb.com

The worldwide supply chain is being reshaped by Industry 4.0 technologies. AI, the Internet of Things, advanced sensing, and big data analytics are changing the way products are designed, tested, manufactured, and distributed. As this trend unfolds, logistics operations are responding not only to these forces, but also to the growth of e-commerce and the constraints imposed by a tight labor market. The result is that distribution and fulfillment centers must find new ways to make their infrastructure and people more productive. Clearly, automation of processes is a vital part of the solution.

One of the most common applications in distribution and fulfillment centers is depalletizing, which consists of picking boxes (technically referred to as “stock keeping units” or SKUs) from a pallet and placing them in an output system such as a conveyor. Essential for major industrial sectors such as the food and beverage industry, pharmaceuticals, consumer durables, and chemicals →01, this often strenuous work is now being targeted by automation companies to improve productivity in logistics. It’s also being introduced with a view to reducing physical fatigue and the incidence of repetitive strain injuries, which can result in reduced efficiency, medical leave, and increased insurance costs.

center

But although automated systems are already available that can perform limited depalletization, they suffer from high error rates and thus still require the support of manual labor. Typically, today’s distribution centers and warehouses handle a mixture of stock keeping units that may vary in terms of materials, size, shape, and color. Depending on the type of boxes being processed, two cases can be differentiated:
• Single SKUs →02a, ie, pallets consisting of boxes of the same size on each layer.
• Multiple SKUs →02b, ie, boxes of different sizes randomly organized on a pallet.

02 Examples of single and multiple stock keeping units.

The problem with traditional approaches Most existing automation systems apply a three-part approach to depalletization:
• Data acquisition: A 3D-camera mounted above the depalletizing area takes images of the boxes in the picking environment.
• Image processing: The acquired images are processed to find the position and size of the boxes to pick.
• Picking: The system defines the most suitable grasping position and orientation to pick items without damaging or dropping boxes.

This traditional approach uses segmentation and classification algorithms to identify and isolate boxes from their background, as well as 2D filters and masks to detect box contours, position, and orientation. Once a box has been identified, the system calculates its position and orientation and sends the resulting data to a robot for picking.

But while mixed SKUs are relatively easy to differentiate from one another, single SKUs present a significant problem since all the boxes have the same size and height, thus making it difficult to distinguish where one box ends and the next one begins – a task that can be difficult even for human eyes, especially if boxes are wrapped in plastic. Indeed, laboratory tests confirm that while geometrical algorithms result in segmentation errors only circa 2 percent of the time when handling mixed SKUs, they result in errors circa 33 percent of the time with single SKUs.

How machine learning is changing the picture
While similar to traditional systems, ABB’s approach introduces a game-changing element to solving depalletization: machine learning. The solution’s principle elements are a 3D vision sensor, which was developed and manufactured in-house, a control unit with machine learning-based vision software that’s applied to both single and mixed SKUs, and a robot equipped with a suitable gripper to pick boxes.

Machine learning makes it possible to build algorithms that rely on a collection of examples to improve the picking success rate. ABB’s example images are acquired with its own vision sensor and as the number of examples has grown, the system’s success rate has steadily improved. Known as supervised learning, which is a training function that maps an input to an output based on example input-output pairs, ABB produces models of single and mixed SKUs that take a vector x as input and output a vector y with useful information to the user →03 top. In other words, in the case of depalletizing, the input is an image acquired by the vision sensor and the output is a list of binary images, where a box’s location in a 2D image is indicated by white pixels on a black background.

center

In supervised learning, the dataset is the collection of labeled examples {(xi,yi)}Ni=1 such that x represents images acquired by the vision sensor and y is a collection of masks for each box in an acquired image →03. The labeling process mainly consists of the identification of box contours from a collection of acquired images. If the machine learning model is well trained it should be able to predict the position and orientation of previously unseen examples from a new image acquired by the sensor, consequently allowing a robot to accurately pick boxes of previously unseen sizes and geometries →04.

center

In ABB’s case study, the training dataset comprised many thousands of images taken by the vision sensor. Some of the images were taken during the first stage of training in the laboratory while the vast majority were taken in a real production environment. After image processing with geometrical algorithms and filters, data analysts labelled the examples of the datasets and identified the contours of the boxes. The result was that segmentation errors in processing single SKUs dropped to below 0.25 percent while being reduced to below 0.02 percent in the case of mixed SKUs. All in all, ABB’s case study demonstrated that the use of machine learning algorithms in combination with geometrical algorithms was able to dramatically reduce the error rate for picking boxes from pallets. 

References
[1] https://www.businesswire.com/news/home/­20170606005883/en/Global-­Automatic-Palletizer-Depalletizer-Market-Segmentation-Forecasts.