Context and Challenges
In modern automated warehouses, Warehouse Management Systems (WMS) can pinpoint the exact location of any item, guiding human operators to the correct shelf for picking and placing tasks. However, the final “last-mile” action of manually handling an item remains a critical source of error. An operator might accidentally pick an adjacent item or misplace a product during replenishment, leading to inventory discrepancies and fulfillment delays.
We were tasked with building a system to verify these manual actions in real-time. This presented a formidable set of challenges:
- Zero-Shot Generalization: The solution had to be universally applicable to any warehouse environment without prior training on specific products, shelf layouts, or operators. It needed to be robust against variations in lighting, operator habits, hand colors, and the use of different types of gloves.
- Extreme Hardware Constraints: The entire system had to operate on a single, compact processing unit (an NXP i.MX 8M Plus with a Hailo Hailo-8 accelerator). This unit alone had to serve 5+ cameras simultaneously to cover the full length of a warehouse aisle.
- Computational and Bandwidth Bottlenecks: Processing multiple high-resolution video streams in real-time on such limited hardware created significant computational and data bandwidth hurdles that conventional approaches could not overcome.
Solution
To address these complex requirements, we engineered a highly optimized, multi-view computer vision system capable of understanding operator actions with precision.
- Advanced Hand Gesture Recognition Models: We developed lightweight yet powerful deep learning models for hand and action understanding. Trained on diverse datasets, these models achieve the required zero-shot performance.
- Dynamic Computational Budget: An important part of the innovation was a system that intelligently allocates processing power. Instead of treating all camera feeds equally, the system identifies which cameras are most relevant to the operator’s current task and dynamically retargeting its computational budget, thus overcoming the hardware and bandwidth limitations.
- Action Verification: The system cross-references the position of a detected “pick” or “place” gesture with the target coordinates provided by the WMS. This allows it to instantly confirm whether the operator has interacted with the correct item and location.
Results
The implemented system successfully provides a critical layer of validation for manual logistics operations. By intelligently managing resources across multiple camera streams, the solution runs seamlessly on the specified compact hardware.
Most importantly, the system has significantly reduced the human error rate in both picking and placing tasks, boosting inventory accuracy and operational efficiency.
This was achieved without adding any friction to the operator’s experience, allowing them to work at their normal pace with a newfound level of accuracy.
