The Technological Challenge: Combining People, Activity, and Spatial Reference

In the ZIM MatrixLotse project, we developed an AI-based software solution for analyzing and optimizing manual matrix assembly processes. The goal was to detect people and body movements using cameras, classify activities, evaluate movement paths and process data, and derive automated optimization suggestions for workplace layouts.

Robust Camera and 3D Foundation

The technical foundation is a stereo vision setup consisting of GenICam cameras. After testing various resolutions and optics, 5-megapixel cameras with 4 mm lenses were selected. The cameras were mounted in parallel on a shared axis with a horizontal distance of 124 centimeters and tilted downward by 30 degrees. Overlapping fields of view eliminate blind spots and ensure that relevant areas are captured from at least two perspectives. For calibration, the system utilizes a checkerboard pattern and approximately 15 images taken from different positions. The extracted camera parameters allow for the correction of perspective distortion and the transformation of image data into a unified coordinate system. This ensures the setup remains adaptable to new assembly areas.

Image 1: Calibration of the stereo vision setup using a checkerboard pattern

Multi-Person and Depth Detection

During pose estimation, initial testing revealed that MediaPipe lacked sufficient scalability for industrial multi-person scenarios. The project therefore switched to YOLO v11. This model detects multiple individuals simultaneously, supports real-time streaming, and provides 17 body keypoints. During testing, detection remained stable even with workwear, challenging lighting conditions, and temporary occlusions.

Image 2: Multi-person detection with bounding boxes and skeleton keypoints

‍

Depth Anything and Depth Anything v2 were integrated for depth estimation. These models generate depth maps directly from camera images. Combined with stereo reconstruction and YOLO skeleton keypoints, a 3D representation of individuals in the workspace is created. Missing skeleton keypoints can be reconstructed over time series to reduce jumps and dropouts.

From Body Pose to Activity Recognition

The second AI model classifies activities into standing, walking, and working. It is based on synchronized stereo videos annotated along a timeline in continuous activity segments. The reconstructed 3D poses are then transformed into a person-centered coordinate system. This removes global positional shifts and makes movement patterns of different individuals more comparable.

Image 3: Comparison between camera-centered 3D pose and person-centered normalization

‍

During model selection, recurrent and graph-based architectures were evaluated. A Temporal Convolutional Network (TCN) was chosen as the target architecture. TCNs process temporal sequences in parallel, capture short- and long-term movement patterns, and run efficiently on industrial server hardware. Heuristic post-processing further stabilizes predictions. Temporal smoothing, hysteresis rules, and confidence-based gating reduce label flickering and short-term misclassifications. Additionally, interpretable features such as joint speeds and torso bending angles are taken into account.

The Solution: An End-to-End Software and Data Pipeline

High-performance GPUs on proprietary hardware were used for training. Model deployment takes place in a containerized environment using Docker Compose. Compared to Kubernetes orchestration, this approach reduced complexity and simplified maintenance and configuration.A deployment tool developed in Python and Django manages models and camera perspectives. It supports the planning of overlapping fields of view and scaling to larger assembly areas. An automated interface ensures that updated model versions are integrated into the production pipeline. Additionally, a generic REST interface with OpenAPI specification was developed. In JSON format, product variants, manufacturing steps, process times, workstations, available employees, and production quantities can be transferred from an MES. This links movement and activity data directly with the respective production context.

Concrete Results: From Movement Paths to the Zone-Action Matrix

The analysis pipeline delivers several immediately actionable evaluations:

• Movement paths are calculated from the 3D position of the hip center and displayed as a top-down trajectory, including the total distance in meters.
• Workstations and process cells are modeled as spatial zones; dwell times per zone can be calculated and visualized as a heatmap.
• A Zone-Action Matrix shows the time spent working, walking, or standing within a specific zone.
• Variant-specific evaluations allow for the comparison of spatial work patterns across different product variants.

Image 4: Heatmap of spatial occupancy with defined work zones and dwell times

‍

These key metrics support the interpretation of real assembly processes. A high concentration of work time can point to a bottleneck, increased walking time to layout inefficiencies, and longer standing times to waiting or idle periods.

Automated Layout Optimization with Reinforcement Learning

A policy-based Reinforcement Learning approach was selected for the third AI model. The workspace layout is represented as a grid-based configuration. The agent shifts stations incrementally and evaluates each change using a cost function that accounts for spatial constraints, workflow dependencies, and realistic path lengths. Invalid configurations—such as overlapping stations or boundary violations—are excluded or penalized. In model tests, the system reduced the internal layout cost function for a small layout with three workstations by approximately 99.94 percent. For a larger layout with four workstations, an improvement of around 56.89 percent was achieved. The results also highlight current limits: as the number of stations increases, the search space grows significantly, making convergence behavior less stable.

Image 5: Initial and optimized three-station layout along with the progression of the internal cost function

‍

The developed frontend allows users to graphically configure room dimensions, stations, and workflow sequences, and to initiate optimization without needing in-depth knowledge of the underlying Reinforcement Learning. Initial and optimized layouts, as well as cost progressions, are stored in a structured format for full traceability.

Image 6: Frontend for layout configuration and initiating the optimization process

‍

Added Value for Industrial Manufacturing

MatrixLotse combines analysis and optimization within a single system, offering the following key benefits:

• Objective evaluation of movement paths, activities, and zone occupancy,
• Reproducible analyses through saved models, parameters, and result artifacts,
• Flexible integration via cameras, container deployment, and standardized MES interfaces,
• Transparent optimization suggestions with visualizable layout changes,
• GDPR-compliant operation, as no personal data is stored and no tracking of individual employees is intended.

The project milestones were successfully achieved: A confidence level of 99 percent was confirmed for body movement detection and tracking. The activity model achieved a performance of 92 to 96 percent even without heuristic stabilization; with correction mechanisms applied to the test dataset, the required confidence level of at least 99 percent was also met.

Conclusion and Outlook

MatrixLotse demonstrates how camera-based process analysis can evolve from passive observation into a resilient decision-making tool. Multi-camera tracking, 3D skeleton data, activity recognition, MES context, and Reinforcement Learning seamlessly interlock in an end-to-end pipeline.The result is a fully functional system whose value for automated capture and optimization of manual assembly processes has been validated by test users among client companies. The next development step is transitioning to series maturity, focusing particularly on scaling to larger and more complex layouts, broader training scenarios, and additional ergonomic, safety, and domain-specific constraints.

‍