Sensor Fusion & Physical AI

Multi-sensor data acquisition, sensor data processing, and data fusion are the origin of FusionSystems; this is where the company’s name comes from. Today we combine these methods with machine learning: learned models detect objects in the data of the individual sensors, and the fusion merges the results into a picture of the environment.

Why one sensor is not enough

Every sensor has strengths and blind spots. A camera recognises what an object is but suffers from backlighting. A laser scanner measures distances precisely but says little about the type of object. Satellite positioning provides a position, but not in a parking garage. The result only becomes reliable when the sources are merged, weighted against each other, and checked for plausibility.

This includes tracking over time. An object that is tracked across many measurements is detected with greater certainty than one from a single measurement, and its movement can be predicted.

Example: lane-accurate position

A state estimator from the Kalman filter family merges the sources. In addition to the position, it also provides a statement of how accurate that position is.

Testing on the road

Our test vehicle captures its surroundings all around: with a multi-camera system for three-dimensional capture, a 360° laser scanner, and a surround-view camera. The measurement data is evaluated in the vehicle in real time. During measurement drives in Chemnitz, road users, roadway, traffic lights, and traffic signs are detected, localised, classified, and rated with a measure of reliability. More on this in the project AI-Based Environment Perception for Autonomous Public Transport.

When the human is part of the scene

The same technology captures people. A sensor chair with pressure sensors and a camera observe the same movement from two perspectives; their data is merged so that the system reliably recognises what the person is doing at that moment. In the MotionComposer, the recognised movement immediately becomes sound. Here, latency counts alongside accuracy: the feedback must come so quickly that it is experienced as a consequence of one’s own movement.

What we mean by Physical AI

A sensor that already evaluates while capturing is a smart sensor. When control and actuators are added, the result is a system that acts on its environment independently. Physical AI is our term for AI that works in such systems: in vehicles, automated guided vehicles and mobile machinery, and interactive devices. It must decide in real time, cope with incomplete measurements, and detect the faults of a sensor.

We describe the mathematical foundation for this, state estimation and the parallel tracking of multiple interpretations, under Hybrid AI & Decision Intelligence.