SensorLM: Real-Time Sensor Fusion Using LLM Reasoning
Its wheel encoder shows it's slowing down unexpectedly Its thermal sensor detects a warm object ahead Its motion prediction model predicts the wheels should still be moving normally.
What if the robot’s system structures this into real-time schemas like:
{
"sensor_type": "rotary_encoder", "metrics": {"wheel_speed_m_s": 0.1}, "recent_values": [0.8, 0.7, 0.1], "metadata": { "units": "m/s", "mount_position": "rear_left_wheel" }
}
A similar schema from the thermal sensor, showing a 37°C object 0.6 m ahead Another from the prediction model, showing expected speed of 0.8 m/s
The LLM than receives these and reasons:
“Observed wheel speed dropped sharply from 0.8 to 0.1 m/s. Thermal sensor shows a warm object directly ahead. Predicted speed was 0.8 m/s — but reality diverged. Likely obstruction from human or animal. Initiate 5-second pause and begin visual recording.”
I'm thinking of building a lightweight library to make this plug-and-play:
You feed it raw, filtered, or predicted sensor data, and it outputs structured, LLM-readable snapshots — ready for reasoning and decision-making.
Not launched yet — just validating.