Anomaly Detection in an Open World by a Neuro-symbolic Program on Zero-shot Symbols

Mobile inspection robots are typically tasked to find anomalies and report them, such that proper action can be taken. They operate in an open world, in which they encounter previously unseen situations in changing environments. We take the robot’s goal, its context, prior knowledge and uncertainties into account to find anomalies that are relevant for the operation. Prior knowledge is expressed by logic formulas. E.g., a tool should not be left on the floor. These symbolic formulas describe anomalous objects in terms of predicates and variables (symbols) that can represent various concepts in the real world, their attributes and their relations. This knowledge can easily be adapted during the robot’s operation to a new anomaly that is considered relevant. Reasoning is performed in a probabilistic, multi-hypothesis framework. A neuro-symbolic program evaluates the symbolic formulas against probabilistic, imperfect observations of the symbols. New anomalies require new symbols, which are measured in images by zero-shot language-vision models and their extensions for objects and segments. Starting from the symbolic formulas and predicates that describe the anomaly, our method infers what objects are involved that need to be detected, extracts the probabilistic information from the language-vision models and reasons about that via a neuro-symbolic program in order to find the anomaly of interest. Our contribution is the integration of the neuro symbolic program and language-vision models. We show the effectiveness of our method to find anomalous situations in a robotic inspection setting.