A vision towards automatic inference of hostile intent from sensory observations

Amsterdam, The Netherlands, 27th of May 2010. The national security authorities call out to
secure the area around the central train station. 150 Security officers, both military and police, are
instructed to guard the area and to pick out those people with hostile intent. Although the officers
were very effective in assessing people’s nervousness and where they were looking, - a lot of other
indicators were hard for them to recognize. In many cases the officers did not recognize particular
indicators. At TNO, we have a vision on the technology to support them. So how to solve the
puzzle of discovering somebody with bad intentions?
Evidence exists that hostile intent can be detected from particular behaviors. Psychological
experiments with CCTV videos demonstrated that humans can very well assess whether particular
behaviors hint at a future escalation. Also, from analysis of other attacks, we have learned that
some clues were visible from the surveillance cameras. This observation led to a compilation of
so-called ‘deviant behaviors’. The list is categorized by indicators that relate to respectively the
appearance of a person, his or her specific behavior and objects that are carried, touched or
exchanged.
With these ‘deviant behaviors’, the problem of ‘intent recognition’ can be decomposed into
realistic challenges of early detection of specific behavioral indicators. We distinguish between
four types of features: (1) trajectories (e.g. where did somebody move, with whom did the person
interact), (2) appearance of the person as a whole (e.g. the pose and entropy), (3) body parts and
spatiotemporal features, (4) physiological properties (e.g. the change of body temperature, sweat
or heart beat). For category 4 we investigated whether vital life signs can be monitored at a
distance, together with a renowned health-care innovator. Whereas this is still very rudimentary,
current research focuses on categories 1, 2 and 3. These properties demonstrated to be very
discriminative, especially when they are combined. Inspired by trained police and soldiers, we
have build software to combine behavioral indicators on the spot, the ‘0+0+0=1’ principle.