HECTOS - Harmonized Evaluation, Certification and Testing Of Security products - D5.4 Testing and evaluation results for the weapons and explosives detection case studies. Part II – People screening portals

HECTOS is an EU FP7 security research project that is exploring the issue that there are very few test, evaluation and certification procedures in Europe for physical security products that are mutually recognized by different Member States. As pointed out in the EC Communication on Security Industrial Policy, this leads to fragmentation of the market, with negative impacts on both suppliers and users.
HECTOS will identify mechanisms to evaluate the performance of security products, as well as compliance with interoperability, regulatory, ethical, privacy and other requirements. The project will develop elements for a roadmap for the development of new harmonized product certification schemes.
By conducting two case studies in the priority areas “Biometrics” and “Weapons and Explosives Detection” HECTOS will enhance, and experimentally investigate elements of the evaluation and certification schemes developed in WP3.
The objective of this deliverable D5.4 is to elaborate a detailed test methodologies (TM) for Weapon and Explosives (W&E) detection devices (portals) for people screening. This TM is then used to carry out different tests for investigating elements of the harmonized certification scheme. It concerns topics identified in D3.2 as specifically important to be included in such a scheme:
1. Determination of the ROC-curve, with the objective to investigate the application-driven evaluation;
2. Repeatability, with the objective to identify important aspects that ensure inter-lab and intra-lab repeatability;
3. Verification of technology independency, with the objective to identify elements of the test method that enhance or impede technology independency.
For the first topic, performance evaluation methodologies have been specified, based on the general test method developed in D5.3. The second topic will be reported in Part I and version 2.0 of Part II of this deliverable and the third topic is assessed by means of a theoretical study.
The main conclusions from the evaluations are:
• Test person-based testing gives a more realistic assessment of the WTMD than the tests without test persons. However, due to a number of deliberately uncontrolled parameters, this more realistic results show a high variance and come hence at the expense of accuracy and repeatability;
• The spatial discrimination of WTMD is not good enough to support performance assessment with zone indication (CTA, TFA, IFA). It is much more convincing just to consider the WTMD as a simple binary detection system with a 2 x 2 contingency table (alarm / no alarm);
• Mixed threat item / innocuous item testing gives a more realistic impression of the actual behavior in the field because of the interference that can occur between two or more items. An important consequence of this interference is that full divested screening might result in lower detection rates;
• The ROC-curve based evaluation seems to be especially useful when operated in partially divested mode. For non-divested mode full detection is obtained, even at the lowest sensitivity and with and without threat items. For full divested operation, the False alarm rate is zero and no ROC-curve is obtained;
• The ROC-curve approach enables type certification based on its intended application, i.e. divestment level and sensitivity setting;
• Scoping tests should be used to determine the regions where the detection system has sensible performance, so that testing efforts can be focused on those and to avoid wasting time on regions where the performance is virtually zero or virtually perfect;
• A one-protocol-fits-all for people screening detection systems is not realistic. It must be ensured that the assumptions made in the development of a test method are valid for all the different types of equipment that may be tested. A process would be built in to the product evaluation process to examine the operating principles of the device under test and determine whether the proposed test method is suitable and how it should be adjusted to fit those operating principles.