Detecting Discrimination in Job Vacancies: A Critical Reflection on the Potential of AI Language Models

Explicit discrimination in job vacancies by using terms that refer to the candidate's background is illegal. Yet, it is still present in numerous vacancies, as was recently observed in the Netherlands. Labour market authorities have organized efforts for the detection of explicit discrimination, which are based on the detection of terms such as "young" or "male". However, many non-discriminatory phrases also contain these terms, such as "we are a young company" or "working with male patients". This results in a labour-intensive task to identify discriminatory job vacancies and act on them. AI language models are seen as promising innovations that may improve efficiency. Yet, their use by governmental bodies raises concerns and requires caution. In this paper, we critically examine the potential of AI and language models to support labour market authorities in detecting explicit discrimination. We do this through an investigation of the potential efficiency gain whilst centring user needs. For this, we first create a labelled data set concerning gender discrimination and investigate a variety of models in their ability to detect known and unforeseen discriminating terms in context. Results show that these methods can support detecting explicit gender discrimination by bringing substantial gains to precision and make sensible suggestions for new terms to detect in vacancies. We complement this with a critical reflection based on interviews with ten experts. They state that considerations on responsibly using AI and language models go beyond efficiency, emphasizing the importance of the underlying goal of discrimination detection. Is this goal reached within a reasonable investment and with acceptable side-effects? In conclusion, this applied use case demonstrated that AI and language models could meaningfully bring efficiency to labour market authorities' efforts to detect explicit discrimination in job vacancies. However, we advocate that, even for technologies used for common good, critical reflections beyond efficiency are needed to decide between AI and non-AI alternatives.