AI getting hacked – systemic vulnerabilities of a booming technology

The use of Machine Learning (ML) systems is increasing rapidly. However, practitioners often do not understand, think about, or protect against an emerging class of vulnerabilities: “adversarial ML”. This threat encompasses any targeted exploitation or hacking of AI systems, leveraging ML-specific vulnerabilities.¹

Together with recent progress in Deep Learning,² research interest in adversarial ML has increased notably.³ Professional hackers are not only able to trick models into making mistakes or leaking information. They can also harm model performance by corrupting training data and/or stealing and extracting ML models. 

The following examples showcase the diversity of potential ML vulnerabilities:⁴

1. Data/model poisoning (backdoors): A backdoor responds to a trigger (specific data pattern) and produces a pre-determined outcome (eg, an exceptionally high creditworthiness or insurance score). Backdoors can be introduced using malicious pre-trained models (used as a basis for specialized ML) or malicious data, for instance by a disgruntled employee.

2. Model evasion: Attackers could use adversarial ML to produce patterns that mislead ML systems and are difficult for humans to notice.⁵ For example, a visual pattern that – if applied to a car as a sticker – leads an automated motor claims tool to misjudge damage.

3. Membership inference attacks can leak sensitive information used for the original training of ML models (eg, a confidential dataset that was used during a system’s learning phase). By constructing a series of targeted queries, attackers can extract training data points with high probability. This raises concerns over data protection issues.

The use of complex machine learning systems is becoming more widespread. By design their outputs are difficult to check for mistakes (both by humans and other ML). Furthermore, the systems are inherently vulnerable to adversarial attacks.⁶ These vulnerabilities pose risks to insurers and businesses alike, and are a challenge for regulators also.⁷

Regarding risks relevant for insurers (beyond cyber insurance), there are increasing opportunities for fraud, as well as potential claims in professional indemnity and errors and omissions (E&O) lines for ML failures and data breaches. Furthermore, adversarial ML attacks leaking to the media (however small the impact might be) could directly impact the reputation of insurers and/or their assets. Model-stealing could lead to intellectual property (IP) loss. Beyond that, data leakage or non-compliance with current and upcoming data and AI regulation⁸ might trigger fines. In some instances ML malfunction could cause harm or accidents (autonomous car crashes, medical misdiagnosis⁹ etc) triggering casualty or health covers.

Strict access management, suitable usage limits and data governance can go a long way in reducing attack surface (areas of vulnerability that can be attacked). While not always applicable, simply not exposing models to the internet, and using only trusted (eg, not automatically collected) data, are powerful risk mitigation strategies. Getting all this right is by no means an easy feat, however. It entails making security and data governance a core feature of ML development and deployment, and striking a balance between usability and privacy/IP-protection. Starting this transition now will make ML applications more resilient in the future.