Can we trust the systems we now rely on?

Article by Adam Green, freelance journalist

As a speck of dust lands on a camera lens, inside the neural network interpreting the road ahead, a 30mph speed limit sign registers as 70. The vehicle, trusting the system entirely, accelerates.

“When vehicles are making autonomous decisions, we just let them drive,” observes Luca Arnaboldi, Assistant Professor of Cyber Security at the University of Birmingham. “But how do we ensure we can trust them?”

With automated systems taking on increasingly high-stakes decisions across society — from vehicles to hospitals, financial systems, and critical infrastructure such as energy, transport, and communications — society requires new risk frameworks that put dependability at the centre, ensuring AI remains trustworthy even in the face of uncertainty and failure.

A sticker on a signpost can fool AI

Despite the growing sense that AI is beginning to “reason”, its decisions are fundamentally statistical: models learn by absorbing vast amounts of data into general patterns and predicting what’s most likely, on average, for any request, whether or not that input actually makes sense.

In a vehicle travelling at speed, where accidental or adversarial variations far exceed what AI can be trained or tested on, small errors don’t stay small. A sticker on a stop sign, worn lane markings, dust on a lens, a spurious LiDAR reflection: each deviation is trivial for a human observer, but sufficient to push an AI system into misreading a sign, drifting out of lane, or failing to brake at exactly the wrong moment.

These kinds of real-world failures are exactly what the risk frameworks that automotive engineers currently rely on — threat assessments and safety standards like ISO/SAE 21434, which were written before generative AI entered the picture — fail to capture, as they mostly rely on predefined deterministic failure modes for individual components. “How do we certify all these components when they work together [in the AI lifecycle]? That’s where the real exposure is,” warns Arnaboldi, reflecting a broader line of research at Birmingham on verifying the integrity of complex systems. “If I can’t write the spec, I can’t check if the output is correct,” he adds. “And because these models are so large, we often can’t even understand how they reached their conclusions.”

The consequence is a false sense of assurance: systems certified as safe can behave unpredictably under real-world conditions. Even in the controlled settings of hacking competition Pwn2Own, the limits of that assurance are clear: in 2024, a Tesla electronic control unit (ECU) was compromised in under 30 seconds; in 2025, researchers discovered 49 previously unknown zero-day security vulnerabilities across automotive systems in just three days.

Outside the lab, the barrier to exploitation is rapidly falling: cyberattacks targeting automotive and smart mobility organisations more than doubled in 2025, with 92% conducted remotely and fewer than one in six requiring any proximity.

Trust, failure and the rise of dependability

That AI vulnerabilities exist in certified systems is not a surprise to Arnaboldi; it is the predictable result of treating the two disciplines designed to prevent them as separate: safety engineering, which estimates how often a hazard might occur and models acceptable risk, and security engineering, which asks what happens if someone deliberately causes that hazard.

“I can’t calculate the probability of an attacker deciding to put a rock on the road. But once they have, the chance is 100%, and the effect on the vehicle is identical to an accident,” explains Arnaboldi, highlighting how a hazard is the same from a system’s perspective, whether it arises by chance or by design. “I don’t believe safety and security should be separate,” he then adds, sharing his view that “it is a mistake when we have a safety team and a security team not talking to each other, pretending they’re different things” — a divide sustained more by organisational habit than by logic.

Regulators are beginning to reflect this shift. The UNECE WP.29 cybersecurity rules, mandatory since July 2024 for new vehicles in tens of markets and in the process of being localised in the UK, require manufacturers to address cybersecurity across the vehicle lifecycle, treating security as a core component of safety.

Yet aligning safety and security is only the tip of the iceberg. When trying to reconcile what researchers understand with what industry builds, Arnaboldi points to a deeper requirement: dependability, or the ability of a system to remain trustworthy through failure, and not just before it. Dependability is a long-established field in computer science, going back generations of work on fault-tolerant systems. Arnaboldi takes those principles and applies them to AI: in one such example he contributed a formal proof that safety and security must be co-engineered if dependability is to hold. The starting premise is one engineers often avoid stating out loud. “We have to assume software will break, edge cases will arise, attacks will succeed,” he says. “The question is how we keep things working when they do.” From this standpoint, the goal shifts from preventing faults to sustaining function under them — through mechanisms such as graceful degradation, fail-safes, and recovery layers, systems can remain dependable even under fault or attack.

Cybersecurity has an unusually unforgiving relationship with reality: ideas that hold on paper can fail instantly when exposed to real systems, real users, and real adversaries. “The real world doesn’t care about your proofs,” Arnaboldi argues. “You need to find actual attacks or actual failures.” Translating cybersecurity research into tools that industry can use has always been the harder half of the problem, and the reason real-world contact is not optional in cybersecurity.

Birmingham’s location matters here. Major UK automotive manufacturers and engineering partners — including JLR, TATA Motors and Horiba MIRA — sit close to the Edgbaston campus, and they have historically shared close collaborations with the Birmingham Centre for Security and Privacy.

A methodology that travels

As AI takes on more of the decisions that used to require human judgment, that combination of rigour and urgency may be exactly what the field needs most. The methodology running through Birmingham’s autonomous systems security and privacy work — formal verification, protocol analysis, real-world testing — extends across the group’s research into contactless payment security, post-quantum cryptography (work so influential that it has informed protocols recommended by GCHQ, the UK’s intelligence and cybersecurity authority), embedded microcontrollers at the heart of vehicle ECUs, and the privacy risks inside AI training pipelines themselves.

In previous work alongside collaborators at Edinburgh, Heriot Watt and Strathclyde, Arnaboldi helped develop tooling to formally verify AI safety, extending from machine vision to natural language. One illustrative case sat at the boundary between formal verification and law. Germany’s Medienstaatsvertrag — the Interstate Media Treaty in force since 2020 — requires that content produced automatically by software, including by text-based AI systems, be identifiable as such rather than presented as the work of a person.

The EU AI Act has since reinforced the principle: under its Article 50, AI systems interacting with people must be designed so that those people are informed they are dealing with an AI. Verifying compliance turns on a deceptively hard question: across the open-ended ways a system can be prompted or queried, can it be relied on to disclose its automated nature?

There are infinite ways to phrase the question in language, and no test suite can cover them all. Arnaboldi and his collaborators set out to formally verify the property. Their analysis was thorough but incomplete: they could not prove the system would respond correctly in every conceivable case. The lawyers they were working with read the result differently. From a legal standpoint, the analysis could be argued as best-effort compliance — and therefore as satisfying the requirement. A complete proof had not been achieved; a useful one had.

That expertise has carried into how Arnaboldi communicates AI risk outside academia: he is one of the academic experts shaping Business in the Community’s Responsible AI Framework — practical guidance, sponsored by Deloitte and Verizon, that UK business leaders are using to deploy AI inside their organisations.

What Arnaboldi values about Birmingham is the culture more than any specific technique. “People here just want to do good work, without ego. That’s the kind of culture I value.”

Arnaboldi’s ‘formal pragmatism’ takes rigorous logic off the whiteboard and puts it to work. By bridging the gap between theory and results, he is moving AI safety from a statistical hope toward a verifiable reality — building technology defined as much by its resilience as by its intelligence.