Rock, Paper, Statistics

Mass screening for rare problems endangers society

This text is adapted in part from previous posts (e.g., on Chat Control, Chat Control again, the AI Act, how the usual liberty versus security framing is bogus, and why the regulatory challenge isn’t just about AI). 

What’s the Problem?

We can’t protect everyone from everything — and can do harm trying. Armed with new technologies to promote public interests including health, safety, and truth, we hit old limitations from universal mathematical laws — laws whose implications are not widely recognized. Thus, as technologies advance, proposed programs keep hitting the same wall, known in probability theory as Bayes’ theorem. Bayes’ says the probability of an event changes depending on the subgroup. 

When applied to mass screening for low-prevalence problems (MaSLoPP), Bayes’ rule implies that these programs backfire under certain conditions: 

• Rarity. Even with high accuracy and low false positive rates, the sheer volume involved in screening entire populations for low-prevalence problems generates either many false positives, or many false negatives (the accuracy-error trade-off). 

• Uncertainty. When real-world scientific testing cannot validate results, inferential uncertainty persists. 

• Harm. When secondary screening to reduce uncertainty carries serious costs and risks, resultant harm may outweigh benefit for the overwhelming majority of affected people. 

MaSLoPP occurs across diverse domains including security, health, (mis)information, and education. Although rarely recognized as a class, it shares a common mathematical structure that may endanger society. A currently controversial case illustrates the problem: Proposed mass digital communications scanning for child sexual abuse material (CSAM).

Case Study: Chat Control 

Consider Chat Control — a new AI tool designed to scan digital communications for CSAM. Imagine if, every time you use email, messenger, or chats, governments require the companies running the infrastructure to scan your communications for evidence of abuse, and report hits to the police on the basis of some algorithm that analysts don’t understand well. Proposed laws across Europe (Chat Control), the UK (the Online Safety Bill), and the U.S. (the EARN IT Act) would do just that. 

Decades in the making, this transnational initiative made headway when the UK Parliament passed the OSB on September 19, and the U.S. Senate Judiciary Committee for a third time sent the EARN IT Act to Congress for consideration in May. This month, the European Parliament’s Civil Liberties, Justice and Home Affairs Committee is expected to vote on Chat Control. Implementing these programs would destroy end-to-end encryption, create mass surveillance infrastructure – and endanger children. Here’s how. 

Reapplying public statistician Stephen Fienberg’s classic application of Bayes’ theorem from the National Academy of Science’s polygraph report to mass digital communications scanning: Assuming a 1/1000 CSAM base rate, an 80% detection threshold, and a .90 accuracy index, scanning 10 billion messages – a fraction of daily European emails –  would generate over 1.5 billion false positives. There’s a 99.5% probability (1,598,000,000/1,606,000,000) that a message flagged as abusive would be innocent. There’s an almost 16% chance (1,598,000,000/9,990,000,000) an innocent message would get flagged. Yet, 20% of the time, abuse would evade detection. Meanwhile, there would be almost 200 messages mistakenly flagged for each abusive message found (1,598,000,000 false positives / 8,000,000 abusive messages). 

Technology can’t escape math. Setting a lower detection threshold to reduce false positives leaves too much abuse undetected; raising accuracy to detect more abuse generates too many false positives. That’s why the National Academy of Sciences concluded mass polygraph screening of National Lab scientists offered “an unacceptable choice.” Mass digital communication screening faces the same dilemma. 

Subjecting entire populations to screenings like Chat Control implies following up many uncertain results with investigations – potentially traumatizing a large number of innocents, including minors. Will society jeopardize many children’s well-being to save a few? How did minors consent for their private communications’ use in training this AI? How will minors’ sexual images and texts, no longer protected by end-to-end encryption, be secured against misuse? What about privacy? Who will measure possible harms from this program, and how? 

A Matter of Digital Governance

As digitalization and AI increase infrastructural capacities to deliver public services, new MaSLoPPs may appear to improve on old ways of advancing the public interest. Their high accuracy and low false positive rates – probabilities – may sound dazzling. But translating the identical statistical information into frequency formats – body counts – shows otherwise. The common (false positives) overwhelms the rare (true positives) – with serious possible consequences. Ignoring this fact is known as the base rate fallacy. 

Nontransparency and perverse incentives often further trouble MaSLoPPs’ reported accuracy rates. For example, Chat Control’s accuracy estimates are based on unpublished analyses of unknown data from commercially interested sources. Chat Control has been heavily promoted by Thorn, a group registered in the EU lobby database as a charity that has made millions from U.S. Department of Homeland Security contracts for its AI tools. Across domains, quoted MaSLoPP accuracy rates are often generated by researchers with economic incentives to inflate them — and cannot be verified when the tests cannot be validated in the real world. 

Chat Control exemplifies a broader class of signal detection problems where MaSLoPP produces uncertain benefits and massive possible damages. Other such programs across diverse domains all face the same accuracy-error trade-off under the same universal mathematical laws. Rare problem incidence, persistent inferential uncertainty, and secondary harms doom many well-intentioned programs to backfire. 

This case study highlights the need for better information security and statistics education supporting international cooperation to enhance cybersecurity. Experts from industry and academia agree that the proposed program would create more digital infrastructural vulnerability than it would mitigate. What is not being discussed, however, is how the same mathematical structure also characterizes mass screenings that tech companies conduct on their digital platforms for misinformation at the behest of governments. 

In a step toward recognizing MaSLoPP’s risks, the EU Parliament’s recent draft AI Act recommends banning several AIs sharing this structure. The implications of probability theory apply equally to these programs – emotion-recognition AI (recently piloted in border crossing screening tech iBorderCtrl), real-time biometrics and predictive policing in public spaces, and social scoring. But we need a new legal and ethical framework beyond AI, regulating the class. 

Policy Recommendation

Policymakers should apply scientific evidentiary rules to evaluate the likely impact of proposed interventions including MaSLoPPs before implementation:  

• Proponents bear the burden of proof to establish that new interventions will do more good than harm. 

• Independent reviewers evaluate the evidence of claimed costs and benefits. 

• Relevant data must be public, including information about its production, storage, analysis, and interpretation. 

Applying these rules to policymaking may prevent massive societal damages from well-intentioned programs that are unknowingly trying to break universal mathematical laws that we cannot escape.