How Much for Mass Surveillance?
As the EU political process continues to neither approve nor kill Chat Control, dirty tricks are confirmed -- and proponents of client-side scanning try buying mass surveillance in U.S. court
Law-breaking, lies, dangling large payments before vulnerable groups for playing a political part, plotting to undermine policework at a massive scale. You might think that describes seasoned criminals’ dirty tricks playbook. But it’s a summary of what mass surveillance proponents have been up to lately in promoting their preferred solutions to child abuse — solutions that would undermine end-to-end encryption, the structural foundation of digital civil rights.
Earlier this month, the European Data Protection Supervisor reprimanded the EU Commission for having targeted left-leaning Dutch citizens with X (formerly Twitter) ads promoting Chat Control, an EU proposal to scan all digital communications using AI for child sexual abuse material (CSAM). This so-called microtargeting broke the bloc’s own rules against processing sensitive personal data without people’s explicit prior consent. It was part of a broader pattern of misinformation including spinning propaganda as survey research to promote mass surveillance.
This anticipated reprimand comes on the heels of Chat Control’s latest EU political process defeat: the Hungarian Council Presidency, like its predecessors, couldn’t secure a majority for its version of the proposal, and so took the regulation off the agenda in October. But that didn’t get it off the EU Council agenda in December (h/t Patrick Breyer’s excellent negotiations timeline). This Zombie bill back-and-forth has been going on since the EU Commission proposed temporary legislation allowing Chat Control in 2020 — raising the question of how much has been spent over the last four years promoting a proposal that can’t pass, but won’t die…
It’s a question of how much mass surveillance would cost in Europe today. But there might be a deal in the works: A new lawsuit against Apple in California makes it look like wholesale Western mass surveillance could come pretty cheap, after all. The lawsuit scales up a similar lawsuit filed in August from one victim to a group.
For a mere $75,000, the Heat Initiative underwrote a lawsuit seeking monetary damages for a group of child sexual abuse victims. The minimum damages entitlement is $150,000, and there are 2,680 eligible victims. Their cumulative award could exceed $1.2 billion.
Apple’s misdeed? Not implementing the client-side scanning system “NeuralHash” to detect CSAM in iCloud. NeuralHash was an algorithm that Apple designed to scan hardware running its operating systems for CSAM. The company abandoned it when scientists reminded it that it can’t escape math.
Chat Control is one example of mass screening for a low-prevalence problem — a dangerous mathematical structure. It requires breaking end-to-end encryption, the technological bedrock of digital privacy. Such a move would make mass surveillance cheap and easy again, like it was before post-Snowden reforms normalized encryption.
Who wants to buy mass surveillance?
According to Sam Biddle of The Intercept, the Heat Initiative is a nonprofit child safety advocacy group funded by the Hopewell Fund, a Democratic Party-aligned dark-money group using billionaires’ money to undermine privacy. Writing for The Atlantic, Emma Green called its self-described sister “social welfare organization” the Sixteen Thirty Fund “the indisputable heavyweight of Democratic dark money.”
The lawsuit results from Heat Initiative founder Sarah Gardner researching and recruiting a law firm specialized in representing child sexual abuse victims to go after Apple — two years after Apple announced its plan to drop client-side scanning tech development in favor of instead strengthening end-to-end encryption while offering an opt-in message screening feature for children’s accounts designed to help prevent sexual abuse. It’s a political maneuver by an organizational figure, not a grassroots effort to solve the problem.
This maneuver rides a long wave of biased and inaccurate reporting on this topic, like the New York Times’ “In Monitoring Child Sex Abuse, Apple Is Caught Between Safety and Privacy” (Tripp Mickle, Sept. 1, 2023), reporting on the Heat Initiative raising “$2 million for a new national advertising campaign calling on Apple to detect, report and remove child sexual abuse materials from iCloud, its cloud storage platform.” The money went to digital ads “on websites popular with policymakers in Washington, such as Politico,” and “posters across San Francisco and New York that say: ‘Child sexual abuse material is stored on iCloud. Apple allows it.’ ”
Safety versus privacy, security versus liberty — the well-worn frame pits the two values as if they conflict, and we have to prioritize one over the other. The value conflict claim implicit in this frame is so often repeated, even by digital civil liberties groups, that it might not occur to people to question whether it has an established evidentiary basis.
It does not.
Mass surveillance undermines both liberty and security
The maxim of probability theory known as Bayes’ Rule implies that false positives will overwhelm true positives in programs of this structure — mass screenings for low-prevalence problems under conditions of rarity, persistent uncertainty, and secondary screening harms. Under these conditions, even highly accurate such programs backfire by making huge haystacks (wrongly flagged cases, “false positives”) while missing some needles (wrongly cleared cases, “false negatives”).
The real-world numbers are likely far worse than my own previous analysis showed: The base rate is probably at least an order of magnitude lower than the one I used to illustrate the point, dedicated attackers gaming the system would likely deflate the error rate, and artificial testing conditions (and perverse incentives) likely inflate the accuracy rate. My analysis intentionally made assumptions that were generous to the program’s proponents in order to simplify and illustrate the point that it would backfire.
Despite its many imperfections, estimating hypothetical outcomes of a program of this structure using Bayes’ Rule provides a useful correction to the common cognitive bias of the base rate fallacy. Another essential part of calculating costs of such a program entails estimating its implications in the context of finite resources. The false positive flood tends to overwhelm resources needed for correctly flagged cases.
In this context, when finite investigative resources are tied up processing CSAM possession tips from mass scanning, they cannot be used for other investigations. This problem is already causing resource exhaustion, as described by Sven Schneider, the head of the Central Evaluation and Collection Point for Child Pornography (ZASt) at the State Criminal Police Office (LKA) in North Rhine-Westphalia. Schneider told Deutschland Funk in 2022 that his investigators at ZASt were so inundated with NCMEC (National Center for Missing and Exploited Children) tips that they lacked the capacity to pursue other investigations. (NCMEC’s is the same database against which Apple’s abandoned NeuralHash algorithm was designed to compare images.)
This is consistent with the possibility that children are endangered by such mass screening programs exhausting the investigative resources necessary to process tips that have a higher likelihood of being true positives and may otherwise be more relevant to current as opposed to past abuse.
Curtailing targeted investigations that might stop ongoing abuse or bigger-fish distributors in favor of processing mass scanning tips that are overwhelmingly false positives does not serve the interests of vulnerable children or society.
But, by rationalizing the structural change in terms of child welfare — terms that are seen as sociopolitically difficult to contest — it does serve the interests of those who would like a return to cheap, easy mass digital communications surveillance.
Why buy mass surveillance?
What the security state wants, the security state gets. The strange thing is how cheaply it might get it when it comes to undermining post-Snowden end-to-end encryption to reinstitute mass digital communication surveillance. The redeeming hope is that “the security state” is a heterogeneous amalgam of actors, some of whom understand statistics.
According to the implications of probability theory, mass surveillance undermines the security it claims to advance. So we get mass surveillance when decisions driven by ignorance and/or self-interest prevail. Ignorance of the implications of the universal laws of mathematics. Self-interest — perhaps understanding the math, but not caring about what actually promotes security, as opposed to what makes profits (be they literal corporate proceeds, sociopolitical power returns, or both).
We don’t get mass surveillance when people who understand statistics make policy decisions that prioritize security. And when those decisions are actually enacted by the agents of those principals’ choosing. Agents who may not understand or care what the consequences of their actions really are…
In reality, there is widespread agreement among tech and intelligence professionals that security agencies will probably conduct politically actionable telecom surveillance whenever they can. In a best-case scenario, this is a principal-agent problem: Principals (policymakers) who understand statistics better than their organizational (security apparatus) agents are stuck trying to constrain middle management from doing what they (stupidly) think is best. (In a worse-case scenario, principals prioritize their own sociopolitical power, not advancing security in societal interests; abusing power through surveillance and other means is par for the course.)
The principal-agent problem here requires both technological and political solutions. That is, guarding against mass surveillance requires making it structurally difficult and (ideally) prohibitively costly. In this view, end-to-end encryption serves security as well as privacy interests by making it harder and more expensive for agents to undermine principals’ interests.
Client-side scanning requirements do the opposite, allowing states to delegate mass surveillance to companies running services like cloud storage and encrypted messaging apps. Once delegated, it’s relatively cheap and easy to hack or otherwise capture the surveillance apparatus, tapping the flow of information that was engineered to be vulnerable. Hack — like the NSA running Room 641A to warrantlessly wiretap and data-mine Americans’ communications (something that wouldn’t have been possible with end-to-end encrypted traffic). Otherwise capture — once one state directs one major company to implement client-side scanning for one crime (like the U.S. forcing Apple to use NeuralHash for CSAM detection), any other state could conceivably direct the same company to use that type of tech to detect another crime (like the old East German Stasi using surveillance broadly to undermine dissent).
For instance, China could require Apple to use client-side scanning tech to scan for pictures of known democracy activists on the iCloud. Except it doesn’t have to, since the Chinese government already has access to Chinese Apple users’ data and encryption keys, so they can already run the Chinese iCloud against whatever databases they please. They have this access because they required Apple provide it as a condition of doing business in China, and doing business in China is a necessary condition for Apple as it exists today. Apple has already crossed this Rubicon. Far from being required by law to do it again in the West, it should be legally forbidden from undermining Western end-to-end encryption in Western security interests.
Engineering information security vulnerability is obviously bad for security in its own right. Can it even be explained without appearing tautological? Weakening security, weakens security. Undermining end-to-end encryption with client-side scanning creates structural conditions favorable for security breaches including mass surveillance.
That this winds up in public debates as a trade-off between security and liberty or privacy is a testament to the Orwellian nature of contemporary political discourse. There’s no proof that mass surveillance actually cashes this check — and plenty of reasons to suspect that we can have our civil liberties cake and eat it, too.
My favorite is Bayes’ Rule.
How it works
Bayes’ Rule, a maxim of probability theory, describes how the probability of an event changes depending on a subgroup; or, more broadly, how prior knowledge influences the interpretation of new evidence. It implies that mass screenings for low-prevalence problems will return overwhelmingly false positives, as the common overwhelms the rare.
When we can disambiguate true from false positives with high certainty and low costs — as in universal HIV or hepatitis screenings for pregnant women, even in places where these diseases are thankfully rare — that’s fine. But when we can’t know for sure that we’ve correctly sorted needles from haystacks, and/or the costs of trying to do so are quite high, that’s a problem.
Common cognitive biases tend to distort perceptions of programs of this structure. The base rate fallacy keeps people from seeing how the common overwhelms the rare; frequency-format outcome spreads can help mitigate this bias. In addition, reification can lead people to misperceive those outcomes as mapping neatly onto values like liberty and security — representing trade-offs that are not on the table. Communicating the myriad uncertainties packed into these estimates, and emphasizing ways in which costs and benefits to such values can accrue from across categories, may help combat this distortion.
These biases, however, tend to instead be compounded by perverse incentives shaping science communication. Far from being unique to this context, this problem pervades science, just as cognitive bias and psychosocial mooring pervade human consciousness. See, e.g., Sander Greenland’s Dec. 9, 2024 keynote at the International Biometric Conference “Toward Restoring Realism in Statistical Training and Practice,” exploring investigator and cognitive biases in science.
Overall, mass surveillance initiatives like Chat Control fall prey to common cognitive biases, threatening resource misallocation and other unintended consequences that undermine their stated goals.
Stupid or evil?
It’s hard to stop the train of mass surveillance.
But if Western-dominant U.S. and allied security agencies keep trying to get it out of stupidity — not understanding that it undermines security according to the universal laws of mathematics — then it should be possible for good science communication to sway them. It worked for public statistician Stephen Fienberg when he took on the Department of Energy, which planned to polygraph all scientists at the National Labs in the wake of the Wen Ho Lee spy scandal — not realizing that Bayes’ Rule implied an unacceptable choice between too many false positives or too many false negatives in such a program (see, e.g., footnote 9, Fienberg’s polygraph testimony story, and correction 3).
If, however, dark-money organizations keep trying to get it out of venality — calculating (for instance) that it would be politically profitable for the security state to have mass surveillance powers with which to spy on its opponents — then they may well eventually lie, bully, and pay their way to a world where it’s cheap and easy to grab those powers again. Modern history offers much pessimism about such totalitarian propensities.
Ultimately, the pursuit of mass surveillance through client-side scanning not only jeopardizes individual liberties, but also threatens security. Too often, we hear that we have an opportunity to trade some liberty for some security, when that trade is not on the table.
One of the worst parts of this myth is that its purveyors may know that it’s a myth. In the case of client-side scanning for CSAM, powerful social and political networks may manipulate child sexual abuse victims for their own purposes — replicating the index trauma of the powerful using the powerless for their own perverse purposes.
That’s not what justice looks like.
Science communication needed
Despite the implications of Bayes’ Rule in programs of this structure being broadly accepted in the medical and scientific community, the argument that mass surveillance undermines both liberty and security is largely missing in public discourse including media coverage of the Heat Initiative’s Apple lawsuit. Is it simply not widely known, outside scientific circles? Is it taboo to write it, because fear primes authoritarianism and it’s a questioning of illegitimate authority, or because no one wants to look like they’re on the side of child abusers? Or is something more nefarious going on?
In any case, the victims who are part of the lawsuit deserve to know that they’re participating in a political maneuver that risks undermining the very goal they prioritize — child protection. But the agents ostensibly representing their interests — like the law firm representing them in the Apple suit, and the organization that recruited and paid it — have perverse incentives to not inform them.
These incentives are substantial. For instance, the law firm stands to benefit monetarily from the lawsuit. The Heat Initiative and its funders stand to benefit politically.
But, since the victims are anonymous, it’s unclear who else might inform them about the likely effects of the proposed client-side scanning on child safety.
There are no neat solutions to this principal-agent problem. It’s not unique to this context. And neither being influenced by such perverse incentives, nor being affected by the base rate fallacy or other cognitive biases makes the people orchestrating the lawsuit bad people. They’re just people in all our pervasive vulnerability to cognitive bias and perspectival limitations.
There’s another layer of principal-agent problem here, too, that highlights this fact: Insofar as the victims represented in the suit may consider themselves agents of other child sexual abuse victims, present and future — they, too, are subject to perverse incentives (e.g., the large possible monetary compensation) and cognitive biases (e.g., the base rate fallacy).
Maybe Apple will manage to educate the public about statistics through the media, communicating the implications of probability theory to the courts through lawyers and expert witnesses, and to the victims through both the media and the courts. It’s a simple argument that suggests there is no genuine conflict of interest here, only a common set of misunderstandings about how to best go about trying to achieve agreed ends. But, as the old saw goes, it can be difficult to make a man understand something that his paycheck depends on him not understanding.
Principal-agent problem pyramid
Mass screenings for low-prevalence problems are often characterized by compounding principal-agent problems at various levels:
At the base level, the public interest (e.g., advancing shared ends like security or health) conflicts with guild (organized expert) interests (typically for more screenings/interventions), as well as with the self-interests interests of political actors (e.g., reelection for “doing something” — when doing nothing has a better cost-benefit profile).
Other conflicts of interest layer on top, with intra-organizational conflicts (e.g., between middle and upper management) in the middle…
… and conflicts of interest between…
classes of affected individuals and their agents (e.g., victims’ lawyers and victims themselves)…
… and each other (e.g., past and current victims) toward the top.
These are unsolvable problems. The state of the art is to recognize that perverse incentives can influence cognition, shaping motivated reasoning and generating spin science. Mitigation requires estimating the effects of these types of programs — something evidence-based policy should but seldom does actually require.
While programs of this structure (mass screenings for low-prevalence problems) in the realm of security are particularly dangerous to society, the structure is common across issue areas. That’s why society needs to legislate the structure instead of playing Whack-A-Mole, with critics exhausting their own finite resources fighting it program by program.
One possible way to do that in the EU involves specifying proportionality in terms of scientific evidentiary standards. In other words, requiring that there is some logically and statistically sound basis for arguments that policies work might help advance evidence-based policy. And that, more than arguing about whether security and liberty conflict, might actually best promote both.