larlet-fr-david-cache/cache/2021/afd74b165f837caf94dd2098d161627a/index.md

title: Apple’s photo scanning and our state of forced collective paranoia
url: https://interconnected.org/home/2021/08/06/paranoia
hash_url: afd74b165f837caf94dd2098d161627a

<p class="measure-wide f6 f5-l lh-copy black-80">Apple released its plans to <a href="https://www.apple.com/child-safety/">automatically scan phones for child abuse material</a> which on the face of it is good policing – and the response has been loud and angry and calls out the dangerous slippery slope of surveillance. But I think what is also being revealed is a particularly 21st century phenomenon, and that is <em>mass social paranoia.</em></p>
<p class="measure-wide f6 f5-l lh-copy black-80">The slippy slope argument is not hard to see. The plan is for Apple to continuously scan photos sent in messages and stored in iCloud, testing them against a known database of child abuse images, and escalating matching photos to human review.</p>
<p class="measure-wide f6 f5-l lh-copy black-80">But now the mechanism is in place, what else could it be used for? Could the Chinese government coerce Apple to locate dissidents, by adding certain non-child-abuse images to the central database? I mean, what are Apple going to do – not sell phones in China? Or can the GDPR “right to be forgotten” be wielded to force erasure of (say) unwisely shared nudes? Hard to argue with that, and Google hides links from search results under GDPR so maybe not such a stretch. But then why not automate removal of embarrassing photos of celebrities with expensive lawyers?</p>
<p class="measure-wide f6 f5-l lh-copy black-80">The EFF response is far more articulate on the details and <q>mission creep</q> potential of the new system: <a href="https://www.eff.org/deeplinks/2021/08/apples-plan-think-different-about-encryption-opens-backdoor-your-private-life">Apple’s Plan to “Think Different” About Encryption Opens a Backdoor to Your Private Life</a>.</p>
<p class="measure-wide f6 f5-l lh-copy black-80">BUT:</p>
<p class="measure-wide f6 f5-l lh-copy black-80">I have a friend who has worked in positions where she can see the global traffic of child exploitation material, and I’ve spoken with her just a little bit about this in the past. It is horrific and huge. We need good policing, that’s my view, and mechanisms to achieve that, and we can debate how that happens. <em>(We’re a long way from any answers, but my thoughts on a good approach are a whole other topic.)</em></p>
<p class="measure-wide f6 f5-l lh-copy black-80">So there’s a line for society to walk.</p>
<p class="measure-wide f6 f5-l lh-copy black-80">And it <em>really</em> doesn’t help that we have to trust a corporation to walk this line, without democratic accountability.</p>
<p class="measure-wide f6 f5-l lh-copy black-80">Yet this isn’t just a privacy debate.</p>
<p class="measure-wide f6 f5-l lh-copy black-80">It feels different because the photos are being scanned on-device. The surveillance is on our phones. And that triggers a whole other kind of response.</p>
<hr class="h1 xh2-ns w1 xw2-ns ml4 mv4 bb bw1 b--white">
<p class="measure-wide f6 f5-l lh-copy black-80">Cory Doctorow, way back in 2002: <a href="https://web.archive.org/web/20020802094408/http://www.oreillynet.com/pub/a/javascript/2002/01/01/cory.html">My Blog, My Outboard Brain</a> <em>(O’Reilly):</em> <q>Being deprived of my blog right now would be akin to suffering extensive brain-damage. Huge swaths of acquired knowledge would simply vanish.</q></p>
<p class="measure-wide f6 f5-l lh-copy black-80">Clive Thompson, in 2007: <a href="https://www.wired.com/2007/09/st-thompson-3/">Your Outboard Brain Knows All</a> <em>(Wired):</em> <q>This summer, neuroscientist Ian Robertson polled 3,000 people and found that the younger ones were less able than their elders to recall standard personal info.</q></p>
<p class="measure-wide f6 f5-l lh-copy black-80">This feels obvious now, but it was new then:</p>
<blockquote class="bl bw1 pl2 b--light-red ml0 italic i">
<p class="measure-wide f6 f5-l lh-copy black-80">And when he asked them their own phone number, fully one-third of the youngsters drew a blank. They had to whip out their handsets to look it up.</p>
</blockquote>
<p class="measure-wide f6 f5-l lh-copy black-80">So smartphones become, somehow, part of the mind.</p>
<p class="measure-wide f6 f5-l lh-copy black-80">Cognitive scientist Andy Clark makes the point that the mind doesn’t stop at the skull. He lays out the <em>extended mind</em> hypothesis in his astoundingly prescient book <a href="https://uk.bookshop.org/books/natural-born-cyborgs-minds-technologies-and-the-future-of-human-intelligence/9780195177510">Natural-Born Cyborgs</a> (2003). <em>Highly recommended.</em></p>
<p class="measure-wide f6 f5-l lh-copy black-80">He makes the argument that we don’t just <em>use</em> pen and paper to work out a sum, but the tool becomes part of our thinking. Information on the web isn’t just consulted on our phones, but is in a real way part of our memory. </p>
<p class="measure-wide f6 f5-l lh-copy black-80">Humans are special precisely because our brains have this ability to side-load the world into self:</p>
<blockquote cite="https://uk.bookshop.org/books/natural-born-cyborgs-minds-technologies-and-the-future-of-human-intelligence/9780195177510" class="quoteback bl bw1 pl2 b--light-red ml0 italic i" data-author="Andy Clarke" data-title="Natural Born Cyborgs (p198)">
<p class="measure-wide f6 f5-l lh-copy black-80">In embracing our hybrid natures, we give up the idea of the mind and the self as a kind of wafer-thin inner essence, dramatically distinct from all its physical trappings. In place of this elusive essence, the human person emerges as a shifting matrix of biological and nonbiological parts. The self, the mind, and the person are no more to be extracted from that complex matrix than the smile from the Cheshire Cat.</p>

</blockquote>
<p class="measure-wide f6 f5-l lh-copy black-80">Phones are part of us.</p>
<hr class="h1 xh2-ns w1 xw2-ns ml4 mv4 bb bw1 b--white">
<p class="measure-wide f6 f5-l lh-copy black-80">Scanning the photos on your phone isn’t like steaming open the mail and peeping inside the envelopes. It’s like rifling through your memory.</p>
<p class="measure-wide f6 f5-l lh-copy black-80">And when those memories may at any time be silently observed or removed… even if it never happens but there is the possibility of it…</p>
<p class="measure-wide f6 f5-l lh-copy black-80">Well.</p>
<hr class="h1 xh2-ns w1 xw2-ns ml4 mv4 bb bw1 b--white">
<p class="measure-wide f6 f5-l lh-copy black-80">Every culture, big and small, has a feeling that it swims in but is often slow to put its finger on, like the proverbial fish in the ocean unable to see the water. That’s my take.</p>
<p class="measure-wide f6 f5-l lh-copy black-80">I think in the 70s and 80s that feeling was the end of the world. I was pretty sure, as a little kid, that by the time I was my age, now, I would be living in a post-apocalyptic nuclear wasteland. It wasn’t a conviction, it was more like an unspoken understanding. And goodness knows what that did to us.</p>
<p class="measure-wide f6 f5-l lh-copy black-80">Ironically the end of the world <em>is</em> coming, in the shape of the climate crisis, and I wonder how those of us who grew up taking the Cold War for granted are coloured by that experience and how it is tainting our response. We probably feel like the climate crisis, or at least some kind of apocalypse is inevitable somehow? Or alternatively, that if we wait around for long enough then the threat will just somehow… recede? Like the way the peril lifted in the 90s. Dangerous templating for us to have; thank god for the zoomers.</p>
<p class="measure-wide f6 f5-l lh-copy black-80">What’s in the air now?</p>
<p class="measure-wide f6 f5-l lh-copy black-80">We swim in paranoia, I think.</p>
<p class="measure-wide f6 f5-l lh-copy black-80">We’re always potentially being watched.</p>
<hr class="h1 xh2-ns w1 xw2-ns ml4 mv4 bb bw1 b--white">
<p class="measure-wide f6 f5-l lh-copy black-80">RELATED: I ran across <a href="https://youtu.be/MXumVxdfbU4">Zizek riffing on Donald Rumsfeld</a> <em>(YouTube)</em> and specifically developing the concept of <strong>unknown knowns.</strong> Here’s Ted Hunt on Twitter with a quote/summary:</p>
<blockquote cite="https://twitter.com/_ted_hunt/status/1422458768399421442?s=21" class="quoteback bl bw1 pl2 b--light-red ml0 italic i" data-author="Ted Hunt (_@ted_hunt)" data-title="08:26, 03/08/2021">
<p class="measure-wide f6 f5-l lh-copy black-80">”.. the main dangers lie in the unknown knowns–the disavowed beliefs, suppositions and obscene practices we pretend not to know about, even though they form the background of our public values.” – Slavoj Zizek</p>

</blockquote>
<p class="measure-wide f6 f5-l lh-copy black-80">So paranoia is like our culture’s current unknown known. That’s where it sits, somewhere in the social unconscious.</p>
<hr class="h1 xh2-ns w1 xw2-ns ml4 mv4 bb bw1 b--white">
<p class="measure-wide f6 f5-l lh-copy black-80">James Bridle’s 2014 work <strong>The Nor</strong> was <q>an investigation into paranoia, electromagnetism, and infrastructure.</q></p>
<p class="measure-wide f6 f5-l lh-copy black-80">It’s a sequence of essays telling the story of a participatory, documentary act: Bridle’s walk across London, photographing every CCTV camera he passed. SPOILER: It doesn’t end well.</p>
<p class="measure-wide f6 f5-l lh-copy black-80"><a href="https://jamesbridle.com/works/the-nor">Here are the essays:</a></p>
<blockquote cite="https://jamesbridle.com/works/the-nor" class="quoteback bl bw1 pl2 b--light-red ml0 italic i" data-author="James Bridle" data-title="The Nor (2014)">
<p class="measure-wide f6 f5-l lh-copy black-80">The sense of being watched is a classic symptom of paranoia, often a sign of deeper psychosis, or dismissed as illusory. In the mirror city, which exists at the juncture of the street and CCTV, of bodily space and the electromagnetic spectrum, one is always being watched. So who’s paranoid now?</p>

</blockquote>
<p class="measure-wide f6 f5-l lh-copy black-80">And it was this work that really opened my eyes to the pervasive sensation of surveillance. (Which is why art is vital, right?) Especially because Bridle makes explicit the role of the <em>network</em> and what that does: the first essay is titled <em>All Cameras are Police Cameras.</em></p>
<p class="measure-wide f6 f5-l lh-copy black-80">The camera network today is Instagram, TikTok, other people’s phones. It’s the pictures taken at parties, previously private spaces, and it’s the acquisition of the breakthrough facial recognition startup <a href="https://en.wikipedia.org/wiki/Face.com">Face.com</a> by Facebook in 2012, and everything that opened a door to across the industry.</p>
<p class="measure-wide f6 f5-l lh-copy black-80">A lot has been said about the <a href="https://en.wikipedia.org/wiki/Panopticon">Panopticon</a>, Jeremy Bentham’s 1786 concept of a prison where the prisoners are controlled by the mere <em>possibility</em> of being observed, and of <a href="https://en.wikipedia.org/wiki/Sousveillance">sousveillance</a>: surveillance from the same level; we watch one-another. That’s what a networked camera in every pocket leads to.</p>
<p class="measure-wide f6 f5-l lh-copy black-80">The debate, over the last 20 years as this has been happening, has been framed around the loss of privacy and whether that matters: the younger generation has different privacy expectations to us, that’s one statement; the absolutist privacy ideals of the EFF are another part of the debate.</p>
<p class="measure-wide f6 f5-l lh-copy black-80">(And the responses to this shift are fascinating. For me, the go-to here is danah boyd’s work, and I’ve recently been diving into <a href="https://www.danah.org/papers/talks/2011/PDF2011.html">her work on networked privacy</a> from the early 2010s, and the sophisticated ways that teens are finding control and agency in this world.)</p>
<p class="measure-wide f6 f5-l lh-copy black-80"><em>But how does it feel?</em></p>
<p class="measure-wide f6 f5-l lh-copy black-80">It feels like paranoia. You don’t know how the image of you has spread, or your words passed on. You don’t know how it will be interpreted; you don’t know if you’re going to wake up one morning in the middle of a context collapse Twitter pile-on – or be fine as normal – or arrested by the police.</p>
<hr class="h1 xh2-ns w1 xw2-ns ml4 mv4 bb bw1 b--white">
<p class="measure-wide f6 f5-l lh-copy black-80">ASIDE, just to say that Covid-19 is a very 2020s disease, very paranoid.</p>
<p class="measure-wide f6 f5-l lh-copy black-80">Unlike the Blitz in London in the Second World War where the risk was external, and everyone has to pull together. (I reference this simply because it’s the event which is also mentioned here in the UK whenever there’s a new national crisis.) Everyone <em>could</em> pull together because everyone could be trusted. All in the same boat.</p>
<p class="measure-wide f6 f5-l lh-copy black-80">But with Covid…</p>
<p class="measure-wide f6 f5-l lh-copy black-80">Anyone you meet may be infectious. Or not. There’s a risk in every interaction that, later, you find out they have “betrayed” you. Further, there’s a risk that you, yourself, may have Covid. You may be spreading it, infecting your neighbours, your parents – you can unknowingly betray yourself.</p>
<p class="measure-wide f6 f5-l lh-copy black-80">So there’s this questioning of self and one-another, and we’ve responded with surveillance and sousveillance: we continuously monitor one-another with contact tracing apps, ourselves with self-administered tests. We’re reminded to be suspicious.</p>
<p class="measure-wide f6 f5-l lh-copy black-80">This uncertainty about self and other is so similar to social media. When you talk to people online, are they really people or are they bots? Are they stealing your data? Have you exposed yourself, given yourself away? Are you, yourself, tainted – have you fallen into a Facebook rabbit hole and been radicalised… how would you know? Is there a home-administered lateral flow test for extremism?</p>
<p class="measure-wide f6 f5-l lh-copy black-80">I am <em>not</em> saying that the Covid response is inappropriate.</p>
<p class="measure-wide f6 f5-l lh-copy black-80">But what I am saying is that the mutual suspicion and monitoring is (looking at it with this particular framing) a forced paranoid state, which is very in keeping with social media and networked technology.</p>
<p class="measure-wide f6 f5-l lh-copy black-80">And it would be interesting to consider how we would have tackled Covid if we had instead a different dominant social scaffolding to conceptualise “threat,” for example if we had still been in the tail end of the Cold War.</p>
<hr class="h1 xh2-ns w1 xw2-ns ml4 mv4 bb bw1 b--white">
<p class="measure-wide f6 f5-l lh-copy black-80">Credit to the current generation, they are responding to this paranoid milieu of the 2010s/2020s and developing new language to point at it and discuss it.</p>
<p class="measure-wide f6 f5-l lh-copy black-80">The emergence of the term <em>gaslighting</em> has been a joy to see: this new ability to discern when memory is being undermined for the purposes of manipulation and control – well, that’s a word we all needed and thank you.</p>
<p class="measure-wide f6 f5-l lh-copy black-80">Jumping to <a href="https://www.verywellmind.com/is-someone-gaslighting-you-4147470">a definition</a> for a second:</p>
<blockquote class="bl bw1 pl2 b--light-red ml0 italic i">
<p class="measure-wide f6 f5-l lh-copy black-80">Gaslighting is a technique that undermines your entire perception of reality. When someone is gaslighting you, you often second-guess yourself, your memories, and your perceptions.</p>
</blockquote>
<hr class="h1 xh2-ns w1 xw2-ns ml4 mv4 bb bw1 b--white">
<p class="measure-wide f6 f5-l lh-copy black-80">Let me bring this back to Apple, and why I think the initial response to their child abuse material scanning announcement has been so angry and so strong.</p>
<p class="measure-wide f6 f5-l lh-copy black-80">Our phones aren’t computers. They are our outboard brains. Our photos aren’t simply stored; they are part of our memory.</p>
<p class="measure-wide f6 f5-l lh-copy black-80">We live in a state of forced paranoia, developed over the last almost twenty years. We don’t know who’s watching or what will be done with this. But we’ve found accommodations. We’ve managed. We have new language to talk about it.</p>
<p class="measure-wide f6 f5-l lh-copy black-80">Except now somebody <em>is</em> proposing to look at our memories. We won’t feel anything; we won’t hear anything; probably nothing will happen. We all know from previous experiences with algorithms that misinterpretations will happen. And of course there are human monitors involved too, which means we have to consider, at some level, what they will think of us. So now we have to police ourselves, just in case we take a photo of - have a memory of - happen to think the wrong thing.</p>
<p class="measure-wide f6 f5-l lh-copy black-80">And if somebody else is now inside your memories, can you be sure that they’re not being edited? Is gaslighting occuring with these most personal of devices? Even if it never happens… that’s the lesson of the Panopticon, the mere possibility is enough to affect behaviour.</p>
<p class="measure-wide f6 f5-l lh-copy black-80">What the word for paranoia when it’s true?</p>
<hr class="h1 xh2-ns w1 xw2-ns ml4 mv4 bb bw1 b--white">
<p class="measure-wide f6 f5-l lh-copy black-80">Covid, phone surveillance, social media, mass paranoia – all of these are of a type and in resonance; nonlinear sympathetic consequences are kicking off all over the place.</p>
<p class="measure-wide f6 f5-l lh-copy black-80">I don’t know what should be done, what the rights and wrongs are here.</p>
<p class="measure-wide f6 f5-l lh-copy black-80">But I wanted to make the connection.</p>