larlet-fr-david-cache/cache/f98974457e4d16ef6940e03b1e6ee58b/index.md

title: The Hidden Costs of Automated Thinking
url: https://www.newyorker.com/tech/annals-of-technology/the-hidden-costs-of-automated-thinking
hash_url: f98974457e4d16ef6940e03b1e6ee58b

<div class="SectionBreak SectionBreak__sectionBreak___1ppA7"><p>Like <span data-page="page_1"/>many medications, the wakefulness drug modafinil, which is marketed under the trade name Provigil, comes with a small, tightly folded paper pamphlet. For the most part, its contents—lists of instructions and precautions, a diagram of the drug’s molecular structure—make for anodyne reading. The subsection called “Mechanism of Action,” however, <a class="ArticleBody__link___1FS03" href="https://www.rxlist.com/provigil-drug.htm#indications" target="_blank">contains a sentence</a> that might induce sleeplessness by itself: “The mechanism(s) through which modafinil promotes wakefulness is unknown.”</p><p>Provigil isn’t uniquely mysterious. Many drugs receive regulatory approval, and are widely prescribed, even though no one knows exactly how they work. This mystery is built into the process of drug discovery, which often proceeds by trial and error. Each year, any number of new substances are tested in cultured cells or animals; the best and safest of those are tried out in people. In some cases, the success of a drug promptly inspires new research that ends up explaining how it works—but not always. Aspirin was discovered in 1897, and yet no one convincingly explained how it worked <a class="ArticleBody__link___1FS03" href="http://chronicle.uchicago.edu/950817/aspirin.shtml" target="_blank">until 1995</a>. The same phenomenon exists elsewhere in medicine. <a class="ArticleBody__link___1FS03" href="https://www.thelancet.com/journals/lanpsy/article/PIIS2215-0366(15)00206-0/fulltext" target="_blank">Deep-brain stimulation</a> involves the implantation of electrodes in the brains of people who suffer from specific movement disorders, such as Parkinson’s disease; it’s been in widespread use for more than twenty years, and some think it should be employed for other purposes, including general cognitive enhancement. No one can say how it works.</p><p>This approach to discovery—answers first, explanations later—accrues what I call intellectual debt. It’s possible to discover what works without knowing why it works, and then to put that insight to use immediately, assuming that the underlying mechanism will be figured out later. In some cases, we pay off this intellectual debt quickly. But, in others, we let it compound, relying, for decades, on knowledge that’s not fully known.</p></div><div class="SectionBreak SectionBreak__sectionBreak___1ppA7"><p>In the past, intellectual debt has been confined to a few areas amenable to trial-and-error discovery, such as medicine. But that may be changing, as new techniques in <a class="ArticleBody__link___1FS03" href="https://www.newyorker.com/tag/artificial-intelligence">artificial intelligence</a>—specifically, machine learning—increase our collective intellectual credit line. Machine-learning systems work by identifying patterns in oceans of data. Using those patterns, they hazard answers to fuzzy, open-ended questions. Provide a neural network with labelled pictures of cats and other, non-feline objects, and it will learn to distinguish cats from everything else; give it access to medical records, and it can attempt to predict a new hospital patient’s <a class="ArticleBody__link___1FS03" href="https://www.ncbi.nlm.nih.gov/pubmed/28092203" target="_blank">likelihood of dying</a>. And yet, most machine-learning systems don’t uncover causal mechanisms. They are statistical-correlation engines. They can’t explain why they think some patients are more likely to die, because they don’t “think” in any colloquial sense of the word—they only answer. As we begin to integrate their insights into our lives, we will, collectively, begin to rack up more and more intellectual debt.</p><p>Theory-free advances in pharmaceuticals show us that, in some cases, intellectual debt can be indispensable. Millions of lives have been saved on the basis of interventions that we fundamentally do not understand, and we are the better for it. Few would refuse to take a life-saving drug—or, for that matter, aspirin—simply because no one knows how it works. But the accrual of intellectual debt has downsides. As drugs with unknown mechanisms of action proliferate, the number of tests required to uncover untoward interactions must scale exponentially. (If the principles by which the drugs worked were understood, bad interactions could be predicted in advance.) In practice, therefore, interactions are discovered after new drugs are on the market, contributing to a cycle in which drugs are introduced, then abandoned, with class-action lawsuits in between. In each individual case, accruing the intellectual debt associated with a new drug may be a reasonable idea. But intellectual debts don’t exist in isolation. Answers without theory, found and deployed in different areas, can complicate one another in unpredictable ways.</p><p>Intellectual debt accrued through machine learning features risks beyond the ones created through old-style trial and error. Because most machine-learning models cannot offer reasons for their ongoing judgments, there is no way to tell when they’ve misfired if one doesn’t already have an independent judgment about the answers they provide. Misfires can be rare in a well-trained system. But they can also be triggered intentionally by someone who knows just what kind of data to feed into that system.</p>
<p>Consider image recognition. Ten years ago, computers couldn’t easily identify objects in photos. Today, image search engines, like so many of the systems we interact with on a day-to-day basis, are based on extraordinarily capable machine-learning models. Google’s image search relies on a neural network called <a class="ArticleBody__link___1FS03" href="https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Szegedy_Rethinking_the_Inception_CVPR_2016_paper.pdf" target="_blank">Inception</a>. In 2017, M.I.T.’s <a class="ArticleBody__link___1FS03" href="https://www.labsix.org/" target="_blank">LabSix</a>—a research group of undergraduate and graduate students—succeeded in <a class="ArticleBody__link___1FS03" href="https://www.labsix.org/physical-objects-that-fool-neural-nets/" target="_blank">altering the pixels of a photograph of a cat</a> so that, although it looked like a cat to human eyes, Inception became 99.99-per-cent sure it had been given a photograph of guacamole. (There was, it calculated, a slim chance that the photograph showed broccoli, or mortar.) Inception, of course, can’t explain what features led it to conclude that a cat is a cat; as a result, there’s no easy way to predict how it might fail when presented with specially crafted or corrupted data. Such a system is likely to have unknown gaps in its accuracy that amount to vulnerabilities for a smart and determined attacker.</p><p>As knowledge generated by machine-learning systems is put to use, these kinds of gaps may prove consequential. Health-care A.I.s have been <a class="ArticleBody__link___1FS03" href="https://www.nature.com/articles/nature21056" target="_blank">successfully trained</a> to<span data-page="page_2"/> classify skin lesions as benign or malignant. And yet—as a team of researchers from Harvard Medical School and M.I.T. showed, in a <a class="ArticleBody__link___1FS03" href="https://arxiv.org/pdf/1804.05296.pdf" target="_blank">paper</a> published this year—they can also be tricked into making inaccurate judgments using the same techniques that turn cats into guacamole. (Among other things, attackers might use these vulnerabilities to commit <a class="ArticleBody__link___1FS03" href="https://science.sciencemag.org/content/363/6433/1287" target="_blank">insurance fraud</a>.) Seduced by the predictive power of such systems, we may stand down the human judges whom they promise to replace. But they will remain susceptible to hijacking—and we will have no easy process for validating the answers they continue to produce.</p></div><div class="SectionBreak SectionBreak__sectionBreak___1ppA7"><p>Could we create a balance sheet for intellectual debt—a system for tracking where and how theoryless knowledge is used? Our accounting could reflect the fact that not all intellectual debt is equally problematic. If an A.I. produces <a class="ArticleBody__link___1FS03" href="https://howtogeneratealmostanything.com/food/2018/08/30/episode2.html" target="_blank">new pizza recipes</a>, it may make sense to shut up and enjoy the pizza; by contrast, when we begin using A.I. to make health predictions and recommendations, we’ll want to be fully informed.</p><p>Building and maintaining a society-wide intellectual-debt balance sheet would probably require refining our approach to trade secrets and other intellectual property. In cities, building codes ask building owners to publicly disclose their renovation plans. Similarly, we might explore asking libraries or universities to accept, in escrow, otherwise hidden data sets and algorithms that have found a certain measure of public use. That would allow researchers to begin probing the models and underlying data on which we’re coming to depend, and—by building theories—make payments on our intellectual debt before it becomes due in the form of errors and vulnerabilities.</p><p>The growing pervasiveness of machine-learning models, and the fact that anyone can create one, promise to make this process of accounting difficult. But it’s vital. Taken in isolation, oracular answers can generate consistently helpful results. But these systems won’t stay in isolation: as A.I.s gather and ingest the world’s data, they’ll produce data of their own—much of which will be taken up by still other systems. Just as drugs with unknown mechanisms of action sometimes interact, so, too, will debt-laden algorithms.</p><p>Even simple interactions can lead to trouble. In 2011, a biologist named Michael Eisen found out, from one of his students, that the least-expensive copy of an otherwise unremarkable used book—“The Making of a Fly: The Genetics of Animal Design”—was available on Amazon for $1.7 million, plus $3.99 shipping. The second-cheapest copy cost $2.1 million. The respective sellers were well established, with thousands of positive reviews between them. When Eisen visited the book’s Amazon page several days in a row, he discovered that the prices were increasing continually, in a regular pattern. Seller A’s price was consistently 99.83 per cent that of Seller B; Seller B’s price was reset, every day, to 127.059 per cent of Seller A’s. Eisen <a class="ArticleBody__link___1FS03" href="http://www.michaeleisen.org/blog/?p=358" target="_blank">surmised</a> that Seller A had a copy of the book, and was seeking to undercut the next-cheapest price. Seller B, meanwhile, didn’t have a copy, and so priced the book higher; if someone purchased it, B could order it, on that customer’s behalf, from A.</p><p>Each seller’s presumed strategy was rational. It was the interaction of their algorithms that produced irrational results. The interaction of thousands of machine-learning systems in the wild promises to be much more unpredictable. The financial markets, where cutting-edge machine-learning systems are already being deployed, provide an obvious breeding ground for this type of problem. In 2010, for a harrowing thirty-six minutes, a “flash crash” driven by algorithmic trading wiped more than a trillion dollars from the major U.S. indices. Last fall, the J. P. Morgan analyst Marko Kolanovic <a class="ArticleBody__link___1FS03" href="https://www.washingtonpost.com/business/economy/the-warning-from-jpmorgan-about-flash-crashes-ahead/2018/09/05/25b1f90a-b148-11e8-a20b-5f4f84429666_story.html" target="_blank">argued</a> that such a crash could easily happen again, since more trading than ever is based on automated systems. Intellectual debt can accumulate in the interstices where systems bump into each other, even when they don’t formally interconnect. Without anything resembling a balance sheet, there’s no way to determine—either in advance or retrospectively—whether any particular quantity of intellectual debt is worth taking on.</p></div><div class="SectionBreak SectionBreak__sectionBreak___1ppA7"><p>The increase in our intellectual debt may also involve a shift in the way we think—away from basic science and <a class="ArticleBody__link___1FS03" href="https://www.nytimes.com/2014/02/17/world/americas/dispute-over-the-future-of-basic-research-in-canada.html" target="_blank">toward applied technology</a>. Unlike, say, particle accelerators—massive capital projects which are supported by consortia of wealthy governments and run by academic research institutions—the tools of machine learning are as readily taken up by private industry as by academia. In fact, the sorts of data that produce useful predictions may be more readily available to Google and Facebook than to any computer-science or statistics department. Businesspeople may be perfectly satisfied by such unexplained knowledge, but intellectual debt will still be building. It will be held by corporations, far from the academic researchers who might be most interested in paying it down.</p><p>It’s easy to imagine that the availability of machine-learning-based knowledge will shift funding away from researchers who insist on the longer route of trying to figure things out for themselves. This past December, Mohammed AlQuraishi, a researcher who studies protein folding, <a class="ArticleBody__link___1FS03" href="https://moalquraishi.wordpress.com/2018/12/09/alphafold-casp13-what-just-happened/#comment-26005" target="_blank">wrote an essay</a> exploring a recent development in his field: the creation of a machine-learning model that can predict protein folds far more accurately than human researchers. AlQuiraishi found himself lamenting the loss of theory over data, even as he sought to reconcile himself to it. “There’s far less prestige associated with conceptual papers or papers that provide some new analytical insight,” he said, <a class="ArticleBody__link___1FS03" href="https://www.vox.com/future-perfect/2019/2/15/18226493/deepmind-alphafold-artificial-intelligence-protein-folding" target="_blank">in an interview</a>. As machines make discovery faster, people may come to see<span data-page="page_final"/> theoreticians as extraneous, superfluous, and hopelessly behind the times. Knowledge about a particular area will be less treasured than expertise in the creation of machine-learning models that produce answers on that subject.</p><p>Financial debt shifts control—from borrower to lender, and from future to past. Mounting intellectual debt may shift control, too. A world of knowledge without understanding becomes a world without discernible cause and effect, in which we grow dependent on our digital concierges to tell us what to do and when. It’s easy to imagine, for example, how a college-admissions committee might turn the laborious and uncertain sifting of applicants over to a machine-learning model; such a model might purport to optimize an entering cohort not just for academic success but also for harmonious relationships and generous alumni donations. The only way to make sense of this world might be to employ our own A.I.s—neural nets that fine-tune our social-media profiles so that we seem like we’ll fit perfectly into the freshman class.</p><p>Perhaps all this technology will work—and that, in turn, will be a problem. Much of the timely criticism of artificial intelligence has rightly focussed on the ways in which it can go wrong: it can create or replicate bias; it can make mistakes; it can be put to evil ends. We should also worry, though, about what will happen when A.I. gets it right.</p></div>