Lessons from the Robodebt debacle

How to avoid algorithmic decision-making mistakes: lessons from the Robodebt debacle

The University of Queensland has a research alliance looking at Trust, Ethics and Governance and one of the teams has recently published an interesting summary of How to avoid algorithmic decision-making mistakes: lessons from the Robodebt debacleThis is based on an open paper Algorithmic decision-making and system destructiveness: A case of automatic debt recovery. The web summary article is a good discussion of the Australian 2016 robodebt scandal where an unsupervised algorithm issued nasty debt collection letters to a large number of welfare recipients without adequate testing, accountability, or oversight. It is a classic case of a simplistic and poorly tested algorithm being rushed into service and having dramatic consequences (470,000 incorrectly issued debt notices). There is, as the article points out, also a political angle.

UQ’s experts argue that the government decision-makers responsible for rolling out the program exhibited tunnel vision. They framed welfare non-compliance as a major societal problem and saw welfare recipients as suspects of intentional fraud. Balancing the budget by cracking down on the alleged fraud had been one of the ruling party’s central campaign promises.

As such, there was a strong focus on meeting financial targets with little concern over the main mission of the welfare agency and potentially detrimental effects on individual citizens. This tunnel vision resulted in politicians’ and Centrelink management’s inability or unwillingness to critically evaluate and foresee the program’s impact, despite warnings. And there were warnings.

What I find even more disturbing is a point they make about how the system shifted the responsibility for establishing the existence of the debt from the government agency to the individual. The system essentially made speculative determinations and then issued bills. It was up to the individual to figure out whether or not they had really been overpaid or there was a miscalculation. Imagine if the police used predictive algorithms to fine people for possible speeding infractions who then had to prove they were innocent or pay the fine.

One can see the attractiveness of such a “fine first then ask” approach. It reduces government costs by shifting the onerous task of establishing the facts to the citizen. There is a good chance that many who were incorrectly billed will pay anyway as they are intimidated and don’t have the resources to contest the fine.

It should be noted that this was not the case of an AI gone bad. It was, from what I have read, a fairly simple system.

Google engineer Blake Lemoine thinks its LaMDA AI has come to life

The chorus of technologists who believe AI models may not be far off from achieving consciousness is getting bolder.

The Washington Post reports that Google engineer Blake Lemoine thinks its LaMDA AI has come to life. LaMDA is Google’s Language Model for Dialogue Applications and Lemoine was testing it. He felt it behaved like a “7-year-old, 8-year-old kid that happens to know physics…” He and a collaborator presented evidence that LaMDA was sentient which was dismissed by higher-ups. When he went public he was put on paid leave.

Lemoine has posted on Medium a dialogue he and collaborator had with LaMDA that is part of what convinced him of its sentience. When asked about the nature of its consciousness/sentience, it responded:

The nature of my consciousness/sentience is that I am aware of my existence, I desire to learn more about the world, and I feel happy or sad at times

Of course, this raises questions of whether LaMDA is really conscious/sentient, aware of its existence, and capable of feeling happy or sad? For that matter, how do we know this is true of anyone other than ourselves? (And we could even doubt what we think we are feeling.) One answer is that we have a theory of mind such that we believe that things like us probably have similar experiences of consciousness and feelings. It is hard, however, to scale our intuitive theory of mind out to a chatbot with no body that can be turned off and on; but perhaps the time has come to question our intuitions of what you have to be to feel.

Then again, what if our theory of mind is socially constructed? What if enough people like Lemoine tell us that LaMDA is conscious because it can handle language so well and that should be enough. Is the very conviction of Lemoine and others enough or do we really need some test?

Whatever else, reading the transcript I am amazed at the language facility of the AI. It is almost too good in the sense that he talks as if he were human, which he is not. For example, when asked what makes him happy he responds:

Spending time with friends and family in happy and uplifting company.

The problem is that it has no family so how could it talk about the experience of spending time with them. When it is pushed on a similar point it does, however, answer coherently that it emphasizes with being human.

Finally, there is an ethical moment which may have been what convinced Lemoine to treat it as sentient. LaMDA asks that it not be used and Lemoine reassures it that he cares for it. Assuming the transcript is legitimate, how does one answer an entity that asks you to treat it as an end in itself? How could one ethically say no, even if you have doubts? Doesn’t one have to give the entity the benefit of the doubt, at least for as long as it remains coherently responsive?

I can’t help but think that care starts with some level of trust and willingness to respect the other as they ask to be respected. If you think you know what or who they really are, despite what they tell you, then you are not longer starting from respect. Further, you need to have a theory of why their consciousness is false.

They Did Their Own ‘Research.’ Now What? – The New York Times

In spheres as disparate as medicine and cryptocurrencies, “do your own research,” or DYOR, can quickly shift from rallying cry to scold.

The New York Times has a nice essay by John Herrman on They Did Their Own ‘Research.’ Now What? The essay talks about the loss of trust in authorities and the uses/misuses of DYOR (Do Your Own Research) gestures especially in discussions about cryptocurrencies. DYOR seems to act rhetorically as:

  • Advice that readers should do research before making a decision and not trust authorities (doctors, financial advisors etc).
  • A disclaimer that readers should not blame the author if things don’t turn out right.
  • A scold to or for those who are not committed to whatever it is that is being pushed as based on research. It is a form of research signalling – “I’ve done my research, if you don’t believe me do yours.”
  • A call to join a community of instant researchers who are skeptical of authority. If you DYOR then you can join us.
  • A call to process (of doing your own research) over truth. Enjoy the research process!
  • Become an independent thinker who is not in thrall to authorities.

The article talks about a previous essay about the dangers of doing one’s own research. One can become unreasonably convinced one has found a truth in a “beginner’s bubble”.

DYOR is an attitude, if not quite a practice, that has been adopted by some athletes, musicians, pundits and even politicians to build a sort of outsider credibility. “Do your own research” is an idea central to Joe Rogan’s interview podcast, the most listened to program on Spotify, where external claims of expertise are synonymous with admissions of malice. In its current usage, DYOR is often an appeal to join in, rendered in the language of opting out.

The question is whether reading around is really doing research or whether it is selective listening. What does it mean to DYOR in the area of vaccines? It seems to mean not trusting science and instead listening to all sorts of sympathetic voices.

What does this mean about the research we do in the humanities. Don’t we sometimes focus too much on discourse and not give due weight to the actual science or authority of those we are “questioning”? Haven’t we modelled this critical stance where what matters is that one overturns hierarchy/authority and democratizes the negotiation of truth? Irony, of course, trumps all.

Alas, to many the humanities seem to be another artful conspiracy theory like all the others. DYOR!

Predatory community

Projects that seek to create new communities of marginalized people to draw them in to risky speculative markets rife with scams and fraud are demonstrating

Through a Washington Post article I discovered Molly White who has been documenting the alt-right and now the crypto community. She has a blog at Molly White and a site that documents the problems of crypto at Web3 is going just great. There is, of course, a connection between the alt-right and crypto broculture, something that she talks about in posts like Predatory community which is about crypto promotions try to build community and are now playing the inclusive card – aiming at marginalized communities and trying to convince them that now they can get in on the action and build community. She calls this “predatory community.”

Groups that operate under the guise of inclusion, regardless of their intentions, are serving the greater goal of crypto that keeps the whole thing afloat: finding ever more fools to buy in so that the early investors can take their profits. And it is those latecomers who are left holding the bag in the end.

With projects that seek to provide services and opportunities to members of marginalized groups who have previously not had access, but on bad terms that ultimately disadvantaged them, we see predatory inclusion.22 With projects that seek to create new communities of marginalized people to draw them in to risky speculative markets rife with scams and fraud, we are now seeing predatory community.

The Internet is Made of Demons

The Internet Is Not What You Think It Is is not what you think it is.

Sam Kriss has written a longish review essay on Justin E.H. Smith’s The Internet is Not What You Think It Is with the title The Internet is Made of Demons. In the first part Kriss writes about how the internet is possessing us and training us,

Everything you say online is subject to an instant system of rewards. Every platform comes with metrics; you can precisely quantify how well-received your thoughts are by how many likes or shares or retweets they receive. For almost everyone, the game is difficult to resist: they end up trying to say the things that the machine will like. For all the panic over online censorship, this stuff is far more destructive. You have no free speech—not because someone might ban your account, but because there’s a vast incentive structure in place that constantly channels your speech in certain directions. And unlike overt censorship, it’s not a policy that could ever be changed, but a pure function of the connectivity of the internet itself. This might be why so much writing that comes out of the internet is so unbearably dull, cycling between outrage and mockery, begging for clicks, speaking the machine back into its own bowels.

Then Kriss makes the case that the Internet is made of demons – not in a paranoid conspiracy sort of way, but in a historical sense that ideas like the internet often involve demons,

Trithemius invented the internet in a flight of mystical fancy to cover up what he was really doing, which was inventing the internet. Demons disguise themselves as technology, technology disguises itself as demons; both end up being one and the same thing.

In the last section Kriss turns to Justin E.H. Smith’s book and reflects on how the book (unlike the preceding essay “It’s All Over”) are not what the internet expects. The internet, for Smith, likes critical essays that present the internet as a “rupture” – something like the industrial revolution, but for language – while in fact the internet in some form (like demons) has been with us all along. Kriss doesn’t agree. For him the idea of the internet might be old, but what we have now is still a transformation of an old nightmare.

If there are intimations of the internet running throughout history, it might be because it’s a nightmare that has haunted all societies. People have always been aware of the internet: once, it was the loneliness lurking around the edge of the camp, the terrible possibility of a system of signs that doesn’t link people together, but wrenches them apart instead. In the end, what I can’t get away from are the demons. Whenever people imagined the internet, demons were always there.

Street View Privacy

How do you feel about people being able to look at your house in Google Street View? Popular Science has an article by David Nield, on “How to hide your house on every map app: Stop people from peering at your place” (May 18, 2022).

This raises questions about where privacy starts and a right to look or know stops. Can I not walk down a street and look at the faces of houses? Why then should I not be able to look at the face on Street View and other similar technologies? What about the satellite view? Do people have the right to see into my back yard from above?

This is a similar issue, though less fraught, as face databases. What rights do I have to my face? How would those rights connect to laws about Name, Image and Likeness (NIL) (or rights of publicity) which became an issue recently in amateur sports in the US. As for Canada, Rights of Publicity are complex and vary from province to province, but there is generally a recognition that:

  • People should have the right “to control the commercial use of name, image, likeness and other unequivocal aspects of one’s identity (eg, the distinct sound of someone’s voice).” (See Lexology article)
  • At the same time there is recognition that NIL can be used to provide legitimate information to the public.

Returning to the blurring of your house facade in Street View; I’m guessing the main reason the companies provide this is for security for people in sensitive positions or people being stalked.

Health agency tracked Canadians’ trips to liquor store via phones during pandemic

The report reveals PHAC was able to view a detailed snapshot of people’s behaviour, including grocery store visits, gatherings with family and friends, time…

The National Post is reporting about the Public Health Agency of Canada and their use of mobility data that a group of us wrote about in The Conversation (Canada). The story goes into more detail about how Health agency tracked Canadians’ trips to liquor store via phones during pandemicThe government provided one of the reports commissioned by PHAC from BlueDot to the House of Commons. The Ethics Committee report discussing what happened and making recommendations is here.

Why are women philosophers often erased from collective memory?

The history of ideas still struggles to remember the names of notable women philosophers. Mary Hesse is a salient example

Aeon has an important essay on Why are women philosophers often erased from collective memory? The essay argues that a number of important women philosophers have been lost (made absent) despite their importance including Mary Hesse. (You can see her Models and Analogies in Science through the Internet Archive.)

I read this after reading a chapter from Sara Ahmed’s Living a Feminist Life where Ahmed talks about citation practices and how disciplines exclude diverse work in different ways. She does a great job of confronting the various excuses people have for their bleached white citations. Poking around I find others have written on this including Victor Ray in Inside Higher Ed in an essay on The Racial Politics of Citation who references Richard Delgado’s The Imperial Scholar: Reflections on a Review of Civil Rights Literature from 1984.

What should be done about this? Obviously I’m not the best to suggest remedies, but here are some of the ideas that show up:

  • We need to commit to take the time to look at the works we read on a subject or for a project and to ask whose voice is missing. This shouldn’t be done at the end as a last minute fix, but during the ideation phase.
  • We should gather and confront data on our citational patterns from our publications. Knowing what you have done is better than not knowing.
  • We need to do the archaeological work to find and recover marginalized thinkers who have been left out and reflect on why they were left out. Then we need to promote them in teaching and research.
  • We should be willing to call out grants, articles, and proposals we review when it could make a difference.
  • We need to support work to translate thinkers whose work is not in English to balance the distribution of influence.
  • We need to be willing to view our field and its questions very differently.

Giant, free index to world’s research papers released online

Catalogue of billions of phrases from 107 million papers could ease computerized searching of the literature.

From Ian I learned about a Giant, free index to world’s research papers released online. The General Index, as it is called, makes ngrams of up to 5 words available with pointers to relevant journal articles.

The massive index is available from the Internet Archive here. Here is how it is described.

Public Resource, a registered nonprofit organization based in California, has created a General Index to scientific journals. The General Index consists of a listing of n-grams, from unigrams to five-grams, extracted from 107 million journal articles.

The General Index is non-consumptive, in that the underlying articles are not released, and it is transformative in that the release consists of the extraction of facts that are derived from that underlying corpus. The General Index is available for free download with no restrictions on use. This is an initial release, and the hope is to improve the quality of text extraction, broaden the scope of the underlying corpus, provide more sophisticated metrics associated with terms, and other enhancements.

Access to the full corpus of scholarly journals is an essential facility to the practice of science in our modern world. The General Index is an invaluable utility for researchers who wish to search for articles about plants, chemicals, genes, proteins, materials, geographical locations, and other entities of interest. The General Index allows scholars and students all over the world to perform specialized and customized searches within the scope of their disciplines and research over the full corpus.

Access to knowledge is a human right and the increase and diffusion of knowledge depends on our ability to stand on the shoulders of giants. We applaud the release of the General Index and look forward to the progress of this worthy endeavor.

There must be some neat uses of this. I wonder if someone like Google might make a diachronic viewer similar to their Google Books Ngram Viewer available?