Automatic grading and how to game it

Edgenuity involves short answers graded by an algorithm, and students have already cracked it

The Verge has a story on how students are figuring out how to game automatic marking systems like Edgenuity. The story is titled, These students figured out their tests were graded by AI — and the easy way to cheat. The story describes a keyword salad approach where you just enter a list of words that the grader may be looking for. The grader doesn’t know whether what your wrote is legible or nonsense, it just looks for the right words. The students in turn get good as skimming the study materials for the keywords needed (or find lists shared by other students online.)

Perhaps we could build a tool called Edgenorance which you could feed the study materials to and it would generate the keyword list automatically. It could watch the lectures for you, do the speech recognition, then extract the relevant keywords based on the text of the question.

None of this should be surprising. Companies have been promoting algorithms that were probably word based for a while. The algorithm works if it is not understood and thus not gamed. Perhaps we will get AIs that can genuinely understand a short paragraph answer and assess it, but that will be close to an artificial general intelligence and such an AGI will change everything.

Sean Gouglas Remembers Stéfan Sinclair

Sean Gouglas shared these memories of Stéfan Sinclair with me and asked me to post them. They are from when they started the Humanities Computing programme at the University of Alberta where I am lucky to now teach.

In the summer of 2001, two newly-minted PhDs started planning how they were going to build and then teach a new graduate program in Humanities Computing at the University of Alberta. This was the first such program in North America. To be absolutely honest, Stéfan Sinclair and I really had no idea what we were doing. The next few months were both exhausting and exhilarating. Working with Stéfan was a professional and personal treat, especially considering that he had an almost infinite capacity for hard work. I remember him coding up the first Humanities Computing website in about seven minutes — the first HuCo logo appearing like a rising sun on a dark blue background. It also had an unfortunate typo that neither of us noticed for years. 

It was an inspiration to work with Stéfan. He was kind and patient with students, demanding a lot from them but giving even more back. He promoted the program passionately at every conference, workshop, and seminar. Over the next three years, there was a lot of coffee, a lot of spicy food, a beer or two, some volleyball, some squash, and then he and Stephanie were off to McMaster for their next adventure. 

Our Digital Humanities program has changed a lot since then — new courses, new programs, new faculty, and even a new name. Through that change, the soul of the program remained the same and it was shaped and molded by the vision and hard work of Stéfan Sinclair. 

On the 6th of August, Stéfan died of cancer. The Canadian Society for Digital Humanities has a lovely tribute, which can be found here: https://csdh-schn.org/stefan-sinclair-in-memoriam/. It was written in part by Geoffrey Rockwell, who worked closely with Stéfan for more than two decades. 

OSS advise on how to sabotage organizations or conferences

On Twitter someone posted a link to a 1944 OSS Simple Sabotage Field Manual. This includes simple, but brilliant advice on how to sabotage organizations or conferences.

This sounds a lot like what we all do when we academics normally do as a matter of principle. I particularly like the advice to “Make ‘speeches.'” I imagine many will see themselves in their less cooperative moments in this list of actions or their committee meetings.

The OSS (Office of Strategic Services) was the US office that turned into the CIA.

What coding really teaches children

You’ve seen movies where programmers pound out torrents of code? That is nothing like reality. Most of the time, coders don’t type at all; they sit and stare morosely at the screen, running their hands through their hair, trying to spot what they’ve done wrong. It can take hours, days, or even weeks. But once the bug is fixed and the program starts working again, the burst of pleasure has a narcotic effect.

Stéfan pointed me to a nice opinion piece about programming education in the Globe titled, Opinion: What coding really teaches children. Clive Thompson that teaching programming in elementary school will not necessarily teach math but it can teach kids about the digital world and teach them the persistence it takes to get complex things working. He also worries, as I do, about asking elementary teachers to learn enough coding to be able to teach it. This could be a recipe for alienating a lot of students who are taught by teachers who haven’t learned.

Internet Archive closes the National Emergency Library

Within a few days of the announcement that libraries, schools and colleges across the nation would be closing due to the COVID-19 global pandemic, we launched the temporary National Emergency Library to provide books to support emergency remote teaching, research activities, independent scholarship, and intellectual stimulation during the closures.  […]

According to the Internet Archive blog the Temporary National Emergency Library to close 2 weeks early, returning to traditional controlled digital lending. The National Emergency Library (NEL) was open to anyone in the world during a time when physical libraries were closed. It made books the IA had digitized available to read online. It was supposed to close at the end of June because four commercial publishers decided to sue. 

The blog entry points to what the HathiTrust is doing as part of their Emergency Temporary Access Service which lets libraries that are members (and the U of Alberta Library is one) provide access to digital copies of books they have corresponding physical copies of. This is only available to “member libraries that have experienced unexpected or involuntary, temporary disruption to normal operations, requiring it to be closed to the public”. 

It is a pity the IS NEL was discontinued, for a moment there it looked like large public service digital libraries might become normal. Instead it looks like we will have a mix of commercial e-book services and Controlled Digital Lending (CDL) offered by libraries that have the physical books and the digital resources to organize it. The IA blog entry goes on to note that even CDL is under attack. Here is a story from Plagiarism Today:

Though the National Emergency Library may have been what provoked the lawsuit, the complaint itself is much broader. Ultimately, it targets the entirety of the IA’s digital lending practices, including the scanning of physical books to create digital books to lend.

The IA has long held that its practices are covered under the concept of controlled digital lending (CDL). However, as the complaint notes, the idea has not been codified by a court and is, at best, very controversial. According to the complaint, the practice of scanning a physical book for digital lending, even when the number of copies is controlled, is an infringement.

CSDH / SCHN 2020 was brilliant online

Today was the last day of the CSDH / SCHN 2020 online conference. You can see my conference notes here. The conference had to go online due to Covid-19 and the cancellation of Congress 2020. That said, the online conference web brilliantly. The Programme Committee, chaired by Kim Martin, deserve a lot of credit as do the folks at the U of Alberta Arts Resource Centre who provided technical support. Some of the things they did that

  • The schedule has a single track across 5 days rather than parallel tracks over 3 days. See the schedule.
  • There were only 3 and half hours of sessions a day (from 9:00am to 12:30 Western time) so you could get other things done. (There were also hangout sessions before and after.)
  • Papers (or prepared presentations) had to be put up the week before on Humanities Commons.
  • The live presentations during the conference were thus kept to 3 minutes or so, which allowed sessions to be shorter which allowed them to have a single track.
  • They had a chair and a respondent for each session which meant that there was a lot of discussion instead of long papers and no time for questions. In fact, the discussion seemed better than at on site conferences.
  • They used Eventbrite for registration, Zoom for the registrants-only parts of the conference, and Google Meet for the open parts.
  • They had hangout or informal sessions at the beginning and end of each day where more informal discussion could take place.

The nice thing about the conference was that they took advantage of the medium. As none of us had flown to London, Ontario, they were able to stretch the conference over 5 days, but not use up the entire day.

All told, I think they have shown that an online conference can work surprisingly well if properly planned and supported.

Introducing the AI4Society Signature Area

AI4Society will provide institutional leadership in this exciting area of teaching, research, and scholarship.

The Quad has a story Introducing the AI4Society Signature Area. Artificial Intelligence for Society is a University of Alberta Signature Area that brings together researchers and instructors from both the sciences and the arts. AI4S looks at how AI can be imagined, designed, and tested so that it serves society. I’m lucky to contribute to this Area as the Associate Director, working with the Director, Eleni Stroulia from Computing Science.

Knowledge is a commons – Pour des savoirs en commun

The Canadian Comparative Literature Association (CCLA/ACLC) celebrated in 2019 its fiftieth anniversary. The association’s annual conference, which took place from June 2 to 5, 2019 as part of the Congress of the Humanities and Social Sciences of Canada at UBC (Vancouver), provided an opportunity to reflect on the place of comparative literature in our institutions. We organized a joint bilingual roundtable bringing together comparatists and digital humanists who think and put in place collaborative editorial practices. Our goal was to foster connections between two communities that ask similar questions about the modalities for the creation, dissemination and legitimation of our research. We wanted our discussions to result in a concrete intervention, thought and written collaboratively and demonstrating what comparative literature promotes. The manifesto you will read, “Knowledge is a commons – Pour des savoirs en commun”, presents the outcome of our collective reflexion and hopes to be the point of departure for more collaborative work.

Thanks to a panel on the Journal in the digital age at CSDH-SCHN 2020 I learned about the manifesto, Knowledge is a commons – Pour des savoirs en commun. The manifesto was “written colingually, that is, alternating between English and French without translating each element into both languages. This choice, which might appear surprising, puts into practice one of our core ideas: the promotion of active and fluid multilingualism.” This is important.

The manifesto makes a number of important points which I summarize in my words:

  • We need to make sure that knowledge is truly made public. It should be transparent, open and reversible (read/write).
  • We have to pay attention to the entire knowledge chain of research to publication and rebuild it in its entirety so as to promote access and inclusion.
  • The temporalities, spaces, and formats of knowledge making matter. Our tools and forms like our thought should be fluid and plural as they can structure our thinking.
  • We should value the collectives that support knowledge-making rather than just authoritative individuals and monolithic texts. We should recognize the invisible labourers and those who provide support and infrastructure.
  • We need to develop inclusive circles of conversation that cross boundaries. We need an ethics of open engagement.
  • We should move towards an active and fluid multilingualism (of which the manifesto is an example.)
  • Writing is co-writing and re-writing and writing beyond words. Let’s recognize a plurality of writing practices.

 

“Excellence R Us”: university research and the fetishisation of excellence

Is “excellence” really the most efficient metric for distributing the resources available to the world’s scientists, teachers, and scholars? Does “excellence” live up to the expectations that academic communities place upon it? Is “excellence” excellent? And are we being excellent to each other in using it?

During the panel today on Journals in the digital age: penser de nouveaux modèles de publication en sciences humaines at CSDH-SCHN 2020 someone linked to an essay on  “Excellence R Us”: university research and the fetishisation of excellence in Palgrave Communications (2017). The essay does what should have been done some time ago, it questions the excellence of “excellence” as a value for everything in universities. The very overuse of “excellence” has devalued the concept. Surely much of what we do these days is “good enough” especially as our budgets are cut and cut.

The article has three major parts:

  • Rhetoric of excellence – it looks at how there is little consensus around what excellence between disciplines. Within disciplines it is negotiated and can become conservative.
  • Is “excellence” good for research – the second section argues that there is little correlation between forms of excellence review and long term metrics. They go on to outline some of the unfortunate side-effects of the push for excellence; how it can distort research and funding by promoting competition rather than collaboration. They also talk about how excellence disincentivizes replication – who wants to bother with replication if
  • Alternative narratives – the third section looks at alternative ways of distributing funding. They discuss looking at “soundness” and “capacity” as an alternatives to the winner-takes-all of excellence.

So much more could and should be addressed on this subject. I have often wondered about the effect of the success rates in grant programmes (percentage of applicants funded). When the success rate gets really low, as it is with many NEH programmes, it almost becomes a waste of time to apply and superstitions about success abound. SSHRC has healthier success rates that generally ensure that most researchers gets funded if they persist and rework their proposals.

Hypercompetition in turn leads to greater (we might even say more shameless …) attempts to perform this “excellence”, driving a circular conservatism and reification of existing power structures while harming rather than improving the qualities of the underlying activity.

Ultimately the “adjunctification” of the university, where few faculty get tenure, also leads to hypercompetition and an impoverished research environment. Getting tenure could end up being the most prestigious (and fundamental) of grants – the grant of a research career.

 

DARIAH Virtual Exchange Event

This morning at 7am I was up participating in a DARIAH VX (Virtual Exchange) on the subject of The Scholarly Primitives of Scholarly Meetings. This virtual seminar was set up when DARIAH’s f2f (face-2-face) meeting was postponed. The VX was to my mind a great example of an intentionally designed virtual event. Jennifer Edmunds and colleagues put together an event meant to be both about and an example of a virtual seminar.

One feature they used was to have us all split into smaller breakout rooms. I was in one on The Academic Footprint: Sustainable methods for knowledge exchange. I presented on Academic Footprint: Moving Ideas Not People which discussed our experience with the Around the World Econferences. I shared some of the advice from the Quick Guide I wrote on Organizing a Conference Online.

  • Recognize the status conferred by travel
  • Be explicit about blocking out the time to concentrate on the econference
  • Develop alternatives to informal networking
  • Gather locally or regionally
  • Don’t mimic F2F conferences (change the pace, timing, and presentation format)
  • Be intentional about objectives of conference – don’t try to do everything
  • Budget for management and technology support

For those interested we have a book coming out from Open Book Publishers with the title Right Research that collects essays on sustainable research. We have put up preprints of two of the essays that deal with econferences:

The organizers had the following concept and questions for our breakout group.

Session Concept: Academic travel is an expense not only to the institutions and grant budgets, but also to the environment. There have been moves towards open-access, virtual conferences and near carbon-neutral events. How can academics work towards creating a more sustainable environment for research activities?

Questions: (1) How can academics work towards creating a more sustainable environment for research activities? (2) What are the barriers or limitations to publishing in open-access journals and how can we overcome these? (3) What environmental waste does your research produce? Hundreds of pages of printed drafts? Jet fuel pollution from frequent travel? Electricity from powering huge servers of data?

The breakout discussion went very well. In fact I would have had more breakout discussion and less introduction, though that was good too.

Another neat feature they had was a short introduction (with a Prezi available) followed by an interview before us all. The interview format gave a liveliness to the proceeding.

Lastly, I was impressed by the supporting materials they had to allow the discussion to continue. This included the DARIAH Virtual Exchange Event – Exhibition Space for the Scholarly Primitives of Scholarly Meetings.

All told, Dr. Edmonds and DARIAH colleagues have put together a great exemplar both about and of a virtual seminar. Stay tuned for when they share more.