Category: Computers and Education

Humanity’s Last Exam

Researchers with the Center for AI Safety and Scale AI are gathering submissions for Humanity’s Last Exam. The submission form is here. The idea is to develop an exam with questions from a breadth of academic specializations that current LLMs can’t answer.

While current LLMs achieve very low accuracy on Humanity’s Last Exam, recent history shows benchmarks are quickly saturated — with models dramatically progressing from near-zero to near-perfect performance in a short timeframe. Given the rapid pace of AI development, it is plausible that models could exceed 50% accuracy on HLE by the end of 2025. High accuracy on HLE would demonstrate expert-level performance on closed-ended, verifiable questions and cutting-edge scientific knowledge, but it would not alone suggest autonomous research capabilities or “artificial general intelligence.” HLE tests structured academic problems rather than open-ended research or creative problem-solving abilities, making it a focused measure of technical knowledge and reasoning. HLE may be the last academic exam we need to give to models, but it is far from the last benchmark for AI.

One wonders if it really will be the last exam. Perhaps we will get more complex exams that test for integrated skills. Andrej Karpathy criticises the exam on X. I agree that what we need are AIs able to do intern-level complex tasks rather than just answering questions.

The 18th Annual Hurtig Lecture 2024: Canada’s Role in Shaping our AI Future

The video for the 2024 Hurtig Lecture is up. The speaker was Dr. Elissa Strome, Executive Director of the Pan-Canadian AI Strategy. She gave an excellent overview of the AI Strategy here in Canada and ended by discussing some of the challenges.

The Hurtig Lecture was organized by my colleague Dr. Yasmeen Abu-Laban. I got to moderate the panel discussion and Q & A after the lecture.

ASBA Releases Artificial Intelligence Policy Guidance for K-12 Education – Alberta School Boards Association

Alberta School Boards Association (ASBA) is pleased to announce the release of its Artificial Intelligence Policy Guidance. As Artificial Intelligence (AI) continues to shape the future of education, ASBA has […]

The ASBA Releases Artificial Intelligence Policy Guidance for K-12 Education – Alberta School Boards Association. This 14 page Policy document is clear and useful without being proscriptive. It could be a model for other educational organizations. (Note that it was authored by someone I supervised.)

Decker – A HyperCard for the Web

I’m at the CSDH-SCHN conference which is in Montreal. We have relocated to U de Montreal from McGill where Congress is taking place. Jason Boyd gave a paper about the Centre for Digital Humanities at TMU that he directs. He mentioned an authoring environment called Decker that recreates a deck/card based environment similar to what HyperCard was like.

Decker can be used to create visual novels, interactive texts, hypertexts, educational apps, and small games. It has a programming language related to Lua. It has simple graphics tools.

Decker looks really neat and seems to work within a browser as a HTML page. This mean that you can Save As a page and get the development environment locally. All the code and data in a page that can be forked or passed around.

As a lover of HyperCard I am thrilled to see something that replicates its spirit!

The Power of AI Is In Our Hands. What Do We Need to Know?

The Power of AI Is In Our Hands. What Do We Need to Know?

The New Trail has a great feature story by Lisa Szabo on generative AI, The Power of AI Is In Our Hands. What Do We Need to Know? The story features a number of us at U of Alberta talking about the generative AI tools like ChatGPT. It quotes me talking about art and how I believe we will still want art by humans despite what AIs can generate. Perhaps it would be more accurate to say that we will enjoy and consume both AI generated entertainment and art that we believe was generated by people we know.

A groundbreaking study shows kids learn better on paper, not screens. Now what?

For ‘deeper reading’ among children aged 10-12, paper trumps screens. What does it mean when schools are going digital?

The title of this Guardian story says it all, A groundbreaking study shows kids learn better on paper, not screens. Now what? The story reports on a study led by Karen Froud at Columbia University titled, Middle-schoolers’ reading and processing depth in response to digital and print media: An N400 study. They found “evidence of differences in brain responses to texts presented in print and digital media, including deeper semantic encoding for print than digital texts.” Paper works better.

John Gabrieli, an MIT neuroscientist who is skeptical about the promises of big tech and its salesmen: “I am impressed how educational technology has had no effect on scale, on reading outcomes, on reading difficulties, on equity issues,”…

How AI Image Generators Make Bias Worse – YouTube

A team at the LIS (London Interdisciplinary School) have created a great short video on the biases of AI image generators. The video covers the issues quickly and is documented with references you can follow for more. I had been looking at how image generators portrayed academics like philosophers, but this reports on research that went much further.

What is also interesting is how this grew out of a LIS undergrad’s first year project. It says something about LIS that they encourage and build on such projects. This got me wondering about the LIS which I had never heard of before. It seems to be a new teaching college in London, UK that is built around interdisciplinary programmes, not departments, that deal with “real-world problems.” It sounds a bit like problem-based learning.

Anyway, it will be interesting to watch how it evolves.

Huminfra: The Imitation Game: Artificial Intelligence and Dialogue

Today I gave a talk online for an event organized by Huminfra, a Swedish national infrastructure project. The title of the talk was “The Imitation Game: Artificial Intelligence and Dialogue” and it was part of an event online on “Research in the Humanities in the wake of ChatGPT.” I drew on Turing’s name for the Turing Test, the “imitation game.” Here is the abstract,

The release of ChatGPT has provoked an explosion of interest in the conversational opportunities of generative artificial intelligence (AI). In this presentation Dr. Rockwell will look at how dialogue has been presented as a paradigm for thinking machines starting with Alan Turing’s proposal to test machine intelligence with an “imitation game” now known as the Turing Test. In this context Rockwell will show Veliza a tool developed as part of Voyant Tools (voyant-tools.org) that lets you play and script a simple chatbot based on ELIZA which was developed by Joseph Weizenbaum in 1966. ELIZA was one of the first chatbots with which you could have a conversation. It responded as if a psychotherapist, turning whatever you said back into a question. While it was simple, it could be quite entertaining and thus provides a useful way to understanding chatbots.

The Emergence of Presentation Software and the Prehistory of PowerPoint

PowerPoint presentations have taken over the world despite Edward Tufte’s pamphlet The Cognitive Style of PowerPoint. It seems that in some contexts the “deck” has become the medium of information exchange rather than the report, paper or memo. In Slashdot I came across a link to a MIT Review essay titled, Next slide, please: A brief history of the corporate presentation. Another history is available from the Computer History Museum, Slide Logic: The Emergence of Presentation Software and the Prehistory of PowerPoint.

I remember the beginnings of computer-assisted presentations. My unit at the University of Toronto Computing Services experimented with the first tools and projectors. The three-gun projectors were finicky to set up and I felt a little guilty promoting set ups which I knew would take lots of technical support. In one presentation on digital presentations there was actually a colleague under the table making sure all the technology worked while I pitched it to faculty.

I also remember tools before PowerPoint. MORE was an outliner and thinking tool that had a presentation mode much the way Mathematica does. MORE was developed by Dave Winer who had a nice page on the history of outline processors he worked on here. It he leaves out how Douglas Engelbart’s Mother of All Demos in 1968 showed something like outlining too.

Alas, PowerPoint came to dominate though now we have a bunch of innovative presentation tools that work on the web from Google Sheets to Prezi.

Now back to Tufte. His critique still stands. Presentation tools have a cognitive style that encourages us to break complex ideas into chunks and then show one chunk at a time in a linear sequence. He points out that a well designed handout or pamphlet (like his pamphlet on The Cognitive Style of PowerPoint) can present a lot more information in a way that doesn’t hide the connections. You can have something more like a concept map that you take people through on a tour. Prezi deserves credit for paying attention to Tufte and breaking out of the linear style.

Now, of course, there are AI tools that can generate presentations like Presentations.ai or Slideoo. You can see a list of a number of them here. No need to know what you’re presenting, an AI will generate the content, design the slides, and soon present it too.