Lancashire: Literary Alzheimer’s

In the category of things I meant to blog some time ago is Ian Lancashire and Graeme Hirst’s research into Agatha Christie’s Alzheimer’s-related dementia which was written up by the New York Times in their list of notable ideas for 2009. The write up is by Amanda Fortini, see Literary Alzheimer’s – The Ninth Annual Year in Ideas – Magazine. There is a longer article about this research by Judy Stoffman in the Insight section of the Toronto Star, An Agatha Christie mystery: Is Alzheimer’s on the page? (Jan. 23, 2010)

Lancashire’s specialty is the esoteric field of neuro-cognitive literary theory – in his words “what science says about the creative process versus what authors report about how they create their books.” He started to apply computer analysis to literary texts in 1982.

Ian Lancashire has links to the poster that first got attention and to a paper on his home page. He has also just published a book, Forgetful Muses; Reading the Author in the Text that develops his neuro-cognitive literary theory.

NYT: Armies of Expensive Lawyers, Replaced by Cheaper Software

The New York Times has an article about commercial text analysis systems by John Markoff, Armies of Expensive Lawyers, Replaced by Cheaper Software (March 5, 2011, A1 in New York Edition; March 4 online). He describes how companies are building systems that can analyze the immense amounts of documents shared in lawsuits. Traditionally an army of people would comb through the documents, “Now, thanks to advances in artificial intelligence, “e-discovery” software can analyze docuemnts in a fraction of the time for a fraction of the cost.”

Some programs go beyond just finding documents with relevant terms at computer speeds. They can extract relevant concepts — like documents relevant to social protest in the Middle East — even in the absence of specific terms, and deduce patterns of behavior that would have eluded lawyers examining millions of documents.

There is a nice graphic to accompany the article here. Markoff mentions companies like Blackstone Discovery and Cataphora. He also argues that the availability of a large email archive from Enron has made it possible for teams to experiment on a real dataset.

IBM’s “Watson” Computing System to Challenge All Time Greatest Jeopardy! Champions

Richard drew my attention to the upcoming competition between IBM’s Watson deep question and answer system and top Jeopardy! champions, IBM’s “Watson” Computing System to Challenge All Time Greatest Jeopardy! Champions. I’d blogged on Watson before – it’s a custom system designed to mine large collections of data for answers to questions. Here is what IBM says its applications are,

Beyond Jeopardy!, the technology behind Watson can be adapted to solve problems and drive progress in various fields. The computer has the ability to sift through vast amounts of data and return precise answers, ranking its confidence in its answers. The technology could be applied in areas such as healthcare, to help accurately diagnose patients, to improve online self-service help desks, to provide tourists and citizens with specific information regarding cities, prompt customer support via phone, and much more.

TagCrowd

created at TagCrowd.com

TagCrowd is another web based word cloud generator that seems clean and works on URLs, uploaded files, and pasted files. They also offer a commercial version for a small license fee.

TAPoR portal has moved

The TAPoR Portal has moved to a new server at the University of Alberta. The new location will allow us here to start redesigning it and developing version 2.0. (Or is it now version 3.0?) I underestimated how much work it is to move something so complex. We had to work on bugs, we had to warn users, we had to set up hardware here. Kamal Ranaweera worked very hard to do this – Bravo!

Some links related to the move:

NY Times: Humanities Scholars Embrace Digital Technology

The next big idea is data according to a New York Times article, Humanities Scholars Embrace Digital Technology by Patricia Cohen (November 16, 2010.) The article reports on some of the big data interpretation projects like those funded by the Digging Into Data program like the Mining with Criminal Intent project I am on.

Members of a new generation of digitally savvy humanists argue it is time to stop looking for inspiration in the next political or philosophical “ism” and start exploring how technology is changing our understanding of the liberal arts. This latest frontier is about method, they say, using powerful technologies and vast stores of digitized materials that previous humanities scholars did not have.

I’m not sure this is a new generation as we have been at this for a while, but perhaps the point is that the new generation is now looking away from theory towards the large-scale data issues.

What stands out about the projects mentioned and others is that the digital humanities and design fields are developing new and subtler forms of large-scale data mining and interpretation that use methods from other disciplines along with a sensitivity to the nature of the data and the questions we want to ask. The image above comes from Stanford’s Visualization of Republic of Letters project. There is nothing new about visualization or network analysis, but digital humanists are trying to adapt methods to messy human data – in other words interpreting the really interesting stuff so that it makes sense of something to someone.

Perhaps we may be able to show that following theses are true and important to the broader community:

  • Interesting data has to be interpreted to be interesting. Someone has to pose the questions that make data useful.
  • There is too much of data and it is messy; therefore it can’t by interpreted automatically. Real world analysis always involves questions, choices, data curation, mixing techniques, and iterative interpretation of results to generate knowledge.
  • Interesting data always has to be explained to someone in some context. Results are only useful knowledge if they are published in some fashion that makes them accessible to an intended audience.
  • Humanists have been the curators and interpreters of information which is why the subtle skills of questioning, curating, editing, analyzing, interpreting and representing are all the more needed now. Without humanists (and I include librarians and archivists in this category) who are comfortable with digital data and methods we will have only too much data and too many unused tools.

Thanks to Judith for pointing me to this NYT article.

2nd Edition of Icon Programming for Humanists

I just got a notice that the 2nd Edition of Icon Programming for Humanists (PDF) by Alan D. Corré has been up (and its free). This has been made available by Jeffery Books who will also sell you a paperback copy. Donations go to promoting Icon and Unicon programming languages and systems.

I read Icon Programming for Humanists ages ago. It was one of the few how-to-program books that were aimed at humanists with text manipulation examples. I thought the book excellent and was only held back because I couldn’t find an Icon interpreter for the Mac when I looked.

This edition has 2 new chapters that deal with Unicode (so you can analyze texts in different languages), and Markup (so you can work with TEI encoded texts.)

There is a recurring issue that crops up as to whether we should be teaching humanities students to program or just to use tools. Corré’s book would make a good textbook for teaching programming.