Thanks to James K I discovered Twitterfall which shows a waterfall of tweets based on keywords and other settings. An interesting example of what St̩fan Sinclair and I call a Knowledge Radio Рan application that organizes knowledge into a live stream.
I put a photo set up on Flickr for our Hacking as a Way of Knowing project. The set documents the evolution of the project which I’ve tentatively named the “ReReader for the Writing on the Wall”. Thanks to all those who made the project and the workshop a success. Now I have to think a bit deeper about making as knowing and things as theories.
From IBM Labs YouTube presence news of a IBM “Watson” System to Challenge Humans at Jeopardy! IBM’s Watson is a Question Answering system that IBM scientists hope “will be able to understand complex questions and answer with enough precision and speed to compete on Jeopardy!” (From the IBM press release.)
Watson will be designed to deftly handle semantics “the meanings behind words” which will enable it to answer questions that require the identification of relevant and irrelevant content, the interpretation of ambiguous expression and puns, the decomposition of questions into sub-questions, and the logical synthesis of final answers. In addition, Watson will compute a statistical confidence in the responses it provides. Watson will be designed to do all of this in a matter of seconds, which will enable it to compete against humans, who have the ability to know what they know in less than a second. (From Addendum: About IBM’s Watson System)
The language IBM uses around this project is that of a “Grand Challenge.” It is smart how they have taken a analytical problem and used Jeopardy to give a target for the research, both in terms of speed and the types of questioning handled. Jeopardy also gives them a dramatic venue to demonstrate their progress just as Deep Blue playing Kasparov did.
The research is based on a Open Architecture for Question Answering (OAQA) that was jointly developed with Carnegie Mellon.
We all know that the Google display comes indirectly from the Concordance, but I have found in Luhn’s 1966 “Keyword-in-Context Index for Technical Literature (Kwic Index)” the explicit recognition of the link and the reason for drawing on the concordance.
the significance of such single keywords could, in most instances, be determined only by referring to the statement from which the keyword had been chosen. This somewhat tedious procedure may be alleviated to a significant degree by listing selected keywords together with surrounding words that act as modifiers pointing up the more specific sense in which a keyword has been applied. This method of indexing words is well established in the process of compiling concordances of important works of literature of the past. The added degree of information conveyed by such keyword-in-context indexes, or “KWIC Indexes” for short, can readily be provided by automatic processing. (p. 161)
The problem for Luhn is that simply retrieving words doesn’t give you a sense of their use. His solution, first shown in the late 1950s, was to provide some context (hence “keyword-in-context”) so that readers can disambiguate themselves and make decisions about which index items to follow. It is from the KWIC that we ultimately get the concordance features of the Google display, though it should be noted that Luhn was proposing KWIC as a way of printing automatically generated literature indexes where the kewwords were in the titles. In this quote Luhn explicitly acknowledges that this is a method well established in concordances.
There is also a link between Luhn and Father Busa. According to Black, quoted in Marguerite Fischer, “The Kwic Index Concept: A Retrospective View”,
the Pontifical Faculty of Philosophy in Milan decided that they would make an analytical index and concordance to the Summa Theologica of St. Thomas Aquinas, and approached IBM about the possibility of having the operations performed on Data Processing. Experience gained in this project contributed towards the development of the KWIC Index. (This is a quote on page 123 from Black, J. D., 1962, “The Keyword: Its Use in Abstracting, Indexing, and Retrieving Information”.)
From the concordance to KWIC through to Google?
For some historical notes on Luhn see, H. P. Luhn and Automatic Indexing.
Over the last few days I’ve been reading and contributing to one of the HASTAC discussion forms on Digital Textuality and Tools. This is led by Angela Kinney and Michael Widner.
Among other things there was a link to an interesting project, Inventoriana for collaborative annotation of manuscript images.
I attended Workshop 3 of Project Bamboo in Tucson Arizona this week. I think I’m beginning to understand it, though understanding what Bamboo is was one of the favorite subjects of conversation of the meeting (so I’m conscious that . There is a deliberate ambiguity to the project since they are trying to listen to the community in order to become what we want rather than what we suspect. Some of my takeaway thoughts:
The Cornell Web Lab is an interesting example of a high performance computing project in the humanities and social sciences. As they say,
The Web Laboratory is a joint project of Cornell University and the Internet Archive to provide data and computing tools for research about the Web and the information on the Web.
In a paper on the project, A Research Library Based on the Historical Collections of the Internet Archive, William Arms and colleagues point out that the data challenge of the social sciences (and humanities) is that the data is poorly structured and there is a lot of it. The Internet Archive is a case in point; as of 2006 they had 5 to 6 petabytes of data of web pages. While it is amazing that we have such archives in computer (and human) readable form, it is hard to do anything with that much. The Web Lab approach is to provide HPC basic services for extracting subsets of the whole that can then be used by other tools.
Pliny, the annotation and note management tool by John Bradley at King’s College London just got a Mellon Award for Technology Collaboration.
The Mellon Awards honour not-for-profit organisations for leadership in the collaborative development of open source software tools with application to scholarship in the arts and humanities, as well as cultural-heritage not-for-profit activities.
Pliny is free and you can try it out on the Mac or PC. John has thought a lot about how tools fit in the research process of humanists.
NiCHE (Network in Canadian History & Environment) has a useful wiki called The Programming Historian by William Turkel and Alan MacEachern. The wiki is a “tutorial-style introduction to programming for practicing historians” but it is could also be used by textual scholars who want to be able to program their own tools. It takes you through learning and using Python for text processing for things like word frequencies and KWICs. It reminds me of Susan Hockey’s book, Snobol Programming for the Humanities. (Oxford: Oxford University Press, 1985) which I loved at the time, even if I couldn’t find a Snobol interpreter for the Mac.
We need more of such books/wikis.
I just got back from the Tools For Data-Driven Scholarship meeting organized by MITH and the Centre for New Media and History. This meeting was funded by the NEH, NSF, and the IMLS and brought together tool developers, content providers (like museums and public libraries), and funders (NEH, JISC, Mellon, NSF and IMLS.) The goal was to imagine initiative(s) that could advance humanities tool development and connect tools better with audiences. I have written a Conference Report with my notes on the meeting. One of the interesting questions asked by a funder was “What do the developers really want?” It was unclear that developers really wanted some of the proposed solutions like a directory of tools or code repository. Three things the breakout group I was in came up with was:
A bunch of us sitting around after the meeting waiting for a plane had the usual debriefing about such meetings. What do they achieve even if they don’t lead to initiatives. From my perspective these meeting are useful in unexpected ways: