Solr is a “search server” based on Lucene that offers “Advanced, Configurable Text Analysis” and XML handling.
Text fields are typically indexed by breaking the field into words and applying various transformations such as lowercasing, removing plurals, or stemming to increase relevancy. The same text transformations are normally applied to any queries in order to match what is indexed. (Tutorial)
Philomine is an extension to the software Mark Olsen and company developed for ARTFL, PhiloLogic. The Rationale and Design Notes page explains text mining in the humanities with examples from ARTFL. As they explain,
Philomine was a common 19th century French name (particularly Qu?©b?©cois), probably derived from Philomena, patron Saint of “lost causes” amongst other things.
Brad Borevitz of onetwothree.net has developed another visualization of language in presidential State of the Union Addresses at State of the Union. He calls it a “data toy” and it combines a number of different graphs. One nice feature is that if you click on one Address and then another the word cloud for the first appears behind (and in red) the second for comparison purposes.
I have blogged other such visualization toys that use the State of the Union Addresses like State of the Union Parsing Tool and the SOTU Rich Prospect Browsing of the New York Times.
Thanks to Nick for this.
The CNW Group that has it’s Canadian base in Toronto has a new service called MEDIAVANTAGE that has many of the features of a multimedia news crawling, managing, and visualizing service. From the Flash intro it looks like users define keywords to track. Mediavantage then shows you results from different sources. It can send alerts and graphs result history.
The interesting part is that they track TV news and provide text summaries that look like the text off close captioning. Subsets of results can be shared by e-mail and PDF. This is a news mining tool for business that offers a model for what Web Mining for Research might look like.
Thanks to Terry for this.
What’s Web Mining for Research is a white paper I wrote on the TADA wiki trying to define an emerging research practice that draws on the web as evidence of human behaviour. I’m not happy with the phrase, but it hard to know what to call it. Text mining refers to mining large text databases, not the web. Web mining means all sorts of things. What stands out for me as important is that we have in the Web a massive body of evidence for philosophical and cultural analysis, something we haven’t had before. While a chance in evidence may seem trivial, the resulting change in research practices is not.
Shawn Day on his Randomosity blog has links to some good explanations of the Soundex algorithm for finding names that sound the same, but are spelled differently as in “Smith” and “Smythe”. For example, Eastman’s Online Geneaology Newsletter has a post on Soundex Explained that walks you through the algorithm.
Yet another George W Bush, State of the Union visualization tool can be seen at State of the Union Parsing Tool. I commented earlier on the New York Times, State of the Union in Words. It seems that Bush’s State of the Union addresses are becoming the standard text for visualizations.
This one on Style.org colorizes the lines with the found words. You can set the size of the words (and therefore text representation.)
I have set up an account on the TAPoR portal where I have created a list of Humanities Computing Links. Because it is on the portal you can analyze the texts. I have also tagged them so you can look at a subset like, Text Analysis.
The Centre National de Ressources Textuelles et Lexicales (CNRTL) is a centre attached to the Analyse et Traitement Informatique de la Langue Fran?ßaise (ATILF / CNRS) lab at Nancy Universit?©. They have a portal for lexical, morphological and etymological work in French. It takes a French word and will give you synonyms, dictionary entry, etymology and so on. Neat. I wonder if it can be mashed into a TAPoR tool?
This is thanks to Jean-Guy.
Matt asked me if we can use TAPoRware tools to compare different texts so I wrote a short recipe on Comparing Two Texts. This assumes that the texts you are comparing are not different versions of the same text, but completely different texts.
It strikes me that we have to think through how the TAPoRware tools could be adapted for comparison.