(Sparkline of US deficit over time) Sparklines: theory and practice is a thread in Edward Tufte’s Ask E.T. forum (which is a great place to follow discussions on design issues.) The thread starts with images of some pages from Tufte’s new book, Beautiful Evidence (2006) on sparklines which are defined as “intense, simple, word-sized graphics”. The sparkline at the beginning of this entry is from the Sparkline PHP Graphing Library. Another source of sparkline tools is Bissantz sparkline tools. Thanks to Shawn for this link.
So how can sparklines be woven into text anlysis environments? Small distribution graphs could be included with lists of word or KWIC displays in tools like the TAPoRware tools.
Category: Text Analysis
Download Pertinence Summarizer – Text Mining Solutions
PertinenceMining.com is a French company that has a number of neat text processing products built on their KENiA or “Knowledge Extraction and Notification Architecture.” One their products is Connivences.info which produces maps of “actors” in the news with weighted lines to indicate relationships.
Another interesting tool is their Google + Pertinence Summarizer that enhances the results from Google with a “Summarize” button which splits the linked page into sentences and tries to rank their pertinence to the document so you can choose to see only the most pertinent. The interface took me a while – I’m not sure it works.
DadaDodo: Exterminate All Rational Thought
DadaDodo is a text generator or “travesty generator” like Dissociated Press. The code is available and unlike programs that randomly cut up text it “it scans bodies of text, and builds a probability tree expressing how frequently word B tends to occur after word A, and various other statistics; then it generates sentences based on those probabilities.” DadaDodo is described by its creator Jamie Zawinski thus:
DadaDodo is a program that analyses texts for word probabilities, and then generates random sentences based on that. Sometimes these sentences are nonsense; but sometimes they cut right through to the heart of the matter, and reveal hidden meanings.
Zawinski’s page has a “cut up” look with downloadable code and interesting links, many of which are no longer active, alas. The effect of DadaDodo are hard to interpret without knowing what the corpus is that it starts with. I am tempted to create a TAPoRware version so that it can be used on existing web pages.
Communications From Elsewhere »
Communications From Elsewhere is a journal (not blog!) by Josh Larios with some interesting text generators including a Postmodernism Generator which randomly generates “completely meaningless” essays using a modified version of The Dada Engine written by Andrew C. Bulhak.
For more on The Dada Engine see the technical report from Monash University, On the simulation of postmodernism and mental debility using recursive transition networks. The Abstract reads:
Recursive transition networks are an abstraction related to context-free grammars and finite-state automata. It is possible, to generate random, meaningless and yet realistic-looking text in genres defined using recursive transition networks, often with quite amusing results. One genre in which this has been accomplished is that of academic papers on postmodernism.
Josh has collected and connected different “Text Generators” to his journal, including an Adolescent Poetry Corner and a Time Cube screed generator. (For an explanation of Gene Ray’s Time Cube theory see DmitryBrant.com ¬ª On Time Cube. The Time Cube site is another story.)
BBC: Text analysis of texting to catch criminals
The BBC have a story on Texting study to catch criminals by researchers at Leicester University in the School of Psychology. They are studying the individual styles of text messages in order to help with forensic investigations. The BBC had previously reported how Text messages examined in Danielle case had helped in the prosecution of the 15-year-old Danielle Jones’ uncle who seems to have sent text messages from Danielle’s phone in order to throw off the scent.
The Leicester University researchers have a web page that welcomes people willing to submit samples. I wonder how useful anonymously submitted messages will be. I imagine, if they get enough messages, it will give them a control sample for the study of particular suspect messages.
RDUES: WebCorp: The Web as Corpus
The Research and Development Unit for English Studies (RDUES) of UCE of Birmingham has a tool WebCorp: The Web as Corpus which searches google for a term and then goes to the top 199 documents Google identifies and searches them. It takes a while and works like our Googlizer, but produces more verbose results. It produces a concordance organized by document with links to a full word frequency list for the doc. The advanced search form has some interesting features, including the ability to point it at other engines.
UCE Birmingham is strange place from the web. UCE stands for “University of Central England” and you have to go deep to the At A Glance : History Of UCE Birmingham to find this out. (There’s no point explaining it to outsiders anywhere on the web page.) They seem to have been formed out of all the little colleges, polytechnics and schools in the area in 1992.
W3Corpora Search Engine
The University of Essex has a site on linguistics corpora with an interesting W3Corpora Search Engine. The Search Engine has a wizard approach where you make choices and continue to next section. It works on Guternberg texts. They also have a tutorial on corpus linguistics.
RefViz: Bibliographic Visualization
RefViz is a visualization tool from Thomson Researchsoft (who also publish EndNote and ProCite). RefViz lets you visualize “galaxies” of bibliographic references showing clusters of references by keywords. It also has a matrix view where you can see how keywords correlate.
Save time and learn more about what is happening in the literature with RefViz. With this powerful text analysis and visualization software program, you get an intuitive framework for exploring reference collections based on content. (From the Product Info page.)
Rollyo: Roll Your Own Search Engine
Rollyo: Roll Your Own Search Engine is another service that lets you create custom search engines that only search a “Searchroll” set of domains/sites. You can have multiple Searchrolls and you can create small Rollyo search panels for your site.
They are onto something, especially if they allowed hierarchies of shared (public) Searchrolls. The down side of the service is the limit on sites you can include (25) and the advertising that shows up in the results.
Here is one I created form my personal website www.geoffreyrockwell.com:
Amazon Text Analysis
Amazon has recently added a neat feature to the pages on certain books. If you go, for example to, Amazon.com: The Search: How Google and Its Rivals Rewrote the Rules of Business and Transformed Our Culture: Books: John Battelle and mouse-over the image of the book it gives you:
- SIPs: Statistically Improbably Phrases
- CAPs: Capitalized Phrases
- and the ability to search inside the book and get a concordance.
They are providing a simple form of text analysis right on the book page. You can click on a SIP and see what other books (for sale on Amazon) have a high frequency of that improbable phrase.