Motion Capture and Noh

On November 10th I was invited by Dr. Kozaburo Hachimura to watch as his graduate students capture the motion of a master Noh performer. The motion capture was run in a special lab that was specifically built for this. They have a floor that was built to Noh theatre standards and we had to take our slippers off to protect the wood. There is a rig on the ceiling with the motion capture cameras and a sound booth in the back. When not in use for motion capture the room is used for seminars and meetings.

Checking the capture on the screen

The Noh performer they captured is Toyohiko Sugiura. He ran through two sequences three times so that the team have multiple takes from which to pick the best. This work is part of a larger project on the analysis and preservation of dance in difference forms.

Removing the markers from the performer

The technology they use places reflectors on key joints in the person being captured. They then have a circle of lights/cameras that pick up the points from different angles over time. They do multiple takes as at times a reflector might be obscured and not register. Here are some photographs of Sugiura performing.

In motion

At the end of the performance

Dr. Hachimura has published about this research in a chapter titled “Digital Archiving of Dancing by Using Motion
Capture Technology,” in A. Kawashima, R. Akama, K. Yano, K. Hachimura, and M. Inaba eds., New Directions in Digital Humanities for Japanese Arts and Cultures, Nakanishiya Publishing, pp. 167-181, 2009. In that chapter he argues that,

For the research, education and creative practice, recording human body movements of dancing in various cultural environments and providing the data for public use with digital technologies are essential elements.

The digital humanities in the West, has been biased towards text as the bearer of culture. The foundational stories and early concerns of computing in the humanities are around concording and text analysis. Humanities computing has branched out to digitize other cultural forms, but even so we tend to focus on digitizing and creating databases of tangible cultural artefacts like paintings, archaeological sites, movies, and so on. By contrast, as I have written before, in Japan a large percentage of the traditional arts from the Bunrako to Noh are in the class of intangible cultural property. Intangible cultural traditions are supported aggressively in Japan through support to individual masters and organizations to support for preservation activities. The challenge with intangible heritage, however, is that you are not preserving an object, but a process or tradition of teaching a process. As the PDF of some slides on the subject puts it, the idea is,

NOT to order the people who hold those heritage to conserve it,

BUT to help the people to hand their heritage down to the next generation by themselves.

Likewise for researchers in Hachimura’s team the challenge with Noh is that they can’t just make a database of digital representations of an object, but they have to work out ways to intervene using digital tools that benefit understanding and preservation of the tradition. This involves negotiating with educators in the tradition and working with performers like Sugiura.

Applying digital techniques to intangible traditions also leads to some fascinating problems. How, for example, do you search for a unit of motion (kata) if you have a database of these forms? On this subject I heard a presentation by Worawat Choensawat, a PhD student of Hachimura’s on “Integrated dance body motion archiving system with motion capture and dance notation.” They have adapted Labanotation to recording Noh dance and developed a really cool LabanEditor tool with which you can write out a sequence of movements and then ask the system to animate the sequence on a Noh stage with a predefined Noh performance character. This system is being developed in consultation with Noh masters for educational uses.

The Labanotation tool, however, doesn’t solve the problem of searching though a database of motion capture sequences which are inevitably more authentic transcriptions of Noh performance. Choensawat, in his presentation went on to talk about a feature extraction technique for searching and comparing motion sequences in a database of motion like the CMU Graphics Lab Motion Capture Database.

As Hachimura admits in his chapter, there are limitations to the motion capture approach. Much of what makes a Noh performance is not being captured and the act of capturing changes the performance:

For the purpose of accurately recording three dimensional positions of various body parts, a dancer is compelled to perform under unusual circumstances in the sense that the dancer wears special body suits. This may cause a negative influence on their performance. Also, for dance archives, body motion by itself is not everything, and irrespective of the fact that the state of costumes and make-up are important, these cannot be recorded by a motion capture system alone.

I see this an illustration of a larger point – we always lose information when digitizing, even when digitizing text. We are always making choices about what to capture, what resolution to capture at and what contextual information to add as enrichment. In text digitization we have lost sight of the materiality of text and all that is lost because we work in a tradition that sees the string (sequence of characters) as what is important. A new edition of Hamlet with an edited text in a different material form is treated ontologically as Hamlet. This is not the case with animated Noh. It is not a Noh performance, partly because it was not developed in the traditional way. One can imagine that it might be possible to create an entirely automated Noh performance that to a viewer was indistinguishable from a real one, but it still wouldn’t be Noh in the sense of being a work in a tradition of training and performance.

Which brings us back to intangible cultural property. For the digital intervention to authentically contribute to an intangible tradition it needs to be woven into the tradition which is why Hachimura’s lab are working closely with masters. Traditions are not fixed – one of the things that makes an oral tradition different from a recording is that performers adapt to local conditions and change the story with the times. What Hachimura is doing is involving the custodians of the Noh traditions in their digital humanities work so as to find ways in which the digital can be woven into the tradition. The day may come when his tools are part of how Noh is “traditionally” taught or archived.