Text classifying software operating on a "bag-of-words" model of a text will accurately classify movie reviews as positive or negative in more than 80% of cases. When you apply the same methods to political debates, the accuracy rate drops to 65%. What causes the performance drop? Can we construct a general-purpose political opinion classifier?

Text classification works by extracting words or other low-level linguistic phenomena that are habitually associated with particular kinds of writing. We found that the vocabulary of movie reviews provides a more effective basis for text classification because it provides more direct pointers to the author's sentiments. Using a large news corpus as the contrast group, we found that movie reviews have a strong emotional charge while the average affect level of political debates is similar to that of news articles.

If this explains the performance drop, which means the political opinion classifier is not effective, why is it still better than random guess? After examining the part-of-speech groups of both feature sets we found that the learned classifier is a mixture of relevant concepts - opinion, topic, ideology and social network, all partly contributing to the recognition of debate polarity. Opinions and topics are considered as two orthogonal dimensions in movie review classification. The relevant concept interference complicates political opinion classification. Controlling such interference is crucial to evaluate the generalizability of political opinion classifiers. However it has been neglected in current studies.

Daniel Diermeier is the IBM Distinguished Professor of Regulation and Competitive Practice at the Kellogg Graduate School of Management. (http://www.kellogg.northwestern.edu/faculty/diermeier/personal/bio.htm)

Stefan Kaufmann is Assistant Professor of Linguistics at Northwestern University (http://www.ling.northwestern.edu/~kaufmann/sk_cv.pdf)

Bei Yu has a PhD from the Graduate School of Library and Information Science at UIUC and is a postdoctoral fellow at the Kellog Graduate School of Management