Tuesday, June 12, 2012

Bad Pop Social Science

Slate posted a piece on the academic study of pop culture.  It found that academics studied Buffy the Vampire Slayer most.  Well, actually, no, it found that more folks studied the Buffy than the Matrix, the Simpsons, Aliens or The Wire.

This led to a FB discussion of selection bias.  We can discuss the merits of these five and ponder why the Simpsons did so poorly (perhaps we need a consistent plot progression?), or why the Wire is under-valued yet again.  That the 2nd and 3rd Matrix movies sucked so much that they sucked all the air out of the studying the Matrix enterprise? But what is most obvious is that this "study" is that it ignored the big, enduring elements of pop culture that we have been obsessing about for years/generations: Star Wars, Star Trek (which is tossed off as an side), Lord of the Rings, and, more recently, Harry Potter.  Using just the Berkeley source, Star Wars appears to be ahead of Buffy.  Using scholar.google:
  • Buffy produces 6k hits; 
  • Star Trek: 30k; 
  • Star Wars: 51k (affected by Reagan's naming of the Strategic Defense Initiative);
  • Lord of the Rings: 22k; 
  • Harry Potter: 32k.
To get a more systematic view, I used Publish or Perish, which relies on scholar google to build comparisons. If there are over a 100 papers, it just does the stats for one hundred (maybe I am lousy at setting the parameters, but this is a blogpost, not a submission to a journal).  There is plenty of error, of course, but there are some systematic patterns.

Papers Cites Cites per paper Cites per year H index* Most cited piece
Buffy TVS 100 4303 43 269 28 836
Harry Potter 83 6770 81 339 43 1296
Lord of the Rings 100 8339 83 83 33 1961
Star Trek 100 7192 72 153 37 926
Star Wars 100 9915 99 275 37 816
Simpsons 100 3164 32 44 24 507
The Matrix Trilogy 100 973 10 22 14 100
Alien Movies 100 370 4 2 9 54

Star Wars has the most citations but probably the most error given the aforementioned SDI bias.  What this does show is that the classics (old) double more or less Buffy while the new (Harry) has 50% more citations.  In terms of which papers have the most impact on average, again Star Wars  prevails but Lord of the Rings and Harry Potter lead the rest (and probably have fewer accidental cites along the way).  Harry Potter has the highest rate of cites per year since all of JK's stuff came out since 1997.  This produces the highest H-index.  Buffy performs quite well, certainly outclassing the Matrix because, well, quality does matter after all.   The Simpsons do pretty well.  Aliens?  Scary but not worthy of citations.  Star Trek is a steady performer, among the leaders across the border, and with fewer false positives than Star Wars.  The Wire, well, hard to measure since even restricting P&P searches to social sciences gives way too many non-The Wire hits (same goes for Breaking Bad and Mad Men).  I guess folks will have to settle for The Wire remaining the best TV program in most folks' minds and perhaps the best application of social science in a TV show.

Aside from the lesson that Slate does apparently poor pop social science, what can we learn from this exercise (other than my priorities are lousy--I should be doing something other than this--including my summer project of finally watching season one of Buffy)?  That we live in a golden age of pop culture and its over-analysis?  That Harry beats Buffy?  That Star Wars and Star Trek fans have something else over which to fight?  That the Matrix really did suck?  What conclusions do the readers have?

Of course, one could argue that I have some selection bias myself--that this was hardly a systematic study of pop culture.  There might be other books/tv programs/movies that get more analysis than these.  But I did do due diligence, web 2.0 style by crowdsourcing first. Certainly, one can do this better if one is writing a dissertation.  And surely someone is.

*  from P&P: H-index aims to measure the cumulative impact of a researcher's output by looking at the amount of citation his/her work has received. Publish or Perish calculates and displays the h index proper, its associated proportionality constant a (from Nc,tot = ah2), and the rate parameter m (from h ~ mn, where n is the number of years since the first publication).


Mrs. Spew said...

Yes, not only did I marry a geek, I married an academic geek, folks. You're all wrong. The biggest pop culture property studied by academics are the novels of Charles Dickens. And Ray Bradbury's Fahrenheit 451 probably beats out Buffy at least.

Anonymous said...

I think much about these values can be explained by looking at medium. I suspect there are not many Simpsons books that academics are studying. Film scholars will not be looking at Buffy (yes there's a Buffy film, but I mean, really.)

Steve Saideman said...

Dickens has not been pop culture for 100 years, but I take the point. If I had even more time to waste on this, I would have added Stephen King, Bradbury, and Asimov to the searches....