Bad Pop Social Science

Slate posted a piece on the academic study of pop culture.  It found that academics studied Buffy the Vampire Slayer most.  Well, actually, no, it found that more folks studied the Buffy than the Matrix, the Simpsons, Aliens or The Wire.

This led to a FB discussion of selection bias.  We can discuss the merits of these five and ponder why the Simpsons did so poorly (perhaps we need a consistent plot progression?), or why the Wire is under-valued yet again.  That the 2nd and 3rd Matrix movies sucked so much that they sucked all the air out of the studying the Matrix enterprise? But what is most obvious is that this "study" is that it ignored the big, enduring elements of pop culture that we have been obsessing about for years/generations: Star Wars, Star Trek (which is tossed off as an side), Lord of the Rings, and, more recently, Harry Potter.  Using just the Berkeley source, Star Wars appears to be ahead of Buffy.  Using scholar.google:

• Buffy produces 6k hits; 
• Star Trek: 30k; 
• Star Wars: 51k (affected by Reagan's naming of the Strategic Defense Initiative);
• Lord of the Rings: 22k; 
• Harry Potter: 32k.

To get a more systematic view, I used Publish or Perish, which relies on scholar google to build comparisons. If there are over a 100 papers, it just does the stats for one hundred (maybe I am lousy at setting the parameters, but this is a blogpost, not a submission to a journal).  There is plenty of error, of course, but there are some systematic patterns.

	Papers	Cites	Cites per paper	Cites per year	H index*	Most cited piece
Buffy TVS	100	4303	43	269	28	836
Harry Potter	83	6770	81	339	43	1296
Lord of the Rings	100	8339	83	83	33	1961
Star Trek	100	7192	72	153	37	926
Star Wars	100	9915	99	275	37	816
Simpsons	100	3164	32	44	24	507
The Matrix Trilogy	100	973	10	22	14	100
Alien Movies	100	370	4	2	9	54

Star Wars has the most citations but probably the most error given the aforementioned SDI bias.  What this does show is that the classics (old) double more or less Buffy while the new (Harry) has 50% more citations.  In terms of which papers have the most impact on average, again Star Wars  prevails but Lord of the Rings and Harry Potter lead the rest (and probably have fewer accidental cites along the way).  Harry Potter has the highest rate of cites per year since all of JK's stuff came out since 1997.  This produces the highest H-index.  Buffy performs quite well, certainly outclassing the Matrix because, well, quality does matter after all.   The Simpsons do pretty well.  Aliens?  Scary but not worthy of citations.  Star Trek is a steady performer, among the leaders across the border, and with fewer false positives than Star Wars.  The Wire, well, hard to measure since even restricting P&P searches to social sciences gives way too many non-The Wire hits (same goes for Breaking Bad and Mad Men).  I guess folks will have to settle for The Wire remaining the best TV program in most folks' minds and perhaps the best application of social science in a TV show.

Aside from the lesson that Slate does apparently poor pop social science, what can we learn from this exercise (other than my priorities are lousy--I should be doing something other than this--including my summer project of finally watching season one of Buffy)?  That we live in a golden age of pop culture and its over-analysis?  That Harry beats Buffy?  That Star Wars and Star Trek fans have something else over which to fight?  That the Matrix really did suck?  What conclusions do the readers have?

Of course, one could argue that I have some selection bias myself--that this was hardly a systematic study of pop culture.  There might be other books/tv programs/movies that get more analysis than these.  But I did do due diligence, web 2.0 style by crowdsourcing first. Certainly, one can do this better if one is writing a dissertation.  And surely someone is.

*  from P&P: H-index aims to measure the cumulative impact of a researcher's output by looking at the amount of citation his/her work has received. Publish or Perish calculates and displays the h index proper, its associated proportionality constant a (from N_c,tot = ah²), and the rate parameter m (from h ~ mn, where n is the number of years since the first publication).