What are the most overrated films?

“Overrated” and “underrated” are slippery terms to try to quantify. An interesting way of looking at this, I thought, would be to compare the reviews of film critics with those of Joe Public, reasoning that a film which is roundly-lauded by the Hollywood press but proved disappointing for the real audience would be “overrated” and vice versa.

To get some data for this I turned to the most prominent review aggregator: Rotten Tomatoes. All this analysis was done in the R programming language, and full code to reproduce it will be attached at the end.

Rotten Tomatoes API

This API is nicely documented, easy to access and permissive with rate limits, as well as being cripplingly restrictive in what data is presents. Want a list of all films in the database? Nope. Most reviewed? Top rated? Highest box-office takings? Nope.

The related forum is full of what seem like simple requests that should be available through the API but aren’t: top 100 lists? Search using mulitple IDs at once? Get audience reviews? All are unanswered or not currently implemented.

So the starting point (a big list of films) is actually kinda hard to get at. The Rube Golbergian method I eventually used was this:

1. Get the “Top Rentals” list of movie details (max: 50)
2. Search each one for “Similar films” (max: 5)
3. Get the unique film IDs from step 2 and iterate

(N.B. This wasn’t my idea but one from a post in the API forums, unfortunately didn’t save the link.)

In theory this grows your set of films at a reasonable pace, but in reality the number of unique films being returned was significantly lower (shown below). I guess this was due to pulling in “walled gardens” to my dataset, e.g. if a Harry Potter film was hit, each further round would pull in the 5 other films as most similar.

Results

Here’s an overview of the critic and audience scores I collected through the Rotten Tomatoes API, with some outliers labelled.

On the whole it should be noted that critics and audience agree most of the time, as shown by the Pearson correlation coefficient between the two scores (0.71 across >1200 films).

Click for interactive version.

Update:

I’ve put together an interactive version of the same plot here using the rCharts R package. It’ll show film title and review scores when you hover over a point so you know what you’re looking at. Also I’ve more than doubled the size of the film dataset by repeating the above method for a couple more iterations — take a look!

Most underrated films

Using our earlier definition it’s easy to build a table of those films where the audience ending up really liking a film that was panned by critics.

Scores are shown out of 100 for both aggregated critics and members of Rotten Tomatoes.

Somewhat surprisingly, the top of the table is Facing the Giants (2006), an evangelical Christian film. I guess non-Christians might have stayed away, and presumably it struck a chord within its target demographic — but after watching the trailer, I’d probably agree with the critics on this one.

This showed that some weighting of the difference might be needed, at the very least weighting by number of reviews, but the Rotten Tomatoes API doesn’t provide that data.

In addition the Rotten Tomatoes page for the film, shows a “want to see” percentage, rather than an audience score. This came up a few times and I’ve seen no explanation for it, presumably “want to see” rating is for unreleased films, but the API returns a separate (and undisclosed?) audience score for these films also.

Above shows a “want to see” rating, different to the “liked it” rating returned by the API and shown below. Note: these screenshots from RottenTomatoes.com are not CC licensed and is shown here under a claim of Fair Use, reproduced for comment/criticism.

Looking over the rest of the table, it seems the public is more fond of gross-out or slapstick comedies (such as Diary of a Mad Black Woman (2005), Grandma’s boy (2006)) than the critics. Again, not films I’d jump to defend as underrated. Bad Boys II however…

Most overrated films

Here we’re looking at those films which the critics loved, but paying audiences were then less enthused.

As before, scores are out of 100 and they’re ranked by difference between audience and critic scores.

Strangely the top 15 (by difference) contains both the original 2001 Spy Kids and the sequel Spy Kids 2: The Island of Lost Dreams (2002). What did critics see in these films that the public didn’t? A possibility is bias in the audience reviews collected, the target audience is young children for these films and they probably are underrepresented amongst Rotten Tomatoes reviewers. Maybe there’s even an enrichment for disgruntled parent chaperones.

Thankfully, though, in this table there’s the type of film we might more associate with being “overrated” by critics. Momma’s Man (2008) is an indie drama debuted at the 26th Torino Film Festival. Essential Killing is a 2010 drama and political thriller from Polish director and screenwriter Jerzy Skolimowski. 

There’s also a smattering of Rom-Coms (Friends with Money (2006), Splash (1984)) — if the API returned genre information it would be interesting to look for overall trends but, alas. Additional interesting variables to consider might be budget, the lead, reviews of producer’s previous films… There’s a lot of scope for interesting analysis here but it’s currently just not possible with the Rotten Tomatoes API.

 Caveats / Extensions

The full code will be posted below, so if you want to do a better job with this analysis, please do so and send me a link! 🙂

• Difference is too simple a metric. A better measure might be weighted by (e.g.) critic ranking. A film critics give 95% but audiences 75% might be more interesting than the same points difference between a 60/40 rated film.
• There’s something akin to a “founder effect” of my initial chosen films that makes it had to diversify the dataset, especially to films from previous decades and classics.
• The Rotten Tomatoes API provides an IMDB id for cross-referencing, maybe that’s a path to getting more data and building a better film list.

Full code to reproduce analysis

Note: If you’re viewing this on r-bloggers, you may need to visit the Benomics version to see the attached gist.

	library("RCurl")
	library("jsonlite")
	library("ggplot2")
	library("RColorBrewer")
	library("scales")
	library("gridExtra")
	api.key <- "yourAPIkey"
	rt <- getURI(paste0("http://api.rottentomatoes.com/api/public/v1.0/lists/dvds/top_rentals.json?apikey=&quot;, api.key, "&limit=50"))
	rt <- fromJSON(rt)
	title <- rt$movies$title
	critics <- rt$movies$ratings$critics_score
	audience <- rt$movies$ratings$audience_score
	df <- data.frame(title=title, critic.score=critics,
	audience.score=audience)
	# Top 50 rentals, max returnable
	ggplot(df, aes(x=critic.score, y=audience.score)) +
	geom_text(aes(label=title, col=1/(critic.score – audience.score)))
	# how can we get more? similar chaining
	# STILL at most 5 per film (sigh)
	getRatings <- function(id){
	sim.1 <- getURI(paste0("http://api.rottentomatoes.com/api/public/v1.0/movies/&quot;,
	id, "/similar.json?apikey=",
	api.key, "&limit=5"))
	sim <- fromJSON(sim.1)
	#cat(sim$movies$title)
	d <- data.frame(id = sim$movies$id,
	title = sim$movies$title,
	crit = sim$movies$ratings$critics_score,
	aud = sim$movies$ratings$audience_score)
	return(d)
	}
	rt.results <- function(idlist){
	r <- sapply(unique(as.character(idlist)), getRatings, simplify=F)
	r <- do.call(rbind, r)
	return(r)
	}
	# Maybe this could be done via a cool recursion using Recall
	r1 <- rt.results(rt$movies$id)
	r2 <- rt.results(r1$id)
	r3 <- rt.results(r2$id)
	r4 <- rt.results(r3$id)
	r5 <- rt.results(r4$id)
	r6 <- rt.results(r5$id)
	r7 <- rt.results(r6$id)
	f <- function(x)
	(5**x)-1
	# Fig. 1: Number of films gathered via recursive descent
	# of 'similar films' lists.
	dev.off()
	options(scipen=99)
	pdf(4, 4, file="rottenTomatoHits.pdf")
	par(cex.axis=.7, pch=20, mar=c(4,3,1,1), mgp=c(1.5,.3,0), tck=-.02)
	plot(1:7, f(1:7), type="b", xlab="Recursions", ylab="Number of hits",
	log="y", col=muted("blue"), lwd=2, ylim=c(4, 1e5))
	lines(1:7, c(nrow(r1), nrow(r2), nrow(r3), nrow(r4), nrow(r5),
	nrow(r6), nrow(r7)), type="b", col=muted("red"), lwd=2)
	legend("bottomright", col=c(muted("blue"), muted("red")), pch=20, lwd=2,
	legend=c(expression(Max~(5^x)), "Realised"), bty="n", lty="47")
	dev.off()
	r <- rbind(r1, r2, r3, r4, r5, r6, r7)
	# 1279 unique films
	ru <- r[!duplicated(as.character(r$id)),]
	# Films with insufficient critics reviews get -1 score
	ru[which(ru$crit == -1),]
	ru <- ru[ru$crit != -1,]
	ru$diff <- ru$crit – ru$aud
	pcc <- cor(ru$crit, ru$aud)
	# Overview figure: Show all critics vs. audience scores
	# and highlight the titles of outliers
	svg(7, 6, file="overview.svg")
	ggplot(ru, aes(x=crit, y=aud, col=diff)) +
	geom_point() +
	coord_cartesian(xlim=c(-10,110), ylim=c(-10,110)) +
	scale_color_gradientn(colours=brewer.pal(11, "RdYlBu"),
	breaks=seq(-60,40, length.out=11),
	labels=c("Underrated", rep("", 4),
	"Agree", rep("", 4),
	"Overrated")) +
	geom_text(aes(label=ifelse(diff < quantile(diff, .005) |
	diff > quantile(diff, .995), as.character(title), ""),
	size=abs(diff)),
	hjust=0, vjust=0, angle=45) +
	scale_size_continuous(range=c(2,4), guide="none") +
	labs(list(x="Critic's score", y="Audience score",
	col="")) +
	annotate("text", 3, 3,
	label=paste0("rho ==", format(pcc, digits=2)),
	parse=T)
	dev.off()
	tab <- ru
	colnames(tab) <- c("id", "Title", "Critics", "Audience", "Difference")
	# Most underrated films:
	grid.newpage()
	grid.draw(tableGrob(tab[order(tab$Difference),][1:15,-1], show.rownames=F))
	# Most overrated:
	grid.newpage()
	grid.draw(tableGrob(tab[order(tab$Difference, decreasing=T),][1:15,-1], show.rownames=F))

view raw

overratedFilms.R

hosted with ❤ by GitHub