Midterm Forecast Roundup: Semi-Complete Lean/Lock Results

Throughout the election cycle I was keeping track of my predictions for several of the House, Senate and gubernatorial races using Slate’s Lean/Lock game. Readers can get a refresher on my methodology here, and Slate’s rules of the game are available here. Basically, the point of the game was to either “Lean” or “Lock” your predictions towards candidates, and if your picks are copacetic with Pollster.com’s trackers, you get either 15 points per day per candidate that you’ve “Leaned” to, or twenty points per day per candidate you’ve “Locked.” Leans can be changed easily, but Locking implies that you’re basically certain you’ve made the right choice.

I made almost all of my picks on September 7, when I became aware of the game. Unfortunately, the game started August 15, meaning I lost out on a boatload of points. To accommodate latecomers Lean/Lock awarded 7.5 points per retroactive day missed per candidate leaned or locked to correctly, less than half of what I would’ve earned as I never “Leaned,” only “Locked.” However, I’m quite pleased with the performance of my dinky little formula.

The formula correctly predicted the outcomes of sixteen out of seventeen Senate races, five out of six gubernatorial races, and eleven out of eleven House races. By contrast, the average Lean/Lock user correctly predicted the outcomes of twelve out of seventeen Senate races, four out of six gubernatorial races, and ten out of eleven House races. Additionally, most Lean/Lock users appear to have been much more conservative in their estimates, as my score rose much more quickly than the average, implying a higher degree of “Locked” certainty.

Scoring isn’t quite done yet, but as it stands, my score is 41,562 out of a maximum of 45,757; the average score is 23,823, the median is 25,740, and the leaderboard bottoms out at 44,525.

Something interesting is that my errors never overlapped with the average Lean/Lock user’s; this means that this time around, Intrade accounts and public opinion polls had their greatest errors in completely different races. But this also means that, if Lean/Lock users relied on polling averages – a methodology that would guarantee them the maximum daily score – then Intrade users produced markedly more accurate predictions than poll results.

Lean/Lock’s scoring method seemed to rely on the implicit assumption that polls were accurate. Players who felt that polls were inaccurate indicators of final outcomes – as they generally were in the cases of the Wisconsin and Nevada Senate races – sacrificed points for final-day accuracy. This aspect of the scoring was unclear to me when I started playing, but I’m still quite pleased with how I did. My dinky little error-weighted Intrade/poll (“E-WIP,” perhaps?) methodology helped me successfully avoid several incorrect estimates that I would’ve made had I relied solely on polls, and many more errors had I relied on the average Lean/Lock user.

This model has two obvious shortcomings: its methodological rigor is low and it didn’t provide exact percentages, only binary outcomes. The guiding assumptions of my method were, essentially:

  • Polls do not need to be weighted by proximity to elections; they don’t become terribly more accurate with decreased temporal proximity
  • Error rates are stable, or at least predictable – it was unlikely polls would out-perform Intrade this election

That’s it. Despite this, I’m pretty proud anyway. Curious readers can check out The Monkey Cage’s summary of how the models I mentioned in my previous post did. Mine isn’t really comparable to these because of how it was devised, but suffice to say, I had a lot of fun with the process.


Tags: , ,

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: