TL;DR: I made a Naive Bayes based difficulty estimator for t...

TL;DR: I made a Naive Bayes based difficulty estimator for the game Sentinels of the Multiverse.

I really like the card game Sentinels of the Multiverse by +Christopher Badell, +Paul Bender and +Adam Rebottaro. It's a cooperative superhero game, but different combinations of heroes, villains, and environments can make for somewhat hard to predict difficulty level.

Fortunately some folks started a project to log details of a large number of plays of Sentinels, providing the raw data to do a difficulty estimator. I wrote a Naive Bayes win/loss "classifier" to yield a scoring system which lets you predict the likely difficulty of a particular combination.

http://x.gray.org/sentinels-of-the-multiverse-difficulty-scores.html

Sentinels of the Multiverse Difficulty Score System Sentinels of ...

Nate Gaylinn - 2013-03-31 21:07:19-0400

Neat! Sounds like that kind of thing would make a great tool for play testing a new game :) I always assumed people were still just tweaking the rules by hand, but maybe I'm wrong--have you heard of others doing analyses like this?

Matthew Gray - 2013-03-31 21:15:22-0400

The problems I'd anticipate with using it for testing a new game is twofold:

1) It really requires a lot of data. Only 2000 games in the training set is enough, but isn't way too much, given the number of parameters.

2) It is hard to compensate for change over time (eg, rules adjustments during development).

And, no I haven't heard about anyone doing this kind of tuning. I've heard of people running their game in simulation to tune certain parameters, but that's a bit different.

Nate Gaylinn - 2013-03-31 21:18:11-0400 - Updated: 2013-03-31 21:24:21-0400

Mmm, good point. The simulation idea seems pretty cool! Doesn't help with the problem of trying to learn how people would play, though, which I imagine is the biggest problem.
Even if it's not much, taking an analytical approach (ie, recording all the details of how each test plays out) for later analysis seems like a cool idea. Perhaps very tedious, but hey ;)

Felix Rodriguez - 2013-03-31 21:41:12-0400

Something seems oddly wrong here... I would not call Miss Information the easiest bad guy.

That makes our most recent play a little crazy:
Miss Information + Final Wasteland + 5 Players + Team Leader Tachyon + Omnitron-X + Haka + Chrono Ranger + Ra. And that scores a ridiculous -337. And, of course, we lost...

Matthew Gray - 2013-03-31 21:52:14-0400

+Felix Rodriguez, I'd agree based on that game, but that's a one-game anecdote. There's certainly possibility of strong sample bias in the data, but in the Sentinels Statistics Project data, players have a >90% win rate against her, with a variety of heroes, over more than 50 games.

Most importantly, we played on advanced, and the (very very sparse) data suggests that Miss Information takes a huge step up (about 100 difficulty points) going to advanced, compared to the typical villain who seems to go up about 75. We also played Time Cataclysm, not Final Wasteland, bumping it up a bit, but still. I get a total difficulty of that game at -180, and we lost, and it wasn't that close. But, some slightly different card draws, and maybe nuking that first Clue and the game might have gone differently.

In my reshare on +mkgray Board Games, I posted some further thoughts on that particular game.

Matthew Gray - 2013-03-31 23:17:45-0400

Btw, +Felix Rodriguez, I actually had all of this ready to post about yesterday, but I had the exact same reaction as you, given that game where she crushed us. So, I held off posting til I managed to double check the numbers, and further verified against other more basic metrics like overall loss rate by villain, which also unambiguously gives her the "easiest villain to defeat" title.

Given how (relatively) badly we lost, I don't want it to be true. ;)

Jeremy Hoffman - 2013-04-01 01:50:58-0400

It seems like an easy way to account for interactions/cross-terms in a simple Naive Bayes estimator would be to feed it with pairs of possible heroes/villains/environments. Although I don't know if the data is too sparse for that.

Felix Rodriguez - 2013-04-01 08:30:49-0400

I think there are two factors at work with Miss Information that are hard to take into account: First, we did play her on advanced mode. And Second, she is unique in the sense that more people makes her harder.

Still...

Philippe Beaudoin - 2013-04-01 08:33:31-0400 - Updated: 2013-04-01 08:35:06-0400

Great idea Matthew! I have used simulated games and simple statistics to tune my designs and they have proven very useful as creation tools. Even with relatively naive AIs it gives a baseline for input/output rate of the various resources to keep the game tight, or for initial adjustments to keep the game fair for each player. I've found that simulation makes the real playtests much more about the core of the game than the small details, which is a big plus...

Anyway, all that to say that any tool that can help with the analysis of a large number of recorded games would definitely be useful to me. :)

Daniel Lew - 2013-05-03 22:19:53-0400

Hey, would you mind if I incorporated some of this data into a SotM randomizer?

Matthew Gray - 2013-05-04 16:12:32-0400

You are welcome to use this data. I'd appreciate a link/credit if there's an appropriate place, but don't require it.

Daniel Lew - 2013-05-04 16:25:46-0400

Sure, will do, thanks.

Matthew Gray - 2013-05-05 13:20:52-0400

+Felix Rodriguez, so I've rerun the numbers now that there's a bunch more games with the new stuff in the logs, and Miss Information on Basic is still one of the easiest villains, but on Advanced, she is one of the hardest (harder then The Chairman on Basic).

Felix Rodriguez - 2013-05-05 20:34:17-0400

Ha! Redemption!

Matthew Gray - 2013-05-06 00:12:06-0400

And, now I've published the updated model.

Daniel Lew - 2013-05-06 07:53:33-0400

Any chance you'd publish an easily parseable format (like JSON or CSV) as well? If it's going to update regularly that would make my task easier.

Matthew Gray - 2013-05-06 09:04:47-0400

+Daniel Lew, good point. I added a link to a JSON formatted version of the data at the bottom of the page. Let me know if there's anything about the format that you'd like tweaked.

http://x.gray.org/sentinels.json

Daniel Lew - 2013-05-06 09:06:25-0400

That format is perfect, thanks!

Daniel Lew - 2013-05-06 09:44:50-0400 - Updated: 2013-05-06 09:46:44-0400

Actually if you want to get technical, it's not valid JSON, due to the trailing commas in the arrays. A lenient parser can figure it out, though. Handy tool if you want to fix: http://jsonlint.com/

Daniel Lew - 2013-05-09 10:26:54-0400

An early build of the Android app I've been working on, if you're interested:

Screenshots here: http://imgur.com/a/ndx0K

APK: http://idunnolol.com/tmp/SentinelsRandomizer.apk