BEHIND THE QUIZ

We Marked Ourselves

…and came close to getting the questions just right.

LLast time we shared what you told us in the survey, and this time we are turning the microscope on ourselves. Before night one we rated every question Easy, Medium or Hard, and afterwards we counted how many of the 15 teams got each one right to see how it actually played. That gave us 51 questions (shout-out questions excluded).

Easy, Medium and Hard

Across all the answers given on the night, 83% of the answers to our Easy questions were right, 49% to our Medium questions and 38% to our Hard questions. All three landed inside their own band, so we feel we got very close on this one, although Hard at 38% only just stayed under the 40% line.

The table lines up what we called each question against how it actually played, and the green cells are the ones we got right. The column to look at is Played Expert, where 4 questions ended up, which means fewer than 15% of teams got them right. We had rated 3 of them Hard and 1 of them Medium. We have put those 4 in red because that is something we will aim not to do going forward. You can also see that 7 of our 15 Hard questions played as Medium.

Round by round

We aim for roughly 60% right in every round, and the night as a whole came in at 57%. Every round landed somewhere between 49% and 66%, which tells us we are pretty close to where we want to be, and that the next step is getting more of our individual questions rated correctly.

Looking question by question, you all did really well on the food and geography questions, which is great as that gives us a bead on how to create sets of questions for future rounds.

Photo, text and a few beers

Sliced by format, Photo came in at 61% across 13 questions and Text (Questions) at 55% across 31, so both sat near the 60% mark where we want them. The blind beer tasting came in at 48% across the 4 beers, which is a bit short, but that is kind of as it should be because the tasting is meant to be harder and it was something unique and challenging.

Video came in at 47% but we only had 2 of those, and Sound came in at 100% from a single question, so we are not reading anything into either until we have run a lot more of both.

Every question

The chart below lists every question round by round with a bar for the share of teams that got it right, how many of the 15 that was, our rating, the rating it actually played at, and a coloured dot for how well those two matched. A question counts as Easy if 75% of teams or more got it, Medium if 40% to 74% got it, Hard if 15% to 39% got it, and Expert if fewer than 15% got it but at least one team did.

Each question then gets a fit, On target if the actual rating matches ours, A bit off if it is one step out, and Way off if it is far enough out that we flag it as mis-rated. Of the 51 questions, 25 were On target, 13 were A bit off and 13 were Way off, so a run of On target dots in a round means we read the room well there and a cluster of the other two means we had blind spots.

REVEAL THE NUMBERS

The ones we got wrong

Here are the 13 we flagged as Way off, and 8 of them played harder than we rated them while 5 played easier. We are not going to pretend this is a science, because deciding that a question is Easy, Medium or Hard is really subjective.

That is the entire point of this exercise, and every one of these helps us tighten our game so that you have a fun but challenging night and hopefully learn something new along the way.

The extremes

The Bookends below show the 3 questions that every single team got right and the 6 questions that 20% of teams or fewer got right, with one square for each team so you can see exactly how close each one came.

I’m a Data Nerd!

If you made it this far, then you and I might be kindred data nerd spirits. 😂

Really though, thank you for reading all the way through, and if you have any questions about a particular chart or data view, please feel free to send me a note through the contact form or email me directly at [email protected].

Looking forward to next time.
-The BnB Team