Phantasmal: The Journey to Full Release in Steam & AB Testing

Phantasmal: The Journey to Full Release in Steam & AB Testing

A while back I decided to do a talk at NZGDC about my experience with releasing a title to Steam for our survival horror roguelike Phantasmal.

While I put up the slides for the general public, I thought I’d write up a recap.

There was a heck of a lot of content so I’ll make this into a multi-part article.

The key things we learned were that:

  • We were in over our heads!
  • We could always do more marketing
  • Proper planning and organization go a long way

When I say we were in over our heads, what I mean is that there were so many moving parts to indie game development.  Making the game itself required competency in programming, level design, 3D Art, UI, audio, testing, analytics and a number of other things.  Each one of those competencies have sub-competencies and specializations as well.

And that is just the game development aspect.  If you take one step up, you also need to worry about marketing, business administration, project management, and all the other joys that go with running a small business.

There is simply no way to master all of these things with a small team, so we had to pick certain things to focus on, but we still had to have adequate cover across the breadth of the project.

For the core team, although most of us were largely noobs at game development, one of our key strengths was having worked in corporate IT for many years.  This gave us mental fortitude from having been through many nightmare projects as well as organizational discipline and planning skills.

So while the game wasn’t a massive financial success, I felt that we were extremely successful in a number of areas.  One of those areas was actually releasing the game on time and in a mostly functional state.

The following section covers some of the things that I believe we did well at and some of the things that weren’t so hot.

1 Things that went well:

1.1 A/B Testing

We used A/B testing in Facebook and Twitter to identify which pictures and videos the general public liked the most.  From the results, we refined a new set of media, and then ran more rounds of A/B Testing.  Once we had the best videos and images, we used these for our final Steam storefront media.

The images below are two rounds of AB testing we did for our capsule image (thumbnail) for the Steam store.


The first row was the first round, and the second row was the second.  We found in the first round that people liked the image on the far left, so for round 2 we made a bunch of variations on that.

Similarly for the videos, we started with 4 variations:

  • A creepy walk
  • A combat heavy video
  • A video with just jump scares
  • A build up to a jump scare

Far more people liked the creepy walk, so we did variations on that:

  • Narrated version
  • No narration
  • Pop-up text
  • A little bit of narration


We found that people liked the version with no narration at all – more people would watch the entire video right to the end.

Here are the results from our second round of video testing in Facebook.


Anyway, it’s a big subject, so I’ll cover the rest of the presentation in subsequent articles – hope you enjoyed it!


Leave a Reply

Your email address will not be published. Required fields are marked *

Wordpress SEO Plugin by SEOPressor