The Value of Open Beta
I was linked to an article with the following tagline: “Doubts raised on the value of public betas.” It’s essentially a commentary on open betas not being useful to the development process. It’s interesting, because I thought we’d figured this whole thing out back in about 2004. I’m not a panel, but I can say fairly definitively that open betas are very useful, in roughly these statistically-irrelevant percentages:
- 70% Marketing: It builds the hype and acts as a game demo, driving players to buy (or not buy) the game based on how positive their experience is during open beta.
- 20% Stress Test: While closed beta provides a good stress test, open beta is more variable and has a larger sampling of different machines and connections.
- 10% Sanity Check: Any hot-button issues that did not come to light during closed beta will become apparent in open beta.
Closed beta, I still believe, is extremely useful as long as you are handpicking the majority of your users. For example, if you target specific guilds or enthusiastic fans, you’re going to get useful feedback and bug reports.
The hype component can outdo the game demo component it seems. A lot of negative commentary can convince people to not even try it for themselves.
Overall I agree with you that beta’s are useful, but after seeing AAA titles like VG, AOC, and WAR come out I can’t help but question the beta process. The cost of failure is so high, and someone obviously dropped the ball somewhere which shouldn’t be happening with multi-million dollar projects.
I think this article is more in-line with closed rather than open beta, But one line which I completely agree with is. “What you get are hundreds of random messages, none in a useful format for the dev team, many without the steps to reproduce.”
There’s a huge gap between the intention of beta’s and the results your actually getting from them, and it’s up to the developers to design the format that will pull out information from beta players that they can work with. Sometimes players offer ridiculous suggestions that just won’t work, and other times I feel developers are afraid to admit the design doesn’t work and refuse to make adjustments. You can’t just say ok here’s your invite go play our game then go spout off on the forum with the problems you have.
I get the impression that developers under-estimate the difficulties of extracting valuable information through the beta process, but with the ones who know what they’re doing it will show up in their work.
I think people who come in to an open beta and expect their feedback to be acted on immediately are the ones who will be disappointed. Open beta is there to see how well the game will handle multitudes of people. That’s why they sell access to it (and / or the last phase of closed beta) with pre-orders or sometimes just make it free and open to all who wish to download a client. All the focused testing is out for major bugs and the only thing left is the durability of the servers and anything that’s a major showstopper.
I agree with your assessment all around.
It drives me nuts that open betas are still used as demos, though — because you’re absolutely right. Their current biggest function is to let players test drive a mostly-finished game and get hype out for the release client. I’ve said it before and I’ll see it again: Make a real frickin’ demo. It does not have to eat into your box sales if done right, and a good demo can be a powerful driver of new players.
There are so many MMOs that I’m not willing to drop $50 on because of how many times I’ve been burned by bad games, where I didn’t want to play the open beta for a variety of reasons (I don’t usually mess with betas unless I’m really hyped on a game), and if they had a simple 10-day trial with limited functionality or access, I might have downloaded it, been interested enough in the gameplay to pay for a copy, and then gotten the 10 friends I usually pull into a new game I’m psyched on to come with me.
I cannot understand the logic of letting everyone and their brother play your not-quite-finished-product and talk about it online, but then not let anyone else try it free without a special code or a special invite for the first 6-12 months of patches and bugfixes.
The biggest problem I see in modern MMO design (you could see sigil as the biggest example of this failure) is discounting beta feedback as “complaints” and not being able to filter the true issues out.
If you look at a majority of the changes to Vanguard that were made after the game had been bought by SOE and lost most of it’s “steam” they are suggestions/issues brought up during limited early open betas.
Developers I find are far too eager and far too able to say “oh that’s just because your not getting it” or “that’s not the concept we’re going for” or just the fallback “there is no time to fix/change/alter that.” Sigil with vanguard had the time and capability to fix many of the issues with vanguard and choose to disregard most beta feedback and steam ahead and paid the ultimate price for it.
Galidin