Comments on: The Value of Open Beta http://www.nerfbat.com/2010/03/18/the-value-of-open-beta/ Game design, development, and industry commentary by MMO Game Designer Ryan Shwayder. Fri, 03 Feb 2012 05:20:22 +0000 hourly 1 http://wordpress.org/?v=3.3.1 By: Galidin http://www.nerfbat.com/2010/03/18/the-value-of-open-beta/comment-page-1/#comment-91927 Galidin Sat, 03 Apr 2010 04:56:19 +0000 http://www.nerfbat.com/?p=1974#comment-91927 The biggest problem I see in modern MMO design (you could see sigil as the biggest example of this failure) is discounting beta feedback as "complaints" and not being able to filter the true issues out. If you look at a majority of the changes to Vanguard that were made after the game had been bought by SOE and lost most of it's "steam" they are suggestions/issues brought up during limited early open betas. Developers I find are far too eager and far too able to say "oh that's just because your not getting it" or "that's not the concept we're going for" or just the fallback "there is no time to fix/change/alter that." Sigil with vanguard had the time and capability to fix many of the issues with vanguard and choose to disregard most beta feedback and steam ahead and paid the ultimate price for it. Galidin The biggest problem I see in modern MMO design (you could see sigil as the biggest example of this failure) is discounting beta feedback as “complaints” and not being able to filter the true issues out.

If you look at a majority of the changes to Vanguard that were made after the game had been bought by SOE and lost most of it’s “steam” they are suggestions/issues brought up during limited early open betas.

Developers I find are far too eager and far too able to say “oh that’s just because your not getting it” or “that’s not the concept we’re going for” or just the fallback “there is no time to fix/change/alter that.” Sigil with vanguard had the time and capability to fix many of the issues with vanguard and choose to disregard most beta feedback and steam ahead and paid the ultimate price for it.

Galidin

]]> By: Cameron Sorden http://www.nerfbat.com/2010/03/18/the-value-of-open-beta/comment-page-1/#comment-91810 Cameron Sorden Sat, 20 Mar 2010 15:09:17 +0000 http://www.nerfbat.com/?p=1974#comment-91810 It drives me nuts that open betas are still used as demos, though -- because you're absolutely right. Their current biggest function is to let players test drive a mostly-finished game and get hype out for the release client. I've said it before and I'll see it again: Make a real frickin' demo. It does not have to eat into your box sales if done right, and a good demo can be a powerful driver of new players. There are so many MMOs that I'm not willing to drop $50 on because of how many times I've been burned by bad games, where I didn't want to play the open beta for a variety of reasons (I don't usually mess with betas unless I'm really hyped on a game), and if they had a simple 10-day trial with limited functionality or access, I might have downloaded it, been interested enough in the gameplay to pay for a copy, and then gotten the 10 friends I usually pull into a new game I'm psyched on to come with me. I cannot understand the logic of letting everyone and their brother play your not-quite-finished-product and talk about it online, but then not let anyone else try it free without a special code or a special invite for the first 6-12 months of patches and bugfixes. It drives me nuts that open betas are still used as demos, though — because you’re absolutely right. Their current biggest function is to let players test drive a mostly-finished game and get hype out for the release client. I’ve said it before and I’ll see it again: Make a real frickin’ demo. It does not have to eat into your box sales if done right, and a good demo can be a powerful driver of new players.

There are so many MMOs that I’m not willing to drop $50 on because of how many times I’ve been burned by bad games, where I didn’t want to play the open beta for a variety of reasons (I don’t usually mess with betas unless I’m really hyped on a game), and if they had a simple 10-day trial with limited functionality or access, I might have downloaded it, been interested enough in the gameplay to pay for a copy, and then gotten the 10 friends I usually pull into a new game I’m psyched on to come with me.

I cannot understand the logic of letting everyone and their brother play your not-quite-finished-product and talk about it online, but then not let anyone else try it free without a special code or a special invite for the first 6-12 months of patches and bugfixes.

]]> By: Wiqd http://www.nerfbat.com/2010/03/18/the-value-of-open-beta/comment-page-1/#comment-91807 Wiqd Sat, 20 Mar 2010 10:01:36 +0000 http://www.nerfbat.com/?p=1974#comment-91807 I think people who come in to an open beta and expect their feedback to be acted on immediately are the ones who will be disappointed. Open beta is there to see how well the game will handle multitudes of people. That's why they sell access to it (and / or the last phase of closed beta) with pre-orders or sometimes just make it free and open to all who wish to download a client. All the focused testing is out for major bugs and the only thing left is the durability of the servers and anything that's a major showstopper. I agree with your assessment all around. I think people who come in to an open beta and expect their feedback to be acted on immediately are the ones who will be disappointed. Open beta is there to see how well the game will handle multitudes of people. That’s why they sell access to it (and / or the last phase of closed beta) with pre-orders or sometimes just make it free and open to all who wish to download a client. All the focused testing is out for major bugs and the only thing left is the durability of the servers and anything that’s a major showstopper.

I agree with your assessment all around.

]]> By: Xorakis http://www.nerfbat.com/2010/03/18/the-value-of-open-beta/comment-page-1/#comment-91769 Xorakis Thu, 18 Mar 2010 18:46:15 +0000 http://www.nerfbat.com/?p=1974#comment-91769 Overall I agree with you that beta's are useful, but after seeing AAA titles like VG, AOC, and WAR come out I can't help but question the beta process. The cost of failure is so high, and someone obviously dropped the ball somewhere which shouldn't be happening with multi-million dollar projects. I think this article is more in-line with closed rather than open beta, But one line which I completely agree with is. “What you get are hundreds of random messages, none in a useful format for the dev team, many without the steps to reproduce.” There's a huge gap between the intention of beta's and the results your actually getting from them, and it's up to the developers to design the format that will pull out information from beta players that they can work with. Sometimes players offer ridiculous suggestions that just won't work, and other times I feel developers are afraid to admit the design doesn't work and refuse to make adjustments. You can't just say ok here's your invite go play our game then go spout off on the forum with the problems you have. I get the impression that developers under-estimate the difficulties of extracting valuable information through the beta process, but with the ones who know what they're doing it will show up in their work. Overall I agree with you that beta’s are useful, but after seeing AAA titles like VG, AOC, and WAR come out I can’t help but question the beta process. The cost of failure is so high, and someone obviously dropped the ball somewhere which shouldn’t be happening with multi-million dollar projects.

I think this article is more in-line with closed rather than open beta, But one line which I completely agree with is. “What you get are hundreds of random messages, none in a useful format for the dev team, many without the steps to reproduce.”

There’s a huge gap between the intention of beta’s and the results your actually getting from them, and it’s up to the developers to design the format that will pull out information from beta players that they can work with. Sometimes players offer ridiculous suggestions that just won’t work, and other times I feel developers are afraid to admit the design doesn’t work and refuse to make adjustments. You can’t just say ok here’s your invite go play our game then go spout off on the forum with the problems you have.

I get the impression that developers under-estimate the difficulties of extracting valuable information through the beta process, but with the ones who know what they’re doing it will show up in their work.

]]> By: JuJutsu http://www.nerfbat.com/2010/03/18/the-value-of-open-beta/comment-page-1/#comment-91767 JuJutsu Thu, 18 Mar 2010 16:29:18 +0000 http://www.nerfbat.com/?p=1974#comment-91767 The hype component can outdo the game demo component it seems. A lot of negative commentary can convince people to not even try it for themselves. The hype component can outdo the game demo component it seems. A lot of negative commentary can convince people to not even try it for themselves.

]]>