[Tutorial] How To Be a Good Beta Tester

Beta testers, one of the most useful tools for developers, and essential for quality assurance. They are a necessity for any self-respecting developer. However, as I’ve started to notice over the years, it seems that the Sonic hacking scene is full of incompetent, clueless and utterly useless beta testers. Aside from a few selection of people, all I get is “it’s good”, “I like it”, and “nice work” when I’m really looking for criticism to improve my work. It, indeed, seems that a popular belief says that beta testing is all fun and games.

It’s not.

When we ask you to beta test, we aren’t looking for you to suck our e-dicks. We rather ask you to help us find flaws in our projects. Bugs, bad level design, something that isn’t executed well, or even just general ideas for improvements. Unfortunately, that’s not what most people are after (helping us), they just seem to be wanting to play something for fun. But the reality is, nobody wants to just share you an early beta of a hack for you to play before everyone else can just for the heck of it. Beta testing is you playing the product over and over and over to find flaws and report them back to the developer(s).

Now! If I made you feel guilty for your complete incompetence at being helpful, good! That was the intention. But fear not, because I will guide you become the opposite: useful. So, without any further ado, let’s start!


Step 0: Can You Be a Beta Tester?

Before we start with the actual help, there are a few things about being a beta tester that are obligatory to for you to be able to do, and without being able to fulfill all of them you should reconsider your capability for the job.

• Time Devotion:
  Being an efficient and helpful beta tester means you must devote a great portion of your free time to the job. It is usually good to be available more often than once per week, and at least an hour or two at a time. It depends entirely on the developer(s), project, and possibly other any other beta testers involved. But generally speaking, two or more times per week, for few hours at once, should be a good average.
• Non-disclosure obligation:
  Imagine a social worker and client relationship. The social worker is legally obliged to not disclose certain types of information about the client to anyone without permission from the client. The very same thing applies to the beta tester and developer relationship. While this, as far as to my knowledge, cannot be legally enforced, it is generally accepted as a rule within workplaces and non-workplaces alike. This means, not even telling your best friend about the most insignificant detail, unless consent by the developer(s) is given.
• Patience:
  Like with anything, patience is necessary with beta testing as well. The developer(s) may want to test or adjust a feature, and that may need frequent updates. If you are in this kind of situation, it may frustrate you a bit, but remember you are only helping by providing feedback and answering questions quickly.
• Criticism:
  This is the main thing all beta testers need to master. If you want to help the product to be successful, you need to be able to know what is good, what is acceptable, and what is outright garbage. I often see people, especially young individuals, lacking the ability to criticize something that is clearly subpar, and in the same breath they can shun something that is actually pretty good, just because they don’t want to mess with their previous opinion. This is completely unacceptable of any beta tester, and in fact it’s usually the best to strive for a mostly objective opinion about something. Now, it’s also good to insert your personal opinion to a degree, obviously. (Unless you genuinely think that Sonic I AM AN UNFUNNY SHITSTAIN's visuals are acceptable for anything else but a joke. Then you should give up trying.)
• Experience:
  Having knowledge of the genre and/or style a product is aiming for is important, so that you can give the developer(s) nudges to push them on the right track. This may be the result of anything from you having played few games alike or even developing your own before. Even if you do not feel you have good grasp on similar games, you can always just play a few to hopefully get an idea to what you’re jumping into. I will later discuss more Sonic hacking related experiences that should help with that.
• Passion for Gaming:
  Unless you absolutely love gaming, playing games, and criticising them, you will sooner or later get tired of beta testing and stop being helpful. It is impossible to universally get enjoyment or even a sense of satisfaction from beta testing, unless you’re the kind of person to explore every nook and cranny of a game until you found every last thing. If you already love hunting bugs for games like Sonic, or even games in general, then it will be more likely you keep an interest in beta testing for a long time, as it’s kind of an ever-changing puzzle of challenges of you finding every flaw possible.

If you have fulfilled most or even all of the above, you should be good to go! If not, maybe educate yourself a little; research, listen to critics, etc. These should help you to pick up on most necessary things.


Step 1: Tools

Now, I can’t really speak any further than our dear Sonic hacking community, because any different work environment or platform will require their specific sets of tools.

Emulators:
Depending on the task you are doing, or the nature of the testing, some emulators will be more useful than others, and while these aren’t rules to follow, they certainly are strong recommendations from my perspective, and are tools I use for said purposes. Note: It is suggested to have multiple emulators available to test on for different purposes.

• Kega Fusion:
  This is really good emulator for testing playability. Not so much to test strength of code, but how the game plays, as it is most common emulator among casual players and likely the main emulator the general public will use once your product is out. If you create something for the sole purpose of having many people play it, you are almost forced to make sure it runs at least on this one.
• Regen:
  Another common emulator for people who play casually; but more so for other hackers. Not only that, but it is vastly more accurate to real hardware in comparison to Fusion, which intentionally isn’t as accurate (to be lighter to run, especially on low end machines). Regen is good for testing most of the code you’ll come across. However, keep in mind there are always those few people who actually play on real hardware, and testing your work on Regen for that is not fool-proof. Also remember that sound emulation in Regen is not fully accurate.
• Exodus:
  By far the most accurate emulator to date. Although it also doesn’t fully replace hardware, and is very CPU-intensive. For anyone able to run it well, it’s definitely the option to go for when testing complex code.
• Flashcarts:
  While not emulators per se, these in combination with real Mega Drives, are by far the best platform to test hacks on. While it limits your options on screen capture, as well as the experience the common user would have (and hence is not suggested to use as the only testing environment), it will help any developer the most, especially keeping their software compatible. The most suggested flashcart is Mega Everdrive, even if it has proven to be not fully accurate, and is rather expensive, it’s the best means we have for hardware testing that I know of.

Screen Capture:

If you want to be a useful tester, you will also want to have a way to capture what you are doing on your end, videos or still images. These methods should allow high quality capture from your computer. Note: Never save the images as JPGs; it is a lossy compression and with small enough images it will make them not only larger than PNG or BMP, but also worsen the quality. PNG and BMP are suggested (though the latter only if you can’t be arsed).

• Print Screen:
  This is a key in your keyboard that allows you to capture image from your entire screen and put it in your clipboard, or while holding ALT and pressing Print Screen to only get an image from active window. This is a quick and effective way of taking an image. And as a bonus, imgur.com allows you to directly paste it into the site and upload directly. Some keyboards may also use the abbreviations, and MACs may not contain this key at all.
• Built-in screenshot features:
  These are emulator-specific features which allow to output correct aspect ratio and size images from your emulator window directly. While it’s a little slower to do, it will also give a better quality than the PrintScreen technique, and will make pixel-accurate outputs.
• Open Broadcast Software:
  OBS is a handy tool for recording any window or area, with any sound from your stereo mix or microphone. While this may not give optimal quality for your emulator, it will allow to quickly capture any emulator with fair ease. An alternative is gyazo.
• Gens Rerecording:
  This is easily the best way to record videos of you playing a hack or a game on the Mega Drive, Mega CD or 32X. It gives high quality AVI output, allows to record a playback movie for you to later decide to record or not record an AVI out of it, and is a good for Tool-Assisted Speedruns. This is not a good emulator for accurate testing however, so sometimes an additional tool may be needed to catch a bug that may not occur in Gens Rerecording.
• Text Editor:
  You will also need a text editor. Preferably one which produces clean ASCII text output, not any “advanced text formats” (such as RTF, DOC, etc.). This will help you when taking notes and sharing said notes with the developer.

Step 2: Beta Testing for the First Time

Eventually the time will come that you are asked to test something. Or the opposite happens, your request to receive help from someone was accepted. So... what should you do? Well, perhaps there isn’t a straightforward answer, but I can share my recommendations.

Try to fetch as much information about the project as you can. What are its goals? What it is supposed to feel like? What does the developer want to achieve with it? These are the type of questions you should be asking. But, try to avoid getting any spoilers. Why? Because the first thing you should do is having a blind playthrough. This is likely the most helpful information for any developer, as most people will be playing with not much knowledge of the product in question, so the first impression is important. It is also good idea to get to know a little bit about who you are working for. Not like personal details, but more along the lines of their personality and work motives. It should help you later in case of disagreement or conflict.


Step 3: Different Beta-Testing Methods

There is quite a bit of variation involved when it comes to ways of beta testing, and some are more helpful than others. I can’t give you an exact plan of what is right and what isn’t, but I can give you vague concepts to mix and match, and develop your own methods based on them.

• Take Notes:
  Taking notes in your text editor of choice, listing pros and cons, or maybe just your general thought, issues, etc.
• Screenshots and Explanation:
  Another method is showing a screenshot of something you see, and then explaining why you took it and what the developer should pay attention to. Be it a bug, something off with a thing, or even you noticing something positive and surprising! It’s also sometimes easier to just show a picture than explaining stuff with words.
• Video Play:
  Or, you can record yourself playing the hack, whether to find bugs, or just to do gameplay, and show this to the developer to analyze what a casual playthrough may look like.
• Review:
  This would you be you, doing a concise or not review about the hack, pointing out your thoughts and maybe listing a few things of what you would like to see in the future.
• Report:
  You doing a short and on-to-the-point observation about the hack, pointing out flaws you find along the way.

Speaking of flaws, don’t be afraid to point out everything, no matter how minor it may seem, or even if you have pointed it out before and it has been neglected. But it is also good to learn the difference between engine issue and user-made issues. Someone new, who doesn’t quite understand how the engine works yet, may not be able to fix those engine issues, but merely be able work around them. Usually good developers want to know about issues, even how minor or obscure they may be. But it’s good to prioritize issues that are larger and easier to trigger, and move to more and more difficult and obscure ones as more issues get fixed.

It’s also good to mention about how you represent your thoughts. It is important that you don’t sugarcoat things much, but that you also don’t be too harsh on the person either. It may be hard to find a good middle road between the two, and can affect how your suggestions are taken. Usually, it’s helpful to try to be neutral about it. It’s also good to note, that the developer may have bad response to even reasonable criticism. If they don’t think it’s important to fix something that clearly needs to be fixed, you should show them how much it matters, even if by annoying them about it a little. If they still refuse to listen to you, or even take offense, it’s good to reconsider if you should work for them anymore.


Step 4: My Last Stupid Thoughts

Here are a few bullet points about how I often work:

• Videos about new features:
  Whenever I’m told to check out a new feature, or major change, I strive to make a video about me testing it, to show how I would approach it as a player.
• Reporting Issues:
  I’m often just lazy, take a screenshot, and write short text about what I think is wrong in my picture. If I can’t get a snap, then I try to explain by text only, but I avoid doing that.
• Finding Bugs:
  I often intentionally try stupid, absurd and unexpected things, just to see if the game would handle it well, or not. Even trying known tactics to make the game freak out (Such as common glitches from the original games or previously found bugs). I do it over and over again to ensure nothing slips by. Of course, sometimes the more obscure things will.
• Dedication:
  When I’m beta testing for something, I easily spend hours, even for something that isn’t too mandatory. I try to do many different things to put the game to test. It takes a lot of effort and sometimes just may be boring, but getting complimented for being helpful is easily worth it. Plus, it’s fun if I can find something ridiculous.
• Interaction:
  I also talk to the developer frequently and suggest things to them what I think could enhance the product, as well as offer ideas as to how to solve issues. Probably not everyone appreciates this, but I like to do so anyway.

Step 5: Final Words

Overall, being a beta tester is definitely not an easy job to do, but it is really useful one. Don’t worry if you feel like you aren’t helping much, because the opposite is true. You will be irreplaceable for the development team at some point. But don’t let it get to you, because you aren’t the only one capable to do the same job.

Most importantly, don’t make beta testing a competition!

Oh, and thanks for KingOfHarts and especially Selbi for helping me make this.

 

This is excellent. I've already had 1 or 2 beta testers who failed to do anything useful for me. I'll ensure that if I ever ask for private beta testers, they read this first. Thanks for making such a comprehensive guide

 

This is a well written guide, and if and when I need beta testers for any of my own projects I will direct them to your guide before hand to make sure they are well qualified for the job.

 

Glad ya'll liked. And remember to thank Selbi as well, as he helped a lot. Also would like if any of you had ideas for improvements or more additions, as at places I feel this lacks good points. When ever I have enough motivation and time I will be (hopefully with help of Selbi) be adding more to this. Granted it wont be easy while keeping it clean but I believe we can do it.

 

This is a decent enough basis for a prospective tester to learn how to start off. That being said, there are some thoughts and nitpicks that I feel are necessary to mention.

• Macs for years have included a method of screen capture. Cmd+Shft+3 captures the entire screen, Cmd+Shft+4 allows selection of a specific area to capture, and pressing Space while in selection mode allows the user to just select a window to capture. It immediately spits out an image file for use.
• Criticism is an art, and one that shouldn't be overlooked. While you caught the basics, criticism is about not only being able to identify where something isn't right, but also being able to identify ways to alter what's wrong to make it right. In addition, criticism, by nature, is subjective. It is fundamentally an opinion, and it must be recognized as such. That doesn't mean that criticism can't carry legitimacy by any means, but it does mean that people do reserve the right to disagree with them because they subjectively see differently. Objectiveness is something to strive for, of course, but in the end, criticism is ultimately always subjective, no matter its legitimacy.
• Non-disclosure is often a sign of respect and a demonstration of integrity. NDAs are able to be legally drawn up and enforced (take it from someone who's taken peeks at some NDA paperwork...), but let's face it, this is a hobbyist community modifying ROMs of games that are often older than a sizable chunk of the community. Instead, non-disclosure is essentially seen as more of a gentlemen's agreement of sorts, and its upholding is seen as being respectful and demonstrating a respectable integrity and trust. When a developer asks you not to share what you've seen while testing their work, there's nothing stopping you from doing just that. However, the honorable and respectful thing is to abide by the developer's request.
• You failed to mention Gens in your list of emulators, yet bring it up for capturing gameplay.
• Your mention of first-time testing has some good points, but there is another side to the coin that wasn't quite so well-discussed. Due to the very nature of what Sonic games are, the gameplay itself is supposed to be intuitive. If there is a requirement for extensive explanation beforehand to make sense of how to play, then that is a major concern about design, albeit one that I know a number of people would contend. Part of the art of development and design is execution with simplicity. Rather than entangling with far too much, many games are strongly praised for being able to be just picked up and enjoyed without having to be instructed how to do everything. The ultimate thought that has to be on your mind during testing is that the game is supposed to be played by a normal person. Not a developer. A normal, off the street person. An utter casual. If the game fails to be something accessible to them, then the project is losing ground before it even gets a foothold.
• You may want to touch on capturing footage directly from hardware, as that is a vital source of information for those aspiring for proper compatibility.
• Experience and passion, in some ways, intermix. Those that understand the core of a game style best are those who are most passionate about the subject. Your most passionate can give you some of the most insightful and detailed analyses on what does and doesn't work within the confines of the style of game that the project is meant to fit.
• There are miscellaneous grammatical issues, but that's honestly being compulsively obsessive to the point of pettiness, which is why it's way down here and not further up.

That's just what I can riddle off at the moment. Do what you will with them.