Choosing the right automated testing framework

By  Walter Finkbeiner - Automation Expert @MQS

A common mistake at the time of designing an automated testing framework is that the Engineers in charge of designing it accept what the client wants to use instead of showing them a proposal that can be a better solution, or choose the tools in which they have more experience, without con-sidering many factors that will affect the Return of Investment in the future. It is not that easy to choose the set of tools that will become part of the framework. Many other items have to be taken into consideration:
• Budget: This item is very important, because the list of possible solutions can be shortened very quickly, and time can be saved. The decision must take into account that a license must be pur-chased, or plugins must be bought for some tools. Luckily, there are a lot of very good, free and open source solutions out there.
• Needs: Take into consideration if it supports all the necessary Operative Systems, browsers, if it provides with good reports, if it allows to run tests in parallel, integration with project management tools, etc.
• Team members: Another item to consider is the available team members. If they are experts on a specific language or tool, it will make sense to lean towards that, instead of choosing a tool that nobody has heard about before. But this should not influence a hundred percent to make the final decision. Also, it is important to know beforehand how easy it will be to get more people with expertise in those technologies if the project grows.
• Learning curve: Even if a tool or language is awesome, some team members might have never heard of it. So the learning curve must be considered. If the benefits of learning are greater than the time invested, and if the time allows it, maybe it is worth it to continue taking it into considera-tion.
• Stability: It is really important to know if the chosen tools are stable enough to avoid having problems in the middle of the project.
• Documentation: It is vital that there is enough good documentation to learn and overcome any obstacle, without having to try random possible solutions on StackOverflow
• Maintainability: The tools must encourage good practices and patterns so that the designed framework and tests are easy to maintain.
• Simplicity: All the complicated logic, if any, should live inside the framework, and testers should not need to make constant changes there. It should be easy for testers to write, read, and execute the tests.
• Reusability of code on tests: Code should be reusable. Don’t Repeat Yourself!
• Reusability in other projects: Take into account that in the future, another project might come up and have to be automated. The framework should be flexible enough to be used in other pro-jects.
• Reusability of existing code inside the framework: There might be already existing code in-side the System Under Test that can be reused in the framework. For example, an HTTP client might already exist inside the application’s project, and it might be a good tool for testing Web Services.
• Non-technical members involved: There are projects in which non-technical members like Product Owners, or manual testers without programming knowledge might be involved in writing and executing tests. There are some tools, for example, BDD frameworks, that allow non-technical members to write tests in English (Gherkin).
• Dev team involved: If the development team members have expertise in the chosen languages or tools, they will be able to help by doing code reviews, sharing the team good practices, and by giving tech support.
• Test reports: It is really important that the framework provides good, trustable and easy to read reports. That is what the management will read. As they might not have too much time, they should be concise, informative and quick and easy to read.
• Continuous Integration support: Continuous integration is very important nowadays to give rapid feedback to the team. A test automation framework must have Continuous Integration sup-port.
Designing a framework is not that easy after all. It requires a lot of planning, investigation, knowledge and communication. There are a lot of very good tools out there, but all projects are different. They have different budgets, deadlines, requirements, people, etc. And it is really im-portant to take all these variables into consideration before rushing to make a decision.