I used Maestro for one year after we switched from Detox. It's awesome to start end-to-end tests with and definitely the most accessible. However, in the end, we had to switch to Appium. While it's great to get started quickly, I definitely wouldn't recommend it for a serious production system pipeline. We encountered several issues:
- When attempting to write logic using JavaScript, there were issues such as error stacks providing no useful information and a complete lack of console logging. The injection of variables and the custom fetch also made linting ineffective. At least Maestro now supports ES6 via GraalJS.
- Coordination of test flows is lacking. I wish I could retry each flow individually. Ultimately, I had to create a wrapper (director) around Maestro to provide things like recordings, retries on failing tests, and (relevant) JSON output for our CI. I also needed to write custom reporters for Slack and other integrations. While these are not a core need for a testing tool, when I switched to Appium + Webdriver, most of these tools were available out-of-the-box (though they come with their own issues as well).
- When we updated XCode or iOS to newer versions, things tended to break. We often had to freeze the pipeline versions and wait until fixes were released, regularly checking GitHub issues to see if updates became available.
- We also experienced strange, random test timeouts which started to be frequently enought to break even with retries. These happened even though we had frozen versions of Maestro, XCode, and iOS, so it wasn't Maestro's fault, but it was problematic enough that we decided to move away from it, because we couldn't isolate the root cause.
I definitely miss the simplicity of Maestro. Appium takes more time to set up and comes with its own set of issues.
I still follow the project (and also the Flashlight.dev project, which uses Maestro for performance measurement), and looking forward to updates, which the team does constantly.
If you are a startup with no QAs I definetly would go with the Maestro route, but avoid it for a complex app pipeline use case, at least for now.
> "Appium takes more time to set up and comes with its own set of issues.
Hi, Appium project creator here. I'm working on something new (complementary to Appium) to address Appium set-up and other issues. If you ever wanted to chat, would love to hear more.
- When attempting to write logic using JavaScript, there were issues such as error stacks providing no useful information and a complete lack of console logging. The injection of variables and the custom fetch also made linting ineffective. At least Maestro now supports ES6 via GraalJS.
- Coordination of test flows is lacking. I wish I could retry each flow individually. Ultimately, I had to create a wrapper (director) around Maestro to provide things like recordings, retries on failing tests, and (relevant) JSON output for our CI. I also needed to write custom reporters for Slack and other integrations. While these are not a core need for a testing tool, when I switched to Appium + Webdriver, most of these tools were available out-of-the-box (though they come with their own issues as well).
- When we updated XCode or iOS to newer versions, things tended to break. We often had to freeze the pipeline versions and wait until fixes were released, regularly checking GitHub issues to see if updates became available.
- We also experienced strange, random test timeouts which started to be frequently enought to break even with retries. These happened even though we had frozen versions of Maestro, XCode, and iOS, so it wasn't Maestro's fault, but it was problematic enough that we decided to move away from it, because we couldn't isolate the root cause.
I definitely miss the simplicity of Maestro. Appium takes more time to set up and comes with its own set of issues.
I still follow the project (and also the Flashlight.dev project, which uses Maestro for performance measurement), and looking forward to updates, which the team does constantly.
If you are a startup with no QAs I definetly would go with the Maestro route, but avoid it for a complex app pipeline use case, at least for now.