Are Simulators & Emulators the VHS Camcorders of Mobile App Testing?
I’ve recently heard some argue that virtual devices, such as emulators (Android) and simulators (iOS), are no longer relevant. They say that only real mobile devices should be used for dev and testing.
I disagree. Strongly.
They are not antiquated, irrelevant, or obsolete, like our once-great VHS camcorder (RIP).
Virtual devices bring a ton of value to both developers and test automation engineers, but the main point here is to have a good BALANCE!
In this post, I will give a quick 101 on simulators and emulators show why they are EXTREMELY VALUABLE when used properly. Lastly, I share some tips for getting a golden mix of real & virtual for better dev and testing.
What Virtual Devices Are Great For?
Using a virtual device can provide a lot of value to mobile app developers.
Velocity
Shift left testing and getting fast feedback fall under this category. Simulators and emulators are fast, always updated with the latest OS version from the platform vendors (Google, Apple). They provide a great platform for early-stage development and testing.
Scale
Using virtual devices (through parallel testing and comparison across platforms) adds tons of value, productivity, and quality assessment earlier in the development stages. It serves as a preliminary phase for real device regression testing.
COO (Cost of Ownership)
It is a fact that testing and developing against a virtual platform is cheaper than a real device when looking at cost of ownership and ongoing maintenance. With that in mind, teams can use these platforms more frequently and at a lower cost.
There is a great value in using such platforms — for both developers and test engineers.
What Virtual Devices Are … Less Great For?
While we established the benefits of using virtual devices, it is important to know and understand their limitations.
End Users Don’t Carry a Virtual Device Around 😊
The end users of any given application or website are running on real devices with real network carrier connectivity, real mobile device hardware, sensors engagement (audio, camera, Bluetooth, etc.), and a permutations of OS versions (latest, legacy, beta).
Automation Coverage
Virtual devices cannot mimic and automate each scenario that a real device can, that means, there is always going to be a subset (IMHO — 30–40%) of scenarios that can only be complemented and developed on a real smartphone or tablet.
Platform Coverage
Virtual devices cannot cover all the real devices that are out there (Huawei’s, LG’s, Motorola, OnePlus, Oppo, Vivo, etc.). The end users of any application are running on more than the usual Samsung, Google, and Apple devices. Therefore, it’s important to cover the other devices with the real capabilities against a wider plethora of devices.
Mobile App Dependencies
Mobile apps by nature rely on users’ contact lists, carrier service for 2FA (two-factor authentications), push notifications, security privileges and much more. Getting the “real” experience and capabilities are critical to assess the app functionality, security, and quality.
There are important considerations and limits to virtual devices that only real devices can cover. Not to mention the importance of a go-no-go decision that is a MUST to perform on real devices for customer experience confidence. We have seen a lot of app-store ratings coming from lack of coverage and testing on real devices.
Getting the Golden Mix of Devices
Want a solid pipeline that is scalable, high quality, and considerate of budget? Use a strategic mix of real and virtual devices. You can grab the benefits from each of these and have a sustainable plan as the mobile market continuously changes.
Consider the above visual, which takes into account the cost of an escaped defect, for a recommended practice of mixing and matching real and virtual devices.
The beauty of a strategy like this is that practitioners can easily use test automation frameworks like Appium, Espresso, and XCUITest across both types. Then they can drive the same tests across the mix of platforms to gather the right feedback based on the right trigger (bug fix validation, smoke testing, UI testing, etc.).
The Bottom Line
The mobile landscape constantly changing. We have:
- Growth around progressive web and Flutter apps.
- Constant releases of new mobile devices and OS versions.
- New capabilities being added to the mobile apps to drive more user engagement.
By combining both real and virtual devices in a strategic, well-managed, and balanced approach, teams can deliver their mobile apps with confidence.
