Mobile app testing device matrix: what to ask a vendor
Updated 2026-09-24
Build the device matrix from your own analytics, not from the vendor's lab list. Group devices into tiers by how much of your traffic they carry and how risky they are, decide per tier what runs on real devices and what on emulators or a cloud device service, and write the matrix into the statement of work so coverage is a commitment you can check.
What to check
- Export operating system versions, device models, screen sizes and app versions from your own analytics for the last few months.
- Group devices into tiers and write down which tests run on each tier: full regression, smoke tests or exploratory checks only.
- Decide which checks must use real devices, for example camera, biometrics, push notifications, payments and battery or network behaviour.
- Ask who pays for any cloud device service, whose account it runs under, and whether your team can use it after the contract ends.
- Put the matrix, and the rule for updating it, into the statement of work.
- Check a sample defect report for device, OS version and build details.
Ask the vendor
- Which devices would you test our app on, and are they physical devices you own, emulators or simulators, or a cloud device service?
- How would you build and update the device matrix from our analytics, and how often would you review it?
- Which of our features need real devices in your view, and why?
- If you use a cloud device service, is the account ours or yours, and is the cost inside your rate or billed to us?
- How do you record the device, OS version, app build and network conditions on each defect and test result?
Start from your analytics, not the vendor's lab
Many proposals include a device list. It usually reflects the devices the vendor owns or the ones its cloud service offers, not the ones your users hold. The UK Government Digital Service manual gives the same advice for browsers: analyse your own users' operating systems, browsers, versions and screen sizes, and revisit the choice as behaviour changes.
Export from your product analytics or app store console:
- Device models and manufacturers
- OS versions
- Screen sizes and densities
- App versions still in use
- Regions, languages and carriers, if relevant
Use a recent period that reflects normal use, and check it against the minimum OS versions you officially support.
Build tiers
A flat list of devices treats every device the same. Tiers let you spend test time where your users are.
| Tier | Which devices | What runs |
|---|---|---|
| 1 | Devices and OS versions carrying most of your users | Full regression, new features, exploratory testing |
| 2 | Next group by usage, plus the oldest supported OS versions | Smoke tests and key journeys each release |
| 3 | Long tail, tablets, unusual screen sizes | Periodic checks, crowd or ad hoc cycles |
| Risk | Any device family with known crashes or past incidents | Targeted tests on the affected areas |
Name devices by model and OS version, not by labels such as "latest iPhone" that change meaning over time.
Real devices, emulators and cloud services
Each option fits different checks.
Emulators and simulators run quickly and cheaply in CI and cover many OS versions. They are a good fit for layout, navigation and business logic checks.
Physical devices are needed where hardware or the real platform matters: camera, biometrics, GPS, Bluetooth, push notifications, in-app payments, background behaviour, battery and memory limits, and real network conditions.
Cloud device services give remote access to many physical devices. They add breadth, but they cost money, they may be under the vendor's account, and your data passes through a third party. Treat the service as a subprocessor if test data includes personal data.
Ask each bidder to map its approach to your tiers, and to say which checks run on which option.
Automation on mobile
If the vendor will automate mobile tests, ask:
- Which framework it proposes. Open-source options such as Appium, Espresso and XCUITest are common; a proprietary framework can create lock-in, see test automation vendor lock-in.
- Where the tests run: your CI with emulators, a device service under your account, or the vendor's own lab.
- How flaky device tests are tracked and fixed rather than rerun until they pass.
State coverage in the SOW
A device matrix in a proposal is a description. A device matrix in the statement of work is a commitment. Include:
- The tiered matrix, with model and OS version per row.
- What runs on each tier each release.
- Which checks require physical devices.
- Who provides devices or the device service, and under whose account.
- How often the matrix is reviewed against your analytics, and who approves changes.
- What the test report must show per device: pass, fail, not run, and why.
ISO/IEC/IEEE 29119-3 expects a test plan to state the test environment. Treat the matrix as that part of the plan. The statement of work guide shows where it fits.
Defect reports need device detail
Ask for a redacted defect report during selection. Each mobile defect should state device model, OS version, app build, network conditions, steps, expected and actual result, and evidence such as a screen recording or logs. Without that, developers cannot reproduce issues and the report adds work instead of saving it.
Keeping the matrix current
New OS versions and devices arrive every year, and your users move with them. Agree a review rhythm, such as each quarter and ahead of each major OS release, where the vendor compares the matrix against your latest analytics and proposes changes. You approve the changes, and the SOW says whether a larger matrix changes the price.
Questions for the RFP
Use the RFP generator to list mobile platforms and attach your analytics summary, and ask bidders to respond with a tiered matrix. The questionnaire includes a question on devices, real versus emulated, and who pays for any device service. If crowd cycles are on your list for long-tail coverage, read crowdtesting vs managed QA first.
The selection path shows when to settle the matrix: before the RFP, so every bidder prices the same coverage.
Tools for this
Questions buyers ask
- How many devices do I need?
- There is no fixed number. Cover the devices and OS versions that carry most of your users, the oldest versions you still support, and any device family with known problems in your app. Your analytics sets the list.
- Are emulators good enough?
- For functional checks of layout and logic on many OS versions, often yes. Hardware features, real network conditions, performance and some platform behaviours need physical devices. Split the matrix accordingly.
- Can crowd testers replace a device lab?
- They can add breadth on real devices and networks for a cycle. They do not replace a repeatable regression run on a stable set. See the crowdtesting guide for the trade-offs.
Sources
- ISO/IEC/IEEE 29119-3:2021 Software testing, Part 3: Test documentation, ISO/IEC/IEEE
- Quality assurance: testing your service regularly, UK Government Digital Service
- Designing for different browsers and devices, UK Government Digital Service, Service Manual