Flutter: Automating shaders warm-up by an E2E UI performance test
There is a well-known animations jank problem in Flutter apps that is recommended to be fixed by the Skia shaders warm-up technique.
It goes without saying that manually going through the steps described in the documentation could be a tedious process, so it’s advised we automate it by a UI integration test(s).
With such integration tests, you can easily and reliably get the new SkSLs when the app code changes, or when Flutter upgrades. Such tests can also be used to verify the performance change before and after the SkSL (Skia Shader Language) warm-up. Even better, you can put those tests into a CI/CD system so the SkSLs are generated and tested automatically.
Please note that theintergration_test package is now the recommended way to write integration tests instead of the outdated flutter_driver package. See updates on Flutter testing article and this integration testing documentation page for details.
So, let’s see how an E2E UI integration test may look like on the example of my pet app.
A problem
There is a dialog transition animation in my app that appears to be janky only on the first run, so it looks like the shader warm-up should be of help there.
Goal
I’d like to have a UI integration test that covers a simple user flow triggering the problematic animation along the way. Also, this test should produce performance measurements and a JSON file containing the captured shaders. The same test is to be used to measure the performance difference after the app is built with the shaders file produced in the first test run.
Implementation
To make the test more maintainable and easier to read I used given_when_then and page_object packages from eBay. For more info and examples you can look at flutter_glove_box and flutter_global_summit_21 repositories. I also used the HTTP recorded session technique described in this article to be independent of my back-end.
Having some page objects and harness extensions written I got the following test that I placed into integration_test folder. The file has open_training_details_test.dart name. I decided not to create variables to reuse the page objects to make the test shorter and simpler for reading. Please note the _binding.watchPerformance wrapper, by default it writes gained results in a file named performance.json.
Even though the UI test itself uses the new integration_test package and can be run by flutter testcommand, we’ll still need the flutter drive command to be able to record performance metrics and produce the shaders file out of the test run. So in order to do that, I added integration_test.dart file under test_driver folder as the mentioned documentation recommends.
After it was done I needed to run the test 2 times in profile mode, so I connected my devices and did a few runs.
This is the command I used to run the test producing a shaders file. Give a read to this thread to learn why --no-dds argument is used.
And this is the command to run the same test but with the shaders file bundled in the app.
The following video shows how the execution of this test looks like (on iPhone SE).
Results
I ran the test on 3 different devices (iPhone SE, iPhone 12 Mini, Sony Xperia M) with 2 Flutter versions and got some interesting results. The performance measurements are in JSON format and look like this.
I summarized them in the following table to ease the comparison.
The moment of truth was in the animation I aimed to improve. Here are the 2 gifs in slow-motion (0.25x of the original speed) from iPhone 12 Mini (the app was built with Flutter 2.8).
I hope you can notice the difference.
Conclusions
Even though I gained significant frame rasterizer time improvements for the older devices (iPhone SE, Xperia M), I was surprised by the results on iPhone 12 Mini with Flutter 2.5.3: the average time was improved but the worst time got worse. Interestingly, the picture changed a lot when I upgraded to the just-released Flutter 2.8.0.
This is an interesting topic and I’ll continue diving into it. Having this automation in place makes the overall performance measurement and improvement process more maintainable.
