Snapshot tests fail when running on different simulators

[KaneCheshire]

One of the main goals of this tool was to enable running snapshots on any simulator without having to remember which resolution the snapshots were recorded at.

Unfortunately this doesn't work properly due to the fact that although the images are the same size, iOS renders stuff slightly differently on different scale devices.

I've tried manually looping through all subviews of the view to test, and setting the contentScaleFactor manually. This actually improved the diff a lot but it didn't completely fix it, so the tests still failed.

[ghost]

This is definitely a challenge. Seems overwhelming.

[KaneCheshire]

It’s not too bad when you think about the fact we only really care about @1x, @2x and @3x.

Width and height of snapshots are defined by the test rather than the device, but the scale is what causes a problem when running a test recorded on a @2x device on a @3x device.

My first naive thought was to force the recorded snapshot image to be recorded at a fixed scale regardless of simulator but sadly that didn’t work since the frames of the view were laying out a fraction of a point differently.

Very frustrating.

[ghost]

Indeed a pickle ! 🤯

[ghost]

These guys seem to have a stable snapshot testing suite. The source may have the answer you're looking for.

https://github.com/uber/ios-snapshot-test-case

Do you use renderInContext: ?

[KaneCheshire]

Haha no the Uber/Facebook solution has the same problem and was the main driving factor behind writing my own in the first place. Theirs is very limited.

This is the code to render in context: https://github.com/KaneCheshire/PixelTest/blob/master/PixelTest/Extensions/UIView%2BPixelTest.swift

[ghost]

Apologies if my phrasing was a little insensitive. I didn't mean to suggest that another team had done a better job. It certainly reads that way. I sometimes send messages to humans after several hours of talking to machines and I forget myself.

I was pretty confident that you would know straight away if that link was a good resource or not. I didn't look into the subject deeply, but to be honest I didn't feel like it was necessary. Given that such a serious flaw isn't listed front and centre in the README, or mentioned in issues (open or closed), I presumed it was rock solid.

This team use it which also suggested to me that it was pretty rock solid.

To be clear, when this project or Uber's project is used in a typical continuous integration configurations, how much of an impact does this bug have on day to day?

[Jon889]

In the project I'm working on we have wrapped the verify method (from the Uber framework) to check if the scale is @2x and if not fail the test. It's not ideal but at least the error is descriptive and the user knows to switch to a 2x simulator.

[KaneCheshire]

That's basically what we're doing as well. Really annoying!