OS specific resemblejs tolerance issues #60
Description
What are you trying to achieve?
Hello, first thanks a million for this great helper! It makes my life easier :)
I am experimenting Visual Testing based on resemblejs, and I face a strange issue:
The tests pass when I run them locally on the computer that originally created the screenshot (Mac OS), weather HEADLESS is on or off.
However, some tests fail from a CI pipeline (bitbucket pipelines) based on Linux (HEADLESS).
I have 2 main questions:
- is this expected? Where can I find information about the visual comparison "tolerance" and overall options? I do not find a reference to it in their documentation: https://github.com/rsmbl/Resemble.js
- what is the intended approach when it comes to update screenshots? Is there an "update" option somehow?
What do you get instead?
Provide console output if related. Use
--verbosemode for more details.
MissMatch Percentage 3.82 Scenario Steps: - I.seeVisualDiff("homepage-mobile.png", {"tolerance":2,"prepareBaseImage":false}) at Test.<anonymous> (tests/visual-regression/homepage.test.js:15:5) - I.saveScreenshot("homepage-mobile.png") at Test.<anonymous> (tests/visual-regression/homepage.test.js:14:5) - I.amOnPage("/") at Test.<anonymous> (tests/visual-regression/homepage.test.js:12:5) - I.resizeWindow(320, 480) at Test.<anonymous> (tests/visual-regression/homepage.test.js:11:5)Provide test source code if related
Feature('Screen comparison of the Home page'); Scenario('Compare desktop Home Page @visual-test', I => { I.amOnPage('/'); I.saveScreenshot('homepage.png'); I.seeVisualDiff('homepage.png', { tolerance: 2, prepareBaseImage: false }); }); Scenario('Compare mobile Home Page @visual-test', I => { I.resizeWindow(320, 480); I.amOnPage('/'); I.saveScreenshot('homepage-mobile.png'); I.seeVisualDiff('homepage-mobile.png', { tolerance: 2, prepareBaseImage: false }); });Details
- CodeceptJS version:
codeceptjs@2.5.0&codeceptjs-resemblehelper@1.9.0 - NodeJS Version:
12.14.0 - Operating System:
OS X 10.15.3vshttps://hub.docker.com/_/amazonlinux/ - puppeteer:
puppeteer@2.1.1 - Configuration file:
require('dotenv').config(); import { setHeadlessWhen, setWindowSize } from '@codeceptjs/configure'; const headless = (process.env.HEADLESS || process.env.CI) && process.env.BROWSER !== 'firefox'; const profile = ['e2e', 'vr'].includes(process.profile) ? process.profile : 'e2e'; // turn on headless mode (not opening browser) setHeadlessWhen(headless); setWindowSize(1600, 1200); export const config = { name: 'my-project', tests: { e2e: 'tests/end-to-end/scenarios/**/*.test.js', vr: 'tests/visual-regression/**/*.test.js' }[profile], include: { I: './tests/end-to-end/steps_file.js' }, output: 'test-reports/end-to-end', bootstrap: null, mocha: { reporterOptions: { mochaFile: { e2e: 'test-reports/end-to-end.xml', vr: 'test-reports/visual-regression.xml' }[profile] } }, helpers: { Puppeteer: { browser: process.env.BROWSER || 'chrome', url: process.env.TEST_BASE_URL, show: true, waitForNavigation: 'networkidle0', waitForAction: 200, pressKeyDelay: 50, chrome: { args: [ '--no-sandbox', '--ignore-certificate-errors', '--disable-dev-shm-usage', '--disable-gpu', '--disable-setuid-sandbox' ] } }, ResembleHelper: { require: 'codeceptjs-resemblehelper', screenshotFolder: './test-reports/screenshots/', baseFolder: './test-reports/screenshots/base/', diffFolder: './test-reports/screenshots/diff/' } }, plugins: { allure: {}, autoDelay: { enabled: true }, pauseOnFailure: {}, retryFailedStep: { enabled: true }, screenshotOnFail: { enabled: !process.env.CI } } };