Skip to main content

Debugging automated tests

Sometimes we need to understand why a test is failing during the build. We need to do some steps to figure out the reason, it might be due to the test being flaky, or it can be a bug that was found by the automated tests, or it may be a change that is expected to break the test, and we need to refactor it. To debug them, we can follow some steps:

  • Run the test in your local machine
  • Execute the test scenario manually
  • Reproduce the error on a clean install

Run the test manually in your local machine#

For this step, you'll need to have Docker installed on your machine. Find the environment used by the test you want to run (you can check that on the .github/workflows/build-dev-artifacts.yml on the e2e section). After that, you can run the bash bin/envs/init.sh command with the environment you want to run, so if you're going to run the default environment, you should run the bash bin/envs/init.sh default command. After the setup, you can navigate to the e2e-tests folder and run yarn cypress open to open the Cypress Test Runner and execute the tests you want to run. If it fails on CI but fails locally, it might be something with the setup being affected by other tests, try to make it more independent; you can always ask Augusto for help.

How to run cypress tests locally without Docker#

This alternative is useful for when you don't want to set up Docker, or you need to change the theme while testing:

  • First, you will need to set up the environment on your local machine and have wp-cli installed. Assuming that your local URL is http://neve.test you can CYPRESS_BASE_URL=http://neve.test bash bin/envs/local.sh or if you want to install a specific environment, you can use CYPRESS_BASE_URL=http://neve.test bash bin/envs/local.sh amp where the amp is the environment inside bin/envs/amp
  • Then, you can move to the e2e-tests folder and run yarn install --frozen-lockfile to install the dependencies.
  • After this, you will have to open cypress with the proper base URL, as CYPRESS_BASE_URL=http://neve.test npm run cypress:open and run the spec that you want.

Execute the test scenario manually#

Check the test scenario that runs on Cypress, and try to execute it manually on a local instance, using the same Docker environment used on the CI. Try to reproduce the error in the environment to check the possible causes. It might be failing due to missing selectors or a selector that changed. If you can reproduce the error on your local machine, you can fix it locally and submit a commit to the PR fixing it.

Reproduce the error on a clean install#

Set up a new instance to test. You can see how to do it on the Setting The Environment on Manual Tests section. Then try to reproduce the error. You can even try to achieve the same error with different setups and different plugins if it is an error related to compatibility. After that, if you can reproduce the error within this environment, then it's a problem with the plugin/theme under test; if not, it might be something with the test code or environment setup.