Testing¶

Test Selection¶

Additionally, on some platforms you may need to increase the maximum number of open file descriptors as some tests open many files to test the server.

ulimit -n 1024

To run all the basic python unit tests, run the following command at the top level of the repository:

py.test -m unit

Note that this includes unit tests that require Selenium to be installed. To exclude those unit tests, you can run the command:

py.test -m "unit and not selenium"

To run just the BokehJS unit tests, execute:

py.test -m js

Alternatively, you can also navigate to the bokehjs subdirectory of the source checkout and execute:

gulp test

You can run all available tests (python and JS unit tests, as well as example and integration tests) from the top level directory by executing:

py.test

To learn more about marking test functions and selecting/deselecting them for a run, please consult the pytest documentation for custom markers. The list of currently defined test markers is below:

• examples: an examples image-diff test
• integration: an integration test that runs on SauceLabs
• js: a javascript test
• quality: a code quality test
• selenium: a test requiring selenium
• unit: a python unit test (implicitly assigned for tests otherwise unmarked)

Code Coverage¶

To run any of the tests with coverage use the following:

py.test --cov=bokeh

To report on a subset of the Bokeh package, pass e.g. -cov=bokeh/models.

Other Options¶

To run any of the tests without standard output captured use:

py.test -s

See the pytest documentation for further information on py.test and its options.

Examples tests¶

The examples tests run a selection of the Bokeh examples and generate images to compare against previous releases. A report is generated that displays the current and previous images, as well as any image difference.

To run just the examples tests, run the command:

py.test -m examples --report-path=examples.html

After the tests have run, you will be able to see the test report at examples.html. Running locally, you can name the test report whatever you want. On TravisCI, the examples report is always examples.html.

The examples tests can run slowly, to speed them up, you can parallelize them:

py.test -m examples --report-path=examples.html -n 5

Where n is the number is the number of cores you want to use.

In addition, the examples tests generate a log file, examples.log which you can view at examples.log in the same directory that you the tests were run from.

Integration tests¶

Integration tests use the selenium webdriver to test bokeh in the browser. Some of the selenium tests run on Firefox and can be run locally.

To download a specific version of Firefox, go to https://ftp.mozilla.org/pub/firefox/releases/ Unzip the release and note the location of the application under bin directory.

To run just the integration tests, run the command:

py.test -m integration                  \
    --driver Firefox                    \
    --firefox-path /path/to/firefox/app \
    --html=tests/pytest-report.html

The --html is optional, but it will allow you to see the same report that is generated on TravisCI.

Many of these tests can be run locally, and you will see browser windows open and close on your machine as you run them. When we run the tests on TravisCI we use the selenium service SauceLabs which provides free testing for open source projects.

It is strongly recommended to run python setup.py develop before running the integration tests to ensure that the latest version of BokehJS (with any changes you may have made), is available for the integration tests.

Some of the integration tests are screenshot tests that take a screenshot of the bokehplot and compare it against a reference image that is stored in the repository. These tests must be run on SauceLabs so that comparisons can be made consistently.

To run the integration tests on SauceLabs, run the command:

py.test -m integration --driver=SauceLabs --html=tests/pytest-report.html

For this command to be successful you must have the following:

• SAUCELABS_USERNAME environment variable
• SAUCELABS_API_KEY environment variable
• Sauce Connect tunnel running

To start up the tunnel, first download Sauce Connect. Next, extract the files and navigate to the install directory. Then you can establish the tunnel by running:

bin/sc -u SAUCELABS_USERNAME -k SAUCELABS_API_KEY

To obtain the SAUCELABS_USERNAME and SAUCELABS_API_KEY please contact the developers.

Python Unit Tests¶

Python unit tests maintain the basic functionality of the Python portion of the Bokeh library. A few general guidelines will help you write Python unit tests:

absolute imports

In order to ensure that Bokeh’s unit tests as relocatable and unambiguous as possible, always prefer absolute imports in test files. When convenient, import and use the entire module under test:

• GOOD: import bokeh.models.transforms as bmt
• GOOD: from bokeh.embed import components
• BAD: from ..document import Document

markers

By default any unmarked test is considered part of the unit group. If a unit test needs an additional mark (e.g. selenium) then the unit marker must be supplied explicitly:

@pytest.mark.unit
@pytest.mark.selenium
def test_basic_script(capsys):
    # test code here

pytest

All new tests should use and assume pytest for test running, fixtures, parameterized testing, etc. New tests should not use the unittest module of the Python standard library.

JavaScript Unit Tests¶

These tests maintain the functionality of the BokehJS portion of the Bokeh project. The BokehJS tests are located in bokehjs/test. They are written using Chai “expect” style. If new test files are added, an appropriate entry in the directory index file should be added.

Integration Tests¶

To add a new screen shot integration test, first make sure you can run existing screen shot tests, for example tests/integration/annotations/test_whisker.py. New screen shot tests should follow the general guidelines:

• Be as simple as possible (only include things under test and nothing extra)
• Prefer the bokeh.models API

Once a new test is written, a base image for comparison is needed. To create a new base image, add --set-new-base-screenshot to your the standard py.test command to run the test. This will generate an image with the name base__<name_of_your_test>.png in the appropriate directory. Use git to check this image into the repository, and then all future screen shot tests will be compared against this base image.

Configuration¶

There are a number of files that affect the build configuration:

• .travis.yml

  Defines the build matrix and global configurations for the stages described below.

• conda.recipe/meta.yaml

  Instructions for building a conda noarch package for Bokeh. This file is the single source of truth for build and test (but not runtime) dependencies.

• setup.py

  Used to build sdist packages and “dev” installs. This file is also the single source of truth for runtime dependencies.

• setup.cfg

  Contains some global configuration for build and test tools such as versioneer and pytest.

Build Stages¶

Etiquette¶

TravisCI provides five free build workers to Open Source projects. A few considerations will help you be considerate of others needing these limited resources:

• Group commits into meaningful chunks of work before pushing to GitHub (i.e. don’t push on every commit).

• If you must make multiple commits in succession, navigate to TravisCI and cancel all but the last build, in order to free up build workers.

• If expensive examples tests are not needed (e.g. for a docs-only Pull Request), they may be disabled by adding the text

  [ci disable examples]
  

  to your commit message.