Play list

Each playbook contains multiple playlists. These are used to execute certain test cases in a specific order. Within a single playlist, the test case or scenario execution is undetermined, which mean that you cannot trust one test case to be executed before the next. It is important to understand that test cases should not be dependent to each other. If its important to run a specific test case before another, you can split them up in different play lists. The order of the play list execution is determined and adjustable.

Within a playlist, you can add test cases and/or scenarios by their tags. This means that you can setup a playlist with a certain tag, e.g. “calculation”, and during the start of the play list, it collects the test cases and/or scenarios that fall in that criteria.

This means that you can add new test cases and/or scenarios to the play list by adding the tags to the specific test case/scenario. This way, you can setup your playbook one time, and the test cases will be added automatically, based on the given tag-filters.

Parallelization within playlists

Within a single playlist, the execution order of test cases and scenarios is not guaranteed. The system executes them as soon as capacity becomes available. This means:

You cannot rely on a fixed order within the same playlist.
Test cases must not depend on the outcome of other test cases within that playlist.
If order matters, move the relevant test cases into separate playlists. The order between playlists is deterministic and configurable.

Each playlist also has its own parallelization settings (e.g. maximum parallel tests, ramp-up). A more detailed explanation can be found in our dedicated blog on parallel execution: https://www.josf.nl/introducing-parallel-test-execution-in-josf/

Tag queries with AND/OR logic

A playlist can be populated manually, but most of the time you will use tag filters. These filters support logical operators to build flexible selection rules:

AND – all specified tags must be present
OR – at least one of the tags must be present
( ) – to group logical conditions

Examples:

regression AND web-shop
→ Includes only the test cases that contain both tags.
(smoke OR critical) AND payment
→ Includes test cases that have smoke or critical, but always the payment tag.

This allows you to create dynamic playlists: whenever a new test case is tagged correctly, it will automatically be included—no need to update the playbook configuration.