A few different responses here:

beaker-rspec should check which roles have been defined and then spin up an individual virtual machine for each of the roles. These should be done either serially or (better) in parallel.

Beaker-rspec doesnt have the ability to use a completely new SUT (system under test) per test suite. The only way to do this right now is create a new _spec.rb file per test, then run each test in a new process eg. bundle exec rspec spec/acceptance/profile_test_one.rb && bundle exec rspec spec/acceptance/profile_test_two.rb

For running in serial or parallel, it's probably best to do that within your CI platform. So for Travis (or Jenkins with the Matrix plugin) for example, you'd create a test matrix with different tests per profile, so they can be run in parallel. This requires significantly powerful testing machine, especially if you're using chunky VMs.

How can I reduce the time for each run, especially if having a rather large Puppet setup with multiple roles and different OS versions etc. Maybe select the type of tests (“only role tests” or “only profile tests”).

When you say reduce time, which part to you mean? How long the Puppet run takes, or how long to get the machine to a testable state?

Using Docker instances significantly speeds up the machine provisioning state.

To speed up the Puppet run, it depends on what's slow. If it's computation, make the SUT bigger, give it more cores or RAM. For network dependant issues, you can use caching. For example on Mac, you can use Squid to cache all the required RPM/APT packages

I usually don't do acceptance testing for roles. Roles are basically just meta-profiles: they should not have logic and are just a collection of profiles.

The only benefit of testing the role would be detection of conflicts when using multiple profiles, but conflicts and dependency checks can be better detected using rspec-puppet rather than an acceptance test.

So for example, a role like:

class role::foo_blog { include profile::base include profile::nginx include profile::php include profile::mysql } 

It would be easier to just test each of the four profiles with acceptance tests, and write an rspec-puppet test to make sure the different profiles don't have dependency issues or conflicts.

A few different responses here:

beaker-rspec should check which roles have been defined and then spin up an individual virtual machine for each of the roles. These should be done either serially or (better) in parallel.

Beaker-rspec doesnt have the ability to use a completely new SUT (system under test) per test suite. The only way to do this right now is create a new _spec.rb file per test, then run each test in a new process eg. bundle exec rspec spec/acceptance/profile_test_one.rb && bundle exec rspec spec/acceptance/profile_test_two.rb

For running in serial or parallel, it's probably best to do that within your CI platform. So for Travis (or Jenkins with the Matrix plugin) for example, you'd create a test matrix with different tests per profile, so they can be run in parallel. This requires significantly powerful testing machine, especially if you're using chunky VMs.

How can I reduce the time for each run, especially if having a rather large Puppet setup with multiple roles and different OS versions etc. Maybe select the type of tests (“only role tests” or “only profile tests”).

When you say reduce time, which part to you mean? How long the Puppet run takes, or how long to get the machine to a testable state?

Using Docker instances significantly speeds up the machine provisioning state.

To speed up the Puppet run, it depends on what's slow. If it's computation, make the SUT bigger, give it more cores or RAM. For network dependant issues, you can use caching. For example on Mac, you can use Squid to cache all the required RPM/APT packages

I usually don't do acceptance testing for roles. Roles are basically just meta-profiles: they should not have logic and are just a collection of profiles.

The only benefit of testing the role would be detection of conflicts when using multiple profiles, but conflicts and dependency checks can be better detected using rspec-puppet rather than an acceptance test.

So for example, a role like:

class role::foo_blog { include profile::base include profile::nginx include profile::php include profile::mysql } 

It would be easier to just test each of the four profiles with acceptance tests, and write an rspec-puppet test to make sure the different profiles don't have dependency issues or conflicts.