Frequently Asked Questions

Is SBOX a SaaS / Cloud service?

No, SBOX is a solution running inside your corporate network on your own infrastructure making it extremely secure and scalable. No external access to the application under test is required.

What are the main advantages of SBOX over building and maintaining your own Selenium Grid? 

Building and maintaining your own Selenium Grid is complex and a high level of resources. The open-source Selenium Grid provides basic functionality for running cross-browser tests. However, it is missing the scalability and enterprise-level functionality that many organizations require today. With every new browser, driver, and Selenium release, there is significant effort involved to ensure that the new browser versions are running properly in the Selenium Grid. SBOX is a maintenance-free solution, as the browser, driver, and Selenium updates are managed by us. As a client, you can fully focus on building tests rather than test infrastructure.  SBOX also provides enterprise-level features like live view, video recording, access control via LDAP, and more.

Do I need to maintain new browser, driver and Selenium versions myself?

No. SBOX automatically downloads and installs new browsers, drivers and Selenium versions after they are tested and released by Element34. This makes the SBOX completely maintenance free.

Am I locked in with SBOX or can I also run tests on another Selenium Grid Service? 

No. SBOX implements the standard Selenium/web driver protocol. You can switch to another solution at any time.

Do I need to check the compatibility of new browser releases with Selenium and the browser drivers? 

No, we do this for you. When we release new images to the SBOX, our customers can be sure that the versions are compatible with each other.

Do tests share browser resources?

No. Each test is run in an isolated environment and does not share any resources with other tests. A new browser is started for each test. After the test finishes, the browser is closed and the environment is destroyed. Each test is completely isolated and there is no chance of side effects between browsers/environments.

Is there a logging/monitoring capability built into SBOX?

Yes. SBOX has an advanced logging and monitoring mechanism, which can easily be customized and integrated into the customer environment.

How long does it take before the latest browser versions are available on SBOX?

After we conduct our tests, new images are usually released to the SBOX within 24-48 hours of the initial release.

Is one SBOX enough, or do I need multiple installations (i.e. per team)?

Most of our customers have a central SBOX instance running where the various teams connect. SBOX is designed to handle a high load from various sources (i.e. CI system, developers).

Can I restrict the visibility of tests in case I don't want everyone to see all tests? 

Yes. Access control and restriction are a built-in feature of SBOX. If you are using an identity provider i.e. LDAP you can easily set up access groups and restrict the visibility of tests to a certain group of people.

Is there a limit on how many tests I can run? 

No, there is no limit on the number of tests that you can run. Based on your license tier, there may be a limit on the concurrency (the number of tests you can run at the same time).

Can I run tests with production data on SBOX?  

While this is entirely up to you (and may depend on your policies), you can run tests with any kind of data on SBOX. The test and test data never leave your network with SBOX. SBOX runs completely inside your network and with that, it is safe to run tests with production data.

Do tests run faster on SBOX than on SaaS solutions?  

Typically we see an increase of factor 3-5 in execution times when the same test is run on SBOX vs. a SaaS solution. This is because SBOX runs inside your network and is close to the rest of the CI/test infrastructure. Latencies are minimized hence the significant increase in speed.

Background: Selenium / WebDriver is an HTTP-based protocol that requires a roundtrip from the client -> server -> browser -> application under test and back for each command. When running on SaaS solutions the browser is running outside of the customer network with potentially large latencies in between.

Is there a limit on running tests in parallel?  

No. You can run as many tests in parallel as your hardware resources are capable of. If you want to run more tests, you simply need to add more CPU / RAM to the SBOX system.

What are the technical requirements to run SBOX on-premises?

SBOX runs on any Linux system. Most of our clients run RHEL or CentOS. SBOX typically runs in one or more virtual machines but can also run on bare metal. We can provide detailed technical requirements upon request.

How does the update process work?

After new browsers are released by the browser vendors we thoroughly test that the browsers are compatible with the driver and Selenium versions. Once all the tests pass on our end we release the bundles to the SBOX installations. SBOX immediately picks up the newly released versions and makes them available for use.

Does SBOX provide a user management/access control mechanism?

Yes. SBOX integrates seamlessly with your corporate LDAP where you can define usage and access rights for your organization.

Can I use self-signed SSL certificates?

Yes. SBOX supports self-signed SSL certificates.

Can I do a proof of concept in my environment?

Yes. This is our recommended approach. Larger companies often have specific requirements in terms of infrastructure, network, etc.

What do I need to prepare for a proof of concept?

We will send you a detailed list of requirements for the proof of concept. On a high level, you need 1-2 Linux machines (ideally VMs) with 8 logical CPU cores each; 16 GB RAM per VM and 150 GB of disk space. We can send you scripts and documentation on how to check that you fulfill the requirements.

Can I see my test(s) running in real time?

Yes. SBOX provides a live view feature which allows you to watch tests in real-time while it is running.

Can I use the debug console of any browser interactively when I run my tests?

Yes. You can manually interact with the browser where the test is running. This also allows you to open a debug console and debug in real-time.

Does SBOX provide a report of the passed/failed tests?

SBOX provides a sophisticated report to analyze your tests while running or after completion of a run. At the same time, SBOX seamlessly integrates into your client system in case you already have reports set up and want to plug in the SBOX tests.

What is the difference between a cloud service and SBOX?

The fundamental difference between a cloud service and SBOX is that SBOX runs completely in-house or in your private cloud. This results in superior performance over cloud solutions and allows for much tighter integration into your surrounding systems such as CI, identity management and reporting. In addition you have better debugging, monitoring and reporting capabilities as well as greater stability and support for custom requirements i.e. self-signed SSL certificates, unlimited scaling option.

Which browsers are supported?

We support Chrome, Firefox, Internet Explorer, Edge, MacOS Safari, WebKit and OpenFin in the most recent and all older versions.

Can I test on older browser versions as well?

Yes. After a new browser or Selenium image is released to the SBOX, older browser versions are still available. Tests can be executed on the latest as well as older browser versions.

Do I need to write my tests in a special way to execute them on SBOX?

No. SBOX uses the standard Selenium / WebDriver protocol without any modifications.

Does SBOX meet the requirements for a banking / financial environment?

Yes. SBOX was designed to operate independently from outside access. Banks, insurance and security sensitive organizations are our main client base.

How does the license model work?

SBOX is based on an annual or multi-year subscription.

Is HTTPS supported?

Yes. SBOX has built in End to End HTTPS support.

Does SBOX also support mobile?

Yes, SBOX fully supports the Appium protocol. You can run your tests on Android and iOS native apps, mobile web and hybrid apps.

Is there a limit on running tests in parallel?

No. You can run as many tests in parallel as your hardware resources are capable of. If you want to run more tests, you simply need to add more CPU / RAM to the SBOX system.

Do I need Windows machines to run Windows-based browsers?

No,  all browsers (except OSX Safari) run inside the Linux host. You don’t need to provision and manage any Windows machines for running Internet Explorer and Microsoft Edge browsers.  This is one of our key technical advantages that we are able to run Windows browsers natively in a Windows OS inside Linux.

Is SBOX a hardware appliance?

No. SBOX is a software solution that runs on your corporate servers or in your private cloud.

Can I see a report of my tests in SBOX (e.g. passed/failed)?

SBOX is a browser and mobile execution environment. The decision if a test is passed or failed is not taken by SBOX but rather by the client side (e.g. CI system). Therefore, it makes most sense to use a client side reporting tool (e.g. Allure, Extent) to visualize the test results. SBOX also provides an option to pushing the test results from the client to the SBOX. We however discourage this option in favour of client side reporting.

What are the key differentiators of SBOX vs. SaaS solutions?

Security (because everything is running inside your network. No data leaves your network, so there are no implications regarding data privacy etc.)Performance (tests are executed extremely fast. There are no latencies, because SBOX is running inside your network and close to the rest of your infrastructure)Integration (SSO, proxies, APIs, authentication)Scalability (you can run as many tests in parallel as you want)

Does SBOX support testing with Safari browsers?

SBOX supports Safari in two ways. You can either use Apple hardware as an executor in which case you can run native Safari browsers and iOS simulators. Alternatively, you can also run a Safari/WebKit browser on Linux which for functional testing purposes provides a safe target to test Safari. Using this approach means you can use your existing SBOX executors and don’t need to provision Apple hardware as executors.

WebKit is the open-source web platform implementation used by Apple Safari and Epiphany. Running a WebKit browser on Linux differs from Apple Safari in the following ways:

- it uses a non-macOS network stack
- it uses non-core Animation to composite scenes and produces image raster. This means that screenshots on Linux will not perfectly match screenshots from macOS.

In terms of running functional tests via Selenium or Playwright the same WebKit code will layout the page and run JavaScript—therefore it will match how WebKit works in Safari.

To summarize, running Safari/WebKit on Linux will use a different compositing pipeline than macOS, but it can be safely used for e2e testing of web applications in WebKit-powered browsers.

How much effort is it to operate SBOX?

SBOX is a fully self-contained system. There is no effort required to manage the SBOX when running in auto-update mode.

Does SBOX support custom proxy setups?

Yes. We fully support custom proxies based on your network architecture and requirements.

Can I run SBOX on AWS, Google Cloud, etc.?

Yes, SBOX is fully cloud compatible and can be run on physical servers, VMWare clusters, AWS, Google Cloud, Azure etc.

What is the benefit of a SBOX proof of concept?

A proof of concept will give the certainty that SBOX satisfies your needs BEFORE purchasing. You will have a fully functional system available for evaluation. During the evaluation period you can run your own tests on SBOX to ensure that you are seeing the results that you are aiming for.

Can I take videos of my tests?

Yes. A video can be taken on a per-test basis. The videos can be easily integrated into the reporting system or CI system.

Can I interact with my tests manually while they are running?

Yes. You can turn on manual interaction for your tests. This allows you to control the browser and debug your tests in more depth.

Does Selenium show a stream of commands that are executed?

Yes. You can see all the Selenium commands executed in the live view window.

Can I run tests against my local build (i.e. on my developer machine)?

Yes. Because SBOX is hosted in-house, it is no problem to also run tests with any supported browser against a build on your local machine.

Can I run Chrome in headless mode?

Yes, you can also run Chrome in headless mode, if you wish.