Overview
Slow loading or non-responsive web pages have an impact on financial revenue because frustrated users will most likely not return once the issue has been solved. Therefore, performance testing has become a fundamental part of the development chain and the demand is still growing.
Performance testing platforms provide a broad range of load simulation methods such as HTTP, headless, and real browser-based. In this paper, we will outline the main aspects of those followed by a comparison matrix, which you can use for choosing an appropriate simulation approach.
HTTP Based Load Simulation
In the early days of the digital age, HTTP-based testing was very popular. With the rise of rich web client technology the HTTP-based simulation approaches have become increasingly outdated. A typical HTTP-based test driver executes service requests and parses responses. Modern web 2.0 applications consist of many client-side scripts, which are totally ignored, and not measured in this type of test execution. In worst-case scenarios, complex use cases cannot be simulated on protocol level due to a shortage of client side generated ids.
Due to their request and response-based nature, the overhead on the load injection machine is very low. A typical load test server can simulate up to 800 simultaneous sessions. Complex protocol based use cases can be difficult to implement. A performance engineer needs to deal with cookies, session IDs, and other dynamic parameters. Depending on the type of system being tested, some web form names often change once a new version has been deployed which will cause the HTTP-based script to fail.
Headless Browser Based Load Simulation
With the rise of web 2.0 technologies, the testing business was faced with serious challenges. Rich browser applications could no longer be tested or simulated on the protocol level due to the missing client side logic during script replay. Therefore, several headless browsers have been introduced such as HtmlUnit, PhantomJS or SlimerJS. They are often built on top of WebKit, the engine behind Chrome and Safari.
Headless Browsers are similar to real browsers without the GUI. Many test-automation and performance testing platforms are using headless browsers to simulate traffic.
Headless browsers have their own pitfalls; as new browsers enter markets headless browser kits need to catch up to all the performance and feature enhancements of the real browsers.
The simulation of client side rendering of is not for free. A typical load injection server can simulate up to 10-12 simultaneous headless browser sessions, versus 500 of HTTP based sessions.
Test script implementation and customization is not too difficult. If you have basic coding skills you will be able to create simple scripts. Not all headless-browsers provide visual replay features and without visual replay script debugging or error analysis could become very tricky.
Real Browser Based Load Simulation
Web2.0 applications are full of JavaScript, Flash, Ajax and CSS. Without a full browser, it’s not possible to measure the actual end-to-end response times of the whole web page. Real browser-based performance testing allows you to verify the site’s functionality and speed as perceived by the end-user.
A typical real browser performance test solution collects loading times of images, JavaScript, CSS and more. Often they provide waterfall charts, which visualize the load time of those components.
The footprint of a real browser based browser is slightly higher. Considering the fact that headless browser simulation does not deliver 100% realistic response times it’s fair to say that real browser based simulation should be preferred. In real life scenarios, headless browsers perform only 20% better than real browsers. So, if average-sized single load injector can run 10-12 headless browser sessions, the same load injector can run 8-10 real browser sessions.
Implementation and maintenance of test scripts is easy because user actions are directly reflected and the visual replay makes debugging easy. In the sample script below the browser opens a URL, inserts user and password and clicks the login button.
Performance Test Types
Component Speed Tests
In recent years software development methods have moved in the agile direction. Short release sprints are essential. Developers and test engineers automate their quality assurance and performance checks. Typically, they implement service based performance tests on protocol level or they simulate real browser based performance checks to compare end-to-end response times with agreed performance boundaries.
Objectives
- + Repeatability
- + Automated interface and end-to-end performance checks
- + Compare response times with agreed thresholds
Load Tests
For several reasons load tests are the ideal testing method when it comes to verification of non-functional requirements. One being that the response times can be verified under reproducible conditions. Another being that those tests allows verification of response time thresholds. Realistic response time measurement is essential in load test scenarios. Therefore, test engineers use headless or real browser based user simulation for their load test settings.
Objectives
- + Reproducible load simulation
- + Verification of response time thresholds
- + Identify bottlenecks under production like load conditions
- + Realistic end-to-end test scenarios
Stress Test
If you have to test the reliability of your application under peak load conditions, run a stress test. In this type of test you specify mainly the max number of users and the time over which the ramp up and the steady state load should be on your application. The goal is to identify the breaking points of your application under test.
Objectives
- + Proof scalability and stability
- + Simulate peak load conditions
- + Exact reproducibility is not important
Comparison
Obviously, there are good reasons for protocol, headless or real browser based user simulation. The matrix below provides some guidance to choose the appropriate approach.
| Criteria | HTTP | Headless Browser | Real Browser |
|---|---|---|---|
| User simulation | No client side rendering | Some client-side elements are simulated | Real user simulation |
| Script implementation and customization | Difficult when web sites are complex | Developer skills required to build robust scripts | Simple scripts, easy to customize |
| Script replay | Low level analysis required | Depending upon the engine used, visual replay is possible | You see what you get |
| Script maintainability | Programming skills required | Errors in un-rendered sections are tricky to solve | Easy because you see failures during replay |
| Multi Browser Support | Some tools emulate web browsers, but this is not comparable | Yes, but some elements are often missing | Some support other versions and different browsers |
| Footprint on load injection machine | Low, up to 800 sessions per server | Medium, up to 10–12 sessions per server | High, up to 6 sessions per server |
| Recommended for DevOps | Depends on actual test scenario | No, often complex scripts | Yes, easy to use and realistic figures |
| Recommended for Load Tests | No, client-side processing is skipped | Yes, better than HTTP simulation | Yes, realistic user simulation |
| Recommended for Stress Tests | Yes, because there is a low overhead on the load generator machine | No, overhead on the load generator machine is too high | No, highest overhead on load generator machine |
| Costs | Low | High | High |
| Recommended for high volume, low cost tests webserver stress tests (where possible) | Not Recommended | Recommended for real-life complex application simulations |
Web pages used for this paper:
- https://developers.google.com/web/tools/chrome-devtools/device-mode/testing-other-browsers
- https://watirmelon.blog/2015/12/08/real-vs-headless-browsers-for-automated-acceptance-tests/
- http://news.softpedia.com/news/what-is-a-headless-browser-and-what-s-it-good-for-485162.shtml
- https://github.com/dhamaniasad/HeadlessBrowsers
- https://circleci.com/