fbpx

Load Testing: HTTP vs Headless vs Real Browser

An outline of the main aspects of load simulation methods such as HTTP, headless, and real browser-based followed by a comparison matrix, to help you choose an appropriate simulation approach.
performance-testing

Load Testing:

HTTP vs Headless vs Real Browser

performance-testing
Overview:

Slow loading or non-responsive web pages have an impact on financial revenue because frustrated users will most likely not return once the issue has been solved. Therefore, performance testing has become a fundamental part of the development chain and the demand is still growing.

Performance testing platforms provide a broad range of load simulation methods such as HTTP, headless, and real browser-based. In this paper, we will outline the main aspects of those followed by a comparison matrix, which you can use for choosing an appropriate simulation approach.

HTTP Based Load Simulation

In the early days of the digital age, HTTP-based testing was very popular. With the rise of rich web client technology the HTTP-based simulation approaches have become increasingly outdated. A typical HTTP-based test driver executes service requests and parses responses. Modern web 2.0 applications consist of many client-side scripts, which are totally ignored, and not measured in this type of test execution. In worst-case scenarios, complex use cases cannot be simulated on protocol level due to a shortage of client side generated ids.

Due to their request and response-based nature, the overhead on the load injection machine is very low. A typical load test server can simulate up to 800 simultaneous sessions. Complex protocol based use cases can be difficult to implement. A performance engineer needs to deal with cookies, session IDs, and other dynamic parameters. Depending on the type of system being tested, some web form names often change once a new version has been deployed which will cause the HTTP-based script to fail.

// sample protocol level script 

transaction TMain 

var 

hContext: number; 

begin 

WebPageUrl(“http://lab3/st/”, “Greetings”); 

WebPageStoreContext(hContext); 

WebPageLink(“Join the experience!”, ” – New Visitor”); 

WebPageSubmit(“Continue”, CONTINUE001, “Main menu”); 

WebPageLink(“Products”, “ShopIt – Products”); 

WebPageLink(NULL, “ShopIt – Product”, 3); 

WebPageSubmit(“Search”, SEARCH001, ” – Search”, 0, NULL, hContext); 

  end TMain; 

dclform 

CONTINUE001: 

    “name”            := “jack”, 

    “New-Name-Button” := “Continue”; 

SEARCH001: 

    “search”          := “boot”; 

The sample script seen here is a showcase for the very technical nature of those scripts. If a technical attribute of your application changes, then you have to rewrite the whole script.

After all, protocol level scripts are good for component level tests in continuous integration environments and the perfect setting for stress testing due to their low footprint on load injection machines.

Headless Browser Based Load Simulation

With the rise of web 2.0 technologies, the testing business was faced with serious challenges. Rich browser applications could no longer be tested or simulated on the protocol level due to the missing client side logic during script replay. Therefore, several headless browsers have been introduced such as HtmlUnit, PhantomJS or SlimerJS. They are often built on top of WebKit, the engine behind Chrome and Safari.

Headless Browsers are similar to real browsers without the GUI. Many test-automation and performance testing platforms are using headless browsers to simulate traffic.

Headless browsers have their own pitfalls; as new browsers enter markets headless browser kits need to catch up to all the performance and feature enhancements of the real browsers.

The simulation of client side rendering of is not for free. A typical load injection server can simulate up to 10-12 simultaneous headless browser sessions, versus 500 of HTTP based sessions.

Test script implementation and customization is not too difficult. If you have basic coding skills you will be able to create simple scripts. Not all headless-browsers provide visual replay features and without visual replay script debugging or error analysis could become very tricky.

// sample phantomjs script
“use strict”;
var page = require(‘webpage’).create(),

server = ‘http://posttestserver.com/post.php?dump’,
data = ‘universe=expanding&answer=42’;

page.open(server, ‘post’, data, function (status) {

if (status !== ‘success’) {

console.log(‘Unable to post!’);

} else {

console.log(page.content);

}
phantom.exit();

});

In the sample script seen here a simple post request is executed. You need Java programming skills to customize such test scripts.

Real Browser Based Load Simulation

Web2.0 applications are full of JavaScript, Flash, Ajax and CSS. Without a full browser, it’s not possible to measure the actual end-to-end response times of the whole web page. Real browser-based performance testing allows you to verify the site’s functionality and speed as perceived by the end-user.

A typical real browser performance test solution collects loading times of images, JavaScript, CSS and more. Often they provide waterfall charts, which visualize the load time of those components.

The footprint of a real browser based browser is slightly higher. Considering the fact that headless browser simulation does not deliver 100% realistic response times it’s fair to say that real browser based simulation should be preferred. In real life scenarios, headless browsers perform only 20% better than real browsers. So, if average-sized single load injector can run 10-12 headless browser sessions, the same load injector can run 8-10 real browser sessions.

Implementation and maintenance of test scripts is easy because user actions are directly reflected and the visual replay makes debugging easy. In the sample script below the browser opens a URL, inserts user and password and clicks the login button.

// sample real browser-based script 

transaction TMain 

begin  

    BrowserStart(BROWSER_MODE_DEFAULT, 800, 600);   

    // navigate to the login site 

    BrowserNavigate(“http://demo.com/TestSite/LoginForm.html”); 

    // set the authentication for the secure site 

    BrowserSetText(“//INPUT[@name=’user’]”, “User1”); 

    BrowserSetPassword(“//INPUT[@name=’pwd’]”,     “Password1”); 

    // Login 

    BrowserClick(“//INPUT[@value=’Login’]”, BUTTON_Left); 

end TMain; 

After all, real browser simulation is useful for realistic end-to-end load tests, but it may get expensive for stress testing high user volumes, because the footprint on the load injection server is too high.

Performance Test Types

Component Speed Tests

In recent years software development methods have moved in the agile direction. Short release sprints are essential. Developers and test engineers automate their quality assurance and performance checks. Typically, they implement service based performance tests on protocol level or they simulate real browser based performance checks to compare end-to-end response times with agreed performance boundaries.

Objectives 

+ Repeatability 

+ Automated interface and end-to-end performance checks 

+ Compare response times with agreed thresholds 

Load Tests

For several reasons load tests are the ideal testing method when it comes to verification of non-functional requirements. One being that the response times can be verified under reproducible conditions. Another being that those tests allows verification of response time thresholds. Realistic response time measurement is essential in load test scenarios. Therefore, test engineers use headless or real browser based user simulation for their load test settings. 

 

Objectives 

+ Reproducible load simulation 

+ Verification of response time thresholds 

+ Identify bottlenecks under production like load conditions  

+ Realistic end-to-end test scenarios 

Stress Test

If you have to test the reliability of your application under peak load conditions, run a stress test. In this type of test you specify mainly the max number of users and the time over which the ramp up and the steady state load should be on your application. The goal is to identify the breaking points of your application under test. 

Objectives 

+ Proof scalability and stability 

+ Simulate peak load conditions 

+ Exact reproducibility is not important 

Comparison

Obviously, there are good reasons for protocol, headless or real browser based user simulation. The matrix below provides some guidance to choose the appropriate approach.

Criteria

HTTP

Headless Browser

Real Browser

User simulation 

No client side rendering

Some client-side elements are simulated

Real user simulation

Script implementation and customization

Difficult when web sites are complex

Developer skills required to build robust scripts 

Simple scripts, easy to customize 

Script replay 

 Low level analysis required 

Depending upon the engine used, visual replay is possible 

You see what you get 

Script maintainability

 Programming skills required 

Errors in un-rendered sections are tricky to solve 

Easy because you see failures during replay 

Multi Browser Support 

 Some tools emulate web browsers, but this is not comparable 

Yes, but some elements are often missing 

Some support other versions and different browsers 

Footprint on load injection machine 

Low, up to 800 sessions per server 

Medium, up to 10-12 sessions per server 

High, up to 6 sessions per server 

Recommended for DevOps 

Depends on actual test scenario 

 No, often complex scripts 

Yes, easy to use and realistic figures 

Recommended for Load Tests

 No, client-side processing is skipped

Yes, better than HTTP simulation 

 Yes, realistic user simulation 

Recommended for Stress Tests 

Yes, because there is a low overhead on the load generator machine  

No, overhead on the load generator machine is too high 

 No, highest overhead on load generator machine 

Costs

Low 

High

High

Recommended for high volume, low cost tests webserver stress tests (where possible) 

Not Recommended 

Recommended for real-life complex application simulations. 

Stress Test

If you have to test the reliability of your application under peak load conditions, run a stress test. In this type of test you specify mainly the max number of users and the time over which the ramp up and the steady state load should be on your application. The goal is to identify the breaking points of your application under test. 

Objectives 

+ Proof scalability and stability 

+ Simulate peak load conditions 

+ Exact reproducibility is not important 

Comparison

Obviously, there are good reasons for protocol, headless or real browser based user simulation. The matrix below provides some guidance to choose the appropriate approach.

Criteria HTTP Headless Browser Real Browser
User simulation No client side rendering Some client side elements are simulated Real user simulation
Script implementation and customization Difficult when web sites are complex Developer skills required to build robust scripts Simple scripts, easy to customize
Script replay Low level analysis required Depending on used engine is visual replay possible You see what you get
Script maintainability Programming skills required Errors in not rendered sections are tricky to solve Easy because you see failures during replay
Multi Browser Support Some tools emulate web browser but this is not comparable Yes, but some elements are often missing Some support other versions and different browsers
Footprint on load injection machine Low, up to 800 sessions per server Medium, up to 10-12 sessions per server High, up to 6 sessions per server
Recommended for DevOps Depends on actual test scenario  No, often complex scripts Yes, easy to use and realistic figures
Recommended for Load Tests No, client side processing is skipped out Yes, better than HTTP simulation Yes, realistic user simulation
Recommended for Stress Tests Yes, because there is a low overhead on load generator machine No, overhead on load generator machine is too high No, highest overhead on load generator machine
Costs Low High High
  Recommended for high volume, low cost tests webserver stress tests (where possible) Not Recommended Recommended for real-life complex application simulations.

 

Share on facebook
Facebook
Share on twitter
Twitter
Share on linkedin
LinkedIn
Share on email
Email
Share on print
Print