A Sandbox in the Clouds: Software Testing and Development in Cloud Storage

The words “testing and development” bring to mind engineers in white lab coats marking clipboards as they hover over a buzzing, whirring machine. The reality in app development is more often a DevOps team in a rented room virtually poking and prodding at something living on a server somewhere. But how does that really work?

Think of testing in the cloud like taking your app or software program to train at an Olympic-sized facility instead of your neighbor’s pool. In app development, building the infrastructure for testing in a local environment can be costly and time-consuming. Cloud-based software development, on the other hand, gives you the ability to scale up resources when needed without investing in the infrastructure to, say, simulate thousands of users.

But first things first…

What Is Cloud Software Testing?

Cloud software testing uses cloud environments and infrastructure to simulate realistic user traffic scenarios to measure software performance, functionality, and security. In cloud testing, someone else owns the hardware, runs the test, and delivers the test results. On-premise testing is limited by budgets, deadlines, and capacity, especially when that capacity may not be needed in the future.

Types of Software Testing

Any software testing done in a local test environment can be done in the cloud, some much more efficiently. The cloud is a big sandbox, and software testing tools are the shovels and rakes and little toy dump trucks you need to create a well-functioning app. Here are a few examples of how to test software. Keep in mind, this is by no means an exhaustive list.

Stress Testing

Stress tests measure how software responds under heavy traffic. They show what happens when traffic spikes (a spike test) or when high traffic lasts a long time (a soak test). Imagine putting your app on a treadmill to run a marathon with no training, then forcing it to sprint for the finish. Stress testing software in an on-premise environment involves a significant amount of capital build-out—servers, software, dedicated networks. Cloud testing is a cost-effective and scalable way to truly push your app to the limit. Companies that deal with big spikes in traffic find stress testing particularly useful. After experiencing ticketing issues, the Royal Opera House in London turned to cloud stress testing to prepare for ticket releases when traffic can spike to 3,000 concurrent users. Stress testing in the cloud enables them to make sure their website and ticketing app can handle the traffic on sales days.

Load Testing

If stress testing is a treadmill, load testing is a bench press. Like stress testing, load testing measures performance. Unlike stress testing, where the software is tested beyond the breaking point, load testing finds that breaking point by steadily increasing demands on the system until it reaches a limit. You keep adding weight until your app can’t possibly do another rep. Blue Ridge Networks, a cybersecurity solutions provider based in Virginia, needed a way to test one of their products against traffic in the millions. They could already load test in the hundreds of thousands but looked to the cloud to scale up. With cloud testing, they found that their product could handle up to 25 million new users and up to 80 million updates per hour.

Performance Testing

Stress and load tests are subsets of software performance testing—the physical fitness of the testing world. The goal of performance testing is not to find bugs or defects, but rather to set benchmarks for functionality (i.e., load speed, response time, data throughput, and breaking points). Cloud testing is particularly well-suited to software performance testing because it allows testers to create high-traffic simulations without building the infrastructure to do so from scratch. Piksel, a video asset management company serving the broadcast media industry, runs performance tests each night and for every new release of their software. By testing in the cloud, they can simulate higher loads and more concurrent users than they could on-premise to ensure stability.

Latency Testing

If stress testing is like training on a treadmill, latency testing is race day. It measures the time it takes an app to perform specific functions under different operating conditions. For example, how long it takes to load a page under different connection speeds. You want your app to be first across the finish line, even under less than ideal conditions. The American Red Cross relies on its websites to get critical information to relief workers on the ground in emergencies. They need to know those sites are responsive, especially in places where connection speeds may not be very fast. They employ a cloud-based monitoring system to notify them when latency lags.

Functional Testing

If performance testing is like physical training, functional testing is like a routine physical. It checks to see if things are working as expected. When a user logs in, functional testing makes sure their account is displayed correctly, for example. It focuses on user experience and business requirements. Healthcare software provider Care Logistics employs automated functional testing to test the functionality of their software whenever updates are rolled out. By moving to the cloud and automating their testing, they reduced their testing time by 50 percent. Functional testing in the cloud is especially useful when business requirements change frequently because the resources to run new tests are instantly available.

Compatibility Testing

Compatibility testing checks to see if software works across different operating systems and browsers. In cloud testing, as opposed to on-premise testing, you can simulate more browsers and operating systems to ensure your app works no matter who uses it. Mobile meeting provider LogMeIn uses the cloud to test it’s GoToMeeting app on 60 different kinds of mobile devices and test their web-based apps daily across multiple browsers.

Smoke Testing

In the early days of technology development, a piece of hardware passed the smoke test if it didn’t catch on fire (hence, smoke). Today, smoke testing in software testing makes sure the most critical functions of an app work before moving on to more specific testing. The grocery chain Supervalu turned to cloud testing to reduce the time they spent smoke testing by 93 percent. And event management platform Eventbrite uses the cloud to run 20 smoke tests on every software build before running an additional 700 automated tests.

Advantages of Cloud Development vs. Traditional Software Development (and Some Drawbacks)

• Savings – Only pay for the resources you need rather than investing in infrastructure build-out and maintenance, saving money and time spent developing a local test environment.
• Scope – Broaden the number of different scenarios you can test — more browsers, more operating systems — to make sure your software works for as many users as possible.
• Scalability – Effortlessly scale your resources up or down based on testing needs from initial smoke testing to enterprise software development in the cloud.
• Speed – Test software on different operating systems, platforms, browsers, and devices simultaneously, reducing testing time.
• Automation – Easily employ automated software testing tools rather than dedicating an employee or team to test software manually.
• Collaboration – As more and more companies abandon waterfall in favor of agile software development, the role of development, operations, and QA continues to blend. In the cloud, developers can push out new configurations or features, and QA can run tests against them right away, making agile development more manageable. For example, cloud testing allowed the Georgia Lottery System to transition from releasing one to two software updates per year with waterfall development to 10+ releases each quarter with agile.

Moving your testing to the cloud is not without some drawbacks. Before you make the move, consider the following:

• Outages – In March of 2019, Amazon Web Services (AWS) suffered an outage at one of their data centers in Virginia. The blackout affected major companies like Slack, Atlassian, and Capital One. For a few hours, not only were their services affected, those companies couldn’t test any web properties or apps running on AWS.
• Access – The nature of cloud services means that companies pay for the access they need. It’s an advantage to building infrastructure on-site, but it puts the onus on companies to determine who needs access to the testing environments housed on the cloud and what level of access they need to keep cloud testing affordable.
• Lack of universal processes – Because each cloud provider develops its own infrastructure and systems (and most are very hush-hush about it), companies who want to switch providers face the burden of reconfiguring their internal systems and data to meet new provider requirements.

What Does Cloud Testing Cost?

Most cloud service providers offer a tiered pricing structure. Providers might charge per device minute (in mobile testing) or a flat fee for unlimited testing. Flat fees start around $100 per month up to $500 per month or more. Many also offer private testing at a higher rate. Start by determining what kind of testing you need and what tier makes the most sense for you.

Who Uses the Cloud for Software Testing?

As shown in the examples above, organizations that use the cloud for testing are as varied as they come. From nonprofits to grocery chains to state lottery systems, any company that wants to provide a software application to improve customer service or user experience can benefit from testing in the cloud.

No longer limited to tech start-ups and industry insiders, testing in the cloud makes good business sense for more and more companies working to bring their apps and software solutions to the world.