Load Testing Too Late: Don’t Miss Opportunities to Reach Your Customers. (Pt. 1 of 5)

Load Testing Too Late


You’ve walked out of a store empty-handed, gotten frustrated with a website, or given up on a tedious checkout process before, so you know how bad customer experiences can cost you money.

In this five part blog post series, we will take a look at the ways using Blitz can save you money by helping you get ahead of website scalability problems before your customer has a bad experience with your application.

We will take a look at the some of the hazards that Blitz can help you proactively avoid, and also offer some solutions that will keep your revenue streams flowing instead of trickling to a stop right when your audience begins to grow. Load testing doesn’t have to be a burden. Blitz makes it easy to send mass amounts of scheduled traffic to your website or application to help you see how your app fairs under the most bone-crushing, load-bearing conditions.

In this five part series, we will cover:

  1. Introduction – Don’t Miss Opportunities to Reach Your Customers
  2. The Abandoned Cart
  3. Loss of Revenue
  4. Bad User Experience
  5. Brand Reputation

Throughout this blog series, we will cover these topics more in-depth to show you how seemingly small problems can have a negative impact across your site, and how a little proactive load-testing can help troubleshoot those problems before your customers have to experience them. Anything from buttoning up your checkout process to maximize cart conversion, to ensuring your customers have a stellar encounter with your brand when they visit your site, the Blitz team wants to help you make the most of your web presence and keep your customers happy by giving you the tools to test the limits of your website’s performance.

The ingredients to a good customer experience: a good first impression, an easy to navigate site, building trust, providing security, presenting a well-oiled machine, and now, offering a seamless web app experience. Though technology has advanced, customers essentially care about the same qualities. They just experience them in different ways.

Imagine taking a car to market without testing its performance, its durability, its ability to take tight turns, or stop on a dime. What if its safety features weren’t suitable to protect people in accidents? What if the car began to fall apart the first time it hit a bump in the road? What if the car salesman can’t get the engine to start on the showroom floor? Will the customer stick around? Suppose he gets the car running, and the customer notices the car has some small flaws during the test drive? Will people buy this car? And suppose they do, and the first wave of people to buy the car quickly begin to realize they’re having a terrible experience with their purchase. What will they say about your car when people ask? How much will you lose on the showroom floor before people even walk out of the dealership? How much will you tarnish your brand’s reputation by not testing the vehicle’s performance before it hits the market?

Web apps are the same way. Neglecting the load-testing phase before going to market possesses all of the same types of pitfalls as failing to test the performance or safety of a car. We know your app has solid potential, and your customers want all the benefits that it provides, but simple scalability and performance issues will keep them from pulling the trigger. They will keep them looking at your competitors, and they will keep them from trusting you to deliver your services in exchange for their hard-earned cash. Don’t let poor scalability or performance be the reason you lose out on customers.

You’ve done the legwork to get your app this far, use Blitz to take it to the next level. Visit www.blitz.io and sign up for a 14 day free trial.

Load Testing on your Schedule

Make it even easier to get fast, accurate results when launching a load test by using our new scheduling feature.

  • Users have the option of turning on the scheduler when in it is important to run tests for specific times.
  • Schedule tests to run after routine site updates
  • Choose dates and times in addition to selecting a multi-region sprint to ensure accuracy in a particular time zone before going live with a site.

Simply select your date and time from the drop down when setting up the test or opt for a recurring schedule!

Load Testing on your Schedule

Blitz + ArmorHub

Blitz + ArmorHub

Secure your website and maintain optimum performance through scanning and load testing with the all new Spirent ArmorHub and Blitz. A perfect combination to increase scalability and security at an affordable price! Just this month, we announced our newest addition to our cloud platform- ArmorHub!  Check it out with a free trial or sign up for as little as $15 a month and schedule recurring vulnerability scans for your website!

With ArmorHub, anyone with a web presence can scan their sites for the top trending vulnerabilities including Heart Bleed. Built in features include the option to schedule recurring scans at specific times and email the report to your inbox. Blitz and ArmorHub together make a great pair –If you’re making changes to your website, use our newest scheduling feature with Blitz to load test and then schedule a website vulnerability scan to ensure your site is still safe with ArmorHub! – All under the same username as password you use with Blitz! Learn more at www.ArmorHub.com

Blitz and Red Hat’s OpenShift PaaS Partnership Program

Here at Blitz, we’re all about making things simple, effective and fun. It’s the driving force behind our load-testing tool, and it’s why we have the partners we have.

That’s why we’re excited about our involvement with Red Hat’s OpenShift PaaS Partnership Program, as they announce the extension of their program to include more partners in line with the launch of their on-premise OpenShift Enterprise PaaS product.

What does this unique partnership mean?

It means that existing OpenShift users can deploy Blitz onto their platforms for capacity planning, scalability, optimization, and performance monitoring. Blitz brings a simple form to OpenShift users so they can better understand the performance of their apps. At any given time, OpenShift users can see what they need to do to optimize their website or application.

And on the other side, this partnership enables users who don’t yet have an application or website to load test can see how the end-user experience of Blitz works. Essentially, we’ve worked with OpenShift to pre-deploy a number of different apps so that people can run load tests against them, even though those apps don’t belong to them. They get to experience Blitz firsthand without needing a product to test.

So, as you can see, this partnership makes a lot of sense, for both Red Hat’s users and for those who want to use Blitz. We’re happy that we can continue to provide a well-rounded, complete user experience for our partners’ users and especially for our own Blitz community.

Announcing Blitz Credits

Blitz creditsLast week, we talked a lot about the freemium model and the lessons we have learned so far in our journey with Blitz. We also asked that you stay tuned for some exciting news that demonstrates our biggest lesson learned: be flexible.

The number-one complaint we’ve heard from our customers doesn’t have to do with the Blitz product, but rather, our pricing model. Currently, users purchase by the hour to test on our system – this is what we call a wall-clock system – basically, time starts at the swipe of a credit card. Our users found this inconvenient for two reasons: 1.) they have to be ready to start testing as soon as they swipe their credit cards, and 2.) they were never able to test for the full hour. If a mistake occurred at the beginning of the test, they would spend their time fixing it, instead of running the test.

Our customers started asking to pay by testing runs instead of by time. And since everything in testing is project-based – whether new code is being introduced for validation or a website or app is launched – we realized that it made sense to change.

So we’re introducing our new pricing model – a Credit-based system that will allow users to pay by test, which allows customers to fine-tune their apps in-between tests without worrying about the clock.  Moving from a wall-clock system to a credit-based system gives the customer control of when they want to do their testing, and ensures they get what they pay for.

Here is how our credit based system works:

  • Pay by using credits; each credit is a 1-minute, 1,000-concurrent-user rush for $1
  • Customers can take advantage of bundles that offer savings for buying in bulk
  • Credits don’t expire and are available whenever customers are ready to start testing
  • Existing customer accounts will be converted to this Credit-based plan automagically.

You can also earn FREE credits:

  • Blogging earns 100 credits!
  • Referring friends earns you a credit per friend—are you popular?
  • Monthly top-up; everyone starts the month with at least 10 credits!

We think that this pricing model will take performance testing to the next level. Starting October 30th, everyone will be able to do their load testing based on this new plan. We’re so excited to help our customers reach their goals and succeed with Blitz. As always, we’re here for your questions, concerns, and comments. Email us at hello@blitz.io, or give us a shout on Twitter @Blitz_io.

Blitz and New Relic Connect

Today, our partner New Relic announced the launch of their New Relic Connect program. New Relic provides a SaaS-based application and performance management tool for Ruby, PHP, Python, Java and .NET web apps.

Connect is designed to make our users’ – and admittedly, our own – lives a little bit easier by letting partners with complimentary technologies easily combine New Relic’s rich app performance data with their own solutions and services. It’s the latest addition to New Relic’s world-class partner program and we’re excited to be among the first to participate.

We’ve actually been integrated with New Relic for a little while, but today we’re making the partnership official.  The integration is now fully supported and, as an additional perk, all Blitz customers who sign up with New Relic will get New Relic Standard free of charge!

So how exactly does it work? New Relic gives you insight how your Web application is performing in real time.  When you integrate your Blitz account with New Relic, you can monitor the performance of your application from the inside while running a load test.  New Relic metrics will be displayed along with the Blitz results so that it’s easier to find and fix your performance bottlenecks.

So, it’s easy to see how complimentary this partnership is. And coupled with some exciting news we’ll be announcing on Monday that will make Blitz easier and more effective for our customers, this integration doesn’t just make perfect sense, but it also is extremely beneficial for both Blitz and New Relic users.

Connect New Relic to your Blitz account your Plugins page.

Sign up for a New Relic account and get New Relic Standard free of charge!

Lessons Learned and Looking Forward with Blitz

We’ve been in business for a year now, and what a year it’s been. We’ve grown and learned so much in such a short amount of time that we felt it wouldn’t be fair if we kept all of those lessons to ourselves. So we wanted to pass on to anyone who is trying to start their own freemium-based products some of the things we’ve learned along the way.

When we first started, we launched Blitz as a load tester in the cloud. Blitz takes advantage of two major trends that are becoming more and more prevalent within software and application development. The first has to do with “where” – development is moving into the cloud. This means that the “where” development done is no longer tied to specific organizations or dedicated environments, opening access to software development in an unprecedented way.

The second trend in development deals with the “who.” The actual process of development testing is now done by a new group of folks: the developers. What used to be two disciplines – performance engineering and development – the ever-so-popular DevOps movement means that things are blending together, and developers are now the ones doing the testing. Which also means they are on the hunt for tools that help them do this.

Another key aspect of Blitz, outside of the unique needs that it addresses for developers, is the “freemium” model on which it is based, which allows developers to apply load tests, otherwise known as “rushes,” of up to 250 concurrent users for up to 1 minute, for an unlimited number of times, at this level.

Keep in mind that freemium isn’t for everyone. You really have to run the math and see if it makes sense for your application. But, if you’ve done your homework and you think that the freemium-model path makes sense for you, we at Blitz have done some of the dirty work for you, and here are some of the things we’ve discovered throughout the process:

Invest in the Correct Outreach Efforts

You may have noticed, we don’t do traditional media and marketing. Our users know our product, and they don’t need a cheerleader bombarding them about how great it is. What we do care about is our community, and what they are telling us – not vice versa. That’s why we’ve put our resources into social media, which allows us to actually communicate with our customers rather than communicate at them.

Think Big Picture

In a world of instant results – when a delay of two seconds puts you behind 90 percent of the population – it’s naturally difficult to think longer-term, especially at the expense of short-term.  And of course, it’s great to give all customers everything they want, but that’s pretty close to impossible. For example, we’ve had users who have requested that we add some functions like specific encryption or testing that would pull us away from making things simple and easy for them to use, and we’ve unfortunately had to say no. Again, it’s not like it gives us pleasure to not be able to offer our customers the moon and beyond, it’s just not realistic, and in the end, these requests didn’t serve our Blitz community as a whole. You have to always look at the bigger picture and trends – it’s not about short-term gains, but about long-term success.

Referral Programs are Your Best Friends

We implemented some successful referral programs in our pricing model, which ties back to our first lesson about investing in appropriate outreach methods. The best thing about social media is that it gets the word out about your product, and encourages customers to talk about you, which leads us to creating referral plans that have kept our existing users happy, while simultaneously helping us grow that user base. For instance, if you refer a friend to Blitz, we give you an extra 25 users for life for your rushing.  In this way, our customers are spreading our name through word-of-mouth, and we get to help those loyal customers keep testing at an even higher level.

Be There

It seems like this point would be obvious, but seriously, having a product or solution that is based on do-it-yourself doesn’t mean that your customers should do it by themselves….and only by themselves. The DIY method means that you as a provider need to have good documentation, good content, and at the end of the day, have a live person available to help your customers when they need it. Be responsive and let them know that you’re there when they have questions, concerns, or need your assistance.


No, we’re not suggesting you do anything inappropriate to garner customers for your freemium model. We’re suggesting the principle: Keep It Simple, Stupid. We want to urge you to have a clear problem for which your product or service is answering a clear solution – and stick to it.  It’s this simplicity on which we’ve based our product – and our freemium model – that has given us our success.

So given all of those hard-learned lessons we’ve discovered over the past year, we have to say we’re pretty happy with where we are now. Blitz has emulated 1 billion virtual users from the cloud, generated from seven different locations around the world. We’ve run 250,000 load tests for almost 30,000 users, testing 17,000 domains. Our convergence from freemium users to paid users is around 2 percent. Not bad for one year!

But despite those lessons, and despite our successes, the most important lesson we learned through our journey so far is to be agile and responsive. And it’s this lesson that we’ve applied to the top feedback we’ve gathered from our users.

So looking forward, we’ve got some great new changes down the pipeline that are going to help our users become more efficient and more productive. We’re really excited to announce these changes – stay tuned for more!

How we use DynamoDB and what we’ve learned along the way

amazon datacenter

Recently, we started to store a history of tests run by our Blitz users.  With over a billion hits so far, we’re growing quickly and we want to make sure that we’ll continue to be able to keep up with our users’ needs.  Eventually we chose Amazon’s DynamoDB as our storage database for this task because it frees us from tedious operations like backups, monitoring, and increasing capacity.  You can see how it performs by going to our http://blitz.io/dashboard.  After running a few tests, they will all show up in the dashboard for later reference.  We had a few surprises as we were bringing up our DynamoDB, so we thought we’d share some lessons learned. Read on…

Lesson 1. DynamoDB doesn’t have JSON types

DynamoDB is a schema-less NoSQL database, with the data stored as key-value pairs. Unlike other NoSQL databases that we’ve used, such as MongoDB or CouchDB, which operate on JSON documents, DynamoDB only accepts Number, String or Binary values.  So, to store Hash or Array objects, we must handle serialization ourselves.  We convert these to JSON strings before sending to DynamoDB, then back from JSON on each fetch. This works well, but you need to be careful not to exceed the 64KB maximum item size.

Lesson 2. Break up your data into fine-grained attributes

To use DynamoDB we need to provision read/write throughput capacity.  Amazon charges a flat hourly rate based on the capacity we reserve.  To keep costs low, we want to retrieve only the data we need on each query.

The configuration and results of our tests have a good amount of data.  And in the dashboard page, we only show the most relevant information, such as response time, error rate and hit rate.  So, before we send the data to store in DynamoDB, we extract or compute those fields and put them into their own key-value pairs.  Then when we query for data to show in the dashboard page, we just need to specify those keys to retrieve, which greatly reduces the overall size of each query result.

Lesson 3. Use the :select option to boost query performance

We designed our tests table with a composite primary key.  The :hash_value is a UUID of the user, the :range_value is the timestamp which is the start time of the test.  For such tables in DynamoDB, we can use the query API to query the test items.  It takes in some options and the :hash_value is required.  We can either specify different kinds of range values or use the :limit option to filter down the tests we want to retrieve from the table.

In our initial implementation, we were doing something like the following:

tests = items.query(
  :hash_value => "user UUID",
  :scan_index_forward => false,
  :limit => 20)

tests.each do |test|
  render test.attributes.to_h

This will retrieve the last 20 tests run by a user.  It provides the desired results, but it was unexpectedly slow… really slow.  By default, the aws-sdk gem will lazy-load the attributes for each item.  So, calling `test.attributes.to_h` will trigger a new request to DynamoDB to retrieve each attribute.  In total, it will make 1 + 20 * num_attributes requests to retrieve our data!

To remedy this, we want to use the :select option to specify the attributes we want, such as:

tests = items.query(
  :hash_value => "user UUID",
  :scan_index_forward => false,
  :limit => 20,
  :select => ['timestamp', 'url', 'response_time'])

tests.each do |test|
  render test.attributes['timestamp'], 

Now, this query will retrieve all the attributes we want in a single request.  With this change, we saw the total load time reduce from as much as 7 seconds to a few hundred milliseconds.

We’ve been running happily with thousands of test results in our DynamoDB for some time now.  We’ll be sure to share any additional lessons to be learned as our datastore grows into the millions, so stay tuned…

Happy Blitzing!