Thursday, July 07, 2022

Don't Forget SaaS Performance Testing!

Your Strategic Sweet Spot

I wrote about application performance a few years ago in my article “It's Never Too Early to Think About Performance”. Since then I have learned and witnessed a few things that warrant a new article on this topic.

As more and more companies leverage SaaS for their business applications the number of companies that suffer from serious performance problems is growing by the day,

SaaS solutions like for example Salesforce, Workday and S/4HANA can be implemented in a way that performance is decent. They can also be implemented in a way that your users will wait for a very long time after each click or keystroke they make. The latter will lead to much frustration and high opportunity costs. 

How is that? 

Well, if the solution is deemed too slow, people will find ways to not use the solution unless they absolutely have to. This means adoption will be terrible, and your original business case will be not worth the paper it was printed on. As I have written before; any system is only as good as how well it is used. 

Still many projects that implement such SaaS solutions ignore performance and do not include client side performance testing into their scope of work. And this is a very big mistake.

Companies assume that the SaaS vendor is responsible for delivering a fast enough user experience. But this is not the case. They can’t. Vendors can only be responsible for the things they have under control.

Your devices, your security stack, your browser, your internet connection, your network, your location. They all have a significant impact on the performance of your SaaS solution. None of them is under the control of your SaaS vendors.

Also your configurations, your customisations, your number of concurrent users, and your amount of data have a significant impact on the performance of the solution.

Trust me, running a revenue recognition run in SAP S/4HANA with 100 or 100.000 WBS elements makes a big difference. As there is a big difference between running a report on 50 or 50.000 opportunities in Salesforce.

And these business applications cannot live in isolation. They need to be talking with each other. They need cloud integrations. See “The Biggest Challenge of Postmodern ERP - Cloud Integrations” for some additional thoughts on this topic.

Each of these cloud integrations need to be tested individually, but even more important are the end-to-end throughput times. How long does it take before your new employee in Workday shows up in your Oracle ERP? What if you have 1000 new employees?

Most of these cloud integrations are not part of a standard SaaS solution, and therefore not under control of your SaaS vendor.

So before you even think about going live with any SaaS solution you should do the following tests:

> Load Tests: you need to understand the behaviour of the system under a specific expected load. Testing with 250 invoices does not make sense when you know you will have 250.000 invoices in the solution within a year. When you know you have 5.000 concurrent users working on the system every day you should test this before you go live. Not after… 

> Stress Tests: you need to understand the upper limits of capacity within the system. And if this capacity is lower than the capacity you expect you need then you need to deal with this before you go live. Not after…

> Soak Tests: you need to determine if the system can sustain the continuous expected load. This can be as simple as running your load tests for a number of consecutive days. 

> Spike Tests: you need to determine if the system can sustain a suddenly increasing load generated by a large number of users and/or new data in the system. Businesses are mostly seasonal and have spike periods. For example your Year End Closing in SAP/Oracle or Black Friday for your sales. You will need to know if your solution can handle this before you have to do the work. Not after…

Fixing performance issues whilst you are working with the solutions is much harder as fixing them before you go live. See Going Live Too Early Can Be Worse as Going Late for some additional thoughts on going live checks.

In a nutshell: make performance testing mandatory for every SaaS implementation project and define decent performance as a killer criterium before going live.

Read more…