gavanm

I work in performance testing, and in the web space you essentially have two different main approaches:

* Protocol Testing - where you use servers to generate lots of HTTP/S traffic that is correctly structured to simulate user traffic. Generally, this is focused on capacity and server response times.

* Browser Based - where you need the complex logic present in SPAs and Javascript to accurately create test traffic. This also allows for as-close-to-possible real user response times. This is essentially "headerless browsers" of varying types. This requires more performance test compute to process - so often a combination of both types are used together.

I find that Testing application security is one of the most technically challenging aspects of performance testing. Often some parts of the security infrastructure have to be disabled to allow testing to occur. For example, Rate Limiting by source IP, any form of Captcha, 3rd party (OpenId etc) services have to be disabled - which increases the risk to application availability because sometimes there are components that haven't been tested exactly the way they will work for actual users.

Luckily most the 3rd party services we use are already significantly tested by their vendors - but it is something that I worry about.

There's a book that takes this general idea in a direction. It's definitely intended as a commentary on society and the future, but you might enjoy reading it.

https://marshallbrain.com/manna1

I work in the corporate performance test / performance tuning space in a brownfield environment with a lot of co-dependent systems supported by a lot of different teams. My background prior to this was a programmer / DBA.

It's all hard - but it's not any harder than building and testing end to end solution working in the equivalent environment - there's a lot of similarities, benefits and also disadvantages. Overall performance testing provides

"Performance Testing" often covers quite a few different areas:

- User Experience (more difficult with SPA and Mobile applications)

- Server Response

- Capacity Planning / Utilization

- Performance Tuning

- Enterprise / Systems Architecture

- Production Support for Performance Problems

- Monitoring (both for testing purposes and planning monitoring for production)

Like programming, most the time actual coding / scripting isn't the challenge. Some of the bigger challenges I deal with are:

- Test Data Setup, Availability, Size, Distribution, Consistency, Security, Backup/Restore over multiple systems

- Understanding / Confirming Scenarios

- Complex Application Authentication (more secure authentication methods can be harder to test or work-around)

- Data integrity / security (difficult to use SaaS solutions for regulatory reasons)

- Environments, Sizes, Accuracy (Prod Like), Support (Application and Infrastructure), Monitoring, Limitations, Architecture

- Load Planning/Estimation/Reporting (both in Performance and Production)

- Deployment Planning / Management

- Investigating Performance Problems

- Working with lots of different development teams.

A lot of the recent trends in DevOps / Agile / IaaS / PaaS are helping improve the situation with significant improvements to deployments and scaling.

Good commercial tools and OSS tools are available. The move to web / HTTP based systems has actually made testing much simpler because there's really only one protocol to target these days (HTTP) - older applications platforms using non-HTTP based protocols often make things more difficult. There are still some smaller niche areas for industries such as call centres where there are physical requirements (such as generating call traffic), and Desktop Virtulization / Remote Access (due to COVID) that don't fit the more common testing patterns. Mobile App performance testing (which is closer to User Experience testing) is also relatively immature compared to SPA and HTTP testing. There are options out there though.

I also view there being (at least) two different traditional paths into performance testing - Functional Testing (Non Technical) and Developers/Tech Admin (Technical) who bring different strengths and weaknesses to large performance testing efforts.

I'm a fairly big fan of trying to push most performance testing down into individual application teams as they know the technical details of the applications and often have better technical skills than a lot of performance testers. However, you still need cross-system / whole environment perf testing where you have complex interconnected systems and dependencies.

It's difficult trying to balance everything between purely scientific / methodological and pragmatic (just prove it's capable of meeting the volumes) approaches.