I wonder why they chose Java for substring search. Why not C (strstr) or grep?
Edit to add: "good enough is not good enough" -- very well put! Yes, it would be good to move this loop to native code. However, with our current design, the overhead of calling out from Java to native code would probably outweigh any benefits. We're planning to move our core database storage to off-Java-heap memory buffers, at which point it will become much more feasible to call out to native code.
The inner loop speedup on this particular code will be more like 2x than 3x, and the speedup of the overall system will be quite a bit less than that. But your point is still correct.
Anyways, congrats on your success. I would love to read more about the business side - finding clients, profits, expenses, etc.
A faster implementation of a string searching algorithm could only save a few milliwatts of CPU power, it wouldn't make it faster nor require less hardware.
It's possible they are bound by the SSD speed.
I can't find much on AWS SSD max speeds.
You're going to have to choose one of those or the other...
Once there's a backlog of requests, faster requests on proportionally fewer machines will not help the queuing latency.
But in general it's not a good plan to be in that situation for long, because the back pressure that stops the queued requests expanding infinitely is people leaving your service in frustration.
Lets just make up toy numbers. Suppose you have a code that can process (say) 1000 lines per second, per machine. You have 3 machines and you need to process 90k lines. Each one gets assigned 30k lines and it takes 30 seconds overall right?
Suppose you find a 3x speedup in the code. Now your 90k lines takes 10 seconds. Alternatively, you can ditch who machines, and process the whole 90k on one machine in 30 seconds, the same as the original time.
To me it seems like this is what tacotime was saying, you can either have faster responses or fewer machines.
The metric that is improving is the latency experienced by a user.
Imagine that you have a web app that takes one second of server CPU time to render a page, and you have three servers which process three hits a second in total. All three servers are thus on 100% CPU load, dealing with one hit a second each.
Each time somebody visits your site, they are going to experience a 1 second latency (in addition to communication latency), as they wait for one of your servers to build the page.
If you then optimize your code so that it completes in a third the time, 333ms, then your servers are going to suddenly be at 1/3 load; they execute their one query for 333ms, and then sleep for 667ms.
Not just that; but the user now only has to wait for 333ms for the page to render on the server, so the site gets a lot more snappy for them.
Then you can shut down two servers leaving one; it will sit on 100% load, but you still keep the shorter 333ms latency experienced by your users.
You are doing the same amount of work as before - 3 hits a second. But previously, when the tasks took 1s each, three would be running in parallel, each being completed more slowly. With the faster run time, they are running in series, being completed quickly before switching to the next one.
Now, this does not apply if your requests are queuing. Because you're not able to do any more work than before (after shutting down the other two servers), if your hits/s exceeds the capacity of your servers to deal with them, the backlog will grow just as fast as it normally would have, and the latency caused by this will sky rocket.
Your example doesn't fit because multiple machines aren't ever used to process front end web requests in parallel; you don't render half of a template on one machine, and the other half on another, for example. If you can set up a system like this and see gains from it, then what I've written above will not apply.