Ask HN: Question from self-taugh developer, my app is slow. What do I do now? | Hacker News Reader

Ask HN: Question from self-taugh developer, my app is slow. What do I do now?

12 pointspototo66630 comments

I built a web app. To be precise, it is a web app inside Wechat ( I am in China).

Every day there is around 200 users, and it generates me some ramen profit.

Users report that my web app is sometimes slow.

I know that there are many plausible causes. But before I jump into the rabbit hole, I want to ask for your kind advice.

What is your mental framework to deal with such a problem?

FYI, my app:

- is built with NodeJS

- ran on a cloud server with 2 core CPU, 2G Ram, and 1M bandwidth

- is a CRUD app

I have learned about building apps from the web, mostly from Udemy and Youtube. Someone has told me to not worry about performance. `Make it work, then make it better`.

Now that I have learned how to make it work, how can I learn how to make it better? Do you have any links or materials to share?

Thanks a lot.

View on HN

Your next steps are to figure out 1. what's slow, 2. why it's slow, and 3. how to fix it.

1. You can find that from basic logs - just logging the duration of each request should be enough - either at the app or the web server level. Figure out which endpoint takes more than expected time and in what situation (with specific parameters, specific user, etc.)

2. You should be able to use some tracing library or APM to figure out what's slow. It could be a call to the database, processing some data, or something else. If it seems obvious what the suspect is, you can also take the timing with a trivial `start=current_time(); the_suspect_thing(); log(current_time()-start);` For APM, datadog has a free trial: https://www.datadoghq.com/product/apm/ but there are other options too.

3. Figure out how you can improve that. For database, often indexes help. For data processing, you can try to do less of it. (filter before processing, cache results, etc.) Whatever is slow, you may need to research more in a specific area. No resource will cover everything, but asking on Stackoverflow is not a bad place to start.

Thanks four your clear steps. It is very practical :)

> Users report that my web app is sometimes slow.

One wise move is to validate that your app really is slow. Do you experience the same slowdowns? If not, it could be network latency, which is completely dependent on where your ends users are working.

I work with many rural end users, and we have noticed that specific areas of the country tend to be where we get complaints of slow apps. We've worked with ISPs in that area to identify and correct old hardware on their network, which improved performance of our app.

If you can reproduce the slow behavior, by all means, follow everyone else's advice. But just like you would reproduce a bug before fixing it, reproduce the slowness.

IF you do discover that it's purely user-side network issues - that doesn't mean there's nothing to be done. You could work on caching certain request and making your app more functional offline (if possible).

Yes, I have experienced the slowness

Thanks for your wise input, this makes sense: reproduce a bug before fixing it.

For a comprehensive review of performance engineering, I’d suggest Brendan Gregg’s book, Systems Performance: Enterprise and the Cloud, now in its second edition. Chapter 2, “Methodologies,” describes several methods—mental frameworks, if you like—for analyzing and resolving performance problems.

Thanks, I tried some passages from Chapter 2 of the book you mention. It is awesome, just exactly what I need right now.

I could not live without a log file.

You should create a log file (text file is fine), where you write as much as possible about the program flow: user logged in, user navigates ..., request started, query submitted with parameters, query responded with 1100 records, result processed, result formatted, result delivered, finished request.

The log should be appropriately detailed.

Correlate the log entries with the user ID, session ID, request ID, thread ID, etc, whatever appropriate.

Timestamp the log entries with millisecond times (2021-12-27 14:21:33.512)

Watch and observe the request flows over period of days.

Perhaps you will see where the problems are.

1) Too many records fetched from DB.

2) Too slow DB fetch (missing indexes)

3) Cold DB caches

4) Some exponential loops

Instead of text file, you can use a structured log, but a simple text file will quickly give you results.

Thank you sir.

I only log program exception. Your advice really makes wonder why I have not done what you suggested at first.

You will need to identify the slow responses and break down performance of each part to then identify what part of the stack is slow.

You don't have a lot of resources (RAM/CPU) in use so it is possible that's the constraint but you have a real learning opportunity to understand better how your tech stack works and learn some performance tuning. Your cloud provider should have some stats for you to see if those things are hitting limits, some simple charts to glance at would at least give you an idea.

You mention its a CRUD app so first thing I would look at and spend time on is your SQL queries and analyzing them. The majority of the time in web requests for CRUD apps is in the DB - because it could be loading and reading a file from disk (where the DB stores data) which is typically slower than receiving bytes on a network socket. Learn what `explain plan` means for your database flavor. I am going to venture a guess that you may not have database indexes on some tables. An explain plan for your queries should tell you what indexes, if any, are used.

For your front-end you could look into caching services for any JS framework libs, images, etc. assuming there is a way to do that on your network in China. That would reduce the bandwidth usage and if those services work properly, your end users would be downloading those assets from a faster server than you provide.

Thank you, Matt.

I have never analyze SQL queries. This seems to be the time to learn how to do it :)

Node does not prioritize performance.

vCPUs don't prioritize or guarantee performance.

2GB physical ram may not be a lot.

1Mb bandwidth (~100KBytes/s) may not be a lot if multiple simultaneous users with bulk payloads.

Storage may not be performant.

Personally as a first step I would just migrate it to a better, dedicated server. Consider the cost of your time to research it. In the West a midrange older dedicated server starts at ~US$30/mo. All kinds of potential badness about being in a vm on a shared resource are eliminated.

If not, or if that doesn't buy you a solution, you must characterize which of the vserver itself, cpu, storage, or bandwidth is the bottleneck.

Assuming it's linux, I would ssh in and set up prometheus or similar monitoring so you can see it over a week.

If no problem, hit it with concurrent transactions until there is a problem. Compare what you have to do to see a problem on a setup on your laptop vs the remote server, you can judge your vserver then.

For cpu, look with top how it acts when you give it transactions. Who eats cpu?

Memory, top + vmstat.

For storage, iostat.

Network, netstat -s

> Personally as a first step I would just migrate it to a better, dedicated server. Consider the cost of your time to research it.

I strongly believe that's a terrible idea as the first step. It may be a solution once you know the reason for the slowdown, but if it's something trivial like a lack of database index, or an accidentally quadradic algorithm, you'll just run into the same issue on a bigger host in a few days with more traffic.

I understand that POV, but if you're on the cheapest possible vcpus, gaining a difficult-to-acquire deep understanding of how overcommitted they are and how sporadically unusable it is as a platform is not worth anything.

Sure it's the wrong move to blow $2K/mo on some Xeon, but it's not the wrong move to spend $30 on bracketing the extent of the platform's problems.

Thanks sir :)

1) put a be cdn in front of it

2) set your caching in assets and images to preserve bandwidth

3) test if your db is the bottleneck.

Eg. Some applications have their performance log in a response header ( db: 20 ms, logic : 10 Ms, total: 50 ms)

Definitely start profiling and try to up you bandwidth depending on your current online users.

Thanks NicoJuicy,

I just finish setting up CDN a few minutes ago. And my bandwith is 2M now.

No problem.

Have you found the bottleneck and has the experience improved?

I don't know the specifics of your app. But minimizing assets could also reduce bandwidth, it's mostly done by default. So that's what's why i didn't mention it before.

I haven't started profiling yet.

According to my cloud server's monitor, there is a big chance that the slowness is due to bandwith too crowded. I want to see if there is more slowness report or if I can experience slowness again, before I spend time to learn profiling.

But I guess sooner or later I'll have to do that. Now that I know what is in my armory, I feel more at peace :)

Monitoring is part of the toolset ;)

Perhaps it's a good time to mention that I've implemented health checks for a lot of applications.

It returns healthy when everything is okay ( eg. Select 1; on the db) and unhealthy when something is off ( eg. Hard drive 80% occupied, CPU at 80% or memory at 85% ) and returns failure when....

If could be just a library in nodejs and adding an endpoint.

I agree with other comments suggestions:

- Use the app, see what is slow from a subjective UX perspective

- Profile your code, find hotspots

- Use system monitoring to determine if there are any hardware/network bottlenecks

One thing I would add on these, perhaps a bit more specific: You mention you are using NodeJS, and it is a CRUD app. What is your database? Are you using an ORM? Sometimes the choice of especially an ORM can cause some performance issues behind the nice opaque interface. I have no specific reason to believe this is causing the performance issues for you, but I've had issues myself in the past so thought it might be a helpful suggestion to look into (for example, N+1 query issues).

Thanks Creatornator.

My database is mongodb and I use Mongoose as ORM. My crud operation is quite simple, therefore I doubt ORM causes the performance. But like you said, I need profiling.

Apart from what others have mentioned.

If you are using NodeJS, it is optimized for I/O and not computation. Since it's single threaded and async at it's core, any blocking computation will impact entire service.

One approach could be to profile your app using a myriad of node js profiling libraries. (https://nodejs.org/en/docs/guides/simple-profiling/)

If you want a solution, try deploying your JS code as AWS Lambda or GCP Cloud Functions. They may address the issue at lower cost while you fix the issue.

Thanks.

I have never profiled my app before. Time to learn new skills :)

An APM tool could help you find the real cause of the slowness instead of guessing.

I used datadog recently to find the exact line of code that would bring CPU to 100%. Since it's quite pricey, I only used it for the couple of weeks I needed it.

> An APM tool

Thanks :)

Learn how to use a profiler, so you know what is actually slow. Find your inefficient hot loops, your N+1 queries, etc.

I don't know what the standard tool is for node, but there must be one.

> Learn how to use a profiler

Thanks :)

First find out why it is slow. If you are not aware of the reason and also the performance got worse, go for any APM solutions that suits your requirement. Atatus has been doing a great job in monitoring with real time insights lately.

Break down each piece of a request -> response trip and add logging or metrics logging to your DB. Use that to find out the weak points and optimize those.

Reply on news.ycombinator.com