Hacker News Reader

Comment by Abishek_Muthian | Hacker News Reader

If it's helpful,

Our Node.js, MongoDB, Python servers all with significant network traffic didn't have any measurable impact after KPTI patches on Amazon Linux on T2.medium(burst), M4.large, T2.large(burst) respectively.

Our impact is lesser than the figures suggested by redhat's advisory - https://access.redhat.com/articles/3307751

View on HN

"Significant network traffic" tells me almost nothing, that's a totally subjective description and you don't describe your workload at all.

A back-end MMO game infrastructure server I have some understanding of in terms of the kind of load it is handling.

It tells us his workload is more network bound than computationally intensive.

Exactly, in my company's SaaS monitoring, I excluded db/external calls, and I saw a 1-2ms increase (on-top of 15ms), so yeah that's a 10% increase, but with external calls make the overall transaction 50+ms, so 1-2ms is trivial.

Similarly, the CPU on our Azure cloud box holding a half dozen dockerized Java apps I can't really see went up at all. Hovering around 10% CPU usage before and after patch.

More specifically, the meltdown patch primarily affect memory bound applications, not necessarily computations, although those usually overlap for CPU bound applications.

We're using KVM on OpenStack. We use a number of the services and PostgreSQL is our main database platform with our main backend platform being C++ based.

I have performance trending going back 1 year and honestly other than Postgres, the performance #s are like rounding errors. PostgreSQL is getting about a 7% hit on our databases that don't fit in memory.

We also run SAP and again we're not seeing anything. The SAP Database runs on POWER Servers, so obviously unaffected at least by Meltdown.

In our case, there was a major kernel version change, so we're just at the tail end of QC before I can release things to production to begin the rolling restarts tomorrow morning.

Depending on when you started your instances, you may have already been patched. I've read reports of AWS performance drops as early as Dec 15th.

Would you be able to post some ballpark figures, please?

Actually, there is not any measurable difference. Our architecture is completely state-less and when compared with equivalent load there's no difference at all. I guess, default network throughput bottle-neck itself is higher than the random memory cache bottle-neck in my case. There was no impact on the latency either.

Fearing automatic update by AWS resulting in performance issue, we rushed to update all our serves and gladly there weren't any performance impact.

I don't think not every one has been fortunate, there's still lot of variables reg performance impact of KPTI. This user here using PHP server mentioning 50% performance - https://twitter.com/timgostony/status/948682862844248065 and of-course OP issue is being covered here.

Thanks - this is what's worrying me: the result of patching seems to be all over the map, and it's somewhat hard to predict. My biggest worry is SQL Server, so our plan is to fix a performance issue we isolated recently before we patch - I'd like that system running optimally before we take the hit.

Judging from our MongoDB instance, I'm optimistic that your SQL server wouldn't face much impact either. Even Redhat tests put the risk at modest.

But for something like Redis, the case could be different.

If you are on a cloud provider, do check that automatic update is not scheduled; if so it's better to update manually to follow up with mitigation if necessary.

This is almost entirely subjective and workload dependent.

We know that the biggest performance factor in both of these patches (spectre, meltdown) is the kernel boundary.

So saying something along the lines of "Redis is more affected than Mongo" is.. odd.

MongoDB (I HOPE) will save files more frequently than redis, since redis is primarily in in-memory DB with some lazy IO access (depending on configuration).

So if you have a lot of small network calls to redis (which, is a kernel boundary event.) then it will be more impacted than a mongodb which takes only a few very large requests.

My gut is that the opposite would be true for the same workloads on both, MongoDB needs its files to return data, redis doesn't.

Even the vfs-cache is in kernel "space", so if your entire data sat in vfs cache it wouldn't help.

But that's my gut, and your exact usage may not mirror the 1:1 of this scenario.

> Actually, there is not any measurable difference.

That doesn't strike you as a little odd?

It's the same result as Google apparently observed across their fleet. It doesn't seem like it should be that odd.

I would agree that so far there doesn't appear to be any measurable difference for a typical web application. We have some heavy postgres data crunching jobs that might be affected but based on the examination I've done so far I'm inclined to say those weren't affected either.

It's not necessarily helpful, since "Python" doesn't indicate 'single-threaded' or 'multi-threaded', and whether or not your threading is constrained on outside resources (MongoDB) or purely internal worker calculations, and etc. But it's interesting!

Are you using PV or HVM instances?

All of them on HVM.

IIRC, XEN said that 32bit HVM VMs are not affected by Meltdown, and so probably don't get impacted by AWS' patches. They still require Linux kernel updates to protect the kernel space, so changes might still be seen there.

Yes, but XEN said 64bit PV aren't affected either because they already run in KPTI like environment. So I assume 64bit HVM like ours aren't impacted for our work load.

They are affected but there is no fix yet.

64-bit PV is unaffected and won't suffer a performance penalty (more precisely, it was already suffering it!!!), hence my original question, but a 64-bit PV guest can use Meltdown to attack the hypervisor. The fix is to update Xen, though I am not sure if fixes are already publicly available.

Reply on news.ycombinator.com