Hacker News Reader

Comment by destroy-2A | Hacker News Reader

Try working in banking for 20 years, stuck behind at least 1 layer of citrix living in citrix inception. Latency for every keystroke, your brain starts to add latency to latency that is not there to compensate for a life lived wearing citrix latency goggles.

View on HN

I was a "Citrix consultant" for about two decades.

I'd walk into customer sites for the first time, meet people, and within minutes they would start ranting about how bad Citrix is.

I suspect only dentists get this kind of feedback from customers before a procedure.

Having said that, 99% of the time the problem boils down to this:

The guy (and it is a guy) signing the cheques either doesn't use Citrix OR uses it from the head office with the 10 Gbps link.

The poor schmuck in the backwater rural branch office on a 512 Kbps link shared by two dozen staff gets no say in anything, especially not the WAN link capacity.

I've seen large distributed orgs that were 100% Citrix "ugprade" from 2 Mbps WAN links to 4 Mbps to "alleviate network congestion" in an era where 100 Mbps fibre-to-the-home is standard. With 2 Mbps you can watch PDF documents slooooowly draw across the screen, top-to-bottom, line by line. Reminds me of the 2400 baud days in the early 90s downloading the first digital porn, eagerly watching the pixels filling the screen.

Don't blame Citrix. Blame the bastard in the head office that doesn't give a f%@$ about anyone not him.

I agree in general but I do blame Citrix for some foot-guns. The Citrix admins at my employer have never figured out how to configure it to get keyboard latency below ~120ms (on a gigabit LAN), and the silly health meter always reports the connection as excellent. This is mostly on them - in classic enterprise IT thinking, if it’s not down your job is done - but I’m somewhat disappointed that it’s even possible to configure it to have latency twice that of a modem.

A 120ms should feel immediate. IIRC anything under 300ms feels instant.

This is just flat out wrong. Any seasoned gamer can feel the difference between a few tens of milliseconds.

300ms would render most video games unplayable.

I see this claim a lot and it's making me want to build a website that gives you some common interactions (moving a mouse cursor, pressing a button) with adjustable latency so people can see just how big of an impact seemingly small amounts of lag have on how responsive something feels.

https://aresluna.org/keyboard-secrets/typing-delay/

After using xterm for years, I don't like gnome-terminal anymore because its lag while typing has become noticeable. It's right around 30ms on this site, and xterm around 10-20ms.

This is great, thanks. I'll have to remember it next time someone makes that bizarre claim.

Just running my display at 60Hz vs 30Hz is enough. The pointer feels extremely laggy at 30Hz, despite that being a higher refresh rate than a movie.

Movies always get brought up in framerate discussions but they are a completely different beast compared to interactive computer applications because

a) movies are not interactive so latency is not a concern, only fluidity is

b) movies come with pre-applied motion blurring to hide the low framerate (which is different from fake motion blur applied in some games)

c) 30 FPS is atrocious even for movies and I wish higher framerate movies had gotten more common

30 vs 45 fps on my steam deck feels night and day different, it's amazing how much small jumps like that can help.

Then have an estimation challenge mode, where it picks a random latency and you have to guess within 50ms what it is. Seriously though, that sounds both fun and useful.

If you had 300ms latency, back when I played League of Legends "your ISP is having problems today and you cannot play". Anything above 70 is considered very bad

Sounds excellent. I would send that link around to a lotta people.

That was a bad post. The figure of 300ms was from memory. I guess it's complex but for games shmup games (https://www.pubnub.com/blog/how-fast-is-realtime-human-perce...):

...for Massive Multiplayer Online Gaming (MMOG), real-time is a requirement.

As online gaming matures, players flock to games with more immersive and lifelike experiences. To satisfy this demand, developers now need to produce games with very realistic environments that have very strict data stream latency requirements:

    300ms < game is unplayable
    150ms  < game play degraded 
    100ms < player performance affected
    50ms   > target performance
    13ms    > lower detectable limit

But this is real-time gaming. Typing should be less demanding, I'd think.

Edit: also https://stackoverflow.com/questions/536300/what-is-the-short...

> Typing should be less demanding, I'd think.

Not really, unless you're the kind of guy working in Cobol and who is used to typing with latency.

I've seen Cobol developers just ignoring the latency, keeping typing because they know what they've typed and it doesn't matter that it's slow to show up on screen.

Working with latency like that also requires the system to be predictable. If you're expecting auto complete but not confident in what it'll show, you've got to wait, if you're not sure if the input will be dropped if you type ahead too much, you've got to wait. If you need to click on things, especially if the targets change, lots of waiting.

If the system works well, yeah, you can type all the stuff, then wait for it to show up and confirm. 'BBS mode' as someone mentioned.

> I've seen Cobol developers just ignoring the latency, keeping typing because they know what they've typed and it doesn't matter that it's slow to show up on screen.

I used to do that (not in COBOL), typing into a text editor in a terminal over a 2400-baud modem. Like the other commenter said, you get used to it, but it requires a certain predictability in your environment that you don't get in modern GUIs.

Generally I think of it in terms of number of frames @ 60 fps.

Anything below one frame (16.66ms) and whether or not any sort of real feedback is even received (let alone interpreted by the brain) becomes a probability density function. With each additional frame after that providing more and more kinesthetic friction until you become completely divorced from the feedback around 15-20 frames.

Just a heads up for others trying to read this, I think the < and > are backwards.

That’s off by about an order of magnitude – highly skilled humans can see and react in less than 120ms. One thing which can complicate discussion on this is that there are different closely related things: how quickly you can see, understand, and react is slower than just seeing which is slower than seeing a change in an ongoing trend (that’s why you notice stutter more than isolated motion), and there are differences based on the type of change (we see motion, contrast, orientation, and color at different latencies due to how signals are processed starting in the cortex and progressing through V1, V2, V3, V4, etc.) how focused you are on the action (e.g. watching to see a bird move is different than seeing the effect of something you’re directly controlling). Audio is generally lower latency than visual, too.

All of this means that the old figures are not useful as a rule of thumb unless your task is exactly what they studied. This paper notes how unhelpful that is with ranges from 2-100ms! They found thresholds around 25ms for some tasks but as low as 6ms for some tasks.

https://www.tactuallabs.com/papers/howMuchFasterIsFastEnough...

Keyboard latency is one of the harder ends of this spectrum: the users are focused, expecting a strong (high contrast, new signal) change in direct response to their action, and everything is highly trained to the point of being reflex.

When I’m typing text, I’m not waiting for the change to hit a key outside of games but rather expecting things like text to appear as expected or a cursor to move. Awhile back I tested this and the latency difference between VSC’s ~15ms key-to-character was noticeably smoother compared to 80+ms (Atom, Sublime) and the Citrix system I tested at 120-150ms (Notepad is like 15ms normally) was enough slower that it forced a different way of thinking about it (for me, that was “like a BBS” because I grew up in the 80s).

n.b. I’m not an expert in this but worked in a neuroscience lab for years supporting researchers who studied the visual system (including this specific issue) so I’m very confident that the overall message is “it’s complicated” even if I’m misremembering some of the details.

Not my experience. 300ms is noticeable and very annoying. 120ms does not feel instant to me.

300ms is a "long press" on a key on Android, and an eternity on an actual keyboard.

The parent comment may be talking only about the network or Citrix components in the critical path. You also have to wait to get keyboard input (often 10s to many 10s of ms) and for double-buffering or composition (you might get updates and render during frame T, flip buffers to reach the OS compositor for frame T+1, have the compositor take another frame to render that and send it to the screen for frame T+2, though this is a bad case for a compositor, you may be paying the double buffering or flu latency twice). And it can take a while for modern LCD screens to process the inputs (changes towards the bottom of the screen take about a frame longer to display) and to physically switch the pixels.

120ms end-to-end without Citrix would be quite achievable with many modern systems (older systems (and programs written for them) were often not powerful enough to do some of the things that add latency to modern systems). So if Citrix 120ms we already get up to your ‘not immediate’ number.

But I think you’re also wrong in that eg typing latency can be noticeable even if you don’t observe a pause between pressing a key and the character appearing. If I use google docs[1] for example, I feel like I am having to move my fingers through honey to type - the whole experience just feels sluggish.

[1] this is on a desktop. On the iPad app I had multiple-second key press-to-display latency when adding a suggestion in the middle of a medium-sized doc.

Divide those figures by 10 might be closer to being accurate. 120ms is quite noticable. I know as I need to adjust latency out of Bluetooth headphones for recording. Recording with those latencies sounds like a disaster and is very very much noticable even with sounds let alone vision

While my post was wrong, in fairness the context was specifically about keyboards. Nothing to do with audio. I suppose I should have been explicit but the context was keyboard entry.

In my experience visual and feeling type things like typing have even stricter tolerances for timings is what I meant to say. If audio has a delay, visually noticing a delay will at least be as equal if not more noticable at a specific ms

We aren't talking about website loading speeds. This is about how quickly your mouse cursor moves in response to mouse movements and that latency needs to be 16ms or less.

Personally I can get latency down to 200ms over the internet into a remote datacenter with WebRTC. The challenge however in practice is that running a CPU without a GPU will eventually starve the CPU because it has to do intensive things like run a 1080p video at 60fps which aren't feasible on a CPU only machine. This CPU load will then slow down the video encoder and overall responsiveness (no, responsiveness doesn't mean a mobile layout here) of the remote desktop.

That contradicts fairly long-held understanding:

https://www.nngroup.com/articles/response-times-3-important-...

Anything above 50 ms is absolutely noticeable and should be considered a bug.

Under 100ms feels immediate. More doesnt.

I recently had a bit of a rant about security people and how 70% of the truly dumb decisions in our industry can be attributed to them.

Your description is exactly why. Security people wedge themselves into the halls of power and then start making decisions that don't actually negatively affect them all that much.

I've literally seen a CISO that insisted everyone worked in a way they themselves did not.

sadly, the job of a CISO typically isn't "make the most pragmatic decisions possible to keep our infrastructure secure and running smoothly". In many industries, it's more lke "join as many compliance programs as possible to expand the ability to capture revenue from regulated markets".

The CISO didn't make the decision to enforce password rotation- the compliance programs your sales team asked for did

To your point, password rotation is considered an insecure practice because it causes people to append 1, 2, 3, etc to the same password.

But I've seen so many companies that still insist on it.

I'm the IT guy for a new non-profit. We aren't separated yet from the company that created us, but we're in the process of separating. I get to decide all this fun stuff.

I had a very brief talk with the IT team for the larger parent company when I started and explained this stupid password rotation thing, as I came from a security background, they wanted nothing of it. Set in their ways.

For the new non-profit that I'm helping spearhead, I'm not sure I'll get away from the password rotation entirely, but I can certainly set it to something more reasonable, like every 365 days, rather than every 60 days or whatever travesty most are dealing with. I'm pretty pleased about this.

NIST agrees, as if their update a few years ago.

> Verifiers SHOULD NOT require memorized secrets to be changed arbitrarily (e.g., periodically). However, verifiers SHALL force a change if there is evidence of compromise of the authenticator.

https://pages.nist.gov/800-63-3/sp800-63b.html#memsecret

This is a really useful thing to keep in mind because even if you aren't directly bound by a requirement to follow the NIST standards, being able to point your policy people at that is handy if you can shift the conversation to “bring our policy in line with NIST” where there's a question about whether they'll later look bad for _not_ having done so. Typically these conversations are driven by risk aversion and things like federal standards help balance that perspective.

Thanks for the direct link, putting this in my back pocket when the discussion inevitably takes place.

Aside from password rotation being a very questionable practice, it actually can cause productivity loss. In a big organisation like mine it can take up to 48 hours for a password change to synchronise across all the internal services. There's also the issue where some endpoint software still uses the old password behind the scenes and fails to log in too many times - causing your account to be locked. I guess you can see my frustration coming through.

I had the joy of dealing with some endpoint software like this in an organization that had mandated password changes every 30 days. Very predictably, people set recurring "change your password" reminders for the 1st of the month and the organization lost an entire day of productivity each month as they locked themselves out of their accounts en masse. So the beginning of the month was always a panicked, all-hands-on-deck day for the help desk as people were waiting on hold for hours to get their account unlocked.

Our penetration testers suggested we add password rotation, and I had to quote them the latest NIST guidelines which state "Verifiers SHOULD NOT require memorized secrets to be changed arbitrarily (e.g., periodically)."

If they don't know better, it's not surprising other companies don't either.

> To your point, password rotation is considered an insecure practice because it causes people to append 1, 2, 3, etc to the same password.

A good solution to discourage this would be to have heuristics that'd make sure that the new password isn't too similar to the old one, but doing that without having plaintext in there somewhere is pretty difficult.

Another solution would be mandating that all of the passwords should be randomly generated, but enforcing that would be difficult, because everyone who isn't used to having 99% of their new account information being in KeePass databases with randomly generated passwords, probably would find that too cumbersome to remain productive.

This seems like a people problem that makes being secure essentially impossible, due to how people use passwords (e.g. "I just use one password across X sites because remembering multiple ones is too difficult" or "I just add a number at the end of my current password").

And others also mentioned the productivity loss, for when people are slowed down by the need to change their passwords. You might easily rotate Let's Encrypt certificates thanks to automation but when it comes to people, things aren't so easy.

At that point, you might just stick with whatever passwords you have, do some dictionary checks in the future, maybe have infrequent password rotation and otherwise stack on more mechanisms, like TOTP through whatever application the user has available, or another means of 2FA, because relying just on passwords isn't feasible.

> causes people to append 1, 2, 3, etc to the same password

It’s either that or they write them down. Because people are going to forget a password that changes every month, especially a password that has to comply with the complexity rules.

> The CISO didn't make the decision to enforce password rotation- the compliance programs your sales team asked for did

And it's the CISO job to resist unnecessary overcompliance which is just for the happiness of the sales team.

You don’t make the company lose business just because compliance is unnecessary. You’ll (rightly) get overruled every time.

Isn't that just a characteristic of how they're evaluated? Any security error is the CISO's fault, "heads must roll", etc

Given that, they're likely to give you what you are asking from them: a brick with no functionality which will do nothing. You can't do anything with Brick, but Brick has zero outstanding CVEs

CISO often stands for Chief Sacrificial Officer...

It seems to me that the reason why so many bad enterprise solutions are bought is because the buyer is not the user. It’s such a funny thing to me that people would spend tons of money without firsthand experience or at least someone they trust using it.

I've never used Citrix but I remember when I had a T-1 (1.54Mbits for the younglings) and I left a Remote Desktop session open on a laptop. Some days later I went back to the laptop and used it for an hour before I realized I was in a RDP session to a machine in another state. I wonder what Citrix screwed up to make their UX so different. Of course a decent T-1 back then probably had better latency than today's consumer HFC connection.

Yeah the T1 easily had enough bandwidth to smoothly send the 800x600 16 bit color desktop you were probably running at the time (guessing the timeframe based on usage of a T1). Frame to frame diff was probably much easier as well with less shadows and graphical effects that modern Windows or Linux DEs have.

I don’t doubt Citrix has gotten worse as well but the job it had to do back then was much easier.

> Don't blame Citrix. Blame the bastard in the head office that doesn't give a f%@$ about anyone not him.

> The guy (and it is a guy) signing the cheques either doesn't use Citrix OR uses it from the head office with the 10 Gbps link.

If you were sure about this you could have as the consultant told this sentence or made this entire comment as your 'first page' of powerpoint/PDF (to make sure other hn-ers are happy!)

How long ago was this? 4 megabits would be pretty good... back in 1998!

This _very_ much depended on where you are. I had symmetric 10Mbps at home in 1998 but when we moved to New Haven in 2008 Verizon couldn't deliver more than ISDN / T1 to large chunks the city (we literally could have used a WiFi antenna to hit their regional headquarters, too). There's so much deferred maintenance around the world.

True, true! I had a 3 megabit cable modem at home, back in 1998 (3 megabits down, 128kbits up, if I recall.)

My office at the time had dual T1's... a little over 3 megabits shared with roughly 500 people.

The last time I saw a place migrate the remote offices from a less than 10Mb/s network was around 2015. That same place replaced its mainframe at 2011 because of an enormous price hike.

2017

I quit a job because of citrix. Exactly like you said, very noticeable latency. It ate into my productivity as a part of my mental energy was going into waiting for feedback to my actions to appear on screen.

> part of my mental energy was going into waiting for feedback to my actions to appear on screen

This should not be underestimated. I was in a situation like this and I though my short term memory stopped working. I forget what steps I already did because some actions took 10-15 seconds. I often switched to another task in the meantime and could not recollect the last step I did 10 seconds ago. Such delays are poison for for intellectual task where you would need concentration for.

There is no excuse for any modern device to make such pauses. It is also far too expensive for any company. The price for hardware is too low to let any user wait.

That's exactly it. Instead of tasks going "1, 2, 3" in my head, it was more like "1,...,1,...,1". I had to keep reloading every task into my working memory, with lots of brief pauses to think "did that click register", or "when I typed those words, was the context on that text box?". It's a truly torturous level of friction.

I didn’t deal with citrix but I did have to frequently SSH into cruise ships at a job some years ago. Goodness was the latency frustrating beyond belief. I didn’t last more than 6 months at that job.

Every single command input/key stroke could take 2-5+ seconds to display on my screen. Imagine trying to troubleshoot something critical in that type of environment. Luckily, I didn’t encounter anything truly critical, it was mostly maintenance tasks and such.

> part of my mental energy was going into waiting for feedback to my actions to appear on screen.

Sounds like my life as a developer too. =D

[ Disclaimer - I am responsible for a Citrix environment, but I'm reasonably proud of how well it works for our company ]

The technology behind remote desktops is fundamentally limited but I'm amazed at how good the user experience can be on a modern well-configured Citrix environment.

- The protocol responds well even on low bandwidth as long as latency is OK. On the office LAN it feels like a local computer.

- There is offloading for Teams[1], media streams[2] and even entire web browsers[3]. The tech behind this is impressive and it works pretty well (mostly!)

- For most staff it's easier to use a thin client or a minimal laptop.

- I can keep the Citrix environment patched and managed much more easily than a proliferation of laptops and home devices.

It can be a struggle at times and it's definitely not the right fit for developers. But it's got a lot of advantages and most of the time it works amazingly well.

[1] https://docs.citrix.com/en-us/citrix-virtual-apps-desktops/m... [2] https://docs.citrix.com/en-us/citrix-virtual-apps-desktops/m... [3] https://docs.citrix.com/en-us/citrix-virtual-apps-desktops/m...

I am at a law firm that uses a remote system like that. Have definitely gone two Citrix’s deep for some things, so I feel this.

Honestly, though, it’s better than the laptop the other firms have given me. One took over 10 minutes to boot, iirc. It wasn’t just the hardware, there was just so much … stuff, multiple layers of antivirus seemingly hooking all of the system calls and fighting with each other, and a document management system with blocking I/O everywhere that was somehow so embedded in Windows that it could seem to freeze the whole system.

The thin client setup may have latency, but at least it is convenient and it gets there eventually. Though I would swear it’s getting slower, or maybe my patience is waning.

O.o Rarely see people like us here! :-)

For me what worked is a setup where I used an Arch linux laptop, ran f5vpn in docker and used the citrix client with some tweaks through that vpn connection.

It was a lot faster than my colleagues' Mac / Win client, and even better, it was automatable to start up and run everything.

Ha! I did document this beautiful setup: https://github.com/kmARC/f5vpn-in-docker

My employer blocks Linux clients for whatever reason. Even if you pass through the initial checks there is some kind of system on the Remote Desktop that detects your local setup and kicks you out.

So I use a KVM Windows machine with Virtio drivers. QXL seems to be the best video solution.

Does everyone here just suffer from exceptionally shitty IT departments? I've used Citrix for years and not experienced any of the chronic issues described here. Remember Citrix was developed in the 1990s... the days of Windows NT 3.5/4.0 [1] & Dial-Up connections and to be able to function well in these low bandwidth environments (we're talking kilobits here people, a 10 Mbps LAN was considered glorious at the time). For years ICA was superior to RDP due to its better compression over such connections. It sounds more like whoever setup your environments didn't know what he was doing and the results are what you would expect.

[1] https://www.youtube.com/watch?v=SNJiWPU4HEU

Citrix performance can depend a lot on the apps - older win32 apps work really well as the object caching masked the latency on windows and buttons. Newer apps seem to somehow make the caching not very effective.

Traumatically well written lol

Ergh! You're giving me PTSD. I still recall those days shudder Best of luck with that

You simply get used to it. In many industry sectors (think CPU architects), multiple layers of inception is the norm (crossing multiple operating systems), and it is not strange for a keystroke to take 2 seconds, and for a menu to open and finish rendering in 10 seconds. This "experience" is probably the reason why I can still comfortably work over a DSL link with just network X (even though I still find NX much more comfortable).

You really just adapt your way of interacting, and start planning more carefully every one of your actions instead of simply clickety-clacketing everywhere like if you were trying to win a game of Starcraft. It's practically subconscious and it really changes you.

I always think it must be much, much worse for blind people.

It also reminds me of people who complain that 5-minute build times "impair their productivity". How do you even work on _any_ mid-sized commercial codebase then ? It's not that uncommon for a build to take hours (e.g. games), and in engineering it is also not that uncommon for builds to take _days_ even on powerful server farms.

You're a CPU architect and you wait 2 seconds for a keystroke? And you stay in that job? You must be one of the dumbest geniuses I have ever met.

That's absolutely ludicrous that anyone would be expected to work that way.

There is a long queue of geniuses waiting for this genius to quit.

This is me right now, but only for a short time (I hope). I'm at an agency and currently on my first ever banking client. I'm on a Mac but I use Citrix Viewer to access a Windows 10 machine. The part I dislike the most is the context switching between Mac and Windows. First off, windows doesn't natively let you customize the keys (I can't install anything obviously, it's a bank client). Also, for some reason, the alt key doesn't work in the Citrix Viewer so I have to change a lot of my usual VSCode shortcuts to sone custom ones. I've googled the issue and some people on Mac use a program called Karabiner[1] but I didn't want to install yet another program, I'm just dealing with it for now.

Our agency has another banking client that I hear sends you a laptop, I much rather have that.

[1] https://karabiner-elements.pqrs.org/

Hah now imagine using Teams through Citrix workspace.

One thing I've learned about Citrix is that its a startup company with limited resources to handle all the bugs and crusty corpocrapware layers. The client craps on my HDR setup. It install a ton of crap you don't need and it relies on crap like HDX software running on your machine that last time I've checked it didn't had ARM binaries but this tech is also unavailable for the iOS clients. Meanwhile RDP can do semi-decent multimedia stuff without any of this crap.

> One thing I've learned about Citrix is that its a startup company

Founded 1989, 2021 revenue over $3B

  </sarcasm>

Amusingly Microsoft purchased RDP from Citrix (like 30 years ago). See version 4:

https://en.m.wikipedia.org/wiki/Remote_Desktop_Protocol

Maybe I'm immune to it, or just lucky, but two hops (Logging it at home to a Citrix Network Desktop to Remote Desktop to the PC in my office) has been shockingly fine. I live very close to the servers in question though, so speed of light isn't a limiting factor, and I have solid and reasonably fast home internet. It can work fine.

Don't worry, I'm sure that will all be folded into microsoft teams soon :)

More seriously, I'm reminded of something a friend always said.

You need to have a response time of 1/10 of a second or less for something to be interactive. I remember that but I wonder if the brain fixes it like it ignores your blind spot.

Worked with Citrix for years at different customers. Think it is more about setup. Server capacity, bandwidth and so on. Often Citrix was used for connecting to a bastion or a jump host. Then an extra hop to the target machine. Some setups were laggy. Some worked just fine.

I work in banking. Everything company-hosted I access via a split tunnel VPN. Everything else goes through the normal internet connection, with a company root CA inserted to sniff HTTPS traffic.

One of the lucky few I guess :-)

This has been my experience with Citrix also, although I have heard it can be set up to work better. Has anyone had experience with HP Anywhere/Teradaci? I am curious how it would compare.

What kind of banking? Tell me people aren't doing excel modeling through a laggy pipe

Printing worked great though, right?

Or healthcare.....

reminds me of the days of dial up modem.

Reply on news.ycombinator.com