askldjd

I spent a little over 3 years in USDS and it changed my life. If you are interested in working on high impact projects that affects hundreds of millions of Americans, I would strongly recommend that you give it a try.

Our team is very careful at choosing technology that is stable and well proven. In fact, it took a lot of debate for us to move from Rails static pages to React SPA.

As for government being slow and only change once a decade, you might be underestimating engineers in the government.

Here's the disability claim processing project I worked on when I was on SSA Digital Service. https://federalnewsradio.com/ask-the-cio/2017/01/ssa-turns-c...

The team I worked with managed to pull off React/Node.js/Redis/Zookeeper microservice framework over AWS using Jenkins as CI/CD pipeline. And most of the team came from an older enterprise stack using Java and IBM WebSphere. They were able to adapt to the new framework.

Be conservative, yes. But being overly conservative is part of the reasons why government is so far behind today.

Nice! I was looking for it, and it predates git. Thank you!!

Yup, it would. Disabling the TS option was just a stopgap measure to make our deployments stable for the time being.

I am turning 34 coming this September. Not much of a kid anymore (even though I act like one).

Not true. GS15 caps at 161k at DC area. It is a very respectable salary. http://www.fedweek.com/pay-tables/2017-gs-pay-table-washingt...

For engineering, USDS predominately hires senior engineers with years of experience. The reason is because we help troubleshoot some of the biggest crises in the government.

I actually did look at the TCP layer early on. However, I didn't pay close attention to the TS Val. From the packet dumps, it just appeared that the TCP window had stopped sliding. I couldn't conclude that NSOC's router was at fault.

Getting NSOC on-board is a big deal. After all, they deal with the entire VA network with 100,000+ employees. If you think about it from their perspective, why is USDS' TCP connections so special?

You are dead on. We do have a bug where we are not recovering the Oracle connectivity correctly. It is on our radar to address the issue. https://github.com/department-of-veterans-affairs/caseflow-m...

However, There is actually another 50% of the story that I never posted. VACOLS is a really old Oracle DB (from the 80s) that is out of our control. Somehow, it has a "feature" where you can only make one TCP connection to it every 2-3 second. So if we lose connection to the database, it will take many seconds to recover. At that point, our ELB health-check would've fired and restarted our EC2 instances. This is why recoverability of the database connection is not an immediate priority.

Here's how we preallocate the VACOLS connection pool to workaround this throttling feature. https://github.com/department-of-veterans-affairs/caseflow/b...

The infrastructure we operate in are very challenging (and interesting) because of legacy systems. That's why common sense engineering often may not apply in USDS.

Thanks. I think the fact Cisco routers fail to route TCP packets bothers me even more.

/you had one job

The government's current IT infrastructure crisis is not caused by any one administrations. The root cause goes back decades. Things like "Improving Veterans' lives so they don't have to wait 5-10 years for an Appeals decision" shouldn't be political.

I can honestly say that the projects I've been involved with in USDS are the most impactful and meaningful projects I've worked on in my entire life.

Author here. Completely agreed. My jaws dropped when I saw the INITIAL_JIFFIES. The kernel developers really saved our butt.

I could not imagine debugging this problem if INITIAL_JIFFIES was randomized. It may takes days/weeks/months for this bug to appear.