Am I weird because I actually muttered 'ARP caching issue' halfway through your article? :-)
Love the technical write-up -- thanks!
Love the technical write-up -- thanks!
Instead we try to maintain a SOA which is agnostic to what underlying tools back our services, something that we've found Docker is useful for.
That being said, we generally just use Ubuntu, and have some other AWS-specific setups around networking. If you like hacking on Cisco routers, I think it's perhaps not the right position :)
More modern and highly specific technologies like HAproxy, Salt, Node, Redis are not going to be known by every experienced, competent applicant though, and while you can often pick them up on the job if you have general experience in the area, there's always going to be a learning curve.
Seems they are more dev then ops for devops role.
Just today: Servers with a static IP address getting it changed to an APIPA just because some other 3rd party device sent a unicast ARP request to it that should've been a broadcast as per RFC4436. I personally find ARP to be one of the funniest protocols out there and I love the faces people make when they understand how ARP glues L2 and L3 and suddenly everything makes sense.
It is funny to see articles like this, because the network seems to be the "last frontier" for IT companies/workers. Only a handful seem to be brave enough to work on it (and actually like it). It is supposed to work like the lights when we switch them on and any interaction with a network engineer is just to tell them about an issue :)
In my opinion, the network guys are the geeks among the geeks. Don't get me wrong, it is easy to understand networking, it just takes dedication.
Nice write-up. Too bad I'm not in SF. :P
As part of the cloud-init scripts on new machines we would install arping and have the newly provisioned VM automatically send gratuitous arps so that arp caches would get updated.
I haven't seen the issue on my RHEL 7 VM's though, nor in my current environment, but we don't spin up/down as many VM's as we used to.
1. something silly like: "arp -a | awk '{print $2}' | xargs -n 1 arp -d". It's been a while and I don't think I was the original culprit.
# arp -s 192.168.0.1 00:18:4d:f8:a4:6e
Wait your working over SSH? Oh ummmm, talk to the data center guy and have him break out his USB to serial dongle and a RS232 cable.
The intermittent connectivity to server A is what makes this problem so fun to diagnose. :-)
He ended up disabling arp caching though, rather than clearing it.
Yeah, that seems like a really bad idea. That will drastically increase the latency and cut throughput.