Comment by junto | Hacker News Reader

Comment by junto | Hacker News Reader

Parent Full thread

True story about IBM Global Services. I was working with a client that was integrating web services with another third party. IBM GS were responsible for managing third party's network.

Our client gave IBM the IP addresses to be white listed. They filled in a lengthy time consuming form that got sent through layers of IBM management till some ops team in India finally implemented the change one week later.

We started testing. It didn't work. We went back and forth for weeks. Finally we put a conference call together with all parties. Live on the phone we sent a series of SOAP requests over the wire. "Yes, we can see the requests", came the answer. We saw no response.

Finally whilst on the phone call, one of the IBM Tech's piped up.

"Did you want this traffic to go in AND out?", he asked.

"Well, HTTP traffic isn't going to be of much use if we don't get a response to our request you idiots", was the answer I wanted to give.

They pulled out the initial firewall request form. The request was to "allow traffic from our IP addresses in through the firewall to reach the client".

They took that absolutely literally, and nobody thought to question it. They also absolved themselves of any blame.

I detested IBM from that day on.

Could also be a compliance thing where they were told "If it's not on the form, don't implement it". And it's up to the requestor to specify everything they want opened. Having the network team second-guess what was requested could leave unwanted security holes when they have to make assumptions about a request.

Though I don't understand the problem you experienced -- HTTP runs over a bidirectional TCP connection, if they got your request, then packets for that session were already flowing in both directions (the TCP session wouldn't set up otherwise). The only way I could see what you described would be if they had a proxy or some sort of deep inspection firewall in the middle that was blocking the return response.

> Could also be a compliance thing where they were told "If it's not on the form, don't implement it". And it's up to the requestor to specify everything they want opened.

Sure thing, do exactly what you're told.

That doesn't preclude you from firing back a message to say "Hey, this request is unusual and almost certainly wrong, are you sure this is what you wanted?"

When a nurse sees an order for 10000 mg of a drug that's never normally administered above 200mg, it's not their job to correct anything, but it is their job to ask the doctor "Hey, did you mean to write 100.00mg? Is this actually correct?"

I get the compliance thing but surely in cases where they can see that a request is very likely to be wrong I don't think it would be unreasonable to expect them to send off an email or give you a call to double check if it is really what you wanted.

This is not surprising given IBM developed the Consulting Industry Complex standard for blaming the client for failure: The Rational Unified Process.

That's not just IBM's outsourced vendors. The desire to do the minimal amount of work to adhere to the exact wording of contracts seems to be prevalent. For us, it takes us 3 to 4 weeks to have a single VM created. And even then its usually done wrong. When asking them to do the partitions different, the response is that they're unable to change the script. A month to simply have a script run.

How does .. anything happen ..

This goes a long way to explaining that Gartner report from last year which say Marketing departments IT budgets will exceed IT departments IT budgets by 2017.

We're seeing this a lot - doing projects directly with marketing department, largely so they can completely circumvent internal IT department's and their apparent stonewalling on seemingly simple requests.

(Of course it also means we're seeing things like completely outlandish requests that indicate fundamental security/privacy misunderstandings ("What do you mean you can't import our entire customer database into the WordPress site you're building and hosting cheaply for us?"), as well as last minute compliance intrusions ("Oh, you need this to live on a big-corp.com subdomain? That requires a security audit and risk assessment and mitigation document! How on earth have you got this far without knowing that?")

It happens very slowly. I have a similar story but with simple VLAN changes taking weeks and endless forms. Sorry to hear people losing their jobs but yeesh what a horrible company IBM is. This is a company that pivoted to a services company as well.

It doesn't.

They bill by the hour.

I was their Corporate Webmaster in the 1990s (effectively CTO, but not quite). The networking support teams had this in place back then. If you did not say you wanted bi–directional traffic, then you would not get bi–directional traffic.

They were incredibly dependent on their filter rules, and their process assumed the filter rules were always correct and accurate.

One night I got confused about which window I was in and ssh'd into my desktop machine (on the internal 9/8 network) from one of the ibm.com backend servers (on the open internet). A "routine update" caused routes for 9.0.0.0/8 to be advertised on the open internet. Although I had a short escalation path it still took an hour from when I discovered the hole to get anyone in Global Services to accept it as a problem.

> "Did you want this traffic to go in AND out?", he asked.

Question sounds reasonable - could be a fire & forget scenario.

Problem isn't the question but that it takes a week between iterations. If you had to order lunch and it was a week between asking for your steak and being asked if you want it medium or well done then the question isn't the problem

UDP perhaps for fire+forget, but TCP is a bidirectional protocol.

I had that happen with Infosys. They do exactly what you tell them to do, good, bad or stupid. You never get feedback that something doesn't make sense. So in the end you look stupid to your managers because you didn't write a good spec.

Now I avoid dealing with outsourcers as much as possible unless I have 100% understanding of the issue and can describe it precisely.

Had the same experience with a large communication equipment manufacturer (who you'd think would understand firewall rules better!). This is pretty normal to be honest.

I doubt there are many companies who are 100% on top of things, but I get the sense that's more down to the department or the individuals you're dealing with. There are people who don't think things through everywhere. I certainly wouldn't bear a grudge against IBM for too long!

Haha, that's pretty similar to my experience with Indian tech workers in general. Very literal interpretation of English phrases like that.

Maybe it's a language barrier?

Agreed, this sounds like an Indian outsource tech center problem that is universal rather than an ibm specific problem. Cultural differences rather than language?

My experience is with Infosys. They definitely stream information from architect->tech lead->coder one-way. There is no channel back if something doesn't make sense.

So basically like any other big enterprise.

I was about to say just this. It is why I will not work in large enterprises again if I can help it. Something that should take 5 minutes can end up taking weeks and still not be done correctly.

I'm not sure why you're being down voted if other big enterprises didn't have this issue they would not put up with IBM having this issue. It is all about minimal work, CYA, and minimal thinking. It is why I always tell people that anyone who can really do IT (probably most on this site), will always have a job.

It not only IBM, there are quite a few other ones that I could write endless books about such war stories.

Hell, as someone client-facing who works at IBM, I've had experiences with a ton of other vendors who do the same thing. I'm having a hard time getting one firewall change put in through a client's outsourced datacenter from one IP address to one other IP address, one port. Ticket keeps getting closed as complete, until we say it's not and they reopen the ticket. Rinse and repeat. It's been three weeks of this.

If you're outsourcing your operations, that's what you have to expect. They're not going to be as invested in your success as your own employees would be.

That is terrible. Counting the meetings off-shoring actually cost them money.

It usually does.

Well, it costs their customers money.

Hey I am sorry if you are impacted by downsizing, but there are better ways of handling this than disparaging the tech community in India. I have worked in both US and India for more than one company and can tell you there is no discernible difference in the quality of the work that gets done between US and India. And even if I take your point that IBM are fools who doesn't know what they are doing, how to you explain Amazon, Microsoft and Google hiring thousands of developers in India. Microsoft's largest developer shop outside US is in India. Your experience can be due to a variety of different things from person at the other end doing exactly what he is supposed to do just to not poke holes in a secure infra to assuming that the firewall access list is state-full and allows return (HTTP response) traffic to pass by default. And there is no guarantee that a person based in US won't do the same thing.

The difference is in motivation. What motivated me to go into tech is that I liked it. What motivated many of my Indian colleagues over the years to go into tech was, by their own admission, societal pressure. Which one do you think produces better outcomes? This is the case regardless of the occupation, lawyer, plumber, programmer or teacher - if you're passionate, chances are you're better at what you do than someone who isn't.

RE 'no discernible difference in quality': there's a fair amount of literature, both anecdotal and factual, that says otherwise.

http://www.wsj.com/articles/SB100014240527487035155045761420...

http://www.theatlantic.com/international/archive/2013/10/beh...

http://www.rozinskiy.com/why-india-outsourcing-is-doomed/

Those are from the top of the fold of the Google search "quality of Indian tech workers."

There are bad coders everywhere and it takes years of deliberate practice to be good at anything, including programming. Those people who did not put in the work, don't make it through grueling interviews at Microsoft, Amazon, Google and other such companies. People who do make it however are just as good as programmers anywhere else in the world.

Well I had similar experience IBM people. We had one small change request for our IBM websphere MQ setup. It was raised when I joined the project and they could not complete it by the time I left after 1 year. Multiple times they canceled our request ticket. And this whole setup was managed by IBM in their data centers.

That which is not required is forbidden. That which is forbidden is not required.

That is why they are cheap.

Maybe cheap in what you get but they are one of the most expensive places around.

Reply on news.ycombinator.com