Comment by IST-Throwaway | Hacker News Reader

Comment by IST-Throwaway | Hacker News Reader

Fascinating to see an article about IS&T, definitely not something I ever expected to hear about on HN!

I was part of a team inside Apple that developed internal-facing tools for the retail teams. Our team was formed because of the extremely high cost in dollars and time that IS&T wanted to charge to develop some fairly straightforward tools. My team was taken from the departments that used the tools, and we developed things that were very custom-fit to what those departments needed.

Eventually politics overcame us, and IS&T finally managed to take over our team. We all became contractors in order to support the migration to their infrastructure, and continue development of the tools. A bit later we all became fired as our project was outsourced to contractors in India, managed by IS&T PMs who had no idea what our tool was used for and had never been inside a retail store. The tools all died shortly after that, some killed by IS&T, some petered out due to lack of use now that they no longer worked correctly or were a good fit for the users.

As far as we could tell, IS&T was run as a unique company inside Apple, which did it's fair share of price gouging in order to make itself money to keep going. Its purpose never appeared to be helping Apple customers or Apple employees, it was simply to get bigger, absorbing more money and power wherever possible, with no apparent reigning in from the parent org.

Although my experience is several years old, everything in this article rings true. The contracting companies they had us working for were taking a huge cut, the quality of the code they produced was dismal, (as soon as we were no longer allowed to re-write their code major things began breaking almost immediately) and people getting transferred around constantly and having no time to understand any one project was common. (rkho's comment about their hiring process seeming like it was simply a beard for a nepotistic contractor conversion was something we definitely saw a number of times.)

All in all it was an extremely eye-opening experience. Considering how "do it the Apple way" every other department we interacted with was, being in the IS&T buildings was like landing on an alien planet.

When will companies learn that cheap development labor always ends up being more expensive than starting with quality? This same story has been playing out over and over for decades.

Our remote team (in SEA) adds a tonne of value to our local Australian based team; but they mostly do maintenance work and are heavily integrated into our local team (they’re quite literally just treated as remote devs, rather than their own company). It’s worked really well for about 6-7 years now

I didn't say Asian labor, I said cheap/hasty labor. As the article makes clear, you can get that in the States too.

You didn't say "cheap/hasty" you said "cheap", and I can assure you teams in Australia outsourcing dev work to SEA are doing it because it's cheap(er).

A lot more people live in SEA than Australia, so it's probably both easier and cheaper to find people. SEA also speaks English and I'm sure there's good programmers around there somewhere.

Another good reason to have remote support in Asia is it's a different time zone. Not for Australia obviously, but that's a harmless thing for a US company to do.

> SEA also speaks English

South East Asia’s a big place, and outside of some of the Philippines, Malaysia, and Singapore, English is bad.

Would love to read more details how the integration and management works within these organisation.

Around 12-13 years ago, I was working (as a team of one on a highly technical upgrade, 32-64 bit upgrade, new compilers, OS changes, etc. Basically everything was changing). Naturally, I was falling behind debugging obscure issues largely due to 32->64 bit (this was C++ cross platform between windows and Linux).

My director offered to toss some offshore resources my way to help. Bluntly told him: I think I have 3 months work ahead of me. Give me offshore resources, it'll more likely be a 9-12 months, because by the time ive described the problem in sufficient detail, I could have just done the work. Instead, id have to review a broken "solution" that doesnt work, probably makes things worse.

Offshore is fine when you need grunt work, pounding out CRUD interfaces or already having a detailed spec. But, offshore doesnt work when you're flying by the seat of your pants.

Most of the big ossues I jad durong that release werent even my own. A few I remember: an x86/x64 inline ASM bug, use of an uninitialized variable in a DB lib that led to random failures dealing with NULLs, an overflow in a 3rd party datetime lib and ABI mismatch because of wrong datatypes used in an OSS ODBC wrapper that worked in 32-bit, but failed horribly on 64-bit.

None of the offshore talent I worked with at the time could have identified, let alone fix those issues. Several of those took reading the actual generated assembler to see what is going on. I dont write much x86/64, but I can read enough to know when shits going to hell.

The 2 big agencies I delt with were Salient and Accenture. I could work with Salient, most of their asserts were Indian and spoke English. Our Accenture resources were all Chinese. It was a pain in the ass because none of our engineering resources spoke or read/wrote English. Everything had to go both ways through a translator. Something was always lost in the translation. We ended up eating the contract and backing out with Accenture.

> Offshore is fine when you need grunt work, pounding out CRUD interfaces or already having a detailed spec.

My partner works at a startup with a very simple CRUD project. They outsourced one of the early sections of that UI. Even in such a simple case, integrating that component has been a huge time-sink and maintaining it is basically out of the question. They're working on a plan to rebuild it, as so often happens with these things.

Fire-and-forget software simply doesn't work.

Any monkey can write code. Writing the right code critically depends on communication.

It's been 20 years since I worked a corporate job, but a good friend and brilliant colleague / friend told me about his own horror story with Accenture / Chinese consultants. He told me the English skills and attitude of the Indian consultants he used to work with far outweighed the pain of dealing with Accenture consultants, in spite of their tech skills. Language problems and ego / combativeness from the stories he told me. Hearsay, I know... but this guy's work is in products that 90% of the people reading this use, so... yeah.

“My director offered to toss some offshore resources my way to help.”

Brooks’s Law in action. Mythical Man-Month should be required reading for all developers and managers.

I’ve seen that done just as a gesture of good will, in a "if it helps, we can spend more money"-kind of way, even though everyone involved knows it would just complicate things.

The first company I worked at had a great team in all departments until we got a new head of technology. Immediately all new projects went to outsource teams, some occasionally good and others bad, so much so that we (existing devs) began to “run out” of work. Eventually they tried to replace all the devs but me, the junior, and use me as the US based engineer for customer calls and maintaining legacy products. I said no and left on good terms with everyone but the new head who didn’t understand why I wouldn’t want to stay.

Within 6 months the new head was gone, the legacy code hadn’t been up kept and no new products had made full releases (of a planned 4-5). I did some maintenance and rewriting/launching one of the products I had originally started on as the only dev until they could hire someone else, but it did cost them 3x what they were previously paying me.

I would like to ask _why_ does this nightmare exist, but is that the right question? What is the right question to ask here.

I wonder if it is because it lacks critical importance and sexiness.

So it lives out of the limelight.

Also expectations of internal tools are low.

Think about the sexiest jobs in tech. They are working with the newest technology, or developing the funnest resulting product.

If you have someone working on the latest machine learning technology, management has a vested interest in keeping those folks happy.

If you have someone working on computer games, the employee has a vested interest in working hard because they're doing something really cool.

But internal tools development? Expectations from the rest of the company are low, it is viewed as a necessary evil by management but not a revenue generator, and to many employees working there, it is a stepping stone.

Which is really a shame. Good internal tools can really foster a good culture. Shitty tools really take people's minds off the ball.

The worst is when some shitty off-the-shelf tool is site-licensed and shoehorned into an organization and people just suffer. I vaguely remember working somewhere where some sales tool was used as the bug-tracker or trouble-ticket system. gah.

Yep, that's what I mean. Wouldn't it be better for all and everything to have a higher-functioning tools team?

This is the ad-absurdum[0] conclusion of the "Cost Center". Cost Centers don't make money, they cost money. Therefore any money you spend on them is lost.

Along with this you have the Iron Law of Bureaucracy - organizations tend (tend!, it's not absolute) to further themselves ahead of their stated goals or directives.

The right question everyone should be asking is how does the parent organization actually make money, and how to help them do that?

0. https://en.wikipedia.org/wiki/Reductio_ad_absurdum

1. https://en.wikipedia.org/wiki/Jerry_Pournelle#Pournelle's_ir...

Reply on news.ycombinator.com