Hacker News Reader

Comment by MattGaiser | Hacker News Reader

I’ve had four jobs in my life and have always been stunned by what is still done by hand. Every company had a task which could be heavily automated but was instead consuming 4-30 people.

So if you think the software market is large, consider all the stuff software never reached.

View on HN

We have an "internal systems" team at my current company to solve this problem. The product manager for the team knows everyone and keeps tabs on what's being done that could be automated. It's not used to replace people (at least it hasn't been yet) rather it's used to allow the team to stay the same size and scale up their operations to meeting the growing business demands.

I think many companies could benefit from a team like this.

One of my four jobs was on such a team. Unfortunately, they canned it (after I left) as they couldn't figure out what it did.

If my team of 8 devs had just one dedicated to improving infrastructure and processes, we could probably increase our overall output by 30% in a year.

Ha, I'm happy this exists. I created this department at the company I founded at and directed it for some time. We call it the internal tools department. It still exists today and is run by a former employee of the department. It's very valuable for us.

This is great. How do they deal with the on-going maintenance of the tools they build? Do they move to the main product team or does the internal systems team continue to maintain them?

I wrote a piece of code last year in probably no more than a few hours.Showed it to a few guys in my team.Instead of sharing my excitement,they were like "if you run this,what are we going to do then?". She was right,if I'd do it, the size of the team would be reduced by half.But then,what I'd get? Do I get a bonus the size of those employee's salary for 6 month? No.I'd get a grand or two and instead and a pat on a shoulder,while those guys won't find a job for momths.So what's the point?

The point is getting that thousand bucks before someone else does. Or to make the company efficient enough to survive in a market where the other company's already efficient.

helping your company save costs (at scale) is almost always stupid compared to selling that knowledge to other companies.

Within the company, there is often no point. You want to be the guy selling it outside the company. You can easily claim tens of thousands for the same work then.

Management has to be ok with reducing headcount. You would think that all managers would be incentivized to made their groups more efficient but there is a high correlation between headcount and political power within an organization and people don't like to give that up for free.

You're in a delicate situation, since you'd be to blame for your co-workers potentially loosing their jobs. Nevertheless, doing nothing is also risky, since someone else from another department could show up and automate away your jobs leaving you in a hard place, and you won't have any control over that.

If you propose the automation yourself, and plan it's adoption within your team, you may have a chance of taking advantage of it, and even try to bring in more responsibilities within the company to your team to secure your teammates jobs.

Isn't really anything your team could do? Usually there are lot of fronts people could work on and bring more revenue to the company. Try to find those to redirect your team and implement your solution.

I've seen this even at technology companies. It's like our devs hours are valuable so we can't invest them in something everyone is comfortable doing with a spreadsheet already.

Internal access to developers is absurdly structured a lot of the time.

One company a friend worked for billed internal developers at twice what IBM would charge for projects.

At other companies, dev time is allocated without any kind of cost analysis whatsoever. It reminded our Romanian business analyst of the Soviet Union.

This happens to me a lot, but it's basically a kind of technical debt. Most of the time the upfront time investment to automate something properly versus doing it manually is hard to justify. Especially if it's a task that only I understand and now I need to document the problem and the automation instead of just the problem.

Business logic doesn't make for elegant APIs that I can hide behind a script and throw on a server, even if it's repetitive or tedious.

There's also the risk of allocating a team of folks with spare cycles, having the get half way through the work, then getting pulled into billable work and leaving it unfinsihed.

We are always looking for ideas!

Can you give a few examples, please?

Not OP but I had a friend who has interned at a few record labels. He's not really a programmer, but knows enough about programming to know what kind of task should be automated. He described a portion of his job to me that he and a bunch of other entry-level workers spent a few hours a day doing (basically scraping and storing data off the internet to be sent to business people for analysis).

I worked with him to develop a Python script do basically automate half his job in a few hours that could have saved the company tens of thousands of dollars. The company didn't want to dedicate/hire an actual programmer to maintain it, so they kept paying employees to manually scrape the web.

The crazy part is the next summer he interned at a different company and literally the exact same thing happened. He re-made the original script, saved tons of work, and the new company still didn't end up using it after he left.

Does he get paid at the internships?

Yep, but not as well as tech internships. Still, the company was throwing money down the drain because they didn't know that even a semi-competent programmer could automate those jobs.

Oh, many companies have only themselves to blame for being blown away by technology.

Are these companies irrational? Perhaps, but there might also be an important point.

A manual process is robust, fault tolerant, and easy to adjust to new circumstances or changing business requirements.

A Python script is fixed, and they day it breaks down due to some slight change in scraped data, you have to pull in a developer to fix it. For a company without dedicated staff to support such automation, it is a high risk.

Ad-hoc automation can turn workflows into Rube Goldberg machines. A Python script here, an Excel macro there - soon nobody knows whats going on and nobody dares change anything.

Developers often think only one step ahead: It was manual, now it is automated with a script. Win! A business have to consider that larger picture - future support, maintainability, risk etc.

Exactly - intern labor is cheap, flexible, and there's a "wax-on wax-off" mentality where they may learn by proxy but are also in a safe environment where they won't hurt themselves or the company. A script and the associated automation could become the expectation or could lead to a key-man-dependency-risk.

It may be lucrative for the friend to take the script and turn it into a product. but then they'll need to market, sell, document, optimize, and do a million other tasks.

There has always been a dead zone between "automated script that makes me work a bit more efficiently" and "product that can make everyone with this job work a bit more efficiently". Fortunately, new technology is narrowing this gap.

Bulk downloading and resizing photos in weird spreadsheets.

Managing attributes and taxonomies for ecommerce.

Capitalizable hour analysis for software projects.

That's just stuff from the last few weeks. None of these are hard to automate but the workflow integration is the tricky bit.

Do you know why they don't just use an existing API that does image resizing automatically? Presume it's the integration challenge, but I'm curious if you know /why/.

Also, what does "Capitalize hour analysis" mean?

I doubt they know such an API even exists (or what an API is!). Most companies aren't run by hackers, assuming there's an API that does the thing they want to do isn't the natural response.

Sure

A tool where you are given a picture/scan of a table and it spits out a table that can be pasted into Excel. This was at a bank where numerous people spent half their days manually typing in scanned documents into the same table in Excel. You would double a $70,000 a year financial analyst's productivity with this.

A portal to manage employee and employer documents. I.e where both you and your employer can easily access your T$ (Canadian tax form), offer letter, certifications, police checks, etc.

Software to manage which options a client ordered and what their selection was of that option as well as autogenerate the appropriate SQL queries from a client config template. The current process consisted of a mega word document with people fiddling around manually with it.

Happy to provide more.

That first one can be done with teseract ocr pretty cheaply. I think a lot of people just don't know how much software can do. You just need to standardize the form and then there's a human data validation step in there.

The challenge we faced (for the two days we tried to hack together a solution) wasn’t so much the OCR but the fact that we couldn’t standardize the form as we were onboarding people with the documents from other banks.

Our requirement was that the the software would need to read the number of separating lines in the table and output accordingly.

Maybe it can do this, idk. I was an intern at the time.

We were using tesseract for our attempt too.

There are commercial OCR software packages that can do it. FineReader is one example (and it's been around for ages).

FineReader seems like it would have done the job.

I worked somewhere that did something similar. Their product would scan and extract data from documents. Even working with the banks and companies who produced the documents, there were very stringent requirements (like identifying marks on pages to tell what document type we are looking at, pages being fed in the correct order, etc). Plus the ability to handle degraded, damaged, or bad copies. It's apparently one of those things that seems easy at a glance but the edge cases end up employing a whole department (they had people to review the scans and press OK or correct the data). It was a legacy system and I think a newer rewrite could get further but I'm wary about saying it could be fully automated.

90% automated is sometimes 90% as good as fully automated. It's not always the case, there are cases where hitting that 100% enables you to do things that were just not possible at all before, but sometimes it is. Recognizing those cases where a 90% solution is a 90% improvement at a 10% cost is a valuable skill.

Yes it's also part of the reason I prefer coding for myself. I can ignore a lot of "what ifs" and edge cases because I'm always there to pick up the mess if something happens. I get to focus on the stuff that makes an impact and forget the idiot-proofing :)

Or said another way, it can be fun and also effective to "Do Things That Don't Scale" [http://paulgraham.com/ds.html].

Several years ago I came across a pile of interesting-to-me US government data that consisted of scanned PDFs of paper forms (some filled out by hand, some with a typewriter, more recently with a computer). The paper forms change in layout and level of detail a few times over several decades, there are hundreds of sub-schema variations within each time period (yes, that means that a particular variant might have a single record from 1983, a dozen scattered over time, or thousands across several decades), some variations are due to a simple category misspelling, others are more substantial.

Some of the scanned forms have marginal notes (probably okay to ignore these), but I've noticed some cases where the form had prefilled data crossed out with handwritten data replacing it.

Please let me know if a system exists that can deal with this idiosyncratic mess and coaxed to produce structured data of some sort without basically using MTurk behind the scenes. I would be interested in re-releasing the data into the public domain.

Can you not use software for the 80% that are amenable to it and humans for the other 20%?

We used tesseract ocr, plus other image processing libraries to improve ocr's chances, and although we were close to usable results there were still many challenges. One major problem was that different data sources had completely different page layouts, and often times the original documents were out of reach and the client only had available the scans of fax transmissions!

The bottom line was that we were able to achieve very good results consistently, but the margin of error was not acceptable for our specific use case.

Reply on news.ycombinator.com