A decade into a project to digitize U.S. immigration forms, just one is online
washingtonpost.com
washingtonpost.com
The root problem is not bureaucracy either. Companies like Amazon, Google, and Microsoft have fairly large bureaucracies yet they routinely successfully complete massive software projects.
The root problem is not "vested interests with complex requirements" either. When coders with large-scale-project experience were put in charge of revamping Obamacare's healthcare.gov, they figured out how to cope with a wide range of demands from a wide range of constituencies in an efficient manner, and got the job done: http://www.thewire.com/politics/2014/02/how-silicon-valley-s...
The root problem, I think, is that the people who are put in charge of these massive software projects -- typically, MBAs or lawyers by training -- often don't know anything about how to develop software at scale. It's like asking people who don't understand rockets to lead development of a new kind of rocket that has never been built before.
Edit: added more detail and context to the third paragraph.
The main difference between small and large companies was it simply took longer for people who got nothing done to be let go. In the worst cases for companies, money stops coming in the door and cuts are forced.
Contrast the above with the government where it sometimes literally takes an act of congress to fire someone. If the populace is unhappy with how the government is spending its tax dollars they cannot simply stop paying taxes and go somewhere else.
Another difference with government is the intractable complexity.
This plays to the difference between digitization and automation. Automation is the removal of human judgement. It requires simplification of the rules and processes.
My local government revamped their payroll system. The first effort tried to digitize existing rules, and failed utterly. The second effort reworked the payroll rules (automation), allowing them to use software closer to off the shelf, and succeeded.
I consider the projects I worked on for higher ed to be complete failures. Publishing course catalogs, registering for classes, auditing degrees, etc. Conceptually simple stuff. Alas, management tried to implement every single psychotic rule and mutant edge case for each and every stake holder, preserving paper-based processes over 100 years old. Such fun.
My work for a telco wasn't any less nutty making. There's a reason your phone bill is too high: bureaucracy. Their internal IT is terrible. They can't even provide conf call services or roaming wifi to themselves. I sometimes fantasize that Apple or Google will introduce ubiquitous flat rate service, gut the dinosaurs, and making another gagillion dollars just thru operational simplicity.
My impression, is that government projects often go like this: 1) We want to digitize and make some process X more efficient, 2) Oh, now that we're automating we can avoid "errors" by removing case-by-case handling (X is of course now set up for individual evaluation by a professional bureaucrat). The result is that not only do you get feature/scope creep (handle all the things for all the cases, or declare failure), if you succeed, the system ends up being much worse (it now follows outdated written documentation to the letter, doesn't allow for any leeway).
The obvious way around this, is make a system that automates the easy 80% of the cases. Allow the system to give good information to, the now much fewer, bureaucrats, and allow them to handle the 20% on a case-by-case basis.
After a year, evaluate, and reduce the 20% to another 80/20.
Give up the idea that you should automate policy. It's completely insane, even if it might look good on paper.
It's like with accounting systems: you should always have a way for a clerk to enter in and account for unforeseen things in a way that a) accommodates the system ("-500 towels missing/destroyed") along with a comment-field to elaborate, and to facilitate revision ("Bus crashed through the window, gave away 500 towels for use as bandages").
This is probably the most important part - every digitization effort I've seen fail, failed because they tried to digitize the existing process, rather than look at what information needed to be collected and creating a new digital process to collect that information.
That is certainly true and one major facet to the problem.
> cannot simply stop paying taxes and go somewhere else
Plenty of people leave the US every year. It's generally really easy to do. At a local level choosing a city or state based on tax rate is even easer.
ED: Unless you meant the moving was the hard part, which seems reasonable.
http://travel.state.gov/content/travel/en/legal-consideratio...
Persons intending to renounce U.S. citizenship should be aware that, unless they already possess a foreign nationality, they may be rendered stateless and, thus, lack the protection of any government.
There are also a lot of exceptions to that IRS rule, depending on the type of income.
PS: As to renouncing Citizenship; Social Security is an example of long term obligation that transcends where your living in a single year. If your willing to give up Social Security by renouncing Citizenship then things get simple.
Getting a second passport takes significant time and investment in most cases.
People really complain about paying taxes while living outside the country. IMO it's a really complex situation, but the tradeoffs are surprisingly reasonable if you consider it in the long term from both the individual and state level in light of various free rider problems.
Large IT systems fail in many ways, small and large and I don't think it's possible to isolate a single root cause but I do have one observation which I think is often overlooked.
When a company like Amazon or Google release a new service or a new product, they generally have the luxury of choosing which customer requirements they want to meet. The more customers they satisfy, the more customers they can hope to win.
This freedom means that these companies can follow the common trajectory of releasing a simple realization of their product which is then honed by iteration. But companies contracted to produce government and corporate systems rarely have this freedom in my experience, but for different reasons.
In the case of government systems it is politically (and often morally) unacceptable to exclude specific classes of end users from a system; the requirement to make a system available to nearly 100% of possible users while at the same time catering for the labyrinthine complexities of local and national government is simply very difficult.
In corporate environments, there is a similar pressure to meet 100% of customer needs, but in this case the users are often staff and though the problem is similar at a technical level, the source of the pressure is different:
Google is a large corporation, and when it releases a product like Gmail, users can choose to use it, or not: they can't singly demand changes, so Google has the power to stand up to users and, in short, be a good custodian of the user experience.
When developing software for corporate customers, this relationship is inverted, and very often, every wrinkle in every corporate business process is expected to be modelled in the software. As a contractor, you know you could produce better software for the corporation, if they were willing to adapt some of their processes in the same way customer adapt their behaviour to using Gmail, but they have the power.
The result is the same in both cases: excess complexity.
The MBA (who has no other expertise) is capitalism's answer to the Soviet "apparatchik."
https://en.wikipedia.org/wiki/Apparatchik
"Management" is a secondary skill in the sense that it only makes any sense if one also has a primary skill. An engineer might learn to manage and thus become an engineering manager. A baker might learn to manage and might manage a baking company or a restaurant/cafe. But only knowing "management" alone is absurd and vapid. I truly cannot comprehend how anyone could purport to manage something they do not understand at the ground level.
To some extent I blame academia for pushing the idea that one can get an MBA and manage things without understanding what one is managing.
Programming is particularly bad. There are no physical analogies for it. It's so alien to ordinary human experience that asking someone with no experience in it to manage it is like asking a blind person to direct a silent film. Actually the latter is far more likely to be successful. I can't actually think of an analogy right now that suitably captures just how inept a non-developer is at managing developers.
From the link:
"Members of the "apparat" were frequently transferred between different areas of responsibility, usually with little or no actual training for their new areas of responsibility. Thus, the term apparatchik, or "agent of the apparatus" was usually the best possible description of the person's profession and occupation."
"It is often considered a derogatory term, with negative connotations in terms of the quality, competence, and attitude of a person thus described."
I also don't believe programming is as alien as you think it is.
I will grant you that most of the diploma mill MBA programs are worthless. But the MBA's going to work in F500 companies aren't just idiots.
Are you saying that there are some great MBA programs, that will produce an MBA that adds value without other experience, or are you saying that the MBAs going to work in F500 companies have experience beyond an MBA diploma?
If it's the latter, isn't that pretty much what parent claimed? If it's the former, is that because those MBA programs are cross-disciplinary with real project work loads?
I took a couple MBA courses at Kellogg when I was at Northwestern in another grad program. The ones I took weren't real project work loads, but the courses delved into case studies of what companies did wrong and right during critical points in their history. Students would use real world experiences from their past to explain things. I took an IP asset management class with one of guys who sued Zuckerberg for allegedly ripping off HarvardConnect to make Facebook. Unfortunately he never talked about it. I was very curious.
Another large value of an elite MBA is the networking. How many billions of VC bucks have been dolled out to MBA grads at Stanford by people they met there?
In fact, the whole idea of an MBA program is somewhat similar to what YCombinator does.
Where is this pushed? The advice I've seen is "go work in industry for a few years and get practical experience before getting an MBA." The ads on the subway for MBA programs that I've seen are usually in the terms you describe of going from a baker to manager of a bakery.
I'm an MBA. Are you suggesting that I have no other expertise? Do you think that we undergo some sort of specialised lobotomy at business school to remove the knowledge and experience we accumulated before enroling in business school, and to prevent us from learning anything thereafter?
I'll allow it this time. But I'm keeping my eye on you. ;-)
When healthcare.gov was turned around by a select set of motivated individuals, it only happened because the people who were involved had a high degree of motivation. IMHO I believe they were motivated by eliminating personal frustration (or having empathy for those who are frustrated) and the upside of a highly visible software project (with none of the downside...it was already failing). Money wasn't a focal point here.
But when individual private companies screw up, other companies do what they tried to do better, and in the end the problem gets solved.
> The root problem, I think, is that the people who are put in charge of these massive software projects -- typically, MBAs or lawyers by training -- often don't know anything about how to develop software at scale
Sounds like a... branch problem. What is the root cause of government projects being run by incompetents?
The incentives tend not to reward competence at achieving a task very well. They tend to reward political climbing. They don't necessarily preclude competence at a task, but energy expended competently performing a task is energy not expended playing politics, which means it tends to get outcompeted.
This is not, strictly speaking, a characteristic of "government". Large corporations that get sufficiently rich and powerful that they end up detached from market discipline develop the same problems.
Security, wealth, and comfort are even more dangerous to large organizations than they are to individuals.
Their interview process was: "can you write computer code?.. OK great, you are hired. Come on in." I should have known something may be amiss..
The dev server took about 5 minutes to start on a souped up computer. That and several static variables hiding up high in an uber base class. These variables were of course designed to hold per session data. I tried submitting a form twice simultaneously. This crashed the server.
I left that and years later the project was I believe cancelled.
I wonder why the government does't just use a forms as a service thing?
> The root problem is not bureaucracy either. Companies like Amazon, Google, and Microsoft have fairly large bureaucracies yet they routinely successfully complete massive software projects.
>...The root problem, I think, is that the people who are put in charge of these massive software projects -- typically, MBAs or lawyers by training -- often don't know anything about how to develop software at scale.
I don't think these explanations are mutually exclusive. If you're right about the root problem being the technical expertise of leaders, then one can still argue that the Amazons of the world slowly rise to the top and displace the K-marts, but an incompetent government can persist indefinitely because it has a monopoly on governance.
Of course, the first one would matter a lot more if: a) people voted based on previous performance, not vague future promises; b) there were more than two options in competition.
As for the second. The second works quite well, but not a time scale that yet cares about well-designed websites... fail to figure out say steel alloys and gunpowder, though, and you'll see this competition in action... ;)
I get it, that won't be a popular sentiment here because the hidden racism backed into immigration to fill holes created by incompetence and ignorance of long term problems by filling them with short term solutions is still quite popular, let alone not even realized. It's kind of like simply going to the title loan place to leverage the future to pay back the credit card of bad past choices.
Digitizing the immigration forms is really more of a sad reminder of how racist our country is that we refuse to provide for our own people, while we are more than willing to deprive source countries of success by making people build industries and technologies in their own country.
I know, I'm all wrong ... screw those mostly black poor kids in poor neighborhoods and screw those other countries, America needs to undermine their success in order to dominate.
EDIT: Xenophobe.
The humongous forest that pro immigrant people can't see for all the trees is that immigration is a core driver and reason for many of not only America's problems, but also the lack of source countries from gaining any traction. All the way from sapping out pressure for social change by motivated people leaving for places where they can mooch over the fight that was already won by others, to extracting the skill and knowledge to do the thing that the do from their society. What pro immigration types don't quite get is that immigration to the USA is like kryptonite or more like a witch that sucks the life force out of the source countries and societies.
You may and probably confuse what I said with xenophobia and hatred and are angry because I am not saying your favorite thing, but reality is that I am right, whether you or others realize it. I'm fine with that, but it needs to be said. The point is not about "go back to where you came from" it's "why didn't you put your energy, drive, knowledge, and skill towards your own country and society and instead went to make the rich, richer and the powerful more powerful" Don't complain about America's actions and policies around the world if you are going to support our life force driven economic, political, and military engine.
Here's something funny. In last few years, all the reputed lawyers (Murthy, Ron Gotcher and others) have setup an online portal where you, as a client, fill in everything online. This includes scanning various documents and uploading them. The online portal provides a great workflow which minimizes data entry and avoids data errors. The same portal allows you to chat with your lawyer/paralegal if you have any questions. At the end of this process, the lawyer firm will generate correctly filled in immigration documents which they will print and send it to you. You sign them and send it back so that they can forward it to USCIS! I know it's not practical or possible, but when I was working through this portal, I wished USCIS would just buy this system so that I don't have to pay thousands of dollars to laywers.
- The forms themselves are poorly designed (needlessly complex, come with misleading/incomplete instructions, tons of "gotchas," etc). Between me and my wife we have three degrees and a masters, are native English speakers, yet we still had to Google to figure things out.
- Instead of having one form, they have three, and you cannot submit the latter two until the proceeding one is accepted (even if they're completely independant). They don't actually check the proceeding forms until the visa is complete (we know that because they flagged something months later).
- Paying fees is a nightmare.
- Even their embassies know how terrible it all is. They often hold workshops for people going through the process and are pretty frank about the bureaucracy. They describe the NVC as a "blackhole" (where things often get lost/delayed/mistakenly rejected).
Digitalisation is useful but the US's house is in such disarray that it seems like the tip of the iceberg. I legitimately believe that the US uses bureaucracy and delays as a way to discourage people from applying for visas and or push visas into the next calendar year. It is the only explanation that makes sense.
To be honest the entire US visa process could be done, end to end, in one day if they wished (ex. background check). And in fact US embassies in exceptional circumstances actually do accomplish the entire visa in 24-48 hours (yes, even for immigration visa).
Why these make no sense:
- They take up a good chunk of a page and consume both paper and the reader's time.
- These forms aren't exactly optional, so I am going to spend time filling them out anyway. They would be easier to fill out if they were electronic, and if I wasn't repeating the same entries for the Nth time. The long disclosure adds nothing to usability and is only taunting.
"Paperwork Reduction Act Statement: A federal agency may not conduct or sponsor, and a person is not required to respond to, nor shall a person be subject to a penalty for failure to comply with a collection of information subject to the requirements of the Paperwork Reduction Act unless that collection of information displays a current valid OMB Control Number. The OMB Control Number for this information collection is <number> Public reporting for this collection of information is estimated to be approximately <n> minutes per response, including the time for reviewing instructions, completing and reviewing the collection of information. All responses to this collection of information are mandated by <statute>. Comments concerning the accuracy of this burden and suggestions for reducing the burden should be directed to the <agency> at: <addr>."
But yeah, otherwise I agree with your statement. When I filled for my green card extension I had to put my application in a box, not an envelop, a box.
It is beyond...well I don't know...beyond something that our federal government proves itself time and time again so incompetent at this sort of thing.
Yes, I understand in Washington that the political process drives everything and that contractors have turned milking-the-cow into an art form, but certain important issues, such as immigration reform, should (pardon me) trump this stasis.
[1] http://qz.com/182303/these-three-management-rules-helped-fix...
I wonder how much of the delay can be attributed to lobbying from the immigration lawyers that profit from the complicated, paper based process.
I have no problem with not passing any cost to the American taxpayers (and even that could be debatable, yes I'm an immigrant but I've never been unemployed and I believe I create wealth so it's a full net win for the country), but the system is really hostile and dysfunctional.
Would you believe that the NY State Police aren't a certified fingerprinting service for INS? (or at least they weren't 20 years ago)
My wife had her prints done by the state police, but Immigration rejected them on that basis. Instead, we had to drive an hour to a camera shop that had the appropriate certification, where the clerk - an Indian immigrant himself, by appearance - took her fingerprints.
It boggles the mind that they'd prefer a clerk in a camera store rather than the state police. It seems clear that the goal isn't proper documentation or security or anything like that, but more along the lines of collecting certification fees and perpetuating the bureacracy.
* A permanent residency application based on my marriage.
* An application to remove conditions on said permanent residency
* An expedited naturalization application
We didn't bother getting a lawyer: the paperwork was easy. The problem isn't that the paperwork is hard. The problem is that there is so much paperwork to file.
Have you undergone the interview yet?
I too decided not to get a lawyer. It went fine. But it was not "easy" it was long, painful, and trivial to make an error (the paperwork is extremely poorly designed, we used online guides instead of the official instructions to avoid tons of gotchas, in particular with the financials).
People just need to be extremely careful without a lawyer. It isn't "easy" but it isn't so hard that you NEED a lawyer either (which could cost thousands).
I commented on this above, but it nearly happened to me, too. We picked an immigration attorney to look over our paperwork, just to make sure we didn't make any glaring mistakes. While we did get some good suggestions, they missed something pretty big that we found on both the official government documentation and a number of forums, and had decided not to go with their services anymore. Luckily we caught it prior to submission.
[0] http://www.uscis.gov/working-united-states/permanent-workers...
Also, what tends to happen with a business process is that it involves many people doing their specific part and handing it off to the next people. Properly automating a process can involve eliminating many jobs. And since people like to keep their jobs, the more streamlined redesigns for systems and processes are often not really considered as they would result in too many people being out of work.
That can be especially difficult when decisions are being made by the business organization that wants to protect its power.
Another issue is that business processes just naturally tend to become more and more complex over time. And its hard for people to accept that they need to be streamlined. Because business people generally don't understand user interface design principles and have lots of rules and laws sent down to them from above. So all of those constraints and great ideas from the higher-ups tend to eventually get passed on as over-complex forms and processes.
It seems likely that some of those issues may have factored into this.
When government agencies like this talk about a "cloud-based system", what do they actually mean?
> the initial $500 million contract had been awarded to IBM
So IBM accepted a pile of money for the project and didn't get it done. The blame lies solely on them. Sure, there were bureaucratic obstacles, and government officials who didn't do things necessary for the project, but that's no excuse; IBM was responsible for making it happen.
I keep hearing people talk about government inefficiency as though it were actual government employees who were failing. But it's usually like this: a failure of the bidding process and of accountability, where a large corporation (it's always one of the same few) collects money for a job and doesn't do it.
"The bidding process and accountability" is a function performed by actual government employees, so its failure is a failure of actual government employees, even if that failure is visible through the failure of a government contractor.
A big problem is that government often does not have staff that understand enough about IT projects to effectively execute them, so it hires other people to do them: but the same skills that they lack that motivate outsourcing means they also don't understand IT projects well enough to do any of (1) effectively specify what it is that the contractor is supposed to do to allow contractors to bid properly, (2) effectively evaluate potential contractors, or (3) effectively oversee the contractor once one is selected.
> Agency officials did not complete the basic plans for the computer system until nearly three years after the initial $500 million contract had been awarded to IBM
Nothing wrong with a company bidding on a project of undefined scope. They likely properly laid out what that money bought -- hours, months, years of consulting / implementation. If the client did not have the interested in getting to implementation in short order, that is the client's issue. The company (IBM's) clock starts immediately and the dollars start disappearing.
The only way to stop this would be to punish the contractors of the failing projects in some extreme way.But that, of course, will never happen.
Anybody want an F-35?
An in-house development team would know the domain, the managers would be much closer to the business and have more authority to make day-to-day decisions.
Plus, if the developers are permanent staff, they're aware that they will be responsible for the system once it's live.
That's much better for everyone, except the politician, whose wife has shares in IBM.
Its not really any easy problem. Its about 6 million applications per year.
What we should really do is make an open source app or website (perhaps p2p, perhaps run on donated servers) and completely eliminate any part of U.S. Citizenship and Immigration Services related to processing applications, as well as all of the relevant laws. The laws are too complicated, and the bureaucracy of USCIS uses a very large budget of more than 3 billion dollars. A short (probably 100-1000x shorter) set of streamlined replacement laws can be designed by experienced lawyers within a few months.
We can make a contest that provides 5 million dollars for the best 3 designs for protocols and schemas for a common data exchange system, then another 5 million for the best 3 apps that allow people to process data. This will spend 30 million. We can use 100 million to buy off the politicians or companies that try to stop us. That should leave hundreds of millions left to donate to poor immigrant families and/or unemployed federal workers.
The app or site can be staffed by about 200-1000 online volunteers. This will be plenty assuming most routine applications can be processed within a few minutes.
For example, clouds and datacenters in large orgs should have three different groups offering competing solutions. If one (inevitably) underperforms, then it gets axed and a new datacenter option is bootstrapped.
So when you decide to 'axe' the underperforming org, what does that entail?
You could redistribute everyone, but if the error is category #2 or #3, then you saddle the higher performance groups with the causes of the problem. But if you fire everyone and the problem category #1, now you have to hire a bunch of new people and they are still doomed to failure.
How do you solve the problem where the organization needs to be self-aware enough to figure out if it has a category 1, 2 or 3 dysfunction?
Nokia had competing internal smartphone groups, which led to much internecine corporate warfare but precious little in timely smartphone development
Also... waterfall.
The Waterfall method has not been successful for 40 years