Whatever happened to programming?
reprog.wordpress.com
reprog.wordpress.com
Don't get me wrong, I'm grateful for all of it, but it just doesn't seem like what I was promised when I followed SICP for the first time. It just feels like I spend most of my time scouring through documentation and trying to remember umpteen different sets of syntax and class names rather than actually thinking in code.
Back in ye olden days, most programming tasks I performed felt quite natural and painless, just a quiet little chat between me and the compiler. Sometimes longwinded, sometimes repetitive, but I just sat and though and typed and software happened. The work I do these days feels more like being a dogsbody at the tower of babel. I just don't seem to feel fluent in anything much any more.
We talk about 'flow' quite a lot in software and I just have to wonder what's happening to us all in that respect. Just like a conversation becomes stilted if the speakers keep having to refer to their phrasebooks and dictionaries, I wonder how much longer it will be possible to retain any sort of flowful state when writing software. Might the idea of mastery disappear forever under a constant torrent of new tools and technologies?
It's the pride in me. The closest I came to it a long time ago was when I animated a bunch of search algorithms for a class project in college. That was one of my favorites because I could show my non-geek friends what I was doing and they could appreciate it.
It's a big reason why I am now focusing on my web app. The web is at a point where I can build something and get (potentially) millions of people using it. I'll sacrifice some hands-off-ness in my programming for that.
The thing is, most of the languages you mention are pretty good at what they do. Python is a good general purpose language, and Django is just a Python framework, right? (Not a Django developer.) XHTML does a good job describing hierarchical content to be displayed. CSS is a good language for styling content separately from the structure on a site wide basis. JQuery and Javascript can be looked at as a single thing, and if you know the syntax for Javascript data structure literals, you pretty much know JSON. Google App Engine seems to potentially offer a lot for the learning curve required (not an App Engine developer, just read some tutorials).
XML, well, there's just no excuse for that. :)
Some people complain about all the parts you need to know to build web applications as compared to desktop applications. But I think that the different languages each doing their thing well makes for a better development experience. The other great thing is the diversity of choices for the backend technology stack, like Viaweb deciding to build web apps just so they could avoid Windows and use Lisp (...and some Perl and C++, I think).
Ever worked with Spring / Hibernate / Struts / EJBs / Oracle? If pain should have a definition when related to programming, then that's it.
If you really want a more unified development method, look into Pyjamas http://pyjs.org/ . It's a port of google web-toolkit to python. You just write everything in python and then it compiles the javascript front end for you.
This is the price you pay for being able to work with other people. If you never wanted to reuse anyone else's code or talk to anything else on the Internet, you would not need to know any of this. To stand on the shoulders of giants, you need to speak their language.
But at least now we don't have to take a stack of punched cards to the Priest with an Offering (bribe) and wait a day just to get a printout saying that the damn thing dumped core.
The reality is that "back in the day", computers weren't useful to very many people. Now they are.
I agree that all this "overhead" does have utility, I'm not saying we shouldn't have it -- but it's less pure/fun compared to the old stuff (IMO).
The problem is that today we demand much more than what those earlier tools can give us. Such demands result in an inevitable (?) increase in complexity of the aggregate toolset. The tools are capable of more, and give you more options. So it seems that learning and using those tools effectively would be more challenging.
But it's not a one-way street. Along with this increase in complexity have been efforts to simplify, streamline, and structure these tools and ease their use. But both the demand for increased ease of learning/use and the demand for power are moving targets. Hopefully with time both could be maximized. We're just not there yet. Remember, computer science is still in its infancy.
Please don't tell me each building project reinvents concrete, or steel girders, or cranes, et cetera, et cetera. These are all technological wonders, and using them means standing on the shoulders of giants.
It just looks different in physical construction.
Anything else, where you start having dependencies on components sourced from elsewhere then you are in this situation.
Personally, I don't really see what the problem is - looking at and using other peoples software is usually a great learning experience.
Though that doesn't mean we don't use third party libraries, I just think we have far more flexibility in not using them.
Yes, Cocoa is a massive library, but it has 99% of the things that I need. I think in objective-c and cocoa.
When it comes to web development I'm doing 100's of tiny little things in 2 different languages with 2 different markups to get something done.
We were talking about why some people love working in Objective-C and why others abhor it and came to the general conclusion that it is more of a philosophical issue than a technical one.
That the bits given to you by Cocoa provide the minimum functionality required while the frameworks of other environments try to cover every piece of functionality imaginable.
The former suits the makers and the latter, the assemblers.
I think there is room for both in this world (although I admit I belong to the first camp) but treating both types of "programmer" the same rubs both the wrong way.
On the other hand, Django/jQuery is pretty sweet. I think it's a matter of working with orthogonal and well-designed systems, more than the number of systems.
The situation seems ripe for a disruptive technology to emerge.
I seem to spend way too much time on what I call "guessing the magic password" - tracking down that one bit of punctuation in some obscure value in some xml configuration file that makes my whole system fail silently in one environment. I might spend 5 hours beating my head against a wall, only to finally figure out that this brittle system over here needed me to put some proprietary prefix on some host name in this one context. And at the end of it, I don't feel accomplished. I feel like I'm wasting my life! Figuring out which parameters to pass to which APIs in what order is a step above "guessing the magic password" - but it's not that big a step.
A few things come to mind. You can try to make sure you are one of the people writing the libraries. You can implement the next platform and use all the aesthetic sense you have in you to try to make it a joy for others to use. Put yourself in the position of the programmers who write the systems that make the rest of us age faster, and try to do a better job. Or content yourself with getting your programming joy from your elegant side-projects.
It's always been this way, but lately it's gotten worse and worse.
You need to fix a problem. Your friends tell you about technology X which seems exactly what you need. So you pick it up, take a quick look at the samples, and wham! Within minutes the thing is working exactly like it is supposed to!
Well almost. There's this little part that's broken that ruins the whole thing and then you spend hours or days (yes, perhaps weeks) digging up obscure configuration documentation and tweaking APIs in ancient Klingon, sacrificing turtles to Ba'al or whatever else it takes to get the damn thing working like it's supposed to.
Then it's on to something new.
The funny part of this whole story is -- when your friends ask you about how programming in X was, you are more than likely to brag about how easy it was to solve your problem. After all, the bulk of the problem was fixed almost instantly. Don't want to look like a dweeb. And so the cycle of pain continues.
As a beginner, I didn't appreciate big libraries (like Cocoa). I couldn't wrap my head around the idea of having tons of prewritten code and stitching it together - I understood programming as writing code - so I wanted to write my own damn code, not just manipulate massive libraries.
So I turned to PHP and developed my own webapp (a stripped down Google Docs), from scratch. I quickly stumbled upon the purpose of frameworks and libraries. After writing my 20th SQL query I realized I was wasting time. About 10 hours later, I made my own light PHP framework. Now I didn't have to worry about my SQL syntax (it was automated based on the data model variables) and I got to start working on really cool/interesting programming issues rather than the mundane blah code.
Now I've started to investigate Cocoa again and I realize that libraries and frameworks have not hurt programming, but dramatically catapulted it into the realm of science rather than plug-and-chug. Today, our software is more complex than ever and it will only get more complex in the future.
I think if you see programming as just "stitching libraries together", then you've lost your passion or never had any to begin with. Programming has evolved into a science where you have to learn the foundations (languages) AND the pre-existing solutions (libraries) in order to figure out what's been done for you so that you can focus on exploring new and more complex issues.
The SQL I kept repeating in that application was simple CRUD functions on MySQL tables. I know that SQL is so much more than that.
"select name, pass, hash, email, realname from user where id = ?"
versus
session.get("User", x)
Oh dear god, a blasphemer!
The former can cause performance problems if you have blobs as well as being an unnecessary security hole, the latter can cause performance problems over large datasets as it often forces the execution plan away from a set based solution.
Nobody debates session.getUser(id)
But SQL syntax is a dog. Especially when it's sitting next to real code, wrapped up inside strings and making everything feel dirty.
If you're using an SQL database from within another language primarily just to dump objects and pull them out, and especially if you're doing a lot of it, I have trouble thinking of a situation where you wouldn't prefer to abstract away the SQL bits. Maybe if you're working in a language with built in SQL syntax support? Even then, it's not exactly compact, and you've got quite a bit to do even after you get your results back from the database...
The problem is all the other crap that you have to layer on top of it for it to be really, really useful.
Also, abstracting away SQL doesn't mean you abstract away set algebra. It means you write "$current_user->messages( text => { -like => 'foo%' })->recipients" to get a list of people the current user sent messages containing the word "foobar" to, instead of some 200-character-long embedded program in another language. A good ORM is an essential abstraction.
I wouldn't call myself a programmer yet ( Joe Hewitt has made that apparent to me: http://twitter.com/joehewitt/status/9813494038 ), but it seems like modern software engineering is a form of alchemy now. I thought a software product is a combination of different mathematical techniques, but translating that into code seems so cumbersome (taken into consideration that I've learned and had experience in my choice of programming language and framework, which is also a major task on its own).
From my somewhat inexperienced standpoint, I think programming has gone from instructing the computer to complete a task, to telling the computer to create a logic and handle events based on a complex set of conditions. The former was simple - 1, 2, 3, done - while the latter requires you to design a system, then make that system function through code.
I see old-style programming as following a map: go straight, turn right, turn left, and then stop at the destination. Modern programming is more like telling the computer to use this logic to find its own way on the map, then follow through.
Basically, I see programming slowly morphing into the creation of artificial intelligence.
No, programming is a result of mixing math and linguistics to achieve information processing. Of course that's my inexperience evaluation and I bet a veteran of this field would quickly tell me that I too have an incorrect notion. But see, that's the thing. It shouldn't be this hard to comprehend! Sadly, as Scott Rosenberg says in his book Dreaming in Code, the reason why programming is imperfect is because people made it that way and people are imperfect.
Early-phase technology workers with those sophisticated machines were true 'engineers', and were paid well. Over time, things evolved (of course). That history confirmed and taught that as technology matures, it becomes commodity. Economies of scale emerge, markets develop, etc. make it such that the technology practitioners ultimately become installers and configure-ers. I think it started to go awry, in my timeline, when the term Application Programming Interface (API) hit the streets. :-)
However, the above is really only true in the 'application development' space. Most software development opportunities are in that space; and therefore, yes, most opportunities to develop software are boring, non-engineering jobs -- rather blue-collar, in fact. To one steeped in a true CS background, that's a real disappointment when considering a lifetime career. This is true for those in their 20s as well as their 40s.
So, I agree with other posters here: get creative! Reject the direction of the herd, if it seems fitting. Do it anyway, just to experience and learn.
One solution is to move to a new area of software development that offers challenging problems.
I am currently, like many others it seems, working with creating enterprise CRUD-apps and integrating them. I try to find joy in reaching for a skill level where I can produce solutions quickly, solutions that are easy to change, solutions that immediately lets you know where and why an error has occurred, solutions that are secure, solutions that scale, solutions ran in the cloud instead of on servers that it takes it operations months to set up etc. There really aren't that many people out there that can do all of the above well (me included) and there are lots of money to be saved for your employee/clients doing those things well. I think we would see a lot less attempts at off-shoring if more people in enterprise-it would strive for that kind of knowledge.
I guess what I'm trying to say is, maybe one shouldn't have too high expectations of the technical challenges in problemscapes that have been mostly solved already but that, at least in my area, there are still challenges to be found. They are just of another kind and maybe you can find them interesting? They keep me interested enough at least.
sigh Why am I posting on hacker news again?
If you are building applications for end users, though, it's hard to justify not leveraging libraries and frameworks that already supply much of what your users want. Especially if the software does not directly make money for your organization.
The lack of fun is, in my opinion, a side effect of using weak languages that force us to write boilerplate and ceremony along with using half-baked non-Turning complete XML languages (don't forget the "L" in "XML" stands for "language").
Pun intended?
Maybe we should be more creative in our application's core functionalities, should we not?
Working with well-designed frameworks, though, is a true joy. It means I spend a minimum of time on plumbing and spend the majority of my efforts solving problems for my customers or otherwise doing high-value work. I understand there are folks out there who love socket programming, but I've done socket programming, and I have no desire to ever touch it again. Good HTTP clients, REST, etc make that totally unnecessary for the types of problems I routinely have to solve. (Speaking of good frameworks versus bad frameworks -- even in Big Freaking Enterprise Java there is a titanic gulf between ways to implement web services. RestEasy is probably the best architechted framework I've ever seen -- it is so good you can forget you're using a web service at all.)
My next business will need to make telephone calls. There is a whole lot of very ugly coding necessary to make that happen. Happily, Twilio has apparently already done it for me. I'll just snap together Twilio and Rails to be able to make the customers' phone ring, and concentrate on delivering the smart part of the service, which is that making sure what happens after the phone rings solves a real problem that is preventing their business from making money.
We're just different. For you, and most modern programmers, there isn't much value in that stuff, and you don't want to have to do it. And, in the modern world, the truth is we don't have to except in the rarest of circumstances.
In the modern world, developers (they aren't called "programmers" anymore, it seems) have the freedom to worry about prioritizing "high value work" to solve their customers' problems. But it wasn't always that way.
For some of us, that kind of sucks.
However, we (web developers at least) are writing software that is supposed to run on just about any hardware / OS / browser combination, and making that happen and doing complex and interesting things in that context is going to involve lots of arbitrary details.
That being said, if I think back 10 years, this type of work has become dramatically more pleasant thanks to a lot of open source software that has matured in that time.
If the complaint is that we can't all write our own virtual memory system or our own b-tree, I suppose I can't disagree that that is becoming less of a reality. However, solving interesting problems still requires an understanding of these things, and using existing libraries in my experience saves me time and allows me to have a bigger-picture view and solve larger problems. It is still very satisfying. If this article is written from the perspective of someone automating simple and mundane business processes, putting together forms, etc, then I believe the complaints are misdirected.
There's a fascination with shiny new things and it's tempting to think that the latest and greatest download from some smart guy solves the problem you are solving. But after a couple years working in that manner I'm not so sure.
I think it's tempting to view frameworks and/or libraries as an implementation of software patterns, and also as a way to avoid learning your trade. That's probably not concious thought behind it, but you see it in so many things... if you just learn this framework or use this library, you won't have to learn html, or css, or sockets, or synchronisation, or searching and sorting, or...
A Pharmacist (not sure what you call them in the US) spends years at Uni learning all about drug manufacture, how the body works etc, and now all they do is type script in the computer and hand over bottles of tablets
What does a signwriter do now? It used to be a skilled craft, now all it requires is typing text in a computer that then cuts the letters from a roll of vinyl sheet
Or Carpentry or any number of other trades?
On the other hand, nobody is forcing you to follow this "glue libraries together" -method. Disconnect from the internet and just start coding. As a bonus you'll get rid of a lot of distractions (email/IM/twitter/whateva). It can be quite refreshing, and who knows, maybe you'll even create something new and awesome.
I guess it depends on your motivation. I don't program because I love creating complex data structures or innovative new algorithms, I program because I want to create innovative new applications that change how people live, think, work, learn, etc. I signed up for programming so I could change the world for the better in my own little way -- not improve how an electronic machine inputs and outputs binary input.
But to each their own. It seems everyone likes to work at different levels of abstractions. I think we can both agree that those electric engineers are weird :)
As the languages and libraries improve, difficult problems become trivial, while impossible problems become merely difficult. Given enough time, problems eventually become trivial enough to be solved by non-programmers.
There will always be a frontier - problems that have barely just become solvable thanks to improvements and technology, and these 'very difficult' problems will be what the best and brightest will be drawn to.
Maybe the author is working on problems that aren't challenging enough.
If you want to program something fun, nothing is stopping you. Thanks to the libraries, you can concentrate on what interests you, rather than the details of how some standards committee thinks the word "referrer" is spelled.
For example, I try constantly make less work for myself. So much of the code we write is boilerplate so I have various tools that generate as much code as possible. I have written mini languages that specify the important parts and then generate all of the plumbing around it. I won't run out of interesting problems until I can generate an application by specifying the important parts (to the business) and generating everything.
So why complain? Go do something interesting. If the problems you are solving as your daily routine are themselves boring why would the process of solving them be any more fun?
Simultaneously, we're expected to be into the newest and coolest technologies.
Put these together, and we're supposed to systematically glue everything that's new.
Seriously, it's not all bad, though. I like being able to generate a test UI in under an hour or a basic web spider in one python file. The art's just different. It's no longer packing your entire logic into beautiful lines of BASIC. It's now more about choosing the right platform to do something that the user will appreciate.
If you're into the pre-library styles, I recommend looking at alternative hardware. When I wrote my first CUDA project, I had to figure out how to get random numbers with sufficient randomness for our physics simulation. You might also look at robots or embedded devices. There's plenty to tinker with on devices that don't have libraries available.
More seriously, abstraction is not the issue. The issue is poorly written libraries and languages that require you to worry about the small details. Your brain can handle many small details or a few big ones, but not both. I recommend picking the latter, and using libraries and languages that allow you to do so.
Hint: Lisp is really pretty swell.
I couldn't agree with this more.
But I know writing algorithmically interesting problems are not likely to appear at work, and I've made my peace with that, along with the paycheck I get for writing enterprise systems.
Our lives are going to suck in 10 years.
If you want to program as they did in 1985, go work for a game company. Or find an investment bank looking for C++ developers; they're out there.
Those were two excellent reasons that I had no interest when their recruiter came calling to ask if I was interested in applying to work there.
(My experience with people that know math and finance is that they do all their work in Excel. That's because simple programs become 100-file boilerplate monsters in C++ and Java, and someone told them that Perl and Python are not real programming languages. Sigh.
OTOH, turning spreadsheets into production software is profitable and enjoyable. Let the programmers program and let the finance people do finance.)
We are not talking the same sort of people. We're talking the type of people who require huge super computers to run their algorithms.
Also, my friend has a math and econ degree, never programmed, got a new job at a trading firm, guess what the first thing they want him to do is: learn C++.
Even if the industry goes in the shitter I guarantee programmers will be better off than 90% of the economy.
There are as many interesting problems today as there were in 1960 or 1985 or 1998. There will be demand for new languages (and new compilers for those languages). Operating systems may or may not be "solved" for now, but VMs are going to be of interest in the next 10 years.
The problem is that, outside of high-risk startups and possibly academia, it's very rare to get permission, resources, and support to attack the interesting problems. Blue-sky research is gone. This is because of the next-quarter mentality; there's a significant likelihood of failure if you're doing something interesting, and interesting projects cannot easily be done to deadline. And everyone wants to be on those projects. So a first-year programmer at an average company is not very likely to get to do an AI research project that may or may not lead to anything.
"Gluing" projects, on the other hand, can usually be completed in a predictable amount of time and have a high likelihood of achieving success for sufficiently mediocre definitions of "success".
Personally I am glad the same thing happened to other industries like shoemakers and car builders. They have been commoditized: so what? It's good for people who buy their product. The same is true of typical software engineering work.
Although I disagree about the extent to which our work has been commoditized. I don't think we're anything like factory workers yet, although it may someday come to that.
Probably the majority of programmers are working for internal corporate IT departments cranking out CRUD apps because the company mistakenly believes it's a precious, unique snowflake that couldn't possibly make do with off-the-shelf software for routine business processes, even though they derive no competitive advantage from custom software. Most of these jobs simply shouldn't exist.
The larger problem is that too many people profit from the dysfunctions of the software industry. From the empire-building CIO to the outsourcer in India, there are plenty of people who benefit from shovelling bad programmers at bad software, and would lose out if companies bought cheap, standardized software from independent companies.
In the end, you end up with a custom solution layered on top of a specific implementation (so, no standardization even if you started from the vanilla off-the-shelf). Implementing a tool that's specific to the needs of the company can turn out to be a reasonable decision.
Why don't you use the knowledge you gained of prolog and lisp and metaprogram the tasks you're given to get your work done even faster.
This talk about crud is boring, libraries are boring, is almost ludicrous. There are 200 million jobs in the united states alone that can be automated by software. Try to automate those jobs - that's cool and lucrative stuff.
If you are programming for work sure its going to be easier to grab a library to do a certain task for you even if you have to mangle your task a little bit to use it but when you are learning or even when you are programming for fun that is dull and really really not fun.
Now, such programs seem so lame and pale in comparison to the state of the art, while the amount you need to know and learn to match that standard is daunting, there must be a cost in terms of programmers lost to the profession at an early age.
Also, every program is either an interpreter or a compiler if you squint just right.