spidaman

I led a counter case where a rewrite was very successful. There had been a major component of the architecture that just approached the problem wrong at its inception. Changes that one would have thought should take a few hours or day to make took weeks. And once you dug into the code, you learned why. It lacked adequate tests, proper componentization, error handling and operational visibility. It was far beyond refactoring, it was just ill conceived. It's poor functioning had cascaded into other systems; they were riddled with hacks to compensate for the problem system's deficiencies; technical debt had become a cancer that spread around the architecture. At a certain point, we declared the technical debt had reached technical bankruptcy and acquired buy-in across the organization (execs to engineers) that we needed to start a new code base.

However, part of making the rewrite succeed was sucking it up and doing continued maintenance on the legacy system. It was no fun but it had to be done. Things that couldn't be implemented in a reasonable time with the legacy system but were high priorities were implemented in the new system to assure that the win wasn't just one of purity of essence, it was enhanced functioning. Enough was learned from what worked and what didn't in the legacy system that we had a good deal of clarity on what requirements we wanted to fulfill. The hand wringing over excessive feature creep and other foibles that can make rewrites fail were attacked with discipline.

I've heard of many big rewrites that failed but don't buy the argument that they demonstrate that it can't be done. It can.

Sure, just don't judge a book by it's...

I don't know what, if any, magic Qwiki has cooking but I think the OP is excessively harsh. My point was that a lot of startups have needed a big round to go to the next level from a limited proof of concept where there's nothing special on their public facing service. So what? Hopefully the vision and potential is evident under the hood and that's what the investment was predicated on.

The author needs to get a grip, a lot of technology bootstraps on top wikipedia with a capacity to scale from there. Powerset unveiled their natural language search on the wikipedia corpus and were acquired by Microsoft for $100M a few months later. I'm sure MS wasn't just acquiring an index of wikipedia.

Filo was a total hacker, reportedly still is. At least into the late 90's, he was active on the FreeBSD mailing lists with encyclopedic knowledge of SCSI card and NIC drivers and assorted other hardware and FreeBSD stuff. He was oft found sitting on the hallway floor in the first colo I used (ISI in Mountain View), screwdrivering the chassis' of Yahoo's early servers.

Regarding "It seems safe to say, then -- no big surprise -- that aside from considering a phone's manufacturer, going with a higher profile, top-of-the-line device will improve your odds of receiving regular OS upgrades..." The Epic 4g has been Sprint's flagship phone, still running Eclair. I'm regretting not going with the Evo at times.

Some galaxy S handsets have Froyo (see http://pages.samsung.com/ca/froyo/English/?pid=ca_home_subba...) others don't, like my Epic 4g. Is it really politics or QA challenges; no official word has been forthcoming.

I agree that debunked pseudo-science should needs to be called out as such. However, I'd rather Salon re-publish the article bookended by the counter-evidence rather than take the article down and have the URL return an HTTP 410 (Gone) error. Here's the article in Google's cache http://webcache.googleusercontent.com/search?q=cache:DEZc9b-...

Yea, I was also hoping for something insightful about Scala adoption. Doh!

As far as premature optimizations, if they're truly premature and moderate/high effort then yea, you're usually better off getting things working quickly first. But I've seen the admonishment of premature optimization often used to turn a blind eye on the technical debt that can result from simply going fast. Successful projects require doing the simplest thing possible that will work while maintaining a conscious effort to keep track of refactoring candidates, performance inefficiencies and other messes that'd otherwise languish under the rug and, maybe, just improving them before proceeding.

Google has proven to be very effective data collectors (see "What they know" stuff from WSJ e.g. http://online.wsj.com/article/SB1000142405274870330970457541...) - it may on the one hand be surprising they don't use this data more effectively to identify and thwart spam. OTOH, they're making a lot of money on spammy SERPs.

Yea, I knew the post would be vacuous when I hit "tyranny", but I guess the flourishes makes good HN bait to speak in sweeping terms like that. There's little about on-disk consistency and data loss risks, scale characteristics yet there's coverage of the wire protocol? Meh. Sure, I'm glad to have a lot of these tools available, the bad old days of rolling your stuff with Berkeley DB or NDBM seem safely behind us but the reality is that there are many classes of problems for which SQL is still and will remain the most sensible solution. Get over it.

I also don't understand the 15 years figure either, is that a reference to when MySQL was initially released? I hope the original poster understands that SQL is older than that.

Initially, it won't speed you up. Developing the right provisioning formula requires some investment of time but I think it's time you get back many times over because you don't need to follow a bunch of manual steps and making changes to the infrastructure is easier when it's all code. I prefer chef simply because it's easy to develop customizations; the DSL is just ruby. I have a pretty non-standard application stack but automating the provisioning is pretty straight forward. I can bring up the whole thing from bare machine images in minutes. Check out http://wiki.opscode.com/display/chef/Launch+Cloud+Instances+...

I don't know if there's a puppet equivalent to chef-solo but using it with vagrant and VirtualBox is my favorite way to develop chef recipes. I did short presentation about this a few months ago, see https://docs.google.com/present/edit?id=0AVIsoyl05MRmZGNwY2N...

BTW, the OP talks about needing a separate security group for the puppets but last I checked, the default security group on EC2 makes the machines you own visible to each other and nobody else's that you haven't explicitly permitted. So I think all of that was unnecessary.

Seems like the comments here on HN are talking about pair programming far more than Mark's OP. The top line of his post was that he prefers to work alone, that the arguments and social overhead of collaborative development have made him unhappy. Pair programming isn't the only way (and often isn't the best way) to collaborate; reading each other's commit messages and talking about them is often sufficient.

And sufficiency is at the core of what makes software development fun and productive: a sufficient number of eyes on the code, a sufficient issue tracking app (and/or other collaborative apps) and sufficient amount of automated test coverage is often all you need.

Having inherited code developed by lone wolves a few times, I have come to believe that healthy code is like healthy DNA; it needs a variety of contributors. Yes, it can be difficult at times. It can involve struggles but I have no sympathy for the complaints about them, the struggles make you stronger.

There's a big difference between the cost to install and operate and the cost to scale. If your hardware utilization characteristics are easy to calibrate for and capacity requirements are predictably steady, you can certainly optimize your own racks at a colo. However, hardware changes quickly, businesses typically change even quicker and capacity requirements are often highly variable. Unless the above conditions are met, it's insanely risky to go into a contract for a colo. Let us know how that one year investment recovery works out for you.

Once I got up and running with sbt, I really took to it. You only need to be concerned about the net when you add a dependency that you want it to resolve by fetching remotely. If you want alternative web frameworks, I think scalatra looks promising.

Stuxnet is a worm that targets net-isolated computers running Siemens control systems propagated by USB drives. All things old are new again, this time with sneakernet viruses.

Quite the dilemma. When this happened to me, a former coworker who succumbed to cancer. I thought about unfriending him just to make the suggestion go away but that gesture is just too cold. I'd rather FB provide a mechanism to mute their "reconnect" suggestions.

Hate to be the Monday morning quarterback but hey, it's Saturday morning and I'm stuck in a Starbucks in Benicia. So, this is a nice walk through the optimization process but it is a fundamentally unscalable system. When the workload triples next year, does it make sense to scale up the hardware? (No)

This system doesn't scale out without ad-hoc partitioning. I think Etsy's approach here should have been informed by the New York Times' project a few years ago. They converted 4 TB of scanned tiffs (their article archive) into PDFs on a hadoop cluster running on ec2 and s3. Parallelizing the process across hardware nodes is the scale free way to do this, exactly what hadoop was intended for.

All that said, I know there are smart folks at Etsy. I'll give 'em the benefit of the doubt; there may be a good reason not to go that route but this write up didn't make that clear.

CouchDB is an MVCC document store, it isn't concerned with scaling the way Cassandra, HBase and Voldemort are. This is a pretty good overview of the landscape http://www.vineetgupta.com/2010/01/nosql-databases-part-1-la...

I'll grant that monetizing advertising without scale may be futile. However, the flip side of this is that monetizing too late shapes a company into a VC addict, always pushing the fund raising lever for another scale pellet. Few media sites make it at all, hoping they can scale their way to viability. At a certain point, entrepreneurs need to tell VC's to get out of the way with their the-first-hit-is-free scale advice.

No so fast with the yacht, he didn't say they're doing 230k saves/sec. That'd be the case if Warhamemr Online saved statuses for 7M concurrent users every 30 seconds. The basho guy said 7M daily users, not 7M concurrent ones.

The app I've been working on doesn't use replicas for read scaling, just cache-money (ergo, activerecord). We're partitioning for write scaling but no slaves (well, we're doing master/master replication but not for read capacity).

After having had to contend with replication lag, lots of instances spinning and related headaches for years I'm pretty convinced that scaling reads with binlogs is in my past for good; it's way more expensive than having a good write-through cache.

Ping seems generally confused about how to discuss commercial support of open source projects, not just Cloudera. Note his reference to "Northscale, parent company of Memcached" is even sillier. AFAIK, the only "parent" I'd think of for memcached, Brad Fitz, is still ensconced at goog.

I've talked to the Northscale guys, sounds like their onto good stuff and I'm not faulting them (just as I wouldn't fault Cloudera), Ping just needs a copy editor.

Among the problems I see with most of these services is that they don't provide anything meaningful to socialize about that is unique to mobile and location based data. For general social banter, facebook seems to have won the turf. To your point, if you're no longer in the hooking up or carousing with friends demo, these apps have seemingly nothing for you.

All that said, I think there's tremendous value potential for overlapping virtual and physical communities. I just think all of these players have been too early and lacking imagination to strike a hit. Now that location aware devices are nearly ubiquitous, the next steps are commodifying and standardizing location data, APIs, etc. It seems inevitable. If there's anything that's going to yank me from consulting/contracting back into full-time startup hackery, it'll be something along these lines.

Yea, boto is a great library. IME, SQS seems to be a good option only when you have low throughput. I tried running a high volume of jobs through SQS and found that I needed a lot more workers (EC2 instances) to do the work because each message was so slow to pickup. That made things cost prohibitive. I'm currently working on another AWS based project. It was initially using ActiveRecord based db queues but that, of course, failed at high volume. So I replaced it with kestrel, which works with any memcached client, and that's been great. However, kestrel doesn't do topic/group subscriptions where lots of receivers can get the same message. That's where these message servers seem to be potentially valuable options.

If you've never needed asynchronous systems and messaging, perhaps you should consider yourself lucky.

I think everybody that has to write applications that deal with URLs as core identifiers have asked this. It's also hairy 'cause leading part of a URL (protocol and hostname) is case insensitive but the trailing part (path, query string and fragment) isn't. At my last job I created a framework for normalizing and canonicalizing URLs as well as storing them consistently (with the hostname components reversed), it was a big improvement for retrieval and duplication detection accuracy and performance.