Tell HN: 2 weeks ago, I didnt know programming. I just wrote a web scraper. | Hacker News Reader

Tell HN: 2 weeks ago, I didnt know programming. I just wrote a web scraper.

34 pointsnaithemilkman41 comments

Feeling really chuffed and had to share. Great way to end the year.

To all business MBA guys learning to code: you can do it! just get started!!

Happy New Year guys :)

P.S It was in C# in case anyone is wondering.

View on HN

> To all business MBA guys learning to code: you can do it! just get started!!

No, all of you can't. But it's not a bad idea to try and see if you think you'd like it!

I am reminded of the half-joking book title: "Learn to Program in 21 Years".

I agree: most of you MBA guys will suck, fail and never figure it out. Some of you will. Everyone should try though.

"Teach Yourself Programming in Ten Years"

http://norvig.com/21-days.html

Congrats. Programming is made out to be a hard thing and for people who don't think logically it is. But to a logically thinking person it's like the greatest game you'll ever discover.

I was intimidated for the longest time about picking it up. Partly because I wasn't spectacular at math or algebra or statistics and all the arcane calculus.

Part of the perfectionist in me was also trying to understand everything that happened. Like what actually happens when I enter Console.WriteLine("Hello world");? That is always going to be a losing better so I've since subscribed to the "I know how to drive but I don't know how a car operates" school of thought.

Somewhere along the way, I realised programming isnt about hardcore math but more like just grouping actions together in a logical manner.

Perhaps programming doesn't resemble your perception of the maths you've been exposed to, but honestly all of maths is grouping together actions in a logical way. Programming is very much a certain kind of mathematics (though if it helps to think of them as distinct, by all means do so!).

I relate very much to your perfectionist statement. Keep in mind that each thing you learn about how the underlying machine operates will make you a more effective programmer. Learn about the craft of programming (patterns of design, abstraction, etc.), but don't lose sight of the fact that programs run on physical machines, and our bound by the properties of those machines.

Even for seemingly logical people, until you've done battle with a computer by programming, you probably don't realize how sloppy your thinking really is. Programming will make you more logical just as push-ups will make you more muscular.

it also tends to teach you just how sloppy your thinking usually is with uncanny precision, probably more than you like. Computers invariably do what you tell them to do with great precision and speed but almost never what you want them to do. The number of times that I've stared at a single line knowing that I was looking at the source of the problem and still not seeing the wood for the trees is best left unmentioned...

Programming is a harsh mistress, but the rewards are worth it.

Way to go. Now make sure you save your code and take a look at it in a year. That's when you'll really see how far you've come.

Congrats, glad to hear you had success. I'm curious as to why you started with C#. Most people these days are steered the way of Python or Ruby it seems. Care you elaborate on your goals?

My cofounders were C# guys. One new benefit I thought of recently was that C# is similar to Java and thats what mobile apps are coded apart from Objective-C. Who says the MVP has to be a webapp? :)

In anycase, my thinking was just get started first. Im probably going to dabble in Python or Ruby (probably Python) just to see what its like at some point. But for the time being, just getting my head down and writing code and asking really n00b questions on stackoverflow is definitely the way to go.

If you're interested in web scraping, take a look at Mechanize (it's available for Perl, Python, and Ruby, at least, though I prefer Ruby) - it makes web scraping trivial.

Im not interested in web scraping per se but in my previous gig, I was doing erm some 'marketing' through social networks and I was frustrated at not being able to contribute more actively to the process. The marketing work was essentially a programmers job.

So its nice to be able to look under the hood now and see whats going on!

Visual studio is one of the best IDEs out there, so not a bad choice.

Just learn ruby on rails as soon as possible so you aren't tainted by the horrible sample code of c#/java/php. Ay least get into codeigniter.

I would venture to say that you knew programming, you just didn't know coding. Organizing a good program is not all that different from organizing a good business.

I want to say you're totally wrong and ignorant, but in case you have a good point, please explain. How is organizing a business like organizing a program, practically speaking?

I was trying to keep my comment succinct, but what I mean is that at the fundamental level, programming is problem solving, and so is running a business. The ins and outs of particular businesses and particular programming languages or computing platforms may vary, but if you're skilled at problem solving (and it is a skill) then that skill is equally applicable in both arenas.

I did almost this exact same thing back in 2005. I had not programmed anything in YEARS and I had to help write part of a system that performed DB queries, screen scraped, file transfers, etc.

In a 24 period I tried Java, Perl and Python and in the end Python won by miles and miles. I now use Python daily for all sorts of tasks. I love it.

You mean in 24 hours? I think its gonna take me at least a few days lol

Good for you. Personal development never ends.

Yeah, that was what I did when I didn't know anything about programming, too.

The next step is to learn why web scraping is discouraged, and how to use public APIs instead.

What if the site doesn't have an API? Or what is the API is not user friendly? Case in point: address book scraping employed by many companies bypassing APIS. e.g dropbox

> The next step is to learn why web scraping is discouraged, and how to use public APIs instead.

Right, just like google uses public api's to access all those sites. Web scraping is responsible for a very large portion of all the traffic on port 80 and the number of sites that have published APIs is extremely small compared to the total.

I know, it's odd how some people act like all web scraping is a terrible thing, when people have built such large businesses based on it. Companies try very hard to get scraped by Google, and don't like being scraped by other random entities so much - I suppose it depends on your intent. Poorly written spiders are a problem for system admins.

If a resource does have a public API, some very positive things about using that instead are that not only is it less resource intensive for them, it is much easier for you to create and maintain. If you're relying on scraping to get specific info from a site, it's a huge Odin when changes to their HTML break your code.

Yep, I access HN via a powerful rest API.

Well done! If you write a detailed account of how you did it, you are bound to inspire a lot more people to try it out.

Your next step on the path to being a hacker is to share your work. Where can we see your "web scraper" in action?

I don't know where can I post it? Probably post it to my blog once I've commented it up. Heading out for dinner with the gf so check later! FYI, it looks like a thing of beauty to me but it probably looks like dogshit to the pros. lol.

https://gist.github.com/

Github! If you're coding, you need to learn source control yesterday, and it makes sharing your code easy to boot. :)

Or just paste.ly

You will learn a great deal by posting your work for others to see. At first it will feel uncomfortable. You will be concerned that others will mock your work (and they might). But, by the 3rd or 4th time you post your work you will start to feel comfortable with it. The important point is that others will benefit from your experience. Karma.

What a nice thought. Someday someone is going to benefit from the code I wrote. LOL!

I feel like 10,000 miles away from that right now.

Does this 'URL' help? I think you have to be a member of github to see this. Nice viral mechanism lol.

git clone git://gist.github.com/761348.git gist-761348

This one does :-)

https://gist.github.com/761348

Fairly idiomatic code too; particularly impressed with the use of XPath.

Are you scraping glogster or are you with glogster? ;-)

I had to stackoverflow a lot of it so I claim no credit but thanks anyway :) BTW, what do you mean by idiomatic code? UPDATE Ok quick google for idiomatic code. Gotcha. I wonder what non-idiomatic code for c# looks like?

If you'll permit me to be annoyingly geeky, "That's good. You have taken your first step into a larger world." :)