Hacker News Reader

Comment by djtriptych | Hacker News Reader

Great memories with this library, one of my all time favs.

It is fast? no.

But it had a fantastic mission: extracting data from malformed HTML.

Might be less common now but back then (~10+ years ago) it was still rampant. Many if not most parsers would barf on any deviation from the standard, leaving you to hand-roll regex solutions and ugly corner cases.

BS covered a LOT of these cases without forcing you to write terrible code. It mostly just worked, with a reasonable API, and stellar, well-written, example-laden docs.

View on HN

"Might be less common now"

It is less necessary now. One of the most important parts of the HTML5 standards, IMHO, is that it specifies how to parse HTML that doesn't conform to the standards in a standard way. In principle, every bag of bytes now has a standard-compliant way to parse it that every HTML5 parser should agree on. I don't use this to know how many edge cases the standard and/or implementations have, but it's a lot better than it used to be, and means that every HTML5 parser has many of the capabilities that Beautiful Soup used to (nearly-)uniquely have for parsing messy HTML.

I suspect Beautiful Soup was a non-trivial aspect of how the decision to implement such a spec was decided upon. It proved the idea to be a very valuable one at a time when most languages lacked such a library. Basically, BS won so hard that while it wasn't necessarily directly adopted as a standard, the essence of it certainly was.

Not just HTML standards, but I've used their detwingle function because one of the sites I'm scraping has a mixture of Windows-1252 and Unicode. It was clearly stored correctly, but encoded differently depending on what page view you were looking at. For example titles in an outline were broken, but on the actual individual pages fine. Their rendering also treated multibyte characters incorrectly during truncation.

I work in finance and this thing is still indispensable for us. A LOT of financial info is only presented on a dynamically rendered HTML page that was last written in the bad old days.

Absolutely. Until you have had to deal with this kind of problem first-hand, you have no idea how much of a relief it is that it exists.

Sometimes, finding and using the right library can completely turn around a f'd project.

True.

And BS has been that library for me on at least 2 such projects.

Any poignant examples?

It's actually pretty trivial to speed up, if you have multiple documents to parse, you can use multiprocessing.

    from multiprocessing import Pool
    
    def parse(html):
      result = []
      soup = BeautifulSoup(html, 'html.parser')
      for p in soup.select('div > p'):
        result.append(p.text)
      return result
    
    with Pool(processes=16) as pool:
      for texts in pool.imap_unordered(parse, my_html_texts):
        for text in texts:
          print(text)

it's a pretty common case to be parsing thousands of locally-stored pages. There are only so many cores and the task of scraping an entire site can still easily still be CPU limited.

You might want to try a different parser as well, I tried a basic performance test comparison in another comment (https://gist.github.com/MercuryRising/4061368) and the html.parser was very slow compared to the lxml-xml or xml parsers for bs4

    ==== Total trials: 100000 =====
    bs4 lxml total time: 110.9
    bs4 html.parser total time: 87.6
    bs4 lxml-xml total time: 0.5
    bs4 xml total time: 0.5
    bs4 html5lib total time: 103.6
    pq total time: 8.7
    lxml (cssselect) total time: 8.8
    lxml (xpath) total time: 5.6
    regex total time: 13.8 (doesn't find all p)

If you want fast HTML parsing in python+lxml, use html5-parser https://github.com/kovidgoyal/html5-parser

Have literally been building something with BS today. It’s very much still a current library, I imagine in some areas people have moved on but I will continue to reach for it.

> Might be less common now

That's because it's often deeply buried under more fashionable abstractions.

absolutely. i wrote this mobile allergy data thing and getting the data was mostly scraping from news websites that did terrible things with javascript to keep scrapers and ad blockers out. Beautiful Soup worked past all of that easily. Probably my favorite Python library.

I remember that I needed to do something involving performance with beautiful soup. Switching the HTML parser backend (as they mention in the docs in BS4) gave me an order of magnitude speedup...

I'm not sure why you're using the past tense, the current version is only a year old, and I will never stop using it because panning messy datasets for hidden gold is my thing.

What type of datasets?

Things with janky APIs or built on abandonware like imageboards, niche forums, dead comment communities, data dumps etc.

doing this professionally? Just curious..

Yes, I'm a subject matter person who can code rather than a coder selling into a rewarding market. Someone with better business skills than I might find it more financially rewarding.

That sounds interesting.

Same. I had to parse HTML written by eBay sellers. They were professional sellers, not webmasters.

the jquery of backend

Reply on news.ycombinator.com