Hacker News Reader

Comment by ekidd | Hacker News Reader

Shortly before Linus wrote this article in 2002, I wrote an XML-RPC library in C that used reference counting. By the time I was done, I'd written 7,000+ lines of extremely paranoid C code, and probably eliminated all the memory leaks. The project cost my client ~$5K.

The standard Python xmlrpc library was less than 800 lines of code, and it was probably written in a day or two.

Was my library about 50 times faster? Sure, I could parse 1,500+ XML-RPC requests/second. Did anybody actually benfit from this speed? Probably not.

But the real problem is even bigger: Virtually every reference-counting codebase I've ever seen was full of bugs and memory leaks, especially in the error-handling code. I don't think more than 5% of programmers are disciplined enough to get it right.

If I'm paying for the code, I'll prefer GC almost every time. I value correctness and low costs, and only worry about performance when there's a clear business need.

View on HN

I Guess Linus also made another suggestion which he didn't pursue on this paragraph:

Generational garabage collectors tend to never re-use hot objects, and often do the copying between generations making things even worse on the cache

The suggestion? Just reuse hot objects.

Note, this post is from 2002. Garbage collectors have improved a lot over the last 9 years. Don't evaluate new technology with such old arguments.

Linus was quite specific about his problems with GCs. Just saying that "garbage collectors have improved" without specifics, and calling his arguments "old" doesn't add much information.

Can you tell us how they have improved in ways that address Linus's specific concerns? And explain how a generational garbage collector can effectively re-use freshly freed objects before they leave the cache? Which 2011 generational GCs take advantage of that idea?

Reusing hot memory is a pretty ancient GC technique, though the details have varied over the decades.

You can Google "nursery generation" for some examples:

http://www.google.com/search?q=gc+nursery+generation

The original theory was that most objects in a functional language are extremely short-lived, and you want to reclaim them quickly. But if you GC your nursery generation after every few kB of allocations, you'll also keep the memory in L1 and L2 cache.

The ideal size of a nursery generation (and the value of reusing hot memory) should be measured empirically.

Of course, most popular scripting language GCs are non-copying, reference-counting collectors, which means they ignore techniques that were well-understood back in the 80s. However, this has not been a barrier to the adoption of Python, Ruby, etc., because the business value of high-performance code is often minimal.

You can bypass the GC entirely, and keep the objects that really should be in cache live, and just reuse them. Not a pretty solution, but it's doable.

Yes I can:

- [1, 2] The pauseless collector also has the highly desirable side-benefit of producing improved memory locality. This happens because the algorithm works by attempting to relocate groups of tightly-coupled objects into regions of adjacent memory, providing excellent paging and cache locality properties for those objects. This can also benefit highly multithreaded applications that operate on distinct sets of object data.

If you read the whitepaper on Azul[3] you will get a glimpse of how things got a lot more involved. And all of this gets a lot more complicated when programmming for multiple cores[4].

But not everybody is using the Erlang VM or Sun's JVM (HotSpot). The point is generic: don't assume things are on your platform by watching other platforms. Don't use old data to guide your search, no matter how brilliant/famous the author is.

[1]http://java.sun.com/products/hotspot/docs/whitepaper/Java_Ho...

[2]http://java.sun.com/docs/books/performance/1st_edition/html/...

[3]http://www.infoq.com/articles/azul_gc_in_detail

[4]http://www.javacodegeeks.com/2011/04/erlang-vs-java-memory-a...

Linus suggested in the OP that automated reference counting (where the language implementation handles reference counting for the programmer) is preferable to GC.

Python uses such reference counting (although it also has a GC fallback to ensure cyclic structures are collected).

And it is one of the reason why python is slow and difficult to scale on multiple cores (the main difficulty by far of removing the GIL is reference counting).

Scaling Python on multiple cores is easy: you just run multiple Python processes, each with its own per-core GIL.

You are correct, though, that ref-counting overhead is one of the main reasons Python is slow.

Thank you for making this point. To my great dismay I see more and more people think that you need concurrent threads in one process in order to scale. It seems as if many programmers believe this as scripture and subsequently have difficulty thinking about distributed systems (not just multi-core, but multi-host).

The drawback with multiple processes is that they each have all the compiled bytecode on their heap. When I "import nltk" my heap goes from ~12 to ~36 MB. Add a few more dependencies and you end up wasting a non-trivial amount of RAM on python heaps.

You could do the import and then fork the child processes. That should share all the memory used by the nitk module between all the children (for as long as the memory is not modified by anyone, which triggers a copy-on-write allocation of the affected pages).

most forks do copy-on-write and every access to a python object meddles with reference count, which is - you guessed it - a write.

The context of the copy-on-write win was the bytecode for modules. I don't know that you'd have anybody meddling with the reference counts for that...but I haven't really looked.

Bytecode is stored in function objects which IIRC are reference-counted.

Aside from ad-hoc classes (defined within the scope of a function, say), does it ever make sense to garbage-collect a module or a class?

Say you do "import smtplib" in your main file. Now that module is imported -- forever. I don't know the internals of Python well enough, but I bet that the module reference has strong references to its contents, so that even if nobody is actually calling anything in smtplib, it will be there in case someone does. The same should be true about modules importing other modules; they stay visible at the scope-level, so they are permanently loaded.

So for those cases it would make sense to keep them separate from global garbage collection. I'm pretty sure that the method tables of all the classes in the system take up considerable space, probably in the order of megabytes for many apps.

That's a valid point, but you can do things to mitigate this, like splitting up your program into processes that have different dependencies. For instance, if you have a UI process and a core logic process, you won't need to import your giant UI library twice.

If you split the code horizontally instead of vertically you have to deal with IPC, which could be problematic (and slow). I would do that only if there was already a clearly defined interface between the modules, not just for the sake of concurrency. Moreover, taking into account growing number of cores in modern machines such strategy doesn't seem very future-proof.

Or you could just use a language whose implementation doesn't blow.

And suffer from 5x slower development time. Yeah your deadline now has to be extended from 2012 to 2015 but at least the language is FAST and is technically excellent, right?

False dichotomy...

This is a bit like saying, "Of course C has full type safety. Just use a Haskell implementation written in C." Both your statement and his are technically accurate, but yours is on a subtly different topic.

How is it a bit like that? He complained that Python didn't scale to lots of cores because of the GIL; I said his Python didn't scale because of threading, which interacts badly with the GIL. If you use a shared-nothing approach to parallelizing your Python code, you don't run into that problem. It's not as if shared-state threading happens magically — you have to rewrite your code to use it, too.

Python doesn't do code analysis to determine when it's effectively doing obj.refs++; obj.refs-- repeatedly.

This sort of analysis is useful, and if the interpreter had been designed to do any optimization along with jitting, would probably come nearly for free. Reference counting could be far far cheaper than it is in python. (How much cheaper? I don't know - it'd need work to figure it out)

Recently, there has been some work on removing redundant reference count operations in the Python interpreter. The following paper describes how it can be done: http://portal.acm.org/citation.cfm?id=1869631.1869633.

Regarding the performance impact of reference counting, the following facts are important:

- Switching from immediate reference counting to deferred reference counting (L.P. Deutsch and D.G. Bobrow, 1976 [1]) eliminates about 90pct of all reference count operations in Smalltalk (Berkeley Smalltalk '82, that is) [2]

- A very good account of reference counting can be found in either Dave Ungar's excellent PhD thesis [3] and Dave Ungar and Dave Patterson's in-depth analysis of Smalltalk performance [4].

[1] An efficient, incremental, automatic garbage collector (http://www.cs.umass.edu/~emery/classes/cmpsci691s-fall2004/p...)

[2] High performance storage reclamation in an object-based memory system (http://techreports.lib.berkeley.edu/accessPages/CSD-84-167.h...)

[3] The Design and Evaluation of A High Performance Smalltalk System (http://www.eecs.berkeley.edu/Pubs/TechRpts/1986/5376.html)

[4] Berkeley Smalltalk: Who knows where the time goes? (Chapter 11 of http://www.iam.unibe.ch/~ducasse/FreeBooks/BitsOfHistory/)

This sort of analysis does not come easily. Say you have:

    def bar():
        return some_constructor()

    def foo():
        b = bar()

Here the decrement is in bar() and the increment is in foo(). You have no way to elide the operation without doing inter-procedural analysis, which is hard.

You don't need to catch all the cases to see a big improvement. Just the most common ones.

The problem is in a language like Python, where everything is broken up into little functions, you're not going to catch the most common uses without doing inter-procedural analysis.

is the ref counting slow because it has to be a protected operation? or is it just the amount of them. if it is due to being protected by locks there are lock free ref counting systems (as described in here: http://www.google.com/url?sa=t&source=web&cd=1&v...) that essentially allow batch refcounting opterations to accumulate in a per thread counter that are only reconciled occasionally. assuming you design your data structures correctly (no false sharing) you can get very high write performance because all your counters are cached and there is no cache invalidation due to competing writes on other processors. of course it takes more memory so it is only suitable for highly shared objects, but it can make a significant difference in a reference counting system. (the article points to scalability issues in the linux ref counting system and how to resolve them)

That's awesome! I knew there were faster approaches to ref-counting, but I didn't know about that one, "sloppy counters".

The most common approach to speeding up ref-counting is to ref-count bigger objects — modules rather than individual variables, say.

The simplest way to speed up ref-counting transparently is to statically analyze the code and remove redundant increment and decrement operations. This can be tricky in practice, and I haven't heard of anyone actually doing it.

It seems to me that the best way to do this would be to put the analysis into a tracing JIT. since the compiler knows exactly what will happen in a long code sequence removing redundant incs/decs would be fairly trivial.

Just for the record: Pramod Joisha did static analysis of redundant reference counting operations in 2006 in the Bartok C# research compiler (http://www.hpl.hp.com/personal/Pramod_Joisha/Publications/is...). IIRC, doing it improves performance significantly.

The problem with most methods to improve ref counting speed is that it generally breaks existing C extensions. For that reason alone, I don't expect cpython to significantly change its way of doing things in that area for a long time.

You are playing with words here: of course you can run multiple python instances to scale your application on multiple-cores, that's a trivial statement. But I was talking about python the interpreter (more exactly cpython). There are legitimate cases where multi-threading is the natural, elegant solution, and cpython, mostly because of reference counting, prevents that.

CPython prevents you from using shared-state multithreading to scale your application on multiple cores, but it doesn't prevent you from scaling your application on multiple cores. That's not "playing with words".

It is indeed unfortunate when the limitations of our platforms force us to contort our code to improve performance, but that is just as true of multithreading as of multi-process programming. The difference between the complexity of the two is small.

C and Java apps with high contention don't scale well onto multiple cores, either.

The key to speed is to not share state, which Python can do fine. It's called fork.

Sometimes you cannot easily not share state. The fact that you can use multiple processes to scale is not an interesting statement when comparing python to other languages because it is true for every language out there.

I think the GIL has made people in the python community too defensive: the GIL does not prevent from building scalable architectures in many cases, but it still sucks, and it would be better without. That's a limitation (and a tradeoff because it made development and integration with C easier). And there are scalable architectures based on threads (example: http://www.mailinator.com/tymaPaulMultithreaded.pdf) - "thread suck" has became a meme which slightly bothers me in general. Not a panacea, but a good solution when applicable.

I thought the key was not managing mutable state. Sharing lots of immutable data can probably shave a lot of performance over ipc.

~$5k for 7 KLOC of bug free C code is a steal, that's impossible to do in less than a couple of months

It's nearly impossible to do in any timeframe. I can think of perhaps two examples in human history where I think it's been done: qmail and seL4. And there may still be bugs in qmail. There may be a few other non-public projects that have achieved less than one bug per 7000 lines of C, but probably not more than one or two.

http://www.dt.e-technik.uni-dortmund.de/~ma/qmail-bugs

There are also some DNS-related bugs that are not on this list.

Thanks! I hadn't seen those, although I knew of 1.4. I think 1.1, 1.3, and 1.4 are actual bugs, if the reports are accurate; I'm pretty sure Dan disagreed with Wietse about 1.1. Three bugs in about 15000 lines of code doesn't quite rise to the level of less than one bug per 7000 lines of code, so maybe that's only been done once, in seL4.

What are the DNS-related bugs?

Beg to differ on both points.

Ignoring the 7 KLOC metric (which is fairly useless in estimating the project value for the customer), $5K at $100/h rate of a senior C programmer on a contract works out to about 6 working days, which is a reasonable timeframe for writing well optimized XML-RPC protocol parser. In other words it's neither a steal nor does it take two months to write.

(edit) YMMV by a programmer, but that's what it would've taken me to write it (though I have written a good deal of protocol parsers).

1+KLOCs a day of C? WTF?

It does not need to be 1 KLOC/day :) The complexity of XML-RPC parser is about a week of work for polished code. Go from there, not from the given 7 KLOC of some implementation.

Compared to other languages, C is pretty straightforward. You also have to write more of it to do less. I switched back to C after 12 years of using web scripting languages, and am surprised at how much more code I churn out per day.

Do you now use C for web apps? Like parsing GET, POST, DB, Cookies?

Please don't take this personally, but the fact that your code was N thousand lines doesn't mean the task is inherently an N-thousand line task.

Also, undertaking anything with XML in C is brave, kudos.

If you know of any way to make the code shorter and more concise, please let me know. I'm always looking for ways to improve.

https://gist.github.com/937200 (two sample files, totaling slightly less than half the project)

Posting 10-year-old code to HN is always a little embarrassing, but please feel free to be brutal. I'll live. :-) I was definitely way too in love with the preprocessor back then.

Here's the corresponding Python code, from shortly after it hit 1,000 lines in late 2001:

http://hg.python.org/cpython-fullhistory/file/7b2d701ec404/L...

I like the Python code rather better, if only because there's so much less explicit memory management. The actual ideas stand out clearly, and there's just so much less code to read. In most cases, I'd happily sacrifice the performance.

Was my library about 50 times faster? Sure, I could parse 1,500+ XML-RPC requests/second. Did anybody actually benfit from this speed? Probably not.

Then your (client's) problem wasn't reference counting but premature optimization.

Are there situations where you'd like to have code run fifty times faster than native Python. You bet there are, lots and lots of them - for example, in a Unix-clone Kernel. Sorry if somehow you didn't find one of them.

Hopefully you used Valgrind and Formal language specification to reduce the work required.

And to avoid premature optimization, use gprof to find the bottleneck(s) rather than just diving into what seems to need optimization.

You're mixing up 'premature optimisation' and 'unnecessary optimisation'. The first is making code faster that isn't the bottleneck, or dominating performance factor. The second is making something faster than it needs to be. Profiling helps avoid the first, benchmarking helps avoid the second. Both require working (toy) systems, which is hard when you are evaluating which language to begin work in.

Writing a library in C (for reasons of performance) where performance would have been 'good enough' in Python is unnecessary optimisation. Writing your own GC implementation layer in Python because you think that's the bottleneck would be premature optimisation.

EDIT: typo

Seems like a rather small distinction. Unnecessary optimization is a natural result of premature optimization so I tended to assume that was what was happening. Unnecessary optimization can result from other things as well of course.

Regardless of this distinction, I think the same conclusions follow; the original problem had little to do with reference counting as such.

Actually, geting reference counting right and mostly bug-free is much easier, if you use C++ and do the counting via objects on stack (RAII).

Absolutely not, http://std.dkuug.dk/jtc1/sc22/wg21/docs/papers/2003/n1450.ht... explains why in section D. Implementation difficulty

Actually, yes. The boost::smart_ptr<...> is highly effective. Prior to that existing, I wrote a similar ref-counting library that has been deployed to hundreds of remote sites, running high-reliability industrial control code.

While complex, the ref-counting implementation gave me something that a GC system just can't deliver:

Determinism.

This system is running on a 300MHz embedded x86 hosts, in a solid-state industrial computer, and has to provide sub-millisecond timing with only occasional hiccups >1ms acceptable. I am unaware of a GC system that could deliver this.

These boxes run for years, with no memory leaks. The cost was, of course, ensuring that no "loops" of objects get created. This was a small price to pay for the benefits. Having written large systems using both GC and ref-counting, I still lean toward ref-counting, even if determinism isn't a strong requirement.

Of course, I'm one of those guys who loves C++, so take my opinions with a grain of salt...

I, too, love C++ and the boost smart pointer library.

In fact, it's been adopted into modern versions of the C++ standard itself. std::tr1::shared_ptr and std::shared_ptr.

IBM's Metronome[1] advertises maximum pause times in the hundreds of microseconds range. Cliff Click[2] cites a 10MB max heap for it so definitely not a cure-all but definitely something I'd like to play around with.

[1]http://domino.watson.ibm.com/comm/research_projects.nsf/page... [2]http://www.youtube.com/watch?v=uL2D3qzHtqY

Reference counting is not deterministic. Whenever a refcount is lowered to 0, you don't know how many subsequent references exist further up the chain that will also be lowered to 0. Lowering a refcount to 0 can result in an arbitrary amount of code to be executed depending on the references.

If you mean that reference counting, on average, tends to result in fewer amounts of random/unpredictable pauses than most GCs, then I agree.

I guess on that scale and probability, no code is deterministic.

reference counting is slower* then other forms of gc. My guess why its fairly fast in c++ has a lot to do whith how much you can avoid it by allocating things on the stack(of course you can do escape anaylsis but most gc'd languages don't have explicit stack allocation of non basic types)

*with a lot of collaries

You are probably right, but your experience does not disprove the central point that GC always slows things down. Now this slow down may be worth it in most situations, in fact it may be worth it in 99% of situations, but it is a fact.

In some cases that 50 times increase in speed may be important. In some cases, if something is 50 times slower it would be simply unusable. Linus happens to be working on one of those special cases, so he is quite correct in pointing out that garbage collection will always slow things down.

Context is always important. And there's no such thing as blanket advice that applies to every circumstance. Software development is a hugely varied endeavor, ranging over more orders of magnitude and more variety of application than almost any other industry. It stands to reason that most practices will vary rather considerably depending on context.

Linus writes from a systems perspective. He cares about every nanosecond and every byte. And that's appropriate and even laudable in the context of the linux kernel and core applications. But it's not always appropriate for every situation.

I've used your brilliant XML-RPC in one of my consulting assignments. Thanks.

Agreed.

Reply on news.ycombinator.com