Hacker News Reader

Comment by white-flame | Hacker News Reader

Much of his thrust seems to boil down to being really offended by the cost of safety of allowing concurrent writes while reads are going on.

When dealing with analysis workloads, his line of thinking can yield fast runtimes and doesn't bring that much of a programmer burden. However, it's killingly complex when it comes to live mutating business databases that need to scale. Even many job-based analysis workloads have a lot of concurrent mutation in trying to efficiently figure out what's been done and what needs to be done next when the actual workflow is not a straight pipeline.

I do agree that 'for' loops should be replaced by broader declarations of what needs to be done, instead of these explicit instructions on how to iterate through the data.

View on HN

I take the author's viewpoint as: when the data is actually big, big data operations are for all practical purposes going to be run in batch mode - it's just hard to squeeze enough new data through the pipeline in ten minutes to change the statistical analysis of many terabytes of big data...if the data is so unpredictable that a snapshot isn't going to be meaningful then slowing operations down to allow concurrent writes isn't going to provide better analysis and what is really needed is not big data but some sort of near real-time stream processing.

All this comes with the assumption that much of what people consider big data isn't really big data.

Could you expand on what you mean about for loops?

In the functional world, there are all sorts of precise constructs to operate on lists (maps, folds, unfolds, scans, etc.), and many versions of these that support concurrency. That's what I imagine when you say "declarations of what needs to be done".

The standard "for" structure is necessarily ordered. Either an index is incremented/decremented, or an iterator object's next() comes in. Since there are side effects that can affect either the iterator or the index, and dependent side effects between loop body runs, obeying a "for" loop means strictly ordered (unless the iterator interface abstracts something else), serialized execution of the loop body.

Even maps, folds, etc are often defined in terms of what order they process data. This is useful for early exit, and mutable state dependency between iterations. But again, this demands serialized, specifically ordered processing of elements.

Lists themselves are ordered structures. Their generalized interface does not assume it's just a bag of items easily thrown around in sub-parts, especially if the linked list nodes are to be modified.

Of course, the above applies to impure functional languages. Purely functional of course has far more freedoms, but even the Lisps and such ultimately demand sequential processing for most of their fundamental list operations.

Much of the functional world is concerned with defining structures that are heavily linked to themselves. Tagged data is essentially the norm.

So, it isn't just that maps are defined in terms of car/cdr, it is that the list itself is defined in terms of car/cdr. The "aha" moment is when you can see that an array of items in c can efficiently represent a tree structure where the parent/child relationships are implied, not tagged. The performance gains from this are impressive.

Functional languages have arrays.

Pssst, let the developers that don't open their eyes beyond C and C++ think their languages are the only ones with such features. :)

Every time I see a C or C++ talk discussing value types or arrays, I get to think those guys never saw other programming languages, e.g. Algol/Wirth family.

Another example of the Lady Gaga culture, instead of learning about the world of programming languages.

I did not mean to claim that they do not. My claim is that "much of the functional..." So, referring to things like cons lists is easy, though that one is old. Look at the multitude of pointers in any of the purely functional data structures, though. There is a lot of overhead in building up these structures.

Look into the Scala (and clojure?) vector sometime. Literally a 32-ary tree. Makes the algorithm somewhat pleasant to look at, but are we really shocked when something using a simple array can beat it in performance?

Now, it is a perfectly fair and valid point that optimizing development time may be preferred. We are allowed contradictary valid points in situations that are trade off defined.

I think that no-one will argue that an imperative, transient data-structure will be faster than a functional, persistent data-structure if you use it imperatively.

If you use persistent data-structures the way they are meant to be used, they can and will be faster than a simple array.

Both have valid use-cases, but they are often distinct.

You would be surprised at the arguments you see.

Even the one you are making is... tough. Persistent data-structures are not a sure fire win against mutable versions. Again I find myself refering to the DLX algorithm. The entire point of which is that it is easier and faster to permute through different configurations of a data structure then it is to have all instances instantiated in a persistent fashion.

Does this mean that persistent data structures are without use? Of course not! Again, you may not be optimizing speed of execution. Which is perfectly valid!!

Reply on news.ycombinator.com