C++14: Transducers | Hacker News Reader

C++14: Transducers

vitiy.info

66 pointsingve84 comments

As far as I can discern, transducers are exactly the same as iterator adaptors in Rust: you compose transformations on a stream and get back a computation that pumps one element through the entire set of transformations on each iteration, i.e. it fuses the loops together. Then you can either run this computation and collect results into a data structure of your choosing or compose different stream transformations onto the end.

The C++ implementation of this notion looks like an inscrutable, complete trainwreck, like the rest of the language.

And fortunately for us who cannot comprehend it C++17 brings in ranges.

Transducers do not require that every element that comes out of the source results in a single element going into the destination, it can just as well be 0 or many, they also work well with mutable or immutable destinations (because they're a reduce rather than a map so they can modify the destination).

Perhaps my description was unclear or assumed too much familiarity with Rust, but iterator adaptors such as .filter() and .flat_map support the 0-/many-output use cases you describe, and Vec::extend vs .collect() are a couple of mutable and immutable (resp.) ways to choose the destination of a composition of iterator adaptors.

You're right, and that description doesn't make sense. Let me see if I can explain why I said what I said:

When you call next on an iterator, you expect it to give you the next value or to tell you that there will be no more values. It's not supposed to give you no value 'for now' or many values. With a transducer it can change your destination value in any way it likes according to your final reducer function.

Perhaps it's better to say that iterators are controlled by their destination and so are used in a pull fashion while how reduction functions are used is defined by the source so can be used in push mode. It's easy to emulate pull mode with a push source - you just keep pumping values into one end until one comes out the other, but you can't emulate push with a pull destination, as you can't tell without running your adapters how many values will need to go into the pipe to produce your next value.

Imagine you have an event source, and you want values coming out of it to go into some other data structure as they arrive via the same kind of chain of transformations as you might use on a simple data structure. It'd be pretty ugly to do this with iterator adapters (probably have to have an iterator stream of futures, but then you can't do a normal map/filter, etc.), but the reduce on the event stream can pump values through as they come in, regardless of if any of them ever make it to the end of the pipe.

Kind of. Rust iterator adapters are closer to the previous Clojure reducers namespace and the Java 8 streams API. All three take a "collection" and return a new "collection" which includes additional behavior/transformations when ultimately iterated over.

Transducers separate out the transformation processes into free-standing composable functions. This makes the transformations first-class values, and makes it possible to apply such transformations to less collection-like entities such as async channels.

I'm personally still not sure if it's a good idea or not, but it is an interesting approach.

I'm not sure you're characterizing Rust iterator adaptors right.

Iterator adaptors take an iterator (which usually, but not always, comes from a collection, e.g. "my_vec.into_iter()") and give you another iterator back, without allocating an intermediate collection; the iterator itself stores only the current state of the iteration process. You can apply iterator adaptors to such iterators as the receiving end of a channel ( http://doc.rust-lang.org/std/sync/mpsc/struct.Receiver.html#... ), and you can pass around the iterator and add other computations to the end of it before you "run" the whole thing.

However, I'm not sure what you mean by the transformation processes being functions. In Rust these processes (iterators) are values, and have methods that produce a derived process by appending a step; for a concrete example:

    let fb_chars = "foobar".chars(); //fb_chars is an iterator
    let not_o = fb_chars.filter(|&x| x!='o'); //not_o is an iterator; we got it by applying the filter iterator adaptor
    for i in not_o {println!("char: {}", i);} //run the iterator with a for loop; we could also have run it by calling .collect() and obtaining a collection

What are the domain and range of the functions you mention in Clojure?

I guess Rust's iterators are pretty standard, meaning that you keep calling "next(): T" until there are no more elements, right?

Well that's pretty cool and iterators are generic, but not generic enough. If you'll look closely their protocol has certain constraints. For example the next() call is synchronous. If the next element is not ready, well, tough luck, the current thread needs to be blocked. Another constraint is that iterators produce elements, right? Well, iterators are cool and useful, but many operators like map(), filter() and flatMap() can be applied on things that don't necessarily produce values. As a side note, this is why monads are hard to understand, being a little more abstract than iterators.

What transducers from Clojure are doing is to define a more generic protocol (more generic than that of Iterator) such that you can have defined operators, like map and filter, operate on whatever you want, such that you can reuse the implementation of those operators. You can also compose those operators, before applying them to something concrete.

> Well, iterators are cool and useful, but many operators like map(), filter() and flatMap() can be applied on things that don't necessarily produce values.

You've lost me. You need to apply the function in `flatMap` to something, and `flatMap` by definition will output zero or more values.

So what do you mean?

The signature of flatMap is something like this:

    flatMap[M,A,B](cc: M[A], f: A => M[B]): M[B]

So given that you respect its signature, M[A] is not necessarily a producer of values. It can be anything and it helps if you think about it not like a collection or a container, but more like a context. For example M[A] could be a function of type S => (A, S). And then you could map and flatMap on such functions to produce other functions. This is the "state monad" btw.

On further thought, the genericism of transducers seems to hurt when you want to combine multiple streams, even somewhat trivially:

   zip_flat xs ys = zip (flatten xs) (flatten ys)

I don't see how you could implement this as a transducer, since a transducer isn't able to reason about its inputs.

I could have been clearer. By the scare-quoted "collection" I meant a value which implements the relevant collection-like interface. In Clojure this is any object `reduce` works on, in Rust is any struct implementing the `Iterator` trait etc. In all the examples discussed these transformations are lazy, only execution the composition of any number of transforms when values are requested.

Clojure transducers are different because a transducer is a function which accepts a reducing function and returns a new reducing function, adding behavior by how the input function is composed into the result function. Because the domain and range of transducers are both reducing functions, they can be chained through function composition. A chain is actually applied by transforming a reducing function then using that function to `reduce` (Rust `fold`) a collection.

The cool part is that the transformations don't refer to collections at all, not even through some highly abstract collection-like interface. This makes them applicable to other domains, like the previously-mentioned channels.

> Because the domain and range of transducers are both reducing functions, they can be chained through function composition. A chain is actually applied by transforming a reducing function then using that function to `reduce` (Rust `fold`) a collection.

For iterators:

Because the domain and range of iterator adaptors are both iterators, they can be chained through function composition. A chain is actually applied by transforming an iterator adaptor then using that iterator adaptor to "iterate" a collection.

Defining a transducer involves returing a closure wrapping some logic around calling a function of type (result, element) → result to get a function of type (result', element) → result'. All the while you're composing, I assume you don't know what this element type is?

Defining an iterator adaptor involves defining the state itself (generally you have a field for the state of the iterator you wrap, and some fields for the state of your transformation itself) and its .next() method for the Iterator impl. The .next() method has type (state, input) → (state, optional output), except that in Rust reality it mutates the state rather than returning a new one.

The difference seems to be that in Rust the output is separate from the state. You could convert an iterator adaptor into a transducer by making the optional output a field in the state.

Also, in Clojure it looks like the usage is that you compose the transducers separately from applying to an input, whereas in Rust you generally compose adaptors on top of the input iterator. I assume this is what you mean with "not referring to the collection at all". In Rust, defining the adaptor chain ahead of applying it would look like:

    let transducer = |x: Iterator| {x.map(xfrm).filter(pred).whatever()};

Which is fine, but not particularly idiomatic, and might require some work annotating types.

> makes it possible to apply such transformations to less collection-like entities such as async channels.

Like https://doc.rust-lang.org/std/sync/mpsc/struct.Receiver.html (scroll down to the trait implementations)?

Ok, bad example :-) Rust's mutable-iterator model of abstract collections supports this directly, but Clojure's functional-reduction model does not. I'll see if I can think of a better example, but it might be hard -- mutable iteration is pretty flexible (which is probably why it's what most languages do...).

That's blocking, not async. Of course you could always create a new thread, but you might want to avoid that for performance reasons...

Transducers are kind of an abstraction level above what you're describing. Transducers don't care about the medium / collection / stream. The algorithm of your transformation is decoupled from the data structure, and completely reuseable.

Clojure used to have (/is deprecating) what you're describing. It's being replaced by transducers to avoid having to implement the same functionality whenever there's a new paradigm/idea. I.e this week we're doing lazy seqs, the next CSP or streams - the transducer logic stays the same.

> It's being replaced by transducers to avoid having to implement the same functionality whenever there's a new paradigm/idea. I.e this week we're doing lazy seqs, the next CSP or streams - the transducer logic stays the same.

AFAICT, iterator adapters don't require you to reimplement the "transducer logic" any more than transducers do. You implement a transformation into an iterator (eg. IntoIterator) and from an iterator (eg. FromIterator) and use the standard transformations on the iterators.

Since a transducer is like an iterator adapter, one is comparing these to functions Iterator<A> → Iterator<B>, which can be passed around and reused as first class citizens with no more difficulty than any other object.

I'm not sure, but one difference might be that you can't give a chunk of stuff to an iterator adaptor and say "run as much of this as possible, and then suspend". Iterators are inherently pull-based, so instead you say "I want X values". I can imagine the former being more useful in some cases where the producer is deciding when computation should be performed.

Iterator adaptors such as .filter, .flat_map, etc. are all defined with no reference to/knowledge of the iterators they work on: http://doc.rust-lang.org/std/iter/trait.Iterator.html

They take a stream of values (i.e., an iterator, which is really only enough state to produce the stream, but not an actual collection of the values produced by that stream) and give you a different stream back, e.g. one that looks like the original stream with a function applied to each element or certain elements removed. Rust iterators are "lazy", and iterator adaptors maintain this property.

They also assume that they can always draw from their source iterator if they are called on to produce a value, which means that anything you write for them will not be reusable in scenarios where this isn't true.

Transducers don't have this problem.

Can you describe some situations in which this problem comes up? I'm having a hard time imagining when I would want/expect an iterator adaptor that does something useful on an empty/finished input.

It's not so much empty/finished, it's that values are produced at particular times, perhaps as the user clicks the mouse, or as messages come in from the network or over stdin.

Maybe you're writing some web page analytics software. Let's say you've got a stream of values representing the (x,y) positions of mouse clicks, you want to filter out any that are more than a predefined distance from a particular point, then remove any that occur 200ms or less than the last one, then every time you have 10 of them you want to post them to the server. How would you do this with iterator adaptors?

With transducers, the exact same filter that removes points further than a particular distance away in the event stream can be used on any source (e.g. an iterator).

TL;DR: Higher order function features that have been available in functional languages since the 1970s will soon be available in C++, with more verbose syntax, explicit type decls everywhere, and (probably) enormous error messages when you get things wrong. The "new" feature will interact badly with other language features like exceptions or RAII and in any case you won't be allowed to use them in the subset of C++ that your company permits you to use.

#define c++25 sml

I think the relevant bit is that we're seeing this without GC which makes it applicable to a whole set of problems that weren't previously relevant(which is also why I think Rust is so interesting).

Yeah, we get it that functional languages came first, but rather than looking down on C++ why not celebrate that functional constructs are making it into places where they haven't been before.

What are the problems where GC can't be used? I'm going to guess very tiny embedded systems and hard real time. What are the places where C++ is being used? Unfortunately a lot more than those, resulting in expensive, unreliable, insecure software.

'hard real time' -> multimedia and action games

Not sure what you mean by that arrow, but if you're implying those things on the right are examples of hard real-time, I disagree.

In hard real-time systems, there's generally the idea that if scheduling fails, things will break and/or property (or, worse, people) will be damaged.

Think of things like industrial high-torque motor control and avionics.

Also, hard real-time systems often don't have to be particularly fast. They are just not allowed to ever be late.

This. "The right answer, too late, is wrong."

And it doesn't matter if you got the right answer 100 times faster than needed 99 times if you are late once.

Physical property damage isn't a requirement of the definition of 'hard real time', it's just that common examples are often given that way.

If deadlines are missed in multimedia applications the product isn't meeting its stated and implied requirements.

The consequences may be different, but a television that drops perceptible video frames is just as 'defective' (from an engineering point of view) as an industrial control system which drops motor control signals.

To be fair soft-realtime systems don't interact well with GC's since random multi-millisecond pauses break everything.

There is at least work towards making controlled GC pauses a possibility.

> random multi-millisecond pauses break everything

Then either use a GC that doesn't have multi-millisecond pauses, or make sure your "plain" GC doesn't. Both approaches are used very commonly.

It is hard to have a discussion about GC's in the super abstract because object pools are better than GC's and even malloc. Without additional details about what the implementation will look like it is impossible to know the best memory allocation mechanism is.

I was just pointing out that most GC's that are general purpose have 2-3 ms pauses for collection which is a no-go for soft-realtime systems. I did not say that GC's are incompatible with soft-realtime systems.

> because object pools are better than GC's

Object pools and GCs do different things, and you can have object pools in both GCed environments as in manually-managed ones, and they'd work equally the same. But note that you need a very, very good (and far from trivial) object pool to beat a good GC in multithreaded environments.

> I was just pointing out that most GC's that are general purpose have 2-3 ms pauses for collection which is a no-go for soft-realtime systems.

Well, most free ones. If you're writing a soft-realtime system with stricter latency requirements you'll either work harder to avoid the GC or pay for a low-latency one.

> To be fair soft-realtime systems don't interact well with GC's since random multi-millisecond pauses break everything.

Like these ones?

http://www.atego.com/de/products/atego-perc/

Used for this?

http://mil-embedded.com/news-id/?21468=

Deterministic garbage collection is a completely different beast than what most people call GC. It is not a nearly magical fairy that lets you have infinite memory.

Also IIRC aviation has incredibly strict rules on freeing during flight, so this likely switches to a fixed memory model for at least part of the time.

You aren't allowed to free at all in avionics.

In Ada deallocattion requires use of the unsafe package and in SPARK dynamic allocation it isn't even possible.

However the usage of GC based systems by the military in their weapon control systems, shows that there are GC implementations that able to cover certain types of real time use.

After all a GC pause during a battle situation is probably not something one wishes for.

Any place you care about memory layout, AKA performance. O(n) doesn't matter if you cache miss all over the place. Some GC languages(like C#'s structs) have some tools for this but as soon as you work with refs you've lost control.

To me that's any device with a battery(I'm looking at you smartphones), gamedev, and as you mentioned embedded systems.

There's also another class of programs that's not related to GC. C++(and C by subset) is the only way to write truly cross-platform code for all the modern OS's. If you want to target Android, iOS and Desktop C++ is the only supported path on all those platforms with the right performance guarantees(JS doesn't cut it perf-wise on mobile, yet). That's why you see DropBox and quite a few other shops using C++.

GC and control over memory layout are two completely orthogonal issues. You can have either one, neither, or both.

I don't know that they're completely orthogonal, GC by nature controls the allocation of objects which implicitly means that the layout in memory is controlled by the GC algorithm.

For instance if you have a GC that compacts then by definition you cannot guarantee object relative layout. This object layout is critical in making sure that your traversal order is cache and pre-fetch friendly.

AFAIK struts+arrays in C# are the only construct I've seen that lets you get close to explicitly control ordering of objects. In almost every other GC'd language you end up needing to use a giant byte array(hello emscripten!) in order to make those guarantees. If you work in that manner you're throwing away 90% of what makes the language useful and paying the cost of not working with the objects in their native form.

> AFAIK struts+arrays in C# are the only construct I've seen that lets you get close to explicitly control ordering of objects.

Check Mesa/Cedar, Modula-3, Oberon, Oberon-2, Active Oberon, Component Pascal, D.

All system programming languages with GC.

I knew I should have included the caveat of being widely available on many platforms with a rich library ecosystem :).

I'm sure they fit the definition but I don't know that I'd want to use any in production aside from D. D is so similar to C++ I'd just rather use C++ instead since it's a bit more proven of a stack.

Except for D, those languages died because they were tied to OSes that didn't managed to jump into the mainstream, but they showed the way.

The problem is to have an OS vendor that would care to bring those features back.

So far I see it happening with Swift and C#, where Microsoft is slowly adding features from System C# used in project Midori.

Will they kill C++? Surely not, but they might reduce the use cases one would use it for.

For example, nowadays I don't write full stack applications in C++ any longer, I just use it for infrastructure code.

Yup, that's totally fair. I'm a big fan of a solid C++ core + wrapper language that maps to the domain I'm in. Very common in the games space to see C# for all tooling/editor with the core engine in C++ for portability/performance.

> For instance if you have a GC that compacts then by definition you cannot guarantee object relative layout. This object layout is critical in making sure that your traversal order is cache and pre-fetch friendly.

You do that in GC and non-GC environments exactly the same way: by embedding your objects in arrays.

> AFAIK struts+arrays in C# are the only construct I've seen that lets you get close to explicitly control ordering of objects.

And Java's upcoming value types.

> If you work in that manner you're throwing away 90% of what makes the language useful and paying the cost of not working with the objects in their native form.

I disagree. Even without value types, working with byte buffers in Java may still be significantly less work than C++, depending on how much of your application requires this kind of work. And the GC saves you a lot of work when you deal with concurrent data structures.

The main disadvantage (good) GCs have is their significant RAM overhead when good performance is required.

Arrays work great for uniform objects(in languages that make guarantees about contiguity which Java doesn't). However once you have disparate types(which is very common) that falls apart. What happens if you have a string associated with an object that you don't want to cache miss on? I haven't seen a language that deals with that appropriately other than C/C++.

Value types are nice in Java, when will we see them in production? We don't even have Java8 on Android yet so I probably won't be able to use them till ~4-5 years from now.

Byte arrays may be less work but they still have problems, you'll pay the int->float cast costs for one, which can be non-trivial on some platforms. Also if you want to use Strings you end up allocating a new string on each access since Java doesn't have a nice way to wrap an immutable string(take a look at FlatBuffers if you want to see this in more detail).

I think it's important to understand they we're talking about a specific set of areas that C++ excels in. There are certainly places where I think Java and other managed languages make sense but if I need to make performance guarantees I'd rather not shoe-horn in a GC'd language where proven stacks have been successful.

> What happens if you have a string associated with an object that you don't want to cache miss on?

Then you embed the array of chars in the object, just like in C++. I don't see what having a GC has to do with it. As to embedding arrays in objects in Java, I think it's planned as part of the value-type project, but no prototype yet. I know John Rose (JVM architect) said that the two features they intend to address are arrays of structs and structs of arrays (but not buffers of structs of arrays).

> you'll pay the int->float cast costs for one

HotSpot should optimize it away (not sure about byte buffer usage, though). It sees an integer read and a cast to float, and reads directly into a floating-point register.

> Java doesn't have a nice way to wrap an immutable string

You mean mutable string? Well, there's the CharSequence interface that String and the mutable StringBuilder implement, but it's true that not many standard APIs accept a CharSequence.

> I think it's important to understand they we're talking about a specific set of areas that C++ excels in.

I agree. C/C++ is unparalleled for single-threaded traversals over encoded binary data (e.g. video encoding/decoding and other signal processing) and for performance-sensitive applications running in memory constrained environments . Managed runtimes even try to compete in those domains.

In other cases, the extra performance you can squeeze out is too insignificant (and growing less significant with time) to justify the extra effort. And in some cases of growing importance (interesting multi-threading), C++ has to work really hard to even match GC-runtime environments. The real cost that is always paid is the significant RAM overhead.

And BTW: Android isn't Java. It's compatibility with Java has always been very partial, and, of course, its development is not directed by the JCP.

> AFAIK struts+arrays in C# are the only construct I've seen

or Go, especially with the value types. Now there is no automatic SoA/ AoS conversions but otherwise, you have full control over memory layout.

Go supports this, I think. I am not quite sure what is meant by object relative layout.

In Go:

A struct is laid out as you declared it, with possible compiler added padding. A struct value field (not a pointer) in another struct is laid out contiguously in memory as you declared it. An array (or a slice) of values (no pointers) are laid out contiguously in memory.

You can happily take pointers into any of these larger structures. If you had a data structure where you really wanted to control the layout, you can create a slab allocator as a big array of values, and take pointers into that array to allocate pieces of your data structure. I did this exact thing to write a red-black tree in Go.

It is true that Go's GC isn't compacting. But it will be in the future (or it is planned to be).

But, if I hadn't programmed in Go I would have agreed with your comment above.

I've used a GC language (Java) in hard-realtime systems successfully. Realtime Java (RTSJ) gives you control over which threads can trigger a GC and which operate in an arena. Even hard realtime systems have a very small core that's actually hard realtime, and real-time Java isolates it nicely.

Isn't that effectively the same thing as embedding the JVM inside C/C++ with thread execution control?

From what I've read on the RTJS spec the real-time areas you're talking about use an arena allocator which is a different memory allocation strategy than GC.

> Isn't that effectively the same thing as embedding the JVM inside C/C++ with thread execution control?

I don't see how.

> From what I've read on the RTJS spec the real-time areas you're talking about use an arena allocator which is a different memory allocation strategy than GC.

Yes, but both hard- and soft-realtime threads (that do use a GC) share the same JVM and can share some objects. Also, keep in mind that realtime JVMs guarantee scheduling latency for hard realtime threads of up to a few microseconds. If you can live with higher latencies, you can use realtime GCs.

I can think of a few more...

Operating Systems. Games. Deterministic soft and hard real-time systems (there's a reason you wouldn't be able to certify a control system for a fighter jet written in Javascript). Scientific stacks. Multi-media (do you think Renderman is written in Ruby?).

I'd even argue against using GC'd languages in desktop applications. It's getting to a point where these run-times are gobbling up a few gigs worth of pages from the OS and they still hang or crash when the OS decides to start paging them to disk. Or another process gets priority from the scheduler. The fact is while processor speeds have increased memory hasn't and that's the big bottle-neck on current hardware. Despite popular beliefs I think users do care about performance. They don't notice good performance but as soon as that beachball/whitescreen/whatever comes up and the UI is unresponsive you can bet their face is scrunching up like a walnut. Does that happen only to 1 or 2 users? 10%? We don't measure things like that enough when our happy little GC will take care of it for us.

Tuning GCs for workloads is probably as much or more work, in my limited experience, as designing your program from the bottom up around its specific data patterns. You'll get better performance from the latter approach more often than tuning a GC to your workloads after profiling. The wins in that case are incremental (we tuned our JVM after some profiling and shaved off 1.8% from our inner loop!). Where they can be dramatic with the right data structures and careful attention to locality (we saw an 800x reduction in cycles due to cutting unnecessary allocations and a 300% speed increase -- sweet!).

Even tuning your allocation strategy can net you bigger gains than trying to structure your code around a JIT.

All that being said -- GC's are incredibly useful in many situations. You just have to measure and be thoughtful. Memory is still an issue GC or not.

> Tuning GCs for workloads is probably as much or more work, in my limited experience, as designing your program from the bottom up around its specific data patterns.

If you are coding the full application yourself, probably.

If doing distributed development with 20+ team members from multiple contractors of various skill levels, which is my world, I have my doubts.

> Yeah, we get it that functional languages came first, but rather than looking down on C++ why not celebrate that functional constructs are making it into places where they haven't been before.

Because that's a meaningless thing to celebrate? Having functional concepts available in a poorly integrated way, with super ugly syntax is more depressing than anything, because seeing it in use is a reminder of how much cleaner it would be in a language like Haskell or OCaml.

Why not accept that C++ isn't a functional language, and realize that many die-hard C++ users won't use these features for the same reasons they're not already using functional languages.

I'd rather see these efforts directed towards improving the compilers and libraries for languages like Haskell, OCaml, etc.

> Having functional concepts available in a poorly integrated way, with super ugly syntax is more depressing than anything, because seeing it in use is a reminder of how much cleaner it would be in a language like Haskell or OCaml.

I swear I remember fans of older OO languages saying the almost exactly the same thing with "object oriented" in place of "functional" about the way C++ grafted OO concepts onto C when C++ was new -- and, IMO, with good justification. Still, C++ was not exactly unsuccessful with that, either.

I guess that's why so much of the software you use every day is written in Haskell and OCaml.

> Why not accept that C++ isn't a functional language, and realize that many die-hard C++ users won't use these features

I'm a die-hard C++ user, and have come to love functional programming for a very C++y reason: improved performance that comes from lock-free parallelizability (if you never modify values, you don't have to synchronize access).

But I also love the simplicity that comes from not having to think so deeply about lifetimes and which portions of code can see which objects in which states. The abstraction of data-structure from algorithm is nice, the terseness is nice, the regularity is nice, but it's the lifetime and visibility simplification that helps me the most.

I don't know Haskell or OCaml. I wouldn't know a monad if it slapped me in the face. But functional C++ makes me happy

pony might make you happy too. here's a great presentation on it: https://cdn.rawgit.com/darach/my_little_pony/master/my-littl...

What's worse is that we're using "modern" programming languages for which these things aren't even on the horizon.

It's so far from the mainstream norm to implement these ancient time-savers that you can't help being excited about Swift, even though such a language should be mundane by now.

>The "new" feature will interact badly with other language features like exceptions or RAII

How so?

Sure, but it's faster! You won't need a LISP machine for this baby to fly!

You forgot amazingly fast.

I have been doing C++ for almost 2 decades and at a particular stage I was able to say that I knew the language with a self-proclaimed score of "8 out of 10". Nowadays, I would give myself a "4.5".

The crazy thing is that, to me, some new features of the language feel like they are being pushed because [x] language has it more than the actual need to solve a particular problem that was to difficult to solve using only the constructs available previously.

Two nights ago I was at a C++ Meetup and this guy was explaining how to solve this problem using a crazy-complicated pattern with a new proposed C++ feature. Eventually I had to stop him and ask him why we could not simply do this with a priority heap, a threadpool and some semaphores (simple shit). "Oh yeah, you could do that, too, but this way is cooler".

The same thing is happening all over the industry. Things are growing organically in all kinds of directions, new technologies appearing at a rate and order of magnitude higher that anything that was happening 10 years ago. It is making it much harder to say current and the net benefit (so far) appears to be negative (I still can't believe that node.js made this terrible untyped language take over the world). I am hoping something good comes out of this, but it is certainly not the nice and cozy industry we knew and loved 10-15-20 years ago. Oh well, it too will pass.

Practical javascript can now be typed very easily (TypeScript or Facebook's flow). Any "serious" JS shop is going to be using typed-JS in the next 12 months.

The problem is right there in your response. Do I do TypeScript or Facebook's flow (whatever that is, I don't know)? Or perhaps CoffeeScript? Or wait, that one is not typed. Or is it? Lots of people like it though! Except the ones that perfer EcmaScript 2015. Because it is the future. At least for the next 2 weeks when 2016 starts. But EcmaScript 2015 is not typed either. So much for your promise of all serious JS shops doing typed JS in next 12 months. Also, yeah, great, "serious" shops will be typed. What about "not serious" shops? They don't count? And their users who have to suffer through crappy products built using incomplete tools don't count? Oh, and you know what all these half a dozen most popular and legion of less popular solutions to the problem of "JavaScript sucks" have in common? They add yet another step into the toolchain making the developer's life so much harder. It is nobody's fault that we are were we are, but please, let's call it as it is and not pretend that this pathetic situation is somehow progress or even acceptable.

The net benefit and multiple direction things are what makes it particularly frustrating. I wouldn't mind as much a new technology/technique/framework/etc if it were a clear, direct, step and meaningfully better than what came before. It's especially hard to be considering what technology/stack to recommend as being easy to develop in and maintain for LOB enterprise developers who will be maintaining an app (without rewrites) for ten years or so.

Right now we're using Knockout but it's feeling a bit grumpy in various respects so I've been looking into React. But still it is common to have breaking changes, and then there's the whole ES5/ES6/TS split.

I need something that our devs can do a Google search "How do I do <X> in <Y>" and nearly copy-paste the result. Knockout has been particularly good for that because it's evolved very slowly and every example for it is in ES5.

Microsoft has wedded themselves to Angular for ASP.NET 5, but it looks like it's Angular 1. Meanwhile Angular 2 looks much better but again splits across ES5, ES6, TS, and Dart... If someone searches "How do I do <X> in Angular?" who knows what combination of assumptions might be unlying that answer.

As much as I'd prefer to use something like React, I think shops with the combination of lower skilled staff and longer term maintenance projects (like ours) will need to wait until, hopefully, the industry calms down and settles on a set of new standards.

Microsoft seems to have gotten sucked into the "do everything a million different ways" spaz attack with ASP.NET 5, but meanwhile Apple seems to be doing it right with a narrow, opinionated set of technologies with a well understood evolutionary path and specific tools for specific jobs.

We've been seeing similar attempts recently, all attempting to fill a long overdue gap: the composability of algorithms in the standard library. At this point, there exists a TS [1] for ranges [2] along with an implementation [3]. It will be a thin layer on top of the iterator-based standard library, sometimes dubbed "STL2."

On the one hand, it's great to new attempts to compose algorithms mushrooming, but on the other hand it would be great to see more aligned efforts to collaboratively push forward the TS and its implementation.

[1] A technical specification (TS) can be thought of as a self-contained mini standard that vendors can implement, but do not have to in order to be "standard-compliant". A is a promising candidate for future inclusion into the standard, however.

[2] https://ericniebler.github.io/std/wg21/D4128.html

[3] https://github.com/ericniebler/range-v3

Are ranges essentially the IEnumerable<T> from c# for c++?

Basically yes.

Benchmark showing that the composition is exactly the same in a for loop as unrolled functions would be nice.

The proposed Ranges TS has some nice benchmarks: https://ericniebler.github.io/std/wg21/D4128.html.

See Appendix 1 and 4.

(It's not exactly what you're asking for, but showcases that such claims often hold true in practice.)

To be honest, if I'd make a language, I would make something like C++, break backward compatibility with C, but just much simpler and more beginner friendly.

No templates, but standard maps, sets, arrays. Both compiled and interpreted. No virtual functions. Clean, simple syntax close to C, but with all the nicer stuff C++ brought along, like lambdas.

C++ is industrial-strong, but sometimes it seems it's not taught because it's too expert, in favor of java or C# while C/C++ is the language that should be taught.

I don't understand languages like rust and go which introduce a non trivial new syntax. Why not keep the C flavor ? I don't see how people are going to adopt languages like that. There should be a statically compiled language which has the same philosophy of python.

> I don't understand languages like rust and go which introduce a non trivial new syntax. Why not keep the C flavor ? I don't see how people are going to adopt languages like that. There should be a statically compiled language which has the same philosophy of python.

I don't understand what having "the same philosophy of Python" has to do with not keeping "the C flavor". The delta between Python's syntax and C's syntax is basically equal to the delta between Go's syntax and C's syntax, or Rust's syntax and C's syntax.

I was talking about 2 different things. What is good about C is its simplicity. Python is easy to learn too, and it has nice language facilities and syntax sugar.

About the deltas, maybe you're right, but I disagree about the difficulty of the languages. For example go function prototypes seem pretty heavy to fully grasp and understand. Rust also have expert programming facilities and paradigms.

What would you have for generic programming instead of templates? How would std::vector be implemented?

It would be part of the language standard. Vector is part of STL.

this is revolting code. period.

it does nothing special or novel whatsoever - except maybe to set a quality bar for writing unmaintainable code.

the very first example given is simple, but demonstrates a thorough lack of understanding for what constitutes readable code. if i had to code review this from a junior i wouldn't let it go in without significant changes, and i'd make time to teach them some of the most basic concepts about producing maintainable and readable code... but only because its easier than firing them.

i am skeptical that the label 'transducer' has any solid value whatsoever here. i can understand the concept... its basically to encapsulate stages of algorithms neatly. the fact that it manifests one way in Clojure says nothing about the value of using exactly the same mechanism in C++ to achieve the same...

normal classes, normal functions, and not being rubbish at writing code combine to give a much better result for the same class of problems.

I tried implementing transducers in MSVC about a year ago. I got pretty far in implement a lot of the basic set of transducers one would use in development. However, I ultimately abandoned it because they weren't really a great fit with C++. You could make it all work, but the meta template code is extremely difficult to reason about, and MSVC specifically had a lot of issues processing the code (pretty quickly bumped up against MSVC's type name size restriction!). There are also a lot of issues to think about regarding lifetime semantics - figuring out the right behavior was pretty difficult, and debugging issues with it pretty frustrating, especially when the code was optimized.

The pros were that, at least for my limited testing, performance was essentially equivalent, for optimized code, between the transducer implementation and the equivalent hand rolled code (e.g., doing a map+filter transducer vs. a foreach with an if), and that using transducers offered further optimizations if you applied them completely up and down the stack, by allowing you to, say, early terminate a sequence that would otherwise have been done in entirety behind some opaque function call. I don't really consider code readability to be a benefit here since transducers are, somewhat inherently, a heady subject.

You can see my work here: https://github.com/AndrewGaspar/transducers.hpp

I haven't done a formal write-up or anything on it, but check out the tests if you want to see what consuming them would kind of look like.

EDIT: One of the things I'm proud of was rigging it, without need of dynamic typing, such that a transducer could produce heterogeneous types. Obviously your sink would need to be capable of accepting all these different types being produced by the transducible process, but it allowed for some interesting integration with <iostream> - say, creating a transducer that inserts comma's between integers for output (see https://github.com/AndrewGaspar/transducers.hpp/blob/master/... and https://github.com/AndrewGaspar/transducers.hpp/blob/master/...).

EDIT 2: I notice that early termination was not shown in this blog post. I implemented this by essentially introducing a known wrapping type (https://github.com/AndrewGaspar/transducers.hpp/blob/master/..., sorry I know it's a bad name). The trick with this was I only wanted the returned values of a transducible process to be wrapped by this struct (since it has the overhead of one bool value, and a check to actually determine if it could early terminate) if a single transducer in the chain can early terminate. It also made each individual transducer slightly more complicated to implement, unfortunately.

I think the best part of 'modern c++' is the fact that you don't have to use any of it (I'll make a concession for move constructors, that just makes sense).

Reply on news.ycombinator.com