Ode to a Streaming ByteString | Hacker News Reader

Ode to a Streaming ByteString

blog.sumtypeofway.com

55 pointsjsnell18 comments

Having a bunch of string types is something people like to rag on Haskell for, but I think the reality is that strings are that complicated and most languages kind of sweep the complication under the rug, while Haskell chooses to accept it. Thankfully the practical downside is quite manageable; the most annoying thing you’ll occasionally have to do is call toStrict/fromStrict across library boundaries.

Using [Char] as String was mistake. If Haskell used Data.Text everywhere strings would be fine

If you mean a Text-like representation (basically ropes of codepoints) then sure, but Data.Text has a bunch of i/o including lazy i/o stirred in with it. Lazy i/o in particular is part of the still-lingering schizophrenia in Haskell about how to do i/o cleanly and safely. The current Streaming library is yet another try at that. I hadn't seen it before and it looks interesting, though I'll have to examine it more closely, and it still seems to be Iteratees deep down.

I had the impression a few years ago that Pipes was the most cleanly designed of these schemes, but haven't been keeping track.

Streaming is definitely the most cleanly designed today and also in my experience the fastest. It’s a shockingly simple design. Main downside is that it’s unidirectional, but I don’t think anyone really cares. Bidirectional pipes are tough to use.

Thanks for this recommendation. I'll be sure to check it out in that case.

True - going from Text to [Char] costs far less than the other direction all told.

But [Char] is quite nice for certain uses.

Minor nit:

>ByteString represents a byte buffer and its associated length; in this it is similar to Go’s []byte or Rust’s &[u8] (though it has one fewer datum to track, as the Rust and Go types offer mutable access to the associated byte buffer and thus must keep track of its total capacity).

Rust's &[u8] and &mut [u8] don't track capacity. Mutable access only allows changing the data in the slice, not changing its length. It couldn't even if it wanted to, since it has no idea of what the underlying storage is. It's only Go's slices that do double-duty as a ranged view into an existing allocation but also with the freedom to become the owner of a new allocation.

How does rust do bounds checking for slice accesses if the length is not stored?

Length and capacity are two different things; a slice is a pointer and a length, but a Vec<T> (and Go's slices) are a pointer, a length, and a capacity.

That Rust and Go both have "slices" that are slightly different is, unfortunate, but that's just how it goes sometimes.

I once wrote a brainfuck interpreter in Haskell which executes even incomplete programs as far as possible. It was really easy thanks to streaming ByteStrings and lazyness in general.

The article only seems to concern handling effectful input. But i can't really imagine what the type for writing to stdout would look like.

Would it be something simple as:

writeStdout :: ByteStream IO a -> IO a

Maybe I misunderstand what you mean, but why not just:

    writeStdout :: ByteStream a -> IO ()

Maybe you could even leave out the `a` type since arguably a `ByteStream` should only contain bytes anyway. When writing to stdout you don't really need a return value usually, so that could just be (). The IO monad can encapsulate any amount of side effects in the same function, so you could fit your additional side effects in besides writing to stdout.

Sure, that could work, or something like

mapM_ :: (ByteString -> m a) -> ByteStream m b -> m b

And then you just have

writeStreamToStdout = mapM_ writeBytesToStdout

I find all those streaming I/O libraries (and lazy bytestreams as well) to be more neat-looking than useful: they can produce slightly cleaner code when used for basic tasks (such as those in examples), are fun to play with, and generally seem nice at first. But once you have a bunch of dependencies with their own (byte)string/text types, errors/exceptions, and ways to do I/O, all of which should be combined and work together, those become rather annoying. Additionally, it becomes easier to shoot yourself in the foot (which is mentioned in the alternative title, but doesn't seem to be expanded on much) -- say, by closing a Handle or even merely not draining the input before proceeding with other I/O, if the outside system expects you to.

At the same time, it's not a big deal to handle potentially infinite streams with strict I/O and explicit chunks.

I imagine it is possible to build projects around such libraries, possibly wrapping all the other I/O into it. Similarly to how in principle it's possible to unify all the error-yielding functions, and those working with text and/or byte streams. But I guess it doesn't happen often.

> Additionally, it becomes easier to shoot yourself in the foot (which is mentioned in the alternative title, but doesn't seem to be expanded on much) -- say, by closing a Handle or even merely not draining the input before proceeding with other I/O, if the outside system expects you to.

Hmm, I find the opposite - iteratee-style streaming libraries are pretty much the only way you can implement something like "open this file and transform it in this way" as a library function and ensure it gets used safely, because the stream can be responsible for resource management.

(I'm opposed to Haskell-style implicit laziness though, I agree with strict and explicit chunking being the way to go).

> But once you have a bunch of dependencies with their own (byte)string/text types, errors/exceptions, and ways to do I/O, all of which should be combined and work together, those become rather annoying.

Sure. I think the answer to that is that iteratees are pretty foundational and should be built into the language, in the same way that e.g. monad is not just a library type but something that's fundamentally part of the "platform" and maybe even the standard library. Efforts like the "Haskell platform" help us move in that direction.

There's some completely different programming patterns that are useful when dealing with streams. Streams really come into their own within DAGs. DAGs are to streams what module structures are to programs. You organize streams within a DAG, and can manage different use cases and goals with different branches. Exceptions can be handled flexibly if they are turned into data for the stream. Same goes for configuration.

> it becomes easier to shoot yourself in the foot (which is mentioned in the alternative title, but doesn't seem to be expanded on much) -- say, by closing a Handle

lazy I/O (such as file reads) is suspect and even bad, but laziness + IO (or something like it) is a great way to stream from an API: http://blog.vmchale.com/article/lazy-io

Makes it possible to transcode for one. Basically makes streaming "compose" nicely - bzip2 and lzip otherwise have very different ways of streaming! And one would have to line a lot of things up.

So isn't ByteStream just Stream (Of ByteString)? And isn't all this just iteratees?

Reply on news.ycombinator.com