Haskell in Production: 12 Months Building ApiEngine
mth.io
mth.io
* I have no idea how far through the content I am
* The lack of verbosity means I'm constantly having to apply guesswork to their statements (ie an explained point will tell the readers why Haskell gave them better communication. A slide show like that with just headings mean I have to guess the reason Haskell had those benefits)
* It's scales terribly (at least on Opera, some of the content fell off screen - so I had to maximize the browser. But this might just have been an Opera bug. In either case, this wouldn't have been a problem with a more traditional blog layout).
* The navigation requires guesswork
* And most annoying of all, it doesn't actually save the reader any time by displaying the content like that than it would have done with blog post as the jarring navigation presented with slides require the reader to pause before each page to understand how to read the slide, and then pause after each page to apply the aforementioned guesswork. With a blog post, the information is presented in a more natural communicative way which makes it quicker to digest the information.
It's a great pity I'm having to pick post a negative comment about the presentation of the submission rather than comment about the point being raised - particularly as I've been very interested in Haskell lately. But I found even my interest in the subject wasn't enough for me to put up with this slide show layout and thus I ended up closing the browser tab before the end. :(
Well, herein lies the problem: It's not a web page, it's a slide deck.
But I do agree in general. The slides alone are often not very interesting or useful without the talk. Hopefully, it was recorded.
I suspect the author made the slideshow to display on his/her own system, and generously shared it publically as an afterthought.
Edit: Ooooh, found a button that actually does something. Apparently you have to use the scrollwheel, space or the arrow keys. Just clicking on things doesn't seem to work. Hooray for modern UX design.
He's used Reveal.js[1], a presentation/slideshow tool in JS. Can use node.js if you want to have speaker notes/etc too.
I agree about the near-uselessness of this form of communication. But we should blame the submitter for that.
I am beginning to see what you mean by the easy-negative-up-vote effect.
I hit that presentation and saw the first two pages - Mark Hibberd, and REST API documentation with Haskell. You could slideshow the phone directory at half speed and I would still be doing exactly what I did - go look up more info.
On the other hand, the responsiveness of that deck did feel like a phone book at half speed. So no downvoting from me.
I'm someone who normally hates such posts as well. However the reason I did leave such a comment this time round was out of disappointment rather than wanting to be negative for the sake of "internet points". The topic was one that I had a particular personal interest in and the content looked like it should have been really juicy. But sadly the item failed to offer any substance and that was entirely down to the choice of layout.
However it now seems like this presentation was designed to accompany a talk and likely wasn't expected to be viewed on it's own and without explanation. In which case my post was unfair and I genuinely wouldn't have left those points had I known. In fact the only reason I haven't deleted my post is because I think that would now do more harm than good due orphaning the replies.
So I do actually agree with your disappointment about seeing such a post - even in spite of being the author of it.
(Or woman.)
What strategies did you use to combat the lack of stack traces, especially in production?
I was happy to see that you are fellow Brisbanites. I'm a bit out of the "local loop", is there much of a local Brisbane start-up scene?
What did the OP meant by being robust, when the code crashes unexpectedly?
In which case I could produce an equally "robust" Java program by wrapping everything in a try/catch and not logging anything. Except that in Java, or just about any other language, with a single method call I could obtain that elusive stack trace that OP is sorely missing. Which, in real applications (which, to be fair, I've never been masochistic enough to write in Haskell), I always do, and it significantly improves our ability to track down the remaining unanticipated conditions that our error handling logic hasn't already handled in well defined ways. And that sort of work should consume most of anyone's time spent developing, because the happy path is the easiest piece of the puzzle - engineers handle all code paths, hackers handle the most important ones, and hacks only handle the best case scenarios.
I'm not really getting what Haskell has helped here, apart from making the article more upvote-able.
That said, the presentation slides don't give me even a vague sense what the real thrust of the presentation was, so I'm probably missing the substance. I take issue with this being posted without any additional context, but I'll grant the benefit of the doubt and assume that there may actually have been something worth listening to if we didn't just have the slides to look at.
This causes you to think about you code in much more depth and generally leads to good results without the need to constantly debug.
I find that things written in Haskell that compile generally work first time more often than the law of averages would seem to allow.
People tend to avoid this behavior except in rare circumstances. That said, it's easy to replicate locally with the Cont monad. These local reflows are easier to reason about as well. You can even build it atop delimited continuations for more control. These are great for covering early stopping in searches, for instance.
This is not technically true. You can throw exceptions from pure code, either explicitly or with something like an incomplete function definition (which the compiler should warn you about). However, it is considered very bad practice to do so.
Catching "undefined" from pure code is the danger that leads to massively confusing semantics—it breaks down the "value" concept badly. The `spoon` library is a great example of this and it's scary to see it in places. That said, it's not terrible for wrapping up foreign code that isn't quite unsafe enough to need "IO" treatment.
The best usage of "error" is to mark completely impossible situations. These show up easily when you do dependently typed stuff with GADTs, but can also exist due to various algorithm invariants.
The canonical example here is using a distinct type that can only be produced by conversion functions to ensure that user input is always escaped correctly on it's way to the database or a web page. Code that fails to perform this escaping won't just send dangerous data somewhere, it will fail to compile. With care, the same can be done for ensuring business rules are also observed. In many ways the distinct-types-with-conversion-functions pattern is the tip of the iceberg here.
It's obviously possible to not do this, but that's a bit like using a database by shoving all of your data into one giant string then complaining that the database doesn't help you with anything.
Also, for most dynamic languages, checking the type at runtime goes against the grain of the language. Python or Ruby code that is littered with type checks is often unnecessarily brittle.
Probably they have experienced some exception during development, but have had no exeptions in productin.
http://hackage.haskell.org/trac/ghc/ticket/3693
http://www.haskell.org/wikiupload/6/6c/Hiw2012-simon-marlow....
http://www.reddit.com/r/haskell/comments/1etssw/haskell_in_p...
Parallel to that is the ZResultT monad. It's built on the ZResult monad (it should be easy enough to write a Monad instance for it anyway) which looks like an Identity monad with some "Z-specific" error handling layered in. ZResultT just lets you have a transformer stack interleaved with the ZResults.
Then you combine those two stacks in ZZ which gives you both "Z-related" configuration and "Z-related" error states.
It looks like the whole thing is specialized to make it easy to build functions pure functions which only need a handful of the effects and then weave them together into the ZZ monad which is likely to be run at the top level, right below Yesod.
I'm not a Haskell expert though - so don't take that as truth until confirmed by someone else :-)
Is it really that alien (I know Scheme)?
You could look at these to get why
http://www.haskell.org/pipermail/haskell-cafe/2012-June/1015...
https://www.fpcomplete.com/page/case-studies
http://elaforge.blogspot.com/2011/07/what-haskell-doesnt-hav...
http://haskell.cs.yale.edu/?post_type=publication&p=112
http://www.jerf.org/iri/post/2908
(or a few dozen tutorials: http://haskell.org/haskellwiki/Learning_Haskell
-----------
here's my attempt at organizing the learning process (having lots of places to ask questions (SO, reddit, beginners list, G+) is great
http://www.reddit.com/r/haskell/comments/1b7axp/haskell_home...
Here is one great post about starting in Haskell (via jerf@Reddit) http://www.reddit.com/r/haskell/comments/1e3cl0/haskell_intu...
it would be the exact way if I would summarize my experience over last few months..
but I do feel the lacking of good communicators in Haskell community who know how to talk to non-Haskeller.
My first project I wrote a simple scraping tool that scrapes a certain web site and organizes the information in a database behind a front end and API. Immediately I encountered several problems that would have been a cakewalk in, say, Ruby:
* The HTTP library (Network.HTTP) is not encoding-aware. It ignores Content-Type and returns a String with some undefined encoding. So if you grab a resource which is, say, ISO-8859-1, anything that works with UTF-8 will potentially blow up.
* (And if you do this yourself, there is no built-in way to parse MIME charsets into Haskell encoding names except by writing it yourself, I think.)
* The string libraries are confusing. There is String, but also Text. Both are supposed to be Unicode-aware, but many of the operations you want are in Text, requiring conversions back and forth.
* I could not, and still haven't, figured out how to convert a ByteString containing ISO-8859-1 to an UTF-8 string. Data.Encoding apparently exists for that, but I could not get it to work like I wanted. I ended up with this lovely mantra (which you wouldn't think had anything to do with ISO-8859-1, but it does) and left it like that just because it works:
decodeLatin1 bytes = Data.Text.unpack $ Data.Text.pack $
Data.ByteString.Char8.unpack bytes
* For the frontend I ended up using Happstack Lite because it's small and about as easy to get started with as Sinatra. Unfortunately, the first templating system it recommends, Blaze, is crap. With Blaze you write the template inline, building an XML tree, similar to Ruby's "Builder" gem, but the syntax is ugly and unnecessarily complicated. The other recommended templating systems, HSP and Hamlet, are like ERB or PHP, so a serious step backwards from HAML. In the end, I could not find anything like HAML.
* I haven't delved very deeply into Happstack yet, but like many Haskell libs it seems to make simple things more complicated than they ought to be (eg., the routing system). One annoyance I remember is the fact that HTTP verbs aren't IO, which makes code less intuitive, since obviously in a modern web app, HTTP verbs are going to be doing IO (things like talking to databases), and so you have to use liftIO a lot.
* Perhaps the biggest issue for me was the general lack of high-quality documentation (Happstack itself has almost none). Sure, there are tons of searchable machine-generated docs, which is great, but that documentation lacks the bits that tie everything together. Since Haskell is mainly about applying functions to values of types, it is very decentralized; knowing what function to use is only part of the game. It's just as important to know how its return value plays with the rest of the function space, because so much of Haskell revolves around composing functions in idiomatic ways. This is very different from, say, Ruby, where you know a method returns an object, and an object only has the methods its class provides; this centralizes, clusters and constrains the information into easily digestible pieces.
A veteran haskeller would probably not struggle too much with these things. But as a novice, I expected the path to be slightly easier. The language was rarely a problem, the libraries definitely were.
> * The HTTP library (Network.HTTP) is not encoding-aware. It ignores Content-Type and returns a String with some undefined encoding. So if you grab a resource which is, say, ISO-8859-1, anything that works with UTF-8 will potentially blow up.
Agreed that the HTTP package is rough around the edges. It also doesn't support HTTPS, which seems like a major shortcoming to me. Fortunately, there are several better-designed alternatives (which have the added benefit of being more efficient), e.g. http-streams and http-conduit.
> * The string libraries are confusing. There is String, but also Text. Both are supposed to be Unicode-aware, but many of the operations you want are in Text, requiring conversions back and forth.
This pain can be mitigated somewhat by using `{-# LANGUAGE OverloadedStrings #-}`, at least when you're working with literal Strings, ByteStrings, and/or Texts. The main problem with the proliferation of string types is when using multiple libraries, each of which expect a different type of string (or one expects a strict ByteString and the other a lazy ByteString). Then the conversions can get irritating.
Generally speaking, you should use Text for textual strings, ByteString for binary data, and String in simple cases when the convenience of using Prelude or list functions trumps performance concerns. They all have reasonably well defined uses.
> * There are multiple libs for HTML/XML parsing, and no clear winner. HaXml was buggy and did not parse my HTML correctly. TagSoup and HandsomeSoup had weird APIs I did not like. I ended up with HXT, which uses Haskell arrow syntax extensively and is therefore incomprehensible, even when I understand what and how it does it. It feels a lot like people who write ambitious libs in Ruby with too much "magic". Haskell is a functional language and should be wonderful for parsing HTML/XML using XPath or CSS selectors, but it's a nightmare. HXT's parser is impure by default (!) and you have to google some tutorials to find out how to use its pure version, which the tutorial writers (in the same breath) dissuade you from using.
Have you tried the `xml` package [1]? It's not as sophisticated as something like HXT, but for a lot of uses, it does the trick and is far easier to work with. HXT is intimidating for sure, but apparently very powerful once you learn it (I haven't bothered).
> * For the frontend I ended up using Happstack Lite because it's small and about as easy to get started with as Sinatra. Unfortunately, the first templating system it recommends, Blaze, is crap. With Blaze you write the template inline, building an XML tree, similar to Ruby's "Builder" gem, but the syntax is ugly and unnecessarily complicated. The other recommended templating systems, HSP and Hamlet, are like ERB or PHP, so a serious step backwards from HAML. In the end, I could not find anything like HAML.
Funny, I personally think blaze-html is terrific. I like having a Haskell EDSL for HTML generation instead of having to drop into a specialized and restricted "templating language" for that purpose.
If you want something like HAML, check out `hamlet` [2]. It's directly inspired by HAML and is the main templating engine of the Yesod web framework.
I found Snap to be somewhat easier to work with than Happstack, after trying them both. It occupies about the same level of abstraction. Yesod is a more full-stack Rails-like framework, which some people prefer. You can pretty easily mix and match components of the different frameworks however you like; they are nice and modular. I often use snap-core, acid-state, digestive-functors, and blaze-html together for simple web apps, even though they are not all part of any one framework.
> * I haven't delved very deeply into Happstack yet, but like many Haskell libs it seems to make simple things more complicated than they ought to be (eg., the routing system). One annoyance I remember is the fact that HTTP verbs aren't IO, which makes code less intuitive, since obviously in a modern web app, HTTP verbs are going to be doing IO (things like talking to databases), and so you have to use liftIO a lot.
Frequently having to use `liftIO` is a sign that the library author overspecialized their functions to IO. If more people would write libraries using `MonadIO m => m a` instead of `IO a` that problem would basically disappear. Polymorphism is wonderful, when you are able to apply it.
> * Perhaps the biggest issue for me was the general lack of high-quality documentation (Happstack itself has almost none). Sure, there are tons of searchable machine-generated docs, which is great, but that documentation lacks the bits that tie everything together. Since Haskell is mainly about applying functions to values of types, it is very decentralized; knowing what function to use is only part of the game. It's just as important to know how its return value plays with the rest of the function space, because so much of Haskell revolves around composing functions in idiomatic ways. This is very different from, say, Ruby, where you know a method returns an object, and an object only has the methods its class provides; this centralizes, clusters and constrains the information into easily digestible pieces.
I don't disagree that more documentation is a good thing, and some libraries are under-documented. But I can also point to examples of Haskell libraries with superior documentation to nearly anything else I've encountered. Some authors really go the extra mile; it's great.
My experience has been that the type system plus Haddock docs are frequently enough for me to figure out a library even without any prose documentation at all. This, to me, is a huge advantage of Haskell. The types only fit together one way, and that way is the correct one. When using a library with well-designed types, you basically can't get it wrong.
I certainly understand why this would not be so easy for a newbie. It gets much, much easier with experience, though.
[1] http://hackage.haskell.org/package/xml [2] http://hackage.haskell.org/package/hamlet
> Fortunately, there are several better-designed alternatives
Do those you mention handle content type and encodings correctly?
> Have you tried the `xml` package [1]?
Yeah, it doesn't do selection based on CSS selectors or XPath expressions. I don't want to have to manually write tree-traversal code when those already exist.
Take a look at my current scraping code [1] if you want to see how I use HXT. (Just promise not to laugh at my novice work.) Basically, the arrow syntax gives you a weird portal into a different world where you select stuff from a tree using a predicate syntax. I say weird because within the proc and the "-<" arrows, it seems you can only deal with XmlTree operations.
> Funny, I personally think blaze-html is terrific.
It's a bit much, but not awful for generating XML programmatically. The main problem is the need to use "toValue" and "toHtml" all the time. I guess I don't understand why the element building functions don't accept strings.
But for templating, I just don't like embedding templates in the actual source code of the HTTP verb it's for. The main controller code should be about preparing data for the UI, and the UI should be separate from the controller code. You could put it in a separate file and function and import it, of course, but it's still Haskell code, which must be compiled along with the entire app, which slows down the development cycle a lot.
I haven't looked closely at the various web frameworks to see if they support Rails-style reloading (ie., recompiling and reloading the app on each page load). Do you know?
> If you want something like HAML, check out `hamlet`
Hamlet is like someone looked at HAML and didn't grasp why it's so good, because they decided they had to make it look like HTML. Here is HAML:
div#foo
ul.list
- items.each do |item|
li= item.title
<div #foo>
<ul .list>
$forall item <- items
<li>#{itemTitle item}
And again, it's apparently meant to be placed inline.
The Snap guys (I think) also invented their own versions of SASS, called Cassius and Lucius. Cassius corresponds to the indented SASS syntax, and uses CSS selectors, unlike Hamlet.
> Frequently having to use `liftIO` ...
Oh, interesting. I don't know about MonadIO yet. Will definitely read up on this.
> My experience has been that the type system plus Haddock docs …
Haddock is great, but a lot of it is non-trivial to understand without having to read the docs very closely. The "xml" library is an example of a simple library where, if you browse around, you can piece together how it works, although it's so bare-bones that if you just want to know how to parse some XML, you have to hunt down the right function, which happens to be in Text.XML.Light.Input; nowhere is there a "how to" introduction which illustrates use cases.
With more complex libraries, the machine-generated docs become increasingly obtuse. For example, I would never, ever have figured out how to use HXT with Hackage alone. I mean, look at it! [2] Is the parser in Text.XML.HXT.Arrow.ParserInterface? Nope. In Text.XML.HXT.Arrow.ReadDocument? Curiously, no. The function to parse a document, runX, is apparently not even in the Hackage database. And good look finding out that this is the function you need to use, never mind the semantics of XML arrows.
[1] https://github.com/alexstaubo/pfinn/blob/master/Scraping.hs
I honestly can't say for sure, but I've never encountered such a problem in my own usage. Maybe that just means I've been lucky. I just know from firsthand experience that they do a lot of things better than the HTTP package does.
> The main problem is the need to use "toValue" and "toHtml" all the time. I guess I don't understand why the element building functions don't accept strings.
First of all, if you're writing string literals and explicitly converting them with `toHtml`, you're missing out. Instead, enable the OverloadedStrings extension, and then you can do this sort of thing:
html $ do
head $ title "My Web Page"
body $ do
h1 "A Heading"
h2 "A Subheading"
hr
p "A paragraph of text"
p $ "Another paragraph, with an " <> (a ! href "http://example.com/") "embedded link"
p $ em "Emphasized text" <> " and " <> strong "strongly emphasized text"
The HTML DSL functions don't accept strings because String and Html are two fundamentally different types of data. Enabling the type system to differentiate between the two lets you do some pretty nifty things. For instance, if you were to write
"<script>alert('XSS!');</script>" :: String
"<script>alert('XSS!');</script>" :: Html
Plus, since blaze combinators are just Haskell, and the Html type is a monad, you can use all the usual Monad functions with Html:
ol $ mapM_ (li . toHtml) ['A'..'Z']
> But for templating, I just don't like embedding templates in the actual source code of the HTTP verb it's for. The main controller code should be about preparing data for the UI, and the UI should be separate from the controller code. You could put it in a separate file and function and import it, of course, but it's still Haskell code, which must be compiled along with the entire app, which slows down the development cycle a lot.
Yeah, I just put "view code" like the above examples into a separate module and import it. I don't find the slowdown to be significant, but maybe that's just me. In any case, true template languages have to be compiled too, either with the rest of the program or at runtime, so one way or another you incur that cost no matter what you're using.
> I haven't looked closely at the various web frameworks to see if they support Rails-style reloading (ie., recompiling and reloading the app on each page load). Do you know?
Yesod does this by default, and Snap can, if you use `snap-loader-dynamic` for your development builds. I believe they both use the awesome `hint` package for runtime eval and compilation of Haskell code. I don't think Happstack has any auto-reloading capability though.
> Why invent something that looks like HTML but isn't? HAML syntax is basically (almost) CSS selectors, that's the whole point. And again, it's apparently meant to be placed inline.
It doesn't seem that different from HAML to me, but it's clearly a subjective judgment. :) And it doesn't have to be inline. It uses the QuasiQuotes language extension, which means you can do inline snippets or write it in a separate file, however you like.
Hamlet, Cassius, Lucius, and Julius (collectively, the "Shakespearean" template languages) are developed by the Yesod framework guys, not Snap, by the way. Snap uses a template language called Heist by default, which I don't particularly care for (I just use blaze-html instead).
> Oh, interesting. I don't know about MonadIO yet. Will definitely read up on this.
If you use liftIO, you're using MonadIO. That's what liftIO does, it's an adapter between plain IO values and polymorphic MonadIO values. But if library authors use MonadIO instead of IO, you can omit the explicit conversion. This isn't something you really have control over as the user of a library though.
> With more complex libraries, the machine-generated docs become increasingly obtuse. For example, I would never, ever have figured out how to use HXT with Hackage alone. I mean, look at it!
Fully agreed. HXT is a huge, complex beast! I don't think it's fair to judge all libraries by that standard, though. HXT is clearly at an extreme end of that continuum.
I appreciate the input on the other things. I will check out Snap, I think.
This is (a) exactly true and (b) a huge problem. For the expert Haskeller it's usually fairly trivial to figure out the semantics and operations of a library by exploring the types. This means that few experts are incentivized to explain the concepts of their libraries well. This is doubly compounded by the fact that so many Haskell paradigms are uniform across all libraries so it's easy to say "it's got a Alternative-Bifunctor-Semigroup interface" and assume the user will figure out what that means elsewhere.
Now, genuinely new abstractions like `pipes` get very thorough documentation because it pays to teach new abstractions just once. That stuff really ought to be decorating all of the stable, introductory libraries, though. Without it the entirety of Hackage looks terribly uninviting.
http://holumbus.fh-wedel.de/hayoo/hayoo.html
At least it finds stuff like HXT's runX. Unfortunately, it seems buggy in that you cannot link directly to a search.
Learn You a Haskell is probably the best resource if you want to take a look, although it seems to be down right now: http://learnyouahaskell.com/
Haskell is really not alien. The stuff that usually comes up, such as discussions about monads, make the language seem weirder and more esoteric than it really is.
The base language, if you disregard the more advanced, complicated topics (for example, the mathematical basis for monads), is very straightforward, although some of the concepts (typeclasses, lazy evaluation, partial application/currying) can seem very novel and strange and daunting if you are not used to ML-based languages. I was not, and I had no problem grasping the basics, although I am still not fluent in the more complicated monad-based stuff.
The best intro, by far, is Learn You A Haskell For Great Good! [1]. It starts very gently, and is tailored to people who are more familiar with current imperative languages.