Fighting spam with Haskell
code.facebook.com
code.facebook.com
Facebook also employs Bryan O'Sullivan, an epic Haskell library writer (Aeson, Attoparsec, Text, Vector, and on and on http://hackage.haskell.org/user/BryanOSullivan). Bryan also co-authored the "Real World Haskell" book.
So Facebook has hired two prolific Haskellers and probably others I don't know about.
I'm hugely excited that Facebook is making this investment and giving some of the developments back to the community. There are many smaller companies that would be interested in Haskell but don't have the resources or expertise to tackle some of these complicated problems.
Facebook is blazing the trail for production Haskell and the rest of us can follow.
Oh and seeing how I'm actually responding now anyway, there's one more thing: use of the word "Hippie" does not actually qualify as name calling, you do know that, right? I do hope so; you see, it is mostly used in the cartoon Southpark, by a kid who is both the biggest bully of them all, and the most pathetic one as well. The ancipitality is really obvious, so I'm curious: was the satirical context so lost on you that you decided to copy this cartoon character's behaviour verbatim instead? Because in that case I think you could actually gain one or two social skills from Facebook! Chop chop, be a good lad now and give 'er a try, eh?
[1] https://twitter.com/bos31337/status/446679462835273728 [2] http://www.serpentine.com/blog/2014/03/28/where-credit-belon... [3] https://www.youtube.com/watch?v=X0VH78ye4yY [4] https://twitter.com/bos31337/status/476536457000415232
EDIT: spelling
I dislike the underlying premise, the adverts, and (especially) the "real names" policy.
But... between great bits of Open Source like React, cool infrastructure projects like this, and a technical culture which seems a whole lot more open than many other big companies, it's getting kind-of hard to go on hating. Walk back a bit from the obsession with open plan offices, and I might just cave...
Wholeheartedly agree with you comment, Facebook's engineering blog is truly humbling.
There is not a day where I do not try to think of something that would both make the Internet more decentralized, anonymous and secure while being extremely profitable. The latter is crucial because capitalism would make the change viral and being profitable would let me attract talent.
In short, make something that people want AND that promotes values I believe in.
Surprisingly, their massive open office is the quietest office I have been in. It is a really nice workplace.
Don't worry, most of their code base is still written in PHP.
ouch!
It is a little intimidating, though. I think the perception is "Haskell is hard enough, AND I have to be a C++/GHC internals expert too?" The hard truth is you have to be this level of expert at something to achieve robustness at industrial scale.
Simon's comment in this thread hints at some of the power of Haskell alone:
The "automatic" bit is that we insert the code that consults the map so the programmer doesn't have to write it. The map itself is already invisible, because it's inside the monad. So the overall effect is a form of automatic memoization.[1]
This pretty much sums up appdev in Haskell. Engineer (or employ) amazing abstractions with no funny business (hello Spring!) and no loss in safety or expressiveness.
I'm really happy this team exists, this is the kind of work that will take Haskell into the mainstream. Kudos!
The application that I am trying to put together is a realtime websockets application. Therefore, I am using yesod/websockets for the server. For the client, I chose haxe simply because of the targets available making it simpler to write plugins.
On one of my irc chats or somewhere, I did hear good things about haste, though it is primarily a javascript library, therefore I plan to evaluate it later.
About the client: it is a single page app to prevent any page refreshes and I am a bit sold on the frp (using promHx). I guess I should have elaborated a bit more about this in my earlier comment, because my choices are not indicative of anything amiss in yesod.
Edit: added some repos to set a context. The original comment about 4th attempt hit home as the latest repo is my 4th(I am going to quickly lose count) attempt to achieve at a model that might work/scale. There are still some issues that I would like to solve: whether to use code generators or not. Code gens will save me some typing, but I found myself tweaking printfs or equivalents to fit it into a narrow model.
Edit II: Elm, iirc, is being supported by prezi that also happens to be using haxe quite actively. So I felt a bit more reassured about using haxe.
I would like to know more about this. What is a request exactly? An API call? If so, when an existing policy is changed, do the memoization tables have to change as well? How are the memoization tables shared? If this is running on a cluster, I would imagine that lookups in a memoization table could be a bottleneck to performance.
Memoization only stores results during a request. It starts empty at the beginning of the request and is discarded at the end, and it is not shared with any other requests. It's just a map that's passed around (inside the monad) during a request.
I always thought of memoization as storing the parameters to, and result of, a function call in a memotable. Doing some quick research, I came across this definition of memoization from NIST that sounds more general "Save (memoize) a computed answer for possible later reuse, rather than recomputing the answer." What I understand from what you said is that when a request is processed, it produces a map that is passed around for the duration of the request.
Something like:
Request -> (some processes) -> memoized map -> Policy Filters
How is the memoized map reused?
The "automatic" bit is that we insert the code that consults the map so the programmer doesn't have to write it. The map itself is already invisible, because it's inside the monad. So the overall effect is a form of automatic memoization.
Is it plain impossible to pick the best fit language without implementing a solution in the first place and fleshing out the requirements and challenges that specific to the problem space? Or do the problems evolve fast enough that no matter how well you design the system, it will need to be deprecated once in a few years?
One of the big parts of becoming a professional for me was accepting that code has a lifecycle; code is written to make the business money at the time, but it's entirely normal for it to change and die as time goes on.
(That said, you should just write everything in Haskell and then you won't have these problems. When was the last time you saw a company port code away from Haskell?)
Once you win user-share in the space the application starts having problems with scalability or debugging becomes an issue or a myriad of other problems that come with success. Now you need to consider writing parts (or possibly the whole application) in a development stack that allows greater scalability or has built-in semantics for core functionality in your application, thus allowing less reliance on 3rd party libraries, etc.
The other factor is that the team who authors an application may have a totally different skill set from the team who takes over the mature application. The authors may have been VB developers with good SQL skills. Your current team has more people who have dealt with Haskell in large-scale applications.
There is some parallel to home-building. A homeowner does a room conversion for more space, more privacy, etc. The builders of the home didn't install tracks and sliding-panel walls for better room configuration. At the time, they didn't see a need. The homeowner now has a need, so he goes through a much more expensive (relatively speaking) building effort.
All software tends to unmaintainability, even if you don't touch it (libraries change/deprecate etc). Given enough time, the problems that you are experiencing, say, memory allocation in C, get solved by something else, such as garbage collectors, and the amount of time it takes to solve the headache in what you've got is longer than just rewriting into the new language.
As a mere mortal programmer, who knows a little Haskell, my takeaway is that if you want to run Haskell in at web scale for a a large userbase, you need the language's primarily compiler author to help build the application and to modify the Haskell compiler to make it performant. And you also need your team led by a 20-year-veteran Haskell expert who is one of the language's handful of luminaries whonwrotr a plurality of its main libraries. What are the rest of us to do, who aren't at Facebook?
That doesn't mean other people can't use Haskell to build web applications.
Very few companies do, though, so OP's point still stands.
This article would be a much better ad for Haskell if it weren't written by one of GHC's main author. The article also casts doubt on whether Haskell was picked because it's the best tool for the job or because of the team's familiarity with it.
How? I think you are misunderstanding the philosophy. Like SPJ once explained, its 'avoid (success at all costs)' and not '(avoid success) at all costs'. It basically means, 'don't compromise on your principles in order to achieve success'.
Yes, it might take some effort to scale things across a 1.4 billion users. Now, in what language wouldn't that be a challenge?
I suppose if you're one of the "many" other companies that are approaching 2 billion active users/month, you might need to put some thought into your systems too...
That they could use the system dynamic linker makes me think they're using some form of relatively basic dlopen/dlsym/call-method procedure, or something along those lines. That's fine, though the use of "hotswapping" evokes the image of some more elaborate DSU mechanism.
My guess is actually that, yes, this system has rules (and gets live feeds of them) to block certain political content associated with influence operations. But I was hoping to get a more serious, direct and charitable answer from someone who works on the system.
On another note, what happens if Facebook is able to detect "propaganda content" but it only silences propaganda it doesn't like? This is what I originally meant.
It's similar to how propaganda works in US media. It's not the news stories themselves, it's all about the news networks which are told from above which stories to run. I can see Facebook operating in a similar fashion today or in the future.
The point of having a rule set that can be pushed to like the system described here is that it can be fed information from a centralized task force. Indeed, these systems are used across industries with patterns and rules to detect malware, hacking, and a host of other activity (using formats such as STIX or TAXI).
It's unlikely that Facebook itself would be trying to determine whether a post to some content is propaganda - it's more likely that their automated system will apply the rules it receives from information defense agencies.
As per your other note, the selective censorship of information is absolutely a form a propaganda of its own. Facebook (and the US government) don't have the best history here and have 100% certifiably censored information to prevent effective organization of (for example) May Day protests. I'm not going to go speak about that at length here - hoping to get some technical clarity from those involved.
1. They have CORE Haskell contributors on their payroll to deliver this type of project (what this mean is that no... Haskell isn't any better than other language, it's just that they have people who know Haskell very very deep to the compiler level...)
2. In-house custom language eventually does not scale (the EOL is much much much shorter than other programming languages), plan for that :)
This is just complete snark, that misunderstands the general thesis of the article (intentionally?).
Having a core language writer on your team by no means implies that the language in itself is unusable without the designer himself. Your comment explicitly mentions this, and you could really generalise this to any language at all.
The fact that they hired a haskell contributor over a PHP or Ruby contributor was perhaps something you should consider. Or, you know, you read the article, where they answer some of these basic assumptions.
This is extremely presumptuous for a parenthetical. If this is what you believe it merits a comment all to itself, because I don't see how this follows in the least. To rephrase what you're claiming: if a language has a flaw than it can not be any better than any other language.
Who should they hire instead? Given the opportunity and the means, wouldn't you hire someone of that caliber for your preferred language/environment?
FXL employed some tricks that were sometimes beneficial, but often weren't - for example it memoized much more aggressively than we do in Haskell. Mostly that's a loss, but just occasionally it's a win. When a profile shows up one of these cases, we can squash it by fixing the original code.
What matters most is overall throughput for the typical workload, and we win comfortably there.
It would be interesting to know why some are slower. Perhaps they require very little processing and so the time becomes dominated by FFI transformations? It would be nice to know!
> (facebook.com)
Ha.