Who's still in Scala boat? Are Google or Fb or other big cos known to give back to open-source growing any Scala codebases now?
In the last ~18 months, the amount of CVs we're getting has been constantly increasing. Part of that is probably that our startups hiring matures, but it's also the kind of CV that is changing.
I'd say that 3 years ago, there was an 80% chance that the applicant was highly self-motivated to learn Scala in their freetime, and tried/did introduce it at his/her current workplace. Today, there is an 80% chance that the applicant either "had to" learn it in their current workplace, or learned Scala when switching jobs. (Don't get me wrong, they're still motivated, and they took the chance when it was there!)
So there is a switch from the Early Adopters to the Early Majority (where the Early Majority now has worked 1 or 2 years with Scala at their current job, and is confident enough to look for a new one).
One driving force was definitely Spark, but there are a lot of Enterprise apps, unrelated to ML (usually with higher traffic requirements). The sort that would have most likely been done with Java or C# 5 years ago. It seems a lot of Enterprises introduce Scala when they try to break up their (Java) monolith into microservices.
So it seems that Scala has been carving out it's place in backend/microservices with scalability requirements, and is eating part of Javas cake there.
I've been using Java and Scala exclusively for the past 10 years at two large NY banks and two fintech start-ups, and I literally do not know anyone who has ever compiled a Kotlin program.
Simply extrapolating from one's own experience is fraught with peril.
So most companies actually are more keen to bet on Scala than F#.
OCaml/Reason world lacks the wealth of Java libraries, which are an import away on Scala.
Regarding Kotlin, so far its killer use case is targeting Android, where one is stuck with an ageing Java subset. Outside of Android, it remains to be seen how it will keep up with the Java improvements coming up every 6 months now.
And what subset would that be? The only place you can't use F# IIRC is UWP application which is likely not the deciding factor in choosing F# or Scala.
It is still playing catch up with VS 2017 tooling for VB.NET and C#.
I don't believe most organizations considering adopting F# or Scala are considering them for GUI development so I'm not sure why you'd rule out F# because of that
- Windows form works but not perfect. The editor works if you install the template for it, but not the auto generation of code, like double click on button -> handler generated. You write code programmatically. but is used a lot. I use it in the repl (fsi), to generate chart, custom data visualization.
- WPF the same, works, no editor (but codegen is less needed)
- Xamarin support F# ( https://docs.microsoft.com/en-us/xamarin/cross-platform/plat... ) on Forms, etc
Some good from community+MS, because community tried to adapt techonologies and make it more friendly to use in F#
- Xamarin XAML in Elmish style, really more idiomatic ( https://github.com/fsprojects/Elmish.XamarinForms ) from F# creator itself (Don Syme, who now work on xamarin division too)
- WPF and Xamarin xaml can be used with a type provider too for statically type view at compile type ( http://fsprojects.github.io/FsXaml/ )
The only one not yet supported is UWP, because of of .NET Native.
All that without speaking of the gui stack outside .net framework, like electron+fable or just fable+elmish/react/react.native ( https://github.com/SAFE-Stack/SAFE-Nightwatch )
Never understood the mentality for designing UIs by coding instead of visually.
I care for what comes in the box, and is directly supported by Visual Studio and Blend.
If someone needs to lose their .NET GUI tooling productivity to embrace F#, then better wait while C# keeps getting F# most relevant features.
Even C++ has better UI tooling support on Visual Studio than F#.
Personally, i write c# and xaml, and i dont use the editor (vs or blend), but i edit directly the xaml.
About f# and gui, depends on use case. For example https://fslab.org/XPlot/ to show graphs.
let series = [ "bars"; "bars"; "bars"; "lines" ]
let inputs = [ Bolivia; Ecuador; Madagascar; Average ]
inputs
|> Chart.Combo
|> Chart.WithOptions
(Options(title = "Coffee Production", series =
[| for typ in series -> Series(typ) |]))
|> Chart.WithLabels
["Bolivia"; "Ecuador"; "Madagascar"; "Average"]
|> Chart.WithLegend true
|> Chart.WithSize (600, 250)
But yes, if you use editor like c# version, F# is less nice to use. But again, programmatically allow other things, like https://github.com/fsprojects/Elmish.XamarinForms
So depends how much time you edit the view (and why), vs gains in the logic behind the view. For me the global tradeoff, but depends, so you are right.
In fact, Microsoft has to advocate library writers to actually care about .NET Core and .NET Standard.
https://channel9.msdn.com/Shows/Visual-Studio-Toolbox/NET-St...
Still, for all intents and purposes you are correct - but there's still some growing pains in the tooling that are hard to ignore.
+1000. It wasn't even library support, it was the tooling that was a huge pain in the ass. Half the time it didn't work right, builds broke unexpectedly, build file formats kept changing, cripes what a pain. I swore off it until the early .NET 2.0 standard prereleases when things seemed more stable, and it's been much easier to port my libraries.
Let's face it, Kotlin hurts Scala adoption.
Outlook largely negative, falling in rankings across the board.
Scala is a fantastic language, but it’s not one your average Java developer can pick up in a day or two.
Scala, on the other hand, gives you a huge toolbox down to some really complicated to reason about features like implicit parameters, creation of completely arbitrary operators, etc.
[Insert some funny pun here about Java giving you a simple tool while Scala gives you an incredibly complicated one]. They're both great languages, but they serve very different purposes and audiences - Kotlin happens to fit Java's demographic better than Scala as a result, it doesn't have the magic and complexity to the same degree Scala does (the most confusing new constructs probably revolve around builders/lambda's with receiver types which aren't needed by most developers not writing DSL's).
With scala I seem to have wtf moments about the language every once in a while. (eg: magnet pattern)
The idea that Kotlin is hurting Scala adoption rests on the assumption that the only people who would ever adopt Scala are the people who are trying to get rid of Java.
I know Java since the early days and only used Spring during one month as validation of an architecture proposal.
This statement is wrong. F# can use .NET Core, and it works same as JVM.
I have not listed it, because in spite of spec, it still is a dynamic language.
And maybe not across the board, but my clients are very comfortable with the (very few) F# solutions we've done. Ocaml clients generally use it for stability, or correctness, they certainly aren't competing in the same sphere that kotlin, scala, Java really operate in. Prominent in the financial spade especially, but the trend is the same everywhere.
Reason may eventually help ocaml move closer into those spaces, as people are coming to ML with more openness and are more eager to learn and get involved, so that'll be cool to see in the future.
Anyways, don't count F# out either. I'm no expert, only ever used it with other project leads and with smaller clients, but MS is doing impressive work lately, like NET Core and stuff like that. They at least can show that making net more visible and flexible is part of what they're after.
Java won't be killed. Ever. But people may stop writing it eventually.
In fact with spark 2.3 python UDF, the performance gap has also reduced. https://mindfulmachines.io/blog/2018/4/3/spark-rdds-and-data...
What kind of specific pipelines did you have trouble in pyspark ?
I very, very rarely think of using PySpark (and I have way more experience with Python than with Scala) when working with Spark. In a kitchen setting, it would be like having to prepare a cake and having to choose between a fork and a whisker. I can get it done with the fork, but I'll do a better and faster job with the whisker.
But I'm very excited about pyspark 2.3 UDF bringing grouped map . It will be interesting to hear your views on that https://databricks.com/blog/2017/10/30/introducing-vectorize...
I'd recommend Dask (haven't tried it much but from all I've seen is top-notch) to anyone who wants Python all the way down (at least until you hit the C at the bottom) ;)
In general, I will stay happily in the spark world using pyspark rather than go to Dask right now.
Besides performance and the obvious fact that not knowing scala makes it difficult to understand the underlying Spark code, there are multiple ways in which scala is more natural to develop in (many libraries are for scala only, for example).
I have no doubt that scala is more performant and the "fat" jar mechanism makes dependency management and codeshipping very easy (it's still tricky to install python dependencies on your spark nodes), but the pandas ecosystem is definitely more intuitive to understand.
PySpark is often used for data science experimentation, but is not as frequently found in production pipelines due to the serialization/deserialization overhead between Python and the JVM. In recent years this problem is less pronounced due to the introduction of Spark dataframes which obviates the performance differences between PySpark and Scala Spark, but for UDFs, Scala Spark is still faster.
A newer development that may change all this is the introduction (in Spark 2.3) of Apache Arrow, a in-memory column store engine which lets Python UDFs work with the in-memory object without serializing/deserializing. This is very exciting as this lets Python get closer to the performance of JVM languages.
I've played around with it on Spark 2.3 -- the Arrow interface works but still not quite production-ready but I expect it will only get better.
Many folks are making strategic bets on Arrow technology due to the AI/GPU craze (and an in-memory standard enables multiple parties to build GPU-based analytics [1]), so there is tremendous momentum there.
At some point I expect the relative importance of Scala on Spark will decrease with respect to Python. (even though Spark APIs are Scala native)
[1] https://www.nextplatform.com/2017/05/09/goai-keeping-databas...
Can't speak too much about the others, but I recently looked into using Akka with Java. Akka itself had a good deal of documentation for Java users. However, other related products (look tools for monitoring Akka, etc) were clearly treating Java as a second class citizen. When there was documentation, it would be outdated, and unmaintained. I ran into this over and over again, which weighted heavily on my decision to not use Akka. I'm not quite sure if there is anything about the language that would have made it particularly challenging (doubt it), but it definitely seems like the community built around it is more Scala centric. And that in itself makes it challenging to use with anything else.
Scala has a for/yield construct similar to Haskell's "do notation", which is the perfect way to work with async code - it strikes the right balance of avoiding an unreadable callback pyramid of doom, but still your async yield points visible (the difference between <- and = is about as concise as it gets, but still very visible). And having HKT and allowing you to express concepts like monads means that a whole library ecosystem can build up of standard operatons operations (e.g. traverse, cataM) that work on async futures and also on other contextual/effecty types (e.g. audit logging, database transactions). That's the big advantage it has over Rust and most other competitors, and it particularly shows in things like Kafka/Akka that need to work with async a lot, but also in large programming projects generally. (What it shows up as in practice is that you can do everything in "plain old code" - all the things that need reflection/interceptors/macros/metaclasses/agents/... in other languages are just libraries in Scala - and that makes development so much easier and bugs so much rarer)
Haskell has all that - the trouble with Haskell is that "there's no way to get to there from here". I was able to go from writing Java on Friday to writing Scala on Monday - not great Scala, but working Scala, and I was just as productive as I was in Java. I couldn't've done that with Haskell.
I'm much more of an ML-style programmer, but the thing keeping me away from OCaml is the lack of a multicore runtime. Not only is Scala's concurrency mature and performant, implicits (like having an ExecutionContext) make it extremely easy to use. Even if OCaml released it's multicore runtime today, it would take years to get to the maturity of the Scala ecosystem.
Plus, occasionally I run into areas where OOP really blows away functional programming. It's nice to not have to learn a new language for those occasional times.
LinkedIn, Twitter, The Guardian, Morgan Stanley, Barclays, Zalando. Generally speaking, Scala is used a lot by companies involved with Big Data (because they use Spark).
It's not even clear to me that most companies doing data science in Scala have that scale -- they're just using tools and libraries companies at that scale have open sourced. You could call it cargo culting, but I think it's more nuanced than that. I think engineers can be separated into two camps roughly: those who are passionate about the language(s) they use, and those who are simply trying to get the result they need and don't care what language they use to get it. A lot of data science engineers naturally fall into the second bucket, so using Scala because a library they want to use is written in it comes naturally, even if they could get the job done in Python (possibly with a bit more wheel reinvention).
Python is a good choice for data science even at relatively large scale. I'd question it's suitability for stable, scalable deployment in production (not to say it can't be used then, just that I wouldn't necessarily reach for it first, preferring either C++ or Rust for that).
Scala just doesn't figure into the picture at all. I consider that some "Big" data tools were written in it to be a matter of trivia and not essential to the work of data science.
I think for a particular data science mindset (the category theory toting, bijection loving person) Scala actually _is_ essential to the work of data science. But these people are in a minority.
Anyway, if you're truly in the second category, then the fact that the best library for doing X is in Scala would mean you're going to write some Scala, despite the fact that it's ~accidental that it was written in that vs Python.
Scala at Morgan Stanley: https://www.reddit.com/r/scala/comments/480ral/scala_at_morg... https://www.linkedin.com/jobs/morgan-stanley-scala-jobs/?cou...
Scala at Barclays: https://underscore.io/jobs/2017-11-02-barclays/
Scala at Zalando: https://jobs.zalando.com/tech/blog/why-we-do-scala
Scala open source projects from Twitter: https://github.com/twitter?utf8=%E2%9C%93&q=&type=&language=...
1- https://www.cisco.com/c/en/us/products/hyperconverged-infras...
2- https://jobs.cisco.com/jobs/SearchJobs/hyperflex?3_12_3=187
As for Scala usage. I'm a Scala consultant, so I'm biased, but I'm seeing a lot of adoption. Most finance companies and most media companies are using Scala.
Martin Odersky seems like a really good language designer. I took a look at the Scala 3 languages features, and a more radical departure from historic Scala probably would have helped Scala.
(Did you have specific breaking changes in mind? The most important changes I'd want to make to the language - having a syntax for guaranteed-safe equality comparison and guaranteed-safe pattern matching - could be done in a backwards compatible way, and the only other thing I can think that I'd like would be Idris-style totality checking which could also be backwards compatible.)
Wait what? In the last post you were complaining it was too close to existing scala, now you're saying it's too big a step? Again, what is the change you're actually advocating?
Hint, the stuff that Chris Granger was trying to accomplish.
I liked the idea of Scala and took Odersky's course. But I built a few things in it and it was never not frustrating. What Bruce Eckels said resonates with me: "I’ve come to view Scala as a landscape of cliffs – you can start feeling pretty comfortable with the language and think that you have a reasonable grasp of it, then suddenly fall off a cliff that makes you realize that no, you still don’t get it."
Then I watched this Paul Phillips talk, which convinced me that the problems were deep, not superficial: https://www.youtube.com/watch?v=4jh94gowim0
My hope with Dotty, etc, was that they had learned a little humility and were pruning back enough to make it a decent developer experience. But I could well believe that it was impossible to do enough and still end up with something that could be fairly called Scala.
Maybe. But high-risk. See Perl 6.
https://www.linkedin.com/jobs/amazon-scala-jobs/?country=gb
https://www.indeed.co.uk/cmp/Amazon.com/jobs?q=scala&l=Londo...
https://www.amazon.jobs/en/search?base_query=scala&loc_query...
Its popularity also far surpasses F# and OCaml in jobs available, books, libraries or other metrics that count.
Not a "big" company by global standards, but not small either, with around 120 engineers.
What if you want functional programming without having to go to MS stack and stay on JVM. You only have have two real options clojure and scala.