Deep learning may need a new programming language
venturebeat.com
venturebeat.com
It was probably a good marketing move on their part, a wink towards the Fortran/Matlab crowd, but it certainly damages the language's appeal for potential converts from C, C++ and Python.
I know the arguments for 0 vs 1 indexing, but in practice it's not really an issue since they're both just book-keeping. The only time where things might get interesting is for certain complex recursive expressions (recursive in the mathematical sense) where you want your code to match the equations, but in those cases I just create a new offset-index and work off that. (edit: just learned from another comment that Julia supports custom indices.)
Note: caveat -- my use of indices is mathematical. I don't do any kind of low-level stuff.
If I want to stick with higher-level construct for dealing with "collections of numbers", I can use numpy arrays in Python, the STL in C++, etc.
Do you think Julia's high-level constructs are much better than the competition? E.g., how do they compare with C++20 ranges?
Someone more polyglot than I am will have to comment on C++ comparisons.
Edit: ah, a sibling comment (now deleted) mentions Julia does have some support for arbitrary indices ("offset arrays").
Many of the most used Julia types are defined outside of the standard library, such as StaticArrays (for fixed size arrays) and CUArray (for GPU processing).
Arrays can start at [-100] if you feel like it.
I was specifically interested in the question ‘could Julia be my general purpose language?’. In reality, I am too much into Lisp languages, Haskell, etc. to seriously want a new language, so this was just a two evening experiment.
But what did you think of Julia?
Meanwhile, we have absolutely no problem statically checking Julia code (and in fact we do when generating code for TPU). It's a bit of a non-standard thing to do and we're working on improving the tooling for this, but putting Julia on the same level as python here ignores a significant aspect of the design of Julia.
As for collaborations among thousands of engineers, it is true that we don't really have Julia projects that large yet (other than Julia itself) and I'm sure we'll undoubtedly find things to improve in the language to support collaboration at that scale, but i really would encourage you to look at things like the new package manager or the documentation system. A significant amount of work has gone into the usability thereof and I think at that point they warrant criticism more detailed than an off hand dismissal.
What I don't quite get it's the silly insistence of the Julia fanboys to claim Julia is a general programming language, and that as such it should be used for everything. There is nothing wrong with being a focused and discipline specific programming language.
As you probably know it is hard (and in a precise sense actually theoretically impossible) to design type systems that are both convenient to use (have automatic type inference) and incorporate dependent types (such as shapes of tensors). Julia's solution of using multiple dispatch is not a substitute for such a static type system.
Running into an exception, if there is no method for the particular combination of parameter types you have accidentally produced in a 1000 LOC project might be fine, it will make it pretty hard to refactor anything in a larger project. It results in exactly the same challenges that large python / javascript projects have. Those challenges are well documented and are the reason why large companies move away from those languages to statically typed languages.
This is why I believe a statically typed language with good support for numerical computing will eventually win as the language of choice for large production machine learning applications. Leaving (for now) exotic examples such as Neural Differential Equations aside, current machine learning models would probably actually profit more from some of the type safety features let's say Swift has (using structs for example instead of Tensors of certain shapes) than from more comprehensive support of numerical features. For the most part we are talking about very basic linear algebra after all.
The Swift for Tensorflow design documents argue this pretty convincingly as well.
Regarding the package and module system you might be right, but looking at https://github.com/JuliaTPU/XLA.jl/blob/master/src/XLA.jl and comparing this to how you would assemble a user facing module in lets say OCaml https://github.com/janestreet/core/blob/master/src/core_unix... I personally feel like the later gives me a lot more information how I would use the module and a clear separation from the implementation at the same time, similar things can be said about how this would look like in Swift or even a well written C/C++ header. As far as I can tell package management is still roughly at the level of pip in sophistication, not comparable to solutions like cargo or opam.
The state of journalism today ... their Github repo web pages show that there's more C++ than Python in both repos.
So it's a good editor that's lacking too, besides the fact-checkers.
Facebook will obviously go its our own way. Given their investments in the JS ecosystem, im hoping they end up choosing Typescript for this.
Jest is a Facebook project for testing React. They moved from Flow to Typescript.
https://github.com/facebook/jest/pull/7554
Dont want this thread to go OT. But I do think Typescript will be an awesome counterpart to Swift-for-Tensorflow on the ML side.
Half of this article is about hardware, while only a small part of it is about programming languages.
> There are several projects at Google, Facebook, and other places to kind of design such a compiled language that can be efficient for deep learning, but it’s not clear at all that the community will follow, because people just want to use Python
Are the libraries not implemented in native code? The brief mention that python gets does not really detail what is wrong with it, aside from not beeing compiled I guess. But that is no issue if the libraries are native, so I dont see a reason to move away from python, let alone create a new programming language for it
Python ML code is a bit of a joke. If you're any kind of semi professional developer who sees deep learning code for the first time and doesn't say "what the fuck would you do it like that for", I simply don't know what to tell you.
Im not saying a new language will fix this problem but some new primitives and idiomatic options protecting people from themselves probably wouldn't hurt the industry.
Likewise, something more ergonomic might actually improve onboarding new developers into the space.
I am not a Swift developer but I have looked at Swift enough to know that I like it better than Python.
Common Lisp is another interesting language: the SBCL compiler’s warnings will help you catch many errors early. So, maybe Python would be much better with MyPy type annotations and JIT compilation.
https://github.com/keras-team/keras/blob/master/examples/cif...
"I simply don't know what to tell you." Please try! You can at least provide one example of the jokes you are referring to?
This is a perfect example of what's perplexing me about the "learning" industry.
https://i.redd.it/00824ycw3hwy.jpg
That code also uses pickle to serialize to disk, which is a gross, irritating pattern I've seen from Pythonistas: just pickle it to disk, it works on my laptop.
Seems like Logo programming, to me.
If you write ML code for money, you are a pro, the end. Python is not perfect, but everything else is worse, far worse. What is the product that falls out of a ML pipeline ? ML pro-tip: its not hand-crafted artisanal code.
Kudos, but what if i told you if the tooling authors could have ensured you never wrote poop but still got shit done?
If you can't describe how that model makes its way into a product and is consumed, then you aren't a pro, the end.
From the product's perspective, all you produce is a configuration file.
Which is already done by python libraries.
However ... that's not really true. Humans prefer to think in terms of operations (consecutive steps) and labels for intermediate results.
The most important situation where this is important is if the dataflow is dynamic. A dynamic graph can make it unclear what is assigned to what, at which point in time, and programming becomes a mess.
1. First-class dataflow variables 2. Syntax for creating flows (c/f pipe in *nix shell) 3. Transparent parallelisation
All of these things exist in libraries of various sorts, but can feel cumbersome compared to native syntax.
[0] https://mitpress.mit.edu/books/concepts-techniques-and-model...
As well as that esolang that appeared a few months ago on the front page. https://github.com/mrakgr/The-Spiral-Language
There are two types of ml jobs. One for modeling and another for computation. The former deals with the communication with human. The latter deals with the communication with machines. I would prefer that a statistical logic programing language (maybe extended from prolog) for the modeling part. A strong compiler tool or service that compiles it into any machines, on cloud or edge. Most of the time, what human likes to achieve is small and declarative. But for any small modifications of the program, we have to think about the computational flow, deal with the type errors and tune the low level efficiency of CPU and GPU. It's a waste.
I like the settings of SQL where modelers use SQL to mine data while engineers maintain the performance of the computational engines.
On GPGPU programming certainly not, in fact latest NVidia's hardware is designed explicitly for C++ workloads.
On GUI frameworks, C++ no longer has the spotlight on OS SDKs that it once had.
[1] https://blogs.nvidia.com/blog/2019/02/05/adacore-secure-auto...
I belong to the same fanboy club, which is kind of why I do like C++ and not so much about C.
I just don't see many adopting it without legislation enforcement, which is why the focus is on autonomous vehicles, where Ada already has a good story.
I do everything flow-based these days. It's not the fastest way (and I also have the luxury of not having to please anyone but myself), but it allows me to only think about my domain problem instead of programming language issues.
I came at it from audio synthesis, where modularity and interoperability are priorities and there is relative cooperation between manufacturers and developers on technical standards. I mention this because audio synthesis and the closely associated business of sequencing have a great deal in common with breadboard electronics and super-basic computing like adders, flip-flops, and so on. You can implement simple classic video games like Pong and Asteroids in a modular synthesizer and play them on an oscilloscope if you're that way inclined.
If you find this interesting, I'd suggest Reaktor as the software platform of choice as it's affordable, ahs a large community, and excellent documentation. Flowstone started as audio software and is now aimed at the Robotics industry. It has one of the nicest UIs and allows you to write code directly into modules. I use KNIME for high level data processing because it has an extensive library of database connections/ format translators/ API hooks etc.
https://www.bloomberg.com/news/articles/2019-02-18/facebook-...
https://www.ft.com/content/1c2aab18-3337-11e9-bd3a-8b2a211d9...
One of the major reasons why many researchers never cared for OpenCL.
It is indeed one of the few existing languages appropriate for generic experimentation in numeric programming.
For example, computing the product of two matrices in C or Fortran is exactly as fast by writing three nested loops or by calling a library function. In python, julia, octave, etc, the difference is abysmal. This is a very sad state of affairs, that forces a mindset where you are allowed a limited toolset of fast operations and the rest are either slow or cumbersome. If you want to compute a variant of the matrix product using a slightly different formula, in Fortran it is trivial change, you just write the new formula inside the loop. But in python or julia you are stuck with either an unusably slow code or you have to write it in another language entirely.
"Vectorized" operations are cool, elegant, and beautiful. But they should not be the only tool available. As differential geometers often resort to coordinates to express their tensor operations, so should programmers be able to.
My main gripe with the julia interpreter was that it was ridiculously slow to startup (I was using it as a "calculator" from within a shell loop: each iteration spawned a julia to perform a simple matrix computation). Does this performance has improved recently?
By the way, what do you mean by "removing safety checks should be explicit" ? This sounds like a problem that the language should be able to deal with itself without bothering the programer (e.g. if the bounds of the loop are variables of known value, it can be checked beforehand, so the bounds checks can be safely omitted).
https://www.slideshare.net/MaurizioTomasi/towards-new-soluti...
Sure, I am not calculating the product of two matrices, but something else. In that case, I proved that Julia's performance is entirely on par with vectorized C++. I am indulged to believe that somebody might produce similar results for the test you are proposing (matrix product).
https://bitbucket.org/Maurizio_Tomasi/adass2018-julia/src/ma...
(and no, the download button does not work for me)
People have been calling for the phaseout of C/C++, but even today's most popular DL frameworks have backends written in C++ in lieu of Rust.
Whoever has imagination more rich than mine, how can a programming language can be easier than Python? What's difficult in it?
The only thing I find inconvenient in Python is you can't simply put every function (including every member functions of a class - that's what I would love to do) in a separate file without having to import every one of them manually.
I do believe that he/she meant that they need way higher levels of abstraction than Python.
Let's say Python is current C, they want something X that's like Ruby in compare to current C.
That strikes me as an anti-pattern although you could mimic this behaviour by exposing each one at the package level. Performance is likely to be terrible if you're working on anything sufficiently large.
When you create a .py file, the CPython interpreter treats it as a "module". You can basically treat a Python "module" as if it were a class because the global variables are automatically scoped unless you import them explicitly. You could actually write some code that would allow you to instantiate the entire module in the same manner you would instantiate a class. Though, I have no idea where this pattern would be useful.
but python is really popular and easy to grasp. you'll end up writing a new programming language and people would still use python in the end.
You can pick only two options;
* features: you have an extensive standard library, macros, high level constructs (like generators, asynchronous programming, ...)
* performance: julia is based on LLVM and as long the code is type-stable and avoids memory allocation in hot loops, the performance is similar to C or Fortran.
* usability: you can also program at a very high level like you would do in matlab or python/numpy.
https://doc.rust-lang.org/book/ch02-00-guessing-game-tutoria...
[text_io]: https://crates.io/crates/text_io
return bigEndianToNative!T((cast(ubyte*)&b)[0..b.sizeof]);
Do you really feel this is a language that scores high on usability? Because it doesn't look like it.