Gorgonia: a library like Theano or TensorFlow, mainly written in Go
blog.chewxy.com
blog.chewxy.com
Go is designed for system programming, which is the opposite of scripting. In scientific computing, interactivity is an important feature. Softwares like MATLAB and IPython(now Jupyter) become popular because of this. Go doesn't have a nice REPL like IPython, and it never will. Besides, Go doesn't allow operator overload, which leads to verbose programs. Compare `c = a @ b` with `c := Must(Mul(a, b))`, the former Python statement is much cleaner than the Go one.
Usually, researchers use MATLAB, R, and Python for prototyping and training. The final model will be put into production by using something like TensorFlow Serving or by hand written C++ for performance. Even if the final product uses Go for serving, the Go program mostly runs the model via cgo or RPC.
Write in C/C++, and wrap the library in Python and Go is a much saner option. TensorFlow and MXNet are both good examples.
Also, thanks for informing me about MXNet. I've never seen that before. Kinda cool.
If nothing else, to prove a point that there are solid alternatives to C and C++, in memory safe languages.
After all the OP mentioned that only C and C++ were valid alternatives, hence why it is so important to implement such tools in other safer languages.
If people don't do it, as usual there will be a myth that only those languages are able to produce such type of applications.
Similar to the myth that C was the first systems programming language, when many of us were using something else before UNIX got widespread in the industry.
What I'm suggesting here is that there is nothing really remarkable about it, from either a research or implementation perspective, given the achievements of the Java community in recent years.
https://github.com/motemen/gore
There are a few others too.
Go is not set up to have a REPL; even if someone develops one someday it's always going to be limited compared to a language that was designed to do it. REPLs are actually pretty hard; even some languages designed with them from day one (or at least very early) like Haskell still come with an interesting list of caveats about the REPL.
It is a true REPL (not like "gore", mentioned somewhere in that thread). It doesn't implement the whole Go specs, but there's AFAICT nothing in the Go specs that prevent `igo` to implement the whole Go specs. It's "just" work. Moreover, with the introduction of the "buildmode=plugin" in the yet-to-be-released Go-1.8, import-ing packages on the fly should be easy implementable.
("my" new vehicle to provide a true REPL for Go is at https://github.com/go-interpreter)
Further, I'd suggest waiting for one of these projects to complete before you confidently declare that the REPL will be great. I didn't say it's impossible, I said "it's always going to be limited compared to a language that was designed to do it". I looked into what it would take to make a REPL around the Go 1.2 timeframe. There are many issues, such as the inability to query a package for what symbols it has, or the way the obvious workarounds have their own problems, brought on for instance by the fact that you can't re-export symbols in another package so the whole "just keep compiling a module incrementally" has its own issues. At this point I'm pretty confident that the best reasonably-possible Go REPL (i.e., not one that basically reimplements Go again) is going to come with a list of caveats a mile long. And if you do reimplement Go for the REPL, which a SSA interpreter basically is, you don't really have a Go REPL... you have a Go-like REPL at best.
I don't see how reimplementing Go in Go but with a REPL (on top of, let's say, a SSA interpreter) is a Go-like REPL at best and not just a Go REPL. Could you expand a bit?
AFAICT, once one has a Go interpreter (the x/go/ssa is basically just that), implementing a REPL is "just" providing a program counter to the interactive interpreter layer, together with the ability to modify that PC... (many details have been dropped on the floor, of course)
> Go doesn't have a nice REPL like IPython, and it never will.
> scientific computing
for quite some time, and I've the value of a REPL steadily in decline: what's far more useful is a tested code snippet or function & a debugger, occasionally.
Also, if a scientific stack were available in Go, I'd jump on it, because deploying Go is so much simpler than Python. Python's deployment story for scientific software often starts with either )a) create the perfect storm of dependency versions or (b) use Anaconda. Even MATLAB is way better in this regard.
it automatically creates a CPython-2 extension module out of a Go-1.5 package.
I plan to update it for Go>=1.6 and also directly generate a "cffi" python module so CPython-{2,3} and PyPy can be directly supported out of the box.
Sorry, I have nothing of substance to say, but this deserves more than an upvote.
https://github.com/tensorflow/tensorflow/issues/10#issuecomm...
I wrote Gorgonia: https://github.com/chewxy/gorgonia
If there are any questions I'm available to answer them
x = PastValue(y)
y = FutureValue(x)
In contrast all of the other NN systems I've used have premade layers like LSTM and GRU, and to be honest their methods are so hacky I haven't really been able to work out how to do custom RNNs in any other system.
How does Gorgonia handle it?
So what I do is something like this:
func (m *model) costFn() (cost *Node, n int, err error) {
var prev *recurrentOutput
for ...{
// build your recurrent graph here
}
}
func (m *model) runOne() (err error) {
cost, n err := costFn()
g := m.g.SubgraphRoots(cost)
machine := NewLispMachine(g)
if err = machine.RunAll(); err != nil{
return
}
}
source: actually have various LSTMs running. Some with attention, some with ADHD
That is what I mean by "Manual unrolling fails as soon as the number of repetitions is not fixed"; you can do manual unrolling in every framework but generally repeated graph building makes it unusably slow if you're unable to do the loop "in the system" with a runtime-chosen number of repetitions.
Padding is also not really a solution, since your average sequence length is likely 5 or 10 times less than the maximum sequence length that you want to support, so just padding to a fixed size will mean 5-10 times slower processing.
(Although maybe it conflicts with their desire to profit from it?)
Software and hardware are now a commodity and all the big players know it. So the only way to make money is to add value some other way like making it dead simple to deploy tensor flow graphs in gce.
Which surprisingly is one of the few I didn't refer to when implementing Gorgonia
A =
if xw, err = Mul(x,w); err != nil { log.Fatal(err) }
if xwpb, err = Add(xw, b); err != nil { log.Fatal(err) }
if prob, err = Sigmoid(xwpb); err != nil { log.Fatal(err) }
B =
probs := Sigmoid(Add(Mul(x,w), b))
probs := MustSigmoid(MustAdd(MustMul(x,w), b))
Fixed up GPU work
So I wrote my own CUDA wrapper. Turns out there was a bug. I'm not sure where and how, but it worked for the earlier versions of Gorgonia but not this release.
You may also note that a bunch of assembly for the float32 tensor are missing... they seem to be affected by the same bug, so what I'm doing now is I'm planning to rewrite from scratch
Also, where in the repo is your wrapper? (I don't have enough knowledge to debug s.t. like that, just curious how they work)
No I have not reached out to nvidia. Don't have any contacts there