craigacp

SIMT - Single Instruction, Multiple Threads

The same operations in the same order is a tough constraint in an environment where core count is increasing and clock speeds/IPC are not. It's hard to rewrite some of these algorithms to use a parallel decomposition that's the same as the serial one.

I've done a lot of work on reproducibility in machine learning systems, and its really, really hard. Even the JVM got me by changing some functions in `java.lang.Math` between versions & platforms (while keeping to their documented 2ulp error bounds).

Some of the gradients are present in the TF C API (and thus the C++ API), but it's hit & miss which are in C and which are in Python. There was an attempt years ago to port more gradients to C, but this petered out in the way that most TF related efforts seem to do at Google.

We use the C API to generate gradients for TF-Java, and have some success training models with it. Replicating the Python bits in another language is a huge effort though that we haven't completed.

The encoder's embedding is contextual, it depends on all the tokens. If you pull out the embedding layer from a decoder only model then that is a fixed embedding where each token's representation doesn't depend on the other tokens in the sequence. The bi-directionality is also important for getting a proper representation of the sequence, though you can train decoder only models to emit a single embedding vector once they have processed the whole sequence left to right.

Fundamentally it's basically a difference between bidirectional attention in the encoder and a triangular (or "causal") attention mask in the decoder.

There are a bunch of these things in a word2vec space. I had a blog post years ago on my group's blog which trained word2vec on a bunch of wikias so we could find out who is the Han Solo of Doctor Who (which I think somewhat inexplicably was Rory Williams). You need to carefully implement word2vec, and then the similarity search, but there are plenty of vaguely interesting things in there once you do.

Java has had the ability to run a single code file directly by running `java <path-to-file.java>` since Java 11 - https://openjdk.org/jeps/330. And it's had a Java REPL since Java 9.

There's usually a two or three step training procedure, first training to predict the next word on a huge corpus of text (billions or trillions of words), then possibly some instruction tuning (giving the model question & answer pairs and training on the answer) and then finally RLHF (or RLAIF, DPO etc) where the model is trained to match human preferences. It's this last step that is used to increase the helpfulness & harmlessness of the model, training it to not respond to certain topics.

I did. It depends what you want, for an overview of how ONNX Runtime works then Microsoft have a bunch of things on https://onnxruntime.ai, but the Java content is a bit lacking on there as I've not had time to write much. Eventually I'll probably write something similar to the C# SD tutorial they have on there but for the Java API.

For writing ONNX models from Java we added an ONNX export system to Tribuo in 2022 which can be used by anything on the JVM to export ONNX models in an easier way than writing a protobuf directly. Tribuo doesn't have full coverage of the ONNX spec, but we're happy to accept PRs to expand it, otherwise it'll fill out as we need it.

Both GPU support and SIMD CPU support are being worked on in OpenJDK projects, there are talks on them from last year's JVM language summit - https://youtube.com/playlist?list=PLX8CzqL3ArzW90jKUCf4H6xCK...

ONNX Runtime needed a good demo application in Java and this was fun to do (I maintain the Java API for ONNX Runtime and wrote this SD implementation). I've added SDv2 and SDXL support (and the turbo variants thereof) after the initial release, and I'll upgrade it to the latest ONNX Runtime when that comes out to get FP16 support among other things.

There's a correction to that tweet, larger vocab means fewer tokens for any given sequence (usually, assuming it's not to add other languages or character sets).

As ONNX models are protobufs you can edit them at a Python or Java REPL (or other language but I've personally used those two). Dumping them out as text seems like a lot more work, and a lot less typesafe.

We built model & data provenance into our open source ML library, though it's admittedly not the W3C PROV standard. There were a few gaps in it until we built an automated reproducibility system on top of it, but now it's pretty solid for all the algorithms we implement. Unfortunately some of the things we wrap (notably TensorFlow) aren't reproducible enough due to some unfixed bugs. There's an overview of the provenance system in this reprise of the JavaOne talk I gave here https://www.youtube.com/watch?v=GXOMjq2OS_c. The library is on GitHub - https://github.com/oracle/tribuo.

A conventional neural network, i.e. one using a stack of dense layers, can't unroll across a sequence in the way the transformer does. So while it could compute the relative importance and interaction of the features it sees it wouldn't be able to compute that across arbitrary length sequences without a mechanism for the sequence elements to interact, which is what self attention provides.

Exactly so, there are a bunch of Java interfaces to native ML libraries that have some messy bit hacking to convert between fp16 and Java floats. This API along with the intrinsification (https://github.com/openjdk/jdk/pull/10500) will mean I can throw away that code and replace it with something that performs correctly and is much faster.

Probabilistic programming can be done via MCMC approaches, but you can also infer the necessary quantities by using variational inference (which approximates the distribution described by your program with something that's simpler and easier to estimate).

Basically probabilistic programming is a way of describing a distribution, and then MCMC is one way of inferring the quantities in that distribution.

There is extensive discussion of that topic in the valhalla-dev and valhalla-spec-experts mailing lists, but I think the details are still being worked out.

There are JEPs out for parts of it now - Primitive Objects (which is the new name for value types) https://openjdk.java.net/jeps/401 and Unify the basic primitives with objects https://openjdk.java.net/jeps/402

XGBoost also has a binary file format which is neither pickle nor json. It's shared across their Python, Java and R frontends and parsed by the C++ library.

The reviews for published NeurIPS papers for at least the past 5 years are available from each paper's page on http://papers.neurips.cc.

It's weird to me that people build libraries on top of the ML stack to track provenance, when it's really the ML library's job to do that for its inputs. However it is a right pain building it into the ML library as it affects all the interfaces. We build data, model & evaluation provenance objects into our ML library, Tribuo (https://tribuo.org), as a first class part of the library. You can take a provenance and emit a configuration to rerun an experiment just by querying the model object. It is built in Java though, which makes it a little easier to enforce the immutability and type safety you need in a provenance system.

edit: I should add that I'm definitely in favour of having provenance in ML systems, and libraries layered on top are the way that people currently do that. It's just odd that people aren't working on adding that support directly into scikit-learn/TF/pytorch etc.

You can with some hoop jumping in the current release (e.g. using 1.14 you need to define the model and optimizer in Python - https://github.com/oracle/tribuo/blob/main/Interop/Tensorflo...). With the upcoming alpha release from the TF Java group you can train purely from Java without invoking any Python - https://github.com/tensorflow/java-models/blob/master/tensor...

The Tensorflow Java project is alive and well. The next version of the API based on TF 2 is coming out soon. (Full disclosure, I'm a member of the SIG that's building it). I'm a firm believer in the utility of type systems for building machine learning projects.

https://github.com/tensorflow/java

Actually my group in Oracle Labs launched a Java ML library last week - https://tribuo.org.

We're implementing the extra trees algorithm at the moment, an isolation forest is only a small amount of code from there.

We have a strong focus on provenance, Tribuo models capture their input and output domains, along with the necessary configuration to rebuild a model. Tribuo's also more object oriented, nothing returns a bare float or int, you always get a strongly typed prediction object back which you can use without looking things up. Tribuo is also happy to integrate with other ML libraries on the JVM like TensorFlow and XGBoost, providing the same provenance/tracking benefits as standard Tribuo models, and we contribute fixes back to those projects to help support the ecosystem. Plus we can load models trained in Python via ONNX.

To your direct question, I've not benchmarked Smile against Tribuo. We are very interested in the upcoming Java Vector API - https://openjdk.java.net/jeps/338 - targeted at Java 16, which will let us accelerate computations which C2 or Graal don't autovectorise.

- Short term roadmap is here - https://github.com/oracle/tribuo/blob/main/docs/Roadmap.md, longer term we'd like to see what the community wants.

- Yep.

- There are various efforts on the JVM to build multidimensional arrays, we're talking to many of them to try and figure out a strategy for the whole platform. Ditto for dataframes, though Apache Arrow looks like a good baseline.

- We're not looking at other languages outside of the JVM at the moment, but we're continuing to contribute to Tensorflow Java and ONNX Runtime to improve their Java support. We could look at pytorch inference support based on their Java API, but that overlaps pretty well with the things that ONNX Runtime supports. Do you have any suggestions?

- Not beyond hyperparameter tuning.

We're thinking about the visualisation story. At the moment we use notebooks with the IJava kernel to explore things (our tutorials are jupyter notebooks), but we're still working on the plotting angle.

The announcement blog talks a little about why we built this library, and where we think it sits in the ML ecosystem - https://blogs.oracle.com/datascience/tribuo-java-machine-lea...

I'm the lead developer on Tribuo, happy to answer any questions. The announcement blogpost goes into more detail about why we built it and where it fits into the ML ecosystem - https://blogs.oracle.com/datascience/tribuo-java-machine-lea...