agibsonccc

They picked such a horrible name for this. Yes it's catchy but please understand what it actually is.

You have every ML vendor (disclaimer: My competitors) following google with this name trying to get a piece of the hype pie spreading more confusion in their marketing material. It drives me nuts.

You have folks doing everything from claiming hyper param search is "automl" to transfer learning + grid search + "insert random architecture search" here is magic that will save us all from needing to understand how this stuff works. People it takes more than that. Please read the papers for yourself down below and try to understand the limitations of these techniques.

Granted, it's great that we are attempting this, it's a real step forward, but please call it for what it is.

Here's the papers referenced in the original blog post(https://www.blog.google/topics/google-cloud/cloud-automl-mak...) :

Learning Transferable Architectures for Scalable Image Recognition, Barret Zoph, Vijay Vasudevan, Jonathon Shlens, Quoc V. Le. Arxiv, 2017.

Progressive Neural Architecture Search, Chenxi Liu, Barret Zoph, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, Kevin Murphy, Arxiv, 2017.

Large-Scale Evolution of Image Classifiers, Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Quoc Le, Alex Kurakin. International Conference on Machine Learning, 2017.

Neural Architecture Search with Reinforcement Learning, Barret Zoph, Quoc V. Le. International Conference on Learning Representations, 2017.

Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning, Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, Alex Alemi. AAAI, 2017.

Bayesian Optimization for a Better Dessert, Benjamin Solnik, Daniel Golovin, Greg Kochanski, John Elliot Karro, Subhodeep Moitra, D. Sculley. NIPS, Workshop on Bayesian Optimization, 2017.

My cofounder and I are remote now after having lived together for 2.5 years. I think a lot of remote first companies do this, but it only works based on a good working relationship. 99% of the time, a lot of folks will tell you it doesn't work, but you'll also find the bulk of companies aren't remote and can't get work done without an office in the first place.

It really comes down to your working relationship with your cofounder rather than location.

Honestly, your cofounder doesn't sound like someone I would want to ever work with. We both did what it takes and shared duties like that.

That being said - your situation doesn't sound healthy in general. Working remotely without a proper structure or know how of what you're getting in to usually leads to bad professional relationships and strain. This is due to meetings happening in person and the other guy being "left out". Communication is the biggest problem and something you should know how to cope with in general if you're working remote.

Your team doesn't seem to have the structure to manage this.

I won't comment on what will be better or worse for your living situation, I am merely stating my observations relative to your situation. For context, my cofounder and I live in different countries, not just different cities. We make it work fine and have for the last few years.

Yes and so are the other frameworks? When I said "scratching our own itch" google is building it for their own strategic initiatives ranging from hiring to google cloud. FB uses pytorch for their research, caffe2 for deployment. CNTK is used by microsoft. Onnx was a joint effort between FB and microsoft to compete against TF's file format.

I'm not sure what your point is. What I was attacking was this "conspiracy" that startups and these companies are somehow out to get each other. Of course there is competition and various strategic reasons folks implement their own frameworks. We ourselves implement our own framework that imports all the python frameworks and runs them in production on the JVM and big data stack. Imagine that, I do it to sell licenses.

The "startup" the above was alluding to was PFN in Japan. FB supposedly "stole" ideas from chainer. And yes they did, they even say so as such.

It doesn't mean there's a conspiracy, it's just smart to do. If something is working adapt it for your use case. That's all that happens across any of the major frameworks.

I'm not sure if I undersold myself a bit, but I just want to say this is all I've been doing since 2013. My framework is well used by a great portion of the fortune 500 and all over the globe as well as part of a major foundation. It's not some toy, it's a commercial venture with millions in funding and a decent sized engineering team, open source foundation/community behind it.

I'm more than familiar with the space and even compete with google's business model to a certain extent.I have plenty of incentive to care about these things, but I'm still calling it out for what it is. I talk to other framework authors and have nothing but good things to say about them. We're all out there just building what we need to to suit our purposes. Yes those things have incentives, but it doesn't mean there needs to be conspiracies and trash talk.

I'll call it out again: The users are the ones who blow this stuff way out of proportion. I've seen this play out for years now.

As a DL framework author myself, I can say for a fact we all steal good ideas from each other. There's no conspiracy here.

A lot of folks not actually building the tools like to assume there's some big war going on where we're trying to sabotage each other.

In reality, we're all just scratching our own itch. This goes for pytorch, TF, as well as us.

Yeah there's occasional public debate, but we're not out to go to "war" with each other or anything. It's the end users that make this out to be something it's not.

Just food for thought here.

Few things about netlib-java.

1: It's a read only repository now. It's retired. Lack of maintenance will hurt its long term prospects.

2. The license on net lib java's native binaries are not commercial friendly

3. Net lib java does everything on heap with double arrays, we do everything off heap. There's no copying to worry about, and there's a lot lower latency and flexibility with our data buffers.

4. Due to javacpp we have better control and interop with other c++ libraries like opencv. This makes it easier to write native code and use it from java later on. This allowed us to write and maintain all of our own c/c++ code with the same api (see: nd4j there) - https://github.com/deeplearning4j/libnd4j

So yes it ends up being faster in practice for a lot of scenarios. Aside from that, we also have more control over the blas libaries we pick.

This means we also have access to cublas as well as (see below) more configuration and flexibility.

Net lib java tries to be "pure" which, while elegant, isn't practical if you want to benefit from gpus and DL. We implemented the proper shims to make things "just work" from the user's perspective there on top of having more flexibility (see: mkls opemp knobs etc)

Nd4j has its own built in garbage collector and memory management which means we don't have to worry about any strange work arounds when working with cpus/gpus and we can keep off heap buffers in a managed manner.

See:

http://deeplearning4j.org/workspaces

http://deeplearning4j.org/native

In general, "just blas" isn't enough. I know from personal experience. I wrote nd4j after trying to use every java library for matrix compute and all of them fell flat in terms of speed, interop with other c++ libraries, and the need to use java arrays was highly limiting. Over the years, we built up nd4j to handle harder scenarios.

This includes other features like distributed parameter servers among other things.

Other things aside: I like what breeze attempted but it ultimately didn't scratch the itch for me when I was looking hard at the various java matrix libraries (I've tried all of them)

When I originally built out nd4j, it has this backend architecture:

http://nd4j.org/backend.html

It was so we could just use whatever matrix backend we wanted. None of them worked well enough due to the flexibility we needed.

I also had an inherent problem with java based for loops in any setting. We wrote our own forkjoin implementation as well attempting to make it fast and it just couldn't beat plain c.

We've found especially after matrices of size 128 x 128 or so, we hands down beat every JVM out there no matter what language is. The last bit we are working on are smaller matrices.

The other problem we're working on is our sparse support could use some work. The basics are in there but it's not quite ready for prime time yet.

After that, (I'm obviously biased) I don't see how anything could compete with us. Especially after we add our autodiff/pytorch like stack on top of all these primitives.

Hope that helps!

Hi fellow YC alumni! See the pitch here: https://news.ycombinator.com/item?id=16399326

We'll be supporting PMML and the like as well. The goal is to hit the simple things rather than perpetuating the latest hype like the AutoML stuff people are going on about currently. If you'd be interested, would be happy to have a conversation to go over what we're trying to do. We hope to just provide a platform neutral tool for building and deploying models similar to sagemaker (but cloud agnostic)

This is basically all we do (Disclaimer: This is mainly meant for bigger companies, not startups): https://skymind.ai/platform - reach out if you have any questions. Mail in profile.

One thing I'll say is "just k8s" isn't realistic. You need a lot more than that.

It should run within k8s but should also be minimalistic. Not a lot of platforms provide that. A lot of our docs on the internals are still being built out yet, but we provide everything ranging from an offline install of anaconda to managed connectivity gateway to hadoop and spark clusters.

We also have built in model serving and experiment tracking (which in reality is just a relational database with a rest api automatically integrated in to the platform) - if you're interested in learning more please reach out.

I haven't had this problem either. We also tend to hire more experienced folks though. A lot of startups are hiring junior folks who might not know the best way to structure things. There aren't a ton of incentives to write tests at startups and it's hard to get right. Very different when you write an infra product though :). It likely depends on the kind of product you're shipping.

Consumer might not need that as much as enterprise.

Dislcaimer: I am not fully on bored with k8s and enable clusters that are not kubernetes as well as k8s deployments.

Is pachyderm actually used anywhere though? Especially compared to the hadoop distros?

Why would I use this over just spinning something up on EMR? If I were on prem, I likely already have hadoop installed. K8s is typically a separate cluster (mainly because it's still a bit error prone for anything with data applications, which is I'm assuming what you're claiming to solve?

Beyond that, re: kubeflow. The problem with just assembling some of these things and calling it "production" is you're still missing a lot of the basic things when people need to go to production:

1. Experiment tracking for collaboration

2. Managed Authentication/connectors for anything outside K8s

3. Connecting with/integrating with existing databases

4. Proper role management: Data Scientists aren't deploying things to prod (at least customer facing prod at scale where money is on the table, "prod" could also mean internal tools for experiments), they typically need to be integrating with external processes and different teams.

Many of these things are left as an exercise for the reader (especially in k8s land). Granted, tutorials and the hosted environments exist, but nothing is a "1 click" deploy that is being promised on any of these "See how easy it is!" blog posts that run something on 1 laptop.

A lot of things being promised here just don't line up for me yet. K8s is maturing quite a bit but when the story is still "Run managed k8s on the cloud or spend time upgrading your cluster every quarter" - I have to say it's far from close to anything a typical data scientist running sk learn on their mac are going to be able to get started with.

The closest I've seen to that that isn't our own product is AWS Sagemaker which actually solves real world problems by actually gluing components together in a fairly seamless way.

Let's just be clear here that we still have a long way to go yet. In practice, we have separate teams that need to collaborate. Data scientists aren't going to download minikube tomorrow and go to production without someone else's approval and going through a huge learning curve yet.

We're moving in the right direction though!

Check out LargeVis then. The original github repo by the authors has a sparse mode: https://github.com/lferry007/LargeVis/

TSNE didn't scale well no matter what you did to it (hence the newer technique) - in general though you can't beat cosine similarity of TFIDF vectors as a baseline.

For those interested in this space, folks might like some of the work that went in to visualizing T Distributed Stochastic Neighbor Embedding and the successor to that LargeVis which use multiple forms of KNN trees each with their own trade offs:

https://arxiv.org/abs/1301.3342 (Barnes Hut)

https://arxiv.org/abs/1602.00370 (LargeVis)

One uses Vantage Point Trees which sudivides the space in to quadrants, and the other uses Random Projection Trees.

Both are interesting exercises in using KNN like techniques to visualize high dimensional spaces.

Then use their docs. In general, I would follow a few tutorials specific to red hat. For now, decompose it focusing on using jaxrs for now. The only other thing might be JNDI for accessing data sources. The rest will come from that.

This might be a hard pill to swallow...but it's going to be highly dependent on your app server.

Java EE is just a set of annotations with several different vendors implementing the standard in an app container.

I would look at the docs for whatever your employer is using.

A lot of it comes down to knowing some basic fundamentals about web development. For example: Load balancing/HA can be done in several different ways not tied to Java EE, the same is true of any of the things you're mentioning.

Largely, HA can be done with your app server. An API is just REST which means the jaxrs implementation.

Multi threading dynamics are typically app server specific. Generally "let your container do it" is the right answer though.

Source: Been doing java since spring 2.0 as well as dealing with app servers ranging from glassfish to jboss (now wildfly).

Yeah sorry, I just assume people click profiles :).

No we're a horizontal AI platform vendor with a strong focus on anomaly detection in time series applications. Our main product is a competitor to AWS sagemaker for on prem and hybrid cloud deployments.

I'm a YC founder with 7 mill in funding and 40 employees. We already are :).

Skymind is a weird mix of joint ventures in asia, US and Chinese investment. We're not looking for an exit anytime soon. We are looking to build a big business though.

The "moat" is a land and expand strategy. There is lockin with our tooling. It's a standard on prem play. We're hard to get rid of once you install us. We help in house teams compete with external vendors. Interest actually aligns there. Happy to elaborate a bit otherwise.

We're "horizontal".

We would not fit his definition of a vertical specific startup. (We have also been around a while though) The bulk of what we do is time series.

Applications we do for real paying customers include:

Detecting theft of power on the raw grid

Online payments fraud

Detecting people stealing from the telco network

Detecting faults in assembly line machines

Detecting computers about to fail

Detecting root cause of dropped calls

Kind of researchy, but we've also done robotics with RL to teach a robot to learn an obstacle course.

(Disclaimer: I created and maintain this library which is now apart of the eclipse foundation):

http://nd4j.org/ - in built GPU garbage collector and everything.

If you want raw cuda primitives (not generally recommendended and hard to do right) - you can take a look at our javacpp based (we also maintain this) cuda bindings: https://github.com/bytedeco/javacpp-presets/tree/master/cuda

Unlike jcuda (which people typically recommend despite not being updated as often) we actually depend on this for the nd4j and deeplearning4j projects.

These cuda bindings are meant to be a 1 to 1 mapping to the cuda api as well. Hope this helps!

If you want a fairly small and minimalistic look at the underlying c code which uses cuda take a look at: https://github.com/deeplearning4j/libnd4j

All of this is published on maven central for you and runs on linux, windows and even mac. It's also the same api. All you do is switch the backend.

Email is in profile.

We see folks using our linalg library for this and deep learning: http://nd4j.org We maintain our own c++ and cuda stack underneath this as well. It also allow control of these native components from java. We implement everything from our own garbage collector for cpu and gpu to our own cuda kernels.

Cuda4j is only implemented in the ibm jvm and last I saw was slow to add newer cuda versions.

This is what I did. I hired someone already local and he handed the company to me (didn't want risk, mainly wanted the salary, we were already well funded) Be aware it does take a bit of capital though.

Ahh yes sorry for the typo. Fixed!

And it will be possible but again support isn't actually implemented by anyone yet.

They just "signed on".

Tensorflow is the only supported format right now for tensorrt.

I just want to call out something with onnx. It will be slated to improve this year but be very careful what you try to do with it. A lot of basic things don't work yet.

See: http://pytorch.org/docs/0.4.0/onnx.html

The other frameworks are using (in my opinion) slightly misleading language as to how "ready" it is. onnx to Caffe2 is facebook's use case so I could see that being supported fine. It's hard to go much beyond that though.

Support for onnx will be bottlenecked by what pytorch can export right now. The file format just hit 1.0, but it will take some time for the ecosystem around it (including the export) to mature.

Disclaimer: I am a framework vendor who has spent the last few months messing with it for end users writing model import

for tensorflow's file format as well.

The arrow integration and persistence and connecting this to jdbc would be an MVP to me that would at least allow folks to imitate pandas. While you do have new ideas here if folks can express these ideas in terms of ways they're familiar with that would likely help.

Maybe you could use calcite[1] as an engine or as a base kind of like arrow.

If you do python, maybe look in pyjnius[2].

There's a lot of things that already have connectors. I would continue along your MVP route allowing folks to do basic things with your framework first then you can improve it as you go. SQL databases aren't a bad initial target. Most folks can do SQL.

[1]: https://calcite.apache.org/ [2]: https://github.com/kivy/pyjnius

Ok so you're defining new operations on top of existing primitives. Makes sense! The concepts look more interesting than the library right now (it doesn't move the needle for me production wise yet) but it has potential! Every project starts somewhere. I'm glad you wrote this in java at least.

There's a ton of things I'd be missing to start looking at this.

1. Backend agnostic: let me run on different backends like flink/spark

2. Give me off heap memory please. Let me play dangerous and use pointers directly to optimize interactions with transforms. We wrote our own GC among other things for our tensor lib due to the GC bottlenecks and copying and the like. That's personally what I kind of like about tablesaw.

A library for 1 off adhoc analysis in memory isn't a bad start though, especially since most folks don't actually have that large of problems.

The core looks close enough to dataframes that I'd be curious to know how you compare to tablesaw: https://github.com/jtablesaw/tablesaw

This looks neat but I'm not sure why I would care about this. There's a ton of solutions out there in the ecosystem out there already with a columnar like interface.

Granted, we wrote our own as well[1] that uses the builder pattern that you then toss to an executor (our main backend is spark for this). One reason we wrote this is for persistence purposes. Being able to encode and persist a series of transforms that you can then load remotely has been very helpful for us in machine learning.

We've since migrated this project to the eclipse foundation and intend on doing a rewrite of the interface as well as integrate our baked in tensor library[2] in to certain parts of the pipeline for speed purposes and handling things like computer vision workloads.

In general, I always like seeing new takes on the columnar format processing approach but I'm just not seeing anything novel here. Clarification of intent would be great!

[1]: https://github.com/deeplearning4j/DataVec [2]: https://github.com/deeplearning4j/nd4j

I'm usually a critic of my home state for a variety of reasons but I have to say it's been impressive to watch them change over time.

Whatever folks think of Ford and Quicken Loans as companies, it's great to see the public sector being open to new ways of doing things like this (within reason of course)

I still hate the weather among other things there, so I won't likely be returning but it's heartening to see this change taking place! Even if they don't get the bid, I hope they continue this kind of thinking.

I'll chime in from the other side as a maintainer of a mid size project at a foundation (eclipse foundation, 8k stars, 4k forks)

We get a ton of drive by contributions of people scratching their own itch. That's most of the 3rd party open source contributions out there.

Beyond that, folks I've hired (we're an OSS company) have been grad student hobbyists that started (and stayed) remotely.

Those have also been my best hires.

I wouldn't feel "guilty" about not contributing to open source. These projects aren't built for altruistic reasons.

If you want to contribute, do it to learn something and get something out of it. Make interests align. That keeps it interesting for both parties.

I wouldn't feel "obligated" to do anything as a single developer.

There's only 24 hours in a day, people have kids etc..

If folks are interested in this, I figured some folks might like our c++ version of this (been around for 5 years ish now): https://github.com/bytedeco/javacpp

Also comes with tons of pre cooked projects already ready to go: https://github.com/bytedeco/javacpp-presets

We use it in production at skymind and it powers our whole cpu and gpu stack with built in memory management among other things.

Also has maven and sbt plugins so you can plug it right in to your workflow automatically just maintaining the java code.

One of the coolest features is the name matching so you can get semi automatic mapping.

We auto generate our JNI bindings from this.