thearn4

GMRES/Krylov spaces seem like they ought to have more of a place in deep learning. But maybe thats the mathematical part of me having a hammer and wanting to see everything as a nail.

One possibility that I haven't seen: if you have a neural network graph with a small subgraph that is not 100% explicit (i.e. I have nodes with cyclic dependency that have to be solved collectively via newton-type method), the gradient across this subgraph can be solved at the converged state by solving a standard matrix-vector linear inverse problem applied to the gradients, without needed the AD engine to follow every operation through the iterates of the non-linear newton solver.

For sparsity and conditioning reasons, in my current work we do something like this for graph-based specification of engineering systems using GMRES to find the gradients across these subsystems. We're not doing deep learning, but the underlying machinery is basically the same.

Looks like a reasonable overview of applications of dense linear algebra operations, with very specific applications in mind.

I feel like iterative Krylov subspace methods should be around somewhere, but to be honest I'm not sure what the applications space for linear inverse problems looks like in the deep learning domain. So maybe that wouldn't quite fit (despite me finding it to be pretty cool, and these methods sort of eating the lunch of most other inverse methods).

I'd round out the course maybe with a read through of Trefethen-Bau, “Numerical Linear Algebra” for those new to the topic area.

It seems like a smart space to try to develop something for. On the business side though, most of what I've seen are flashy and would probably get used less than a dozen times before being ditched for being to cumbersome.

Something like this for changing drum brakes in a car would be great. Or changing a baby's diaper.

It's probably not worth a headset for any single one of those tasks, but for one that can be used with some training platform to learn any manual task, that would be great.

An analog to Travis CI for this would be interesting and delicious.

An analog to Docker images would be something like Blue Apron or Hellofresh I guess.

I've had similar thoughts when it comes to recognizing the underlying (potential) simplicity of a phenomena of interest.

For example, consider a toy experiment where you take dozens of high speed sensors pointed a rig in order to study basic spring dynamics (i.e. Hooke's law).

You could apply "big data analytics" or ML methods to break apart the dynamics to predict future positions past on past positions.

But hopefully, somewhere along the way, you have some means of recognizing that it is a simple 1D phenomena and that most of the volume of data that you collected is fairly pointless for that goal.

I'm a mathematician, and never quite got the emphasis on "beauty" that some obsess over. Maybe I'm just too deep into the applied side. But mathematical beauty to me is unexpected terseness or conciseness in a result. Which is similar to how I would define beautiful source code I think.

We saw similar rallying in the year following 9-11 and the initial invasion of Afghanistan. With the invasion of Iraq in 2003, it became more complicated very quickly.

i.e. what is sometimes called the "Alabama Space Mafia".

I think that it would be a good place to develop ISRU capability, to enable us to do further exploration targets in the future.

I'm a bigger fan of putting a semi-permanent ISS 2.0 on the surface of the moon vs. boots on Mars. I don't work directly in exploration systems, but I'm not the only one at NASA who feels this way.

But more than anything I think we and the other executive agencies would take any strong commitment on an exploration and human spaceflight direction from congress that survives across presidential administrations over any specific technical consideration.

I.e. we're waiting for strong elected leadership.

It's pretty cool. Based on the video, by dipping at various angles it looks like the process is solving an inverse problem in the Radon or Hough transform domains, something similar to what is done in medical tomographic imaging.

I almost miss the days when everything was about Hitler. At least then it had the veneer of being historical.

Now it's about ice road truckers, pawn shops, and things hillbillies like.

I guess part of this is doctors overprescribing to prevent patient complaints, based on how much patient satisfaction surveys are used to rate their year over year performance.

I don't know if its the AMA to blame for that or not, but everyone in the system seems to think its an upstream problem.

I think there is room to exchange ideas between the nonlinear programming and ML communities for sure.

Specifically for momentum, if I understand it right it's a particular way of perturbing step size and gradient steps to prevent oscillation. There are some other good examples of this used by many gradient-descent optimizers. For example:

https://www.cs.cmu.edu/~ggordon/10725-F12/scribes/10725_Lect...

There's part of me that wonders if one interesting way forward for deep learning is a minibatch form of BFGS or SNOPT.

It isn't mentioned in the abstract, but this seems to be more of an overview of ML-specific notions of gradient descent, where batch processing is possible due to needing to leverage gradients of a fixed prediction architecture over a large set of training data, with respect to tunable weights.

So each of those training points represents a sort of separable or parallelizable piece of the whole processes, giving you a ton of freedom in how you actually execute the gradient stepping (with one training point, several of them, or all of them). As I understand it, stochasticity in this process interestingly seems to add enough "noise" that local minima seem to be avoided in many cases.

In more general applications of non-linear gradient-based optimization (say for optimizing parametric models in physical engineering), this doesn't necessarily come into play.

This is a great idea for FB and Reddit. For HN, it's a tougher call for me, I value the topics and discussions a lot more than I do most other sites. But it is probably a giant distraction when taken as a whole.

Slack is another one I've found myself trying to stop compulsively checking. I've been collecting a lot of public slack teams lately.

For another point of comparison, has there been a "us vs. them" dynamic in the computer vision community when it comes to deep learning? After all, it seems like deep convolutional networks did sort of railroad there way through a ton of topics in that field. I used to do image analysis research, but moved on before much of this came into play so I lack any sort of inside scoop.

The SR-71 story never fails to put a smile on my face.

Scrum master feels more like a role rather than a job title. Like one thing that you do among other things.

Yes, and frankly it is bad history. Development of intermediate cartridges and their adoption by NATO forces had nothing to do with purposefully limiting lethality. It's about finding a better balance between controllability at increased fire rates, combat loading, and range suitable for conflicts emerging in the 20th century.

https://en.wikipedia.org/wiki/Intermediate_cartridge

> Another problem cited for INSAS was that it performs badly in Close Quarter Battles (CQBs) for which the forces preferred AK47. However, INSAS was a field weapon and its 5.56mm ammo (designed to injure, not kill) was not suited for CQB, very much like the any 5.56mm weapon with NATO ( e.g. M16) ,is not suited for CQB.

I know very little about the INSAS. but I'm not sure I follow the reasoning here. Why does the author not believe that 5.56mm is suitable for CQB? It may not have the momentum of other calibers, but it's most certainty not a less-than-lethal system...

It's such a broad cross-disciplinary area, it's tough to define a good entry point that is reasonable for everyone. But I'd say actually start with getting a solid foundation on statistics (point estimation, and hypothesis testing).

If you're good with that, I'd learn a bit of DSP to get a feeling for how people in that world manipulate and clean digital signals. Some of that helps out later. FFT, DWT, PCA, ICA, Convolutional filtering, Kalman and PID filtering (if you're interested in online systems), things like that. They help especially in feature extraction for classification, but also for unsupervised methods.

Past that, the hierarchy of methods to be learned and in what order is funny, I'm not sure there is a current consensus on that.

The truth is that the best path kind of depends on your own goals. If you have a particular problem in your pocket that you want to solve, that actually helps a ton.

I've commented on this before, but matplotlib is sort of stuck between a rock and a hard place of supporting the cruft of MATLAB plotting syntax, and trying to be pythonic. I'm still a big fan of it because I've grown with it, but I also don't expect that it is the future of python technical plotting.

Its more domain-specific, but I do like the seaborn library.

It's a shame that the internet devolved into cat pictures, pervasive advertisements, and political shouting matches. On the other hand, there are still some great communities out there for great discussion. I consider HN to be one of them.

You're old enough now that you can actually pick who your friends are. Choose wisely.

I use OpenCV for reading and writing real-time video streams (like webcams or video frames) for my hobby computer vision projects, but tend to use other ML or image-specific libraries for actual processing. The cascade classifiers in OpenCV are okay, but it isn't too difficult to set up something comparable in scikit-learn that is more modern and robust. Though a bit less performant if you do need real-time response though.

Best example that I have is a pulse rate detector that I put together, that uses OpenCV for video frame extraction & display but bare numpy/scipy for the rest.

https://github.com/thearn/webcam-pulse-detector

That seems pretty sensible to me. When the educational funding bubble pops in the U.S., I could imagine something like that taking it's place.

That's very cool to hear, I'm a lot more interested in the eGPUs (vs. something like an AWS P2 instance) after reading this. Thanks again for sharing.

Caffeine and walk breaks every hour or two work pretty well for me.