MITSardine

I might be misreading your comment but it seems to me you're linking this too closely to AI, as I believe you'd see a similarly international makeup of researchers in any STEM field in the US. If I recall correctly, something like 20-30% of professors in the US are foreign born. This would probably be even more for younger researchers (PhD students, postdocs) as there's a greater proportion of people willing to spend a few years abroad than their whole adult life.

The US is the largest research hub in the world, and it offers (or offered) outstanding conditions for research. I believe this to be as old as WW2, and it certainly didn't start with AI. Higher salaries, more diverse career opportunities (academia is more porous to industry in the US than many other countries), and the ability to hire more and better candidates for the workhorses of a lab: PhD students, postdocs, technicians, research scientists.

Re: supply side, undergraduate education (including Master's in some countries) has become basic infrastructure in a developed (or developing) country, and countries like China, ex-USSR or the western European nations have solid traditions in this regard, with many offering comparable (or surpassing) education to the best US universities in specific STEM topics. However, save for China, I believe a majority of these countries have not invested in research to match their growing pool of Master's (or even PhD) graduates.

Wouldn't a decent compromise be to extend the capabilities of the C preprocessor, so that we could iterate over types at preprocessing time?

That way, we could instantiate for a great number of types more easily, while keeping implementations in source files. You can always do an instantiation macro, but it still becomes painful when the number of template arguments (and their possible values) increases.

I know there's the Boost PP library but I personally find it incomprehensible once you try to nest two+ loops.

Surely there's a reason the CPP hasn't budged?

What do you mean by this? I've been using those libraries on mac ARM and AMD processors, are you referring to intel-specific implementations? How about the sparse matrix solver, what do you use?

This all hinges on the fact the variance is homogeneous to X^2, not X. If we look at the standard deviation instead, we have the expect homogeneity: stddev(tX) = abs(t) stddev(X). However, it is *not linear*, rather stddev(sum t_i X_i) = sqrt(sum t_i stddev(X_i)) assuming independent variables.

Quantitatively speaking, t^2 and (1-t)^2 are always < 1 iff |t| < 1 and t != 0. As such, the standard deviation of a convex combination of variables is *always strictly smaller* than the convex combination of the standard deviations of the variables. In other words, stddev(sum_i t_i X_i) < sum_i t_i stddev(X_i) for all t != 0, |t|<1.

What this means in practice is that the convex combination (that is, with positive coeffs < 1) of any number of random variables is always smaller than the standard deviation of any of those variables.

Math isn't about memorizing closed-form solutions, but analyzing the behavior of mathematical objects.

That said, I mostly agree with you, and I thought I'd share an anecdote where a math result came from a premature implementation.

I was working on maximizing the minimum value of a set of functions f_i that depend on variables X. I.e., solve max_X min_i f_i(X).

The f_i were each cubic, so F(X) = min_i f_i(X) was piecewise cubic. X was dimension 3xN, N arbitrarily large. This is intractable to solve as, F being non-smooth (derivatives are discontinuous), you can't well throw it at Newton's method or a gradient descent. Non-differentiable optimization was out of the question due to cost.

To solve this, I'd implemented an optimizer that moved one variable at a time x, such that F(x) was now a 1d piecewise cubic function that I could globally maximize with analytical methods.

This was a simple algorithm where I intersected graphs of the f_i to figure out where they're minimal, then maximize the whole thing analytically section by section.

In debugging this, something jumped out: coefficients corresponding to second and third derivative were always zero. What the hell was wrong with my implementation?? Did I compute the coefficients wrong?

After a lot of head scratching and code back and forth, I went back to the scratchpad, looked at these functions more closely, and realized they're cubic of all variables, but linear of any given variable. This should have been obvious, as it was a determinant of a matrix whose columns or rows depended linearly on the variables. Noticing this would have been 1st year math curriculum.

This changed things radically as I could now recast my maxmin problem as a Linear Program, which has very efficient numerical solvers (e.g. Dantzig's simplex algorithm). These give you the global optimum to machine precision, and are very fast on small problems. As a bonus, I could actually move three variables at once --- not just one ---, as those were separate rows of the matrix. Or I could even move N at once, as those were separate columns. This could beat all the differentiable optimization based approaches that people had been doing on all counts (quality of the extrema and speed), using regularizations of F.

The end result is what I'd consider one of the few things not busy work in my PhD thesis, an actual novel result that brings something useful to the table. To say this has been adopted at all is a different matter, but I'm satisfied with my result which, in the end, is mathematical in nature. It still baffles me that no-one had stumbled on this simple property despite the compute cycles wasted on solving this problem, which coincidentally is often stated as one of the main reasons the overarching field is still not as popular as it could be.

From this episode, I deduced two things. Firstly, the right a priori mathematical insight can save a lot of time in designing misfit algorithms, and then implementing and debugging them. I don't recall exactly, but this took me about two months or so, as I tried different approaches. Secondly, the right mathematical insight can be easy to miss. I had been blinded by the fact no-one had solved this problem before, so I assumed it must have had a hard solution. Something as trivial as this was not even imaginable to me.

Now I try to be a little more careful and not jump into code right away when meeting a novel problem, and at least consider if there isn't a way it can be recast to a simpler problem. Recasting things to simpler or known problems is basically the essence of mathematics, isn't it?

We'd be fortunate if everything that gets cited were this short (it certainly often has similar amount of useful information)

While those are certainly indicators of interest, is their absence an indicator of lack of interest? In other words, "has side projects" is sufficient to prove "likes programming", but I don't think it's necessary.

There's only 24 hours in a day, and mastering any serious curriculum already takes the majority of those.

Then there's family: some have parents to support, and/or a spouse they can't just ignore all evening to sit in front of a terminal typing git push.

Lastly, plenty of people learn or tinker on their own, but they don't all have the (socioeconomically loaded) reflex of marketing themselves with a github repo.

Of my whole Bachelor's, Master's and PhD cohorts, I haven't known one person to have a github repo outside of employment. Some were building little things on the side but sharing them informally, never in public.

What you're looking for is people with no social life, no responsibilities towards anyone or anything (even just being an immigrant can be a huge time and energy sink), and with the social background to confidently market themselves by putting what's objectively noise online.

Are junior / new grads getting sponsored for H1Bs, or senior people?

I just wanted to comment on the "out of distribution" solution the author proposes, partly for the young grads on this forum.

Going "out of distribution" in abilities also means your job prospects go "out of distribution". When you specialize, so too does the kind of position you'd be the better fit for. This can mean radically fewer possibilities, and strong geographic restrictions.

To give an example, my PhD topic concerned something "that's everywhere" but, when you look at things more closely, there's only < 10 labs (by lab, I mean between 1 and 3 permanent researchers and their turnover staff) in the world working on it, and around that many companies requiring skills beyond gluing existing solutions together, in which case they'd just as well hire a cheaper (and more proficient) generalist with some basic notions.

This isn't even a very abstract, very academic field, it's something that gets attacked within academia for being too practical/engineering-like on occasion.

I understand the "belly of the curve" gets automated away, but consider that the tail end of the curve - producing knowledge and solutions to novel problems - has been for a long time, since Gutenberg's invention of the printing press, if not oral communication. The solutions scale very well.

A researcher's job is, almost by definition, to work themselves out of a job, and this has been the case since long before AI. Once the unknown has been known, a new unknown must be found and tackled. There's very, very few places in the world that truly innovate (not implementing a one-off novel solution produced in some academic lab) and value those skills.

I don't mean to be overly bleak, but it doesn't necessarily follow from this automation that the freed salary mass will go towards higher-level functions; just as likely (if not more), this goes towards profits first.

I think it's both.

You can see it with smoking. Only 20 years ago (I'll take France as the example), you could smoke indoors in restaurants and many public places and, a few years before that, even in classrooms.

Recently, it was forbidden in public parks and similar places. A perhaps still minoritary segment of the population would like it outlawed even in e.g. restaurant and bar terraces.

Since France remains behind on this compared to the US, you can get a time-traveling experience regarding this right now, by boarding a plane from Paris to e.g. Boston or back. You won't find a terrace where you're allowed to smoke (or even vape) in Boston (at least I didn't) or, if you're American, you might find the French very comfortable smoking right under your nose.

There is no doubt these are good policies as far as public health is concerned. But they outline how, in the tug-of-war between personal freedom and safety, there has been a tendency lately for safety to win.

Other examples, as far as I'm aware, there weren't even speed limits on non-agglo roads before the 70s (France, again). Drinking (under the limit of manifest ebriety, and the tolerance must have been pretty lenient) and driving was not illegal until 1970. You could drive a motor scooter/moped including on high speed roads (though they wouldn't break 40km/h unless unbridled) from the age of 14 without any paperwork, nowadays it requires a minimal license. I'm very doubtful people didn't know this was a dangerous situation, or that they didn't care, I think they simply prioritized freedom and simplicity over living in a zero-risk society.

I point to these safety regulations because they're where the compromise between social order and personal freedom is most evident, but they're only manifestations of a broader change of the Zeitgeist, in my mind.

Another clear tendency seems to be an obsession with so-called insecurity which, according to politicians and a worryingly large portion of the French (judging by election results), is at an unbearable all time high; in fact, violent crimes have never been so rare (for comparison, murder rate is 1/5th that of the US).

Everyone seems obsessed with risk and danger, be it the right of being mugged, or the left of precarity (which I can better sympathize with). It seems one half of the population wants to live in a police state with no personal freedom, and the other wants to live in an absolute socialist state with no financial freedom.

Just to conclude, it also seems the newer generation has been found less promiscuous, which, to me, is not a very adventurous attitude.

This looks really polished. I've always found crowd and traffic simulation fascinating.

The Projects page is worth looking at too.

I don't think that's possible, there's too much data that pairs your real identity with purchases.

Even Delta's own information on your past purchases (recall tickets are nominative) could be enough to size you up.

I don't know who's responsible, but there's clear information sharing related to using a credit card in the US. E-mail address and whether you have Amazon Prime are two examples I've seen (I was never asked for permission to share this information).

Then there's smartphone-based payment methods, by Google and Apple no less. I'm sure that results in no data at all being disseminated.

I don't think it'd be too complicated for a few major retailers, Amazon and airlines to get together and have a good estimate of your income (or at least of your expenses). What are you going to do against that? Pay by cash everywhere?

It's not usually up to businesses to compensate wealth inequality by adapting prices to customers.

I personally find this appalling, not because of the prospect of paying more or less, but because of the arbitrariness and opacity of it all. I'm not too excited for a future where we'll have to spoof various browser identities and try different times of day and VPNs to book plane tickets...

At the extreme, some people could find themselves stuck or in very difficult situations because of this kind of practice. Oh, your parent is on their deathbed? Shouldn't have said that to a friend on Messenger or WhatsApp.

And I don't understand what could possibly justify personalized pricing (towards consumers, not businesses) of non-personalized goods and services being legal in the first place.

What most surprises me with this article is that "personalized pricing" is even legal to begin with.

What can possibly justify this being legal? It seems ripe for abuse.

If airlines are allowed to do this, I don't think there'll be a thing we can do, seeing as tickets are nominative.

Naive question but what would be the advantage over traditional sound isolation? (such as in music studios)

By contact simulation, do you mean solid dynamics? (mechanics is not my field)

I'm intrigued by the use of level set domains here. I've only encountered those in other type of numerical simulation where the intent is in avoiding surface meshing.

I suppose moving an object in this context is as simple as composing its level set function with a translation and rotation. However, deforming is non trivial, especially local deformations, right?

How do you efficiently resolve collisions? At the scale of an element, it seems to be a simple check of nowhere should both level set functions be negative. But how do you select the elements to check? Do you somehow keep track of only the elements traversed by the objects in a time step, or some other method? I would guess your method should be more efficient than intersecting meshes, is that what you've found?

I'm particularly interested by your mention of high-order boundary parameterizations, what do you mean by that exactly?

Sorry to bombard you with questions, I was intrigued by a combination of things I'd never seen together before!

Hey, I had a quick look at your engine's github page, what are you looking to do with surface meshing exactly?

I watched your talk, very interesting! Super inventive idea, I hope it works out.

Is the name a word play with "torgnole" at all, or does it mean something?

I think there's still a unique solution but, on the harder difficulties, you're given very little to work with. (in Towers)

Yeah, that's what I meant. On the other hand, something like Towers has you trying different configurations because there's not always enough information to motivate the next step.

I've had this on my phone for years, it's a great collection of puzzles. I haven't tried them all (games on phones), but it's certainly the best I have. No ads, no useless gamification, but well polished and varied puzzles, and quite a bit of control over the difficulty.

My favourite has to be "Keen", it's a sudoku-like where a grid has to be filled with no repeated numbers on either columns or rows, and arbitrarily shaped cells must be filled to satisfy an arithmetic constraint like "sums to 7", "the product is 84" or "one divided by the other is 3" (if sized two).

Towers is nice too, similar concept (re repetition), but the constraints are now visibility ranges on the boundaries of the grid, as you put down towers of varying height. I find it more difficult.

Some of the games are more mechanical, where you can mindlessly iterate to a solution step by step. Like "Net" (rotate pipes to connect them all to the center). Towers takes some more guess work, and I find Keen is there in the middle.

Could it be those companies are not interested in working with a small video game studio? In case they usually work with the much larger actors of the auto industry.

Or maybe they're operating at a level of fidelity much higher than the game needs? If they're working on simulation e.g. for Formula 1, they might be doing full fledged numerical simulation (FEM etc) rather than simplified models for real time games.

I'd be curious to know more though, what else do you know about this?

Is that actually a null result though? That sounds like a standard positive result: "We managed to show that minimum wage has no effect on employment rate".

A null result would have been: "We tried to apply Famous Theory to showing that minimum wage has no effect on employment rate but we failed because this and that".

If people are so interested, they'd presumably read and cite null-result publications, and their authors would get the same boons as if having published a positive result.

There's some issues, though. Firstly, how do you enforce citing negative results? In the case of positive results, reviewers can ask that work be cited if it had already introduced things present in the article. This is because a publication is a claim to originality.

But how do you define originality in not following a given approach? Anyone can not have the idea of doing something. You can't well cite all the paths not followed in your work, considering you might not even be aware of a negative result publication regarding these ideas you discarded or didn't have. Bibliography is time consuming enough as it is, without having to also cite all things irrelevant.

Another issue is that the effort to write an article and get it published and, on the other side, to review it, makes it hard to justify publishing negative results. I'd say an issue is rather that many positive results are already not getting published... There's a lot of informal knowledge, as people don't have time to write 100 page papers with all the tricks and details regularly, nor reviewers to read them.

Also, I could see a larger acceptance of negative result publications bringing perverse incentives. Currently, you have to get somewhere eventually. If negative results become legitimate publications, what would e.g. PhD theses become? Oh, we tried to reinvent everything but nothing worked, here's 200 pages of negative results no-one would have reasonably tried anyways. While the current state of affairs favours incremental research, I think that is still better than no serious research at all.

Hopefully a future step is enforcing airwave time limits/parity as is done in traditional media, for the countries that have such rules.

How this can be done is another question, not just from a technical standpoint...

An important issue is the lack of transparency in campaigning expenses, which are capped in several countries. This is the principle that underpins the Sarkozy-Kadhafi affair (illegal presidential campaign funding) or the Brexit Cambridge Analytica affair (though that goes beyond that).

This has been my reaction in Oslo. I saw so few cars I thought they might be banned (but I don't think that's the case). Additionally, a great deal of cars in Norway are electric already. From what I can gather, 95+% (!!) of new cars bought in Norway are electric, and 30+% of cars as a whole.

Combine the two and it feels like walking in the middle of a Covid lockdown.

A gradient descent is used to solve optimization problems, those arise in many many cases unrelated to ML. Please research a little the history of this field, and notice how it predates ML by decades (even 180 years in the case of the gradient descent specifically).

A great deal of applied mathematics is related to finding a minimum or maximum quantity of something. There are not always constructive methods, sometimes (often) there's no better way than to step through a generic optimization method.

Some quick examples clearly unrelated to ML, and very common as they relate to CAD (everywhere from in silico studies to manufacturing) and computer vision:

- projecting a point on a surface

- fitting a parametric surface through a point cloud

Another example is non-linear PDEs. Some notable cases are Navier-Stoke's equations, non-linear elasticity, or reaction-diffusion. These are used in many industries. To solve non-linear PDEs, a residual is minimized using, typically, quasi-Newton methods (gradient descent's buff cousin). This is because numerical schemes only exist for linear equations, so you must first recast the problem as something linear (or a succession of those, as it were).

By the way, I might add that most PDEs can be equivalently recast as optimization problems.

Yet another is inverse problems: imaging (medical, non destructive testing...), parameter estimation (subsoil imaging), or even shape optimization. Similarly, optimal control. (similar in that it is minimizing a quantity under PDE constraints)

To summarize, almost every time you seek to solve a non-linear equation of any kind (of which there are many completely unrelated to ML), numerical optimization is right around the corner. And when you seek to find "the best" or "the least" or "the most" of something, optimization. Clearly, this is all the time.

I think I've provided a broad enough set of fields with ubiquitous applications, that it is clear optimization is omnipresent and used considerably more often than ML is. As you see, there is no association from optimization to ML or AI, although there is one the other way around. (much like a bird is not a chicken).

I see!

For your second point, you can decouple ODE time stepping and frame time stepping. I think it suffice to step until the lowest time ODE time step that is greater than the current frame time and interpolate between that and the previous ODE time step.

This technique is used in loosely coupled systems, for instance in mechanics where a rigid body needs a much lower time step (higher frequencies) than a soft body to compute the dynamics of, but you still need common time steps to compute interactions. Often times, the time steps are dictated by CFL conditions, and they may not even be integer multiples of each other.

However your first point is where I see the iterative approach really wouldn't work. Especially if the user might change the parameters before you've done anything with them, it wouldn't make sense to precompute values using the iterative scheme. Otherwise, that could be done, and then values interpolated between steps.

If you have few parameters, one solution that comes to mind if you cannot find a closed form solution is to grid the params space, and precompute the curve at each point of the grid. Then, when the user requests any value, simply localize that in the grid and interpolate from the nodes of the element. It becomes problematic if start and end points are parameters, though... In that case I suppose a linear transform of the curve to fit the end points precisely would be in order. You can consider the end points alone, it wouldn't be very complicated.

An improvement would be an adaptive grid; each time a point is inserted in a hypercube, compute also at the projections of that point on the hypercube's facets (and then the projection of that onto the facets' facets, and so on...), and split it. Consider whether to insert based on an indicator that takes into account the solutions at the nodes of the initial hypercube (if they are too distinct, a split is in order). Maybe this is too much complication for something this simple, but anyways there would be solutions for using more difficult ODEs that don't have closed-form solutions. Note if a single family of ODEs is considered, then this can be done offline, or online but cached.

There are more sophisticated methods like PINNs of course, but I don't know if you'd gain anything in performance versus a bespoke scheme. In particular if the inference step is more expensive than localizing in a kd-tree and interpolating from a few points.

At the same time, the cos and exp are also computed using addition and multiplication. Unless the values are tabulated, I imagine you'd use a series expansion. Or an even more straightforward argument is computers only know how to do addition and multiplication...

In the case of the exponential, if you're willing to start from a known value (say 0, where exp(0) = 1) and need values until the last, then using the very definition of the exponential is even more straightforward... this is the only function s.t. f' = f and f(0) = 1. In other words, step through it! The most natural definition of the exponential is as an ODE to begin with.

Tangentially related, this is one of my favourite articles "Nineteen Dubious Ways to Compute the Exponential of a Matrix":

https://www.cs.jhu.edu/~misha/ReadingSeminar/Papers/Moler03....