The "Soft Maximum" function

Until you get some foundational knowledge, math is one of the topics where you're better off using traditional textbooks than wikipedia. The information is arranged in a logical manner that builds on previous concepts. Unfortunately, most individual topics are better-explained in wikipedia than in most books. A suitable compromise might be using a textbook's table of contents as the order to follow in navigating wikipedia, but the articles also have the problem of not always being consistent with each other, in style, nomenclature, required knowledge, and point of view.

After about one semester's worth each of algebra and analysis, you would probably have enough working knowledge to at least not stumble around like a blind man on 50% of topics. My personal favorite analysis textbook was Rudin's Principles of Mathematical Analysis, however it's... not for the feint of heart. Seriously. It could kill a man. If you want to get the full value out of that book, you will be spending over an hour per page to make sure you understand every step of every argument.

That's where convexity comes in. There are two different things that can be convex: sets and functions. Convexity of sets and convexity of functions is related but not the same. Convexity of sets is well explained by the pictures on wikipedia. A function is convex if for any two points you choose, the value of the function stays below the line that connects these two points.

An example of a convex function is e^x: http://www.wolframalpha.com/input/?i=e^x Take any two points on the graph and connect them with a line. The graph of e^x will stay below that line.

An example of a function that is not convex is sin(x): http://www.wolframalpha.com/input/?i=sin+x You can find two points on the graph such that if you connect them with a line sin(x) will go above that line. For example x=0 and x=pi.

If the function is convex then it will have only 1 minimum. So what you can do to find the minimum is just start at an arbitrary starting position and walk down hill and you will arrive at the minimum of f if f has a minimum. Note that if f had two local minima at different heights you might end up stuck in the higher minimum if you apply this method.

Now there are several techniques to walk down hill. One thing you could do is approximate f by a line. This tells you the slope of f at the point you're standing, so you know which direction is down hill. You walk a small step in this direction and repeat. This is gradient descent.

Another way to do it is to approximate f by a parabola. A parabola has a minimum that can be computed exactly. So you compute the minimum of the parabola and jump to it. Because it was an approximation this minimum will generally not be exactly f's minimum, but close. So you repeat the process to get closer and closer. This is Newton's method.

A complication is that many optimization problems have constraints on the parameters of f. For example f(a,b) could represent the profit you get from investing a dollars in product 1 and b dollars in product 2. But you only have 10 dollars so the constraint is a+b <= 10. If you just start looking for the maximum of f by taking steps or jumping around you will have to keep in mind that a+b cannot go above 10.

Another complication is that f may not be differentiable, which makes it harder to compute a line approximation and even harder to compute a parabola approximation.

There are books written on solving these issues. This one is available online: http://www.stanford.edu/~boyd/cvxbook/

As far as the subject matter goes, a convex function can just be thought of as a function with a global maximum/minimum and no other local maximum/minimum. I.E. The derivative is zero in only one place which is the global max/min. These functions can then be easily optimized by taking their derivatives and finding the zero. It is a really nice way to optimize a problem.

I would suggest the method that works (and is proven to work by the gazillion undegraduate mathematicians and physicists who get it to work every year): get some tutoring in the topic and/or acquire some quality undegraduate (or graduate) textbooks.

Once you have the basic principles down (basically the curriculum of a degree), you will still find it hard to approach a completely new topic like "convex sets/functions/optimisations", but you'll have a method that works.

The problem is that often those applications are so wide and varied that nobody bothers to talk very much about them. You're just supposed to understand the underlying purpose and need on your own. For someone with the background, that's ok, because the background is the context that lets you internalize the math. Without the background, applications are necessary.

Of course, this is well known for basic math. That's why you learn how to add apples and subtract oranges instead of number theory and fields. At some point, though, if you want to understand "real" mathematics, you need to somehow build that critical mass and context shift to where you can get by on the general level.

My advice is to take the math you do know and go read the proofs that make it operate. Use the applications you already are comfortable with and understand the reasons why they work. Instead of diving both into generalized math and a complex new subject at once, just use your strengths.

My anecdote is that I liked math but never really had a grasp on it until I got thrown through the ringer on infinite series, linear algebra, abstract vector spaces all at once. I learned to read and understand proofs (sometimes), I learned how to prioritize when reading complex math (i.e. internalizing definitions is often half or more of the battle), I learned how to synthesize my own general mathematical arguments.

I wouldn't call myself anything close to a mathematician, but by jumping into the language and structure of generalized math, I have a lot of the tools to get somewhere with whatever field I want. Wikipedia still often confuses me, but if I want to learn it I know how to take what I learn and use it as the building blocks to a deeper understanding.

Convex functions: if you draw a straight line between the points (x, f(x)) and (y, f(y)), then f(z) is below that straight line if z is between x and y. (The green line is below the red line in the first picture.)

Optimization: find x so that f(x) is as small as possible.

Convex optimization: Same as optimization, but now you know f(x) is a convex function (e.g., the soft max).

"Do not worry about your problems with mathematics, I assure you mine are far greater." - Albert Einstein

A soft maximum might be useful if you want a differentiable function that closely approximates the max() function. By differentiable I mean you can work out the rate of change of the function at any point. With the hard maximum, the rate of change at the hard edge is not defined. Nicer to have one function to represent the RoC and not have to worry about special cases.

A related problem is that of bounding. Suppose I want to minimize the parameters in a set of nonlinear equations; I can numerically differentiate the equations to get the gradient and the Hamiltonian and then I can minimize that. But I may want to impose additional criteria, like x > 0. I could just say the function goes to some preposterously high number when x <= 0, but then we have this hard corner problem again. Instead if you use a continuous function like a logarithm to impose your bound, it affects the solution space minimally, makes for a solution that will not wander outside your bounds, and is differentiable.

Let k be 1/max(x,y) so that you have:

g(x,y) = log(exp(1)+exp(min(x,y)/max(x,y))*max(x,y)

There are some issues if the maximum value can be negative or zero, though.