Maybe I can connect the dots for you. Optimization is the general problem of finding the minimum of some function f in a region. The problem with this is that it's
hard. One of the reasons why it's hard is that f could have many
local minima. So we look for conditions on f that make the problem easier.
That's where convexity comes in. There are two different things that can be convex: sets and functions. Convexity of sets and convexity of functions is related but not the same. Convexity of sets is well explained by the pictures on wikipedia. A function is convex if for any two points you choose, the value of the function stays below the line that connects these two points.
An example of a convex function is e^x: http://www.wolframalpha.com/input/?i=e^x
Take any two points on the graph and connect them with a line. The graph of e^x will stay below that line.
An example of a function that is not convex is sin(x): http://www.wolframalpha.com/input/?i=sin+x
You can find two points on the graph such that if you connect them with a line sin(x) will go above that line. For example x=0 and x=pi.
If the function is convex then it will have only 1 minimum. So what you can do to find the minimum is just start at an arbitrary starting position and walk down hill and you will arrive at the minimum of f if f has a minimum. Note that if f had two local minima at different heights you might end up stuck in the higher minimum if you apply this method.
Now there are several techniques to walk down hill. One thing you could do is approximate f by a line. This tells you the slope of f at the point you're standing, so you know which direction is down hill. You walk a small step in this direction and repeat. This is gradient descent.
Another way to do it is to approximate f by a parabola. A parabola has a minimum that can be computed exactly. So you compute the minimum of the parabola and jump to it. Because it was an approximation this minimum will generally not be exactly f's minimum, but close. So you repeat the process to get closer and closer. This is Newton's method.
A complication is that many optimization problems have constraints on the parameters of f. For example f(a,b) could represent the profit you get from investing a dollars in product 1 and b dollars in product 2. But you only have 10 dollars so the constraint is a+b <= 10. If you just start looking for the maximum of f by taking steps or jumping around you will have to keep in mind that a+b cannot go above 10.
Another complication is that f may not be differentiable, which makes it harder to compute a line approximation and even harder to compute a parabola approximation.
There are books written on solving these issues. This one is available online: http://www.stanford.edu/~boyd/cvxbook/