Imagine you're in a hilly field and you need to get to the lowest point because some high winds are coming, the lower the better. Normally you could just look across the whole field and see which is the lowest point, but it's pitch black outside and you can't see anything.
Now hills can slope East-West and North-South. You need to move your hand around your immediate area and feel where the hill seems to be going downward both East-West and North-South. When you've found the point that feels to be sloping downward in both directions then you can move that direction and get lower than you were before. You keep repeating this process until you reach a point where feeling around you only seems to take you up again.
Closer to the math: Your height on the hill is your "loss function" (ie how wrong you are), East-West and North-South are two variables: x and y. Feeling where in your immediate area is down for both EW and NS is Partial Derivative (backwards 6 thingie) of x and y in relation to the cost function (your height up the hill). In our hill example we're trying to find ideal physical coordinates, in a NN we're trying to find the ideal weights.
Just like in our metaphor if the Hills are big and smooth you'll find the lowest possible point quite easily. If the hills are very bumpy it can be quite hard to find the lowest point.
Unlike the hill example our problem in machine learning likely has hundreds or thousands of dimenions we have to "feel" in before we can move rather than 2. This example is different than "Hill climbing" algorithms because in Hill Climbing you would move either EW or NS and see if you where lower than before. But Hill Climbing doesn't take into account that you might be going down EW but actualy moving upwards more slowly NS. The way the entire surface seems to be sloping when you put your hand on it is essentially the Gradient.