But then the problem is to parametrize these three vectors' positions. The first one can be define in spherical coords as (acos(phi)sin(psi), asin(phi)sin(psi), a*cos(psi)) but then I need to introduce a third parameter for vector b and write both b and c using this system...
I'll give it another try this afternoon using matrices.
EDIT: Alright, I guess you can write a, b and c as a rotational matrix using Euler angles ( http://en.wikipedia.org/wiki/Euler_angles ). You can probably also cancel one of the angles (let's say theta) by rotating your xOy plane around which doesn't change the problem.
You then just have to calculate S using phi and psi.
S = ||a^b||+||b^c||+||c^a||
and solve
dS/d(psi) = 0 and dS/d(phi) = 0
However, that's way too much trig for me for today so I'll leave it to someone more courageous than me.
You've listed 3 sides of a box. Does that mean that the "fourth" side is part of one of those 3 sides so a = b = c? Or are you asking for the orientation and relative length of the 3 sides such that the fourth side would maximize the area of the box?
... kidding :)
For example, if you think of the sun straight overhead, then the shadow will have area a*b. But if the sun moves over, then the shadow will grow by inclusion of some of the height of the box.