You may mean "convex hull"; you should be able to find 2D and 3D algorithms easily (Wikipedia?).

Graphically:

    X--X  X-X         X--X--X-X
    |  | /  /         |       /
    X  X-  |          X  X   |
    \    --X          \    --X
     X--/              X--/

   Possible polygon    Convex hull
   based on the points of the points
   marked X            marked X