You probably could, but I'm assuming there's just so many things that are pretty arbitrary. Take for example a dog. We breed dogs to display a huge number of differences, and only some of these are skeletal. From bones alone there's no way to tell how jowly the muzzle of the dog is, or what colour and length its hair, how musculature it is. Dogs are bred by humans, but many species have extravagant, non-skeletal features use in mating displays that are bred purely by arbitrary sexual selection and have nothing to do with diet, climate, or any other factor you could determine from fossil remains.