House and urban slaves weren't much worse off than a regular citizen, having a good chance of manumission (release), after which they became freedmen - still not full citizens, but their children would be, and some freedmen became very rich.
Given that those slaves are the ones that we know the most about (from both archeology and written sources), it skews our perception of Roman slavery to being more benign than it actually was.
Rural slaves which worked in agriculture were worked very hard and they were much more numerous than house slaves. Not an enviable fate, though one must admit that ancient agriculture was a backbreaking work regardless of personal status.
Slaves sent to mines were basically sentenced to death. Few of them would last more than five years. Ancient Rome had very extensive mining operations and mines "ate" men by the thousands. These were mostly enslaved men, because they were cheaper and could not say "no". Mining safety back then was basically zero.