TLDR: It is never too late to live up to your perceived potential. Let go of the past, decide what potential you want to live up to, and start making small decisions today to achieve that potential.
I'm sure this will be an unpopular opinion, but it really is in the same vein as what a lot of other commenters are saying here: Stop dwelling on your past. If you really are intelligent and have any sense of drive and desire left in you, stop blaming the things in your past for your "lost potential". Unless you are still in those situations and they truly are stopping you from PHYSICALLY making decisions to get out of those situations, the only one holding you back now is you.
My life was pretty easy growing up so I can not relate to your past and my heart breaks that you had to endure such hardships - no one deserves that. You didn't deserve that.
What I do draw from as evidence that potential is never wasted because of a hard background are my father and maternal grandfather.
My grandfather grew up poorest of the poor in rural US after the depression and had nothing, no money, and couldn't go to school. At 40 years old, after a stint as a Merchant Marine and years of odd jobs with a wife and 9 kids, he went out to home construction sites and asked to work on the sites for FREE to learn the various trades. As he learned to build houses, he built a home and restaurant for his family and my grandmother ran the restaurant successfully almost for 60 years until her death at 93. When my grandfather died in his 80's he was a multi-millionaire.
My father's story isn't as "glamorous" but has more connections to your own story. He grew up in a poor neighborhood with other poor families living in multi-family housing. His parents were heavy drinkers and in-and-out of work, and his father and brother were in and out of jail. When he was young, he seldom had shoes or clothes that fit, he'd have to avoid fights on his way to and from school, and certain teachers and people indifferent neighborhoods would discriminate against him because of his family and their religion. The one thing he was able to cling to during his youth was soccer and even though he did not become the pro he dreamed of, it helped him to avoid the life he otherwise might have fallen into.
As he grew up he saw the writing on the wall and did not want to let the circumstances of his upbringing dictate his future. He moved to the US in his 20's, met my mother and had 3 children. He spent much of my youth coaching soccer at the local and college level. He worked every day to provide for his family, purchased a house he never could have dreamed of as a child, and for all accounts has reached a level of success and happiness that his child self could only have dreamed about.
Both my grandfather and father told me independently that our past makes us who we are but does not dictate what we are capable of.
It is never too late to live up to your perceived potential. Let go of the past, decide what potential you want to live up to, and start making small decisions today to achieve that potential.
CAVEAT: Depending on your past and the trauma you still carry from it, it might be vitally important that you seek therapy or someone else to talk to to help you move beyond it. The baggage we carry, even though we were strong enough to get through it, stays with us. Sometimes we need assistance dealing with so we can clear our mind and soul enough to overcome it.
Good luck. I hope there is still a lot of time for you to achieve your potential.