In math, rec letters are the most important, and whether rightly or wrongly, math professors don't give a damn about the word of anyone other than other math professors.
Figure out how you will be spending your first two years, and the coursework in particular. At good programs it will be killer hard. Are you prepared for this? Your rec letters need to come from people who thoroughly understand how difficult graduate work is, and who can credibly assert that you will succeed in it. This needs to be based on your past success, presumably in your coursework.
In particular, top programs look for not only talent but also thorough preparation. Have you gone far beyond your university's requirements, and absolutely excelled?
If you want to get a Ph.D., and don't get into top schools, you can probably go to a mid-tier place, and if you are really thriving, file transfer applications your second year. Or just finish at the mid-tier place. I went to a non-top-ten university for my Ph.D. and am doing very well now.
Good luck to you!