-anti-matter is incredibly difficult to produce (the total produced so far in labs is about enough to boil 1 litre of water)
-anti-matter is incredibly expensive to produce (currently around 1 trillion USD per milligram)
-anti-matter is very difficult to store and manage
-a lot of the energy will be given off as high energy gamma rays, which will be difficult to convert into thrust
-the high energy gamma rays will be very destructive to the spacecraft and crew (biological or robotic) without lots of shielding
-over a ton of anti-matter would be required to get a ton of payload to a nearby star at ~40% the speed of light
So don't start saving up for a ticket on an anti-matter spaceship. Or maybe do start saving, so your ancestors can take advantage of compound interest to buy a ticket in a few millenia, if we ever manage to develop the technology.