An auto parts store or Ace Hardware should stock fuses.
For your example, from the fuses search page (https://www.digikey.com/en/products/filter/fuses/139) you can apply the 2.5A, fuse type "Cartridge, Glass", mounting type "Holder" and still have a wide variety of options starting at $0.25 per fuse. No free shipping though.