Go to Fastmail → Settings → Masked Email. There, click the "Edit" link next to the address you want to delete. On the following page, there's a big red "Delete" button. Works for me. :)

Update: looks like deletion only works when using the website, though.