In Germany we have a tax software from the government. It is called Elster, like a bird. A stealing bird. A rare moment of humor by the state.
As a Brit I do not associate the magpie with stealing per se but rather with having a fascination with shiny objects like jewellery - which also sort-of works, for the joke.
To me magpies are literally thieves. They take shiny objects which are sometimes valuable. I've heard more than one story of people who lost jewelry to magpies. Of course, even though they are territorial it isn't obvious that magpies understand that they are stealing. But their intention doesn't change the result.