Before it was a settlement it was a military base. It did have walls protecting until 1940, where under WW2 Japanese occupation they were demolished. The name stuck even after. The south gate of the wall remains as part of Kowloon Walled City Park

You can see an illustration of the wall in the infographic here http://archive.today/2omPT