My thinking was that editing columns would be a rare even - you'd set your board up once and then use it with perhaps an occasional decision to add or delete a column.
Your real-estate is valuable and should remain uncluttered - a UI element (be it label, icon, button) must earn it's place especially in prominent positions - the top right of a column is (in the western world) a valuable place plus each column has a limit to how many UI elements they can support before getting confusing.
Thus, the decision you have to make each time you place a UI element is where are the prominent positions where UI can go? What are my most common interactions? Where can I put my less common, yet still essential, interactions?
So - here the obvious thing to do is to concentrate on the cards - creating new ones, moving them from column to column, deleting them. I'd give prominence to these interactions as their buttons are the ones that will be consistently pressed.
Then you are left with what to do with the ui for setting up a board. You generally have two options - chuck the UI into a lesser-used area of the scene (generally a lazy idea) or (my preferred option) add a mode.
Modes get a bad rap as they are considered to be 'non-discoverable' but I find that a good mode is often a simpler solution - you get a clean slate and all your good UI positions are available for reuse.
SO - a good column editing mode would involve an option to switch into a mode where you can no longer edit cards but instead manipulate the board - add/delete columns, move things around.
There are various possibilities for entering the mode - the most obvious of which is a button. However you could play around with clicking the area between columns - as its a large unused target area. The downside might be that it isn't obvious what you'd have to do to discover it - the temptation then is to add instructions.
Have a play - I suspect it might work.