The downsides arise when management injects themselves into the process. "Skip testing", "don't use source control", "don't refactor", "don't start building until the spec is complete" are some of the gems I've heard before.
It's one thing to be lazy or make mistakes - we all do it. It's another beast when you want to deliver professionally but you can't because it conflicts with a company's "way of doing things".
There are basically two ways to approach a career as a programmer, one is to see programming in itself as your passion and solving the programming problem you're facing at any given moment as an ends in itself. The other is to see programming as a force multiplier to give yourself 'super powers' for doing the thing you actually care about.
At least that's my experience of venturing into a non-hardcore SW field (health sector).
The important part is to understand, enjoy and respect the domain you are working in. Then working with domain experts to use software to solve problems you both care about can be both fun and rewarding. And
You are right that you will have to 'battle' their ignorant views of SW development. But those battles are rarely hard to win if you're a bit diplomatic, because people don't care to much. They might not have any version control in place when you start and might not see the need for it, but I've never experienced anyone forbidding me from using version control. Most of the time they even come around to it as being pretty good idea.
I'm just glad to have a boss who understands that I do better work when I am getting paid real money. It's surprising how many people want to boss me around for free.