(I'm not a lawyer.) This might be a small matter of a misunderstanding, straightened out easily, with everyone happy. Corporate lawyers and HR are just doing their job, which is protecting the company's interests, which includes not giving away anything, and trying to keep the upper hand. This posture might seem scary. You need to straighten this out, preferably in a collegial fashion, but don't panic.
Talk with your manager some more, see how much s/he can help you fix this with the company. This might be getting the OK for open source work (future, and retroactive), or it might be only getting you a pass for a past misunderstanding.
If manager decides to make a case for the company supporting/tolerating open source contributions, s/he might sell it as a way to improve employee morale, enhance the reputation of the company for recruiting, improve customer goodwill/reputation for open source savvy, get free open source labor from others on software the company can use internally, or something like that.
You might find it reassuring to consult your own lawyer. Your own lawyer might have a very different perspective on some key things (e.g., reasonable personal accommodations while traveling on business), but hopefully it won't be necessary to debate the matters. If someone asks you to sign something, or your gut says you've been targeted for termination and/or legal action, consult your own lawyer.
Regardless of how that turns out, you might also need to give the leadership of the open source project(s) a heads-up on the status of your contributions, in a timely manner. They might need consider when they need to do anything now, to minimize headaches for anyone.