Is it possible to use it without it?
The MSBuild logic, at least for Xamarin.Mac and Xamarin.iOS has been made completely open source, so all the logic to build your apps + the SDKs are all open source.
Microsoft has not open sourced Visual studio (VS) so similarly they will not open source Xamarin. You can also use .net framework without using VS. Just use command line tools