So if you drill through the crust to the mantle, will magma come up the hole? Will you create a little volcano?
Regardless, the mantle isn't molten. It's different types of rocks, but they're still rocks.
In fact, the mantle is exposed at the surface in a handful of locations (and a good chunk of the ocean floor). We have learned a lot from what's exposed at the surface, but these rocks have been significantly chemically alerted. Many of the minerals we expect to be there are unstable at near surface conditions. We'd learn a _lot_ from "fresh" mantle rocks
If you were to maintain the same temperature but drop the pressure to surface conditions, yes, deep mantle rock would melt. A hole with nothing in it going deep enough would cause a volcano (even if it didn't hit the mantle). However, not all of the mantle is hot enough for that to occur. (Again, the mantle is quite shallow beneath the oceans) We also have to keep the pressure at the bottom of the hole the same as the rock around it, so a well doesn't usually release pressure.
At any rate, in any case where you'd try to drill into the mantle, it would be impossible for a volcano to form.
All that having been said, geothermal drilling in Iceland has hit magma chambers by accident and caused a "mini volcano".
The mantle is solid. You would have to drill way deep into the mantle before you encountered liquid rock.
Volcanoes don't work like that. Even if you drilled directly into a magma chamber, not much would happen. This has been done inadvertently a handful of times. https://phys.org/news/2009-06-scientists-drill-magma.html Basically volcanoes only happen when there's an extreme amount of pressure from below.