I decided to just start with a robust set of media queries and then build my site from there. I can control the code, the UI/UX and pick and choose which plugins or design patterns I want to use. For me, this has been the best option.You can control all the code in Bootstrap. Download it, edit the hell out of it. That's not really an advantage of rolling your own.
Does it take a little longer? Sure, but in the end, I control all the code and can pivot and modify the code a lot faster to suit my clients.
You can modify your own code faster to start with, but really that's just because you haven't learned Bootstrap. Once you know it well it's no harder than changing your own code.
It also saves time having to dig through oodles of documentation to try and find out if some functionality my client requested is supported by Bootstrap and how to implement it.
Digging through Bootstrap's documentation is pretty much always going to be faster than writing your own code.
Building something in-house is very, very rarely a better option than spending time learning something that tens of thousands of people use every day. You simply can't test your own code to the breadth and depth that the community goes to with Bootstrap. This is the same reason why we don't all roll our own jQuery equivalent or SASS compiler. Open source's big advantage is that it's a collective effort that we can all freely build upon.
Lastly, one massive advantage of Bootstrap is that other people know it. Once you get to the point where you're working on projects that need more than one front end developer that is a huge advantage over an in-house library because all the advantages you see for your own code the other developers will see for theirs.