Advanced R - http://adv-r.had.co.nz/
R Packages - http://r-pkgs.had.co.nz/intro.html
I also write and contribute to R packages at work and these two books have been really useful. That being said, Advanced R isn't the best read because a lot of it is just listing exceptions and strange behaviour of the language but it's useful info, especially the chapters on non-standard evaluation and the OO systems. Similarly, figuring out how the package system works can take a bit of experimenting - just stick with it, try things out and pay attention to how things are done in the existing codebase.
Also, your colleagues are right - tidyverse isn't suitable for heavy use in packages, it's for interactive use and EDA.