I fear it will be difficult for me to do this and not make them social outcasts. Part of being a parent is also helping your kids fit in and be successful "socially". I think what you're doing is good. Hopefully enough parents do this that it doesn't seem weird when a little one doesn't have a phone.
My one year old can already steal my phone, hold it, and scroll/mess with buttons with her thumbs, open camera to look at herself... so I try not to use it around her anymore.