It's no different than trusting code in a 3pp library. I'd vet the code before running it. In some cases, you are giving the running container root access to your computer. Take a look at the Dirty Cow to exploit that allows you to break out of the container and control the host.
https://blog.paranoidsoftware.com/dirty-cow-cve-2016-5195-do...
You never know what people will put into their images. I built a tool to help reverse engineer images to find files people put in their images here.
https://github.com/P3GLEG/Whaler
I have found so much weird stuff that people build into the containers. I wouldn't trust them implicitly because it's a packaged black box.