I know that the friend I'm hanging out with will get my jokes - inside jokes, weird humor etc. Now I have to worry about an entire group, some of whom may not like my humor, and now I have to put on the 'you're in public, be brusque and minimal' anxiety filled mask, and can't open up as I would in 1-1 or 1-2 with a friend or just a couple of friends.
Now add that to other people invited who may feel the same way. Maybe they're only comfortable with that one or two friends they hang out with, and not the rest of us they don't know.
Groups aren't my thing sadly. Nor, I assume, are a thing for many young people.