Small stacks used for goroutines don't play well with calling C functions which assume huge stacks. That's price you pay for having nice scalability story. There might be other issues, which come from the fact that Go has elaborated runtime.
The workaround is usually to batch your computations on the c-side (micro-rpc) and minimize the calls (like cockroachdb is doing)