Sounds slow to use RPC with JSON for communication.
It seems that Racket does allow for C-to-Racket calls. You could write a glue program that generates C code that calls Racket functions and returns data, Swift code can then be generated to call that C code. That might be faster.