Who cares if the profiler is making the app run slower. You aren't going to run it all the time in production!
If you can't find the memory leak or performance bottleneck, either you are doing something wrong or the profiler is not giving enough information. Getting a stack trace of where those byte[]'s are being created would be a good start.