It really depends on several factors, and won't be something you crack in 5 minutes. But let's say you have a hunch of what the original data was. If it was plain text and you know what language, you can pull up statistics of which letters appear with which frequency in that language. The longer the encrypted data, the better those statistics will apply. So you just take whatever byte appears most and assume it's an "e" (or whatever) and work your way from there.

If you know the input must be a jpeg, you can look up the file structure, especially the header, so you already know what the first couple bytes of the decrypted file have to look like. Boom, you already know the first couple bytes of the key. And no matter whether you use your original method or the xor method, if you know the length of the key, for example 256, you can also immediately decrypt a few bytes every 256 bytes into the file. You can then work your way from there, using other properties of the file type you're trying to decrypt to.

If you know multiple files have been encrypted with the same key, it gets even easier as you can infer information across all those files.

The xor method would only be truly safe if your key is as long as the data to encrypt, never gets reused and was generated using a secure random number generator.