Sorry, but are you suggesting that transformers have been used by quants prior to the language application? LLM these days implies a transformer architecture.
(1) https://arxiv.org/pdf/2208.08300.pdf
Aug 2022, TRANSFORMER-BASED DEEP LEARNING MODEL FOR STOCK PRICE PREDICTION International Journal of Computational Intelligence and Applications
Something similar was an exercise/project in the NYCDSA "Data Science Boot Camp" I did nearly 2 years ago. I am fairly sure that applying these models to language was secondary, right?
> The transformer model has been widely leveraged for natural language processing and computer vision tasks, but, to the best of our knowledge, has never been used for stock price prediction task at DSE.
In general, all the fancy DL stuff from the last decade started on either images (e.g. CNNs) or text (e.g. RNNs, transformers). Applying them to tabular data is generally an afterthought, and for time series in particular is rarely worth the time and compute.
The paper you posted is not very good and should not be used to inform your own work. It includes no comparisons to baseline models, and even the abstract describes the results as "acceptable".