I can only guess the format for archived tweets is very different from newer one, maybe just a compressed text file, which the engineers have to automate extracting and parsing at scale. Doesn't seem that hard, but the format was probably a hack job back in the day and there are probably a lot of considerations, e.g. metadata like RTs and favorites.

Plus, CEO's comments sound like it's been set up as a side project. The kind of management style books like Mythical Man Month and Peopleware warned us about.