vrudyi

Have you seen any interesting LLM applications besides marketing tools and PDF parsers recently?

Good question. There are limits on the max query execution time (30 minutes) and the number of parallel queries that can be executed at the same time.

There is a separate page about service limits here: https://docs.aws.amazon.com/athena/latest/ug/service-limits....

Thanks a lot, I've just extended the post with this information.

I think there may be better options than Step Functions(or that can be used from Step Functions) for this particular purpose. Depending on the processing required, passing data further to something like recently announced AWS Batch can solve such problem.

Amazing, thank you. I haven't read it, but definitely, will.

Obviously, there won't be a strictly mathematical solution. Neither optimal one. We can try to create something that is good enough for a limited set of problems, though. If it works at least in half cases, then I'm happy.

I don't think we can create a methodology or magic something. But having a repository of estimates for similar projects could be great.