My personal opinion: this is some randomly fixed "we are selling these"-price to show they are not IBMs-"ask for a 6 figure-quote" price. So they might very well be very interesting if you have a budget in the upper six figures (because, the 40k won't be the final offer for sure...); everyone else (like the lower half of the 6-figures) knows: nice, but not my price.
Actually it looks like it's just the opposite of what you describe. Fujitsu build a CPU that's very good at running a wide variety of applications. Unlike x86-64's that need accelerators for good FP or memory bandwidth per node, the A64FX does not need CUDA applications. Plain old fortran is just fine.
Keep in mind the $40k includes support for porting applications, it doesn't mean that if you want 1000 of them that you'll have to pay $20k each.
For another F90 benchmark (Himeno) they got 4 times faster than a dual Xeon 8168 AND 1.1x faster than a Tesla V100 running the CUDA version of the same benchmark. IMO that's pretty amazing, no source code changes and you get BETTER than GPU performance without having to rewrite your code.
So sure a standard Epyc or Skylake refresh will run your code and "just work", the a64fx opens up the possibility of significant performance upgrades with no source code changes. If your code isn't currently rewritten for CUDA getting 10x the bandwidth and 4x the CPU performance would be really attractive.