That does seem to be a concern here. They are replacing Octane with what amounts to a-collection-of-stuff. Without a specific workset (ie. a benchmark) there is no clear way to measure one thing against another.

This seems like a mechanism to hide where you stand. Now they can just say "That's not what we find in our tests".

I'm inclined to agree that becnhmarks do become obsolete as workloads change and development targets benchmark pointscoring. The solution is to make a new standardised benchmark, or indeed many of them. If they are clear about what they are measuring then the relative perfomance of different JavaScript engines can be understood by developers to give a nuanced view of what performs well.