There are a whole bunch of people who insist that lots of the "baggage" of IEEE 754 is really important and must not be abandoned. For most applications it is just extra garbage and for the few there are other ways to do what they want.
The biggest saving you could make is probably foregoing exactly rounded results, and only stipulate that the result has to be within ±1 lsb of the true value. That would save multipliers from computing a bunch of bits that don't end up in the result anyway, except for the rare case where they decide the rounding. That would probably be a good trade-off for most AI chips. For general purpose CPUs I don't think it is worth the breakage.