- it says you should use 64-bit entropy as output from a CSPRNG
- it says Serial number is a signed number
If you use 64bit serial, you clearly lose 1 bit. That means any certificate with 64bit serial is not conforming to the standard.
This has been generally accepted (via discussions on MDSP and in the CABForum itself) to mean that the serial must contain at least 64-bits of entropy. Not that the generation process consumes 64-bits. If it was the latter then `head -c 8 /dev/urandom | head -c 1` (and clamping the top bit to 0) would be a permissible method of generating a serial since it consumes 64-bits of CSPRNG output to generate the serial number.
CAs are given enormous power in the ecosystem and we judge them based on their incident response and process improvement. The barrier to entry should be very high for new entrants as we strive to improve the state of the industry. This incident alone is not a disqualifier for DM's inclusion but should be considered in the context of the other issues (both technical and otherwise) raised during review of the application process.
Regarding your claim that all certificates with 64-bit serial numbers are guaranteed invalid: A certificate with a 64-bit serial MAY be valid under the standard because a CA might choose to implement the spec such that they generate a serial and prepend a null byte if and only if the high bit is 1. To determine whether or not a CA is doing that experimentally you could use services like crt.sh or censys to find all CAs with valid 64-bit serial number certificates and see if they also issue 72-bit serial numbers. If you don't see any >64-bit they're likely violating the BRs and you should submit a report to them (Mozilla maintains a contact list for CAs on their wiki) and also send an email to MDSP. Independent researchers are the lifeblood of the current wave of enforcement across the industry!
In this specific case, they have the option to use 160 bit fields for the serial number. They chose to use the minimum required of 64, apparently out of performance concerns for embedded devices with limited memory/compute.
The CA platform (EJBCA) they use defaulted to this behavior, and from what I was able to understand[0] are in the process of pushing out an update to all their customers to resolve the issue. It is very likely that someone will start going through the CT logs and we will see many reports of insufficient entropy in cert serial numbers.
[0]: https://groups.google.com/d/msg/mozilla.dev.security.policy/...
Mozilla doesn't have a big team of investigators wandering about looking for possible problems. The CAs could dob themselves in (filing a "problem report" that says they broke this rule and what they're doing about it) or an independent researcher could write to m.d.s.policy about a problem they found. Otherwise it may go undetected forever.
Most interest is focused on problems that either directly indicate a grave problem (e.g. some researcher got a CA to issue for example.com when they shouldn't because that researcher doesn't control example.com) or suggest a management problem that could hide further grave problems from us (e.g. Symantec had paperwork that directly contradicted the operational behaviour of CrossCert, so why did we notice the problem before they did?)
The 64-bit entropy rule isn't such a problem. CAs should use random values here because it protects us against a problem with the secure hash function (SHA256 usually these days). If the hash function is slightly broken the random numbers buy us time to get a new one. But if you have 63-bits instead of 64-bits, it's not like that's the difference that makes an attack viable. It's just a technical violation, it's a Brown M&M in that sense. The rules say to do it, and they didn't, so we are more worried than otherwise that they are not obeying the rules.
You have some software to get key for serial from RNG, when you get 64 bit and put it to serial field, you feel it is ok.
This is probably what happened in this case. But if requirements made sure lets say signature part is 64bit random + some extra (serial, identifier etc) forcing serial field to be one more octet, this could never happen.
That's not clear to me at all. A 64-bit signed integer can still contain 64 bits of entropy, it just has to be negative 50% of the time.