A Practical Cryptanalysis of the Telegram Messaging Protocol [pdf]

Core flaw: Telegram uses a block cipher mode that requires padding to the cipher's block length, which means messages end in a variable amount of random bytes. Telegram "MACs" plaintext (really: applies a SHA-1 hash to it), but doesn't include the padding in the MAC, which means attackers can extend messages with arbitrary amounts of attacker-controlled padding.

From that affordance, this paper comes up with an error oracle that relies on message alignment.

There's more not to like than this:

* The protocol is MAC-then-encrypt; it has to do a large amount of work and validation before attempting to cryptographically validate the message.

* They use a nonstandard padding scheme which requires them to rely on a message_length field to strip padding; conventional CBC-like applications (with respect to padding, IGE is like CBC) would use PKCS#7 padding, where the padding itself describes how to strip itself. That also has problems, but the message length solution requires them to do calculations with respect to plaintext length, padding length, and the body all before the fingerprint of the message is checked.

This protocol leaves a lot of room for attackers to invent cute tricks with message lengths, block swapping, and field validation. Modern crypto gets around all these problems by ensuring that messages are cryptographically authenticated before they're encrypted.

The author's point, which is correct and pretty much the universal opinion of cryptographers who have looked at Telegram is that MTProto should have just used a conventional AEAD mode rather than inventing their own weird thing.