Three specific complaints he has involve a belief in infinity (and limits), which he asserts do not exist in the real world; a delay in the publication of a pair of papers on which he worked, due to a variety of circumstances; and a "pernicious" influence of axiomatic mathematics which leads to "stupid" questions such as Hilbert's Second Problem.
The failed publication of one paper is particularly notable, as it claimed a counterexample of Fermat's Last Theorem (according to Dr. Zeilberger's "Opinion 123", on his Rutgers website), and (ibid) was recognized by Andrew Wiles himself as one of three possible counterexamples: the reason given for this oversight was in fact the acceptance of the decidedly non-rigourous statement "it is easily seen ..." (ibid). This particular instance seems to contradict the main thrust of Dr. Zeilberger's rant against a mathematics overburdened by rules.
Unfortunately, little example of what mathematics SHOULD look like is offered - beyond a statement that 'obvious things should be treated as such' and an assertion that infinities and continuities do not occur in real life/nature/the universe. While such a statement may be understandable coming from a respected (and clearly accomplished) combinatorist such as Dr. Zeilberger, physics has yet to demonstrate conclusively that space and time are discrete - undisproven interpretations of quantum mechanics exist which allow for continuities, and 'the size of the universe' is defined as that which we can see (ie, within ~13.8 billion light-years of Earth/Sol) so that actual infinities are ruled out only because of our inability to perceive them.
Perhaps Dr. Zeilberger needs to think outside his own discrete box.