It's not like you have to be a legal genius to know the rules. They are right in the treaty (think of it as the EU constitution) and Google know about the many previous cases interpreting the rules that share similarity with theirs, including Microsoft's case about bundling Explorer with Windows and making it the default browser.
So Google and their hundreds of lawyers are well aware of the rules against abusing dominant position. Nevertheless, they have chosen to ignore the rules - maybe because they stood to gain so much money on breaking the rules.
When Google does business in Europe, they make use of European laws such as the trademark rights that grant them exclusivity to the brand "google", the copyright that grant them exclusivity to their source code, patent rights that grant them exclusivity to their many inventions, and hundreds of other laws that ensure that Google - just like European counterparts - can rely on the rule of law to protect them while they make money. Google itself has likely benefited tremendously from the exact same European competition law, that they are now fined for breaking. Were it not for such laws, Microsoft or Apple could and would likely have toppled Google from the advertisement market simply by forcing their own search engines, browser, maps, emails etc. on users of their operating systems.
Whether the administrative decision of the European commission to fine Google will hold up in court will be seen if Google decides not to accept the fine. So far the European Commission has had a good track record.