One of the biggest problems in building something on top of Drupal is the 2 year cycle of major releases with no backward compatibility. That means you will need to port all your custom code every 2 years. Combine that with the fact that many crucial modules are actually not part of the core distribution and are often not ported for months/years. Some less popular ones are even abandoned with new alternatives coming up. We went through that process of upgrading from D5 to D6 but simply gave up at the D6 - D7 transition.