Hi Andrew,
The Heron rewrite had more to do with what were at the time gross operational inefficiencies of Storm at very high scales, and problems diagnosing failures and bottlenecks. This is Storm 0.9 -- in the years since, I think Storm community in general and the Yahoo folks in particular have been working hard on addressing some of those issues, and a recent blog post from them indicated that some of the stuff we fixed in Heron is on their roadmap. Note that the Heron paper was published a year or so after the first Heron topology went into production inside Twitter. We had a very real problem that we needed to fix very quickly, and writing Heron was faster than making Storm work. Some of that was due to OSS challenges, some due to Storm's architecture fundamentals, some due to the specific people and backgrounds we had on the real-time compute team at that point.
The scales I am talking about are hundreds of nodes, not dozens, and an order of magnitude more messages per second. I am not surprised it works perfectly well for your use case (it worked fine while we were only putting tweets into it in 2013, as well -- that was about the size you are quoting, iirc).
Mesos not only existed when Storm was written, Storm ran inside Twitter on Mesos since before Storm was open-sourced. Mesos went into Apache incubator in 2011, while Storm did so in 2013. I'm not sure why you got the impression any of this was related to Mesos; it's true that we simplified a lot of operational complexity for Storm+Mesos by not writing our own Mesos scheduler, like Storm did, and just using Apache Aurora -- a decision that also meant we were able to use Aurora/Mesos clusters shared with other processes, which was nice; but that wasn't the prime motivation by a long shot.
The API was kept as a trade-off, to make migration of internal customers seamless. There are problems with the API, particularly around back pressure, but at the same time it was a straightforward API that got wide internal adoption, both in raw form and through Summingbird. So it made sense to evolve that part incrementally and deliver our internal customers the performance, observability, and reliability wins first, without having them rewrite a line of code, and tweak APIs over time to address the above-mentioned issues.
I'm biased of course, but I think our track record with open source contributions is still pretty good -- Scalding, Parquet, Aurora, Mesos, Finagle, Zipkin, and many more major projects with very wide adoption in the industry, which we still very actively contribute to and continue to evolve. At the same time, there's definitely a cost to open-sourcing projects, and those decisions get made on a case by case basis.