Earlier in this thread I wrote about data being the top priority [1], the raw material, that we rarely give data the primary focus it deserves and instead focus too much on the processing side. More concretely: dashboards showing instrumented processing clusters give a biased view that does not focus on what matters at the end of the day. What we also need are dashboards showing data flowing between sinks and sources, data quantity, data quality, etc. Sure resource utilization and efficiency matters, but only after we can validate we still have the right output, and that input is of proper quality. If output contains garbage, is it bad processing or is it garbage from the input? And if something is wrong with processing, do we know the impact downstream? In other words instrumentation should include data sensors, not just processing sensors: data counters, validation points, invariants, etc. Because at the end of the day, when the power goes off, do you know what's left to recover? Do you prefer your customers telling you about an unfulfilled order or do you rather want it to be detected earlier? If you get audited for GDPR, do you have a map of your sensitive data? In terms of security, is it about protecting clusters and containers, or is it about protecting the data? Once you get the data side right many things become simpler, but if you get it wrong, as we often do, we create a world of problems. Giving data its proper place in our engineering practices will certainly change our industry for the better and bring it closer to "reality" with less danger of veering into the virtual for the sake of it.
In a world where software is increasingly involved in human activities this would have a great impact. However I don't think we should stop there, as I believe this is part of a larger trend I'm concerned about: the idea that, not just in software development but in most human endeavors, we're increasingly favoring spending time in virtual/man-made spaces and activities, at the expense of the real world, the place and time we're at, nature and the environment. As if we want to escape the physical conditions we're in: whether it's our body, our environment, society, the work we do, etc. When one can't see a way to influence the real world a tendency would be to start operating in a virtual one where we get the illusion to have an effect, make some money, be an expert, etc. Oh and let's not forget this desire to put as much tech between us and the real world, as if we don't want to experience it directly, it's too icky, and instead need devices to offer an indirect perception: wearable tech, navigation by GPS, remote controlling tractors in vast industrialized agricultural fields, etc.
A friend working in a supermarket chain told me this story: he often advised a younger manager to consider better maintenance procedures of their A/C system, but to no effect. Now that my friend is retiring soon this younger manager is proposing to make an excel chart to track energy consumption in order to optimize setpoints, and asks my friend for his approval. Really? Approve perception to be limited to an excel chart? Isn't that the same as leaving the windows open and wanting to change the setpoint?