Nothing has changed. In the last century this also happened and it was called islands of automation. In my bubble back then it was considered a good strategy, let departments first play around, and if they are on to something integrate it.
If there's an xls which has been in regular use for more than 18 months, and it contains macros, then it can be assumed it performs some important role and should be properly documented and checked and could also be rewritten in a "real" language and officially supported. Set up a meeting with whoever made it, and whoever's touched it most. Approach it more like "we're improving your cool thing" than "we're taking away your toys".
Many years ago a company I worked for used to send out a spreadsheet to its suppliers which they would complete with the products they offered and then when it was received back there was a button in the spreadsheet that would automatically upload the data to a central database.
When I first saw this I was curious how it worked and did a bit of investigation - turns out there was VBA behind the button that established the database connection and uploaded the data. What was amusing was that the user had hardcoded the database connection string including username and password. Of course this wouldn't work outside of the firewall - but I'd be careful about letting people get too crazy.
The reality of the situation is with proper IT support, there could be compiled Excel Addins which provide API connections to core systems such that proper authentication also takes place. But that requires a first step by IT. Either that or authentication via a web server to get a temporary connection string. Either way, it requires prior infrastructure.
They exist because they work. You want UX or business analysis? You literally just got that done for you for free if you run into a Excel/VBA application. The hardest part of dev is figuring out requirements, so stop looking at these as toys and start realizing that shadow IT exists because of a gap in development. Full stop. You can argue all day long that you're working your assess off, and you do great products, but the existence of these apps is empirical proof that IT has missed the boat on developing something of importance.
Use that.
The moment IT touches your stuff, your job transforms from solving problems to writing emails and having meetings.
Any change, no matter how trivial, takes dozens of emails, dozens of meetings, and half a year to orchestrate.
If IT wants to help solve more business problems, it needs to fundamentally change its self-concept and purpose away from "prevent hypothetical bad things from happening at all costs" and move it towards "solve more business problems".
You might as well become an immigration lawyer and spend all day begging the government to explain why your latest M-10582-9DJVA-V isn't being processed in the normal time frame, even though it was stamped in triplicate and sent by Certified Mail with a full-color copy of every identification document you own.
The best way to deal with IT is to avoid depending on it ever in the slightest way. If you give it an inch, it will take a mile.
Funny how with computerized process, IT departments are effectively central planners. The lowly workers get to only do what the IT secretariat allows. It is this way because national^Wcorporate security!
Modern IT is more like if your water utility had final say over which faucet you installed and how you used it.
Authorizing use would be akin to the pre-Carterphone ATT model where only pre-approved uses would be allowed ('you can't attach your equipment to our network').
Thankfully, we eventually realized that was a dumb decision and moved to something closer to user freedom + network protects itself + zero trust.
Better to just guide behavior at the pricing level, and let people make their own decisions about use.
If we had to make some sort of water use analogy, I’d go with something like; the corporate network is a somewhat protected environment that needs to be maintained to be useful. So it it is more like a reservoir than a faucet.
It would actually be OK for a couple people to go swimming and even pee in the reservoir. Some people could even boat in the reservoir, if they went out of their way to make sure that their boats are clean, safe, no pollution, etc. But lots of places just have a general “don’t go in the reservoir” rule. Not because a person would damage it, but because everybody doing it would.
It is hard as a residential user to use enough water to damage the reservoir, but hypothetically if you managed to, somebody would check in. Even if you are paying, the town doesn’t want to run dry. If there is a drought, residential users might be asked to use less water.
Price doesn’t work as a signal in corporate IT for individual workers, because it is expected that the company will “subsidize” the worker to the extent needed to do their job. If we want to make the analogy work—at least in some areas, landlords are required to provide water to their customers. In that case, you can use as much water as you want for free, but your landlord will get curious and might find some way to get you on the hook if you pass some reasonable threshold.
You can also do some things as a user like dump toxic waste down your toilet. This would be sort of like running a publicly visible unpatched XP system on the network. It would damage the system, and why do you have that in the first place?
Anyway, that was fun to write, but I don’t know that it is particularly useful. In order to make the analogy fit, we need to bring in as much complexity from the water management system as the IT system has.
Sure you can have all of these. They're just not offered as part of normal utilities. Nobody will care if you build yourself some, except maybe for petrol pipes due to fire/explosion risk.
Example: Imagine a CI/CD pipelines using notebooks.
I hate Jenkins/Hudson style build systems so much I could just spit. I just want to run a shell script.
(Alas, I haven't had the gumption to try this idea out yet. Soon.)
These are probably not impossible to solve for notebook-style, but there are not many efforts to solve them or they are not even acknowledged as problems.
Edit: There is Pluto for Julia that attempts to solve the state-problem. I have not used it in practice though; I've given up on Julia, in large part because Julia community tends to be even actively hostile towards "stateless" development.
By "stateless", I'm assuming you mean functional programming paradigms of immutable, idpotent, and no side effects.
FWIW, for build pipelines, my quarter-baked notion is to use ZFS snapshots (or equiv).
I'll check out Pluto for Julia.
As you know, state is a challenge for "serverless" too.
I've been reacquainting w/ RDBMS tools. There are a few new strategies (implementions) for change tracking. Back in the day, we just banged the rocks together (ook, ook), so I'm very eager to learn the new hotness.
Immutability and idempotencency are good, and related, ideals too, although I think these can get too "unergonomic" if taken too dogmatically (like in Haskell or Redux), they should be used with almost goto-level discretion.
Of course there's the clear (short term) usability benefit of maintaining the memory state in that stuff doesn't have to be recomputed. But we can have that benefit and be stateless with pure functions and memoization. I quite often whip up a buggy and brittle ad-hoc solution to do so. There was also the IncPy project [1] that did this more rigorously, but it hasn't been updated in 13 years.
In general I'm a bit baffled why pure function memoization is so rarely used or proposed. Despite the old adage, cache invalidation is not actually half of the three hard problems in CS. With pure functions it's trivial.
Another baffle is why snapshotting/change tracking (and compressing) file systems haven't caught on. Instead these tend to get implemented badly in any sufficiently complicated application.
Also the "higher-level side effects" apply more or less identically to REPL development.
I wasn't even thinking about REPL style work. Mea culpa: I don't actually know how jupyter et al work, so I'm talking out my hat.
Your explanation reminded me of "prevalent" persistence (vs full orthogonal persistence). I guess I assumed something like that was happening between cells.
I suppose it's analogous to the transition of UI frameworks.
Bad: Mutant components directly.
Good: Mutate thru event queue. Get undo/redo for free. Debugging still sucks.
Better: Pretend it's a simulation and use an entity component system. I think this is what the kids are calling "reactive".
> pure function memoization
Answering just for me: because I'm just a simple bear.
I've been imperative for so long, continuations, currying, and lazy eval break my brain. Yup, a fully functional world would be a lot more simple. Maybe it's time for me to revisit clojure.
Thanks again. This is fun to think about.
But this may not be good for anybody in the long run. For example it tends to lead many students to not understand basic concepts, like variables. Which is understandable because variables don't behave like variables in notebooks (e.g. the same variable in the same notebook may refer to different values in different cells depending on how they are run). For many students this can cause almost insurmountably wrong mental models (which they will of course carry to "production" later on).
But as I argued in another thread here, it doesn't have to be this way. E.g. Pluto does notebooks in a more rigorous manner.
Almost all "software engineering" languages and tools makes getting started and actually getting something done quickly needlessly difficult. Probably uncontroversial that git UI is a total mess, and things are getting even worse with more build tools, dogmatic static typing and general pointless ceremony.
Even though it's a visual environment, everything is strongly text based, from HTML to CSS to JSON payloads.