2,578 karma · joined October 15, 2012
E-mail: sixdimensional ATAT [dimensionsix DOT com] minus the ATAT and brackets.
At the end of the day, we are all surrounded by different scales of organization, whether it be a small family unit, formal and informal networks, societal organizations and institutions, businesses, etc.
There has to be a balance, because many organizations want flexibility, "democracy", the ability to change direction quickly, autonomy in suborganizations, etc. Yet, this must be balanced with the need for some kind of order against pure chaos or harmful organization actors (for instance). This implies some kind of governance, some kind of "ground rules" if you were, for things to function and be reasonably effective/efficient.
I think this is where organizational culture truly matters. I do think culture can be encouraged, and norms and behaviors modeled, and sometimes even formally enforced, but ultimately the net governance that emerges at scale is a sum of the parts (i.e. the whole is greater). I do not know if, at the kind of organizational scale I'm talking about (many thousands of people or more), it is possible to govern everything with absolute precision and control.
Rather, at scale, one can only model the desired outcomes, norms and behaviors, and/or set policy. Utimately what the organization is - all its boundaries and suborganizations and networks all the way up and down the scale of an organization - is determined by the sum of those who participate and represent the organization. It does leave holes where those with personal agendas or sociopathic tendencies (for example) may take hold.
By all this, I mean to say that, I do not believe it is just in recesses where sociopaths grow most influential. I believe they grow wherever their influence has meaning and they have power/control.
The only thing that can protect organizations against such behavior ultimately is the collective will of the members of the organization, at all levels of scale. The culture, and members of the organization are the ultimate tool for protecting the organization against such threats.
Even with laws or policies, everything depends on enforcement. Enforcement also depends on culture, in addition to power and control. I believe though, that in the long run, everything depends on what culture allows, and only culture can stop sociopaths from being ultimately successful and protect organizations from such threats.
I'm not saying a return to pure tribalism here, but I am saying maybe realistic acknowledgement of tendencies and how to make those more effective/efficient and less problematic might be a good idea.
In other words, turn a negative into a positive. Work with the tendencies with incremental improvement.
There's another way of thinking about silos - call them organizational boundaries - and observe a complete shift in thought on the topic.
Some might call this heretical thinking.
I can definitively confirm that just-in-time is about trying to optimize the amount and strategic positioning of buffer needed to support the "right amount of production" to meet anticipated and actual demand, exactly as you describe.
Now, doing that can be rather difficult in the case of black swan events. But the goal is as you describe.
It was somewhat accidental forced unplugging/disconnecting.
Between work stress, COVID, and my own slight Asperger's, I struggle to get myself out of mental loops.
During this time away, my first few days, my dreams and every moment I closed my eyes was a flush of memories or thoughts of work. It almost felt like my brain was trying to process a backlog of stress and that process was "flushing" those thoughts (or brain chemistry) out.
I went hiking, read some, and mostly took it easy, walked around, ate and rested.
It took at least 3-4 days of a 8 day trip for my brain to barely start calming down. I had to keep repeating the "simpler activities" and just put out of mind literally everything else. I literally had to "escape" from my normal life entirely.
I don't say this to be run of the mill, I literally had this anecdotal experience but a few weeks ago.
I felt a little more like I had a handle on myself after this break, but only in a "just almost barely" kind of way. In other words, I had just started to come out of my loops and my fog, when my trip ended.
Just like the work had a lingering effect, so did taking the time off. I am back at work and finding myself going back to some of my loops, but my trip gave me a little break and opportunity to take some perspective.
Therefore, my non-scientific anectodal recommendation is to study your own rhythm and see how it changes when you disconnect or take a break, especially if it includes an aspect of "slightly forced disconnecting". If you can learn to become more self aware of the rhythms you experience or fall susceptible to, it might give you a chance to subtley shift or do something actively to tweak them in a way that helps you.
I think it helped me, at least temporarily.
I found the posture in a similar zero gravity chair super comfy but, I don't think I could sit at a desk with one.
The zero gravity chair idea reminds me of this insane thing: https://arstechnica.com/information-technology/2013/06/seat-...
That is probably one easy way to encourage competition. Eliminate the egress fees.
If it costs nothing abnormal to use services across clouds or on premises, then you can likely pick and choose best of breed anywhere as long as there is a decent network.
I think LA is talking with Pallet to implement their temporary housing, which would be amazing.
There's also the thought that decentralization is seen as a threat by centralized players, therefore, even if it could be more effective, it may be prevented or delayed from taking hold.
I see an interesting alignment between the trends we are seeing in technology and politics. Actually, my senior thesis back in college (2003) was that the politics of information technology are disruptive to organizations. Not necessarily disruptive in a "sided" sense, in that "disruption" can be considered good, or bad, healthy or damaging, or both.
There is a trend we see in information system/technology architectures towards "decentralized" and/or "federated" solutions. For example - "service mesh" [1] or "data mesh" [2], both higher level stack abstractions that came from lower level mesh networking concepts. And we even see "mesh" concepts in the form of knowledge graphs, social networks, peer-to-peer, etc.
If you read up on this you will often see the word "federated" and it is so interesting that federation is both something we apply to technology/systems as well as political. I am quite sure that it is part of the key to what you are thinking about. There's a reason that countries have been referred to as "federations", why we call it the U.S. "federal" government, etc. I believe, among other things, it largely revolves around how one implements governance and ensures a working "system" in terms of standardization/policy.
There is a desire to be autonomous yet interoperable, and this applies as much to "human" systems as well as purely technological systems. There is a link, after all, between how humans implement systems per Conway's Law [3] - organization's systems often are designed to mirror their communication structure.
Institutions are just one kind of organizational unit, and there too you come across all kinds of interesting issues that have perhaps, barely been explored - one such example is "institutional trust" [4].
In summary, my own latest "noodling" on this area is that actually, our technological and political systems are still very much evolving. When you take into context the possibility that it is truly becoming possible within, how about we say, the next 100 years - that humanity my be able to truly exist off the Earth, in space, for longer periods of time - then that is going to open up a whole world of questions about these very systems which we think we have completely figured out.
If there is one constant I think we can count on, it is probably change - but, ultimately survival (a key element of which is adaptability) will probably be the force that drives whatever the next 100 years look like - and that could be both good or bad overall for the development of new organizational structures.
Personally, I feel that all of this is interwoven together in quite an interesting way and all related, as information systems are the backbones of organizations.
[1] https://en.wikipedia.org/wiki/Service_mesh
[2] https://martinfowler.com/articles/data-monolith-to-mesh.html
[3] https://en.wikipedia.org/wiki/Conway%27s_law
[4] https://en.wikipedia.org/wiki/Institutional_trust_(social_sc...
Seeing what is broken, knowing that it can be fixed, and having even the slightest bit of motivation or knowledge of how to do so, and how it goes on forever and ever and doesn't get fixed becomes incredibly tiring.
I suppose, if you go into a dysfunctional company hoping to "coast", just be aware, if you are not good at managing boundaries and passing off the "frustration", it might be a lot worse than you realize.
Just imagine.. if you are spending 8 hours a day, 5 days a week, watching a slow motion wreck.. and powerless to do anything about it... sure, you get paid, but is it worth spending the time?
Similar to some of the concepts of "Linked Data", maybe - https://en.wikipedia.org/wiki/Linked_data.
The problem is getting to a standard, it would essentially need to be federated search so a standard would have to be established (de facto most likely).
Also, indexes and storage, distribution of processing load.. peer-to-peer search is already a thing, but it doesn't seem to be a core function of the Internet.
This is basically the same concept as making an "open" version of something that is "closed" in order to compete, I guess.
A general answer, perhaps... not specific to the case you've specified.
Query optimization is a science with multiple dimensions [1]. I'd wager every problem in computer science plays a role somehow in query optimization.
Query performance is based on a combination of actions you take to optimize the design of your system to get the best performance (e.g. data modelling, index design, hardware, query style, and more), and the patterns the system can recognize based on your inputs and the data itself, with the resources it has available, to optimize your queries.
There are known patterns for optimization that are discovered over the years, many hard learned from practical experience. This is why older "popular" engines sometimes are more mature and more performant - they have optimizations built for the common use cases over long periods of time. That is not to say older engines are always better, just that they have often had more exposure to the variety of problems that occur.
The reason why the engine "can't figure it out" is that most engines, even the best ones, are quite complex - combinations of known rules as well as more fuzzy logic, where the engine uses a combination of information and heuristics to essentially explore a possible solution space, to try to find the optimal execution plan. Making the right decision, well, can be difficult and given the nature of these things, sometimes the optimizer makes the wrong decision (this is why "hints" exist, sometimes, you can force the optimizer to do what you see is obvious - but this is suboptimal for you).
In some cases, finding an optimal execution plan can actually be quite computationally expensive, and/or quite time consuming, or the engine in question may simply have no logic coded to handle the case. Optimization is all about finding the balance between finding the most performant query plan, but in the least amount of time, with the least computational and I/O impact to the overall system, that returns the right result. Optimizers are also highly depending on the capabilities of the engineering teams that build them.
It is not an easy problem, and it is an area which one could liken to almost machine learning/artificial intelligence, in one way. There are so many possible options, the problem space so big, with so many different ways to approach a given scenario, that it can be difficult for the "engine" to decide.
This is why known patterns were created, for example, dimensional data models for analytical queries vs. 3rd normal form. Dimensional data models enable certain optimizations, for example, star schemas [2]. If you take a combination of implementing known patterns, along with optimizers written by engineers that exploit those patterns, you can get to a world of better performance.
However, in a world that is, let's say.. more "open ended" - for example, the world of data in a "data lake", where data models are not optimized, data comes in unpredictable multiple shapes/sizes, then it often comes down to combinations of elegant/complex engines that can interpret the shapes of data, cardinality, and other characteristics, make use of much larger distributed compute and system performance, and in some cases - often brute force to arrive at the best query plan or performance possible.
There are so many levels of optimization.. for example, if you were to look at things like Trino [3], which started its genesis as PrestoDb in Facebook - you will see special CPU optimizations (e.g. SIMD instructions), vectorized/pipelined operations - there are storage engine optimizations, memory optimizations, etc. etc. It truly is a complex and fascinating problem.
Source: I was a technical product manager for a federated query engine.
[1] https://en.wikipedia.org/wiki/Query_optimization
I was pretty bummed, I was hoping for a little boost from this as I had read good things about it, including the NIH articles you posted.
I can also confirm, finding the supplements that are not just mostly the ground brown rice flour is also a little difficult.
The data mesh ideas aren't necessarily new - as this idea that data has been managed/owned by what some would call "federated teams" (optimistic view) and others might call "silos" (pessimistic view) isn't really new - especially in certain industries.
Certainly, the current energy and focus on building collaborative organizational service models, process, standards, etc. and standardizing this around the concept of data mesh is "fresh" and very much on the tip of many people's minds.
I worked at a company that had built a tool which was a data federation engine, and we dabbled with the concepts of "data fabric", "mesh", "network" etc. but we were more talking about tech/implementation vs. architecture and organizational structure. Still, we thought the right tech could enable this architecture/organizational structure and that it wasn't something totally figured out.
Thanks for sharing and mentioning it, and also Terminus. Certainly, I agree that knowledge graph is another approach and one that can also fit very well with the idea of linking data that is both common and specific to domains.
However, I've rarely seen graph databases that had the kind of versioning/time travel that a temporal database (such as relational ones) had built in natively (or anything like "diff").
I don't agree this can't map to SQL tables.. tables and relations (tuples) are just a form of data. People do collaborative data management using relational-based master data management systems already today and have been for many, many years.
That is not to say that graphs aren't a good match for storing common/reference datasets or linking data together. Given that graphs can be considered a superset of relational model, I don't see any reason that you can't do the same thing in graph databases, just that doing diffs and enabling the same kind of workflow might be more difficult in a graph data model - but definitely not impossible.
Lest we forget things like "Linked Data" [1], triple stores, etc. also that have been trying to get at linked data across the entire Internet. However, I never saw collaborative data management as part of that vision, it was mostly for querying.
I respect and honor this man for what he did. RIP Allan McDonald, and thank you for what you left behind.
This quote from the article helped me, right here, right now:
--- excerpt ---
"What we should remember about Al McDonald [is] he would often stress his laws of the seven R's," Maire says. "It was always, always do the right thing, for the right reason at the right time with the right people. [And] you will have no regrets for the rest of your life."
--- end excerpt ---
When people say that it is not just about about what you do, or why you do it, but who you do it with, it is so easy to gloss over that sometimes, it's about doing your part with the right people.
For me, this served as a reminder of high value.
Yes you need reference data for machine learning, but the world isn't only about machine learning. You might want reference data for human-interpreted analytics, table-driven logic (business rule engines, for example), etc.
One of the main use cases you can see them targeting, and that I think makes a ton of sense, is providing tools for collecting, maintaining and publishing reference data sets using crowd sourcing.
For example, they are doing this with hospital charge codes (a.k.a. chargemaster data). Hospitals in the US are required to publish this data for transparency.. however, I have never seen a single aggregated national (or international) data set of all these charges. In fact, such a data set could be worth a lot of money to a lot of organizations for so many reasons. I used to work in health insurance, gathering data from all kinds of sources (government rules/regs, etc.) and it was a lot of hard work, scraping, structuring, maintaining, etc.
This reference data can be used for analytics, to power table-driven business logic, machine learning - to help identify cost inequalities, efficiencies, maybe even illicit price gouging, etc. There are so many reference data sets that have similar characteristics... and "data marketplaces" in a way are targeted at making "private" reference data sets available for sale - so then where is the "open" data marketplace? Well, here you go.. Dolt.
I have often realized that the more ways we can make things collaborative, the better off we will be.
Data is one of those things where, coming up with common, public reference datasets is difficult and there are lots of different perspectives ("branches"), sometimes your data set is missing something and it would be cool if someone could propose it ("pull request"), sometimes you want to compare the old and new version of a dataset ("diff") to see what is different.
One difficult thing about Dolt is, it will only be successful if people are actually willing to work together to cook up and maintain common data sets collaboratively, or if those doing so have an incentive to manage an "open data" project on Dolt as benevolent maintainers, for example. But, I could say then it has the same challenges as "open source" in general, so therefore it is not really that different.
Dolt could even be used as a foundation for a master data management registry - in the sense of you could pop it in as a "communal data fountain" if you will where anybody in your org, or on the Web, etc. could contribute - and you can have benevolent maintainers look over the general quality of the data. Dolt would be missing the data quality/fuzzy matching aspect that master data tools offer, but this is a start for sure.
For example, I work in a giant corporation right now. Each department prefers to maintain its own domain data and in some cases duplicates common data in that domain. Imagine using Dolt to make it possible for all these different domains to collaborate on a single copy of common data in a "Dolt" data set - now people can share data on a single copy and use pull requests, etc. to have an orderly debate on what that common data schema and data set should look like.
I think it's an idea that is very timely.
P.S. Dolt maintainers, if you read this and want to talk, I'm game! Awesome work :)
I think the idea of a "diff" applied to datasets is quite awesome, but even then, we kind of do that with databases today with data comparison tools - it's just most of them are not time aware, rather they are used to compare data between two instances of the data in different databases, not at two points in time in the same database.
I feel one could argue that there is a chicken-and-egg problem here. Nobody could code anything without a reference to start from. But, then, who developed the first language without a reference?
Ah, there - the start of something - those who invented the first programming languages - the very concept of "stored programs" as punches on a card, for example - assembly language, etc. - they did not have a reference. They invented the reference. But, it was also, by today's standards - much "simpler" and a much "smaller surface area" - while probably quite technologically complex and difficult for those at the time.
But after that, everyone else, who didn't invent the technology, but was rather using it.. was therefore - by definition - using a reference. Standing on the shoulders of giants... how else to learn?
For instance, I learned BASIC coding first by studying other people's examples - in magazines, by looking at code that came with my first compiler as samples, reading books. Same way I learned Pascal and C++ - the tooling/IDEs back then couldn't "help you learn" with autocomplete.. you learned when you wrote something, and it either compiled or it didn't. You learned by using "help" or books or talking to people or looking at examples.
I'd argue that even those who invent most things today are inventing it at a higher level of abstraction, and therefore still looking at references for levels below the level of abstraction they work at.
So.. "can you make a basic web app without googling"? Well, actually, yes, sure I can. But that is because I learned HTML, CSS, JS (and supporting languages like Perl, PHP, etc.) when the surface area was still small - from the Internet itself, by looking at examples and reading things like the entire HTML specification when it was smaller. Then I memorized it by doing it.. a lot. Then things changed, but a lot of the basics are still possible to do today and still work. That said - "will that basic web app work in every browser platform or any every server/OS platform that exists"? Well, if you stick to the true basics, possibly yes - but, this raises the question of technology and implementation variability and complexity.
I think the bigger question being posed here, isn't the headline... it would be better to ask, "does it matter if you can't make a basic web app without Googling"? There, I think, is the point of the post - and - yes, it matters in a lot of ways. But if you are coding for the web - presumably you are already using the web.. so.. perhaps that specifically is not as important as the next bigger question...
"Has writing code for modern systems become so diverse and complex, and the surface area so large, that it is difficult to build systems? Has the variability and diversity of our platforms become so large that it is difficult to build once/deploy many with certainty?". There again, I feel, is a much deeper and more interesting question. This too can be answered simply - avoid complexity and choose simpler languages/platforms that accomplish your goals, learn their surface area, and do your work. But, this might be more difficult/non-obvious to those just getting started - and there some good work could be done to make it more accessible.
And here we look to things, for example, like WASM - trying to build simple primitives that can work on nearly any platform, upon which many complex languages can live - engineering solutions to attempt at reducing complexity based on variability and diversity of platforms and implementations.
This question... surely relevant during this time of resurgence of "low/no-code" platforms and in relation to what we might have once thought of as "4GL" (or beyond) programming languages and related implementations/platforms.
People don't realize that it is a very difficult balancing act to keep electronics working over the long term due to the ever advancing pace of technology (hardware and software), complexity of manufacturing, materials availability, customer demand, etc.
One part of that could include rapid design refreshes planned on a schedule to avoid obsolescence issues, for example. When viewed a certain way, pumping out new products rapidly and not supporting old products for long periods of time could be perceived as planned obsolescence, but is possibly a reaction to customer demands..
I am not defending planned obsolescence here (perhaps pointing out a different concept entirely given its current negative definition), but there is a point of view that planning for and addressing obsolescence in a controlled and transparent manner, proactively, may actually be a good thing, not a bad thing. But it could be quite hard to see that perspective as a consumer.
Also, certainly, when done behind the scenes for reasons that only serve the best interests of a private organization and not the customer and/or environment - that is an entirely different story and not acceptable.
I just had a look through your docs and I understand the approach you're taking... similar to "infrastructure as code" concept applied to simplifying workflow. What you are saying makes sense to me. I haven't heard of temporal.io before (cool, hi!), but I get the concept of trying to eliminate the intermediate representation (which I suppose also could be seen as a bottleneck in a fully distributed system and to the "workflow" of a software engineer).
I would point out that, Petri net is a theory and modelling language for proving correctness/soundness of a model - and as I mentioned earlier, I agree that if you don't need that for your use case, yeah - it could be too much. I would like to make it clear for other readers that Petri nets are not specifically the implementation of a technology, they are a modelling technique/concept which could also be implemented/executable through an engine. Having, or not having, an intermediate representation in compiled code or runtime has nothing to do with whether or not you want to model your FSM, or graph, or network of processes, logic and state as a Petri net.
My goal in originally posting the comment was just to share the superset theory of Petri nets, as I don't often see people bring it up in discussions on workflow, FSM, etc.
The comment that Petri nets are one possible representation of FSM is true, with the key difference being that FSM are for single threaded operations and have some limitations in that regard, whereas Petri nets, as the superset, also handle concurrent operations - and workflow is a common use case where we see concurrent operations (but certainly not the only one, and industrial control, etc. is certainly more complicated that organizational workflow). I think this deck is kind of handy for describing some of the differences between FSM and Petri nets [1]. Interestingly, we know that dealing with concurrency, parallel processing, multithreading is difficult, which is why any tools to reduce some of that complexity from day-to-day coding (ex. coroutines/async/await, workflow engines, etc.) make the engineer's life easier.
When we were talking about compartmentalization/encapsulation of state, I see how conceptually you are abstracting that away from the engineer in temporal. I suppose, what I meant when I was talking about WWF was, the fact that the engineer's code and their own state could be "compartmentalized" into the pluggable definitions of the activities provided by the workflow engine. It seems like your team has kind of inverted that a little bit, but roughly the idea is similar. You appear to have the engineer "write the workflow as code" and use a wrapper around the activity to ensure the state management and simplify it for the engineer. A slight difference but interesting paradigm.
I would say, I believe that, while rarely done - one could define WWF workflows completely through code and that there is a way to serialize workflow as an artifact that could be version controlled, etc. But, how they display that visually and how WWF compiles into the intermediate representation and uses that at runtime does sound different than temporal. I suppose though, having the intermediate representation does perhaps allow for running tests against the workflow logic itself - whereas you shift the tests to the code itself.. so, different implementation but similar idea.
Again, I see how temporal simplifies that from the engineer's perspective, so kudos to you guys. I can see how what you are saying would work and be a helpful approach for a lot of use cases.
[1] https://www.cs.ucdavis.edu/~devanbu/teaching/160/docs/petrin...
That is to say, level of complexity in describing process in Petri net "modelling language" might seem higher. I can surely see how it feels more esoteric and not always clear how it ties to the implementation.
Many workflow engines take the approach you're describing, for example, I also used to use Microsoft's Windows Workflow Foundation to do similar things. Essentially you're sketching a workflow process skeleton and then managing the state atomically by compartmentalizing it, so to speak.
Actually, this is exactly what Petri nets propose - state is defined by tokens in places - i.e. compartmentalized.
I don't entirely agree with your comment about state management and argument passing in Petri nets. I do agree it takes digging around to find tangible examples/applications that cover argument passing, but the idea of "tokens in places and how they enable transitions of state" is the part of the puzzle to represent in an abstract way, the tiny pieces of state/arguments that enable transitions to fire. I could represent your code above as transitions which cannot fire until tokens representing the state of your conditions were present in the right places. For example, the passage of time was present in an input place, and the condition trial period = true and value customer ID is not blank, all as tokens that have to be in input places to enable the transitions to fire which trigger those activities.
This is to say that, I agree that representing that graphically using Petri net modelling may not be as business friendly as say, UML activity diagramming. But it also doesn't make the simpler approach any less a subset of what you can do with Petri nets, as it very much is one.
But definitely agree, use the right level of abstraction that fits the need, like the old adage, try to use the right tool for the job.
I'd argue that, the tool you describe could be modeled as a Petri net, but that perhaps you may not wish to have a user do it that way.
Do you agree or have a different opinion?
I have always felt that the key to being an engineer is to understand that theory is a tool, and what makes it fact is science (wow, cool science! LOL) - testing the theory by putting it into practice and measuring the result/iterating. The fact that Petri nets give you a way to talk about the proof as well as model the problem is part of what makes them interesting to me.
So I guess I'm saying, moving fast while applying science.. and the learning and iterating.. that just makes sense right??? If the result is a working system, even better! Science and being agile are totally compatible!
The burden of proof is on everyone, to say, OK.. I am following this technique because it helped me build a working system. I am standing on the shoulders of giants, and the system has been tested and works... but if you don't think the theory is correct, then please, prove it wrong. Make it better, improve, iterate!
In this case, with a genuine and positive attitude, I kindly turn the computer keyboard around to the other person and ask them if they know a better way, to please explain it and I'm happy to learn and adapt. I'll show my proof if you show me yours, and we can discuss, kind of thing.
Like, set logic in relational modeling.. if you abide by that technique, set logic gives you certain guarantees about operational characteristics that have mathematically proven grounding. It gives you primitives for talking about set operations, like union, difference, intersection, join, etc.
Petri nets are a modelling technique. Activity diagrams (UML) are another way of expressing similar processes, but they don't necessarily have the mathematical grounding inherent in the way you model Petri nets.
Perhaps yet another way of saying it is that Petri nets are closer to a visual modelling language of execution that shows more of the logic of the concerned process or system.