We've been thinking about software integration wrong for the past 50 years

It may not be immediately obvious, but this is basically another cri de coeur to the effect of "Why can't we make the Semantic Web work?"

That's a good question. But you need to start from the fact that people have actually tried really, really hard, rather than nobody having tried at all, and it in fact hasn't worked despite rather substantial investment. When you start from there, you may obtain some useful understanding. Perhaps even useful understanding that will lead to solving some of the problem. I've got no problem with people jousting with windmills, I just think that anyone who wants to take such a task on really ought to research previous jousting attempts first if they want to succeed (as opposed to just doing it for fun).

But if you start from the blatently false presumption that you're the first person to ever dream of data that can just flow between systems, well, yeah looking out into the world you're going to be confused at what you see.

We have in fact tried the "correct" way, and it turned out not to be correct. Whereas what we're doing now, however annoying it definitely is, does in fact at least partially work.

It's a clickbait ad for something called "Nerve".

The original idea for "electronic data interchange"[1] was to have a standard format for inter-company invoices, purchase orders, bills of lading, customs clearance, etc. So you only need N implementations, not N^2. This got going in the 1970s. So the formats are ancient. This is a flight ticket availability request in the international EDIFACT format.

    UNA:+.? '
    UNB+IATB:1+6XPPC:ZZ+LHPPC:ZZ+940101:0950+1'
    UNH+1+PAORES:93:1:IA'
    MSG+1:45'
    IFT+3+XYZCOMPANY AVAILABILITY'
    ERC+A7V:1:AMD'
    IFT+3+NO MORE FLIGHTS'
    ODI'
    TVL+240493:1000::1220+FRA+JFK+DL+400+C'
    PDI++C:3+Y::3+F::1'
    APD+74C:0:::6++++++6X'
    TVL+240493:1740::2030+JFK+MIA+DL+081+C'
    PDI++C:4'
    APD+EM2:0:1630::6+++++++DA'
    UNT+13+1'
    UNZ+1+1'

This dates from when data communications were really expensive per byte. It's also from when people were thinking in terms of forms expressed in machine-readable data, not an API which actually did something. There are about five different standards bodies for this sort of thing. There's a mini-industry which converts this sort of thing to and from other formats.

Anyone involved in modern systems for this sort of thing?

[1] https://en.wikipedia.org/wiki/Electronic_data_interchange

This is a neat idea, but the unfortunate reality is that every link in an "integration chain" is almost guaranteed to be lossy. The author notes an obvious version of this (what if one node doesn't support a field that the other nodes do) but there are countless other types of loss in each link - structure (i.e. document data types), precision, limits (max field size, max number), other "exotic" datatypes (geography, binary blobs). Even mutability can cause lossiness - some of the fields in these systems are write-once, and even if creates and updates can be propagated, delete propagation is a Wild West in SaaS applications.

If you're old enough to remember what happens when you make a repeated analog copy of something (fax machines, copy machines, VHS cassettes, etc) you know what happens with this kind of data flow topology. We advise our customers to do the /exact/ opposite of this - follow as few links as possible for any piece of data and move towards hub-and-spoke, ideally with the hub as a highly configurable, low loss node - in my world, this is a data warehouse or data lake, but it might also mean an iPaaS or similar.

I...

I am dumbfounded.

With the current state of the market why in the name of all that is holy would you depend on one vendor to get to another?

I dont need to be at the whim of one vendor to get to the next one. What happens when the vendor in-between goes out of business or wants to present their own service or...

I have no words for how bad this idea is.

An academic architecture almost always has the following basic characteristics:

1) It offers both superior control and convenience.

2) It is simple to comprehend and cheaper to operate.

3) Systems are connected to each other with drawn lines.

4) The architecture "Decouples" systems to make them simpler, not more complex.

5) Data structures map cleanly to each other, with no loss.

6) Data structures are 1-dimensional collections of "fields".

7) The architecture is not used in production. No organizations depend on it economically.

.

On the other hand, a practical architecture can be distinguished by the following characteristics:

1) It is used in production. People rely on it daily.

2) It is expensive, and needs constant human attention to balance control and convenience.

3) It has no optimal state - only varying states of failure.

4) Its systems have myriad ways to connect. Some are incompatible with others.

5) Data structures are messy and arbitrary. Some are provably incorrect.

6) Unicode in particular is a problem.

7) It is not comprehensible by any single person.

.

With apologies to Admiral Rickover :)

This is full of opaque phrases that I guess make sense to the author.

I highly doubt theres a solution to anything in there.

I agree that moving data between arbitrary systems is an admirable goal, but this whole article doesn't make any sense nor solves anything.

Data migration is more than just "a function" that we can "just apply". Maybe this would work if all integrations were written in compatible languages with some sort of common SDK/API.

There is a reason open standards exist, to achieve interoperability. They have drawbacks like slower innovation cycles due to being standardized and now hard to change - hence all our $bigcorp don't give a flying fart and instead keep building up their walls which often don't even provide the ability to implement point to point integrations

Having Salesforce, Zendesk, Hubspot and Pipedrive that share the same data in one company I would immediately start cutting costs instead of thinking about integration. Even if I had to think about it, the only correct approach for data propagation is directly from source of truth (or in "colonial" terms - master system) to dependent systems, and the only middleware in between them being data delivery (simple queue or ESB, workflow automation platform like Camunda or Zapier etc).

We tried this when we were building DAM integrations. It doesn't work. Lossy integrations. You convert a datetime from Zendesk to your format via Hubspot and it goes from 1943-01-01T14:00:00Z to 0 because HubSpot stores that field as a unix timestamp and is very lax around bounds. You pull a decimal and you loose precision. Floats become ints, lists become dicts, strings become xml, binary blobs become urls, etc.

I am always put off by articles titled "we've been doing X wrong for Y years" ("and #3 will shock you!" :)).

I have the feeling that this is the article of someone who has a vision under the shower and shares it with the wrong-doers to bring them wisdom.

Now, I have not read this specific article so it may be revolutionary for once.

The software integration wrong is wrong BEFORE the web, the idea of an OS like a container ship and applications as container on the ship, able to "interact each others only with cut&paste, DnD humanly, or programmatically with some "desktop bus" is a commercial move that fail the entire subsequent IT development.

The original model was an OS-framework where an application was just a bit of code added to the common live codebase.

The web 2.0 try to reimplement such model without admitting the commercial horror we are in. Hence it can only fails despite all the progresses. Web n.x can't solve the underlying issue. The solution we have is simply accept that software as a product and as a service can't exists in a proper IT sound world. Data can be sold, iron can be sold, not more.

Data can be XML, EDI, json, yaml, ... as long as we have the base we can have easy enough integration because agreeing with a real world standard is not something doable up front while having the flexible base, the ease user-level/high-level development of classic systems do solve much integration issue. Try Emacs and you'll understand.

Hey Matt, it's a really neat product idea. I'd really recommend taking a niche and building it for that first. Selling a new Enterprise-wide integration tool is hard. Selling a new data orchestration concept will be harder. I recently moved out of that area, but I worked in this space for about a decade.

I can already see the rejections:

Debugging and Support: Rather than a centralised support system with levels, to support this you'd need a technical user familiar with the integration, or have the field mappings so well-documented and available a level 2 support rep could figure it out. The latter will never happen.

Brittle: This introduces a lot of risk to the organisation, as understanding conditions of failure and what the repercussions are in the event one of those services stops working becomes.. Complicated. There's a reason authN/authZ providers have opted for hub & spoke model, and that's because you need a highly specialised, often under-resourced team to look after it. Sound familiar? If you've worked in integration, it will.

Development and Resourcing: Like any proprietary product, this is another thing to hire for, and support. The uplift required for someone to understand this concept, and run with it for the use-case you've offered which is adopting it across the entire organisation requires.. Executive-level buy-in. To put it plainly, adopting this would introduce too much risk to the business in its current form.

I'd recommend taking a smaller bite with your approach. You're one person. Take a niche, and build something people want for that - people don't want technology for the sake of technology anymore. That money's dried up, and if you're trying to form a business off this concept - you'll need deep pockets to develop it.

Here's one that could be relatively successful: Take SIEM tooling, for example. For small and medium-sized businesses, operating a SIEM can be a massive undertaking. Usually the first thing an organisation wants to understand when they look to do this is: what audit logs do we have, and how can we look at audit logs for a user or employee across a range of services. Every company operating in a regulated environment, or company wanting to ensure they understand what an employee did during their last 2 weeks of offboarding would be interested in this.

If you built a product and demo purely to extract user/employee audit logs from services and expose them in this interface - I'd be lining up to try it.

This might be functional for a startup with an exit horizon of 2 years and nobody needs to inherit/own the mistakes of green biz-ops people who don't know what they're doing.

I've had to integrate new functionality with disasters like this and it is always a huge pain to support. Especially when the one guy maintaining the zendesk application leaves the company and we're like, "what is the purpose of this zendesk app in the django monolith doing?" and six months after we update it to support the new thing we added, we find out that django app's functionality was superceeded by xyz a year ago.

If I found out the mission critical biz ops side of the company was run like this I would not accept the offer.

Author has discovered (f ∘ g)(x) and wants to tell the world about it