Amazon AppFlow
aws.amazon.com
aws.amazon.com
Curious what the HN crowd thinks about this issue?
If you get to a point where you are locked-in it's because you spent years building into a solution that was your most expedient and renumerative option. At some future point in time, a future you or the person who replaces you may want to reconsider a re-platform. Let them worry about that.
I worry about development working on glue code/solve problems to the detriment of their businesses actual needs.
Sure right now it's your easiest option, but in the future the vendor might be the only one offering a compatible version of CoolNewFeatureXY at prices that make your business uncompetitive to the other businesses that have the freedom to choose between vendors.
If one day you want to move off Salesforce to another service, you will just switch off the configuration in AppFlow for the new service. Sure you're locked in AWS, but it takes tech debt away from other things, plus might save you migration costs.
That assumes that the AppFlow code is not buggy. In my experience these services are very hit and miss. This might be a good one, but it's impossible to tell until you use it (or read experience reports of others using it). Reliable 3rd party services can be a big time saver, but Unreliable/Buggy/Limited 3rd party services can easily take 5x more time than something in-house.
* About 70-80% of what you want to do with a service like appflow to integrate with a service like salesforce will be possible. The other 20-30% of your requirements will require building a bugging in house integration from the ground up. Might as well start off that way.
* As soon as you hit a wall with a service like appflow that is usually it. You'll raise a ticket and be stuck until they get around to implementing it. There are fewer roadblocks with home grown implementations of integrations.
I think you’re confusing 2 things. The technical and developmental side, where code is a liability that needs to be maintained, and the business/usage side, where code is definitely an asset that provides your company with additional leverage against competitors.
> Because you might have to pivot away because it might not support your requirements in the future? Then just pay that cost in the future if it ever comes.
That's pretty much the textbook definition of debt.
I used "build in house" as a simple example of a broader principle--if there is a "freedom to choose" option that magically lets you migrate from one platform to another at zero cost while imposing no overhead over the AWS solution, then absolutely you should do it. But generally these solutions require a significant up front investment and rarely actually deliver on their promise of abstracting away the underlying platform (now you're wed to $ABSTRACTION on AWS, for example), but that's another story.
> That's pretty much the textbook definition of debt.
Nonsense. That's not debt, it's deciding to wait to pay for something in cash until you know you want to buy it. It's not debt to wait to spend $30K (cash) on a car until you're sure you want to buy it. The "tech debt" analogy is intended to convey interest--you know you want to change course but you do the expedient thing now knowing that you're going to continually bump into it (each bump is "interest") until you can pay it off properly. This is the opposite--if you build it in house or use a dodgy "platform-agnostic" abstraction, you're going to be paying interest on something you will very likely never even need!
It's also A) frequently not all that hard to avoid B) worth doing for reasons other than sheer cost (e.g. flexibility, ability to unblock oneself more easily, ability do deal with bugs by oneself).
If you work in a company that isn't especially worried about hosting costs it's fine (mine isn't, for instance), but most companies are at least somewhat concerned with these costs.
Multi-cloud is complicated and not cheap.
My clients are all expending far too much energy trying to be cloud vendor neutral with the engineering capacity of your average non-tech large mid-market company. It's killing productivity. They went from VMWare in the data center and open source DevOps tooling to multi-cloud vendor-agnostic container-orchestration all in the name of "cloud but not with vendor-lock in". Meanwhile they could be actually achieving business objectives with these "seductive" managed services.
Here's one simple example, one client is spending about $250,000/yr on what could be accomplished with AWS Cognito for $250/month. But "no", they say, "then we'd be locked in to AWS".
I just can't.
- keep the business logic away from your implementation details
- generally speaking practice clean code
- don't overthink it, if your business goal is solved easily and quickly by using the cloud, use it, otherwise you lose time to market
The reply?
Why? We just want to pick one and stick with.
Customers want the vendor lockin. They even embrace it.
A huge problem however for smaller companies that have little power if the vendor raises prices, discontinues products or throws them off the platform. Going all in with AWS as a small company is not only expensive but can be a risky bet.
You can choose to work with a single vendor but switching to a new one should be trivial in case the prices get jacked. It’s that simple.
Vendor lock in is essentially that it’s more effort than you’re able to afford, in terms of time and cost.
There's always a tradeoff, but being all in on a vendor isn't necessarily a bad thing. Embracing a platform and the services they have to offer is how you get the most out it.
Whenever you pick a technology or service, you need to employ an ongoing effort rather than waiting until it's too late to do a one-time effort.
This is true with any choice of technology, whether your rolling your own or using SaSS, an on-going effort is required to avoid being locked in.
Amplitude, DataDog, Dynatrace, Google Analytics, Infor Nexus, Marketo, Salesforce, ServiceNow, Singular, Slack, Snowflake, Trend Micro, Veeva, Zendesk.
I feel like a notably missing product from this list is Jira. As horrible as it is, lots of companies use Jira for tasking so it should have been supported out the gate.
Basically you can build a “connector” to any REST/JSON API
Disclaimer: Microsoft employee, but not affiliated with Azure DevOps
Clubhouse.io, Asana & PivotalTracker are nice, but a little opinionated (can be a good or bad thing depending on who you are :) ).
I think where Jira really rubs people the wrong way is it's speed, particularly after someone customized it with a byzantine workflow or with dozens of views/reports and no primary source of truth for the team to use. Imagine a PM looking at one report and the dev's thinking another is "the one." A fresh Jira (Cloud verion on corp-name.atlassian.net) with out of the box Kanban board should work for most.
There are some industry vertical plays out there that focus on Agency or Freelancing work.
That’s exactly the problem with Jira (and honestly Bugzilla in the same way a decade ago), it tries to please everyone and thus succeeds with no one. Trying to do everything typically ends up in bloat/complication/slowness, which seems to happen disproportionately for bug tracking and cms systems.
> Trello handed over my personal account to my previous company
Isn't that the point? Businesses and people are way more obssessed about Jira than they should ever be. Jira is supposed to help organize, not become someones job.
At its core Jira is just a tool you use to track who's working on what issue, maybe link it to your version control to see what commits have been done.
BUT, then the Suits come in and start adding timesheets and reporting and 42 different categories for everything and it becomes a hellscape of forms you need to fill just to get some work done.
And don't get me started with creating a GUI so that the user can insert the data for a set of fields you don't know in advance.
Every time a moderately expensive purchase is needed companies get together, write down a list of things that they need the product to do and then compare the options on the market. This process results in a 'winner' who checked the most checkboxes.
It's a process that doesn't take into account the subtler more analog things we should be rating products on - basically how 'nice' they are to use (in a corporate setting I would phrase that as how 'efficient' they are to use, but it's the same thing).
This is one of a handful of very significant deficiencies in AWS's data/analytics portfolio. They haven't had good tools for getting data into AWS from the various places that profit centers store it. I hope they very aggressively pursue extending the number of supported integrations and finish building the product.
This is a huge step forward IMO.
Hopefully, that can change.
This is an odd use case given that Datadog can already store logs in your S3 buckets, and Datadog integrates with AWS Cloudwatch to pull in AWS metrics!
The power of metaphor right there. Although I've never seen that phrase before, I know exactly what it means.
in my experience it's not just the untransformed ones that are like dead fish in a hole.
Also it just sounds cooler.
Also, as I mentioned in another comment, this phrase doesn't work for the metaphor. Dehydrating and rehydrating a lake isn't a good idea; it'd kill the ecosystem, and it'd be tremendously expensive. I'd rather stick to dehydrating and rehydrating something else, like fruit. Here, this looks similar to the fruit soup my in-laws make: https://www.cheaprecipeblog.com/2015/11/norwegian-sweet-soup...
I also smirked at the metaphor. The AWS marketing department must be high fiveing right now
EDIT: Someone mentioned Heroku Connect in one of the comments. We do use this to archive certain objects (> 75 million records) and I have thought about replicating this data to AWS China (Postgres). Could be one option but some of our data is real time (e.g. customer orders), not sure what the "freshness" of the data would be.
So many companies with tons of data trapped in Salesforce's expensive walled garden and no way to get it out. Dumping it to SQL is the first step to freedom.
Heroku Connect is pretty powerful but not only is it super expensive as you point out but also forces you into creating an extra layer in the stack by creating a Heroku Postgres DB. The secret advantage of Connect is that since it's owned by Salesforce it bypasses Salesforce API quotas but that's a whole other can of worms :-)
I've built a few projects that provide real-time reporting across sass applications but it always seems like an uphill battle. Especially when your only interface with the sass is a hypertext API.
Can a data source change while you're completing a multi-part query? Are records hard deleted? If so how do you detect removed records to maintain deltas? How do you maintain deltas throughout lossy pipelines that group or filter?
In the end, it always feels like it'd be easier to build one monolith to replace all the systems and report on that instead...
Interesting product though! I think everyone that's using SaaS solutions has nightmare scenarios about providers screwing up or going out of business, or even seeing versioning of data at the providers. Exporting in a standardized way to S3 is great for insights in changes as well as for backups.
Of course when you starting building triggers that react to changes this creates a huge lock-in to AWS, but that's a cost a lot of people are willing to pay.
I would love to be able to do this with a load of different SaaS solutions that we use (gitlab.com, support tools, even google docs) (I'm in a regulated industry and this would solve some issues for us). Curious to see how this will expand and if custom integrations will appear soon.
Any AWS marketing people that are reading, there's a typo: route cause -> root cause.
Azure Data Factory https://azure.microsoft.com/en-us/services/data-factory/
and
Microsoft Power Automate - https://flow.microsoft.com/
Conversely, who knew there was so much money to be had in performing ETL on API payloads!
There are definitely healthy businesses in different market sectors that have popped up to do DataA <=> DataB
https://www.wsj.com/articles/amazon-scooped-up-data-from-its...
AWS presents so many options that it's a bit overwhelming.
This has always killed me about SaaS-es that don't provide any decent method for backups (and less so: restores), especially for data modifications (eg, accidental deletions) that there's no ability to rollback.
Love how these things keep getting re-invented.
As they are in the same category (Cloud technology), I won't be entirely surprised if Ionic renames their 'AppFlow' name to something else. (Unless they're feeling lucky in winning a trademark lawsuit)
This is a bit simplifying answer.
Segment (or RudderStack) is primarily for routing event-data from your apps to multiple destinations (cloud SaaS, warehouse etc). On the other hand, AppFlow is for syncing data that lives on cloud applications with AWS products like RDS, RedShift etc. So, both the nature of data is very different (event-streams vs database update batches) and also the destinations where they are sent (multiple cloud applications + data-warehouse VS only datawarehouses/databases).
Segment also has a product similar to AppFlow called Segment Sources but that's not what Segment is known for. There are a bunch of other products in that space like Blendo & FiveTran.
Very, very crowded space.
This depends on price and by that time its possible that Google Cloud Platform and Microsoft Azure will start jumping in on the integrations market too, further squeezing the likes of some enterprise-only integrations such as Tray.io which still has less integrations than Zapier in general.
It's still too early to tell, but we'll see how fast Amazon adds support for more API integrations or if GCP and Azure will join in and do the same.
90% of the market for Zapier is people who have no technical knowledge. Those people won't be jumping on AWS any time soon.
Obviously pulling the 90% out of my ass, but I'm heavy Zapier user for my e-commerce business. I have (Or had - Haven't renewed as e-com has replaced my DevOps income) multiple AWS certs and I'd still much prefer Zapier over this. Especially when you can run custom code on Zapier for more complex cases anyway.
I'm sure there's way more I can do as well. I have to find the balance between automating and just telling an employee to do it But automation leads to less mistakes