Take a look at https://pipedream.com/apps/openai/actions/summarize
50 karma · joined February 12, 2019
We do a little work to create the shell of the app integrations, e.g. we add the logo, for OAuth apps we configure the details of the OAuth authorization / refresh process, etc. Then you can develop any sources or actions for the app and publish them for all Pipedream users.
The best place to chat is on our public Slack or GitHub, if you're interested. See https://pipedream.com/support .
We build our own containers and still run them on Lambda. The container base image contains the necessary language runtimes so that we can execute both Node.js and e.g. Python in the same workflow.
Do you think it would make sense to add an OpenFAAS integration to Pipedream? Then it would show up when users searched for it in workflows, and have a dedicated page at https://pipedream.com/apps . Even if we just end up wrapping HTTP requests to kick off a function, we've found it's helpful because users are often searching for FaaS platforms by name when they want to integrate with them in a workflow.
Still in the idea phase, but let me know if that resonates.
If you're interested, join our community and you'll get early updates on stuff we're working on (GitHub integration, TypeScript / other new language runtimes, etc.): https://pipedream.com/support
You can see our pricing at https://pipedream.com/pricing and a deeper overview / FAQ here: https://pipedream.com/docs/pricing/
1. We plan to publish an execution environment (as a container) that simulates the core Pipedream runtime. You could run this locally or as a container on whatever container service you'd like. 2. We get a lot of requests for a fully-self-hosted / open-source version, and we're considering how to approach that as well.
Please follow the issues below, and let us know if you have any more feedback!
[1] https://github.com/PipedreamHQ/pipedream/issues/1635 [2] https://github.com/PipedreamHQ/pipedream/issues/954
* Would technical docs on our privacy and security practices help? e.g. how we use AWS, specific security controls, automatic data deletion policies?
* Yes, the product is still in beta but maturing quickly. Happy to answer any specific questions where you think lack of longevity will be an issue for you.
* Paid plans for individual developers, teams, and enterprises are coming soon. We started with a free tier to encourage experimentation and solicit feedback while the product is in beta. This has worked well, but lack of pricing is one of the top pieces of feedback - I hear you clearly on the concern.
* We've raised the quota for early users who need that added time, and we're happy to do it if you expect you'll need that for your workflows. In general, we expect the 30 min daily quota to stay that way for users of the free tier. You'd then be able to pay for additional time.
Let me know if that helps and reach out to dylan [at] pipedream [dot] com anytime.
Event sources are one piece of that. RSS is one type of source, but we're creating sources for Twitter, Slack, Github, and all of the apps supported on Pipedream today [1]. Sources expose a single API for retrieving events from many platforms. So if you just want to run code on new Twitter mentions, Slack messages, or new Github PRs, Pipedream tries to abstract the API-specific details, so you just
pd deploy
Event sources let you treat an RSS feed like an event stream: Pipedream polls the feed for you, and exposes a REST API and private SSE stream for accessing new items from the feed.
You can also trigger a Pipedream workflow on new items. We built 12 example workflows (linked on the page) for doing all sorts of things with RSS feeds, for example:
- RSS to Email
- RSS to AWS EventBridge
- Run a Puppeteer script to take a screenshot of every new item in an RSS feed
- etc.
Give it a try and let us know if you have any feedback or suggested improvements!
Would love if you had a chance to give it a spin. We're always eager to hear what can be improved.
For example, you can use the CLI to run this script every 15 seconds:
echo 'console.log("Hello, world")' > cronjob.js
pd deploy --run cronjob.js --timer --frequency 15s # cron expressions also supported with --cron
Behind the scenes, we package your Node.js code into a Pipedream component [2] - just a terse way to express Node.js code and its metadata in a reusable way. This interface will improve and evolve over time, but we'd love any feedback y'all have on the component API, too.
If you haven't explored the Pipedream platform at large, give it a go at https://pipedream.com . We let you run serverless workflows, hosted on Pipedream infra, optimized for integrations between services. You can think of us like a Zapier for developers. Check out the docs at
Happy to answer any questions below - thanks for the read!
[1] http://pipedream.com/community
[2] https://github.com/PipedreamHQ/pipedream/blob/master/COMPONE...
Paid tiers are coming soon, and if you hit our daily compute limit before that, we're happy to raise your limit to make sure your workflows run without issue.
Let me know if you have any other questions! Feel free to reach out to me directly at dylan [at] pipedream [dot] com.
Now that we've got a new CSP running in production, it's nice to get violations sent to Slack so we can identify other potential issues.
Tools like https://report-uri.com/ are great for reporting on CSP violations, but we found this was an easy way to roll our own and get some advanced functionality (SQL, running Node code on each violation to filter and format the violation JSON) with little work.
You can copy and extend this workflow in any way you'd like!
Spotify no longer notifies you when a playlist you follow adds new music, so I wrote a little Node.js code to list all the playlists in my library owned by others, and checks them for new tracks once an hour.
The code for diffing tracks is a little complex, but Pipedream makes it easy to connect my Spotify account and run Node code that uses the OAuth Bearer token tied to my OAuth grant. It hosts the code and runs the job on a schedule, giving me inline observability, emails me on errors, built-in state via the this.$checkpoint variable, etc.
Would love feedback on the workflow and the product. Pipedream is still a young product, so we'd love to know what we can improve. Thanks!
It's a great question. AWS is incredibly powerful (we use it at Pipedream), but we think Pipedream is better optimized for building workflows and integrations between developer apps and SaaS services. The idea is that we'll save you a lot of time creating and managing these integrations.
We're trying to strike that balance of giving devs the control they need without having to worry about the stuff that shouldn't matter when they're building integrations.
A few ways we differentiate specifically from AWS and other cloud platforms for the integration use case:
- One click "triggers" that enable you to run a workflow on an HTTP request, cron job, or email [1], vs. having to e.g. Terraform a CloudWatch Events schedule or setup an API Gateway / SES email endpoint.
- You can connect third-party accounts within Pipedream and we manage the OAuth flow, giving you programmatic access to your OAuth access tokens. You can manage API keys in the same way [2]. If you've ever had to manage the OAuth authorization flow yourself, storing refresh tokens and generating access tokens, you know how nice it is not to have to manage.
- When you just need to checkpoint the items you've previously processed (common when you're pulling data from an API every few minutes), you don't have to store the data in external state. You can use our built-in checkpoint API to store and retrieve basic state [3].
Also, while you _can_ write Node.js code, you don't have to. Many of our users don't know Node, and exclusively use the actions we've added that abstract common operations [4]. As we add more actions, the tool should be more accessible to "no code" workflows and less technical users.
We'd love if you had a few minutes to sign up and use the tool! If there are things you still like about AWS and want that control in Pipedream, that's exactly the kind of feedback that would help. Send me a note anytime:
dylan [at] pipedream [dot] com
[1] https://docs.pipedream.com/workflows/steps/triggers/
[2] https://docs.pipedream.com/connected-accounts/
[3] https://docs.pipedream.com/workflows/steps/code/#managing-st...
All of us were early employees of BrightRoll [1], an advertising marketplace that handled 10 million requests / second at peak. We've scaled large systems in the past and we've built this system with scale in mind.
Believe it or not, most workflows are extremely low volume. That, in part with the design of the system, allows us to offer the generous free tier.
We're planning to introduce paid tiers soon. You can see my comment on the parent thread that addresses some of the general questions around pricing asked by others in the thread.
This was a very early release of the platform and we hoped offering the product for free would encourage experimentation. To a great degree, we've seen that, so we think it was the right choice.
https://www.youtube.com/watch?v=9ZGNP1-1vyg&feature=youtu.be
You can load the workflow I created for the video here:
https://pipedream.com/@dylburger/template-workflow-to-proces...
and press the Fork button in the top right to create a copy of it in your own account. You can modify that copy and run it for free.
The video isn't perfectly polished and I made a few mistakes, but we've built Pipedream to make it easy to debug your workflows, too, so I hope the mistakes help you understand part of its power!
A couple of notes on your specific use case:
* There's no webhook event for new commits. A PR (like I show in the video) or a push might be the best event to listen for. You can then run Node code to list commits associated with that push. See the API docs for commits here: https://developer.github.com/v3/repos/commits/ . * I wasn't sure specifically what build process you wanted to run, so I end the video reviewing how to add new steps to the workflow and walk through a few options: you can run any Node.js code or any of our pre-built actions. You have access to the /tmp dir on Pipedream and can use a package like child_process [1] to spawn some commands if you need to run anything on the shell. Unfortunately you don't have access to a full shell but let me know if what we provide works or doesn't — we'd love the feedback.
Let me know if that helps or if you have any questions!
[1] https://www.freecodecamp.org/news/node-js-child-processes-ev...
Your code runs in its own Node execution environment and VM, completely isolated from other users' code.
Happy to answer any other specific questions you have! Feel free to reach me at dylan [at] pipedream [dot] com .
I'd love if you had the chance to sign up. It's free to use and we'd love the feedback.
We've found that for integrations or common automations, building on a cloud platform is too low-level. Frequently you're working with more than just a Lambda. You've got to manage a number of services that aren't core to your use case (e.g. API Gateway, IAM, Cloudwatch Logs, etc.). There's a lot of boilerplate config, and many of these services are tangential to the core logic of your app.
We operate at a higher level and just give you an execution environment to run Node.js code or stitch together pre-built actions, similar to integration platforms like Zapier. We manage the HTTP endpoint, function config, and give you built-in observability. We also provide higher-level services, like built-in app integrations and OAuth / API key-based authorization. Building this yourself on AWS can take time.
This kind of abstraction isn't great for every application, but we think it shines when you're building integrations between services.
I'd love if you had time to sign up and see how this works! It's free to use and we'd love the feedback.
Let me know if that answers your questions, or if you have any more!
I completely empathize with the time involved learning new tools. I'm happy to create a tutorial specific to your use case. We've built a few workflows to process Github events and I'd love to show you how this works end-to-end.
Is there anything specific you'd like to do with the commit / merge events, just so I make sure the tutorial targets your use case?
Feel free to reach out directly if you'd like to talk more at dylan [at] pipedream [dot] com.
I especially appreciate the discussion around how and why Pipedream is free, and the concerns around the lack of a visible, scalable paid tier. Those concerns are valid and I want to give you a little more insight into our thinking on this and what the future holds.
Workflows created by one developer can be forked, run, and modified by others. We all build a lot of the same integrations across companies and believe if this code is shared and executed on a common platform that’s purpose-built for running these workflows, it’ll save us all a lot of time.
We’ve worked with thousands of alpha users to understand how they’re using the product and considered what features enterprises ultimately will want and need to pay for. These include: higher workflow limits, private workflows and actions, SLAs, premium support, and more. We didn’t launch with that today because we’re focused on getting feedback from individual developers who will be the majority of users moving forward.
Of course, you can’t run a business solely on free workflows. We’ve set some limits on these workflows that help us control excessive use [1]. Our team has a wealth of collective experience scaling software companies on the business and tech side, and we have confidence that we’ll be able to retain a generous free tier while building a sustainable business.
I empathize with the skepticism of Pipedream and of hosted platforms in general, and welcome any more specific questions. We truly love feedback. We’ve implemented some ideas that we believe facilitate the developer experience for building workflows but are looking for y’all to test that, validate or invalidate it, and give us specific thoughts as you have time.
dylan [at] pipedream [dot] com
We've found that for integrations or common automations, building on a cloud platform is too low-level. Frequently you're working with more than just a Lambda. You've got to manage a number of services that aren't core to your use case (e.g. API Gateway, IAM, Cloudwatch Logs, etc.). There's a lot of boilerplate config, and many of these services are tangential to the core logic of your app.
We operate at a higher level and just give you an execution environment to run Node.js code or stitch together pre-built actions, similar to integration platforms like Zapier. We manage the HTTP endpoint, function config, and give you built-in observability. We also provide higher-level services, like built-in app integrations and OAuth / API key-based authorization. Building this yourself on AWS can take time.
This kind of abstraction isn't great for every application, but we think it shines when you're building integrations between services.
Let me know if that answers your questions, or if you have any more!