Thank you Anne!!
2,759 karma · joined June 1, 2013
I am large, I contain multitudes.
Email: rememberlenny@gmail.com
Web: rememberlenny.com
I touched on the video APIs above, but regarding transcription - I have a lot of thoughts.
We chose to work with AssemblyAI (https://assemblyai.com) after trying AWS's transcript service, RevAI and Google's Speech-to-text API.
First, we started with doing manual transcripts through Rev, but the cost was unmanageable at scale. We were really happy with the quality, but couldn't charge $100 per video, so we needed a cost-effective automated solution.
I then found an old blog post from Descript's co-founder Andrew Mason, who talked about which speech-to-text API they decided to use. The blog post is old, so the metric used are going to be irrelevant, but I was impressed they decided to use Google's API.
We implemented the GCP option, I was shocked how slow it was and how expensive it was. For one, the quality wasn't that great, and to use the lower cost option (audio-only), you need to do some additional FFMPEG based transcoding, which is very error prone. Because we receive a range of video types from users, it was causing more problems than was worth dealing with. Also the time lost made the cost savings irrelevant.
Enter - AssemblyAI.
I did some research around what other companies were using today, and saw they have great ratings on G2 (https://www.g2.com/products/assemblyai-speech-to-text-api/re...). The CEO jumped on a call on a Sunday, when I was trying to improve our transcript processing time, and after testing the API, I was shocked that the transcript quality was closer to the human done transcripts I was getting, at a cost significantly cheaper than the Google option.
Conclusion - we needed speed, quality, low-cost and support. AssemblyAI won on all these fronts.
Email: lenny@milkvideo.com
In short, I'm just one person building this - so I'm sticking to what I know best. I want video to "just work" without having to worry about some video format/extension/containers that I have no idea about.
There are a number of video processing services, but Mux really is the best. The API is simple. They have a ton of really nice helper functions, that I use a lot (like timestamped thumbnails, preview gifs, and VTT storyboard generation), which I could easily spend a few days on making, and then countless hours maintaining.
I dont doubt that building video infra is a good idea, but just as I'm not about to train my own speech-to-text model, I'm not going to build out video infra.
At least for me, I'm more worried about the end user experience, and the more I can focus on that, the better the overall product will be.
The backend is a Ruby on Rails application that serves the frontend app's API. This interfaces with the user tables, database, and handles all the "state" of the app.
The serverless stuff has changed over the months, but primarily it handles the stuff I don't want Rails to handle: file uploads, video processing and transcription.
First, huge props to the Mux (https://mux.com) team and product. I can not express how easy it has been to build video (and audio) products. File uploads are handled to AWS/GCP (depending on a few things) and then trigger a serverless callback to Mux.com. Mux was the fastest way we found to turn an arbitrary video file (mp4/mov/etc) into HLS format for quick streaming.
Then once the video is uploaded, we have another serverless callback that sends the video for transcription using Assembly AI (https://assemblyai.com). There are a ton of transcription based services and they vary dramatically in quality, based on the media content. I believe Google/Amazon services were largely built around the need to process phone calls, so unless you may for their "enhanced" models, the quality is surprisingly bad (and surprisingly slow).
I *highly highly* recommend Mux and Assembly AI if you are doing anything video/transcription based work.
To get an immediate update to the end user, we actually process two transcript requests - one that is just the first 60 seconds, and then the remainder of the video. This lets us render a preview transcript in the first 15-20 seconds.
We also have a serverless pipeline for generating the videos, but I won't go into that unless you're interested. In short, a serverless function kicks off a Docker instance running on ECS.
The requests to the serverless apps (mostly Node) have a callback to the Rails app, which then updates the end user state using websockets (which are very easy to use in Rail's ActionCable).
The point of the transcript is to lower the bar on who can review video content. One layer on top of that is moving the technical work (cutting video) into an editorial role (picking the parts that are recommended).
We aren't trying to position ourselves to do the clip picking/recommendation now, but we have already done some machine learning based analysis to make this easier to find. We have a video processing task that looks for "scene changes" based on image threshold changes, so the metadata associated to when a new person joins/slide changes/etc is present.
The original thinking here is that we can recommend "templates" that correspond with certain video (ie. multiple speakers vs single presenter).
The inspiration behind making this is to replace Premiere Pro, so I'd love to understand your MVP to solve your problem.
I do think we can be very performant for you, given that all the processing is done on the cloud, and you are only ever interfacing with JavaScript/video tags. That being said, there is work to do!
We don't have speaker diarization right now, but it's just a feature flag for us. Also the start/end content is something that we don't have active, but is planned for next week.
This book resonates with me.
This reminds me of the "data science" book was from Data Smart, from the Mailchimp CDO, talking how to keep orange juice tasting the same all year round (using a seasonal fruit) or calculating the likelihood of a consumer being pregnant - all within Excel.
I am using RTK for the overall app state and Recoil for the on-page state. I make API requests and store the results in the redux store, but the hooks/prop passing is too slow for handling video players/transcript manipulation.
I initially had everything in RTK, but noticed the render cycle for dispatching to and listening to the store was creating unusual issues.
With Recoil, Im able to represent the video player's current time state, and then listen to it in the other parts of the app. Similarly, when I have the transcript updating the time, the React Context based API performs better than the hook/props.
Happy to dig more into this. I'll reach out via Twitter too!
FWIW - they are Webflow templates, but do a great job at making it easy to manage.
Our focus is helping make a visual clip that is engaging, based on the transcript information.
Here are some examples I posted above:
- https://twitter.com/rememberlenny/status/1339618249575714816...
- https://twitter.com/rabois/status/1310644068326629376?s=20
- https://twitter.com/m_cieplinski/status/1356331228954292224?...
Per the podcast point, last night (phew) we launched the ability to upload audio files and work with them.
We are focused on webinars/Zoom recordings, but now you can upload a podcast and create a promo tile.
These are some other links that were made in Milk Video:
- https://twitter.com/m_cieplinski/status/1356331228954292224
- https://twitter.com/rabois/status/1310644068326629376
- https://twitter.com/rememberlenny/status/1339618249575714816
There are a number of new companies appearing in the synthesizing/organizing space, because there was an explosion in the quantity of video being produced, and the software to parse it exists (speech-to-text, object recognition, better-search, etc).
At least for companies, people intuitively want to share all the "best parts", but we found that people won't actually watch them. Instead highlighting one specific piece that visually looks engaging is a better way to capture attention, and then engage.
Our thought process here is that there is only so much organizing/synthesizing most people will do, but there is an endless about of creation someone can do.
Re: government - For what it's worth, the US Chamber of Commerce is one of our paying customers.
We actually drive people to use Descript for most use cases, that aren't relevant.
Think Photoshop vs Canva.
Since speech-to-text APIs have become really good (props to companies like AssemblyAI (https://www.assemblyai.com/), the transcript-based interfaces are going to become much more common.
Our product goal is to solve the use case around making the visual output, when editing/correction isn't the goal. That being said, the editor should be performant and work well, so lots to improve there.
As an aside, there are a few evolving open-source libraries that consume the output of these STT services (https://github.com/bbc/react-transcript-editor) and make turnkey transcript interfaces.
The newest/most developed one I like is based on Slate, and made by a really amazing engineer at the Wall Street Journal named Pietro.
Descript is hands down the leader in any transcript based video/audio editing. They set the standard for detailed editing and magically manipulating audio/video.
We are focused on the workflow around creating something visually appealing, that uses a Zoom recording in it. Specifically, the transcript-based interface is only for speeding up the review process, but our main focus is on visual templates to drop in the video/captions.
One way to think of it is that we took the Descript Audiogram feature, and built out a workflow that creates a wider variety applicable to marketing/sales related needs.
We are solving the problem where you need to quickly take a video recording and make something you/your team can proudly share on social media, that reflects your company's brand guidelines/visual aesthetic.
It's worth noting, the use case you are mentioning is interesting, but not exactly the workflow we are building for.
We are focused on improving the workflow around a marketer/demand generation/sales person at a company who needs to use existing content to get attention on social media/blog/email.
One of the ways we are thinking about this is around increasing the "shelf-life" of quality content, which doesn't get discovered because its long. This is very much a problem that appeared when businesses became content creators, as opposed to individuals.
That being said, if you have any questions, please email either Ross (ross@milkvideo.com) or me (lenny@milkvideo.com) and we will be happy to help!
Context here - before we started working on the current software, we planned to do a opaque marketplace for post-production video work. To vet the idea, we reached out to companies with webinars and manually edited their videos.
In the process, one finding was that its hard to make styled word-by-word highlighted captions. This resulted in a small utility app that turned SRT and VTT caption files into a dynamically sized/styled caption videos, and later evolved into todays product.
I love these applications of CV for accessibility for the blind. This is exactly what technology is made for.
Feels apocalyptic. In an addicting, oddly fun kind of way.
These results are surprisingly good.
This site is an incredible, concise, and comprehensive resource for people trying to better understand how video works.
I attended the demuxed conference this year, and was exposed for the first time into the nitty gritty of how video works behind the scenes.
Huge props for the content not being a sales pitch, and truly being educational and informative.