Hacker News Reader

Comment by leros | Hacker News Reader

I built a service around helping podcasters automatically convert their audio podcast into a YouTube channel. I went through tons of review with Google in order to get access to the YouTube API and make sure everything I was doing was in compliance with their terms - literally months of back and forth. I had been testing in my development and staging environments against their API for 6+ months. I launched in production, got a few videos uploaded to YouTube, and they disabled my API key. I spent months emailing them and never got anything more than the same boilerplate copy/pasted answer. I could have pivoted or something, but I just shut it down and moved on. Lesson learned.

View on HN

I used to not bother with APIs and instead use selenium to drive the websites. It neatly avoided all of the politics around API access.

That may be a CFAA violation, a felony in the United States. See Ryanair DAC v. Booking Holdings, Inc.

This literally made me lol. Not sure if it’s true. I might be true. But come on!

That's because Booking was also committing some type of misrepresentation and taking revenue away from Ryanair through their browser automation. Even then, the infraction was sooo bad that they got a $5k fine.

I think the hiQ Labs vs LinkedIn case is a better representation that scraping is generally allowed: https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn

When it comes to CFAA violations, corporations get a $5K fine, while individuals get hounded to suicide.

Start a company then do my nefarious work under that, got it!

You should do ABSOLUTELY EVERYTHING in your life under a company. And I mean EVERYTHING

My wife hates it when 1099 her for services rendered, especially since she refuses to bill against the PO I set up for her. The kids at least accepted NET15 payment terms. Although the oldest said if I short pay him for lawn care again, he’s going to take away my early pay discount.

Have you considered short paying him again through the same entity? After all, that's one use case of limited liability ;)

Kafka is smiling appreciatively at your approach to marriage

corporations are more people than people.

Unironically, yes.

How does that work, though? Setting up a company has an initial cost and then recurrent costs (accountant, etc). Are the benefits that high for the average Joe?

It's hard to answer without specifics, even if you're not doing anything neferious there are a lot of benefits to putting an entity between yourself and your customers. It depends on where you live and what your business is of course.

> In November 2022 the U.S. District Court for the Northern District of California ruled that hiQ had breached LinkedIn's User Agreement and a settlement agreement was reached between the two parties.

Reminder that the earlier ruling was overturned, it is no longer clear whether scraping is legal or not.

Very Probably No way they could find me.

Just say you train an AI on it and it's all fair game.

Good thing I am not an American, don't live in the US and have no intentions of visiting then.

(I gave you an upvote, I don't think your comment deserve to be nuked for bringing up a possible legal risk)

HTTP is an API of sorts…

> HTTP is an API of sorts…

True, but it is an API that they can't easily deprecate on a whim.

But they can quickly change the "structure" of that API.

I believe this is where smarter tools like Kadoa [0] can be of help. It detects data structure changes for existing workflows, and adapts to them.

[0] https://www.kadoa.com/

A small problem compared to not having an API key and being stonewalled as to why.

I worked for a very early billpay company where you could pay your bills online to vendors, even if the vendor didn't support it. We used API's where we could, but where we couldn't...

We had a whole team dedicated to keeping up the changes vendors would make to their websites that we scraped for info. The team was called, of course, "Scrape and P(r)ay".

Isn't that similar to deleting your API key except you can at least fix the structure of the selenium one.

If you build your scraper to find data on the page based on the shape of the data itself instead of the structure of the page then it will be resilient to most changes that don't materially change what data is displayed on the page.

So, prefer regex over css selectors, and css selectors over xpath, where possible. And don't select based on nesting or position if possible.

You're one CAPTCHA added to the flow from this access being effectively deprecated.

In the AI era, is this as true?

Depends on your development and per-action cost. And on the possible latency. It also changes your whole stack from "send a request" to "emulate each step in a browser while taking screenshots at (hopefully) the right event/delay" - that's a huge difference.

My first Saas was built around Facebook, Twitter and Reddits API.

Facebook broke like monthly and required random updates, sometimes documented, sometimes not. Zero support.

Twitter worker flawlessly until I hit some limits and there was no way to increase them because I didn't hit some other limits. No way they would talk to me.

Reddit just worked until I gave up.

What I learned is to never again do any business based on someone else.

A good lesson, but a more nuanced one is to do sure diligence. There are companies I would never build my business around (Google, Oracle, etc.) for a variety of reasons. There are ones I know I can trust (at least until they change). There are many where I need a contract.

Change is also a constant, and there are plenty of good companies give bad, and a few in reverse. So due diligence is an ongoing process.

That dramatically raises the price of SaaS, so I use it much less than mainstream industry practice, but much more than zero.

> What I learned is to never again do any business based on someone else.

That's a little dramatic, don't you think?

Supply chains are a core of many (most?) businesses out there. They depend on others to get their business done. No supply -> no business.

What you probably meant is not building a business based SOLELY on irreplaceable business relationships. Unfortunately in a lot of cases involving the tech giants and little startups, there are no alternatives.

More or less, however I really try to not build on anyone else ever again.

I can't find a single dependency in my serious money generating chain right now that isn't replaceable with a similar product within a few minutes or hours if necessary.

My most recent project didn't even last a full year as there was no replacement that worked for me for ChatGPT in that moment. It was built to die, a gamble essentially. I wouldn't waste my time for things like this if I didn't had the end calculated in already.

Same experience, slightly different. It made me realise why some parts of an MBA (e.g. strategic place/partnership/spring boarding from a brand) actually have value I never saw before for tech companies. There are essentially king makers in certain fields - either by their apathy at first-movers, or their choice on who they allow to continue.

>Lesson learned.

And that lesson is don't base your business model around playing in someone else's waled garden.

Precisely that.

Funnily, if I understood correctly, he already had that lesson with Facebook. Let's see what comes next...

Their loss of additional content, traffic and ad revenue.

The impact on google losing $10m of revenue is exactly equal to the impact of John Doe losing $10m of revenue

In a competitive market John Doe would go to one of googles competitors. We don't have a competitive market. We have winner takes all market.

Sure, its $10m today, but think of all the other potential startups that they're doing this to, and how many millions they're throwing away, and billions over the long haul. Reddit, Twitter, and company all got big because they had open APIs and people were able to use them extensively for really creative things.

I agree, Google has swallowed up the video streaming market unfortunately.

I keep thinking back to how Vine was basically TikTok, and they threw it away.

>Sure, its $10m today, but think of all the other potential startups that they're doing this to, and how many millions they're throwing away, and billions over the long haul.

Google made $300B last year. "billions over the long haul" is a lot of money, an unimaginable amount even, to you and I. But to Google?

> Google made $300B last year.

No, Google made $73.795B last year. Revenue is not profit, net income is profit and what you ‘make’ as a company.

Revenue is not profit.

I mean, they "make" both. "Make" does not imply profit. If you say a company "made" a revenue figure, then you mean it in the revenue sense. It's contextual.

No, it’s not contextual. “Company X made $N billion dollars” only means one thing, net income.

Sorry bro it's contextual.

100% contextual. In every business where I've been an executive, or had access to the budget, we talk "made" as gross revenue and EBIDA as a stand in gross profit and specifically call out net profit in internal meetings. For example, "We made 10 million with an EBIDA of 1 million and net of 200,000.00." Using EBIDA to talk to potential investors and as a guiding metric if we didn't have a well established gross profit formula that followed GAAP.

That’s like saying a trading company made a trillion dollars because you counted all of the securities sales but not the costs.

I don't think that's nearly as universal an assumption as you're claiming. Like--I made $X,000 dollars last year, but I also have a mortgage and like to eat food.

By that logic I make what I can save, lol

right, which if you follow that assumption you can see that things are not as great as you thought.

The assumtion things were great wasn’t mine. I live in a HCL area and am not able to save much so in this sense the ‘profit’ of my work is close to zero. That is not great at all…

Well yes.

Companies get to write off expenses of existing

People don't

Mortgage accumulates equity. Bad example

Mortgage interest payments don't

Your assets might inflate in value, but that's nothing to do with the mortgage

Even if we subscribe to this interpretation, the overall point stands.

Revenue isn't profit, but your wasted business expenditures are properly measured against your revenue. That's what they're spent from.

First off - how much of revenue usage last year was on fines etc. That is to say profit they could potentially have had.

Second there is lots of revenue you use for "business" because you don't want to pay taxes on the profit and anyway it is giving you something you like to have.

How much of Google's revenue is used to do things for top executives and people with power in the company that is really something they (the executives) should be taxed on but is instead a business expense of google (cars, transport, 'working' vacations, security, super cool chefs preparing meals at the company...) Hard to say really because if you knew the answer it would actually be something they were taxed on. But it's not 0 - sure probably not a billion, but a couple hundred million splashed around wouldn't surprise me.

How much of Google's revenue usage is for wages and other forms of payout to executives etc. (stock) that does not get counted as profit but of course it is amounts those people want to have.

Google made $73.795B profit last year, and expensed slightly over 262B - some portion of which the people who run Google no doubt personally thought of the way we would consider profit in our day to day existence, and another portion of which were fines for things they did in getting the rest of the money.

Literally none of that is relevant for much the corporate entity made (I.e. what the shareholders got).

Obviously the fines, which are the first thing mentioned, is relevant for how much the shareholders got.

Literally the other stuff is relevant for how remunerative the people who actually run the companies, day-to-day, feel those companies are (put in day-to-day because I got the feeling you might give me a lecture about the shareholders actually running the company)

Personally I think you have a decent point since in practice employees of GOOG are shareholders too. Although I don’t know if their interests matter to the corporate entity because I don’t know what percentage is held by insiders.

This is a bizarre nitpick.

1. I know it's revenue - the GP says The impact on google losing $10m of revenue

2. Do you think the 10m that Google would have booked from API usage would be booked as pure profit?

Those startups are their competitors. You don’t have to pay millions of dollars to acquire a competitor that was never started because of your inconsistent API policy.

Google makes enough money that losing several currently non-existent revenue streams that are theoretically $10 million apiece isn’t hurting them. It’s hurting their users.

Google is all about ads - why would they give a shit about the users?

It's a testimate to the health of our free market that the company throwing away millions, and billions over the long haul, is the overwhelmingly dominant market leader.

Take it from someone who watched with interest, Reddit and Twitter did not get big due to open APIs. They got their first big steps before they even had APIs... In the first year of twitters launch it was in the news in the UK at 300k users in press releases about how it was a "new form of communication" etc... 2008 was actually when the original set of API-laden websites began to fade because with the recession, we could not have nice things be free so much. Many API services fell into disrepair in 2009, certainly the peak of APIs for the UK was around 2008. Check out Tom Scotts video on this for a picture of the open API internet many people thought would occur (and which did exist in some places until FAANG began to productise and dominate and use accounts across products which do not incentivize APIs). Yes it is true open network graphs/facebook etc had some API access, but the APIs before were more single purpose and numerous; facebook lets you explore their network and to some extent data, but that's not the same as the useful APIs that became restricted, paid or were cut in the aftermath of the recession.

Competitive markets are by far more the exception than the rule. It's just not how capitalism works (because pretty much any competitive market is ripe for “consolidation”, which increases aggregate shareholders value by reducing competitive pressure).

If you want competitive market in a capitalist economy, then you need very active state enforcement.

>Competitive markets are by far more the exception than the rule it's just not how capitalism works

Empirically this absolutely isn't the case; the majority of listed companies have fairly low margins, especially non-tech companies, which can be trivially seen from their financial statements. A low profit margin means a competitive market (because if it wasn't a competitive market the firm could raise its prices to obtain higher margins).

Might there be other reasons to report lower profits?

> A low profit margin means a competitive market (because if it wasn't a competitive market the firm could raise its prices to obtain higher margins).

Not necessarily: if prices are elastic then even a monopoly can aim for low profit margin (in percentage) in order to increase profit. What matters is how much total profit are being made, margin only matters when measuring risk.

Also, corporations are social structures, and low competition also encourage complacency in the corporate structure itself, which drives costs up and reduces profit margin.

In addition, a low profit margin may increase lifetime profits by deterring potential competitors.

The markets are absolutely competitive. Google has to pour money into youtube to keep it actively developed and popular. Yet they still lose live streamers to Twitch, video uploads to all kinds of niche specific platforms, and their paid content isn’t coming close to things like Netflix, max, prime, etc.

They are now hemorrhaging search to OpenAI that popped into public existence just recently.

To claim there isn’t competition in these markets is completely ignorant. “A big player eats 80%” isn’t anything like a monopoly/duopoly scenario where there literally isn’t any competition or product advancement for decades.

If you wanna see lack of competition, look at government granted monopolies on utilities. Guaranteed but capped rates means you reduce investment right to $0 and cut costs as much as possible since there is no other way to make money. That “state enforcement” you are calling for is how you end up with PG&E and scenarios like all insurances companies pulling out of the state.

> To claim there isn’t competition in these markets is completely ignorant. “A big player eats 80%” isn’t anything like a monopoly/duopoly scenario where there literally isn’t any competition or product advancement for decades.

Please tell me how many “product advancement” in Google search or YouTube over the past decade… From a consumer's perspective all that happened was enshitification and despite being owned by a search engine company, YouTube never managed to ship a functional search on their platform.

> If you wanna see lack of competition, look at government granted monopolies on utilities.

Utilities are “natural monopolies” though, and as such they should be state owned. Making a natural monopoly owned for profit is a recipe for rent seeking, and that's why it was promoted …

> That “state enforcement” you are calling for

No, the state enforcement I'm calling for is proper enforcement of antitrust laws, forbidding consolidation through M&A and disbanding companies megacorps. That is to say, what existed in US's golden age.

> Google has to pour money into youtube to keep it actively developed and popular. Yet they still lose live streamers to Twitch, video uploads to all kinds of niche specific platforms, and their paid content isn’t coming close to things like Netflix, max, prime, etc.

Phew, here I was stressing out about lack of competition, but you helped me relax.

Google has to invest into Youtube lest it loses out to Twitch (Amazon), Netflix ($35bn per year), HBO Max (Warner Bros, $50bn per year), (Amazon) Prime, Plus (Disney, $90bn per year).

My faith in humanity is restored now that there are alternatives.

From other MegaCorps.

That’s competition, don’t shift the goalposts.

I picked well-known names as examples. There are thousands and thousands of niche providers.

That's not competition, that's an oligopoly and they are mostly focus on their own niche for which there isn't more than a minority competitor (there's no competitor to Youtube on for non-live video, Twitch is in a monopolistic competition situation for game streaming, so is netflix on TV series, and Disney and HBO each have a monopoly on their IP).

Claiming that this is a competitive market is a joke.

I was thinking about how to solve this given that one of the primary problems is that of fast, global content distribution. I like the idea of paying people in crypto as part of a ledger transaction to host and serve content, like bittorrent with a crypto payment. Unfortunately I can't also think of a way to prevent such a system from being abused to distribute harmful media such as CP. I guess it's not like this isn't a problem with BitTorrent today though.

Regulation might here - something like minimum mandatory 3 months notice for shutting down the API keys. Considering the average age of our politicians, I doubt they’ll understand what “API” is, much less be willing to take on giant tech corporations even if they did understand the problems

> […] much less be willing to take on giant tech corporations […]

I’d argue that, on the contrary, it seems to be all the rage lately.

Not really. There is a big difference between making noise and actually doing something about the power of big tech.

Yeah, what’s popular is actually to crap all over the first amendment by mandating private companies to allow whatever each politician’s special definition of “free speech” is (but of course probably also not allowing whatever the politician’s definition of “dangerous misinformation” is).

As for the actual anticompetitive power of big tech, absolute crickets.

And yet: https://news.ycombinator.com/item?id=41164240

It isn't much, but I'd like to see that as a start.

>The impact on google losing $10m of revenue is exactly equal to the impact of John Doe losing $10m of revenue

no, it's not, see declining marginal utility of wealth

https://www.economicshelp.org/blog/12309/concepts/diminishin...

https://www.investopedia.com/terms/l/lawofdiminishingutility...

Attempted to build a similar thing a few years ago when living in Tokyo.

Provide a selection of restaurants within a 1km radius and automagically provide 3 recommendations based on my preferences.

We had multiple API providers available, Tabelog, Gurunavi and Hotpepper, all required a paid developer license. We still needed to use the Google maps API to get the user's current location though.

This was also just when Google Maps suddenly raised their API pricing. After spending a couple of weekends building a working prototype, we stopped as we couldn't justify the cost of paying multiple API providers for basically 3 guys looking to save 10 mins deciding where to get lunch.

Also, this kind of app is a common theme if you frequent meetups in Tokyo. There's always at least 1 person that has built such a thing.

> We still needed to use the Google maps API to get the user's current location though.

I’m sure as somebody who actually built a prototype, you understand the situation better than me, but I’m curious: why would just getting the user’s lat/long require anything you’d pay for? Wouldn’t the OS or browser provide that to you?

Oops. I meant that we were using the maps/places API to get lat/long of the restaurants using the street address from the other API providers, then using that to calculate a 'N mins walk away'.

We were just testing the abilities of the various APIs. Google was the only that provided travelling time estimates.

Cool idea, I was almost about to do the same app/service. I didn't know it could end like that. (Naive, I know.)

Have you seen https://repurpose.io? They existed before I started working on my service and they do the same thing.

Do you know if there is any open source tools in this field that a programmer can work with/automate, I have mainly textual and graphic content, a little bit of video, but thinking to branch out to podcasts and music soon, unfortunately the other products supporting text seem pricey and geared to big teams.

on edit: obviously I can write my own, but I am hoping for a project that has already done a bunch of things that I can extend for my needs, as writing my own would be at the point where it would be more cost benefit to purchase.

I run a webapp that uses the YouTube API too.

I believe I am entirely in compliance with YouTube's terms and policies, but I was flagged for some bullshit as well and got the boilerplate responses. So what I decided to do was hide it for logged out users so it looked like that specific feature was entirely removed to the dudes in India that review this stuff... One year later and my fingers are crossed we don't get another audit.

We went through a three-week Facebook API shutdown due to a clear glitch - Meta support couldn't go outside the script, or escalate in any way beyond "a supervisor will email you". Only resolved when our CEO found some VP at Facebook on LinkedIn and got them to escalate it internally.

(The dev community thread is full of people still impacted, so I think they literally just edited our app's flags directly.)

Even tried to invoke GDPR's rights to be exempt from automatic decision making, but their privacy questions email address responds with "nope fuck off" to those.

Basically any hope of solving a problem with a Meta property is to know employees who can escalate your issue. Sometimes your friend at Meta also needs to argue with the bug owner to not close it frivolously.

I've heard that serving them process is another way to talk to a human being.

It's the same with Google.

Within the customer service industry, they'd claim a great "deflection rate". That's a metric that all of these large companies hold around any kind of help/support channel that may involve humans because people are a cost. It's often covered by some kind of satisfaction metric based off post-issue surveys, but fundamentally, if you just go away that's success.

There are often complaints here about what amounts to bean counting affecting other aspects of business. Customer service at larger scale is costly so attracts a section of very analytical leaders. They don't get, or ever prioritize, the human elements. It's only when satisfaction numbers are bad, or another exec outside of customer service takes action, that things improve.

The scale part kills the customer. You can have great support at one size. Once you grow then leadership, structure, and the culture change. These analytical leaders don't want to carry over the culture and structure because it comes at a cost. It really needs force and support from outside of customer support leadership to maintain it.

Had this issue with Ubisoft recently. Installed Trackmania through Steam for free, then had to create a Ubisoft account, then had to buy the Trackmania annual pass for $20. I guess I followed the wrong flow or something, but I ended up with the purchase on my Ubisoft account, but I guess their system also auto-creates a Ubisoft account for Steam accounts? So the purchase on the Ubisoft account ended up on the account not tied to Steam.

Emailed support about it on 13/07/2024.

Got back a reply on 23/07/2024:

> Our Support Team is currently experiencing high case volumes, so we are reaching out with a message to check-in and make sure you still need help!

> If you've found a solution to your problem already, there's nothing you need to do. Just ignore this message and your case will automatically close in 2 days.

Didn't see the reply until the 3rd day.

> As we have not heard from you in the last 2 days your case has been closed automatically.

Excellent customer experience.

> Customer service at larger scale is costly

I think people, even on HN, grossly underestimate how much customer service costs at any major company that works with the public.

Most calls to customer service are easily self-serviceable. My brother used to work for X-Box Live support, and he said over half his calls were for password resets, something people could easily do on their X-Box.

But then there's bad actors. Another significant chunk of his calls were from people complaining about being banned, and he'd see the reports of their spamming racial slurs.

Not to mention how many people try to abuse customer support in attempts to take over someone else's account.

Yeah...customer service is expensive. Even if you automate as much as you can, but offer an appeal process, that appeal process with be absolutely abused by bad actors, so that will end up automated as well.

My sympathy is veeery limited. But once you have an app that does the exact opposite, please tell me!!

And, just in that particular case: I'm veery okay if that would be a cloud service.

YouTube to podcast is way easier than the other way around. I'm sure solutions already exist.

Are you based in EU? If yes, then I would a complaint letter to the competition bureau. And you can CC Google offices in your country.

I am curious if you would be interested to retry the idea? I might have an in with the YouTube team. I feel like it's a shame to let this go. Would you be open to chat? Please reach out r@pehul.com

If only we weren't living in a world run by abusive mafia... o_O

open source your code ?

Couldn't you have bypassed them and used `yt-dlp` or something similar?

Why get Google/YouTube's permission at all about this?

I was creating YouTube channels and uploading videos to them on behalf of my users. That requires using the YouTube API.

Can you let the user bring their own API key?

No. The default API quota is not large enough to upload videos. You would have to contact Google, explain your use case, and jump through several hoops to get the quota increased... which is a huge process.

I once tried using a Python library to upload videos with an API key. All videos uploaded got forced private due to "spam".

Yeah. Uploading legit videos is non-trivial. And if you ever upload the same video twice (which you think you might do during testing, right?), it's a violation of their terms and they disable your access.

reread: The project was going in the other direction, From a pure audio source to a Youtube video.

Reply on news.ycombinator.com