Patreon laid off their security team
twitter.com
twitter.com
- - -
“As a global platform, Patron will always prioritize the security of our creators' and customers' data. As part of a strategic shift of a portion of our security program, we have parted ways with five employees. We also partner with a number of external organizations to continuously develop our security capabilities and conduct regular security assessments to ensure we meet or exceed the highest industry standards. The changes made this week will have no impact on our ability to continue providing a secure and safe platform for our creators and patrons, including the upcoming adult ID verification requirement.”
- - -
I have no idea how big their security team is (five may in fact be the entire group they called “the security team”), but I am not convinced that this is necessarily the end of the world for those who use Patreon. I don’t know how much financial data they even store directly (I would assume best practices entail letting their payment processors store actual credit card and bank account numbers, for instance, although I’m aware that may be easier said than done depending on their infrastructure).
While I do think Patreon is going to have Issues (tm) due to having accepted so much VC money, it’s not clear to me that these layoffs were forced by their funders—they haven’t accepted any money recently, and I haven’t heard anything that suggests their revenues have been dropping to a point where there’d need to do serious belt-tightening.
(Also, to people who seem to think any creator could just slap up “a basic microblog” and get all of Patreon’s functionality, that kind of falls into the “not even wrong” category. Replicating what they do would be a lot of work even for the technically-minded, many creators who use them are not technical, and even if you pull it off, there are things that just literally can’t be duplicated as a solo endeavor: all of your patrons may be supporting multiple creators, not just you; patrons may discover other creators based on who their friends support and who those they support are supporting; creators who support other creators just have those charges automatically deducted from their earnings; etc.)
Reading between the lines, this suggests that they've decided to use an external firm to cover these security domains (for now). Without knowing which firm or firms we obviously can't say anything specific about their reputations, but I will say that I know some individuals who operate in this space and there are some very qualified and talented outside security firms out there.
However, the good firms aren't cheap. Obviously we can't know the details, but I wouldn't assume this move was motivated by cost-cutting reasons. Keeping a top-notch outside security firm fully involved could easily cost more than the salaries of the people who were let go.
I don't want to single anyone out, but a quick glance through LinkedIn leaves me somewhat confused about their former security team's structure. I could be missing something, but the only lead I found on LinkedIn was listed as "acting head of security" with a description that they were covering the Head of Security position for now. It's possible that the security team were good engineers, but maybe not the exact engineers/managers that Patreon needed at this stage of their development. Some times a mismatch like that really can't be overcome easily, so employing a known external firm can cover the gaps while the company tries to find a better match for their security leadership needs internally (if they return to internal at all).
This move could make even more sense if Patreon was aggressively heading toward IPO. In this case, the company would need to demonstrate that they've done their due diligence and made a best effort to cover security as best they can, even if that means paying exorbitantly for an external firm that has helped prepare companies for IPO in the past. If their internal security team didn't have this experience (and maybe didn't even have an official head of security) then they didn't stand a chance.
This upsets me so much. Patreon is a perfect example of a company that could operate as a non-profit and everyone would benefit; but, instead, they need to make so much money that they're valuable on the Stock Market. It seems like a net harm for society, here :/
Everyone except all of the employees who built the company with some equity compensation.
Like it or not, a lot of the current and former employees are equity holders in the company. An exit event gets them paid for their ownership piece.
Of course, anyone could start a non-profit Patreon competitor if they really wanted to and operate it as such. However, cloning Patreon isn't as trivial as it might sound to the casual observer. A lot of work has gone into getting the company to this point, and IMO it's fair to reward the people who took on the risk to make it happen.
My argument is that I wish Patreon had been non-profit from the start and such there would never be a time for them to "Go Public"; and, all their employees would already be focused on the common good that comes out of a patronage platform.
I just... I don't have any money, and if I ever made any money instead of losing money on it (currently), I could never imagine making enough to actually pay employees.
Patreon works because they marketed the hell out of themselves. That costs a ton, and it almost certainly was funded by VC capital. I wouldn't be surprised if they didn't operate at large losses for years.
I guess, as someone trying to build a startup from nothing, bootstrapping is just hard. Lord knows I'm very proud of the growth I've had, but I'm still almost certain that nothing will come of the nearly 3 years I've worked on this site. Unless you're willing to potentially bankrupt yourself to build a nonprofit, or fight on alone incredibly slowly, I really think that nearly every startup with seed funding will absolutely have to sell out big time.
It really does probably start with the kindness of the initial benefactor - or as an in-house job until such time as the "profits" exceeded the founder's "wages" and then they could hire more people and have a reserve of X years of current employees' wages to handle the ups and downs.
It would likely need to start from someone that either has a primary job, or is independently wealthy, though, and then that person would need to remain at the company because AFAIK (I'm not a lawyer), you cannot pay a "board" at a non-profit.
Even stuff like exclusivity contracts can be "marketing" in a sense - it may not land that way on a finance budget (genuinely don't know how that stuff is classified), but a lot of big creators are incentivized to join platforms outside of just choosing what works best for them.
Disclaimer: I work for a Patreon competitor (Supercast) and I'm not saying any or all of the above is definitely something Patreon does, it's just things that are typical in the industry.
They are competing against some of the best paying firms in the world. “Making it easier for podcasters to make money” is not high on the list of causes I’d give up my high paying gigs for.
I may be "a fool", but when I wasn't working for a startup or a FANG, my stock earnings were pretty piddling, really, and my salary was (a bit less) than FANG's pre-stock earnings.
In virtually all existing use cases, I'm strongly against cryptocurrencies, buuuut token issuing and the related Ponzi scheme has been the single biggest new thing they brought to the table which _does_, actually solves this (minus the questionable legality, the Ponzi, and all the current issues the crypto market is having). IE: issue utility-coin for founders & first employees, issue utility coin for public, increase overall utility (and thereby price) of coin, cash out of coin via public. Some coins even carry voting power, which makes the end-consumer part of the governing process. Connecting value-holding with utility _is fairly novel_, haven't seen it being tried in normal corporations before, and might be interesting, if it works.
The other config would be dividend-paying corps; however they went out of fashion on account of growth-above-all-strat playing corps being able to fundraise based on the _biggest hopes and aspirations_ any of the potential investors can dream up; which means, a competitive market have selected for growth-above-all against dividend-paying.
| What do the balance sheets and end of year accounts look like for a non-profit? What happens if they generate surplus profit? Can the CEO and director simply pocket it too avoid "making profit"?
Nonprofits _totally can make a profit_; buuut they can't distribute it to directors, and can't issue dividends. They could have increased salaries; however the governing check&balance on the organization, is that they must have 3 independent people on the board, who checks decisions against the organization's "non-profit mission"; and having high salaries for execs can't really fit that.
I imagine the non-trivial piece is adoption and normalization of the business model. Or do you mean that there is something technically difficult in what they are doing (which would surprise me greatly)?
Now that Patreon has normalized people donating on a subscription basis to content creators, that actually helps competitors.
But certainly one that should be run as lean and mean. After all there is only so much money to be extracted. And the whole system is not that complicated. Entirely reasonable business to run, but not something that will be an unicorn...
Wikipedia seems like the biggest tech counterexample.
Source: https://www.dailydot.com/debug/wikipedia-endownemnt-fundrais... // associated HN discussion: https://news.ycombinator.com/item?id=27339887
For a more practical example; I moderate a wiki project with semi-decent traffic and a decent enough reputation and our operating costs are about ~3k each year (it's like a little more than that) if the donation rounds are any indication. The donations usually occur in November and we get funded within 3 weeks. In terms of expenses, it's really hosting and a small amount of money set aside for a WMF developer (independently) to maintain our custom extensions for the site.
You don't need tons of money if you just want to stay solvent.
Many non-profits don't need to fundraise publicly because they've lined up some sort of stable funding, but then you're naturally much less aware of their funding situation.
Such a model can work for stuff that people's lives dont depend on. But its not viable for people trusting their livelihood on them.
I don't know where you got this weird idea of begging for money every two years. Nonprofits can and do make money from products and services, and they can pay it out in salaries or executive compensation. What they can't do is pay shareholders or VCs. Their revenue can be reinvested in themselves or in some fund, just not paid as "profit" to investors.
And the others have permanent funding from other sources, so they are not really non-profit examples.
If you have to beg every 2 years for money or you go bankrupt, that is some terrible planning. Nonprofits are usually run like businesses, and have revenues and expenses like any other business, usually continuous. It's not like they fundraise every 2 years and then spend money the rest of the time. That's just not how it works. They have boards, managers, staff, salaries, etc. that they plan for day to day, like any other business, and their revenue streams are not limited to donations (there can be grants, sales, memberships, endowments, whatever).
Wikipedia, as an example, uses that bullshit to stir up a sense of urgency as a call to action, but they are actually doing just fine:
https://www.makeuseof.com/tag/wikipedia-millions-bank-beg/
https://www.washingtonpost.com/news/the-intersect/wp/2015/12...
You can also look at their Form 990s for a clearer picture of their financials; https://projects.propublica.org/nonprofits/organizations/200... (it's not FAANG money, but it's enough for what they're doing, and they get richer every year).
The reason typical nonprofits don't get rich is that you can't really "invest" in them via capital injections, because investors can't get a return. That's what makes them nonprofit. But nothing stops them from doing routine business activities to make money on a predictable, cyclical cycle -- not a lump sum every year. That Wikipedia thing is just a marketing gimmick, the same way many charities will use holidays or Giving Tuesday to beg for money, but that's not their entire income stream (or even most of it).
The choice to become a nonprofit is exactly that, a choice. They don't "have it bad", they just choose their mission (well, ideally) over profit. Of course scams exist and some are just tax havens or whatever.
There are also non 501c3 nonprofits, like churches or issue-based advocacy groups, that have their own funding structures (tithes, corporate affiliations, whatever). And nonprofits can and do have their own investments, their own funds, etc. that earn a return, but it just goes back to the org and/or its staff (as salaries and perks, etc.), not shareholder return.
Overall, though, for the most part the primary differentiation is just that they don't have shareholders/owners to give money to. Aside from that they are mostly just businesses with a bunch of small technicalities of IRS rules to adhere to.
Its not. Its the legal setting that prevents non-profits from gathering more operating money than a given period.
> their revenue streams are not limited to donations
They aren't. Except it works the same way - they cant gather any kind of financial capital that could make them a viable competitor for for-profit companies. Otherwise non profits would have steamrolled the for-profit companies a long time ago.
> Wikipedia
Picking one example as poster boy does not make the argument. Wikipedia is the product of a well-connected, famous persona, who already had enough connections to start and float any kind of organization at the point he started Wikipedia. Its also the first and practically the only one of its kind with no practical competititon. Even so, it is dependent on fleeting funding from donations and similar revenue sources.
The sole example of Mozilla can prove how fickle this format is. Its also one-of-a-kind, it was also practically the first in its field, and for a time it had no competition for its major product. Now its on shaky grounds, with funding for Thunderbird being cut, Firefox not doing so fine and so on.
Thats the problem with non-profits in the current legal landscape. The landscape is crafted to prevent any threat to for-profit companies and it gimps many things ranging from non-profits to cooperatives for that end.
And that makes it totally risky for anyone to build their livelihood on such a non-profit. You can still be able to pay your rent next month as a creator if Wikipedia goes down. But if your Patreon goes down, you are in knee-deep sh*t. Because its infrastructure by now.
...
Im totally for non-profits. Actually, an entire economy that comprises of cooperatives, functioning on the infrastructure provided by the democratically elected government is the ideal economy for me. However the proposition of turning such services like Patreon to non-profit is not viable with the current legal landscape unless its changed drastically.
There are thousands if not millions of perfectly functional nonprofits. Mozilla is more the exception, a monster that gets fed by Google and produces little of value. The EFF, ACLU, Sierra Club, whatever are also nonprofits. There are lots and lots and lots.
Nonprofits are a different beast from cooperatives. They don't have a profit motive. If you want a democratic nonprofit there are such things as worker self directed nonprofits. Or if you want shared investments without shared governance there are ESOPs and such.
Yes, I would love to see a more democratic economy too. But that's not held back by the limits of organizational structures.
Please, if you're going to make these extraordinary and (I believe) incorrect claims about nonprofits, provide some sources?
That was what I was told. Even if its 'many years', its still a limit. Even if there wasn't any limit and what I was told was wrong, it still doesn't change the reality:
https://www.councilofnonprofits.org/tools-resources/operatin...
All the examples that people give on behalf of nonprofits are organizations that provide a free, non-critical service which would not harm anyone's livelihood if they went away. From free browsers to Wikipedia. There isnt one single service provider among them.
...
Ask yourself: Would you build your startup by relying on a nonprofit-provided infra like payments, hosting and so on. That is, if you can even find such a non-profit service in the first place.
Recall that the majority of people that use Patreon tend to be YouTubers that want real income but are regularly banned from getting it; artists; non-profit / open-source software development teams; single-person dream-projects; influencers / online personalities; etc.
So, "would I build my [community-facilitated, primary source of income] by relying on a [company that is not beholden to the whims of the stock market, so it can choose what's in everyone's best interest and not just the highest stock value]?"
Absolutely.
What are you doubting, exactly? That nonprofits can't charge for services?
Here's a few examples... some big museums (the NY Moma, the Field Museum in Chicago), the Green Bay packers (and the NFL as a whole until 2015), the Mayo Clinic, the Kaiser Permanente parent organization (which owns the for profit hospitals), etc. Donations are a part of many of their income streams, but not all.
In tech, some other examples include VLC/VideoLan, the Linux foundation, TechSoup, Change.org, the Internet Archive...
None of those have FAANG level money, but that was never their goal. They still manage to survive and produce very valuable output.
Of course I would want a nonprofit payment processor, along with a nonprofit Lyft/Uber equivalent, etc. If I had the money or talent, I'd build those myself.
Don't get me wrong, the nonprofit space has a lot less money overall because it can't attract investment. That doesn't mean they can only survive two years at a time. It entirely depends on their business model (yes, even nonprofits have one). If they don't diversify and depend solely on donations and don't get any, yes they will die. Businesses without income will die too.
I'm not really sure what we're arguing about anymore. That nonprofits are poorer? It's true, they are. That they can't make or bank money? That part is not true. It's harder for them to attract money because you can't invest in them, but once they get it, they can manage it like any business, even buying up for profit businesses as investments.
The average windows enterprise customer who uses MSFT for 60% of the security load and 40% to the MSSP and IT team can get by with this move.
But, knowing people who work at MSSP, a lot of security theater goes on because it’s just on the wrong side of industry trends. Watching is one thing, fixing it is the other. They prob has some cost cutting and a CISO or equivalent who didn’t know how to make the case (or no CISO).
But at SME/enterprise scales, a lot of the success is driven by in-housing tier 2 + security engineering. For tech/product companies, true at even smaller scales. Security is part of a common core product value proposition for the customer: trust baked in. Same but different for say fraud & safety and how that relates to account churn vs growth. Seeing that capability evaporate is scary.
As an analogy, most companies who handed over control of their data plane to palantir engineers & proprietary palantir infra ended up regretting it and wanting to change the relationship. Areas like parts of the US Gov who haven't are now stuck. Doing the same with the trust & risk aspects of an enterprise is scary, especially as each one is now increasingly part tech company.
That being said, I don't doubt a good arrangement can be had with someone like Red Canary to provide services of tier 2 and 3. I've only spoken with them and not other leading MDRs so I'm not sure as to the quality of others. I can easily see a company like Patreon making the decision to use an MDR and it being the right call.
A non-exhaustive look through the LinkedIn profiles of the former security team doesn't show any previous startup-to-IPO security experience. If Patreon is preparing the company for eventual IPO, getting everything security-related aligned with the due diligence expected for an IPO is critical.
I would guess Patreon is simply engaging with an external firm that has past experience preparing security at a company for IPO. The laid off security team might have been otherwise good engineers, but not necessarily the right team needed to prep a company for IPO.
I dunno, either all they did is actually covered by developers and ops, or they're getting rid of a lot institutional knowledge.
This is exactly why I use it - and the only reason why I use it. It's just a convenient portal for me to manage donations. I don't care about exclusive content and I'd rather the people I support just continue to put out content on their primary platforms. The less I have to engage with Patreon's content feed, the better.
A platform like Patreon mainly provides the really hard service of collecting and distributing money while conforming with tax and other laws you run into while doing that. The rest, like the content feed, is trivial.
Centralisation that "literally can’t be duplicated" is no cause for celebration; it's a broken black-hole monopoly and the more it grows, the more you have to use that platform if you want a "fair" shake against other creators; the harder it is for competition.
You're technically right in what you said but I don't think it hurts to give people a little encouragement if they want to strike out and show people there's another way than just paying Patreon 5-12% or OnlyFans their 20%. We need creative solutions for getting away from relying on Amazon, Patreon, etc, etc to find the things we want.
Anyone who tries to do those ends up spending a lot of their time in maintaining and sorting these out than doing their actual activity like art, music and so on. That's why services like Patreon are a godsend to, well, practically anyone. Anyone, because there are a lot of software developers (freeware, shareware, modders) who use Patreon to fund their software development.
Handling payments, subscriptions, fraud, local tax kinks are mostly solved problems with the right intermediaries. Yeah, it's more than letting somebody else handle it, but —depending on your content— it's potentially 15% of your revenue.
Yeah. And Patreon is that intermediary. That intermediary service is the difficult thing to do.
> depending on your content— it's potentially 15% of your revenue.
That's peanuts compared to the effort that goes to doing those stuff by yourself.
Wait, they can't even spell the name of their own company correctly?
It's kind of amazing that that's even a thing, let alone that it's higher quality than a typical read-and-type copy.
[0] <https://news.ycombinator.com/item?id=6156238#:~:text=Languag...>
This is the key part here - whoever they've outsourced this to costs less than those 5 employees and presumably still lets them tick whatever regulatory boxes they need.
Either that or their 5 person security team made a boo-boo recently and this is the fallout.
The key part of the Discord post is this;
> As part of a strategic shift of a portion of our security program, we have parted ways with five employees. We also partner with a number of external organizations...
Xe has commented on this and is suggesting to simply enable 2FA to deal with security concerns [1], as apparently a more measured response to security issues at Patreon. This seems to be more self-serving advice rather than good advice [2], the better suggestion would be to simply move to one of the many alternatives.
The assumption is that the login system is somehow insecure, instead of the bank details themselves being stored insecurely. (Hypothetically) If you suffered a large leak of payment details, when payment processing is your _main_ income revenue, and you find out your security team of X people at probably $100k+ a year for N years have done nothing to secure payment details, you too may fire them on the spot.
Until Patreon explain why they fired their _entire_ security team, the best advice is to remove your card details immediately. You can always add it back at a later date, or use an alternative payment processor.
[1] https://xeiaso.net/blog/patreon-happening
[2] https://www.patreon.com/cadey
(Edit: As pointed out, Xe didn't mention credit card details.)
I realize that my announcement looks self-serving and at some level it kind of is (support on Patreon is part of how I keep my US account solvent because international wire transfers have yearly limits to their frequency, apparently).
However, the facts of the situation are that we do not know what is going on and other creators I know are not in the same kind of financial situation that I am in. At least one of my good friends will probably miss a rent payment if there's a mass dropoff in support (and they just escaped an abusive living situation). My goal is to emphasize a "woah, slow the fuck down and let's wait to see what's going on before making rash decisions" rather than a "oh no my free money fountain is at risk let's stop them from going away". That post exists because I've gotten over 60 inquiries about this today from as many people. I was tired of explaining it over and over so I put it on the blog to avoid constant duplication.
Telling people to enable two-factor auth is the moral equivalent of a free space in bingo. It's something people should already be doing and it's good to repeat that message in case people haven't heard it.
We don't know what's going on. Until we do, it's best to keep an eye on the situation and be ready to react if needed.
Are other alternatives not viable? For example SubscribeStar [1]? It's probably better to have multiple sources of income, rather than a single payment processor?
> However, the facts of the situation are that we do not know what is going on and other creators I know are not in the same kind of financial situation that I am in. At least one of my good friends will probably miss a rent payment if there's a mass dropoff in support (and they just escaped an abusive living situation).
I'm sympathetic to your friend, but I also weigh up the possibility of however many supporters having their credentials exposed. Not for being taken for a few dollars, but potentially having their entire account wiped out.
(On a side note I would also suggest that given the incoming economic downturn we are about to experience, this disposable income from strangers may not be reliable in the near future.)
> My goal is to emphasize a "woah, slow the fuck down and let's wait to see what's going on before making rash decisions" rather than a "oh no my free money fountain is at risk let's stop them from going away". That post exists because I've gotten over 60 inquiries about this today from as many people. I was tired of explaining it over and over so I put it on the blog to avoid constant duplication.
I still think the best action for your supporters would be to temporarily remove their card details until Patreon explain themselves. These can be added back in a few days once Patreon come up with some PR. In the meantime, I would definitely suggest to look for alternative platforms.
Firing your entire security team without warning is not standard practice, it really does indicate something bad. If the security team were caught be surprise, it means there has been no hand-over process, meaning that potentially Patreon currently has no way to handle or detect new security issues. Clearly Patreon themselves were taken off-guard by this, hence the radio silence amid growing speculation.
> Telling people to enable two-factor auth is the moral equivalent of a free space in bingo. It's something people should already be doing and it's good to repeat that message in case people haven't heard it.
As I wrote in another child comment, it somewhat sounds like enabling 2FA is an alternative to starving creators. If there was a breach in the system itself behind the login system, 2FA won't make an ounce of difference.
Anyway, fingers crossed this is all blown out of proportion. My gut is still telling me there is something amiss here though.
Patreon is where the users are. I've evaluated other platforms, but at that point I'm probably going to come up with my own thing on top of Stripe if Patreon falls over. I'm kinda low on ideas on how to make such a migration palatable otherwise.
Also for total transparency's sake, posting that thing about Patreon has actually made me lose patron support.
> As I wrote in another child comment, it somewhat sounds like enabling 2FA is an alternative to starving creators. If there was a breach in the system itself behind the login system, 2FA won't make an ounce of difference.
As a corollary, if credit card information was actually pwned, it's not like people can do much about that either at 10:30 PM EDT on a thursday after the Queen died. The financial system is gonna be kind of tied up. Two factor auth is a more actionable step that can have measurable security benefits. If it really is a credit card pwnage, that's a big enough problem at Patreon's scale that individual actions can't really do anything. It would need a systemic response.
I'm hoping this is blown out of proportion too.
Theoretically it shouldn't be too difficult to periodically charge people and then send this money as a single sum periodically. This is done by companies all of the time. As it's one person you can cut out a lot of the complications.
> As a corollary, if credit card information was actually pwned, it's not like people can do much about that either at 10:30 PM EDT on a thursday after the Queen died. The financial system is gonna be kind of tied up. Two factor auth is a more actionable step that can have measurable security benefits. If it really is a credit card pwnage, that's a big enough problem at Patreon's scale that individual actions can't really do anything. It would need a systemic response.
Patreon just fired their entire security team, they may not be well positioned for such a scenario. 2FA is of course good to have but I somehow doubt accounts have been compromised in this way (if they have been compromised at all).
I suspect that although the Queen has died, financial systems continue to operate near normal (minus the stock market).
What about LiberaPay?
> Patreon is where the users are.
Moving my stuff would make for an incomplete migration. I don't want to deal with that.
There isn't one single sufficiently reliable and reputable subscription payments provider outside of Patreon. I researched various providers for a certain project. From Chargebee to Recurly to Paddle, all the providers either have obnoxious revenue or other requirements for using their platform, questionable reputation and feedback from users, or they are too small. On top of that integrating them to your website or software requires jumping through hoops for most cases.
Patreon looks great compared to all those. Its larger than the majority of them, it has zero requirements to get on board, and it can process payments from even payment providers from the developing world. It feels a bit odd to use it for selling software, but many software developers seem to be doing it already.
Most important is that such services like Patreon take the sales tax processing off of your hands. From California sales tax to VAT. That's a major, major pain in the ass for any business.
The Xe link you cited doesn't speculate about credit card details being hacked at all. In fact, the entire gist of the blog is that we shouldn't be speculating about hacks or other theories.
The 2FA recommendation isn't a response to a potential hack, it was just general good security advice.
True, I accidentally bridged that gap when reading it. I have updated my comment to reflect security more generally.
> The 2FA recommendation isn't a response to a potential hack, it was just general good security advice.
Given the context in which the article is written (the security team being expelled), this is the way I have interpreted it. It's not a standalone PSA on best security practices. It's written as if enabling 2FA is an alternative to homeless creators.
The message under `UPDATE(M09 08 2022 20:40)` is intended to be spread to and understood by less technically apt people. I have intentionally focused it on harm reduction, lessening fearmongering and overall calming down the situation so that we can wait for the truth to be revealed. As I mentioned earlier, suggesting people use 2FA is something concrete people can do right now to help secure access to sensitive accounts.
I'm worried about the kind of situation that would cause this to happen, but I don't know what is going on so I have to assume that there is active disinformation and interpret things in the best possible faith. The last thing I want to do as a technical communicator is lie to people.
I agree there will be misinformation in this scenario, but the facts we do have do not bode well. As far as I know, it is not a good sign to fire your entire security team. There isn't a circumstance I can imagine where this reflects well on Patreon internally.
Then I decided to support someone at Patreon and to do that I had to enter my credit card details. A few days later there were fraudulent purchases on Alibaba charged to my card. I immediately called the bank, had my credit card frozen, reversed the transactions, and requested a new card.
Although I can't be sure, I have a strong suspicion that it was Patreon that leaked my credit card info, as it was the only unusual payment I made, the usual being electric/internet bills, food delivery, etc.
Have had no problems with my credit card since.
From <bingo@patreon.com> To <my@email>
According to gmail both SPF and DKIM pass, so I know my payment went to Patreon.
Mitm attacks possibly, but I highly doubt it.
But since it's the security team, I'm gonna go with Patreon is in dire financial straits and this is the tip of the iceberg.
Like, we HAD TO change our passwords every 30 days.... but our default modus operandi was not having password auth in first place + hardware tokens for authentication, which made whole password thing totally irrelevant, but no, security, change your password you don't use every 30 days.
That's good. Scrum manager as a crontab:
Every week, scheduled per developer 1/2 hour: "[90%: How's your progress? Any blockers? I don't have any updates on any blockers you may or may not have previously mentioned. Your dog is good? You have a dog right? I thought so. See you., 10%: The [CEO;CTO;COO] is pissed. The velocity is not on target. I'm not saying this to anyone in particular but I am saying this to all of you. There's going to be a retrospective and I'm not sure what we are going to say. Look I got another meeting. See you.]"
Every Tuesday & Thursday: "[90%: Look I don't have much, you guys know what your working on I'm not gonna waste your time; 10%: The [CEO;CTO;COO] is pissed. The velocity is not on target. I'm not saying this to anyone in particular but I am saying this to all of you. There's going to be a retrospective and I'm not sure what we are going to say. Look I got another meeting, feel free to stay on if ya'll have anything to talk about.]
Every other week, meeting scheduled with all developers: "The [CEO;CTO;COO] is pissed. The velocity is not on target. I'm not saying this to anyone in particular but I am saying this to all of you. There's going to be a retrospective and I'm not sure what we are going to say. Look I got another meeting, feel free to stay on if ya'll have anything to talk about."
And on top of that, if you use a cloud provider, the data you get back is next to useless because it still thinks hostnames/IPs are a remotely useful way to identify stuff, which they aren't. It looks like they might finally be providing some remotely useful identifying metadata, like instance tags, but only after years of begging for it.
DevSec is next, right?
Not really. If you wanted to find a way to exfiltrate it would be in your favor to have the tool client side that you were trying to bypass.
Alternatively maybe the devops or other team took over their functions. They're not legally required in that industry, while in some industries / publicly traded companies they can be.
I think that's a wrong attitude to have. The entire point of security systems is to be bureaucratic, i.e. to follow proper procedures.
A good chunk of security incidents in the software industry could be avoided by people following procedures rather than winging it, the annoyance is often not justified or an indication that people are playing too fast and lose.
I've borne witness to infosec people so hands off I'm convinced they secretly have two jobs as remote workers.
Good DevOps and dev folks obviate infosec, leaving the latter roles to industries that mandate it.
Being bureaucratic and following "proper" procedures ranges from being a way to achieve that goal to being actively counterproductive. It's not exactly rare for a "proper" procedure to be optimized for CYA, compliance or security theater at the expense of actual security!
I view them as a media hosting company of sorts, like YouTube or Vimeo. The fact that they collect money makes them nothing special, since everyone technically collects money somehow.
I have a Patreon page with hundreds of supporters. If I want to post a video for my supporters, it has to be hosted elsewhere. Patreon will not host and serve it.
a couple additional cases that can cause a security team to be laid off:
1. Outsourcing or Offshoring
2. evidence of significant corruption or incompetence within the team requiring a full reset
It doesn't happen often, but when I get requests to investigate a C-level for misconduct, the first thing I do is update my resume.
"We've never had a security breach, what am I paying you for?"
"We're having a security breach, what am I paying you for?"
A couple years ago, they tried to change their whole payment system so each subscription was billed individually, racking up more fees for patrons. They kept defending it and finally backed down at the last minute after a ton of their biggest creators yelled at them.
no, I don't understand either.
Meanwhile in January they said they had taken in a total of $3.5 billion in creator lifetime funding. At lets say 8% average cut that means they've made a total of $280M in their whole history, although earlier accounts got grandfathered at the original 5% rate I think. We'll go with 8% to be generous. Even at 12% that means total lifetime income for patreon is $420M, around the same as their VC funding.
Then they have to pay all the processing fees, their own development and operational staff (~300 employees in 4 offices), etc. Then they have to actually make money as profit to satisfy investors.
VC funding killed patreon.
A company is worth it’s future cash flows. You are making an incorrect argument about the sunken costs (sunken costs do admittedly have relevance to individual funds, but not to your argument).
You don’t get 10 rounds of investment without a convincing argument (or believable story) for future profitability at each round (even if some rounds turn out to be duds later).
Patreon has had multiple rounds of layoffs since 2020. Their finances aren't public but no matter what they're selling investors, what they're doing shows they're bleeding money all over, and VCs make many mistakes betting on the wrong horses.
A poor analogy. VC is more like putting 20 horses in a race, hoping that 1 horse will win billions of dollars. The other 19 horses usually make approximately $0 in comparison to the winner.
VC is not zero-sum. Most sports are zero-sum and most sports betting is worse than zero-sum due to house take and taxes. Using sports for analogies with business often gives invalid intuitions because of this.
I think you are also making a fat tail error, the opposite of survivorship/selection bias[1]: you see heaps of failures so you are not properly offsetting correctly for the small percentage of big wins which is the theoretical modus operandi of VC[2]. A majority of VC funds fail to return enough for their risk, but that alone doesn’t tell you whether investing in VC funds gives a poor return: perhaps one VC fund returns 100x, perhaps an investor is willing to pay for investment diversity, perhaps other reasons to invest in what superficially appears to be a poor performing sector.
[1] https://en.m.wikipedia.org/wiki/Survivorship_bias
[2] https://techcrunch.com/2017/06/01/the-meeting-that-showed-me...
Profitability might not matter to the VC investors if they can IPO like Uber.
Patreon has been amazing for small and large content creators, providing a big win for society IMHO, and Patreon seems likely to continue in some form even if they don’t meet the VC growth targets. In a perfect world, things would be different, but there there is little surprising here (which is part of the crux of your point?).
Profitability is almost meaningless for a growing company (see Amazon). Perhaps Patreon turns out to be a loser for the investors, or perhaps a loser for society. But we have to wait and see, and meanwhile take the wins we get.
They have sacked some employees, but employees take that risk on when they join a startup, and those risks are obvious to anyone that is paying any attention.
Sure, VCs make mistakes, often big hurtful ones. That’s part of the game. You are calling the game over, and it isn’t half-time (to stretch some sporting analogies badly), and most importantly: you are a spectator and you are not the ref. As far as I can tell, you don’t understand the rules of the game.
Having members on YouTube doesn't impact that at all. The members will pay and discover, same as if they were on Patreon. Non-members might have a harder time discovering, but it's not like it's easy to discover content through Patreon either.
Only if the Visa/Mastercard duopoly approves. It's not a real market.
Cardless Subscriptions are also here (disclaimer, that's my role at a large open banking company) although some more legislation needs to be done before they can be used for ecommerce
For the Creator, even if you exclude the things that interact with Patreon to do the benefit-management bits, you have to also do any KYC/Identity verification bits, re-setup account transfers.
Then you need to reach out to all your supporters and convince them to move over. They're going to lose some percentage of those people -- how much is going to depend on the creator and the fanbase. Some people are just going to ignore it, every single one of them though are going to re-evaluate that math in their head as to whether they really want to continue supporting that person.
Patreon also has major brand-awareness. Someone telling you to move over to SubscribeStar, Liberapay, Buy Me a Coffee or whatever might get a "This seems like a scam" type reaction.
E: see this video https://youtu.be/bGvfYv5nzs0?t=138
"Good UI" doesn't make something a platform.
Doing payments in a fee-minimizing way definitely doesn't make something a platform.
This isn't like a newspaper, finding neither ads nor paywalls will make ends meet. They're not phone app developers, finding users will barely pay $1. It isn't an ad-supported business that can only make a fraction of a penny per user. Their moderation needs are at a pretty small scale, with most user accounts tied to known bank accounts. They don't have to negotiate with rights holders for copyrighted content and get taken to the cleaners. They don't have to store and distribute gigabytes of video per second. They don't face big customer acquisition costs.
Youtube and Twitch are doing a lot more work for their cuts.
Only if the creatives are making their money on their platform.
Since Patreon's entire business model is based on taking a cut of the revenue artists make on their platform, they don't make money if the artists make money elsewhere. Many creatives use Patreon only for fan outreach, and sell merchandise outside of Patreon.
If Patreon charged a fixed rate for their software, they could make more money, but they would lose the business of small creatives. That's the dilemma.
You could be correct, but I posit it'll take 2-3 years minimum before it's used widely.
They have zero reason to lose significant money over the video feature, especially since it's still in closed beta.
You want people focused on security, but separate security department all too often just becomes annoyance that doesn't understand business, or architecture, just drops decrees.
This is why the decrees are made, I guess it strongly depends on the tone of the message and the messenger. A little bit of social engineering of engineers goes a long way.
So they don't seem to have any problem with income. VCs have little reason to pressure them for anything either - this is a startup that is making 125-130 million revenue every year already, and its subscription revenue - the holy grail. Any pressure that comes in from that side would likely be in the direction of pushing for growth. Which may be one of the reasons for the layoff. Excepting some major incident related to the security, of course.
If they are in dire financial straits, they can definitely afford 1-2 security engineers to keep the light on.
I believe it was this was the biggest, most complete and the first time: https://www.zdnet.com/article/patreon-hacked-anonymous-patro...
I didn't dig enough to find the second time but it occurred a few months later if I recall and wasn't as extensive.
Company has its own security team: professionals tell their employers "do A, B, and C, or else you'll be on the hook legally for the consequences"
Company hires outside firm X: "pay us $LARGE_SUM annually and we'll take on responsibility and legal liability for failures, which we can do because we have a lot of customers and we pool the risk".
When $LARGE_SUM < the cost of maintaining your own security team + the potential liability payouts, the choice is easy.
Also, that is one cool badge as their pinned post, I didn't see that one this year. I always miss out on the cool badges at DEFCON.
Such a narrow comment, as if they didn’t outsource the work.
Not every company on the planet needs a team of specialists in one sector.
The article by the co-founder is not shown on any Patreon blog front pages, but it can be searched on their blog. I found it on Patreon official Discord announcements channel.
Along with a statement from their "Head of Engineering."
I published the statement quote, and the article link in my Patreon-related article: https://io.bikegremlin.com/29277/patreon-review/#7
Everyone laid off should be encouraged not judged for sharing the news of their availability with their network, just as much as one might boast when they achieve a milestone or get a new security certification.
And unfortunately ‘I’m searching for a new role’ without context leaves the reasons why open to wild, impersonal internet haters imagination to fill in the blank.
If your business had a broken lock on the door would it be ethical for a disgruntled former employee to tweet that fact out to the world after you laid him off? They may not be breaking in themselves, but they are substantially increasing the chance that someone else will.
These people aren't announcing a vulnerability (like in your broken lock analogy), they're announcing that they all got fired and that they all need references. It's also a strong signal to everyone that the company may not be a good place to apply for, for a security position.
Instead of trying to figure out who the bad actors were and who could still be trusted, the directors simply fired everybody in the team.
I was brought in as an outsider on the Friday evening to reverse engineer their passwords and basic network info so that they could continue to operate on Monday morning.
Those same directors are part of the same organization and are responsible for the actions of everyone under them. That responsibility is supposedly WHY they get to enjoy those big paychecks.
But somehow "everyone involved has to go" still doesn't include them...
My point was about someone higher up applying standards to everyone under them but not even considering having those same standards applied to themselves.
Should the head of IT be in the kill group? Why? It might have just been a couple of their team and they had no idea.
The board obviously says "your team your responsibility and plus we don't know you're not in on it"
Ok but then the owner or the stockholders should be able to apply EXACTLY that same reasoning to that same director or the entire board. If I'm a stock holder, I don't know they're not in on it and "your team your responsibility", applies all the way to the owner/ceo.
It's almost like there should be some other less stupid way to resolve problems...
Everyone else has all the cool stories meanwhile here I am aspiring to live Homer Simpson's lifestyle. Oh well.
This was an legal firm with something like 100-200 staff. The vague memory I have of the incident that triggered the whole thing was that a drug gang bribed someone in IT to get inside information about a rival gang's case.
We got dragged in on Friday mere hours after the entire IT staff (not just SecOps) was walked out the door, with several people apparently escorted by police.
The non-IT staff had no passwords other than to their own normal staff accounts, and there was no accessible hardcopy of any documentation.
I found a plain-text password in SYSVOL with some "server admin" rights, used that to find a plain-text domain admin password in a scheduled task, which we then used to dump out the Active Directory hashes. I have a vivid memory of a nice fancy lunch in a central business district on a quiet Saturday with my colleague while my laptop GPU was making turbine noises back at the customer site running hashcat.
By the time lunch had finished we had about 80% of all passwords, including more than half of all admin and service accounts. The only credential we couldn't obtain by Monday morning was the Cisco firewall appliance root password. We found some documentation and reverse-engineered the rest.
It was an eye-opening experience to see just how fast the security of a network can fall against an "attacker" with LAN access and just some basic tools you can download from GitHub, nothing fancy.
There was another similar incident where an sole IT admin had a rapidly deteriorating mental illness. Similarly he was walked on a Friday to prevent him "salting the Earth behind him", which apparently he had threatened. That network was reverse engineered in a similar timeframe with little effort...
I do general IT consulting, and all that was required was some "general security knowledge".
There's like a couple of tricks that will crack open any large-ish network in seconds given nothing much more than a command-line.
If you want to get fancy, Deathstar[1] will break open the vast majority of Active Directory networks in minutes. In some talk about it I saw on YouTube, one of the authors quipped that it'll grant you "Domain Admin in under 5 minutes, or your money back."
We didn't have to run it. I just used one of the initial attack vectors (scanning SYSVOL for passwords) and went from there.
This "one trick"[2] will get you an admin account password on something like 50% of all Active Directory networks:
dir \\domainname\SYSVOL -Recurse | Select-String 'password'
Reversing password hashes on Windows networks is now hilariously simple, because the NT hash is very old and weak. For compatibility reasons it's almost never replaced with something stronger. Any Windows password that is shorter than about 9 characters should be considered trivially reversible, to the point that it should be treated as "plain text" for security purposes.[3]
[1] https://github.com/byt3bl33d3r/DeathStar
[2] https://adsecurity.org/?p=2288
[3] https://www.tomsguide.com/us/8-character-password-dead,news-...
Elaborate?
- E-mail aliasing and password resets
- Resetting root accounts directly in on prem systems
- Cooperative employees
- When you are the CFO paying the bills, you can usually work with cloud providers to recover access to SaaS solutions (password managers aside)
- Terminals left powered on and unlocked
I used 'hashcat', which is an open-source tool available on GitHub.
A few hours of GPU time will reverse about 50-80% of typical NT-hashed passwords, and 90% or more given a few days or weeks.
Once you have passwords of admin users or service accounts, you can use those to "trawl through" servers looking for plain-text passwords in scripts, scheduled tasks, documentation, source code, etc...
Unless it is the fuck ups of all fuckups, companies that fire security teams for a security gap that got hacked will never be able to hire a security team again that’s competent. They’ll get people trying to break into the industry, but not capable vets.
Behind 99% of security incidents is a lack of funding, staffing, or “no we have to deploy to prod soon, sorry.” The very large sprawl that Series B -> pre-IPO sec teams have to address is staggering. Pre-IPO is usually the latest point that funding, staffing, support shows up.
The thing about slowly firing your security people is that they're high risk; fire them one at a time with long notice periods and you could have a situation on your hands. Fire them all at once (hopefully with generous severance packages) and you avoid that problem altogether.
Is this now enough for you to not trust platforms like Patreon? Regardless, with the light of this failure and the platforms like PayPal stealing funds from the likes of small businesses and locking it up for months; killing them.
Web 2 payments is going just great! /s
They're good, too.
They have 3 clear motivations to exaggerate the situation:
1. Disgruntled employee wanting the employer to look dumb.
2. Oversensationalize the news to attract more attention to their job search.
3. Oversell their role in the company in hopes of getting more interviews.
For what it's worth, "me and the rest of the security team" could be one person with a handful on interns doing an experimental project in the security area. Like, trying to achieve 100% formal validation of a massive codebase, on top of existing engineering practices. Such ambitious projects do get axed at the first sign of a downturn, and it would not be a reason for panic.
I would exercise caution and stick to quantifiable facts before concluding that Patreon is run by idiots.
Patreon would, in turn, have a motivation to deny the layoff, so the truth is somewhere in between. A good independent metric would be other verifiable LinkedIn users confirming that they got sacked as well.
[0] https://www.linkedin.com/posts/emetcalfe_opentowork-activity...
Ironically, the comment you replied to stayed on top of the thread for about 30 minutes, gaining upvote after upvote, and then instantly got kicked down to the bottom, while still showing the original score.
This is very different from the usual time+score-based ordering and looks 100% like a manual override. Strange response to trying to be objective.
For what it's worth, every other comment I looked at which appeared above yours was made after yours (excepting two made an hour prior with dozens of replies and presumably plenty of votes). Other comments can get upvotes too.
Sort of like... claiming there is a conspiracy about downvoting comments? Seems like a non-factual claim with no evidence (also could be called a rumor).
The simple answer would be that some people agreed with you and then later some other people disagreed. Combine that with people posting other comments after you, and other comments having significantly more engagement than yours.
If the person who made the claim is lying, which is going to be pretty obvious with a little bit of time, they've flushed their own career down the drain. Risky move with little potential gain.
Not with you on accusing posters of having those 3 motivations. Maybe they do, but I would most certainly not assume it. Need facts.
Obviously, we don't know if that's the case here, but I wish more companies figured out their security team is security theatre.
It was a member of their team doing a pentest they organized.
Agreed with a lot of the criticism but will first leaven that with devs get up to some wild shit that has to get controlled behind the scenes by the sec team. Can’t say more without doxing, but I’ve have done sec eng at a variety of pre/post IPO tech companies. Staggering irresponsibility with PII management, prod secrets, access prod on a personal laptop and indignantly arguing About it, malicious IDE/browsers extensions given prod creds by a freewheeling dev, and on and on and on. I have lost a non-zero amount of evenings due to this and top it off with an argumentative dev or PM, even if I really do a good job about crafty “this is a blame free situation we’re just trying to solve.”
But, that aside, 10-20% of security hires are engineers who dig security, 20-30% can glue tools together and know the concepts but really stretch the definition of engineer, and the remainder are ivory tower no-machines who somehow refuse to learn how to code. SREs playing a large role in security are a natural result of this. The less technical the company, the more these %s skew badly. Also, the emotional intelligence distribution is all over the place across these categories.
However, keep in mind that behind the stronger “no don’t do this anymore” dictates is not a zero chance it’s a sales RFP or large client audit. If you want to generate revenue and get paid, audits have to get passed, and many hardline rules source from this (as in, not just regulated industries do this).
In complex applications, the person who just yells about security and doesn’t understand code becomes a liability.
> Starting July 2020, Patreon no longer offers the previous “disable” feature. To close a Patreon account, users must submit their account for deletion through our Privacy center.