jadell

For all intents and purposes, this isn't competitive data for them. There aren't really competitors in the space anyway, the barrier to entry is ridiculous. In fact, by law, operators in the industry are required to share this particular data with each other and industry regulators. But they don't share it with outside parties in the aggregate form we need it in. Hence, the scraping.

Yes, we have. And no, they are not interested.

I run a large scraper farm against several large sites. They're not online shops, and we don't compete with them. But they do have hundreds of thousands of data points that we use to provide reports and analytics for our clients, who also do not compete with the sites.

I absolutely would pay for an API that provides that data. I'd be willing to pay 10x more than the cost of maintaining and running the scrapers.

But the sites being scraped have no interest in that.

Alternatively, in a tool that allows multiple channels, create a channel for the specific issue and invite the relevant people into it.

The problem in the essay isn't chat rooms, it's having ONE chat room for all discussions. Branch off into another chat/channel and title that channel for the specific issue being discussed.

It's the gentrification cycle, and it's been going on for decades (perhaps centuries.) It's just now getting to smaller mid-sized cities in the US.

- Some part of $City is a dirty, smelly, claustrophobic place, no one wants to live there so rents are low.

-> Low wage service workers, underserved/oppressed communities, and other "undesirables" move there because they have no where else to go, and form their own community.

-> Artists, actors, students, and others with no money move there because of low rents, and the area become "quirky" and "authentic" (ignoring, of course, the community culture that already lives there.)

-> Patrons of artists and actors like to live among artists and actors so also move to the area (or adjacent surroundings.)

-> Some of those patrons start up businesses to serve the low-rent community (the ubiquitous "coffee shop", bookstore, or gallery.)

-> The area becomes a center for art and creativity, and new amenities draw more people.

-> Middle income people start to move in, bringing demand for services and more amenities, but specifically the ones they are used to from elsewhere, not the unique "quirky" ones that made the area interesting to begin with. (Right around here is the part of the cycle where the city council and police start receiving more complaints than in the past, and usually complaints directed at the community which was there first.)

-> Developers start to take notice, buying land and increasing property values, building cheaply constructed luxury "slums" but charging sky high prices due to desirable location and access to amenities.

-> As property values go up, so does rent, forcing out first the low wage service workers, underserved/oppressed communities, and other "undesirables", and then the artists/actors/students/etc.

-> The area starts to lose what made it "quirky" and "authentic" to begin with, as the coffee shops are replaced with Starbucks, ethnic restaurants are replaced with chain restaurants, and little theaters and galleries can't afford rent anymore. It become sterile and Disney-fied.

And while all this is happening, newcomer residents are demanding that the city & police do something about all the growing homeless and addiction problems brought on by the people who can no longer afford to live there being forced out of their homes.

"Ardent detractors" aren't the people I'm talking about. I'm talking about bad faith actors who understand exactly what is harmful about their position and are choosing that position anyway, because their incentives are aligned with maintaining the harmful situation. These aren't people you can engage in good faith with precisely because they will use your assumption of good faith against you.

This doesn't protect someone (or a whole society) from bad actors who know how to exploit the assumption of positive intent. This might work for arguing with Aunt Tillie at Thanksgiving dinner, but it's a loophole actively exploited by people in power whose motives run counter to what's good, in general, for society as a whole. See: tobacco executives, climate change deniers, those who profit off anti-vaxx sentiment, televangelists, etc.

Picking apart someone's belief through explaining their logical fallacies presupposes that they got to their belief through a logical process.

"Reasoning will never make a Man correct an ill Opinion, which by Reasoning he never acquired." -- Jonathan Swift, 1721.

To rephrase in a more modern way: You can't use reason or logic to get to someone to stop believing something they didn't use reason or logic to start believing.

I've had options at 3 companies that all went through some event.

1 - Company acquired after I left. I paid 3k for my shares when I left, made 3k net. Not bad.

2 - Company was self-funded when I left, paid about 2k for shares. When they took external investment years later, I needed to sell my shares, net 6k.

3 - Company I was still working for was acquired. I'm acquihired at a better salary & benefits, plus retention bonus after a year. Exercised my options at 4k to net 55k.

I've had a run of good luck, and, in each case, the cost to exercise my options was one I could afford to lose if things turned. If it comes up again, I'll have to evaluate where I am at that time. Each event is unique, you have to judge your specific circumstances.

The House has shown the ability to pass it, which means that if the Dems can flip the Senate (using the GA election), then the House can pass it again next session and have it go to a more favorable Senate. This was less about getting it through Mitch McConnell's Senate, and more about signaling voters: this is what we can do if it's no longer Mitch McConnell's Senate.

> The House represents the people. The Senate represents the state.

That was the original thinking, but the 17th amendment turned the Senate into just another House of Representatives, but one where each member has a longer term and a hugely outsized influence compared to the number of people they represent.

To be fair, most minority and underserved communities back in the 80s and 90s wanted the Dems to take a tough on crime stance, which is why the Dems shifted that way. For example, the crime bill that everyone loves to rag on Biden for voting for was written in cooperation with the input of Black community leaders and activists at the time. It's only in hindsight at the implementation and consequences of those policies that those communities are starting to shift away from that stance.

That's not the modern Republican platform. Both major parties want stronger central government in some instances, and weaker in others. They just differ in which instances are which.

And they're like that because most people (voters in particular) are like that. Everyone wants stronger government control and regulation over the things they personally dislike, and less control over the things they do like, and they vote for the party that matches those preferences.

Most Republican voters like and dislike the same things, while Democratic voters tend to have much larger variability (outside a core set of things.) It's why the GOP has an easier time hanging on to power than the Democrats do.

In no way did I claim the massacre was an unfortunate oversight.

In the case of the Interstate Highway System, it was legitimately the stated idea to go through Black neighborhoods because those were considered less desirable than white ones, and the land could be had for cheaper. It was absolutely malicious. In several documented cases, the routes were originally supposed to go through white neighborhoods until they complained, and the highways were rerouted through Black neighborhoods (who also complained, but were ignored.)

And the federal program became a model for state and local programs that did the same thing. Great example is NC 147, which destroyed the prosperous Hayti neighborhood of Durham, NC, and wiped out the city's own Black Wall Street. And it was absolutely done intentionally.

I don't disagree with anything you said in your original comment. I was adding additional commentary that systemic racism isn't always overt acts of violent destruction. Many times it's subtle acts of state-sponsored legal maneuvering using the excuse of "for the greater good."

"We need to build an interstate highway system, to modernize our logistical capacity and connect our commercial centers. But we need land in those cities, and all the prime real estate is developed."

"Hmmmm...we could eminent domain some of the land."

"But from whom? A lot of those commercial centers are owned by white people!"

"How about the equally affluent Black neighborhoods in those cities? Let's just bulldoze those, wiping out generational wealth that took them decades to build, and permanently dividing them off from their cities' downtowns, making it next to impossible to share in the prosperity the new infrastructure will bring with it!"

"Brilliant! Smith, you're a genius!"

It's not always as obvious as firebombing and massacres. Those are the exceptions. It's usually much more insidious and subtle than that.

Crazy how far into the plot they got before they were stopped. They tracked down her private homes, did surveillance and recon, acquired explosives and other equipment, and it all planned out. I don't know whether it would have actually worked (my guess is not), but it's amazing they got that far considering how early the FBI knew about them. https://www.detroitnews.com/story/news/local/michigan/2020/1...

This was not a concocted plot by the FBI. These guys had a plan, did reconnaissance, acquired supplies. It's crazy how close they actually got before they were arrested, considering how early on the FBI knew about them. Affidavit with more info: https://www.detroitnews.com/story/news/local/michigan/2020/1...

Not to mention any medical devices you might have on or in you.

Does it not signal something, that in all of the publication's history, all the world events it has existed through, that now is the time they decide make an endorsement? Instead of bemoaning that they made an endorsement, maybe it would be better to discuss why now? What's different about our current times that the publication has made such an out-of-character move?

We stop former felons from taking certain government jobs. There's no reason we can't do the same for current or former members of white supremacist organizations. You don't get to be in law enforcement while also being part of a white nationalist terrorist group, this shouldn't be controversial.

Using Proxy is key to a lot of bot detection avoidance.

edit: Really like that repo! I use a lot of those techniques as well.

There are services that will solve captcha for you (including Google's) in "real-time", and with convenient APIs that allow for automation.

One day I will right an extensive post (or set of them) about using Puppeteer to bypass sites' anti-bot measures. It's a fascinating (and annoying) cat-and-mouse game. But at the end of the day, almost all bot detection measures rely on using Javascript to report back metrics about the browser, but those measures are running in an environment where the bot completely controls what Javascript reports back.

One of my favorite tricks I've seen employed are detection measures that look to see if common detection bypass tricks have been implemented (like checking the toString output of commonly overridden native functions.)

https://theheadless.dev/posts/challenging-flows/#bot-detecti...

You don't need to poll the DOM yourself. Puppeteer's page.waitForSelector() does this for you. You can specify a timeout of how long to wait.

> The music isn’t your product, the music is your marketing. The shows, the merch, your influence - that’s your product.

In case anyone is wondering why all pop music sounds the same.

If they are not wearing badges and identification of which agency they are operating under the authority of, then the laws should be that they are not legally operating as federal officers. That's the point. I'm allowed (in many jurisdictions) to defend myself with deadly force if necessary against an assailant. Without properly identifying themselves as federal agents, they could be any random person who bought camo and tactical gear at any random gun shop or military surplus store.

Federal agents should be required to clearly identify themselves as such.

I'm tired of this argument that any order given by law enforcement should be assumed to be a legal order and one should just comply or face consequences. That would have such a chilling effect on the exercise of rights. Cops don't like what you're saying? Even if they have no legal reason to prevent you saying it? Just demand you disperse and now they can beat you, arrest you, ruin your life with impunity.

Oh, they happened to be wrong? Well, that's for the courts to decide later. Meanwhile, you've already been beaten, arrested, possibly lost your job, missed important events with your family, etc.

The whole idea that people should just comply with LEO commands and let the courts figure out later if the LEO were wrong completely misses the fact that the consequences for the LEO being wrong in their issued commands are trivial compared to the consequences for most people in complying with those commands.

If someone is wearing press credentials, or is being followed by a huge news camera, or is clearly speaking into said camera with a microphone, then I find it unreasonable that any officer can't assume that person is a journalist.

And if that's seriously the standard they want to hold others to, then I think it's reaonsable to assume a random person in camo without any law enforcement ID on them trying to shove me into a van is a kidnapper and I should be legally protected in using violent force against them to defend myself.

Or we could use completely non-gendered language and say "don't be an asshole."