A Google bot scrapes pricing info by adding items to carts
wsj.com
wsj.com
I always have found that kind of shady but it's probably known to increase conversions.
What I found interesting is that this an open attack vector for e-commerces. Multiple bots can hit a website and start adding items and start the checkout process. This basically creates an unprecedented cart behavior data influx that ruins any possible usage for data coming from legit customers. Maybe cleaning the data wouldn't be that hard but if someone knows what they are doing they can really make it hard (separate IPs, emails and cart behavior)
I doubt Shopify or Magento have anything to prevent this.
Yes but how would you verify this or hold them accountable?
It's usually not possible because you don't know how much the shipping + taxes are until the customer enters the billing information.
Any good UX designer can come up with a solution for this in a couple of hours or less. There's just no motivation to make it happen because this obfuscation of data is particularly optimized and useful for the sellers.
If they're doing that anyways, they should have everything they need to hazard a pretty good guess (and then they have an actual inducement for me to provide it).
You can approximate UPS/FedEX costs by fitting a trend line, they are decently modeled by a linear (base charge + K*distance) function, but when you go to buy the label you might be way off. This puts you in the lose-lose-lose of either eating the difference of wrong estimates, overcharging for shipping and losing conversions, or just making people hate you by increasing the shipping over the estimate. Making a shipping API call is noticeably slow, so most people require an interaction after the address is entered.
Tim Sweeney's hot take was that "the two hardest problems in computer science are cache invalidation and shopping carts!"
It's really not that weird when you consider the US is not a unitary state.
Sure, they show you a different content depending on your IP address and lots of shady heuristics, but when it comes to estimate a shipping cost, it is absolutely impossible: you can just be anywhere, who knows where you are. I say bullshit.
A postal code should suffice, and I'm not providing more personal information if the site is unwilling to say what I'll be charged upfront.
A relatively common example would be shops using the Shopify platform.
Something like: we will allow you to sell our product but only if you don't advertise discounts more than x%.
Just a few customers with alternative browsers being detected as "bots" will poison your reputation and income stream.
I mean yeah, I get that if you inadvertently make the checkout button hard to find, you'll lose potential sales, but I don't think you need intricate data about what your customers are doing to figure that out.
The process may eventually evolve in a cat-and-mouse game, where malicious e-commerce sites try to detect these Google crawlers and serve different price info to them, but let’s hope it doesn’t get this far.
So, yes, you can estimate shipping for a single item but you can't always present the per-item shipping charge as it depends on the context of the whole order.
Yes, a lot of smaller e-commerce platforms could do this, but finalizing order value can be a very complex workflow for bigger merchants with more varied sku mixes.
I’ve worked in multi-billion dollar Ecom companies where the programs to refine the order checkout process gets scoped as a multi year effort accounting for a couple of decades of legacy cruft... even if you separate the “product/tax/shipping” calculations from the “customer/credit/rewards” dependencies. But it’s often not worth separating them because they’re very inter-dependent. Moreso when you involve drop shipping or made-to-order things.
the important bit is to present it as a separate line item (with grand total) so that consumers can decide how much to trust the estimate.
that would be an even clearer shot across the bow of amazon, walmart, and the like, who provide comparison across their own platform merchants, but not across all merchants everywhere.
You are still left with the same scenario as if the store listed the individual shipping price on the front page
Google isn't going to know what other items you _might_ add to show you a "real" shipping cost
There’s a lot of legitimate case were showing shipping price upfront is just not doable or valuable to the customer.
BTW there are a surprising amount of shops for specialized goods that won’t even list the final price at the end. The customer places an order, and they update it with a finalized price after a human looks at the content, and from there the customer is free to pay the transaction or give up the order.
It's the T (taxes) part that may be still a tipping point these days, but it's just between vendor and your state,
In my experience, the most fluctuations were on international shipping by small vendors. Lego bricks for instance, where it makes a big difference if you request 5 small pieces that weight 20g total and can wait 3 months, or if it's 500+g in a middle sized box and you want it in 2 days.
Even with average indication on what to expect, depending on the combination you are requesting the vendor might use a different carrier, different shipping method and so on. They could make it more simple with a range of arbitrary standard fees, but then it costs a lot more to the customer, putting the vendor at a disadvantage price wise. In particular people have visceral reactions to overly high shipping prices.
Add items to card, check the total price and then decide whether to buy it(i remember trying to order some stuff form one japanese plamo store - and it didn't provide exact prices before checkout. I went through the process, but even the cheapest option for delivery was way too high - as 2x price of the whole order)
On top of all this, if all you're displaying is list price you can cache that very effectively and significantly reduce the load on your backend, probably by at least another order of magnitude. As with many things, items loaded on ecommerce sites tend to follow a Pareto distribution, for which caching is very effective. Adding a shipping address to the mix will destroy this caching ability, so not only are your requests 10x more expensive, 10x more of them now make it to the backend. There are various tricks you can do to try to have your cake and eat it too, but none of them are easy or simple. At the end of the day, while this is definitely a useful and desirable feature for customers, it has significant cost for both development time and hardware.
TL;DR this is actually a much more difficult technical problem to solve cost effectively at scale than it initially appears.
I ran an ecommerce platform company for many years and you had merchants with very complex shipping and tax schemes, or you had merchants that made it super simple with a basic rate table. The complex merchants had margin on every order at the cost of processing external API calls. The simple rate table merchants had great margin on some, lost money on others but were happy with their average shipping margin.
Shipping is often highly dependent on the location of the buyer and often involves full estimate calls from each carriers APIs (USPS, FedEx, UPS). The only major data point I would focus on is whether the shipping is free or flat rate.
Tax is even more complicated. Merchants often outsource tax calculations to a third-party service such as Avalara, which calculates unbelievably complex taxing schemes even down to the zip code, as tax laws are becoming increasingly more complex.
Because of these reasons taxes and shipping are not widely useful data points for search engines. That may change in the future, however. I could imagine it becoming another SEO topic to be accounted for, similar to meta tags on product pages.
I don't see how this is relevant because ecommerce sites will change the price in the cart (or reveal it) before shipping is even calculated.
This was also pretty common on Amazon.
Incidentally, for almost every brand in our industry, that number was 1.8X listed wholesale.
We could sell for less, but not list a price of less. Implementing "add to cart to see price" was good enough at the time to keep them happy.
I've heard varying explanations as to why, but at the end of the day it doesn't matter. Adding to the cart is the only way to scrape the price.
The main exceptions I can think of are venue and plane tickets, and hotel rooms. These might put a hold on a specific piece of inventory for a short time. They usually tell you when they do.
All hospitality prices must include cleaning and service fees.
It is only the US were hidden fees and charges may apply and the price regulation, if any, is more tilted towards corporations.
Only upfront shipping charge is tricky because it depends on so many factors.
The only difference is that the VAT is a certain % depending on your country (mostly 15-25%), making it easy for a merchant to calculate it with a single data point. Percentage is also always a fixed round number, so you can calculate it in your head when you stumble upon "+ VAT" somewhere without providing that data point to the merchant for calculation purposes.
My favorite example of how absurd it can get in the US: one side of my friend's street has 9.5% sales tax, the other side has 7.25%. same state, city, and zip.
I used to live in Turkey, where taxes on some stuff are insane but most people won't ever know because it is not as obvious. In a similar vein, income tax is also quite hidden in Turkey, at least from yhe perspective of employee
On top of the other reasons mentioned, the seller may have a contract with the manufacturer that covers a "minimum advertised price"
"The FTC says that the price displayed in a secure or encrypted shopping cart isn’t subject to MAP because it’s technically not advertising."
(https://www.thebalancesmb.com/what-is-minimum-advertised-pri...)
Assuming a simple product, you don't know where the user lives so you can't apply the correct taxes yet. In AUS this is easy because it's the same nationally. But in the US there are dozens of tax combinations that could be applied depending on the location.
Shipping obviously depends on where you live and what you're buying, and few places charge per item shipping these days so it doesn't even make sense to include shipping in a single item cost.
Then you have customer group discounts, some customers get different pricing when they are logged in, even item combinations in the cart can have different prices, you get the idea. It is usually not possible to calculate ahead of time.
Ebay sellers are particularly bad offenders: https://www.ebay.com/itm/Open-Box-Certified-Samsung-Galaxy-1...
Other examples here: https://ux.stackexchange.com/questions/83050/price-too-low-t...
Amazon example from a few years ago: https://lh5.googleusercontent.com/ztyT6xTPaTr9TtP8LwlRJBE6RV...
> When The Wall Street Journal contacted Google in June, a spokesman at the internet giant, after a few days of digging, provided an update: The mystery shopper is a bot of its own creation. The purpose: making sure the all-in price for the product, including tax and shipping, matches the listing on its Google Shopping platform or in advertisements.
it's rather annoying because it creates dozens of "abandoned" carts per day which we have to continually clear out (based on Google's known ip address ranges) so our reps can go through actual abandoned carts.
https://www.thebalancesmb.com/what-is-minimum-advertised-pri...
Because it seems like this mystery shopper is already doing that.
I think they meant, since PMs will never stop using metrics, we should write bots that skew those metrics in favor of an experience for the consumer rather than the perceived increase in engagement.
Not a great argument, but good enough to allow a developer to sleep at night.
Once heard a tale of an angry site owner calling Google (back when Google itself was novel) - Google deleted his whole website! Turned out he had "DELETE" button in each page, which generated plain GET request. So Googlebot visited the site, followed links to every page, and then of course followed every link that generated GET requests - because they are supposed to be safe.
Don't be like that site owner.
But most online stores should be happy to have Google crawling their prices and showing up under the Shopping results.
This bot generates thousands of "Abandoned Carts" on one of our sites... thousands...
We send cart reminders to Abandoned Carts after a few days, sometimes with a coupon offer to complete checkout.
This bot is responsible for thousands of bounced emails each week, which impacts our metrics with Mandrill among other things.
Maybe we shouldn't care, but it's sloppy and ruins all sorts of stats we keep track of regarding cart abandonment rates, recapture rates and more.
If u want help coding this or advice happy to help (for free)
So, effectively, no you cannot cut this bot off.
To make it worse, the bot doesn't always follow the same pattern. Sometimes slightly different names, addresses, etc.
We initially thought it was fraud attempts, but none of them actually attempt a checkout. They just enter all their info on the checkout page, get the final quote, and bail.
It would have been nice if Google told people about this instead of it just happening. Or allowed you to schedule a time slot for it to do what it's going to do.
Empty Abandoned Carts are useless anyway (for stats and other things - we track bounce rates in other ways), so that would be a large improvement from what is going on right now.
For this particular problem, none of those platforms can provide any assistance.
Violate CAN-SPAM Act and risk a $16,000 fine per instance? No legit business is going to do that.
Just ask Papa Johns how painful those fines/settlements can be[1].
[1] https://topclassactions.com/lawsuit-settlements/lawsuit-news...
Besides, the user is opting-in to receiving these emails. They don't have to provide an email address - so some are probably playing the game and seeing if they get a coupon or not.
As an aside - if the internet worked the way SV hipster brogrammers thought it should work, nobody would use it.
Yes, ads are crazy effective - you can ignore or block them, we don't care because enough people don't block them and are happy to click.
Yes, emails are crazy effective - you can ignore or opt-out or never opt-in, we don't care, you just cost us money if you're not engaged anyway so we'd rather you not be on our mailing list.
By entering your email address you are opting-in for transaction related emails, including new order confirmations, shipment notifications, and yes cart reminders. It's spelled out for you.
It's the same for almost every ecommerce site.
That's different than marketing emails, which require a separate explicit opt-in - ie. the user has to go and type their email address into another form and click "Sign up".
It doesn't get any more transparent than that.
Don't call something a dark pattern just because you can't be bothered to understand what you're consenting to when using someone else's website and start entering information like your email address or more. That's entirely on you.
The whole concept of a dark pattern is about UX choices that lead people to agree to things that they don't actually want; it isn't about whether you break your TOS or not.
I am saying, I think that if you asked people point blank "do you want websites to email you reminders if you leave something in the cart?", most people (myself included) would say, "no, I don't want to get that email"
You can put whatever you want in the TOS, but it doesn't mean users like it, and a user agreeing to something doesn't mean they like all the things they are agreeing to.
Whether it is "on me" or not, it is still a dark pattern.
Also, as a user entering my email, I am not promising you that I will always accept email to that address from you. If you try to send me email and I reject it, that is 'on you' to deal with what that rejection does to your spam scores.
You might think people don't want these emails just because you don't want these emails. That's pretty biased.
Ask any ecommerce company - these emails, both marketing and transactional, are wanted by most people shopping on the site. The statistics simply prove your argument is false.
This is a common theme among techies. You don't like something... say ads... so you assume nobody should like them and they should be done away with entirely. That's an absurdly short view of the world.
Also, I don't think any amount of statistics is going to be able to show you if people truly want emails or ads... just because they increase sales doesn't mean people want them. Ads can be both effective and unwanted.
OK fine, perhaps I interpreted your OP incorrectly. That's your prerogative, and you can do whatever you please.
> Also, I don't think any amount of statistics is going to be able to show you if people truly want emails or ads... just because they increase sales doesn't mean people want them. Ads can be both effective and unwanted.
True, but we're not just talking about increased sales.
We, and most companies that are serious about this stuff, track open rates, number of times the same person opened the same email, click rates, text link vs image link clicks, page dwell time after clicking through, session length and bounce rate, which pages they browse, which products they view, were the products related to the email that initiated the session, did they add something to their cart, how frequently this individual engages with our content, how long they were dormant, order frequency, etc.
Basically, we're interested in how "Engaged" you are with the site, brand(s), products and content. People who open every email, click on a bunch of links, hang out on the site for 20 minutes and add stuff to their cart are highly engaged, and are doing actions that indicate they like what they are seeing/receiving.
Remember, the people signing up for marketing emails are the most likely to be engaged with your brand/product/website. They've actively said, yes, please end me content from your company. If they lose interest some day, no problem, either our stats will show this and we'll unsubscribe them automatically, or they'll actively unsubscribe themselves.
Ads might be a different beast - however, you'd be surprised how many people click ads, and then buy products. It's immense. Clearly, the value provided there was getting the right product in front of them, matching what they were looking for, and offering it at a price that's attractive to that customer. In this scenario, I'd say it's wanted too - they got what they were looking for quickly and effectively. Everyone is happy in that scenario.
Everyone else can just run an ad blocker and choose not to subscribe to marketing emails. I do both myself... but I'd never assert these things were unwanted by a lot of people or ineffective.
You realize that you can't send marketing emails without double opt-in, right? "Hey, you forgot to buy these ..." is definitely a marketing email.
That's not true.
> "Hey, you forgot to buy these ..." is definitely a marketing email
Also untrue.
I consider this spammy behaviour, and mark the emails as such. I can only hope this discourages such practices in the future.
That's a win-win.
Maybe you are? Just don't project that onto everyone else.
and more variations...
I mean their normal web crawler identifies itself as such. Here, I feel like they're committing (very) minor fraud by putting in fake shopper information and actively hiding their identity. Not a big deal if it were just some Joe Schmoe somewhere, but at their scale might it border on harassment? The robot equivalent of a prank call?
How is what I'm doing materially different from what Google's doing? Is scale a factor that matters for CFAA?
1. https://www.eff.org/deeplinks/2020/04/federal-judge-rules-it...
Let's imagine I have my online stock linked to limited physical items/assets, ex tickets for a show, which will get reserved for a period of time. This will be preventing genuine clients from buying them.
RIP Aaron
If it was consumer journalist doing it to get the price for a news article (in a for-profit publication) about the product, would it be “rude”? If not, how is it for Google bot?
I didn't read the article due to the paywall, but I assume that the problem is that the problem is that these goods are reserved for that (non)-customer until the shopping cart times out? That is directly costing the merchant money, either in lost sales or having to maintain extra inventory.
So yeah, that bot really should have been programmed to end the session with an empty basket one way or another.
If a customer really wants or needs something they’ll go back and buy it. The world doesn’t need the excess consumption of people psychologically manipulated into buying stuff they weren’t going to.
Is this rude? I really don't care.
Putting something in a cart and leaving it there should be inconsequential to the seller. The only thing it might affect is their analytics. By the way, I consider analyzing my site visits and other data about me to be much more rude than abandoning a cart, especially when most sites don't even tell me they're doing it.
It's also infuriating to sort by price and get inflated fake shipping prices to "make up the total"
Since we were only running search and not payment processing, the tracking pixel/API for "Add to Cart" was a big thing for us. The whole product ran on revenue-share so we were paid per X ATCs
Interesting to see if any of the customers were affected by bots doing ATC and how it was handled if it was.
There's a popular moment of people using it atm https://heavy.com/news/2020/06/shopping-card-abandonment-tik...
eBay is a cesspool. Aliexpress is worse. Random web sites are bad. Amazon isn't perfect, but it's better.
Amazon also has customer service; they've always made me whole. Random web sites, I'm basically SOL. Aliexpress and eBay are random. Someone flips a coin, heads seller wins, tails buyer wins, regardless of who the scammer is.
I mostly buy from Amazon since my odds of not having problems are that much higher.
Bought some cables from Amazon Basic, one ended up not working, another had some cosmetic damage but works fine. They refunded both, sent out replacements, and just told me to discard them, it wasn't worth it for Amazon to pay to have it shipped back.
Of course if you abuse this too much Amazon will ban you. If you are an honest consumer though, their customer service generally provides a great experience.
I still remember a time when everyone was afraid of purchasing stuff over the internet, Amazon has so greatly reduced the friction and concern that sometimes I find myself going from "hmm, I need something" to "it will be here tomorrow" in the matter of a minute or two.
Although more competition in this space would ultimately benefit the consumer, it seems unlikely that Google is going to be the source of that competition. They've got shopping results integrated into their search engine, and it's a feature I've maybe browsed from time to time, but I often just end up searching and purchasing on amazon directly. I don't know if I would be super comfortable purchasing from Google in the same way that I am with Amazon, too many horror stories of App Developers / YouTube Creators / etc getting caught in some sort of Machine Learning Customer Support system.
Curious if others use the Google Shopping thing in the search engine and what their experiences are with it.
Amazon's customer service is a robot, which switches to someone in a callcenter in India, and then finally switches to a local person. I know because I recently had to contact them.
Not sure how this is "difficult to beat".
If I need a widget, and Vendor A charges a buck, while Vendor B charges two bucks, all else being equal, I'll buy from Vendor A. Bad customer service helps both vendors compete with each other, but prevents small companies, collectively, from competing with Amazon.
On eBay, small players do manage reputations, but only for a few weeks. If a product fails (or is discovered to be a fake) after 60 days, the seller is all good. Next sucker! There are things I'll buy there, but far more I won't.
Google itself has the problem that culturally, it relies on algorithms which know better than you do, and is not a service company. It does great tech, but holds human being outside of Google in open contempt. That's find for running a search engine, adwords, or gmail, but it crashes-and-burns for ecommerce.
Back then, the goal was exactly the opposite.
Look at The New York Times in 1921 [0]. Generally the stories are factual and to the point. The entire front page seems to be pure news. There's very little storytelling here, at most there are a few timelines of events.
Look at The New York Times today [1]. There's a bunch of factual and useful Coronavirus information but ~15% of the page is dedicated to "Opinion", the second article appears to be pure speculation, the third article is a bunch of storytime fluff around a little bit of news and the front page has a mix of actual news and opinion pieces being passed off as news.
When did this happen? Why? Did people lose interest in actual news? Is there less actual news to report?
Perhaps this is regional? Take for example the story about the San Quentin prison. NYTimes [2] has the same drawn out nonsense as this Google story while Aljazeera [3] adds a lot of background but sticks to factual reporting.
[0]: https://archive.org/details/NYTimes_jul16_31_1921
[2]: https://www.nytimes.com/2020/06/30/us/san-quentin-prison-cor...
[3]: https://www.aljazeera.com/news/2020/07/san-quentin-prison-se...
We used to have tyrant monarchs - Genghis Khan would roam the steppe and cutting people's heads off and that would be normal every day life.
There are still bad things today, but more people have more rights and a higher quality of life.
Genghis Khan was neither a tyrant nor a monarch, nor was cutting off heads a part of his everyday life. He actually got more Mongol people more rights and a higher quality of life.
I would request you to read a detailed and accurate account of his life by a historian.
For example, Genghis Khan: His Conquests, His Empire, His Legacy by Frank McLynn https://amzn.com/B00X2ZW5ZI
"Campaigns initiated in his lifetime include those against the Qara Khitai, Khwarezmia, and the Western Xia and Jin dynasties, and raids into Medieval Georgia, the Kievan Rus', and Volga Bulgaria. These campaigns were often accompanied by large-scale massacres of the civilian populations, especially in the Khwarazmian- and Western Xia–controlled lands. Because of this brutality, which left millions dead, he is considered by many to have been a brutal ruler"
This is an extremely odd thing to say about authoritarians right now, although this may be my American perspective, since people have been pointing out recently how futile it can be when dealing with police in the US. Still, I kind of think it's universal - being polite to authority is utterly useless once you are a suspect, thought to be dangerous, or have something they want.
It made a lasting impression on me when I was stopped in my car as a teenager and an officer decided that I must be a drug smuggler because I was too quiet/nervous.
Edit: I guess if you interpret "everything" broadly enough, your statement is somewhat plausible, but "everything" includes many situations where an authority asks you something that is theoretically or ought to be voluntary and you have to be aware that it's not really a choice. And it's still not always true.
This is ambiguous about cities or nations that resisted and then surrendered. There seem to be a lot of references to how surrender was followed by everyone being killed or enslaved. I kind of think that's an area of contrast between their standards and today's. You can point to "civilized" countries destroying whole cities in the 20th century, but doing it after your enemy is defeated is much more frowned on.
Also, when you say "beneficial for those who followed him" it elides what it meant when they enslaved the women and children from a conquered population. I don't think they generally allowed individuals to choose whether to be among those massacred or not. And what happened next probably wasn't consistent with modern human rights standards, which again, I think goes back to the original point that started the subthread.
He raided these places in military campaigns; he did not rule them at the time of the raids. So he was not a brutal ruler. In fact, he was a benevolent ruler, distributing loot equally and promoting on merit rather than nepotism.
As another commentator has pointed out, the brutality was an intimidation tactic. It was very selectively applied. https://news.ycombinator.com/item?id=23708182
He also did much to bring modern human rights to his subjects. You can read about it in Genghis Khan and the Making of the Modern World by Jack Weatherford, an anthropologist who lives in Mongolia part-time. https://amzn.com/B000FCK206
But the reason I'd balk here is that entropy as a phenomenon happens in absence of a driving force.
(1) In the present WSJ case, this is not mere indifference on the part of readers: storytelling is deeply a part of what makes us human and readers do actively turn away from flat descriptions of events to storied accounts—the WSJ is just satisfying the demand. Like, this part of the WSJ, the A-hed is actively edited to be a bit more fluff and less hard reporting to try and rope more of these folks in, as observed in comments above.
(2) In the case of shitty politicians, at least in the US, something even more remarkable is happening. The very construction of the US's voting system has a very non-obvious fixed point (as a transformation polity → polity), so that fixed-point theorems from mathematics seem to impose their own biasing force on the candidates. Or to use more economic language, the rules of elections in the USA actively incentivize its politicians to be shitty. Well, the parties are incentivized to be spineless and the politicians are incentivized to belong to one and thus inherit the shittiness—but it's the same thing. [note]
Entropy still has a part to play when you introduce a bias voltage like this, but it is usually the exact opposite. Entropy explains why air molecules are not falling to the floor in spite of the fact that there is a clear bias force—gravity—that makes everything else fall to the floor quite swiftly and the air molecules have certainly had enough time to start hitting the ground. Similarly, entropy may be more helpful in explaining why some news is still buried inside the Wall Street Journal articles or why the US political parties are able to work together on various operations to bomb remote lands.
[note] I can discourse at length about the downfall of the Whigs but the short and fast of it is that they had a spine or agenda of core issues, chiefly economic. They brokered compromises under one Whig president about the issue-of-the-day, slavery, to try and refocus attention on their agenda items. They nominated General Scott in 1852 to represent them. The spineless political party they were against, used the opportunity to nominate a candidate who nobody really knew and who was himself a compromise, a Southerner who was personally anti-slavery, and mirrored their opponents' agenda quite thoroughly. (Something, I should note, you can only do if your opponent has a spine and you don’t.) Voter turnout in the election of 1852 was incredibly low as a result, and that Southerner won the electoral college in a landslide, despite the fact that he did not actually do any campaigning. “I don't know what he is gonna do about the issue-of-the-day but he seems to have something in common with me” actively won out against “he thinks the issue-of-the-day is a distraction from the real problems and will broker compromises on it to try and address those real problems,” which, I don’t know, maybe was a fluke. But what wasn’t a fluke was the aftermath: any landslide loss leads to self-reflection, right, but if you have a spine it is worse: the internal tensions in the Whigs over that issue-of-the-day broke the party into two chunks and suddenly the party was no more, and a similarly spineless party ultimately took its place in the great two party system of the USA. That party’s spinelessness means that it has weathered similarly disastrous losses as the one which broke the Whigs with almost no alarm and has also led to it slowly reversing its geographic constituency from North to South as the opposite has happened to its spineless opponent, which is again much more of an interesting random-walk phenomenon that one could ascribe to entropy.
What is at stake is that you carry a subtle contradiction in your head, assuming you got some pieces of wisdom from your physics classes but not others.
One side of this is the minimum energy principle. Like, this is common sense, you leave a basketball bouncing in your driveway and you expect it to stop somewhere, probably (if it's not a perfectly flat blacktop) downhill of wherever you started it. Heck if you've had a hoop in your driveway you probably have a reflex to run after the ball when it touches ground, otherwise it'll eventually find the road and roll very far away as it chases the downhill. That's the minimum energy principle, dynamic friction reduces kinetic energy in a system while forces tend to make potential energy into kinetic energy, so you would expect if you just leave the system alone it ends up at rest at some minimum of potential energy.
The other side of the contradiction is that we tell you that energy is conserved cannot be created or destroyed. If you are very lucky, we tell you that there is a way of phrasing the laws of physics such that energy conservation is the same as saying that the laws of physics are the same today as they are tomorrow—we call this “time translation symmetry” and did the theorem that connects continuous symmetry is to conserved quantities is Noether’s Theorem if you are looking for something to google here.
The only way to resolve the contradiction is to say that friction is actually dissipation—energy is getting more spread out among the universe but is not being destroyed. So you want to picture a big bucket of water and a thousand little glasses and we empty the bucket only by putting it into all of those glasses. And the idea is that eventually if random processes take over the moving of the water from any of these to any other of these, all of them will have the same water level. You can actually see this if you see demonstrations of the siphon effect, water will actually flow up and down a hose to equalize two water levels in two reservoirs. When energy does this, has the same average occupation in every degree of freedom of a system, we say that the system has thermalized and we can measure its absolute temperature as that energy level. Technically temperature is not uniquely defined in any other context—mostly, we find physical objects whose properties like volume or length or so vary approximately linearly with temperature in this sense, then we use them as thermometers to measure temperatures in other contexts.
Now there is an interesting result, which is that if your bucket is at the same level as the cups, in some sense your bucket never ends up empty. Like there can be a lot of cups and that water can be spread over everything and there is only a tiny film of water in the actual bucket left, but it’s not zero.
This is also a theorem, it is called the fluctuation-dissipation theorem. It says that I can't dissipate energy into some environment without feeling noise from that environment prevent me from dissipating all of my energy into it: I have to accept random fluctuations back from it. In other words, there are no one-way channels for energy.
To bring this back to your question, the basketball only comes to rest on the ground because it has so much more energy than the thermal fluctuation energy which it gets back from the ground. When it eventually settles, it turns out that it is not fully at rest but is moving imperceptibly due to these fluctuations. And those fluctuations are imperceptible because the mass of the basketball is very large, large enough that this disturbs the center of mass by a height way smaller than the size of atoms, which it turns out are way smaller than the light you can see. So like even with a microscope, visible light is too chunky to show you this on a basketball.
But repeat the calculation for how far those 25 meV of thermal energy will launch a 28-amu nitrogen molecule and you will find that the height is roughly nine kilometers [1] which is a pretty good rough estimate for the height of the Earth's atmosphere, that's about where the troposphere ends. Just to be clear, the ground doesn't kick any individual molecules that high, they collide with other air molecules way before they get anywhere near that high, but that energy and momentum does ultimately get communicated to the whole swarm of air molecules and stops the swarm collectively from falling lower than that distance on average, even though everything is one big colliding mess. The first order prediction is actually an exponential decrease in density as you go to those higher heights, and I think that 9km figure is a 1/e decay constant, but the truth gets a lot more complicated as the ultraviolet light coming into the Earth is getting preferentially scattered in the high atmosphere and contributing a second source of energy to the system.
But yeah, the air doesn't fall down to the ground because the sun is shining and keeping our planet warm, and that warmth is imperceptible in the motion of a basketball but several kilometers in terms of the height of air molecules.
[1] https://www.google.com/search?q=25+meV+%2F+%2828+amu+*+9.81+...
It's easy to report that a chicken died crossing the road, it's a lot harder to explain what chain of events led to a chicken dying crossing the road.
But that's not what most of them are doing anymore. It's not so much context as it is agenda. The stories are written to lead you to a particular conclusion. From right now:
NY Times: Republicans are abruptly pushing Americans to wear masks, despite President Trump’s resistance.
Fox News: 'Looked like the Lone Ranger': Trump jokes about wearing mask, supports it
And that's not even getting into how the stories are chosen or prioritized.
NY Times: Some States and Cities Halt Reopenings As U.S. Cases Surge
Fox News: Media narrative of peaceful Seattle CHOP zone turned upside down
They've chosen sides. Maybe they always had, but it seems more blatant now. They're not even pretending anymore.
Now it’s all narrative and the focus is pannational. Not much has a local flavor. It’s not what’s good for Canadians or Ghanaians, it’s what can Canadians and Ghanaians do to make the world better (despite there being lots of things Ghanaians and Canadians could be doing in their own backyards to improve the life of their own downtrodden.
It feels like a recast comintern.
And since Murdoch likes to run his news outlets with an iron fist, these news outlets tend to report the news in a manner that strongly reflects Murdoch's thinking at the time.
Any thing Murdoch has owned and run has always comes with a level of Murdoch bias.
You could say that about a lot of places these days.
> As for NY Times I don't feel see any agendas in the titles, am I missing something?
In the first case they're both covering the exact same story but putting the opposite spin on it. Trump recommends that people wear a mask but then he often doesn't wear one himself.
He says he doesn't wear it when he's only around people who have already been tested, which could be true enough but I suspect a big reason is that it impairs communication. Human communication is lossy and full of redundancy to compensate. If you don't quite hear something but you can see their lips moving you still know what they said. People say as much with facial expressions as with words. But not if you have a mask over your face.
Trump's primary job right now is communicating, so "get Trump to wear a mask whenever he's talking to the public" becomes a priority for anybody who doesn't like him, because they have a plausible-sounding line about him setting an example, so it's a win-win for them -- either they paint him as reckless for not wearing a mask or they get him to wear it which impairs his communication. The media writes a hundred stories about it a week. Fox is only even covering that story to try to defend him, the others are using it as a political weapon and they know it.
A mask also reminds everybody of the pandemic and a pandemic is always going to be bad for the incumbent. Which is the issue with the other story -- not the contents but the priority. Trump has been wanting to reopen the country, so reopening problems in some places are bad for Trump, so that's their top story. Important things happening in Hong Kong right now. I think there's some BLM stuff going on too.
It would be one thing if this was an anomaly, but it's consistent.
The Times literally has a thing on their site right now called "Meet the Supporters Trump has Lost."
So you don't have to worry that it's some broad decline in journalistic standards (at least based on this)... The WSJ is one of the few quite reputable news rooms out there.
You can read about A-hed articles here: https://www.wsj.com/articles/SB10001424052702303362404575580...
And there was even a book published a few years ago with collections of these kinds of amusing stories: https://www.amazon.com/Floating-Off-Page-Stories-Journals/dp...
If A-heads are entertainment then why does it say A-head and not Entertainment? I am not at fault here if I don't know what the hell A-head is supposed to mean and just think that this news site is kinda garbage.
What exactly about the presentation of this article was I supposed to pick up on and see that it is entertainment. It is the exact same presentation than their "U.S. Seeks to Seize Iranian Fuel Bound for Venezuela" article. Is that one news or is it also entertainment?
I also don't want to be everything online to be "hard news", but when I go to a newspaper site I expect it to be news if it isn't otherwise noted. No the cryptic "A-Head" does not count. I am not a news insider and I think it is completely inappropriate to demand from the consumer to learn such cryptic jargon.
You said this:
> one still needs to try and understand the context of the publication and how the article is presented.
The context of the publication is that it is an article on the Wall Street Journal which is a business focused Newspaper and the Presentation of the article is exactly the same as the rest.
How is it my error then that I think an article like this is just garbage. (Hint: If your customers think your product is garbage then you need to do something about it and not just say "You are holding it wrong".)
https://www.wsj.com/articles/baseball-stadiums-major-league-... "Baseball Stadiums Are Closed to Fans—but This Guy’s Balcony Is Open for Business"
https://www.wsj.com/articles/americans-ponder-how-to-throw-a... "Americans Ponder How to Throw a Party"
https://www.wsj.com/articles/how-do-doctors-treating-coronav... "How Do Doctors Treating Coronavirus Relax? By Playing the Game ‘Pandemic’"
https://www.wsj.com/articles/a-scientist-turned-the-coronavi... "A Scientist Turned the Coronavirus Into Music—Here’s What It Sounds Like"
https://www.wsj.com/articles/the-covid-15-lockdowns-are-lift... "The Covid 15: Lockdowns Are Lifting, and Our Clothes Don’t Fit"
https://www.wsj.com/articles/oversize-sneakers-are-hot-but-v... "Oversize Sneakers Are Hot but Very Hard to Wear. ‘I Just Can’t Balance.’"
The WSJ is owned by Rupert Murdoch. The credibility of their newsroom begins being compromised by his owning it. He will destroy its credibility utterly by selling it for politicial influence in news reporting. Just as he has everywhere he has bought media. The particular example of compromised credibility that comes to mind is the Times of London which is now Murdoch propaganda (all be it vastly more polite than fox news) where it used to do credible news reporting. Times reporting now can still be excellent but has a "be cautious" flag on it that it used not to have in the days prior to Mudoch. The man has become vastly worse in the past couple of decades as has everything he touches.
I'm sure at one point they were a thinking man's newspaper. At this point they're just fan service for people who have drunk the koolaid but can't stomach Fox's mass market approach.
The opinion columns slant very far right, and I'm sure Murdoch has played some role in that. I subscribe to the Journal and am outraged every time I see them. But, it's good to see what other people are talking about. It does you no harm to read an opinion column that you vehemently disagree with. I'm sure people feel the same way about opinion columns in The New York Times, which I also read and tend to agree with.
My TL;DR here is that if you are interested in the news, you should get it from a variety of reputable sources. The Journal has proved over the years that it is reputable. I haven't seen anything recently that would change my mind. I have friends that get all their news through social media and they are constantly outraged based on pure misinformation. If they read the Journal, they'd have a much better understanding of the facts. (Same goes for the New York Times, or Washington Post, or The Guardian.)
I think this is the core issue that leads to sentiment like OPs. Real news still exists, but it's the highly editorialized and opinionated articles that are shared more widely. News agencies universally are terrible at obviously differentiating the two to users.
99% of people when they open an article scroll directly to the content. But any discerning features (in this case, a small font A-hed link) are tucked way at the top. In this case, the A-hed link takes you to the A-hed home page but still does not offer any context to what A-hed is.
> So you don't have to worry that it's some broad decline in journalistic standards (at least based on this).
As long as the WSJ does such a bad job at separating "WSJ the proper news room" and "A-hed the not proper news room", their brand will suffer. OP is proof of that, and I think it's safe to assume the average HN reader is more astute than the average citizen. A tiny link to nowhere useful is not enough of a UI change for us to blame the user. The onus is on the news agencies to do a better job giving context for articles.
Do newspapers need to put an explanation of what an opinion piece is at the top of every opinion piece?
It's clear that internet distribution is very different from physical newspapers and people don't pay much attention to the domain and styling.
Old paper newspapers accomplished this by physically having a separate section for opinion pieces. It's a harder problem when people share direct links to an article.
That said, news agencies don't try very hard. I think it's easy to argue they intentionally make it hard to differentiate. If WSJ really cared about making it clear to the user what this article was, they would have done more than hiding the word "A-hed" in a small font in the top corner.
To be fair, they mostly dominate viewership numbers by being the only right-leaning news network available, so they get half the viewing public to themselves while their competitors fight over the other half.
Sort of an indictment of the efficient market hypothesis that nobody has figured that out and gone into competition with them.
Your 1921 NYT example has "Broker is Slain at Bride's Door By Her Gardener", which the Times would never feature prominently today, because it's basically yellow press.
It also features some French pilot's unverified claims about a new record – something that would see HN storm the barricades today.
There's a story there, right at the top, about a group of visitors of the Capitol being disappointed. I'm not entirely sure if it's trying to make fun of the congresswoman it mentions. But it does sound a lot more like today's opinion pieces than a straight news story.
1. Search engines prioritized longer content.
2. People are less aware of longer-lasting events and most people land on news stories from social media as a one-off. The additional context is needed.
As radio and then TV (both of which favor emotion over information) displaced print as the major avenues for “news”, shaping expectations even of print media.
There is no crime in making a creative story that people enjoy reading out of a true event.
Who is "Eric Breindel" ... well Wikipedia includes this, as his legacy:
"In 1988, Spy magazine ran a feature that depicted Breindel as a ruthlessly career-driven opportunist whose career was effectively ended by his drug bust"
And regardless of accuracy, I consider this to be an excellent definition for whatever "Opinion Journalism" must be.
[0]: https://en.wikipedia.org/wiki/Eric_Breindel_Award_for_Excell...
Hint: Never
Read: A History of News by Mitchell Stephens https://openlibrary.org/works/OL1854941W/A_history_of_news
Read: https://en.m.wikipedia.org/wiki/Yellow_journalism
Read: https://en.m.wikipedia.org/wiki/History_of_French_journalism
Social media happened.
People have been made to think that they have got what they needed from a news article from just the title, image, summary as the next shiny news article was just waiting below to scroll.
News media are under tremendous pressure to entertain the few who actually clicked the article and is reading it.
World over only those news media which started out as traditional newspaper have some journalism integrity left and are serving news as it is. They are paying for that by going bankrupt, getting sued, journalists getting murdered etc.
More over people are getting news only from those sources which gives the news 'they like', rather than going to source which reports factual news.
It’s off-topic.
> The purpose: making sure the all-in price for the product, including tax and shipping, matches the listing on its Google Shopping platform or in advertisements. It wasn’t to cause angst to merchants due to thousands of abandoned carts.
> “We use automated systems to ensure consumers are getting accurate pricing information from our merchants,” a company spokesman said. “This sometimes leads to merchants seeing abandoned carts as a result of our system testing whether the price displayed matches the price at checkout.”
You'd think they could have better identified themselves in accounts they were creating rather than creating this mysterious "John Smith" persona. Maybe "GoogleBot PriceVerifier" would have been a better choice.
edit: remove my inaccurate confusion about something, and fix quotes that I'd copied from a plagiarized version of the article.
They're saying if they were to lie to their customers, they'd want to make sure they're deceiving Google. In that case, having an easy way to detect that it's Google would make that trivial.
Is it wise to intentionally refrain from offering the same price to a human you'd offer to the Google-bot? Possibly. Depends on your goals.
English is my first language, I didn't find the writing that terrible. It's certainly no worse than the standard of output from most large news outlets. Could you cite some examples?
Sorry, it was my mistake. Before I saw the archive link, I had found a reposting on some sketchy website and got the tabs confused. That version is basically plagiarized but makes some weird word substitutions.
WSJ:
> For more than a year, online merchants selling items ranging from kayaks to keychains have puzzled over the mystery shopper with the generic name behind thousands of abandoned carts. Each cart has only one item.
https://apkmetro.com/who-is-the-mystery-shopper-leaving-behi...:
> For greater than a 12 months, on-line retailers promoting objects starting from kayaks to keychains have puzzled over the thriller shopper with the generic title behind hundreds of deserted carts. Every cart has just one merchandise.
WSJ:
> The purpose: making sure the all-in price for the product, including tax and shipping, matches the listing on its Google Shopping platform or in advertisements. It wasn’t to cause angst to merchants due to thousands of abandoned carts.
apkmetro plagiarization:
> The aim: ensuring the all-in worth for the product, together with tax and transport, matches the itemizing on its Google Purchasing platform or in commercials. It wasn’t to trigger angst to retailers as a consequence of hundreds of deserted carts.
* Baseball Stadiums Are Closed to Fans - but This Guy's Balcony Is Open for Business
* Americans Craving Contact Ponder New Rules for Throwing a Party in Real Life
* When Your Best Friend in Quarantine Is a Squirrel, You May Be Going Nuts
* Beware of Falling Tofu: China Takes on High-Altitude Littering
* Did You Forget Things During Lockdown? So Did People With Superior Memories
In that context I don't have a problem with the title "Who Is the Mystery Shopper Leaving Behind Thousands of Online Shopping Carts?".
Saved you a click.
I kind of liked the mystery shopper angle, but since there's more than one complaint in the thread, the guidelines win ("Please use the original title, unless it is misleading or linkbait")
I know HN is not very keen on adding features, but this is one that is missing for the sake of transparency (seeing if the original title was editorialized)
I understand that the original title was click-bait trash and this one makes sense, but it would be nice to understand how it changed so certain comments don’t get de-contextualized.
But I guess is the same problem with editing comments.