Hacker News Reader

x0xMaximuson Pandas Should Go Extinct

We run Dask distributed for contents in what their chart covers (10TB+ with 4+ physical hosts) and it's a problem that can't really be discussed without well defined drive read or network latency limitations (if non-local storage). We've found defining recordsize, raid and other filesystem patterns to have an equal or larger impact in itself. Pandas is great because it allows us to validate on a sampling or smaller time window dataset, and then the same engineers can use a very similar implementation for Dask distributed through their scheduler tool.

While throwing hardware at problems isn't always a reasonable solution, it's trivialized to discuss this without IOPS or other read limitations.. and the nvme enterprise market is rapidly developing making this change quickly.

x0xMaximuson Mechanical Turk shutting down September 30

Amazon took 20% of all workers earnings for doing essentially nothing for years and years.. of course I take great joy in giving AWS as little money as possible.

x0xMaximuson Mechanical Turk shutting down September 30

Do you know how much inn-n-out you can buy when saving ~$50 a day and still be in the positive! It came out of building our own ledger system and abandoning any Decimal or float operations early on.. then it sparks "I wonder how they do it and if it's correct??"

x0xMaximuson Mechanical Turk shutting down September 30

Finally you get it. Only exception is that we run our own exchange now to do task bidding so we don’t need to deal exclusively with other companies to middleman. Core business is what’s called yield management (akin to DSP in adtech) where the best survey (is the user qualified for it, does it have the best pay, etc) is selected for traffic in <100ms. I’d only argue the shady companies are the ones paying proxies (like cint.com) and pushing paid user acquisition instead of surveys. We actively fund ontology development for better profiling targeting. but yes, big enterprises/consultancies can use and interpret the collected data however they want, not our responsibility and we have no legal rights over it anyway

x0xMaximuson Mechanical Turk shutting down September 30

Users/respondents lie, and many buyers/researchers are neo-luddites; most C level still come from the telemarketing days. A fun example: mobile targeting is terrible when off wifi because all survey platforms uniq identify users based off their IPv4, so any T-Mobile LTE users in the same city going through the same CGNAT often share profiling data that ends up conflicting, which ends up meaning they don't get sent into the best survey(s). The issue is even worse in heavy IPv6 countries like India+France.

x0xMaximuson Mechanical Turk shutting down September 30

Buyer A is presented with the following rate card:

< 10min targeting gen pop: $3 10 <= 15min targeting gen pop: $5

Of course they'll bid saying their 14 min survey only takes 9 min to complete. Buyers try to cheat pricing strategies of exchanges just as much as respondents try to cheat buyers on survey platforms. Both parties can't be trusted and have adverse incentives.

x0xMaximuson Mechanical Turk shutting down September 30

Wish we didn’t even need one

x0xMaximuson Mechanical Turk shutting down September 30

Exactly, it’s a beautiful thing when the market is rational https://finance.yahoo.com/quote/CINT.ST/

x0xMaximuson Mechanical Turk shutting down September 30

Obviously, a problem not even fully addressed by L2’s datasets. Sure, but the use of proxies to masquerade the identity of users being sold to researcher buyers that paid for a different service is fraud

x0xMaximuson Mechanical Turk shutting down September 30

The industry is so backwards, there are residential proxy companies that are directly paid by one of the largest market research firms. I have discussed this at length https://www.reddit.com/r/Marketresearch/comments/1s2tl0o/ope...

It should be a bipartisan issue that a Swedish company is paying a UAE Residential Proxy company [1] to build tools that allow people from anywhere in the world to take US political polls that are used by both parties to collect election data.

Just for starters, you can't even think about soliciting online work from a panel without a robust residential proxy detection methods. We had to build our own:

`wget -N 'https://grip.net/files/grip-proxy-30d.mmdb'`

[1] https://www.youtube.com/watch?v=eOmeQcwSK3o flagged by Nokia Deepfield and CTRL for it's involvement in botnets

x0xMaximuson Mechanical Turk shutting down September 30

Initially the annotation of biomedical literature (https://pubmed.ncbi.nlm.nih.gov/25592589/) but left academia for commercial use where we transitioned to arbitrage the market research and ephemeral task markets. It was essentially a research panel with a underdeveloped UI for tasks.. so abstract away AMT's tooling so it can be used by any buyer in the ResTech, Political Polling and Data Annotation space

My favorite part of AMT is always going to be figuring out that if we only paid in $0.07 intervals, their commission algorithm would round down to the nearest whole cent when their 20% commission resulted in a fractional cent on the unit transaction level, not the monthly invoice level. Was ultimately worth it to have implemented https://git.generalresearch.com/panels/amt-jb/tree/jb/flow/a...

x0xMaximuson Mechanical Turk shutting down September 30

As AMT's largest requester for the past 10 years, this news was relayed to requesters at the same time as respondents. It's also worth noting that our lead contact, the Sr Program Manager at AWS leading AMT, transitioned to Amazon Bedrock and SageMaker Model Evaluations a ~2-3 years ago.. Leaving behind essential zero team managing the project after they migrated over the stored value accounts to native AWS billing.

x0xMaximuson Stop Making TUIs

aerc is the best program I've ever used. tmux from anywhere. May TUIs never die.

x0xMaximuson Memory prices climb 500% in 12 months

We had last bought 24TB+ (4x 24 racks, and some miscellaneous) of DDR5 4800 and 6400 last year while finishing a move off of AWS entirely. I tried to be responsible and prioritize a cold and warm spares of all our equipment and over allocation to ensure proper redundancy capacity. At the time it was a hard pill to swallow looking at our budget; but a few months ago we've started to offload some DIMMs on various channels. What went from a cost analysis between a cloud provider, to having to manage transit, circuits, equipment, a IP blocks, etc etc and all the operational obligations that come with it.. it ultimately covering our hosting costs entirely for the next ~18months *at least* (while "only" coming at the cost of having slightly less cold spares sitting in the bottom of our cages).

Given we're all 9004/9005 now, and that (from what I understand) Venice will support 6400. Will hopefully buy back in a glut comes ready for the next bubble. It's been wild, I even posted a small 32GB DDR5-4800 that was not getting recognized for $100 on eBay and it was literally purchased within ~15min of being live.

x0xMaximuson Iran strikes leave Amazon availability zones "hard down" in Bahrain and Dubai

I recently wrote a little on this https://generalresearch.com/detail-oriented/how-to-seed-a-cl...

While we're completely at the mercy of datacenters that we can colo out racks / power / upstreams from, it's a worthy discussion for any technology company that wants some amount of digital sovereignty over their presence online and ability to provide their service independent of a hypervisor / cloud provider (or even just a centralized location).

The best option is simply to anycast from any many distinct countries that are either neutral, or unlikely to be involved with any global or regional conflicts at any given time. You don't want them getting bombed at the same time!

x0xMaximuson Alleged M4 MacBook Pro unboxing video

Assuming this is real, I can't help but think this could push Apple to accelerate their reduction on Chinese production. Is there any other realistic way this could have leaked other than through a Chinese backdoor?

Are these being made at Quanta or Foxconn? Since the start of the war, there has been a massive increase in sanction evasion through China and Central Asia. I would have to assume Valentin used a connection that has formed over the past 2 years in order to get a North American SKU as Russia's richest YouTube blogger

x0xMaximuson Shortcomings of Amazon Mechanical Turk May Threaten Natural Language Generation

Technical and Demographic user metrics are federated across the platform (we supply tasks to other labor markets online, essentially where bots and farms typically attack) so we leverage priors and shared similarity weights. Behavioral user metrics aren’t the only way to evaluate a respondent

x0xMaximuson Shortcomings of Amazon Mechanical Turk May Threaten Natural Language Generation

As AMT’s largest requester, we have longitudinal (~3yr) worker quality data for over 85% of respondents. Feel free to DM for details

x0xMaximuson Show HN: A 1000 auto-generated hexagonal SVG logos

Everything Old Is New Again

https://www.metmuseum.org/art/collection/search/691091

x0xMaximuson Ask HN: How are Digital Nomads coping with travel restrictions?

No, I simply meant that a regularly scheduled direct flight to get back home is comforting. Additionally, that wouldn't be possible at the moment as US tourists can't enter Russia

x0xMaximuson Ask HN: How are Digital Nomads coping with travel restrictions?

Entered Russia 3/15 right before they closed the borders, with my tourist stay expiring on 9/15 (and visa expiring on 10/31). On 4/18 a presidential decree auto extended everything, and another extension every 3 months. So with auto visa situation, and the weekly flights back home to JFK just incase, it's been a perfect situation..

The weworks in Moscow are amazing, and everyone is following health guidelines very well. Definitely not in a rush to head home, or at least keep riding it out here

- http://kremlin.ru/acts/bank/45441 - https://gupkrd.ru/novosti/prodleno-dejstvie-norm-ukaza-prezi...

x0xMaximuson The Programmer's Price (2014)

Hi – I'm Max from the article (long haired guy). Happy to answer any questions about what's changed since 2014...

Michael Solomon [1] still my agent, and 10x has stood by me throughout my career. While I've stopped doing consulting work for others through 10x, they still support my contracts related to art and speaking arrangements. I can't speak highly enough of how much they've done for me.

I stopped doing contract work not because of 10x, but simply grew tired of working on, and fixing other people's problems. They were completely accepting and supportive of me to head out and build my own company (since 2017). I know their rooster of talent has grown significantly since then, and I would still highly recommend companies use them to get amazing talent onboarded without any hassle.

[1] While you're looking into 10x, don't forget to check out the nonprofit he started https://www.musiciansoncall.org which brings live music into hospitals!

x0xMaximuson Ask HN: What is your favorite motto?

If not now, then when?

x0xMaximuson Show HN: Stop Covid-19 by taking your temperature

https://www.medrxiv.org/content/10.1101/2020.03.19.20038844v...

https://bmjopen.bmj.com/content/1/2/e000234

https://www.bmj.com/content/351/bmj.h3480.full

x0xMaximuson Show HN: Stop Covid-19 by taking your temperature

Make the data public like https://covidstudy.com

x0xMaximuson Ask HN: Outreach for Covid-19 longitudinal study proposal?

We've been running a longitudinal self-reported diagnosis, conditions, and risk exposure study. There are many challenges in this [0, 1], but plan to keep collecting into the foreseeable future. Currently operating on a global scale, but could be targeted to a specific geographic region.. - M (max@generalresearch.com)

[0] https://www.bmj.com/content/351/bmj.h3480.full [1] https://bmjopen.bmj.com/content/1/2/e000234

x0xMaximuson Show HN: A simplified weekly calendar to record and share wasted time

Hi everyone, Max (the creator) here, I just wanted to note that previously submitted results can be revisited with their unique pages, e.g. https://theslackertracker.com/bsoaztceyfjd/

I’m still trying to address some confusion that might come out of having the cumulative hour amounts shown in the week view, rather than a daily average or percentage of waste for that week as the value shown. I’m likely going to see how it starts to look with real submissions and will adjust accordingly. In addition, still challenged with activity + company name normalization, and general approaches to gamify and make more fun (without sacrificing submission legitimacy).

The idea started from Peter’s line in Office Space, “I'd probably, say, in a given week, I probably do about fifteen minutes of real, actual work.“

x0xMaximuson Rhizome Awarded $600k by the Mellon Foundation to Build Webrecorder

I'm the "css expert" you desire to know. No issues are open at the moment in regards the typography you're referencing.

x0xMaximuson The Programmer's Price: Want to hire a coding superstar? Call the agent

Michael (& Altay) continues to do a lot of me to excel in my other life goals. They provide advice, outlets and scheduling (via assistant) for me to take my art and science career ahead just as much as my software. If they weren't interested in my best interest of being happy on my own standards above all else then I wouldn't be with them for ~2yrs now.

I want to also note here that I leverage 10x to do less recruitment for jobs and more with purely figuring out which are the best opportunities to pursue at that time. Personally speaking, I want to spend as little time on the planning / business side of my tech career as possible so I can use that time to focus on my other pursuits. I trust (& they've proven) to be able to make the most advantageous choices for me professionally with long term goals in mind.