starwind

I made this same transition from data science to data engineering about 18 months ago and I've never looked back.

I hated working with bad code and dealing with arrogant phds who don't value good code. I've seen so many terrible Jupyter Notebooks just copied and pasted into VS Code and the data scientist just washed their hands of it calling it "production ready." Here's a conversation I've had multiple times:

Me: have you ever considered not making every variable global scope

Them: that's just software engineering. We do machine learning

Me: if it's just software engineering, then why can't you do it?

Meanwhile, automated data science tools are getting halfway decent. If you know what algorithm to pick and you don't need to run millions of records through the model every minute, your standard business analyst could probably get a solid model going--at least as well as most data scientists for all the reasons the article mentions.

And I like that I know I can do data engineering. With data science you can never really know if you can hit your target metrics given the data you have. So data scientists end up encouraged to fudge their results or make sloppy decisions. With data engineering I can say "yes this is doable or no that's not" and people believe me.

My prediction: there's value in the massive volume of data but most of it can be had through standard dashboards, some summary statistics, a graph network, or maybe a linear/logistic regression. Most data science is BS and companies aren't getting the return they need to pay for these guys. (And good God, you almost certainly don't need a neural network.) Meanwhile, data engineering will get integrated into software development, and machine learning—by virtue of its proliferation through academia—will just become another tool for software developers. Data scientists won't get laid off enmass but they will go the way of the webmaster: either pick up new skills and evolve or move on til they end up with new titles

From SNL: Start-up airline Boom Supersonic has announced it is going to fly passengers anywhere in the world in four hours or less for just a hundred dollars. So get ready to fly fast and cheap on the only airline named after the sound of an explosion.

On a personal note, this company is based out of Centennial Airport (KAPA) which is in my neighborhood

Not true, string theory made numerous testable predictions of what particles LHC should detect at different energy levels. AFAIK none of the predictions unique to string theory and it's different branches have panned out.

At this point I see string theory as more of a mathematical framework than physics

Our current understanding of gravitational lensing and the speed of galaxy rotations would both suggest that we're between 80-90% of the mass in the universe. It's important to note the missing mass calculated via lensing and the missing mass calculated via the speed of galatic rotations matches pretty well. Whatever this stuff is, we don't see it directly, and we don't know what it is.

Dark matter is the result of observations...of something

Curious, I googled "In the radio astronomy of Pulsars we find that the shorter wavelengths of the leading edge of the pulse arrive before longer wavelengths. The velocity of light, c, is NOT constant but varies by wavelength" and something interesting came up:

"interstellar clouds of electrons retard the velocity of radio waves in direct proportion to their wavelength" in an article from a 1968 edition of Scientific American written by Anthony Hewish...the same guy who discovered pulsars

I'm sure that's exactly what happened here. You can opt out of that but it's a pain in the ass and some companies make it so you have to do it for each merchant

> gaping OpSec hole you have that resulted in the disclosure of your card number to a third party

You mean like the server I hand my credit card to to pay my check at dinner?

Does this relate to the Krebs cycle running backwards thing that was talked about earlier this week?

https://news.ycombinator.com/item?id=32392937

> Smart people will on average probably be better of financially, but that doesn't transfer directly to happiness.

eh, not exactly true: https://www.pnas.org/doi/10.1073/pnas.2016976118

> But what if you identify problems you cannot solve?

then you shouldn't take it personally

I’ve seen some other studies that say the opposite and I’m generally skeptical of results that come out of the GSS (it’s too easy to data dredge). There’s a decent amount of evidence that IQ positively correlates with happiness. Here’s a study I found real quick: https://pubmed.ncbi.nlm.nih.gov/22998852/

Results: Happiness is significantly associated with IQ. Those in the lowest IQ range (70-99) reported the lowest levels of happiness compared with the highest IQ group (120-129). Mediation analysis using the continuous IQ variable found dependency in activities of daily living, income, health and neurotic symptoms were strong mediators of the relationship, as they reduced the association between happiness and IQ by 50%.

There’s also evidence that IQ correlates with education, income, number of friends, lifespan, having a successful marriage, and having the desired number of children you desire (in case that’s not clear here’s an example: if you want 2 kids, it’s more likely you end up with 2 kids the higher your IQ). All that surely plays into happiness

Disclaimer: Individual results may vary

oh interesting. When the article said he was in first gear and the poster above mentioned a handbreak I assumed he had a stick

You're right. If you ever lose your brakes get the car in neutral and try to coast to a stop.

That said, he had his foot on the brake so I assume he had the clutch in

> I see so many people with depressing faces every day, it's hard to deny that we have widespread psychological sanity issues in our society.

No society in history has ever had so much abundance as the West today. We've really moved up Maslows hierarchy beyond survival and safety towards needing meaning in our lives (more than what comes with doing the labor you need to survive).

I wonder if there's just not enough meaning in the world to go around and that's the cause of our society's malaise.

> this is only because I will be locked out of access to certain essential financial products as the result of having no credit history.

Pretty much every bank offers a credit card with a 1% cash-back rewards rate and their underwriters are more generous to their customers with no credit history to help address this exact problem. Get one, use for like 6 months and build up a credit history then go get an AmEx or whatever

As I understand it: before gravity captures light--which moves as fast as anything can move--gravity will have captured everything else and overcame all other repulsive forces crushing the matter into a singularity

edit: pixl97 made a really good point right as I posted. I should say "As I understand the math"

really interesting point about hypersonics and plasmas. Developments in aviation have trickled down and led to other developments in technology—tons of work in modern control theory, material sciences, and aerodynamics was originally motivated by trying to build a better plane. I’d be surprised if we didn’t learn a few very practical things about plasmas while working on hypersonic weapons

Deep geothermal energy. Here's the best explanatory video I've seen: https://youtu.be/8erbvqFZ9M8?t=50

Caveat, I have no idea if these guys can do it but someone can. At this stage, it's about getting done it accurately and profitably.

https://newatlas.com/energy/quaise-deep-geothermal-drilling-...

https://www.vox.com/energy-and-environment/2020/10/21/215154...

Worked in defense so I have thoughts.

Warning: it's a slow moving business. Lots of subcontractors do cool stuff with modern tech stacks but the bigger companies--think BAE--are dinosaurs.

Defense touches a lot of different aspects of life. GPS came out of the US Air Force and is being maintained by Space Force (I think, might still be Air Force). DARPA funding has been critical for the development of modern robotic limbs. The Missile Defense Agency exists to protect North American airspace from bad things that explode that could come flying over the arctic. Satellite operations—communications, orbit analysis, making sure they don't get hit by debris—is a big area and only getting bigger.

So there's a lot of aspects of the defense industry that are a lot more about defending people than killing people.

There's also lots of opportunity to do work where you would work on weapons or military intelligence. Most weapons development today is about making weapons smaller and more precise to limit the number of innocent people harmed. There's lot of work in the intelligence field trying to quickly and accurate identify targets via satellite images or signals intelligence or videos posted on Telegram and Twitter to more accurately assess threats and capabilities and potential targets.

Joining the defense industry doesn't mean that you have to work on any project they give you. I'm very clear that I won't do anything for the Saudi's because of what they've done in Yemen and vocally opposed projects my past companies have done that I found disagreeable. My security clearance goes with me if I resign so defense companies want me happy cause I ain't easily replaceable. If you see something unethical, there are people who'll listen if you speak up

The Coase Theorem in action...

> The Air Force has even opened its own venture capital arm and is actively investing in small businesses.

In-Q-Tel is the VC arm of the CIA. I've never directly done a deal with them but the employees I've met seem really nice and helpful

NBER seems to emphasize changes in Real Personal Income and the unemployment rate when they identify recessions.

Not yet exploited: I'm a big fan of deep bore geothermal energy using a closed loop. Here's an example from a company trying to make it work. Idk if they can actually do it but it can def be done by a large enough entity.

https://www.youtube.com/watch?v=PtQmGPmyLA0

Build some pipes in the ground. Pump an anti-freeze like fluid into the ground so it heats up. Bring it up on the other side and the heat from the fluid heats some steel pipes submerged in water so the water turns to steam. Steam turns turbine and electricity is generated.

You can either send it straight to market or you could store it via batteries, flywheels, compressed air, your choice.

Lots of areas where energy could be lost but the earth is hot so it's not like we're depleting a resource

On multiple occasions I've emailed back and asked to be put on a "do-not-contact" list but it never works :/

After replying to FB recruiters asking to be taken off their contact list one finally added a note to my file saying not to contact me and I haven't heard from them since. So...props to Facebook I guess

> Apple are saying "we will protect you from state actors", which is a role usually performed by states

Not to sound flippant, but defense attorneys do this, too. I don't think it's as big a zeitgeist as you think

> Here's the thing about conspiracies: they have a shelf life. The amount of time you can keep a conspiracy secret is bounded by the number of conspirators, how well you can get them to shut up, and how long it has been since the conspiracy occurred. You can take a secret only you know to your grave, but work in a group of ten and someone will confess before their death. Bring in hundreds or thousands and the conspiracy will be blown open accidentally before you can even execute on it.

That's what they want you think!

Excellent point. As a matter of legal principle in the United States you can't unilaterally disclaim liability for future injuries. Signs that say "not responsible for accidents on these premises" don't hold up when you run a retail store where people are expected to freely to come and go as they please.

More concretely: a perfectly reasonable judge could decide a dangerous step-by-step how-to guide negates the prior "for entertainment purposes only" warning. There's a reason Jackass frequently had skull-and-crossbone warnings and didn't show much of their set up, just the results

I've always been shocked at how crappy the research into nutrition is and how there's very little consensus beyond "vegetables are good" and how shoddy a lot of the professional training is for dieticians. Nearly everybody eats nearly every day, how have we not figured this out?

You are correct. It's likely that the majority of humans have been lactose intolerant for human history. It's primarily people of Northern European descent that are not lactose intolerant, but lactose intolerance is very common outside of those groups

There have been a couple times where I've said something like "I'm sorry, I'm not seeing what's so funny here" when people are laughing about something serious. You can either look like a humorless dick or you people snap out of it and then respect you for acting like an adult

They were farming some crops but wouldn't have been as dependent on them as Romans would have been and would have had a lot more variety including meat in their diets