baazaa

I didn't mean to imply that engineering-values only pertain to things like code-quality. Project management emerged from engineering and it was a lot better when it was done by engineers who embraced engineering-values.

If you look at mid-century project management, when it was engineering-based, a lot of the focus was on things like optimising resource inputs, identify the critical path so scheduling could be done to minimise blocking, figuring out approaches to accurately calculate time and money estimates (i.e. making sure estimates were not uniformly biased towards being underestimates as that demonstrates a failure to make empirically-informed predictions).

While I totally agree that nowadays most projects fail due to bad management, I'd go further and say they fail because management is HR-values based. None of the above has anything to do with kindness, or indeed social skills in general.

Engineers, having systematically deprecated engineering-values, now have their projects managed by non-engineers who go around screwing things up.

If we tallied up all technical projects in the Western world that failed, how many would be failing due to lack of kindness versus, say, straight-forwards incompetence? Because 100% fall in the latter category in my experience.

One of the problems with engineers counter-signalling engineering values (like actual technical competence) is that we live in a world where those values are extremely underrated while every manager, HR-bot etc. are already pushing values like kindness.

E.g. if you ever wonder why government doesn't work it's because they're absurdly skewed towards HR-values and opposed to engineering-values.

Schemas are overrated. Often the source-system can't be trusted so you need to check everything anyway or you'll have random strings in your data. Immature languages/libraries often do dumb stuff like throwing away the timezone before adjusting it to UTC. They might not support certain parquet types (e.g. an interval).

Like I've recently found it much easier to deal with schema evolution in pyspark with a lot of historical CSVs than historical parquets. This is essentially a pyspark problem, but if everything works worse with your data format then maybe it's the format that's the problem. CSV parsing is always and everywhere easy, easier than the problems parquets often throw up.

The only time I'd recommend parquet is if you're setting up a pipeline with file transfer and you control both ends... but that's the easiest possible situation to be in; if your solution only works when it's a very easy problem then it's not a good solution.

Tangentially related: it now takes our data eng team 6 months to mirror some tables (into databricks) due to data vault modelling... presumably to handle schema changes. And then at the end of it everything is riddled with duplicates and missing data because they don't know what they're doing. But none of the source-systems can do schema evolution anyway, so we know the schemas would never change.

I think as an industry we should stop warning juniors of 'premature optimisation' (kids aren't even choosing the right data-structures/algos/architectures and are getting terrible perf), and instead warn them away from premature scalability and premature 'flexibility'.

Yes and I'm talking about the original 2016 house-republican DBCFT proposal which would have eliminated tax on investment in general.

Section 174 is a hack to fix a moronic tax that would have been replaced if not for the left. There shouldn't be a section 174 to even amend. And as I said, software development clearly is development, the change makes perfect sense.

If you want R&D to be taxed (even with deductions which get amortized), and you work in R&D, you don't get to whinge when a loophole gets closed so you get taxed like other R&D expenses. The people getting bankrupted by this deserve it insofar as they opposed the tax reform bill that would have elegantly solved this problem forever.

The best solution is to incentivise investment by setting its tax rate to 0. That's what Trump's cash-flow tax would have done. Higher taxes on the money printing ventures (Google ads etc.), lower taxes on start-ups and companies developing new products. That would be besides all of the other benefits like preventing profit-shifting.

Maybe if Silicon Valley wasn't so intensely ideological it would have backed the right horse when it came to tax policy. I have very little sympathy here because I think most software developer salaries really are 'development', the problem isn't the classification of developer salaries, it's that the current corporation tax is dumb.

The notion of 'culture' does this. It's a meaningless term. No-one ever says 'we fixed the business culture but the company still isn't delivering', because the culture is always inferred from the results. So as a concept it has no explanatory power whatsoever; it only obfuscates, providing a convenient cover for incompetent managers who can say dumb things like 'culture is hard to change'.

I work in the public service where accountability has been transformed from a concrete thing (who is responsible for this?) to a 'value'. So the org chart is set-up so no-one is accountable for anything, the processes are set-up so no-one is accountable for anything, but if you point this out managers will retort that it doesn't matter, because what's really important is that everyone personally commits to the value of accountability which in turn will create a culture of accountability. They've replaced performance reviews with workers writing about how much they embrace accountability (of course the absence of genuine reviews helps ensure no-one can be held accountable for poor performance).

I'm increasingly of the opinion that any manager who talks at length about culture is a moron. It's a fuzzy catch-all that can be invoked to explain anything without understanding anything, of great value to the modern manager who typically understands nothing.

A lot of the design decisions seem to prioritize expressiveness over performance, with the type system and multiple dispatch etc. If they wanted a performant python, they wouldn't have made those choices. So clearly they think Julia is good in other ways, but it didn't make the marketing pitch, which tended to be just 'performant python'.

Now Mojo's going to come along and probably fill that niche, which might at the very least force Julia's community to focus a bit more on its other advantages.

It feels like Julia would have side-stepped a lot of grumbling if it hadn't made its central pitch 'performance'. From time-to-plot to type instability and GC issues, lots of reasonable performance gripes have been amplified because we were sold that everything would be 'blazingly fast'.

Yes and pushing 'when do i update to/from the server' onto the user is exactly what I'm complaining about. No other software I use does this. Even software originally built to do things locally allows you to seamlessly update to/from a server nowadays (e.g. working on a cloud document in Word say). And any software built from the ground up to do this tends to have a great user experience (e.g. Figma).

You can still cache things locally, it's just shouldn't be something the user has to be aware of. What I'm typing now isn't immediately being communicated to the central server every keystroke, there still is local state, but it's still a client-server architecture.

The repos I work on are often smaller than the average webpage, I'm really not buying the idea that in the age of HD streaming bandwidth-concerns are an issue here.

Incidentally none of the teams I've worked on ever even use any git features like bisect. The cult of git means everyone has to use git, even though for most people that just means many hours wasted googling with absolutely nothing to show for it (as in git is strictly inferior than just sharing files across a networked drive for many teams' use-cases).

This always degenerates into a calc vs stats debate when really the problem is the vast majority of the calc curriculum is a waste of time. Even if you become a professional engineer, you're not going to be solving lots of complicated integrals by hand.

If the curriculum started with 'imagine the student has access to a computer' you could easily cover not just stats and calc but a whole lot of discrete math etc. as well, just from the time saved not teaching everyone the huge 'bag of tricks' necessary to solve a few equations by hand (in the real-world most things don't have closed-form solutions so the tricks are utterly pointless).

Note the problem goes far beyond high-school curricula. I studied a subject on queuing theory at one point and realised that everyone in the class was less equipped to deal with any real-world queuing problem than any of my CS friends who could code. Simulating the queue allows you to solve most things, whereas only an extremely narrow class of problems could be solved by hand. The only reason the maths curriculum is the way it is is because it's barely changed since computers were invented.

Totally agree that people spend way too much time complaining about git's solutions but there's not nearly enough criticism of the fact it's solving the wrong problem 99% of the time.

I want to share 5k LOC with a colleague. Anyone who thinks that requires a decentralised solution in 2023 is unwell.

I recently wasted a few hours because I forgot to fetch before running some code. Sure the UI could be better (if you're very tired and you're told you're up-to-date you'll stupidly believe it), but fundamentally the 'feature' that caused this problem is something I'll never use. I'm always going to code with a connection to the internet. I shouldn't need to mentally keep track of local vs remote because that distinction simply shouldn't exist.

Torvalds liked to code on planes back when you couldn't use the internet and that's just about the only use-case I've ever heard where I agree distributed makes sense.

Kids these days can't even code at all without chat-GPT, there's a central server hosting the git repo anyway, the whole architecture feels like it was designed for dial-up.

I can't think of anything that was doable 30 years ago compute and bandwidth-wise, that we can't do today due to performance reasons, except client-server source control...

The massive generational redistribution of wealth means that naturally the interest-rate consumer spending link has changed. Before higher interest rates meant boomers had less to spend after their mortgage, now it means they have more to spend from returns on their investments.

Never understood the hate for pie charts when stacked bar charts have a similar problem (hard to compare groups when they're in different positions). If a pie chart isn't good enough, probably switch to a paired bar chart or some other pair-wise comparison.

The author's point is that poor support for DOTS meant the devs had to roll their own culling implementation which they screwed up.

The problem with the neoliberal narrative is that academics were the best paid, most well-trusted, and highest job-security professionals in existence. They had incredible bargaining power, and indeed traditionally universities were virtually run by faculty. So how did all these (provably very left-wing, many avowedly Marxist) academics let universities be transformed by a few mid-wit administrators copying ideas from the private sector? And how come the results are worse than anything in the private sector?

IMO universities must have been pretty sick to begin with to capitulate so easily. The right-wing counter-narrative at least is internally consistent.

If you want to reconcile the two, the academics I know all suggest the wokest, youngest academics with the least regard for intellectual rigor usually allied with management. Like if you want to teach in the humanities, it's a big help to kick out all incumbents by saying that their courses are eurocentric etc. The more recent generations of academics have nothing to complain about re: universities because they were active agents in its demise.

I don't mind inequality but I hate this is always framed as 'the market' at work. There is no market here, there's no price discovery, and the people actually paying Pinchai's salary (shareholders) have incredibly little say in the matter. No economist would ever assume governments exclusively act in the public's best interest, but it's bizarrely common to assume boards and senior executives act in the shareholders' best interest.

I'm in Australia and data engineers here are overpaid, afaict slightly better paid on average than software engineers who are better paid than data people.

So you end up with bizarro stuff where a $150k AUD data engineer who basically can't code or even set-up FTP telling some maths PhD data scientist on $100k what he can or can't do. Managers, unable to gauge technical ability, assume the data engineer must be highly skilled because they're getting paid so much.

The wages and lack of skill I think are symptomatic of there being no real entry-point in data eng. There's a very slow osmosis of people from genuinely skilled areas into data eng, the main drawback being that the entire culture of data eng is defective with so many unskilled hacks.

The lack of a roadmap is disastrous in terms of professional quality IMO. A lot of orgs have data scientists who are good at coding and maths who can't do their work because the data engineers, who got their jobs because they kinda knew SQL a decade ago, are incompetent.

I'm sure this is less of a problem in tech companies which are skimming the cream of the crop, but in other organisations it's apparent that a third-rate data scientist or software engineer is probably much better at their job than a second-rate data engineer. And indeed if I were running an IT dept I'd probably just fill the data engineering roles with decent coders and tell them to reskill a bit.

I get the impression that this is a relic of how calculus developed. Basically the creators had a vague intuition of an infinitesimal, it was later replaced by supposedly much more rigorous limit definitions. Then it turns out just creating a non-standard system in which infinitesimals exist is perfectly fine, just like imaginary numbers are useful.

The equivalent to i = sqrt(-1) might be something like the nilpotent infinitesimal where epsilon^2 = 0 but epsilon =/= 0.

This is a problem across the civil service, although not as stark as this example usually.

The cult of the generalist has long meant that specific skills are considered almost worthless. Instead of the civil service being an assemblage of people with different skill-sets, it instead consists chiefly of administrators with no more skills than a high-school graduate (although they do have tertiary degrees). Jobs which require actual skill are then farmed out to contractors by these 'generalists'.

The problem is baked into the pay structure, which tends to be based on seniority and how many people you manage rather than skill. You can't just up salaries across the board because then you'd be paying all the unskilled 'generalists' far too much.

They need to completely reorient the civil service away from the notion of generalists, because they're not true generalists. If someone can't code in this day and age they're not a generalist. And if they have no other specialist skill-set besides (say law or accounting) then they shouldn't even be considered professionals. What we have today is a civil service made of pseudo-professionals who in fact do exactly the same sort of work once done by much cheaper clerical staff.

Human sociality was, in our evolutionary environment, built on reciprocity. Tribes were complex webs of interdependence and mutual aid. The closest to that today is inside firms, where transactions are not mediated through markets but instead depend on relationships, personal favours, power hierarchies etc.

It's hardly surprising that work plays an increasingly central role in everyone's lives when the alternatives are ersatz forms of sociality where people who share no common interest whatsoever get together under the pretence of 'having fun'. Anecdotally a lot of people I know who lived through long lockdowns realised that they didn't really miss having to make small-talk with strangers at parties. It might look like lockdowns greatly impaired their social life, but their social life was mostly a chore anyway. The weekend life of meeting people for drinks or hobbies etc. is failing not because people don't put in enough effort, but because that's not how humans bond, and they've in fact put in a lot of effort in an approach which hasn't paid dividends.

IMO it's the reverse. I could swear some of these games intentionally keep the gap between good-play and bad-play very small so as to hide the deficiencies of AI.

The lack of choice in buildings, reasonably inconsequential bonus tile yields, minimalist tech tree where you have to research everything, lack of synergies between buildings-resources-terrains, etc. all make a lot more sense when you see what horrendous decisions the AI makes even on hardest difficulties.

Agreed.

Sure many academics today lack the job security to stand up for themselves, but that's only because a whole generation of ultra-powerful tenure-track faculty before them allowed this to happen. Also you'd have to be an idiot not to realise that universities were intentionally over-producing post-docs to create an easily exploitable labour pool. The lack of job security is not some act of god, but the product of a corrupt system in which academics once had absolute power.

The whole institution should be considered a damning indictment of the sort of people who aspire(d) to become academics. As in there is something seriously wrong with them at the characterological level. Usually institutions decline due to various external pressures which creates internal faults. Whereas in this case an omnipotent clerisy simply spent a lot of time not working, exploited underlings and took credit for their work while fornicating with the more of attractive of them (and no, not in a 'and then they married and lived happily ever-after way', as in every year profs would sleep with multiple very young women, that was a perk of the job for boomers).

People who make important decisions get paid a lot and have a lot of power. Why on earth would they then delegate that decision-making to data-peeps and make themselves entirely redundant? Data scientists are always going to be funnelled into make-work projects just so companies can claim they're doing well on the data-side of things.

The same is true of automation incidentally, there's lots of big companies doing a lot of easily automatable work, but the guy who manages all those people doing easily automatable-work is hardly going to be scrap his own area by calling in some SWEs.

On a completely unrelated note, the author's data eng sounds like nothing I've seen. Hell veto power over code? I'm not even sure all of them can code. They're just glorified sys-admins who now can provision some cloud infra. Somehow data engs are probably even more incompetent on average than data scientists, and the reason you move upstream is because it's an easier job and you don't need to spend your weekends grinding through maths-problems or learning new languages while probably still having higher value-add.

I feel like this is guaranteed to happen because everyone involved in the decision making can make a lot of money from it very easily. Easy to get caught insider-trading when it's shares and there's a broker, but buying up proof-of-stake currencies? This is a zero-risk windfall for politicians and you should invest accordingly.

He's air-gapped.

Critical thinking is such a nebulous term there's not even an agreed-upon way of measuring it, much less old tests of it to compare against. It's largely been used by the educational establishment to justify why students come out of 13 years of compulsory schooling functionally innumerate and illiterate. Oh sure they don't know anything, but their critical thinking skills are off the charts!

If we use the slightly better-defined term of problem solving skills, there's no worse way of teaching that than putting a teacher up in front of 20 students teaching a standardised curriculum and giving out standardised tests. In the CS context, most students first learn how to think through a problem when they learn to code. What were they doing in school? Certainly not thinking through problems.

Edit: Also you completely misunderstood the import of those two questions. Anyone with a passing familiarity with world literature can answer the Dostoyevsky question, and anyone with a passing familiarity with geography or Russian history can answer the Urals question. You're note supposed to memorise the answer, the answer comes naturally if you're not pig-ignorant about the world. I wouldn't expect most students to know them, I sure as hell would expect more than 0.00% to know them. Kids know nothing because schools imbed extreme anti-intellectualism into even the brightest of kids.