DataDaoDe

great stuff, thanks so much! ! I've been looking for python internals resources.

They have a lot of examples in the appendix, just scroll to after the references.

It’s probably a good thing Tom Lehrer didn’t know about this paper…

As someone who knows nothing about the field of archeology, it would interst me to know, what are other methods of dating they coud use to improve the estimate, or address to some degree the concerns you raise?

Came here to say the same thing. This is not a paper on mathematical artificial intelligence, at last as I understand it, not in the way Goertzel or Hutter write about it. I was actually thinking this would be something like Hutter's universal artificial intelligence.

Here's some information about nuclear in terms of safety and environmental cleanliness in the context of other common energy sources (https://ourworldindata.org/safest-sources-of-energy).

From what I can tell from this data as well as other sources I've seen in the past, it seems to be by far the most environmentally friendly as well as the safest form of reliable i.e. non-intermittent energy.

Beej's guides are amazing with a memorable and entertaining writing style. I remember reading his network guide almost two decades ago when I was a kid, had just heard of this thing called a socket, but the only connotation I had for one at the time was a wall socket. His guide helped me understand networking but more importantly it made it a fun experience for a programmer just starting out.

This made me wonder if inflation is enough to account for price increases. For instance, it could be the case - I don't know since I'm not an acquainted with the field/data - that the costs of important goods such as homes, cars, education, health care, etc. has increased more or less that what that 10k approx. 70k figure tells you. For instance, maybe its the case that for a generic 10k its worth 70k distributed over all goods and services, but for those that matter most maybe its 250k or 20k, I don't know. Does anyone know more about this topic?

I think they use a basket of goods to calculate inflation, but how has that basket changed / been updated over time? Can any economist weigh in here?

Thanks! That blog post was an interesting read, age adjusted mortality rates was something that hadn't occurred to me as someone unfamiliar with this type of data.

One thing I would like to know, and I don’t know enough about mortality measures so maybe someone who does could chip in, would be essentially what is the probability of observing a year like 2020 with respect to all cause mortality. Is it in 99th-percentile or are we looking at an observation within a standard deviation of the expected value,etc.

Agreed. Its fantastic we have any such place. However, having had conversations with friends and family outside of Germany, they usually hear the costs and then imagine the system in the USA or their home country just with much lower costs. And this is just not the case. Our German system is a very different system from the USA at all levels which complicates direct comparisons.

Regardless, I think initiatives like this one which in general provide more opportunites to more people to better their lives is something I would like to see more of, not just in Germany but all over the world.

> The neat thing is that any foreigner can study any subject at any University in Germany for free.

This statement is not true. It is the case though that many if not most public German universities have costs that run in the very low thousands i.e. 1-5k for total degree costs.

However, there are some drawbacks to the way this system works. For one, you will be hard pressed to find online degree offerings or continuing studies options from public Universities (i.e. these types of low costs offerings) that allow you to continue your education while working fulltime. Most of these come from private institutions or specific programs which have higher costs typically running in the 10k+ cost range area.

Its great if you are a student graduating from high school and going to university - you won't be burdened with high costs. However, the public system largely still neglects (this lone University being the only example in all of Germany which I am aware of) many other situations where people might want to further their education.

We do have something called Volkshochschulen, which is in some ways similar to Junior Colleges in the United States and provides some level of extended studies at a public level, but anyway, I won't go into details, suffice it to say the systems don't translate one-to-one onto each other and there are cons with our system as well.

3n + 1 is interesting because it seems so tantalizingly easy and beautiful at first glance. Its like looking at a stream where you can clearly see the stony bed and it appears to be ankle deep but once you take a step you realize the lighting has fooled you and you fall head over heels into icy waters.

Speaking for myself, the biggest problems in my thinking are as follows:

1. Thinking you know anything with an even remotely high degree of certainty.

2. Confirmation Bias - thinking you are right and searching for evidence to prove you are right, reaffirming your own beliefs in the process and actively ignoring or providing excuses for counter examples and contradictory evidence.

3. Difficulty/Inability of looking only at the facts without interpreting them as you would like them to be or not to be.

3. Inability/Difficulty of admitting you are wrong/made a mistake and also remember it and learn from it.

4. Inability/Difficulty of acknowledging you are an irrational, flawed monkey stumbling around trying to make sense of about a billion things you don't understand and never will understand.

Trying to still make progress given all the above.

First, this is a nice resource, so good job!

As someone who has worked for the past several years in this space, I'd say the biggest problems in data engineering are wholistic in nature. Sure, you need to know Python, SQL, Data Warehouses, Data Modeling, etc., but to me by far the biggest problems have to do with the entire architecture i.e. How do you extract data from potentially unreliable data sources, pull that data into some staging area, build further workflows that base off of this raw data to reliably update or create data warehouses/marts or deploy ml models. How do you allow everyone in your company to access and work with the data in a compliant and secure way? How do you test any of this? How can distributed teams, sometimes technical, sometimes more business oriented interact with the architecture and add/control data and release it into the overall company data stream? Has anyone found a reliable and maintainable way to setup CI/CD for company data architecture/pipelines/projects?

To me these are the big problems. And if anyone has any resources for any of these topics I would be super interested, since I deal with these problems daily :)

I couldn't agree more. Once you realize that energy is the fundamental unit which powers everything else in the economy, not only from every movement of ourselves to those of our machines as well, you realize the more we advance in our ability to produce more and more energy at cheaper and cheaper costs in increasingly reliable ways, the more we can accelerate, without exception, every other industry.

I can't remember his name but there was a researcher (I think he was German) back in the early 2010s who predicted that as renewables became cheaper and gained more market share the price of energy would go up. This is because renewables are intermittent i.e. unreliable energy sources which means you need to maintain multiple energy infrastructures, a fallback for whenever renewables can't meet demand i.e. its nighttime or the wind isn't blowing, in addition to the renewable energy grid and switching back and forth has inefficiences. This is all if I'm remembering the paper correctly - it has been years since reading it.

If you think about the scientific lifecycle: Gather Information => Form Hypothesis => Test Hypothesis => Analyze Data => Interpret => Repeat.

Then I would say the hardest parts are the "Gather Information" and "Test Hypothesis" phases. But its like this in every scientific endevour and this is nothing unique to data science.

One interesting point is, perhaps, that we as data scientists are aware that our sources for gathering information and our means for testing hypothesis are often tied to man made software or hardware systems - as opposed to dynamical real world structures. This means that theoretically and practically there is only ingenuity and will-power keeping us from building better and less time consuming ways for automating away the tedius (data cleansing/prep/etc.) parts of those processes.

I would like to add to your remarks that science actually says something along the lines of "Grant me these assumptions/axioms/miracles(i.e. space-time, causation, etc.)" and then I can give you a theory of gravity, statistics, etc. This I think is hinted at in Gödel's core insight from his incompleteness theorem.

Very interesting. Thanks for the link :)

> Governments often abuse their power. They have absolutely no problem with that either.

The way I think about it is this. Governments and companies are both organizations made up of people. Governments have additional powers (for instance, military force, police power, etc.) over companies. The checks and balances for gov't abuse of power depend on the type i.e. representative democracy, republic, etc. And the checks and balances for company abuse of power are however the markets are configured.

There are different incentives at an organizational and people level both in gov't and companies. But in general, I tend to think its easier to prevent "company abuse" of power b/c ultimately they have to deliver a product which people can choose to buy or not - in this sense we all have a direct democracy with companies, unlike Gov't.

For instance, if I choose to not buy any more Google products, then however small and insignificant it may be Google is going to notice my change of preference on the bottom line of financial statements just as will the other company whom I deem better aligned to my preferences.

This type of direct control is currently not possible with governmental institutions unfortunately.

I just finished reading "The Victorian Internet" by Tom Standage and would highly recommend it. The book is an informative and enjoyable read about the rise and fall of the telegraph in the 19th century. The development of the telegraph has a lot of parallels to the modern internet that are worth contemplating and this book provides a point worth noting - the modern internet was not the first means of rapid exchange of information through networks spanning the entire planet.

Here is the actual study [1]. This is n=3, so we are talking about highly anecdotal data. Nonetheless, I think this statement is misplaced:

> Low carb diets can help type 1 diabetics, but it's not a cure

Likely, as in most cases, the real picture is much more complex. It could be the case that LCHF diets can cure some percentage of individuals, can help some percentage and can be detrimental perhaps even fatal to some percentage. All of these could easily be simultaneously true.

Indeed, in case 1 it seems to have possibly "cured" the patient, though he could have also easily been a false positive or there could have been any number of other confounding factors. I quote from the study:

"He was able to stop insulin completely 15 days later (after diagnosis), and has not used insulin since March 2015." and "In the 4 years since his diagnosis, he has had no diabetes-related complications".

Here is patient #2's summary:

"He stopped insulin therapy at the beginning of the diet and has not used insulin since, except for one infectious episode." and " At 1 year after the diagnosis, he had no diabetes-related complications".

Also of note, both of these patients had started exercising 1-2 h a week, and 2 and 2-3 times a week respectively, before the diagnosis.

1. https://www.sciencedirect.com/science/article/abs/pii/S12623...

> I'm specifically interested in whether or not tools exist to help hold police accountable.

I'm interested in this too. And haven't been able to find much of anything.

> I am imagining an app that has a real-time, location-aware feed of media references to such instances to corroborate and publicize, potentially supported by facial recognition of offenders to gather evidence of repeat abuse.

I think the implementation and specific measures need careful consideration. Remember justice according to Rawls is a form of fairness among all members in society. What about the injustice going on that the police are tasked with stopping on a daily basis and their respective part in ensuring a just society? What about injustices against the police themselves? If we ignore such data points and issues, society will develop a false impression and injustice will supplant the justice we are striving for. IMHO any social justice software should be "context aware" in as broad a sense as possible (history of officers actions - good, bad, everything in between) and probably a history of the "offenders/recorders" public actions (again good, bad, everything in between) as well as specific encounters. Of course, this brings problems such as rights of privacy, among others.

A software which only captures specific encounters, and only makes them public when they are negative is something that on the whole might not be a net benefit for society. I'm unsure and I think it needs some consideration at the least.

> is that what is best for society? to create elite clubs based on who you know vs merit, capability, output volume etc.

I don't know if finding ways to expand someone's network and connections necessarily means we have to create elite clubs - having networks is vital to success in any way, especially in a system based on merit.

So maybe I was one step ahead of what you were saying. I read it as more along the lines of given the entire set of X people in our education systems, how do we ensure a system that is fair and allows for the maximal human flourishing of each individual member while taking into account our psychology and the negative externalities / side-effects of emergent system dynamics. (Of course this last point, is something we sorely need in many fields - not just education).

I've spent a fair amount of time and energy working and researching in this area. I'd agree with most of what you are saying. To point #1 I worked on a project that had almost the exact same design you mention (collaborate, git versioned edits of learning modules, etc.) - the biggest problem is the mounds and mounds of red tape and regulation that stifle any truly innovative solution. There are some movements for high school materials that are tackling parts of the problem (see: https://www.ck12.org/student/, https://www.khanacademy.org/), but it definitely needs to be on a much larger and national scale and extended to universities. I think, most all undergraduate courses could be done this way. If someone with enough leverage could ever push it through it would virtually completely remove textbook, materials, and lecture costs from the system and if done right, provide a treasure store of knowledge.

To #2 This is already a technique in education research its called flipped classrooms (https://en.wikipedia.org/wiki/Flipped_classroom).

To #3, I'm sure there are organizational structures you could implement that would augment every students network - there's a lot of stuff from management science you could try out.

Overall though I found it extremely frustrating trying to innovate in education because you really need to be able to implement radical changes but there are so many regulations at every level that you have to creep forward, hoping for minor incremental improvements when what you need is complete system overhauls. This was extremely frustrating for me.

> Probably in time to come Rust will be used for some core library development which brings substantial benefits like scipy, not something rewritten as wrapper around C library with unsafe code which can be done with Python much better.

I think a nice situation would be to have all the core scientific libraries and tools written in rust with clean APIs, independent of any particular scripting language but with the clear goal of providing a clean API to wrapper languages. Then any scripting language can wrap these libs and go - essentially something like a universal numpy/scipy/scikit, etc. This would mean Ruby/Python/JavaScript/Julia, etc. would all be able to get to state of the art immediately while minimizing the duplication of effort across languages and creating a standard "ML" or "Scientific" API.

You can have your opinion, but to answer some of your questions, I use vscode and it works amazingly with python, so yes it does know what environment and python version you are using and can show source code, do type checking (if you use the "typing" module, and tons of other nice things that just work out of the box).

As far as the CI-system knowing, there is not much difference in installing a particular version of rust/cargo say in a docker container and having a particular version of python installed in a container, so I don't really know what you are getting at.

Could it be better? Sure, I do some programming in rust and IMHO the best build tool I've ever used is cargo. But compared to some of the other stuff you mentioned which I've used on projects as well (npm, lein specifically) I would say python is on the same level, and I prefer python over them but that's probably just my bad experience with package-lock/caching and long jvm load times speaking.

I solve this whole issue by using pyenv + direnv. I can have a project specific .python-version file and upgrade to a newer python version when necessary. Then I just run python -m venv env once on project creation and have direnv source env/bin/activate and can use pip as usual

The articles arguments against this and for always using python -m pip don't convince me, personally. I never work on windows, so that's a mute point, and b/c I use direnv to automatically activate my env when cd'ing into the directory, I'm never not in the project environment. So basically, its impossible to not be using the correct pip version i.e. I have never once had this problem and I am constantly working on multiple python projects + code bases in various envs and versions.

I completely agree with the sentiment here. As an engineer tasked with building data driven systems and architectures, I'm well aware of the amount of buzzword / enterprise nonsense floating around in this space - and it is enormous. You can couple that with the fact that its really difficult to find any practical books and resources that you can actually apply to solve real-world problems when working in small-teams and on quick deadlines. Getting business to act in a data driven and analytic way, and building the architecture for it - is a non-trivial task. This looks like it could be one of those rare good resources in the space.

Great work by the guys from chartio!