I think databricks and Apache Spark are probably the most successful at this. An open-source project, whose maintainers offer a hosted/managed version.
181 karma · joined September 7, 2013
osx includes a "non-framework" build of python making it hard to use matplotlib
Actually just now finding out about "venv" in the standard library introduced in python 3.3
This is the actual virtual internship I did, which doesn't actually seem accessible from the workatastartup website: https://www.insidesherpa.com/virtual-internships/prototype/o...
There's are frontend, backend and analytics modules.
Would recommend to others. The tasks aren't very difficult, there are PDFs that hold your hand pretty well.
Also completed the JP Morgan Chase one recently for fun. Wonder what the law ones are like.
https://runestone.academy/runestone/books/published/pythonds...
Grokking Algorithms
Classic Computer Science Problems in Python
Lulu https://objective-see.com/products/lulu.html
Haven't used a block-first, prompt to allow firewall on windows or linux. Similar to noscript in the browser
The interactive crossfilter part in particular
https://altair-viz.github.io/user_guide/interactions.html
https://vega.github.io/vega-lite/examples/interactive_seattl...
My girlfriend and her family have Taiwanese, Chinese and US citizenship/passports though. Wonder how that works
Devotion, a small Taiwanese horror game, is an example of people abusing these types of control on steam
>Devotion, which came under fire over the weekend because of the presence of a piece of art that appears to mock Chinese president Xi Jinping. That quickly led to a massive review-bombing campaign (that has now bled over into Devotion's predecessor, Detention)
https://www.theverge.com/2019/2/25/18239937/taiwanese-horror...
https://www.pcgamer.com/taiwanese-horror-game-devotion-has-b...
giantbomb.com sells premium subscriptions and merch and does okay
https://www.withouthotair.com/c1/page_5.shtml
>The climate-change motivation is argued in three steps: one: human fossil- fuel burning causes carbon dioxide concentrations to rise; two: carbon dioxide is a greenhouse gas; three: increasing the greenhouse effect in- creases average global temperatures (and has many other effects).
>Fossil fuel burning increases CO2 concentrations significantly. But does it matter? “Carbon is nature!”, the oilspinners remind us, “Carbon is life!” If CO2 had no harmful effects, then indeed carbon emissions would not matter.
>However, carbon dioxide is a greenhouse gas. Not the strongest greenhouse gas, but a significant one nonetheless. Put more of it in the atmosphere, and it does what greenhouse gases do: it absorbs infrared radiation (heat) heading out from the earth and reemits it in a random di- rection; the effect of this random redirection of the atmospheric heat traffic is to impede the flow of heat from the planet, just like a quilt. So carbon dioxide has a warming effect. This fact is based not on complex historical records of global temperatures but on the simple physical properties of CO2 molecules. Greenhouse gases are a quilt, and CO2 is one layer of the quilt.
> One last thing about the climate-change motivation: while a range of human activities cause greenhouse-gas emissions, the biggest cause by far is energy use. Some people justify not doing anything about their energy use by excuses such as “methane from burping cows causes more warming than jet travel.” Yes, agricultural by-products contributed one eighth of greenhouse-gas emissions in the year 2000. But energy-use contributed three quarters (figure 1.9). The climate change problem is principally an energy problem.
Watched Merchants of Doubt recently, point that stuck with me most is that people against global warming are motivated by libertarianism, and anti-communism. lots of the contrarian scientists worked during the cold war. Will be checking out the book version soon
I don't think you even need the climate change motivation to get behind moving off fossil fuels (supply chain security is another huge one, I find climate change deniers often also in the peak oil camp). The way David MacKay presents the numbers allows people to come up with realistic plans to move off fossil fuels on their own instead of reciting someone else's dogma
Teasing out cause and effect is difficult especially with something that we can't perform randomized experiments on. I'd like to read more about counterfactuals and causal analysis of climate change
Judea Pearl has a great little bit about this in The Book of Why
>Until recently, climate scientists have found it very difficult and awkward to answer questions like “Did global warming cause this storm [or this heat wave, or this drought]?” The conventional answer has been that individual weather events cannot be attributed to global climate change. Yet this answer seems rather evasive and may even contribute to public indifference about climate change.
> Counterfactual analysis allows climate scientists to make much more precise and definite statements than before. It requires, however, a slight addition to our everyday vocabulary. It will be helpful to distinguish three different kinds of causation: necessary causation, sufficient causation, and necessary-and-sufficient causation.
> Using these words, a climate scientist can say, “There is a 90 percent probability that man-made climate change was a necessary cause of this heat wave,” or “There is an 80 percent probability that climate change will be sufficient to produce a heat wave this strong at least once every 50 years.” The first sentence has to do with attribution: Who was responsible for the unusual heat? The second has to do with policy. It says that we had better prepare for such heat waves because they are likely to occur sooner or later. Either of these statements is more informative than shrugging our shoulders and saying nothing about the causes of individual weather events
> Climate scientists can get counterfactuals very easily from their computer models: just enter in a new number for the carbon dioxide concentration and let the program run.
and one of my favorite quotes from MacKay:
> Please don't get me wrong: I'm not trying to be pro-nuclear or anti-wind. I'm just pro-arithmetic
If you want some more recs, my two favorites are Chris Albon's Machine Learning with Python Cookbook and Joel Grus' Data Science from Scratch: First Principles with Python
>in that every exchange necessitates a double coincidence of wants (i.e. when Jack offers Jill some Team Fortress 2 hat in exchange for a couple of keys, the trade will go ahead if, at the same time, Jill also prefers that particular hat to her two keys).
>[...] throughout history, whenever the number of transactions (and ‘assets’) grew in number, one of those assets soon emerged as a numéraire – a basic form of money that is. Once the numéraire acquired currency, suddenly the prerequisite of some double coincidence of wants vanished and people could trade anything for the numéraire–asset which they could then use in order to buy whatever else tickled their fancy. In short, as economies grew in sophistication, they ‘monetised’ and ceased functioning on the basis of barter
>Initially, I had expected that a similar pattern would be replicated in digital economies, like Valve’s. I was expecting to find that some item or asset would emerge as currency in the context of games such as Team Fortress 2. However, a close study of our Team Fortress 2 economy revealed a more complex picture; one in which barter still prevails even though the volume of trading is skyrocketing and the sophistication of the participants’ economic behavior is progressing in leaps and bounds.
from a comment-er (brandon):
> i know that the trading system in TF2 his been in place for roughly a year now but i don’t know if i would call it a bartering system. as a player i’ve seen scrap, reclaimed, and refined metal act more as our currency more than anything. when people talk, on servers about buy or selling item the price is usually is translated into an amount of metal. like keys having a selling price of 2.66 (2 refined and 2 reclaimed worth of metal), although keys can be bought with actual money, scrap metal and the likes is not something you cannot buy in the store.
I'd like to hear Varoufakis talk more about his time at valve. I played 1000hrs+ (mostly idling) and got lucky when trading first started and got a few sam and max hats. Ended up accumulating ~3k$ for like 5 month's work lol.
Jeffrey Elman (with others) wrote a successor to the PDP books called Rethinking Innateness: A Connectionist Perspective on Development (1997)
His paper Finding Structure in Time (1990) adapted backpropagation to take time into account, backpropagation through time (BPTT):
https://crl.ucsd.edu/~elman/Papers/fsit.pdf
https://en.wikipedia.org/wiki/Jeffrey_Elman
>Elman's work was highly significant to our understanding of how languages are acquired and also, once acquired, how sentences are comprehended. Sentences in natural languages are composed of sequences of words that are organized in phrases and hierarchical structures. The Elman network provides an important hypothesis for how neural networks - and, by analogy, the human brain - might be doing the learning and processing of such structures.
https://web.stanford.edu/group/pdplab/pdphandbook/handbookch...
>Here we briefly discuss three of the findings from Elman (1990). Elman's work was highly significant to our understanding of how languages are acquired and also, once acquired, how sentences are comprehended. Sentences in natural languages are composed of sequences of words that are organized in phrases and hierarchical structures. The Elman network provides an important hypothesis for how neural networks - and, by analogy, the human brain - might be doing the learning and processing of such structures.
>The concept ‘word’ is actually a complicated one, presenting considerable difficulty to anyone who feels they must decide what is a word and what is not. Consider these examples: ‘linedrive’, ‘flagpole’, ‘carport’, ‘gonna’, ‘wanna’, ‘hafta’, ‘isn’t’ and ‘didn’t’ (often pronounced “dint”). How many words are involved in each case? If more than one word, where are the word boundaries? Life might be easier if we did not have to decide where the boundaries between words actually lie. Yet, we have intuitions that there are points in the stream of speech sounds that correspond to places where something ends and something else begins. One such place might be between ‘fifteen’ and ‘men’ in a sentence like ‘Fifteen men sat down at a long table’, although there is unlikely to be a clear boundary between these words in running speech.
> Elman’s approach to these issues, as previously mentioned, was to break utternances down into a sequence of elements, and present them to an SRN. In his letter-in-word simulation, he actually used a stream of sentences generated from a vocabulary of 15 words. The words were converted into a stream of elements corresponding to the letters that spelled each of the words, with no spaces. Thus, the network was trained on an unbroken stream of letters. After the network had looped repeatedly through a stream of about 5,000 elements, he tested its predictions for the first 50 or so elements of the training sequence.
Schmidhuber developed the LSTMs, LeCun developed CNN, the ideas were refined and processing capabilities developed and Hinton revived these connectionist ideas leading up to Imagenet in 2012
What you're talking about sounds like transaction costs and information asymmetry, which are interesting. Hayek's work is interesting to me, even as someone who would self describe as a socialist.
His decentralized market solution to the economic calculation problem, the information from prices (price signals) tell producers and consumers to increase/decrease supply and demand.
But markets aren't perfectly competitive, there are monopolies. Markets are embedded in societies, they don't exist outside of them. Markets don't guarantee an optimal equilibrium. 50% of R&D spending is by the government.
He also did early work on neural networks, published The Sensory Order in 1952, 3 years after Hebb's The Organization of Behavior
The methodology of Austrian economics seems completely unscientific, I believe a part of praxeology is disregarding empirical evidence(?). Reminds me of something like Ayn Rand's egoism. It just seems to me you can't persuasively argue a philosophical theory without empirical justification
I have a more favorable view towards Polyani's methodology, who largely draws on historical sources. The historical approach seems at least some what grounded compared to the pure theory used in much of economics
Economic theory and statistics can't answer questions like "How much of an effect can we expect if we were to raise minimum wage by one dollar an hour".
A randomized control trial is the gold standard, and the way forward seems to be more experiments like the RAND Health Insurance Experience and the Oregon Health Insurance Experiment. Even these results and their policy implications are subject to debate, so how could pure theory even get close?
However, randomized experiments in the social sciences often aren't feasible for cost or ethical reasons. So econometrics has developed tools to work on natural experiments, or even observational data, like Differences-in-Differences, Instrumental Variables, Regression Discontinuity designs.
Even a brief skim of methodological considerations in economics reveals how much uncertainty there is. I am only a fan of economics (only high school and college microeconomics) so I'm likely wrong
Department of Transportation topics for 2018: https://www.sbir.gov/node/1413255
Even if the research isn't done by government itself, it's likely the result of government grants. VC usually step in and reap profits way later
binder is kinda similar for python jupyter notebooks
https://www.digitalocean.com/products/one-click-apps/
https://blog.heroku.com/heroku-button
edit: after looking at the github, they actually have this with YunoHost (whatever that is): https://install-app.yunohost.org/?app=peertube
Think it's possible to self study the exams? Not a likely path for me, but interested in seeing if anyone has done that
It connected with what i've heard Chomsky say about trying to develop laws of physics by filming what's happening outside the window. We need to do experiments and interventions to learn the dynamics of a system
"What do you think the role is, if any, of other uses of so-called big data? [...]
NOAM CHOMSKY: It’s more complicated than that. Let’s go back to the early days of modern physics: Galileo, Newton, and so on. They did not organize data. If they had, they could never have reached the laws of nature. You couldn’t establish the law of falling bodies, what we all learn in high school, by simply accumulating data from videotapes of what’s happening outside the window. What they did was study highly idealized situations, such as balls rolling down frictionless planes. Much of what they did were actually thought experiments.
Now let’s go to linguistics. Among the interesting questions that we ask are, for example, what’s the nature of ECP violations? You can look at 10 billion articles from the Wall Street Journal, and you won’t find any examples of ECP violations. It’s an interesting theory-determined question that tells you something about the nature of language, just as rolling a ball down an inclined plane is something that tells you about the laws of nature. Scientists use data, of course. But theory-driven experimental investigation has been the nature of the sciences for the last 500 years.
In linguistics we all know that the kind of phenomena that we inquire about are often exotic. They are phenomena that almost never occur. In fact, those are the most interesting phenomena, because they lead you directly to fundamental principles. You could look at data forever, and you’d never figure out the laws, the rules, that are structure dependent. Let alone figure out why. And somehow that’s missed by the Silicon Valley approach of just studying masses of data and hoping something will come out. It doesn’t work in the sciences, and it doesn’t work here."
- https://www.rochester.edu/newscenter/conversations-on-lingui...
It is actually a really interesting subject, marketing people doing a/b tests for ads/features seem at least a little closer to the experimental ideal, not just fitting curves to data
For further reading, I'd recommend the epilogue of Casuality (Pearl 2000), it's from a 1996 lecture at UCLA:
I'll try to get my hands on human use of human beings, probably give cybernetics another run through too