alew1

Good question. Disclaimer: I’m in the lab that made Gen & was on the paper, so not impartial :)

Anglican is implemented in Clojure, and can be extended (by writing new Clojure code) to support new general-purpose inference engines. Creating those extensions requires an understanding of both the statistics and the PL concepts used in Anglican’s backend; you are essentially writing a new interpreter for arbitrary Anglican code.

Gen provides high level abstractions for writing custom inference algorithms for _specific models/problems_ (not entire general-purpose inference engines). Those abstractions don’t require reasoning about PL concepts like continuation-passing style, nor do they require the user to do any math by hand. Of course, since Gen is just Julia code, you can still reach in and implement new inference engines (just as in Anglican/Clojure) if you’re an expert. But I wouldn’t expect people who are not probabilistic programming researchers to do this (in either Anglican or Gen).

This is really cool! And awesome that (anecdotally) it seems to have been an effective tool for teaching high school students about logic and proof. Excited to dig in and learn more at some point about how this relates to formal proof assistants.

Hmm, I'm not sure I see the difference. Why is it not "symbolic"? The symbols that construct the neural network are what encodes translation invariance -- not some vector of reals.

As I was reading, I kept thinking, he should really look at The Little Typer and Pie. Then that was Section V!

(Really, many of the books in that series do this. The Reasoned Schemer for logic programming, The Little Prover for ACL2-style first-order logic.)

We need a digital representation of real numbers. One possible choice is: a real number r is represented as the source code of a function which takes a natural number n to the nth digit of r. This seems no more like “cheating” (and to me, it is less like cheating) than to represent a real number r by a line segment of length r.

I taught high school freshmen using HtDP for three years and loved it. Definitely works for high schoolers!

Or maybe more analogous to “differentiable programming” or “logic programming”: a style of programming that supports a broader range of operations than “running.”

Also, it’s not quite that you are using a DSL to encode Bayesian models traditionally represented in other ways; you are using a full-featured programming language to express a set of models larger than the set that is easy to express using existing formalisms. (That said, there are also some models that are easier to express using those other formalisms; you can think of them as being DSLs in a Turing-complete probabilistic programming language, which make it easier to express certain limited classes of programs.)

Great connection. Here's one way of looking at it: when you add stochasticity to your program, you make it easier to search the space of possible solutions (and you also relax the criteria for what it means to be "a solution").

In (deterministic) logic programming, the default search strategy is backtracking. We search the space of "inputs that could have produced this output" exhaustively. Each set of inputs we try either works or doesn't, and doesn't tell us anything about whether similar inputs will behave similarly.

In probabilistic programming, we can use the information we have about probability densities to tell whether we're on the right track; as we search, we can tell if we're getting "warmer" or "colder." Suppose I have the simple probabilistic program:

   def f():
     a = normal(0, 1) # mean 0, stddev 1
     b = normal(0, 1)
     c = normal(a+b, 0.1)
     return c

In this generative model, I choose two random numbers, add them together, and then pick a third number very close to their sum. This could be thought of as a probabilistic version of the deterministic program f(a, b) = a+b. If I want to "invert" it, and condition on c=2, say, I can walk around the space of a priori likely (a, b) values, "scoring" them based on how likely "c=2" would have been given the particular (a, b) under consideration. The tight bell curve around the value c=2 can be seen as an implicit "loss function," telling us how good or bad the current solution is. This is very different from a logic programming approach, which would be able to exactly invert the + operation; but as you pointed out, it's nontrivial to formulate `+` correctly as a logic program, whereas here, we are free to use an imperative implementation of `+` (or much more complex deterministic operations, like a 3D renderer).

There is research in probabilistic logic programming languages, which can be seen as combining some of the strengths of both paradigms, though that's not covered by the introduction linked in the OP.

"Updating priors" is quite complex, and arguably describes every interesting task you'd want to do as a learner, reasoner, analyst, etc. One view of probabilistic programming languages is that they allow you to express your priors—i.e., uncertain beliefs about the world—as programs, and help you to answer all sorts of interesting questions in the contexts of those programs.

As a concrete example, you might write a program expressing your beliefs about how the scenes you see come to be: some random objects are placed at some random positions, lighting sources are placed, and then a rendering algorithm converts that scene description into the light intensities perceived by your eye. If you write this up in a probabilistic programming language, you can then ask questions like: supposing I see this image, what are some likely scenes (positions and shapes of objects) that might have produced it? How does that distribution over scenes change if I know that the light source was here, or that there was a small chair there, or that my vision isn't that great and I should trust the image's pixels a little less? Section 1.3 of the introduction linked above lists many more applications for which probabilistic programming turns out to be useful: breaking CAPTCHAs, constrained procedural graphics, automatic program induction/synthesis, recursive multi-agent reasoning... and, as the authors write at the end of this section, "the list goes on."

Of course, solving the inference task in full generality, for any program and query that the user might write, is very difficult; you can design pathological cases where it boils down to solving the halting problem, and is therefore uncomputable. Different languages take different approaches to addressing this difficulty. They might limit you to writing certain kinds of programs for which a limited class of inference algorithms can always be made efficient; they might try to plan a fancy inference algorithm based on the specifics of your model; they might just expect you to learn which programs will be tractable and which won't; or they might provide abstractions that make it possible for you to safely experiment with writing new inference algorithms (or combinations of existing ones) specialized for your use case. Probabilistic programming is still a young field and this is an area of active research.

It is, but the research that happens there is still funded in large part by public grants.

Did the article change? The rest of the “hard to analyze” sentence you quoted above is “linear systems respond proportionally to changes in variables, whereas nonlinear systems have more complicated relationships.“

I’ve seen 30 since signing up at the end of November. Not sure how typical that is, but my friends with MoviePass also see a bunch!

I highly doubt they spent weeks training on the toy example. More like five minutes, probably. Again, that the weights learned (quickly) for the toy example can also be set by hand is not surprising and is evidence of a good toy example. The paper’s main result is not the toy example, but the real experiments (for which I doubt you could hand-code the network weights).

This doesn’t seem like a particularly fair criticism.

1. As others have pointed out, the ImageNet experiment is presented as evidence that (as you’d expect) adding coordinate channels doesn’t affect performance on image classification tasks. That’s a good “sanity check” experiment to have done.

2. The paper proposes a simple idea, and it may not have been necessary to give it a whole new name (CoordConv). But if you’d asked me if I thought that adding coordinate data to the input would have led to significantly better object detection, I wouldn’t have known the answer, so the results of their experiments—that it does help on tasks like object detection—is not trivial. Not only that—a lot of people have tried to do object detection, and yet nobody has reported adding input channels for storing coordinates before. A lot of ideas seem simple after someone thinks of them.

3. Toy examples are useful for testing intuition (and building intuition about why this trick may be helpful and for what kinds of tasks). The fact that we can easily imagine what sorts of weights we’d expect the network to learn is one of the things that makes it a good toy example. (Of course, the paper wouldn’t be worth publishing if it only had the toy example.)

Question: in the list comprehension example, is `x` defined outside the loop comprehension? Will it overwrite an existing `x`? Or is it like `z` and only visible inside the loop comprehension? The latter would make more sense, I think.

I’ve spent the last three years (straight out of college) teaching high school computer science. I’m headed to graduate school in the fall, but I’ve loved teaching, and think it’s a shame that fewer CS people go into the field. (Pay isn’t as good as in tech, obviously—I think teachers at my school generally make in the $50-80k range—but from a meaningful-work perspective, things are so much better!) A few points:

* My colleagues and I sometimes gripe about parts of the job, but we all agree we’re never bored. We are shifting to a new environment (new classroom) every forty minutes; there’s a ton of meaningful human interaction; and the life of the school depends so much on the particular kids in your classes / the whole school each year. And the pedagogical questions are really interesting, too (more on that in a moment).

* I’m lucky to teach at a school that supports a four-year computer science curriculum and has given me the autonomy to to do whatever I’d like with it. There are plenty of areas I didn’t study in my undergrad CS degree (networking & concurrency, operating systems, distributed computing/cryptocurrencies, combinators and monads) that I’ve had the chance to dig into (read about, plan lessons about, develop assignments and demos about) in teaching my more advanced students.

* CS pedagogy nationally is in a strange place. There is a political push (and a good deal of funding) to get CS in every K-12 school. But very few people trained in CS go into teaching, and there is little consensus about how it should be taught or what curriculum should look like. Companies like College Board, Code.org, as well as corporations like Apple are stepping in to provide curriculum and training to teachers, but these curricula (in my opinion) leave something to be desired, partly because these companies have other interests. For example, Apple wants everyone coding Swift on iPads, and CollegeBoard wants people paying for AP exams. Things are in such a state of infancy now, though, that by getting involved in the community (going to conferences, blogging, creating high-quality materials, setting and example in your classroom), you have a chance to join the debate & help build a better consensus around how CS can/should be taught. There are a number of people doing awesome work already, and several really stellar CS programs at high schools across the country. But we need more!

* It’s really amazing having diverse colleagues whose interests span many disciplines: I have colleagues who have PhDs in physics and English and Classics, who direct plays, translate poetry, write novels... In the tech industry, the majority is young, male, white or Asian, and consumes a similar “cultural diet” of news sources, books, movies, hobbies, etc. It’s nice to be somewhere a little less homogenous.

After playing with this, I can make my brain hear either — and even in rhythmic patterns (Yanny, Laurel, Yanny, Laurel, Yanny, Yanny, Yanny, Laurel). It’s a neat reminder of how much “preprocessing” our brains do before we experience a sound (filtering out what are perceived to be the irrelevant frequencies).

Good response by aphyr. If I’m reading right, the original poster left because LLVM partnered with an organization designed to create internship opportunities for groups traditionally underrepresented in tech. In particular, LLVM agreed to take on one intern from that organization.

Everyone is entitled to their own reasons for leaving. But I do find it frustrating that the left gets saddled with names like “snowflake” when such small & reasonable actions routinely ruffle feathers on the right.

I think here they just meant, show that no rational number is the square root of 2. Not rational = not expressible as a ratio. You assume that sqrt(2) = p/q for some integers p and q, then derive a contradiction.

It seems likely that the two years of effort were essential in getting him to the point where he could have the breakthrough.

An analogy that is used in the YouTube debate linked from the article is that of a scorekeeper and players of a game. The government is the scorekeeper, and operates according to rules we collectively agree on. Those rules can incentivize certain kinds of behavior. The points that the scorekeeper tracks are "money."

If I, as the scorekeeper, decide to award 100 points to the person who accomplishes X task, I don't "lose" 100 points, nor do I have to "get them" from anywhere. I don't "have" a fixed number of points to give out. I create them by awarding them, and I destroy them by collecting them or allowing you to trade them for things. Privately, you (the players) may also trade points among each other; in this, you are constrained by traditional "budget" thinking (how many points am I getting, and how many can I afford to spend).

For your "society of 10 people," imagine a group of 10 roommates who institute a point system to incentivize good behavior at home. Every month, each roommate has to "pay taxes" of some number of points. When we collectively choose to "pay for public services" with points, what we're doing is setting up rules like "whoever cleans the bathroom gets two points" and "taking out the trash is worth one point". Some people don't want to do chores; perhaps you prefer to create some piece of software for your roommate in exchange for some of his points. If there's ever unemployment--people who can't make enough points, because all the chores are already being done and no one wants to give up their (scarce) points for private services--this indicates that we could have a better "point policy," either by paying for more chores (public services), which the "unemployed" roommates can use to earn their points (all while providing useful services for the entire apartment), or by cutting taxes for those who just barely have enough points now, so that it becomes more reasonable to pay some points to the unemployed for non-essential goods/services.

Bootstrap is working on a high school physics curriculum for ninth graders, which integrates functional programming as a fundamental component. They only have a landing page up at the moment [0], but I attended a very cool workshop they gave this summer. They’ve designed the programming language to avoid any “incidental complexity”—Physics teachers only have to teach the CS that’s absolutely necessary for the physics education goals they’re trying to achieve.

[0] http://www.bootstrapworld.org/materials/spring2018/courses/p...

Makes sense! I tend to take the "wrapping your point in uncertainty" approach, without all the ego-stroking. Nothing against ego-stroking; it just seems hard to make it feel genuine enough for it to work. Most comments (mine included!) contain little enough insight that any praise more intense than "good point!" or "agreed!" or "that's totally right!" feels somewhat disingenuous.

I wonder if it could be done in a less grating way? I agree with the general idea of affirming the good in others' viewpoints, but I do find GP's phrasing patronizing. Why not something like:

I agree, and I think there'd be a lot of developers who feel the same way (re: wanting to help, valuing freedom of access to scientific information). I wonder, though, if there's really that much we can do as developers. There are of course some vulnerabilities that can be mitigated with good code on the server, but most attacks are impossible to prevent without the resources that a company like CloudFlare can marshal. Unfortunately, the state of the art for responding to DDoS attacks requires a big network, a lot of bandwidth, and active management.

Not sure what his source is, but from the article:

Andrew Clement, a computer scientist and expert in surveillance studies, has been mapping the NSA’s warrantless wiretapping activities since before Snowden’s disclosures. He strongly believes the agency would not be restrained in their uses of speaker recognition on U.S. citizens. The agency has often chosen to classify all of the information collected up until the point that a human analyst listens to it or reads it as metadata, he explained. “That’s just a huge loophole,” he said. “It appears that anything they can derive algorithmically from content they would classify simply as metadata.”

Hi all! Coded this up over the holidays, and curious to hear thoughts & suggestions!

In order to find the "next word" w to say, the program uses a normalized set of word vectors and tries to maximize the product of the cosine similarities between w and each of the two context words. (The idea to maximize this product, rather than finding the word closest to the mean of the context words, comes from the paper Linguistic Regularities in Sparse and Explicit Word Representations [1].) Possible answers are then filtered based on some simple heuristics (avoid saying a word that's been said before, that's too infrequent or too frequent, etc.).

It's come up with some fun answers — van + agree --> Accord, facebook + sore --> viral, cursive + movie --> script — but can also sometimes be frustrating. Any thoughts on how to make it more "human-like" would be appreciated :)

There's also a discussion on Reddit [2].

[1] http://www.aclweb.org/anthology/W14-1618

[2] https://www.reddit.com/r/MachineLearning/comments/7o8q8v/p_c...

Whether or not the economics are right, this author doesn’t understand the technology:

> To explain: essentially, “blocks” are segregated, vast bundles of data in permanent communication with each other so that each block knows what the content is in the rest of the chain. However, only the owner of a particular block has the digital key to access it.

I know journalists need to simplify the technology, but those simplifications should basically be correct. This is just... wrong.

Thanks. I can see that this is a useful experiment, and I'm glad that the authors did it. It's just not clear to me that their model is new or needs a special name; isn't this single-stream approach just the simplest interpretation of "using a WGAN to generate video" (just generate all the pixel data)? If anything, the two-stream architecture you link to seems like the special case/new idea.

Am I missing something in the paper, or is iVGAN actually just a vanilla Wasserstein GAN? Is the contribution of the paper “we tried GAN on video and it worked”? That’s still a useful result, but what justifies the new iVGAN name? Perhaps the specific network architecture they’re using?

How do blind people navigate these spaces?