Hacker News Reader

Comment by Xcelerate | Hacker News Reader

I do scientific computing, and Python is one language I never actually got around to learning for some reason. However, as a long-time hobby, I do have an interest in programming languages so I like exploring things like Haskell, Clojure, Lisp, etc.

One language I'm really excited about for scientific computing though is Julia. From a language-design perspective, it's beautiful. It was actually thought out rather than kludged together. I've been trying to gradually use it more and more for my research, but the only problem I've found so far is the large mental context-switch I make going from my usual languages to Julia. It's hard to tell what Julia code will be the most performant because there's many ways of doing the same task. I saw someone in the comments on this page mention that you can hand-tune the LLVM generated output within the REPL itself. I imagine this would be very useful if I can get around to learning it (anyone know a good tutorial?)

View on HN

The big question for me is whether Julia will be able to maintain its "purity" as it gains adoption.

R probably started out "beautiful" and "thought out" but has lost that edge with years of community driven development. It's also what make it so damn useful -- you can pretty much find anything on CRAN, often multiple implementations of it.

Sorry, no. Even the core S language would not be described by very many users as "beautiful" according to modern standards.

R is actually one of the most pure languages out there; it basically says "I have vectors; they can have missing values, be nested, and can have other vectors as attributes. And I have functions with lexical scoping. Now go and build the rest as you like." So people did this, one better, one worse -- but the core and beautiful stuff here is that all those approaches will work together and just do the job.

Once all of Bioconductor is ported to Julia, I will probably shed some joyous tears.

R is really nice for some things, because of how easy it is to work with table-like data, and operations like applying a function over a vector or a matrix, etc are cake. However, it's also slow, leaky (and sometime seems to use more memory after gc()), and often difficult to debug because so many things fail silently. If it wasn't for so many standard bioinformatics packages being available only for R, I'd probably be using Python. Once we stop running microarrays, I'll probably be using R much less.

python is a deliberately-straightforward language. i don't see how anyone couldn't become highly proficient in it after writing one or two scripts

The straightforward design makes it fast to go from zero to having a working proficiency. But I'm not sure about "highly proficient": even experienced Pythoners get tripped up by things such as mutable function arguments, and it's often not clear why some simple-looking code is running slowly, and how to speed it up.

Deliberately-straightforward -- agreed. But "highly proficient" after writing one or two scripts? That's quite a stretch.

For instance, one of the questions I give in phone screens is for the candidate to write a program to count the number of occurrences of unique words in a text file. The "after writing one or two Python scripts" approach is something like this:

    counts = {}
    f = open('test.txt')
    lines = f.read().split('\n')
    for line in lines:
        for word in line.split(' '):
            if word:
                word = word.lower()
                if word in counts.keys():
                    counts[word] += 1
                else:
                    counts[word] = 1
    f.close()
    count_items = [(count, word) for word, count in counts.items()]
    count_items.sort()
    for count, word in reversed(count_items):
        print word, count

Whereas the "highly proficient" (and much simpler and more Pythonic) approach might look something like this:

    import collections
    counts = collections.Counter()
    with open('test.txt') as f:
        for line in f:
            for word in line.lower().split():
                counts[word] += 1
    for word, count in counts.most_common():
        print word, count

Actually that's not 'highly proficient'. This is:

    def read_words(words_file):
        return [word for line in open(words_file, 'r') for word in line.split()]
    len(set(read_words('test.txt')))

R version for comparison (;

    length(unique(scan('test.txt',character(),sep=" ")))

Very clean. 1 small change to make it count of frequencies-

table(scan('test.txt',character(),sep=" "))

smashing tons of crap together isn't necessarily "highly proficient". In some cases it makes things harder to read and/or harder to maintain and many times certainly harder to edit.

True that. But there is also another side to this story, where such nested calls translates naturally to what we are thinking, i.e.

length(unique(text)) comes from the thought "How many(length()) unique (unique()) words are there in this text(text)?"

Don't know when set comprehensions and resource handlers were introduced so that might not run everywhere

    def read_words(words_file):
        with open(words_file, 'r') as f:
            return {word for line in f for word in line.split()}

len(set(open("test.txt").read().split()))

Oh, I like this one. Not quite what OP asked, but very clean.

GP asked for counts, not uniques.

Oops, you're right.

    words = open("test.txt").read().split()
    [(k, len(list(g))) for k, g in groupby(words, lambda x: x)]

Doesn't that require an "import itertools"?

Yup. Sorry.

Oops, should have been this:

    words = open("test.txt").read().split()
    [(k, len(list(g))) for k, g in groupby(words, lambda x: x)]

except that's wrong, all you're doing is counting the number of unique words, and you didn't even consider Foo and foo as the same in your example. Part of proficiency is understanding the problem.

  for word, count in Counter(open('test.txt').read().lower().split()).most_common():
      print word, count

I like this one. Though I would do it like this to keep the lines under 80 characters:

    words=open('test.txt').read().lower().split()
    for word, count in Counter(words).most_common():
        print word, count

(edited per child comment)

Yeah, those are nice -- and may actually be more efficient on smaller files, as you're only doing the lower() once on a big string. However, for big files you don't necessarily want to read the whole thing in at once.

One nitpick: it's Pythonic (I think) to just name the list of words "words" rather than "word_list".

Yes that's a classic tradeoff, a proficient programmer will have to pick one.

Personally I always read entire files into memory first unless I have reason to believe memory will be an issue or need to program defensively against malicious/careless input. The code is always much cleaner and easier to read and if you need to do a second pass on the data you don't need to re-read it from disk.

My solution:

    lines = [line for line in open("bible.txt")]
    words = [word for line in lines for word in line.split()]
    counts = {word:0 for word in words}
    for word in words:
        counts[word] += 1

No imports needed. Linear time. A bit inefficient in the dictionary comprehension, but easy to read.

The "lines=" and "words=" can be compressed into one line, but I figure this is a bit easier to read for people who aren't familiar with nested list comprehensions.

> A bit inefficient in the dictionary comprehension, but easy to read.

If you allow one import to sneak in, I think a `defaultdict` would do nicely. :-)

I'm not too familiar with Python, but this is case sensitive unlike benhoyt's examples. Would this be case insensitive?

    lines = [line for line in open("bible.txt")]
    words = [word.lower() for line in lines for word in line.split()]
    counts = {word:0 for word in words}
    for word in words:
        counts[word] += 1

yes!

How does this type of question go during a phone screen? Does the candidate send the code to you afterward?

that first version is hilarious. i'd consider it hideous at any proficiency level in any language :D

I'd like to jump in with a little R here- it not all that difficult in "that" either!

open(con <- file('text.txt'))

text = readLines(con, n= -1L) # n is number of lines to read, -1L means read all of it

words = strsplit(text,split = " ")

counts = table(unlist(words))

I put this in because the good thing about R is that it provides functions for many such mathematical operations. And along with this, I'll say something any self-respecting pythoner will know- Less is better than more.

Here's mine for what it's worth, since this is one of the Google python assignments. Admittedly, i'm not very proficient at all.

   def get_file_words(filename):
     file = open(filename, 'rU')
     words = {}
     for line in file:
       for word in [word.lower() for word in line.split()]:
         if not word in words.keys():
           words[word] = 1
         else: words[word]+=1
     return words


   def print_words(filename):
     wordcount = get_file_words(filename)
     for word in wordcount:
       print word, wordcount[word]

What are your thoughts on Clojure's suitability as a go-to language for scientific computing (except in lower-level, high-performance scenarios that might recommend Julia)? I think it has serious potential in this field. It's not there yet, but it's getting there, and the core.matrix standardization helps a lot.

Its functional. Looking at the way bioinformaticians code, that will make it too complicated for many of them. SQL seems too much for many of them.

I agree with collyw; Lisp is too much for most scientists, who just care about getting their research done and can't be bother to learn all this weird FP/paren stuff [1]. Supplying an obvious, familiar syntax that looks like math written on paper for, e.g., matrix operations, is critical. This is actually one of the core tensions in Julia development imho: balancing having a sane, well designed language (from a programmer's perspective) vs. having a tool that allows scientists to quickly and easily crank out results.

[1]: Note; I am not insinuating that Lisp is bad, I like it personally. Just relaying the response you will get from most practicing scientists who are not trained as programmers.

Just relaying the response you will get from most practicing scientists who are not trained as programmers.

Is it that hard of a gap to cross? I was a math major who took a couple CS courses, and wouldn't say I was "trained as a programmer". I found Scheme pretty easy to get from the go.

I tend to think that anyone who can get a PhD in the physical sciences can become a half-decent programmer-- if the desire is there, the intelligence and work ethic being established (one hopes, at least) by the degree.

It's less a matter of ability and more a matter of having the time/motivation. If you took a bunch of practicing natural scientists and/or engineers and forced them somehow to enroll in a course teaching Scheme/Clojure/etc., would many of them do well? Almost certainly. If you gave them the choice between a Lisp and something like Julia or MATLAB to use in their everyday work when they need to do some computation? They'll probably choose the later because it's easy and familiar and doesn't have a huge up-front time investment; scientists tend to be very busy people.

Reply on news.ycombinator.com