Scipy - the embarrassing way to code | Hacker News Reader

Scipy - the embarrassing way to code

vetta.org

66 pointsdood37 comments

I've done lots of Matlab development, and this is too true.

Python is going to replace Matlab, but the pattern in the same.

People aren't really used to thinking about matrices and vectors as primitive types and operations.

By the way, this is an excellent resource for learning Matlab. http://www.andrew.cmu.edu/course/16-720/handouts.html

Look at the "working with images" section. If you manually touch every pixel in an image, you're doing something wrong. You code will be 5X as long, and 100X as slow.

The bottleneck in writing code isn’t in the writing of the code, it’s in understanding and conceptualising what needs to be done.

Good observation, succinctly stated. Most of my days are spent trying to figure out what I need to do, not how to do it.

Exception: interfacing with 3rd-party libraries.

This is exactly why I have urged people to learn Matlab (on this forum and others). Specifically, I'm sick of seeing hundreds of lines of code written in Java that are essentially a poor (slow) implementation of a one liner in Matlab that uses a 20 year old fortran library. I have to go back all of 8 days to find the following quote in response to my plea:

"...and I question the usefulness of learning MATLAB unless you actually need it."

The problem is, most people don't realize they need it and just plod along writing hundreds of lines of python, perl, ruby, java, etc. implementing a decade old numerical computation.

When you write a lot of matlab it becomes a different way of thinking. Instead of the for loops multiple levels deep you can write it all as operations to multi-dimensional matrices.

I would agree that many problems people spend time on can be found as single commands in matlab.

The comments here on Matlab have me intrigued. I'm ignorant and have a few newb questions. How does it compare to the APL family of vector languages (J/K/Q)? How does it compare to Mathematica? What roughly are the licensing terms for commercial use (i.e. if you want to build a product with it)?

in terms of interface, it has more in common with mathematica (both good and bad) than do the APL style languages, basically the language semantics are C style with dynamic shape checking for array operations and a nice syntax for many different linear algebraic operations.

It'd be nice if there was a nice type system to aid in getting comfortable with matlab's semantics, in my opinion

it's an algol language with array manipulation. APL languages start with array manipulation as a primitive in the design of the language, so they are much better at these sorts of things

Matlab is for engineers, Mathematica is for scientists. Matlab is designed for speedy matrixy stuff. Mathematica uses a lisp-like structure 'replacement' pattern matching evaluation scheme, and intrinsically supports symbolic mathematics and is the the only language i would really call multiparadigmatic

Thanks. My project is calculation-intensive and performance is important. I would love to use kdb. Arthur Whitney's work is stunning. Unfortunately, their licenses are draconian. Can you offer any insight about these various options from the standpoint of building a commercial product on top of them?

truthfully, i have no idea. it's certainly unfortunate that these languages/environments are not widely usable. i don't know of a more powerful scratchpad than Mathematica

have you looked at R? it's an open source matlab. though i don't know what kind of database options it might have

R isn't "an open source matlab". R is an open source implementation of S-plus. If anything is the open source Matlab it is octave. However, I agree they can be used interchangeably for solving many problems.

Perhaps I can help, but first. What calculations are you doing (e.g. are they all linear algebra?) What language is it currently written in?

Think of it as similar to a spreadsheet. The calculations are mostly linear algebra and are represented visually in 2d arrays.

What language is it currently written in

Common Lisp (at least, the calculation part). It's a great language for developing the product, but we don't want to greenspun our own vector-based calculation engine if we don't have to.

I would suggest looking into NumPy/SciPy instead of Matlab. Matlab has a very good library for numerical computation but in my experience, Matlab is absolutely terrible as a language for anything except matrix stuff.

NumPy gives you a good matrix syntax while building on an excellent general purpose language. Personally I felt NumPy is a lot cleaner and Python a lot more fun than Matlab though Matlab has many many "toolboxes"/libraries not in NumPy.

I second this sentiment. The difference in natural expressiveness is almost painful

matlab costs boatloads of money for commercial stuff. At least from my perspective.

I wish there were an open alternative to matlab that was fast. Matlab is kinda fast if you do exactly the right thing, but its interpretedness gets in the way. Most of the people I know doing computational stuff write everything in c using some outside libraries to speed things up. Often they just code up all their own algorithms from scratch. Its lame, but they usually halve the computation time. I could imagine that SciPy and NumPy is much much faster than Matlab.

Matlab is crazy easy to code for though. Its really handy for prototyping.

> I could imagine that SciPy and NumPy is much much faster than Matlab.

SciPy and NumPy tend to be just about as fast as Matlab, when used in their default manner. When you drop into inline Blitz/C for your inner loops using Scipy though, you can achieve orders of magnitude improvements.

Check it out: http://www.scipy.org/PerformancePython

I think the deeper idea to take away from this is that knowing more theory is always a good thing

which begs the question of why it seems that even in pretty technical domains, theres still an aversion to aggressive use of theory tools?

What does this have to do with theory?

if you don't know any of the theory behind a problem, you don't even necessarily know if theres a solution thats useful in practice.

A good example in Computer Science is the various reductions (efficient or not )between different problems, where you relate the difficulty of the problem to any other computational problem which you find you're able to encode within the structure of the orginal problem.

The article seems to have two messages: one that it's better to fully comprehend the problem before you start coding; the other that learning a powerful library will save you pages of code. Both well rehearsed truths of computer programming.

As for myself I often find that I have to fully explore the problem on my own terms before I can maturely optimize my source.

it's better to fully comprehend the problem before you start coding

Often the best way to understand a problem is to try to implement a solution.

That's a very true observation mhartl and I try to touch on it in my last sentence although it is enigmatically worded as I was trying to sneak a "Premature optimization is the root of all evil" reference into it. Alas not all wordplay is destined for upmods!

Would it be advisable to try to implement a solution on a chalkboard first? Mathematical notation and diagrams can be amazingly succinct.

give J (an APL descendant) a try, it has some good tutorial thingies that appear right in the console thingie:

http://www.jsoftware.com/stable.htm

and here's a post about it:

http://www.squidoo.com/jprogramming

The introduction to J that peaked my interest: http://prog21.dadgum.com/14.html

Haven't done anything with it yet, but it's on my radar.

> give J (an APL descendant) a try,

J is not open source. Might be a factor to consider.

From the J site,

"Jsoftware licenses source for all the Jsoftware binaries. There are several factors in determining the cost of source licenses. Primary is the source itself: J Engine (portable C); Windows GUI support (C++); and Unix GUI support (Java). Secondary is the scope of use: internal; distributed products; and competitive (to Jsoftware) products. And finally the platforms: Windows; Unix; PocketPC; etc. The price range is from $10,000 to $400,000. Regular updates to our current source levels are available for separate update fees."

careful, not free: http://www.jsoftware.com/source.htm

I can't believe nobody has mentioned Sage Math (http://www.sagemath.org/).

It's sort of a set union of every good free math program.

I've noticed a similar conceptual pattern while investigating meta-programming. I think there's something fundamental going on here. Something that conventional programming constantly misses. Something remarkably simple, consistent and powerful. It's where code is data along a continuous path of transformations.

Read Godel Escher Bach. Given the comment above, you'll like it.

This is a familiar story, regardless of technology.

The more you work, the less you appear to have.

The efficiency paradox.

"Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away." -Antoine de Saint Exupéry

Bulletin board material.

I have it posted at my desk in a nameplate frame. I try to adhere to it. I often fail.

Great article. Learn the hard way and it'll stick. Plus it's fun.

Reply on news.ycombinator.com