Blaze: Next Generation NumPy
speakerdeck.com
speakerdeck.com
What I don't want: Just slides no audio. I also don't want video of the talk uploaded to youtube, because sometimes the slides aren't visible in the video.
My talks involve slides that make no sense without the audio. If there is video capture of the talk, it's nice to include some of the video too. But I guess I would be content with just audio + slides.
I do not want to have to do complicated video editing; I want an existing solution for this problem.
[edit: clarifications]
Our site Lanyrd encourages the creation of a page-per-talk which can embed slides, video and other talk materials - here's an example: http://lanyrd.com/2012/jquery-san-francisco-conference/swmdt...
I love this, and every time I am pointed to slideshare or speakerdeck I hope they have implemented something like this :(
Slide hosting is eaassyy, just post the file and you're done. Recording and then syncing adds about an order of magnitude amount of work for the poster.
Both projects are designed to work very well with each other, although both have applicability outside the region of integration. Numba will compile regular Python functions that consume Numpy arrays, and will not require Blaze. Blaze will produce lazily-evaluated expression graphs and dataflows on disjoint and out-of-core arrays, and can evaluate these efficiently without Numba. (This is what the Array VM is all about; you can think of it as an improved Numexpr.)
As I understand it from a quick non-expert skim (please correct!): the idea is to use a fairly flexible and expressive metadata language under the hood to specify the physical memory layout of a multi-dimensional array -- more expressive than numpy's existing support for strides. Sounds like this will allow arrays to be reshaped and joined together without copying, support arrays larger than core memory, etc.
I wonder if the language is too powerful, if this could make memory management (e.g.: can I free this contiguous chunk of memory?) quite hard to reason about.
Also: seems like this makes it easier to push the cost of array layout manipulations in a somewhat lazy fashion from the producer onto the consumer of results. At which point does one decide that traversal is now excessively complicated and the thing should be copied into a flat layout. Especially since this might affect one's ability to use certain BLAS routines, take advantage of certain hardware optimisations etc. Is the idea that a compiler makes this decision for you, or you make the decision explicitly or implicitly?
Interesting stuff anyway, look forward to hearing more about it!
No, but those will be forthcoming as we build out more of it. We are targeting an end-of-November preview release, as the slides indicate, and it will give people a much more concrete idea of the things Blaze can do.
> the idea is to use a fairly flexible and expressive metadata language under the hood to specify the physical memory layout of a multi-dimensional array
Yep, that's exactly right. The challenge is to constrain this is that it is not too general, i.e. enters into PhD research land, but still useful for a large number of use cases.
One of the core ideas is to more coherently represent the metadata about location, locality/compute affinity, and index spaces. Numpy has the beginnings of some of these ideas in its various flags and View object semantics, but it's all stuck in its very rigid, contiguous-memory origins.
> At which point does one decide that traversal is now excessively complicated and the thing should be copied into a flat layout.
This would be a great Master's research project. :-) The hope and the belief is that one can punt on resolving this in generality, and still solve a number of interesting, concrete problems in an efficient way.
Incidentally I found the use of topological manifold terminology (atlas of coordinate charts, homeomorphisms, ...) in the slides quite intriguing, are there topological connections here or is that more just an analogy?
Good work, Continuum Analytics!
chatted with stephen about Repa and hpc code engineering the evening before the talk, its all haskell ideas ported to python land under the covers :)
(as the slide deck admits!)
Well, that's not entirely true. It's based on many of the same ideas which underlie some of the good Haskell ideas, and have a natural expression in Haskell because it's a type algebra on index spaces.
An incomplete list of the inspirations: Paralation Lisp, Merrimac, APL / J / K, Manticore, FORTH, Chapel & ZPL (and VCODE), SISAL, OLAP & MDX, LINQ, various PGAS approaches, etc., etc.