Now Serving: Full-Text Sampler in IPython Notebook Format | Hacker News Reader

Now Serving: Full-Text Sampler in IPython Notebook Format

miningthesocialweb.com

42 pointsptwobrussell27 comments

That's really amazing, I think I can speek for the all IPython team by saying that we are Really happy with this.

We've cross a few authors that were writing books for O’Reilly with IPython Notebooks in the last few month and we have plan to help convert those to (and from) many format. We would be extremely happy to work with people from O'Reilly to simplify the conversion process.

We also have plan to integrate with publication in academic world (see https://github.com/ipython/ipython/issues/4119) but we really need more people involved than the few of us (Core Team).

Even if IPython seem like a big complicated project, moving forward in area like document conversion is not that hard, but just needs lots of eyes and tester to catch all cases.

As for people wanting to convert notebook to other format than those currently supported, it is perfectly possible using nbconvert with template. We can spit the TeX that is just between begin/end document so you can include in your preferred dissertation template. It should be pretty easy to also just write a converter that export to asciidoc or doocbook.

For those interested of the idea behing how nbconvert I wrote this quite some time ago: http://nbviewer.ipython.org/urls/raw.github.com/Carreau/post... and https://github.com/ipython/ipython-in-depth Offer some example too.

Hope this will push some people to contribute, we'll always be happy do guide you in your first step.

I'm working on an O'Reilly book using IPython Notebook, I'm basing my workflow on Olivier Grisel's script (with a couple of tiny fixes): https://github.com/ianozsvald/ipynbhelper which extracts the Notebook's 'markdown' blocks (which contain asciidoc which obviously won't render in the browser) & code blocks, these get exported as asciidoc for O'Reilly.

It is clunky but I'm hoping we'll get better control as nbconvert evolves, so we're experimenting with this approach.

Most of the code examples are not 'live', they're pasted in along with analysis results (the book is about high performance and parallel computing: http://shop.oreilly.com/product/0636920028963.do ), as lots of the examples are best run from a fresh VM.

It is a very practical idea. How many times do you want to test an example? With pdf and other formats you could make a copy-paste but many things could go wrong. But the really important improvement is the use of Vagrant and a virtual machine to give an accurate workspace.

+1 mherradora - I got the first version right after twitter changed the API >_< and busted one of the examples in chapter 1. Not anymore with Mining the Social Web 2e :). Bonus style points for Vagrant so that the author doesn't have to host (i.e. pay for) a VM and the massive bandwidth.

I purchased Mining the Social Web, 2nd Edition (O'Reilly, 2013) a few weeks ago and have been loving the IPython Notebook setup they're using to distribute the code examples. Anyone can clone the following git repo and play around with the code:

https://github.com/ptwobrussell/Mining-the-Social-Web-2nd-Ed...

I work with scientists that love the IPython Notebook technology. Some claim the IP[y]: Notebook to be the best thing since the Mosaic web browser and the most important development in scientific computing in a decade. They are replacing a variety of tools (Perl, Matlab, R, etc.) with the notebook technology. I tend to agree, it is a revolutionary technology and the idea of executable papers is tantalizing. But I also see some limitations. Setting up a Python environment with all the necessary libraries can be a real pain in the neck even with technologies like pip. Also taking advantage of multicore parallelism is not trivial because of the Python Global Interpreter Lock. I have also worked in JVM environments where parallel computing is becoming significantly easier and I don't see that happening in Python anytime soon. I would love to be proven wrong, of course.

I would totally agree on the first point about how much of a burden it can be to get any moderately complex Python environment up and running (whether IPython Notebook is part of the install or not.) Unfortunately, it's just not as easy as it should be yet.

IMHO, this is what makes a duo like Vagrant + IPython Notebook so appropriate for creating a minimum barrier to entry. In other words, what you really need is a VM that has IPython Notebook and all of the necessary dependencies installed that you just start up as a guest machine. Then, on your host, you point your browser to http://localhost:8888 and get to work with the the ipynb UI.

The approach I described is exactly what I did with Mining the Social Web 2E, and I elaborated on it a bit here: http://miningthesocialweb.com/2013/08/24/reflections-on-auth...

The cool thing about the way Vagrant works is that you can run a VM locally or you can run it in the cloud just as easily with the appropriate plugin.

There's a video that demonstrates how easy it can be to use Vagrant and IPython Notebook to get started here - https://vimeo.com/72383764

By the way, I really like the metaphor of describing IPython Notebook as "executable paper" - that's clever and concise.

It should be noted that the IPython Notebook is being developed to support multiple backends. People are working on backends for Haskell[1], Scala [2], and Julia[3]. I'd love to see a Clojure backend.

All of this does not obviate the challenges of setting up IPython, but that could also be (somewhat)resolved via cloud services[4][5][6].

[1] http://gibiansky.github.io/IHaskell/

[2] https://github.com/mattpap/IScala

[3] https://github.com/JuliaLang/IJulia.jl

[4] http://www.windowsazure.com/en-us/develop/python/tutorials/i...

[5] https://www.wakari.io/

[6] http://blog.picloud.com/2012/12/23/introducing-the-picloud-n...

$1.15M (plus my opinion) says that IPython is the bee's knees: http://ipython.org/sloan-grant.html

YES!

I would also love to see a publishing workflow that could render IPython Notebooks for print on dead trees.

The 1.0.0 version started bundling nbconvert, which does a pretty good job of exporting to html, pdf, latex, etc. Have you tried it out yet?

I haven't started meaningfully playing with nbconvert yet, but I think one of the really promising features are the nbconvert templates[1]. From what I gather you can make custom templates with jinja and invoke them on the command line. It would be interesting to create a docbook, msword, specifically formatted pdf templates which would allow IPython Notebooks to flow into existing print publication workflows (if I recall O'Reilly uses docbook, yes?). I haven't been able to find documentation about the nbconvert templates so I'm dragging this out of memory from an email or a tweet I saw once upon a time.

One thing I'm hoping to figure out is a way to transform IPython Notebooks into PDFs that satisfy my university's dissertation formatting guidelines (one inch margins, 12pt times, etc).

[1] https://github.com/ipython/ipython/tree/master/IPython/nbcon...

Wow, you definitely have some great ideas here, and given the enthusiasm about ipynb, I would not be surprised to see these things happen in the next 6-12 months.

O'Reilly does use docbook internally as part of the toolchain, but asciidoc is coming into style as well. I've done most of my work in docbook with a docbook editor, but did some of my latest boook in asciidoc, which was much easier. Just use a standard text editor of choice (like Vim) and get right to it. Much easier for me, anyway.

One thing I'd really like to see is ipynb supporting asciidoc instead of or in addition to markdown. You can go to/from asciidoc and docbook in a lossless way, IIRC.

asciidoc support would require an ansciidoc renderer written in javascript, which does seem to exist[1](although I haven't used it). This also highlights some of the problems with markdown because it is an insufficiently rich language for generating semantically meaningful markup for use-cases beyond blogging. There has been some really interesting work on "scholarly markdown"[2][3], but given the social and technical complexities of markdown implementations and the community[4] seems like adding asciidoc is easier than fixing markdown.

[1] http://asciidoctor.org/news/2013/05/21/asciidoctor-js-render...

[2]https://github.com/scholmd/scholmd/wiki

[3]http://blog.martinfenner.org/2013/06/19/citations-in-scholar...

[4]http://www.codinghorror.com/blog/2012/10/the-future-of-markd...

+1 mcburton

What an awesome idea -- I can think of some sociologists/biologists who had to suffer through MS Word equation editor with excel charts who would love something like this. The internal hyperlinks would be awesome too.

For now, at least you can "decode" ipynb to LaTeX and use the %%latex magic.

Sorry, forgot the link. It's what I used to export the interactive ipynb file to html for preview on GitHub, for example: https://rawgithub.com/ptwobrussell/Mining-the-Social-Web-2nd...

I would love to see this start to happen in the academic world too. Being able to play around with a very complicated algorithm you are trying to wrap your head around would be a godsend.

There are some awesome notebooks out that demonstrate exactly that. Here are a couple of good links with combinatorial optimization and TSP:

http://nbviewer.ipython.org/urls/raw.github.com/fonnesbeck/B...

http://nbviewer.ipython.org/url/norvig.com/ipython/TSPv3.ipy...

This is the first I've looked into IPython Notebook, so apologies for the newbie question: Is a notebook an html page with Python code snippets interspersed? Are these supposed to be interactive at all? Or do you have to copy the Python locally, then install some packages before you can finally run it?

I guess I'm asking, what's the difference between an HTML page that shows a Python implementation of an algorithm vs those links? Or what's the appeal?

I'm certain IPython Notebooks are valuable, since everybody seems to love them. It's not clear from glancing at those examples what the big deal is, though, so I thought I should ask in case more newbies come along and incorrectly conclude that there's nothing especially interesting about those examples beyond their content. (To be clear, the content is excellent, but why is IPython Notebook to thank rather than the author? Could those examples not have been done via some other method, or is the purpose of IPython Notebook just to make it really easy to generate such pages?)

The nbviewer links are just HTML pages, but if you were running your own IPython notebook server, it would be the actual python interpreter running your live code!

Setting up that IPython notebook server is easy: check out this 2 min video (http://vimeo.com/72383764). Bonus: if you use the one provided in github in that link, it comes with all the python imports you'd already need like networkx, response, etc. :)

The links that you're viewing are "read-only" version of the notebooks. The great thing about them if you install IPython Notebook and "run" them is that you are able to read the content, and then when you encounter code, you execute it interactively - kind of like Mathematica if you've ever used that?

There are some videos that show this in action. One that I put together is here if you'd like to see a quick demo of it as it (rather generically) relates to Mining the Social Web - https://vimeo.com/72871930

The IPython Notebook is part of the IPython interactive computing environment[1]. You must to install IPython locally(or in the cloud) and run the notebook web server to load and execute the notebook's code live in the browser. These are static HTML representations of the notebooks, think of them as previews, but you can download the JSON .ipynb file and load it up in your IPython Notebook environment and execute the code.

The benefit of the IPython Notebook is sharable JSON file and interactive REPL-like environment that blends code, results, and narrative prose.

[1] http://ipython.org/ipython-doc/stable/interactive/notebook.h...

edit: I should have refreshed the page before I wrote this redundant reply.

I think it's your opinion that is really the deciding factor there ;)

lol, don't just take my word for it: http://blog.fperez.org/2012/09/blogging-with-ipython-noteboo...

Really interesting blog. I would love to follow it but I don't see an RSS (or atom) feed: did I miss it?

On bottom right: http://miningthesocialweb.com/feed/

One of the fun examples shows how to mine an RSS feed ;)

https://rawgithub.com/ptwobrussell/Mining-the-Social-Web-2nd...

Reply on news.ycombinator.com