Hacker News Reader

Comment by liuliu | Hacker News Reader

How they scale automatically on python (a non-functional language)? I am puzzled.

They probably spawn several python processes

Indeed, seems like a single cloud.call will not give you any parallelism, you need many calls. So the whole parallelization challenge is left completely up to you. Still, this is a nice approach, and I'd like to see more companies in this space.

Agreed, a very cool idea. I believe you are right on the parallelism front, cloud.call just offloads the processing on their servers, but you'd have to call it multiple times to get any parallelism.

PiCloud dev here.

That is correct. We also offer a mapping function, cloud.map(func, arg_list), where every func(arg_list[i]) will be evaluated in parallel.

This is just a way to call a function and run it on a different server(s). You still have to design your program to handle parallelism.

Python is a functional language in the sense that it has first class functions and you can pass them around.

Yes but it's not purely functional in that you cannot automagically make a script scale to n threads, as is the case with Haskell. This is what the OP is referring to.

You can't really do that either in Haskell. Well, in theory you can, but in practice nobody has yet written a way to autoparallelize Haskell that actually gets you reasonable speedups, because it requires predicting which parts are worth parallelizing.

Instead, the Haskell devs are mostly taking the same approach, of providing constructs for the programmer to explicitly write parallel code: http://www.haskell.org/ghc/docs/6.6/html/libraries/base/Cont...

Correct. cloud.call, etc. are all higher-level functions.

PiCloud does treat cloud.call(func) dispatches as completely functional. The function evaluated cannot have side-effects in terms of the program space; within the program, you access its output via cloud.result(). (Note: You can within the func have external side-effects - e.g. opening a connection to a database and modifying entries).

- edit : nevermind, found the answer in the 'pitfalls' doc : yes you can create picloud call from within picloud, but they of course go against your maximum count of parallel calls -

I have read (quickly) your webscraper example and i still have one question : Is it possible to launch piclouds jobs from within a picloud job ? i.e. from picloud download some list of url as csvs and then start webcrapers then have the original function return the results of these scrapers ? Thanks for your answer.

Reply on news.ycombinator.com