Show HN: Phantom JS interactive editor | Hacker News Reader

Show HN: Phantom JS interactive editor

phantomjs-pad.ozaur.com

120 pointsjakozaur39 comments

A shameless plug: https://github.com/scrapinghub/splash is an open source tool similar to Phantom JS, and it has an interactive web editor built-in.

The concurrency model in Splash scripts is not the same as in Phantom JS scripts: Splash allows to use loops/ifs/functions in async code without recursion tricks or things like "casper.thenOpen". See http://splash.readthedocs.org/en/latest/scripting-tutorial.h... for more info.

We haven't done a public announcement, and Splash is not yet feature complete, but it is used in production for more than a year, there are hundreds of unit tests and it is in active (though a bit sporadic) development - so give it a try if you want something different :)

Installation is just "docker pull scrapinghub/splash" - see http://splash.readthedocs.org/en/latest/install.html.

Looks nice! I created a service called Ghost Inspector last spring that runs PhantomJS under the hood: https://ghostinspector.com/

It's not open source, but there's a free tier and lots of nice features built in like video recording, screenshot comparison and a GUI editor.

If anyone is planning to write PhantomJS scripts directly, I strongly recommend checking out CasperJS: http://casperjs.org/ -- It gives you a really nice layer on top that makes Phantom much more wieldy.

There's also Automately (http://automate.ly/) it's a pretty interesting platform and it also allows for use of browser automation but differently than PhantomJS it's a little confusing to get setup at first but once you're there it's easy from there on.

It doesn't look like Automately is open source, but their API looks nice (based on examples from http://blog.automate.ly/).

We will actually be open sourcing our core very soon! We will also be including the source code for our SDK and Automately Pi (unreleased Raspberry Pi integration).

interesting but seems like just a selenium in the cloud. wouldn't it make more sense for someone to host automation scripts on their own servers?

It's actually more than that, our API doesn't only provide access to the Browser module. We provide an SDK that can help people build real-time applications that can connect with almost anything. We're getting ready to release our Raspberry Pi integration. We are also going to be going open source fairly soon.

how does the real time application benefit from automately?

We've basically created a connected platform that makes it very easy to connect things. You can have your web application talk to your server-side automately scripts. We've made it as simple as including some JavaScript on your web page. Our server-side APIs make it very easy to communicate with APIs and WebSites. Your Automately scripts can store data via the DataBus and they can talk to other execution jobs via the MessageBus.

Edit: Here's a very simple example http://searchproxy.automate.ly/

interesting idea! however, I am struggling to see when and how someone could use this, because almost always a developer such as myself would simply create an endpoint for something like this directly on the server.

It would be interesting to see what type of customers or use cases your solution might be of use.

I am not fully sure what you mean but if you mean running the automately stack on your own servers then you would be interested in automately-core our open-source core. Shoot us an email at open@automate.ly.

Cool. Haven't heard about it.

Will have a look. PhantomJS development seems stagnate a bit. Try to contribute to it myself.

I was under impression PhantomJS 2.x is rather active, but of course Splash contributions are welcome, that'd be great!

Splash Web UI is quite similar to PhantomJS Pad UI - by default it shows a CodeMirror text editor (with a simple autocompletion) and a screenshot of the webpage. Also, by default it shows information about requests/responses (like a "Network" tab in Firefox/Chrome development console) and a textarea with rendered html.

PhantomJS Pad UI ideas are good; tabs for images, an editor window which is open by default and a field with console logs make a lot of sense.

Cool, comes in quite handy!

You may want to up your security-game though. Check the ~/FIXME file on your sever for more info :-)

Yeah, know it, but didn't spent that much time on it and it suddenly attract so much traction.

Will definitely do something serious about it.

Fixed. More security related improvements are coming.

Since the formal semantic failed to become massively used, everyone is trying to make sense of the Web using scraping. It's something most viable sites should want to get behind, from ecommerce where you can find the best price faster to education. But I suspect if a very good shared scraping system were created we'd see countermeasures pretty fast, from timid content producers, as well as the Googles who want to have exclusive access to this level of information. Yet it's a natural layer for a shared commons, and what can be built on that, when finally get past the mess of arbitrary representations, is very exciting. Anyone know of any standard to create interoperable scrapers? In Javascript, of course. Here's one submission: https://github.com/zotero/translators

You are spot on with this. Data acquisition is the foundation of data science which in turn is the foundation of [fill in the blank]

Neat. Did you have a real use-case for this, or just fun? I'm curious, what dependencies does it have that would make setup difficult?

In the Yelp example, I was confused for a second by the variable name of 'pizza' when doing a Yelp search on seafood. Renaming that would make the example more clear. :)

What does the browser-view show? I didn't try, but if the code traverses multiple locations, is it a screenshot of the last step?

Right now it's for fun, but the real-use case is to make more productive. E.g. be able to write test way faster.

Browser-view shows at the end and whenever you write page.render().

Making setup smooth is solve able problem, I just haven't spent time on that. It uses Meteor and PhantomJS.

I tried with typekit with no success: http://phantomjs-pad.ozaur.com/DrDvxMADoP2TdF652

I was curious with this example, as I have had zero luck getting webfonts to render using PhantomJS so far. Have you had any success rendering webfonts?

The thing with phantomjs and webfonts is that it has an old webkit version which has some bugs with webfont support.

Theres a branch from a phantomjs contributor on github that has bugfixes for webfonts support backported:

https://github.com/Vitallium/phantomjs/tree/fix-WOFF-file-su...

I use phantomjs for html to image rendering and I managed to get webfont support working with a custom build from that branch (working on Heroku)

PhantomJS is falling further and further behind. It also doesn't, for example, support scripts triggered DOM events.

The more time goes by, the more I'm considering removing it from our testing suite.

Working example: http://phantomjs-pad.ozaur.com/DrDvxMADoP2TdF652

You need to exit at the end: phantom.exit()

Hm, that's what I added but I guess the link did not update like other pastebin sites. Oh well...

Looks like the site is overloaded, so a lot of times it end up in timeout problems.

This is really cool. Just getting deep in to phantomjs myself, I'm sure this will help a bit :)

Author here: oh no Google banned traffic from my AWS instance and google.com shows blank :-(.

Hi. Small CodeMirror-related nitpick. If you set your editor to be the height of the panel it is in, and let it take care of its own scrollbar, it'll look better (the gutter will span the whole panel) and be more efficient in case of big documents (it'll render only the visible part).

Yeah, they'll do that :)

Can we download this and run it locally? Is the code open sourced somewhere?

Not yet open-sourced or downloaded, but consider doing that.

Right now the setup would be too hard to make it smooth.

Put it in a Docker container! :)

At first I was like "I don't get it". Then I opened it in Chrome and now I get it.

Just an FYI… the editor is collapsing in Safari.

Appears to show for me.

I am a big fan of CasperJS which sits atop PhantomJS. This tool certain looks like a fun way to get introduced to Phantom.

would love to write the script using http://www.nightmarejs.org instead ;)

phantomjs is great for rastering svgs (d3.js visualizations)

That's exactly what I've used it for. My only complaint is that the documentation is not as nice as it could be.

Reply on news.ycombinator.com