PhantomJS: Stepping down as maintainer
groups.google.com
groups.google.com
Even an acquisition[1] of PhantomJS would totally make sense, then let him keep working on it but based on headless Chrome and with real resources.
[1] Careful how you spin it for this route, learn from TJ/Express https://medium.com/@tjholowaychuk/strongloop-express-40b8bcb...
I played around with PhantomJS for something at my current job - it ultimately didn't work out for us (and we went a different route), but it was interesting and fun to learn about.
So if an open source project is being sold:
- Make sure everyone knows it's because it'll be better handled and not because of the money.
- At least get a decent amount of money! According to TJ it was half a month worth of money.
- If you are the company buying it don't leave it unattended afterwards.
Anyway, now TJ is surely remembered with a lot of gratitude in the Node.js community and I'm so happy he helped it grow so much!
This is all better explained here (and read the comments for TJ opinion): http://thefullstack.xyz/history-express-javascript-framework...
We can only be thankful for all the good work that went into PhantomJS, and wish the maintainers the best of luck in their next endeavors.
Cheers!
I've been using Selenium Driver with Chrome in xvfb for my headless testing needs, and I've used PhantomJS for some automation things in the past where it was great, but since I switched I really haven't looked back!
I had things breaking subtly that I couldn't fix, and they did not manifest as problems in the Chrome browser or Selenium. I still don't know what was wrong, I just know that my Rails app won't pass its JavaScript functional tests if I use PhantomJS. When I did the evaluation of 3 test drivers, I found that the one with the actual browser in it was the one that worked most reliably.
I started writing in Selenium instead of CasperJS (a PhantomJS frontend) because PhantomJS experienced intermittent bugs on the page I was trying to access. I think you're right that "real browsers" are still much more reliable for complex use cases, but the low profile of PhantomJS is definitely nice when it works.
Things like reading the HTTP response code, detecting 404s on assets and catching JS errors in the console are all not possible with Selenium/ Webdriver, and I relied heavily on those capabilities in my Capybara tests.
While headless chrome might be able to replace PhantomJS for many use cases that doesn't necessarily mean the APIs will be comparable. In fact I'd more likely expect the Chrome folks to say "the webdriver API is it, because it's a standard." [1] Sadly IMO it's lacking compared to what PhantomJS was capable of.
After we realised that we hadn't seen a "one-browser" bug for 2 years in our massive angularjs app, we got rid of all browsers but PhantomJS in our karma suite. PhantomJS' slowness, lagging-behind in webstandards and just my general gut feeling (these facts above made me question the point in running JavaScript tests in an actual browser at all) made me port our karma test suite to jest w/ jsdom and haven't been happier since we, years prior, got rid of our gnarly Selenium test suite that caught 0 bugs but was the major cause for maintenance headache.
Really? When I was writing JavaScript I thought I'd hit one ~ once a month. As a matter of fact I hit a v8 bug just yesterday which apparently doesn't support /Etc/GMT[-+][0-9]{0,2} timezones.
Phantom 2 switched to QTWebKit which I'm sure was a tremendous amount of work. Probably at the end of that he was hoping things would get "easier", and it sounds like it hasn't. It's just too much work for one person and if companies aren't will to pay people to do it, I'd quit too.
If you sum up the very real value PhantomJS has delivered to very real companies over the last several years, napkin math tells me we wouldn't have the project being abandoned for being a "bloody hell" to work on.
This as opposed to open source tooling which has no such hurdles.
Just because you can download and use it on your local machine, doesn't mean you're not violating your corp policy and procedure.
Kind of defeats the whole reason to have a closed source system when you're still on the hook and charging out $500 per hour to clients.
"You've gotta be careful with those free tools though... Never know what will happen when they break. You can't get any support. Besides, we need logging and alerting too."
"Oh, $currentTool does logging and alerting?"
"Well, yes... But it's currently not working. We've got a ticket in to fix it."
That was months ago. Yesterday it broke and there was no alert. There's also still no logs.
Gotta be careful about that "open source" stuff though.
Besides, now it's some abomination wrapped inside a proprietary program that doesn't work right in the first place, only this time we're out the $x,xxx licensing costs and the entire process is opaque, with absolutely zero hope of sorting it out on our own. That's not a huge gain...
Product Manager has $X per team member. Is encouraged to coordinate purchases and pool purchases together for licenses. Licenses can be proprietary or Open Source licenses X, Y and Z.
They do it for research, which is another thing that would otherwise probably not pay for itself.
I guess if they started doing it a lot of commercial companies would complain to them about the competition.
If you poke around https://government.github.com/community you'll find a lot of government created projects but checking those organizations/ contributors will often turn up a ton of forks of popular tools. One common thing is improving security defaults or accessibility, which are tedious but mandatory for government.
If you value this, make sure to let your elected representatives know: I'm sure they hear from the major contractors regularly.
Here's a nice blog about how the UK deals with this.
https://governmenttechnology.blog.gov.uk/2016/12/15/next-ste...
As to infrastructure work, easiest way to help and still profit is to hire one or more developers that explicitly work on a set of FOSS libraries. That way you have that knowledge in-house and a connection into the community. Also, you'll have some highly motivated employees.
This is probably the reason it worked so good.
So, do we necessarily need governments for other positive externalities in the list?
Open source software is not a public good, in the economic sense. There are two criteria for being a public good: non-rivalry and non-excludability. Open-source software satisfies the first criterion (my using it doesn't prevent you from using it), but it's fairly excludable (I can prevent you from using it legally).
As developers, our instinct might tell us that it's not excludable because "if the source is there, nothing prevents me from using it", but when we're talking about goods which fall under copyright law, the legal aspect matters as well as the practicality. And in fact, open-source licenses (such as the GPL and the Apache licenses) can contain provisions which prevent people from using the licensed software under certain circumstances, while still being considered both free and open-source by the FSF and OSI respectively[0]
The real classic example of a public good is national security. Practically, there is literally no way that national security can be applied to people within a country on an individual basis, as opposed to a geographic one. For most threat models (e.g. espionage, (counter-)terrorism), the mitigations are things like "prevent terrorist attacks from happening". You can't apply the benefits of that only to people who have paid for the service - a terrorist attack either happens or it doesn't, and you can't choose who's a victim of it.
> These are a classic example used by economists of a natural place for government in the economy.
Even for things which are actually public goods, like national security, that's overstating the case greatly. Public goods are used as an example of a good for which an individual market cannot exist, but that doesn't mean that the only alternative is a government one.
The so-called "tragedy of the commons" is an appropriate (and ironic) example - despite the way that most people use the term, the town commons was actually something for which there were plenty of well-established codified rights, and these were not always negotiated or enforced by a government entity.
[0] For example, the Apache license contains a patent retaliation clause, which terminates your right to use the software in the event of a patent lawsuit. (Technically it doesn't revoke your right to the copyrighted code, but it does revoke your right to the underlying patents, which amounts to the same thing, because presumably the copyrighted code utilizes the underlying patents, or else it wouldn't be covered by the license in the first place).
I'm imagining a clause like: "this source and any modifications is irrevocably eligible for use by all, provided that its creation did not break any other laws".
Similarly, critical software intended for developers is impractical to deny to them; doing so has some serious negative effects on the production process (hard to get good feedback/PRs from customers, for example, unless you expose the source to them to a level that closed-source manufacturers find dangerous to their business model).
WRT the original commons, you can make such things work in a tight-knit community that can enforce social norms (which, by the way, would take on a lot of what are now considered "government functions" in a modern society), but in a larger capitalist economy with actors that aren't inside the community, the government is the only actor that has the authority to enforce public ownership of the commons.
The regular market rules don't really apply to Open source software. A lot of viable (or even thriving) open source projects would be dismal failures as stand-alone businesses or startups. Paradoxically, the only way they can provide real value is in their current form of open source projects.
I would have given CyanogenMod as an example, but the amount of inept management at the startup there would cloud the issue.
You can make your software "source available", i.e., not open-source under activist definitions but still have a GitHub repo and all that, and restrict [heavy?] commercial use. I think it'd be interesting to see more open-source devs take that route and stop giving away the farm.
This will still allow people to use your stuff, developers will get familiar with the tooling and expect to be able to use it at work, and companies that have the dough can be compelled to pony up for a license.
On Windows, there is still an underappreciated market for cheap early-90s-shareware-style applications that are < $100 a pop, but I think most of them think that sharing the source means they have to enter the poor house, which is sad. We should show people that there's a way to share your source without bankrupting yourself in the meantime.
The GPL almost gets there, as it makes large-scale commercial use undesirable due to its infectious nature, which allows for dual-licensing, but with everything server-side nowadays, those stipulations are much less effective (have to go AGPL).
Not giving away something for free (as in freedom) but asking for free legal advice may sound ironic but it's not about the money, it's about having something reliable for a very common use case.
BTW, please mind the distinction between open-source software and Free Software. You don't have to like the FSF, but the distinction they recognize is important.
I don't suggest that the Linux kernel be distributed under non-commercial terms, and I agree with you that such a project wouldn't have done well.
>BTW, please mind the distinction between open-source software and Free Software. You don't have to like the FSF, but the distinction they recognize is important.
Right, so there are 3 levels of "purity" here. For the record, I didn't run afoul of any of them; I intentionally distinguished my suggestion as "source available", not "open-source".
There is "Free Software", which is software meeting Stallman's "Four Freedoms".
There is "Open Source", typically referred to as the improper noun "open-source", which activists insist refers solely to license approved by the OSI (Open Source Initiative). Because these include permissive licenses, the FSF considers them potentially non-free, and makes the point that Open Source isn't good enough; it must be Free Software.
Then there is "source available", insisted upon by the OSI people, to indicate that while you can download and modify the source, it is not distributed under copyright terms they like. This would be source distributed only for non-commercial use, for example. Jef Raskin's Archy project (apparently now dead) [0] was distributed under CC BY-NC-SA and made this distinction.
Does it have to be a market?
In this context, I would interpret "google's Moloch" along the lines of: Google is net-bad for the world, because of privacy issues and problems with centralisation and so on. Using Google's software (and services) makes them more powerful, so people don't want to use Google's software. But because everyone else is using Google's software, the world is optimized for Google users in a way that it isn't optimized for non-Google users, and so it's difficult to escape. And so Google grows yet stronger, and it becomes more difficult to escape.
(To clarify: this is my interpretation of grandparent's use of the phrase. It's not my own position, and there's a decent chance that I'm completely off-base and it's got nothing to do with grandparent's position either.)
I just read the character description for Moloch in Paradise Lost, and don't see any obvious theme you might tie to Google.
In fact, I'm still not quite sure what the original comment was trying to say. Apart from some fuzzy notion that capitalism is sort of like Moloch and we keep feeding it with our "children". Where children is what? Privacy? Money? Open source tools?
Do you have another hypothesis about what the term means in context? I note that "it's a reference to Paradise Lost" is not very descriptive: for example, if I talked about "Google's Frodo" you might ask what I mean by that, and "he's a character in Lord of the Rings" does not answer the question.
Take a look at, say,
http://www.nybooks.com/search/?s=moloch&option_match=&year_a...
Lots and lots of Moloch. Your hypothesis is 'it's a reference to some logorrheic blogger'. Sure, it's possible you're right but it's one hell of a weirdly specific guess. It's not like the three people to ever mention Moloch were Milton, Ginsberg and internet-man-addicted-to-his-own-typing.
Phantomjs is less resource heavy if you're constantly spooling up and down lots of instances but I prefer SlimerJS w/ Firefox since it lets you keep up to date with a modern version of Firefox (rather than relying on sporadic QTWebkit updates from Phantomjs).
If you're using Casperjs, SlimerJS is virtually a drop in replacement for Phantomjs (though I worry about how long/well Casperjs will continue to be maintained).
I did too, then found out you also needed a dev license for users being able to run your app. Supporting Mac/OSX is damn expensive if your app is free.
I can understand why stepping down is the right decision, maintaining such project by himself is an amazing feat on its own and even more when it proves to be useful for so many companies. Sadly when it becomes your second job you might always be on the lookout for a clean exit and such opportunity just became a reality.
Good luck in your future projects Vitaly!
Wait, can someone tell me where to download this doesn't-eat-memory-like-crazy version of Chrome? Activity Monitor is showing me 2GB of Chrome processes right now and that's even with The Great Suspender having paused almost all my tabs.
The blog post is somewhat old[1] and not in sync with the version that is in Git[2], you might find a way to do this without Docker (I was using an old version of Docker and kernel that couldn't get it to work. But I need the old version of Docker for reasons.)
Chrome will aggressively consume any memory you give it (up to a point?) to "make your browsing experience better" somehow. You're not wrong. But there is modern technology that can make it better. If you have a fast SSD, then Chrome can still use Swap to make your experience better. The later version in the Dockerfile linked on Git also leverages swapaccount with the seccomp setting.
This may be one great use of Docker for people that wouldn't yet have been convinced to use Docker for any serious reason.
[1]: https://blog.jessfraz.com/post/docker-containers-on-the-desk... [2]: https://github.com/jessfraz/dockerfiles/blob/master/chrome/s...
But won't you run into issues with child processes, of which Chrome tends to spawn a zillion? (I'm reading that each one gets its own limit under ulimit...)
And potentially get your browser killed when it hits the limit? I haven't tested this and I don't really know how ulimit works, but I think it's a less effective solution than the cgroups for at least one reason.
Or something pretty close to it.
Thanks to the maintainers for all the good work!
To me, it's always a sad occasion to see diversity diminished. Nothing against Chromium, but I hope it won't be the one browser to rule them all. It's always good to have alternatives.
https://groups.google.com/forum/#!topic/phantomjs-dev/S-mEBw...
( posted by askmike in https://news.ycombinator.com/item?id=14105613 )
The reality is very different. WebKit/Blink rendering is intimately tied with the graphics system of each platform, in particular through the use of native widgets and native window system compositors.
For example, on the Mac, a lot of compositing within the browser window is done using Core Animation layers. This is a really good idea for performance, because it leverages the work done by Apple to improve their GUI performance.
The downside is that capturing the output becomes very tricky when the browser doesn't do the final compositing. Previously this didn't really matter because 99.99% of browser rendering is for end users and they don't need to capture the output (or if they do, they would just use platform GUI functionality like screen capture).
An increasing demand for headless rendering has effectively forced browser engine teams to rethink some of the internal APIs so that a pipeline can be built to capture the final rendering.
I'm wondering, are there any examples out there for generating screenshots on headless Chrome?
If that is true, the takeScreenshot() should work as normal hopefully.
https://gist.github.com/masonmark/2332c1238a2fa70b5e4fcfffdc...
Of course, that is only talking about wide-scale scanning. If you're only looking to scrape a single target, for whatever reason, then having an instrumented headless browser will greatly simplify things. Headless Chrome should be more efficient than running it in a (virtual) framebuffer. Plus the whole setup for a powerful crawler is reduced to "install Chrome, start it, point $crawler at the API endpoint". My guess is that we might see turnkey crawling / automation tools appear where you supply a list of URLs and the library + Chrome does the rest. Then, browser-based large scale scanning will be within everyone's reach, only limited by their resources.
Background: I created https://urlscan.io which will simply visit a website and record HTTP interactions (annotating the resources with some helpful meta data). I've been preaching the power of headless instrumented browsers for the better part of a year now ;)
I had to implement a "wait between actions" feature to handle it, while Phantom had no such problems. I'm assuming this will not be an issue with headless Chrome, since I think half of the problem was due to graphical rendering.
Thank you, if you're reading.
So, cheers to you Vitaly and anyone else who's helped make Phantom & Poltergeist into my favorite Capybara web driver!
chromium-browser chromium-browser-l10n chromium-codecs-ffmpeg-extra cpp
cpp-4.8 fontconfig fontconfig-config fonts-dejavu-core hicolor-icon-theme
libasound2 libasound2-data libatk1.0-0 libatk1.0-data libatomic1
libavahi-client3 libavahi-common-data libavahi-common3 libcairo2
libcloog-isl4 libcups2 libdatrie1 libdrm-intel1 libdrm-nouveau2
libdrm-radeon1 libfile-basedir-perl libfile-desktopentry-perl
libfile-mimeinfo-perl libfontconfig1 libfontenc1 libgdk-pixbuf2.0-0
libgdk-pixbuf2.0-common libgl1-mesa-dri libgl1-mesa-glx libglapi-mesa
libgmp10 libgnome-keyring-common libgnome-keyring0 libgraphite2-3
libgtk2.0-0 libgtk2.0-bin libgtk2.0-common libharfbuzz0b libice6 libisl10
libjasper1 libjbig0 libjpeg-turbo8 libjpeg8 libllvm3.4 libmpc3 libmpfr4
libnspr4 libnss3 libnss3-nssdb libpango-1.0-0 libpangocairo-1.0-0
libpangoft2-1.0-0 libpciaccess0 libpixman-1-0 libsm6 libspeechd2
libthai-data libthai0 libtiff5 libtxc-dxtn-s2tc0 libx11-xcb1 libxaw7
libxcb-dri2-0 libxcb-dri3-0 libxcb-glx0 libxcb-present0 libxcb-render0
libxcb-shape0 libxcb-shm0 libxcb-sync1 libxcomposite1 libxcursor1
libxdamage1 libxfixes3 libxft2 libxi6 libxinerama1 libxmu6 libxpm4
libxrandr2 libxrender1 libxshmfence1 libxss1 libxt6 libxtst6 libxv1
libxxf86dga1 libxxf86vm1 x11-common x11-utils x11-xserver-utils xdg-utils
fontconfig-config fonts-dejavu-core libfontconfig1 libjpeg-turbo8 libjpeg8 $ docker run -it ubuntu:16.04 sh -c 'apt-get update && apt-get install phantomjs'
134 newly installed. After this operation, 339 MB of additional disk space will be used.
$ docker run -it ubuntu:16.04 sh -c 'apt-get update && apt-get install chromium-browser'
202 newly installed. After this operation, 615 MB of additional disk space will be used.We are using it for reports and we are tailoring our HTML specifically to the reports so we haven't had that issue. If I needed Javascript it may not be the best solution.
I like that it generates full PDFs (with real text objects) though not just a static image. I'm not sure if you can generate a PDF like that with PhantomJS. I haven't tried.
Headless Chrome might solve this issue once and for all though.
Seems to work well so far.
I recall fondly informally introducing colleagues to chrome dev tools, injecting jquery via a booklet, and querying the dom like xml xpath. Then taking this headless (server-side) with almost minimal wrapping due to your work.
Hail you and damn regexp.. :)
"Use headless chromium with capybara and selenium webdriver - today!" -- March 30th
http://blog.faraday.io/headless-chromium-with-capybara-and-s...
Better than it being closed source and you being told to go away.
I pulled a Node repo that ran tests using Karma (why people use Karma is a complete mystery to me). I pulled the repo, ran `npm install` and then `npm test`. Sure enough Karma explodes out of the gate.
Phantom can't start. I'm on Windows 8.1. I debug for an hour, eventually finding a magic custom binary Ariya created. I then have to copy this binary to the `/node_modules/karma-phantom-launcher/node_modules/phantomjs2-ext/bin` directory.
All this to run some Jasmine specs.
If Chrome headless support is really as good as "works just like Chrome without the GUI" then I will be one happy camper.
In my last job, I used PhantomJS with highcharts to provide a web service for generating charts. And used it with the poltergeist gem for headless testing.
It has saved me days of effort over the last year.
Unfortunately, we had quite a few compatibility issues with it leading us to migrate to Chrome (with xvfb) one year ago. Since then, we must confess that we are very happy of this choice. Chrome is indeed very stable, fast and more importantly for us, always up-to-date.
Are there people who earn money by getting it to automatically fill out forms, enter competitions etc?
Also, the darker stuff like click fraud and all the other kinds of fraud where you pretend there are humans doing something when in fact there's just a bot.
Think "what could I accomplish with a browser, with the slow human replaced by a fast program" and let your imagination run wild.
The space is interesting enough that people have jumped through a lot of hoops to make it work in the past; this makes it one less hoop.
Oh, and if you ever wondered why web captchas are a thing, one of the reasons is headless browsers.
You could run such a thing as a microservice using a headless browser or PhantomJS. There are probably better ways to do this but that's one of the first things that popped into my head!
It's possible to create pretty advanced layouts with maps, graphs using that. Even embedding IFrames with e.g. Google Maps works.
Or you can redo the layout in latex and maintain two layouts.
The full print css is actually pretty complete, problem is the only browser that fully supports it is PrinceXML. None of the major browsers seem to care much about print layout.
Are you in the US?
But yes, I'm talking about the US.
Some way of authorising API access to read-only access to things like statements would be fantastic, to the extent that I'd consider changing banks over it, if you know of any UK banks that offer it.
There is Kantu: https://www.a9t9.com/kantu/web-automation
It uses screenshots and OCR to automate web browsing and scraping, to you do not even have to "touch" the DOM. What you do is you simply draw a frame around the areas that you need to have extracted and OCR'ed. It also works with PDFs.
Yes. Tiny ex. https://news.ycombinator.com/item?id=1165680
It's way cheaper to launch a browser headless at all the resolution we need and grab screenshot to visually compare them at glance, instead of goin one by one at hand
- in Ruby, poltergeist (https://github.com/teampoltergeist/poltergeist)
- in Elixir, hound (https://github.com/HashNuke/hound)
• A script that would go to Comcast's TV schedule for my area and make a list of all movies upcoming in the next two weeks on all channels that are included in my subscription. I could then grep that for a list of movies I've been looking for.
I couldn't just grab the page with curl and parse it, because JavaScript does most of the work. JavaScript fetches the listings, and when you advance the listings it it fetches the new listings and replaces the old ones on the page.
• A script that goes to the FCCs license information site and gets a list of all ham radio callsigns issued recently [1].
• A script that given a URL to a tactics problem on lichess gets the FEN for the position. I'd use this if I was doing tactics training there on my iPad and did not understand why my answer was wrong or why their answer was right.
I'd mail myself a link to the problem, and then later on my desktop I'd give that URL to this script and it would go to lichess to the problem page, and then from there to the board editor page for that position, and grab and give me the FEN, which I could then use to set up the position in Stockfish to analyze.
(This is no longer useful. They have made some changes at lichess and now they have a browser-based version of Stockfish on the problem pages, so I can answer my questions right there).
• A script that goes to everquest.com and gets the server population levels from the server population display on that page.
I don't think that there was anything in this one that actually needed a headless browser. As far as I recall it could have all been done with getting the page with curl and parsing it. It was just easier to do it in JavaScript using the DOM. (The lichess one may also have been that way).
It was the first time I wasn't able to successfully reverse engineer a site enough to scrape what I needed with just requests/beautifulsoup. I was however able to get it working just fine using phantomjs via selenium via splinter. It was a fun exercise, but part of me still feels like it was cheating.
PhantomJS loaded up an HTML file written earlier in the pipeline. The HTML consisted of a big slug of JSON containing all the relevant data (which would vary from one run to another) and a bunch of scripts and templates (which were fixed for any given report type). The scripts built into the HTML file would chew up the JSON slug and build up the DOM required for the report. Then the PhantomJS script would identify all the images in the DOM and replace all of them with data URIs, strip out the JSON slug to prevent giving away more data than contained in the DOM, and strip out all of the templating JavaScript, leaving behind only the JavaScript needed for the interactive features, which was inlined.
We went with PrinceXML for PDF generation. I was briefly nervous because I saw people praising PhantomJS' pdf generation capabilities... but then I saw the people saying, "we used PhantomJS for pdf generation, we used wkhtmltopdf, then we just paid some money to get something that wouldn't produce weird output some of the time." CSS Paged Media Module FTW, y'all.
This is also a great example of why it's smart to use Selenium or something like it for scripting the tests. You can easily swap out for another backend in Selenium, but if you wrote tests in pure PhantomJS, you're now stuck with a codebase that depends on unmaintained software.
if they wanted to do that, theyd already do it. this change doesnt seem to be all that big.