Pigshell – Unix the Web | Hacker News Reader

Pigshell – Unix the Web

pigshell.com

333 pointsnmcfarl78 comments

Author here.

- Please use a desktop browser (Chrome preferred). While mobile browsers kinda-sorta work if you're just clicking through the examples, keyboard input doesn't work on mobiles (yet).

- Pigshell has few users (in fact, using the plural is a slight marketing exaggeration) so please bear with me for any teething issues, broken links, inadequate documentation...

- One of the practical use cases is copying files across Google accounts, and backing it up to your desktop. (Details here: http://pigshell.com/v/0.6.2/doc/gdrive.html) The nice thing about a unix approach is that the exact same cp command can be used to copy GDrive to Dropbox, or any other supported filesystem.

- Some more D3 visualization examples here, using Mike Bostock's gist viewer: http://bl.ocks.org/ganeshv (Do use the "Open in a new window" links, like this one: http://bl.ocks.org/ganeshv/raw/46cc526d41d16995ccdc/).

Good work, congrats!

I played with the same idea a few years ago, but never went that far :)

cf. https://github.com/hbbio/webshell

Hi tubelite, on mobile safari and chrome IOS7.04 with an ipad the keyboard comes up after I clicked on this example command: pig:/$ cat http://pigshell.com/sample/photos/bchips.jpg I can type in stuff and after enter the keyboard disappears again. A external bluetooth keyboard might work.

Yes, that happens, but it is awkward and unreliable. The CLI is a CodeMirror instance (version 2.something), which has known issues with iOS. I need to check if later versions work better on iOS.

I looked through the four documentation links on the right and was unable to find out how to cat a Google spreadsheet as anything consumable by the shell. I took a guess and tried cat -o gdrive.fmt=txt and csv, but those were not fruitful. Is this possible currently?

edit: found mimeMap.js and saw tsv was an option. That didn't work either.

IIRC the Drive API does not support those conversion options. The way forward would be for an xlsx parsing JS library to convert it into CSV or a stream of Javascript objects. I didn't find any GPL-compatible library last time I checked.

See https://developers.google.com/drive/web/manage-downloads for a list of export format possibilities. For spreadsheets, the only options are xlsx, odt and pdf.

Google Drive document can be embedded as iframes. Wouldn't this solve this problem?

Here's a proof: http://www.cellmaster.com.au/example.html

not for retrieving the juicy data inside the spreadsheet as objects.

Oh, I didn't read the whole thread, I was just thinking about my own experience with `cat gdrive/something` showing me a lot of unreadable bytes when it could show an iframe.

Are write operations possible with pigshell? If I connected my Facebook account and want to post a new status update or add a picture to an album, it doesn't seem like I can do something like this:

pig> echo "Testing status update" >> /facebook/me/posts

That should be

  echo "Testing status update" > /facebook/me/posts/

I tried it and it WorksForMe (TM)

That said, two caveats:

- The FB filesystem hasn't been worked on in 1+ year.

- AFAIK v2 of the graph API will become mandatory sometime next year and does not allow apps to get lists of friends. Friend list returns only friends who also use the same app... which is pretty disappointing.

Trailing slash?

Fitting web apis into a filesystem metaphor isn't always straightforward. For instance, it is common to POST into a container URI (which we represent as a directory) and have no control over the name or id of the object which is created. This is the case with Facebook posts.

  echo "foo" > /facebook/me/posts

would imply overwriting the posts directory, which is not allowed (either on pigshell or on unix)

OTOH, asking the user to do a

   echo foo >/facebook/me/posts/post-11oct2014

sets up the expectation that the post will be named that way, when it will most likely be a long numeric id. Thus, the trailing slash. Not perfect, but the best compromise I could think of.

The great-grandparent suggested:

  echo "Testing status update" >> /facebook/me/posts

I think appending content to a file path (with or without the trailing slash) is clearer than the analogy of a truncating write to a directory path (where the trailing slash is required).

The current model seems similar to the REST interpretation of CRUD, where > equates to 'create' and >> to 'update', and creating a resource is done at the collection URL. Nothing wrong with this, but it seems slightly unintuitive to me.

My suggested mapping would be:

  >/facebook/me/posts - Unsupported (or delete all posts, and make a new one)

  >>/facebook/me/posts - Add a new facebook post

  >/facebook/me/posts/long_numeric_id - Update post (truncate)

  >>/facebook/me/posts/long_numeric_id - Update post (append)

In this case, both > and >> are context dependent, but more flexible. Would this strategy mesh well with the rest of the implementation?

Also, nice work! The project reminds me of Plan 9's /net.

So if you are running psty, here is a fun sequence of commands to try out:

  cd http://dilbert.com/strips/comic/1995-06-24/
  ls
  cat 21021.strip.zoom.gif

Never really liked the colorization of Dilbert. It was so much nicer as a line drawing. Best we can do is remove the colors.

  cat 21021.strip.zoom.gif | to canvas | pixastic -d

And to save your result,

  cat 21021.strip.zoom.gif | to canvas | pixastic -d >/home/dilbert-unix.png

Or to GDrive or Dropbox or even the local RamFS by changing the target directory.

(Scott Adams is fine with personal usage. I checked. And you need psty because it proxies HTTP requests for sites which don't have CORS enabled)

Wow, this is a really interesting way to think about web resources. After about two minutes of playing around with this, it gave me a more pleasant experience of the web than I've had in a while.

This is what it was like when I first got on the Internet (around 1992); there was no web then, instead you FTPed everything and used resources like Gopher, Archie, and Veronica to find documents.

I find PigShell interesting from a historic point of view but think it also points to a flaw in how the web currently works, with far too much glue and eye candy, which is a direct result of the ad business model.

I completely agree. I actually deleted a bunch of my comment before posting it, because I started getting carried away pondering about what the right level of abstraction is for dealing with web resources.

At the risk of sounding like one of the semantic-web xml-standards people, it really does make me wonder if all this horrible, incompatible, site-specific frontend crap is just a fad that will give way to tools (like psty) that ignore all that and just deal with the raw semantic content (and HTTP verbs) at a higher level of abstraction.

Woah, man. We're like...surfing the Matrix, man.

Things like this really make me miss the old Firefox experiment "Ubiquity". It was truly a marvel for the time it was up, and I often wonder why they did discontinue it.

You could search websites, send emails, look at maps and much more (you could build your own commands) without ever actually opening a webpage. It had 200.000 users when they shut it down.

Wouldn't you just be able to fork it and continue as a hobby project of your own? Who knows, maybe you find other enthusiasts and it becomes a real project again. Mozilla licenses are (I believe and could be wrong), quite lenient about these things.

This is what I found by looking at its Wikipedia article:

https://bitbucket.org/satyr/ubiquity/downloads

How is the procedure for "installing" new programs? Adding a local directory with javascript commands to the PATH will do it?

What about saving the state of things. I want, for example, to `mount` my local directory with psty everytime I open pigshell, and I wanna have my PATH set automatically. Will a `/local/.config/.pigrc` do this? (This raises another question: file editing. Where is the file editing tool? Maybe a web-based editor somewhere). Well, maybe a gist will do it.

There are two kinds of commands: Javascript commands (like ls, cat) and scripts (ps, kill, man, csv2js, table2js).

Currently, Javascript commands have to be built-in at "compilation" time. (I need to add a load command which will evaluate a new JS command inside the pigshell closure)

Scripts can be anywhere in PATH.

There is an rc mechanism for doing exactly what you want. At boot time, we run /etc/profile, followed by /local/rc.sh - this is a file saved in the browser's localStorage. Mine is very simple:

  HOME=/home
  mount http://localhost:50937/ $HOME

You can create it simply by

  echo "HOME=/home\nmount http://localhost:50937/ $HOME" >/local/rc.sh

or running "edit /local/rc.sh". Edit provides a simple inline CodeMirror-based editor (actually even the CLI is a CodeMirror editor instance). I use vim on the desktop to do any serious editing. If you run psty, /home is a desktop directory visible inside pigshell, so edit on desktop/run in shell is a low-friction process.

If you manage to mess up /local/rc.sh to the point where pigshell is not "booting", then you can boot it in "safe mode" by appending #norc to the URL, i.e. http://pigshell.com/#norc

I downloaded http://pigshell.com/v/0.6.2/psty.py, started it with "python psty.py -a" and did "mount http://localhost:50937/ /dir" successfully.

Unfortunately only "ls /dir/file" works but neither "cat /dir/file" nor "cp /dir/file ." although "file" was marked with "chmod a+r". I get the message "Cross-origin request denied (check if psty is running)". What do I wrong?

Argh! This is a bug. Has been fixed in master (https://github.com/pigshell/pigshell/commit/f402ef445977015a...)

As a workaround in 0.6.2, please use a trailing slash in the name of the mount point. ( /dir/ rather than /dir)

Alternately, use http://pigshell.com/v/0.6.3-pre4/ which would have the fix.

Unfortunately these suggestions don't work also. Could the problem be in psty.py? Or is my Python 2.7 obsolete? There is a strange error message in psty:

EXCEPT timestamp out of range for platform time_t Traceback (most recent call last): File "psty.py", line 404, in send_head self.send_header("Last-Modified", self.date_time_string(entry["mtime"])) File "/usr/lib/python2.7/BaseHTTPServer.py", line 468, in date_time_string year, month, day, hh, mm, ss, wd, y, z = time.gmtime(timestamp) ValueError: timestamp out of range for platform time_t

This is a new bug. I'm guessing it got exposed because you have a 32-bit machine, and I've been testing so far on 64-bit. Edit line 404 of psty.py as follows:

  -            self.send_header("Last-Modified", self.date_time_string(entry["mtime"]))
  +            self.send_header("Last-Modified", self.date_time_string(entry["mtime"] / 1000))

If it still doesn't work, maybe we can continue the conversation on the google groups forum?

I have a 64-bit machine but your fix works great :-)

Thanks for this, and thanks for sharing your amazing pigshell!

More in this comment by the author: https://news.ycombinator.com/item?id=8436716

Interesting. I'm trying to really go out all the API for my project and was thinking about something like a Mathemathica-like notebook for the local API. E.g. say you want to execute an API call which normally does GET /api/v1/x/y/z; in the "API shell" you type x/y/z and the JSON collection is summarized as a small datatable (not showing all of the 500 rows / 40 columns that might have, but an interesting summary) that you can later interact with.

I'm thinking that such an approach can give the user their own way to explore/modify the data ,especially if you combine this with some kind of pipe lines that let you analyze/mutate/visualize the data without having to write some Python code to access the API (even though with a REST API plus e.g. requests it's simple).

I like the psty idea to integrate with the local system. Sounds like it could also be used to start e.g. emacs on a remote page. Say you have a site with an API that lets you read/write files, you mount it and then can fire up emacs on it via pigshell.

Is this approach similar to something like flickrfs[0], to make one's flickr account locally accessible? It always seemed like a cool idea (though I never had much luck with flickrfs itself), and it is neat to see the broader utility of psty.

[0] https://launchpad.net/flickrfs

The psty links in README file are broken. A working link is http://pigshell.com/v/0.6.2/psty.py

As mentioned in the README, you need to run psty server locally and establish websocket connection (mount in pigshell). This will enable you to harness local filesystem from pigshell.

I just had a glance about the functionality, but for my understanding, this is the best idea right behind ifttt. Pipelining all the web's things with well designed interfaces on a shell. Even if the product itself would be unstable, alone the idea is great enough for a gold-medal. Keep it up!

So in many ways, this is like having a FUSE/Hurd translator implementation directly from the browser, though it also provides a mini-Unix environment due to needing to implement corollaries to some Unix utilities.

The scope of this project is definitely what makes it nifty.

It's very cool, gets back to using the power of the simplicity of UNIX. I found though, that X middle click paste doesn't work.

I made a directory at /mnt/mccomsoft then ran Firefox and didn't see it which is good. :) Then I tried mounting that using your HttpFS with:

  mount http://www.mccomsoft.com/index.html /mnt/mccomsoft

It gave me:

  Cross-origin request denied (check if psty is running)

Which is also good. :) If I load Pigshell from my domain though then it should mount fine.

One user commented below it's like the Internet was a shell and you've cracked it open. I agree.

Congratulations!

The way how you implemented "man" ("man ls" for instance) shows how a convenient shell can look like. Pigshell could be a foundation for a completely new kind of toolchain for web development. I am thinking of seemless integration of JS editors, D3 graphics for pipe output, etc. ... Wait - it's already working :-)

http://bl.ocks.org/ganeshv/raw/b971989c337e958c0531/

If it just were possible to access the bare bone hardware so that we wouldn't be restricted to the JS world (in a closed environment, of course).

The way the man page is formatted is great but I don't like the scrollbar inside a scrollbar effect. It's been traditionally a worst practice. It's better to have the man page take all the vertical space it needs in the main frame and add a Close button to the end of the man.

By the way, from http://pigshell.com/v/0.6.2/doc/psty.html "Websocket server: Every unix app on the desktop which uses stdin/stdout can participate in a pigshell pipeline" so maybe your request has already been implemented.

> By the way, ...

Thanks, meanwhile I have realized that also :-)

Kudos, tubelite. This is really cool. I'm not sure I'd ever really use it but it's been fun to play with. Definitely made me think that I'd like to use the web in a more command based fashion.

Have you given thought to exposing this to the regular shell with FUSE?

The concept of lazy-stream-of-JS-objects and HTML as a display engine underpin pigshell.

I guess it would be possible to adapt the filesystem code to fuse4js. It won't be a drop-in; unix FS ops differ from pigshell's, but it shouldn't be too hard. Personally, I find psty (http://pigshell.com/v/0.6.2/doc/psty.html) a better way to incorporate (a part of) the unix filesystem into the web rather than fuse - which is the other way round.

Isn't this -- at least in spirit -- what IPython notebook tries to do? In IPython, you can use shell commands with a bang, e.g. `!ls -l`. And `ipython notebook` uses ZMQ to connect the server and client.

At any rate, awesome work. I've been thinking for a while, offhandedly, about the idea of piping web URLs around. But I don't know how in fact it would work under the hood. I write a lot of API interfaces in my work, and it's all in the details -- every API is different in small but important ways.

Yes, it is similar in spirit to IPython.

We have been developing pigshell off and on since ~2012, and I was unaware of IPython notebooks until ~6 months ago. When I'd last used IPython, it didn't have any of the notebook stuff. So it was a bit of a jar to see that they had been there, done that, got the T-shirt, published the book, etc. etc. :)

Pigshell can run in the browser with zero installation, and has a low barrier for casual usage. "Notebooks" aka gists can be shared without requiring a backend. The focus of pigshell is more around providing file adapters for web/cloud data stores.

That said - more power to IPython! I love the conversational, exploratory, CLI style of interacting with a computing environment. The more ecosystems of this sort, the better.

I do love this idea, but it bothers me from a security standpoint. Is there a way to do something like this without giving the author explicit control of all of my online data?

The author has NO control whatsoever of your data.

- Pigshell is a 100% client-side Javascript app. There is no server side. pigshell.com is a dumb static html/js/css server.

- All data (username/password as well as user data like photos and files) travels directly between the user's browser and the provider (e.g. google.com, facebook.com). pigshell.com is not involved at all.

- Pigshell is free software - you are free to download the source code, examine, modify and run it locally yourself. (https://github.com/pigshell/pigshell)

- Our privacy policy is minimal because we know NOTHING about you - not your name, email address, facebook id, nothing. Only your IP address is recorded in apache logs, which is the case with every website you visit.

What is more, we don't want to know. Pigshell has been deliberately designed so as to afford maximum privacy and freedom to the end user.

I don't really doubt you or your intentions, but I guess I might just not understand how authentication works for something like this.

When I try to sign in with Google Drive, it tells me that PigShell (developer email: xxxx@gmail.com) would like access to my Google Drive, as well as my photos and videos. Does that give your API key access to my account, or does only code that I run that's hosted on pigshell.com have access? What do the permissions get tied to? And how would it work if I were to host it locally?

We use OAuth2 client-side flow. Many pure client-side apps use the same technique - pigshell is not blazing any new trails here.

Only the code you run that's hosted on pigshell.com has access. The permissions are tied to the app-id which is embedded in the code. The access token is persisted in your browser either as a cookie or explicitly in localStorage.

We don't and won't support OAuth 1 (Twitter, Flickr etc), which lacks a pure client-side flow, just to avoid the issue of users having to trust the pigshell.com server to generate (and not leak or misuse) the access tokens.

Here is a rough guide to local setup:

- Check out the git sources, run "make" (some more details here, but reading the Makefile should help)

- set up apache to serve the virtual host pigshell.com (if you want to use Dropbox, you need to create a self-signed SSL certificate and set up https as well)

- modify /etc/hosts and set 127.0.0.1 to point to pigshell.com

This way, static assets as well as redirect URLs from the OAuth2 server will hit your local server rather than pigshell.com

one way to look at it ...

the "application" Pigshell (a copy of the javascript downloaded from the "site" pigshell.com and running in your browser is an instance of the "application") requests and say is granted access to your GDrive.

so all permissions/rights are associated with the running instance of the "application" pigshell in your browser - cookies apart, end of story.

other than having served the javascript, the "site" pigshell.com has no further role to play here. so you could just as well have sourced/hosted these javascripts locally.

Hosting locally makes sense I guess, or at a server you control and using your credentials.

> Error loading root filesystem at http://pigshell.com/v/0.6.2/root.tar: Forbidden

I like the idea. If I'm playing with some facebook data, right now I use whatever language I'm confortable with and use its HTTP request functionality, and than manipulate the data how I want in that language. To achieve the same thing in pigshell I have to write a shell script that manipulates the data. What can pigshell do that I can't just do in another language?

In theory, the availability of APIs should be enough for the technical user to write their own data extraction routines, connect them to other APIs and so on, using their preferred language and toolchain. In practice, reading up the documentation, writing and debugging code sets up enough of a barrier to entry that it discourages casual usage without a powerful incentive.

The availability of a file abstraction covering many popular cloud services, web resources etc. enables casual, low cost experimentation across these domains, without having to read up each one of the APIs. If something clicks, then it may make sense to write a better "native" implementation.

With some practice, or even better, some support libraries, this looks like a great way to quickly explore viz prototypes. Great work!

Attempting to copy photos out of Facebook albums into GDrive gives me "Exception: SecurityError: The operation is insecure." I'm not running psty locally.

This is super cool, though. There's an undocumented template engine built in it seems? (see /bin/mostliked) which is neat.

Does not work in Internet Explorer 11: Exception: The property "slice" of an undefined or null referance cannot be accessed (Exception: Die Eigenschaft "slice" eines undefinierten oder Nullverweises kann nicht abgerufen werden.)

Sorry, Internet Explorer is not supported currently. Please try Chrome or Firefox, in that order.

This is awesome. It reminds me of a talk I saw recently, linked from hackernews at some point in the last year or so. Something about re-thinking internet infrastructure into a "directory-tree" like model. Does anyone remember it?

I don't, but I am interested if you find it. We already have a hierarchy in a way.

The domain tree goes from top to bottom this way <---

The resource tree goes from top to bottom this way --->

So, using numbering for order, you could look at a URI like:

0002.0001.0000/0003/0004/0005

Meanwhile, in an alternate reality...

    $ ls /net | shuf -n 16
    coffee
    net
    рф
    한국
    org
    dating
    pw
    中國
    institute
    இலங்கை
    ninja
    com
    みんな
    su
    政务

Wow, this was like the nutshell cracked and the universe spilled out ...

Ah! Nice feeling that whole of internet as a single (unix) machine.

reminds me of Plan9 sorta.

I like the sentiment, however I think it'd be cooler if you could get the Facebook, Google, and other APIs to behave in the "everything is a file" tradition of UNIX.

Looks a lot like webshell (http://webshell.io). I wonder if there are other similar tools out there?

I'm curious about how are people using this. Is it just a visualization tool for data or what else? Is anyone using it as their primary navigation tool?

Like I said in an earlier comment, there are very few users right now (cough OK just me cough).

Mostly I have fun with it, e.g. trying out visualizations of interesting data (e.g. http://bl.ocks.org/ganeshv/raw/fada13e56d2c5b7b1a78/)

My Serious Use Case is full and incremental backups of the contents of my Google Drive, both personal and company, to my desktop. Running the Drive desktop app does not backup Google Docs, and you can only sign in as a single user at a time.

perhaps stating the obvious here, but with the abstraction pigshell provides of resources on the web, it should be reasonably straight forward to build arbitrary "internet applications" too.

with deferred pipes, split and joins (that pigshell already provides/supports) it should be possible to implement arbitrary work/data flow definitions one desires.

coupled with psty, one could build/harness special purpose "transformers" on the web

pushing it further, one could conceive of setting up pipes between transformers on the web.

of course getting there will require significant work on a security model as well.

[and then someone will figure out a way to interject ads]

> of course getting there will require significant work on a security model as well.

It could probably be easier than you imagine. The security model could be implemented in psty completely by granting/denying requested resources whose permissions are stored in a database on the psty server, and managed by using long security IDs in the URLs. This way the security model could even be fine-grained. In other words, pigshell itself could be embedded into a secure environment, covered by the permissions of psty.

CouchDB is already compatible.

There are so many possible uses cases. For instance,

- Web development toolchain.

- (Multiple) Server administration.

- Control embedded systems. No need for special GUIs there. For instance, simply psty on a Raspberry Pi would be sufficient to present collected data in a nice web graphics on your PC. However for embedded systems I would recommend a reimplementation of psty in a compiled language like C or Nimrod.

My connection is shaped at 128 kbps at the moment - am grateful that this loads really quickly. Awesome stuff. HN itself is really low bandwidth too.

This is beautiful!

I have no idea what I would use this for, but I really wish I did.

Very cool.

How do I know what are all the fields available on a file?

Here is one way:

  ls -d /path/to/file_or_dir | printf

This command prints the stack of file objects. The last one printed is at the "top" of the stack and tells you the fields available. You can then print a field

  ls -d /gdrive/user@gmail.com/somedoc | printf "%(raw.ownerNames)s"

Sorry, a lot more documentation is needed. Working on it.

I typed in `oink` but the command was not found...

    rm -r facebook

ah, that feels good.

holy crap this is awesome

A new toy! Thanks!

Reply on news.ycombinator.com