Show HN: PyFilesystem 2.0 – A Python interface to filesystems of all kinds | Hacker News Reader

Show HN: PyFilesystem 2.0 – A Python interface to filesystems of all kinds

willmcgugan.com

139 pointsbillowycoat21 comments

Oh, this looks really nice. Thank you for posting it!

A while back, I had an idea for modelling a filesystem as a dict, so maybe this will inspire me to get off my ass and actually write it.

Basically, the idea is that this:

    foo = rootfs['home']['amyjess']['stuff']['foo.txt']
    rootfs['home']['amyjess']['stuff']['bar.txt'] = bar

would be equivalent to this:

    with open('/home/amyjess/stuff/foo.txt', 'r') as foo_file:
        foo = foo_file.read()
    with open('/home/amyjess/stuff/bar.txt', 'w') as bar_file:
        bar_file.write(bar)

I really want to implement that now.

It's funny you mention that, I have literally that functionality in my "blob storage" python wrapper. Currently it's just like 50 lines of code I use for internal ad-hoc storage because as it turns out, reasoning about storage as a hierarchy is really nice. (I also let you index with a list, where the list is the hierarchical path; the key part being composibility, which I imagine you can do reasonably with both models).

Interesting to know that it's a more common pattern. I will have to think a bit on where that may go. I had always shot down any "value" with "it's just a thin transformation on top of path.join". (yay self-deprecation or something?)

There is already pathlib in Python's stdlib that do something similar:

>>> p = PurePath('/etc') >>> p PurePosixPath('/etc') >>> p / 'init.d' / 'apache2' PurePosixPath('/etc/init.d/apache2')

And yeah, this uses operator overload. However it does look quite nice.

You might want to look at pathlib:

    with (Path.home() / '.config' / '1234.conf').open() as fd:
        print('Hello', file=fd)

Granted, with / precedence you'll often need parens to group / before .

Nice idea!

You can also do this with PyFilesystem...

    fs.gettext('/home/amyjess/stuff/foo.txt')
    fs.settext('/home/amyjess/stuff/foo.txt', bar)

There are equivalent bytes methods.

I once made something like this to mount json files as a fuse filesystem. It's not maintained and one of my first projects, so the code is somewhat questionable, but it does work.

It basically mounts a dict on a filesystem, the exact opposite of what you want :)

https://github.com/yhekma/datamounter

My big question is why? you wanted to navigate or display a JSON dataset using filesystem-viewer tools? i.e. "Explore" the JSON dataset?

Well, mostly because I wanted to play with fuse, but it started out as an ansible thing (doesn't work with >2 though).

The ansible setup module returns the system information in json, and if you mount it with the --realtime flag, when you open a "file" like ram for instance, ansible fetches the current value for you. That way you have your infrastructure mounted so to speak. Sort of a /proc filesystem for your linux infrastructure.

I never got further than a working poc though. It works, but there are some bugs and there is no regard for security.

I was inspired to write a very basic working proof-of-concept based on your idea: https://gist.github.com/tcdent/742f62f7082bd4a0b958865a893cb...

You might want to look at how Common Lisp abstracted the filesystem 20-odd years ago as an example of how to do it right and also how to do it wrong (e.g. lack of iterators in those days)

Come to think of it, I have already implemented something like this in Moya.

This would insert "bar/baz.txt" from filesystem "foo" in to a template:

    ${.fs.foo.bar.baz.txt}

pyfilesystem is great!

My only suggestion would be to improve the search optimization for pypi.

It's the ~10th result for https://pypi.python.org/pypi?%3Aaction=search&term=pyfilesys... and a page down on https://pypi.python.org/pypi?%3Aaction=search&term=fs&submit...

It's really not clear what it's called since the package name is 'fs' but the project is 'pyfilesystem'.

> pyfilesystem is great!

Cheers.

> My only suggestion would be to improve the search optimization for pypi.

Fair point. Not entirely sure how to fix that. I'll look in to it...

PKG-INFO, line 2:

Name: fs

Looks like it's also registered in PyPI as "fs", from the URL: https://pypi.python.org/pypi/fs/

Maybe have a proxy https://pypi.python.org/pypi/pyfilesystem that has `fs` as a requirement? That way even if people install with `pip install pyfilesystem`, you're good.

Hmm. That's an idea. Feels like a hack though. Do you know if there is a precedence for that?

We're planning on something similar to move a big open source library somewhere else (so that dependencies don't break).

If you run pip install py.test it fails with a message that you should run pip install pytest.

As long as you can recursively process folders and files without stopping early due to an UnauthorizedAccessException-equivalent (which .NET didn't straighten out 'til v4).

http://stackoverflow.com/a/2663779

Would this work with any object implementing the PathLike interface introduced in Python 3.6 ?

You generally won't need to. PyFilesystem hides the differences between paths from you (https://pyfilesystem2.readthedocs.io/en/latest/concepts.html...).

An exception would be in the constructor for OSFS Objects, which take a system path. I'll look in to adding support there...

Reply on news.ycombinator.com