cogs

No I'm not. It's possible I have done in the past, but not currently or recently.

Different computer, same house. So it wasn't browser cookies.

Supra-national organizations help prevent war. The original purpose of the EU was to integrate the countries so tightly that war .. which happened twice on this continent not so long ago, would be unthinkable. Now - when the world is looking particularly unstable - is not the time to damage the organisations that keep us close.

From an economic perspective, we live in a world of supra-national companies. To have leverage when negotiating with the likes of Amazon or Google the UK needs to work with other countries to avoid a race to the bottom on working conditions, taxes and so on.

Its the same for trade. When we're not tied by EU legislation, the UK may keep going economically, by building a low tax imitation of the US with shorter holidays, poorer healthcare, welfare etc. But I think the world is a better place for having the EU as standard bearer for a fairer kinder society. Not perfect for sure, wasteful definitely (as is all government). But also holding hope that as human beings we're able to identify with a bigger picture than the tribal nationalisms we grew up with.

How does this compare with Apple? I haven't seen so many articles about what MacOS slurps, is that because it is better behaved?

We built Crab so we could run SQL queries over the filesystem. It's for Win and Mac, free for personal use, and lets you slice this anyway you want and plot the data how you like. I like the flexibility.

http://etia.co.uk/win/about/

e.g.

  SELECT sum(bytes) FROM files
  GROUP BY extension 
  ORDER BY sum(bytes) DESC;

You can GROUP BY any file attributes

I don't see we should have to learn a new language for this at all.

The set based paradigm of SQL lends itself very well to reporting and processing files, and is known by 7 million people.

SELECT extension, count(*) FROM files GROUP BY extension ORDER BY extension;

We built a tool, free for personal use, that does SQL at the command line

http://etia.co.uk

The syntax kind of gets in the way of the logic here.

We built a SQL tool to run at the shell prompt, Crab. It's free for personal use, and designed to make this easier.

http://etia.co.uk

We've only tested on personal systems so far, just a few TB.

The theoretical limit is around 9PB, but we don't know what performance would be like at this scale.

A one TB spinning disk drive takes about 30 mins to scan, several thousand files per second, and query performance is very good. Most people scan project directories on demand, and the whole disk once a week or so. SSDs are much faster of course.

There is a restriction on the maximum number of files acted on by one query (e.g. moved, deleted, renamed), as the exec() function caches the list of files in memory. On macOS we're ok with tens of millions of files, but the first Windows release, due any day now is 32bit (we're fighting compiler issues), so the limit is around a million files.

Crab - SQL for the filesystem.

Bash is such a pain because of all the incompatible utilities. Its much nicer just to think about logic than to be searching for command switches and dealing with corner cases like .. file names that contain spaces(!)

Free for personal use, $5 / month commercial

http://etia.co.uk

We did look at Log Parser, and osquery, before starting to build Crab, but neither of them have an exec() command to run programs on query results - such as deleting them, unzipping them, or moving them. And Log Parser doesn't support joins either, which are useful to compare the contents of one directory or file with another.

We built a SQL query tool for the filesystem (macOS) last year, and will be launching for Windows in the next two weeks.

Finding files is pretty straight forward using the LIKE operator against text in the filename, path or file contents. And the set-based logic of SQL works very well for identifying sets of files to work with: we have an exec() command that lets you run commands on file paths returned in query results.

License terms are free for personal use and $5/month commercial.

http://etia.co.uk/

It lets us turn on and off different functional parts, to see the effect, so we can reverse engineer it.

We don't have good general theories about how brains work, so a good place to start is from the bottom up.

Wordiness isn't really a problem, because I'm a fast typer and most systems have tab completion and history recall.

For me the benefit is that for lots of problems, whether databases or the filesystem, thinking in terms of sets feels really natural. I can't imagine why you'd want to pipe grep results to grep -v rather than say "and not"

I guess we all prefer the language whose syntax we know the best.

Blatant plug: My pain at all these unnecessary hieroglyphics & man page work is why I built Crab, so I could use SQL to find things and an EXEC command to run commands on them.

etia.co.uk

Really it shouldn't have to be this hard.

Crab doesn't pick up metadata during the scan, because this would slow the scan too much.

There is a wrapper function 'metadata' which returns a specific metadata item from a file at a given path at query run time, but this basically runs mdls under the hood.

Crab can handle extended attributes using the EVAL function function which runs an OS command and returns the result as a string. But you have to parse the string, for example to return the size of a resource fork:

select bytes + coalesce(matchedgroup(eval('ls','-l@',fullpath),'.com.apple.ResourceFork\t(\d+)'),0)

We don't have any linux builds at the moment, if there's demand we'll make some.

We can scan mounted drives, and Crab has a command line switch to treat names as case sensitive or not. SHA-1 calls are probably not practical across a network to compare file contents, even scans can be a bit slow, but you might do something by comparing file sizes.

The example does check the contents, by using the SHA-1 of the files

We looked at Log Parser before building Crab. The syntax was far from standard SQL, it didn't have joins etc.

It didn't use string matching to identify subsets of files to query.

And maybe the most important thing, it doesn't have the exec function to run operating system commands on the files you get back in your query results.

We're working on Windows, the query syntax and the functions are going to be the same.

Crab is aimed specifically at the filesystem, rather than querying devices or processes.

For example wildcard matching on paths, combined with the exec function to run OS commands on the files you get back.

Crab has tab completion, so there's not too much extra typing, and the repacement of hieroglyphics by words does help readability.

It's especially handy to see what is going on if you wrote it a month ago, and don't remember all the command line switches.