Linux Productivity Tools (2019) [pdf] | Hacker News Reader

Linux Productivity Tools (2019) [pdf]

usenix.org

1,239 pointsketanmaheshwari269 comments

I do my development work and run most of my agency (with multiple employees) with Linux. For some of the business things that most people don't assume Linux can do, I use:

1. Xournal to annotate PDFs (aka sign contracts without printing them and scanning them back).

2. LibreOffice of course for most document related work

3. OBS Studio for recording webcam videos along with screen sharing

4. Audacity for audio editing (heck, I used this even when I was on Mac OS X)

5. Technically I've tried video editing with OpenShot, but do find myself back at Final Cut Pro X on my now 6 year old Macbook Pro for that for now

6. pdfsandwich and Tesseract OCR for OCR/turning PDFs into searchable files

7. Chrome/Firefox are both first class and run all the modern day web application stuff

8. Tons and tons and tons of command line stuff that Linux is well known for

9. QEMU/KVM for hosting arbitrary virtual machines with almost native performance

10. GnuCash for double entry accounting for personal and volunteer society finances. I used GnuCash for a while to run the S Corp accounting when we were on Freshbooks and Harvest, but we've since graduated to QuickBooks Online for better invoicing and CPA office professional services support.

11. GIMP for photo editing

12. Inkscape for messing around with vector graphics

Once you get past locating the tools to do your job. Linux has everything.

Could try out Blender for your video editing. It also appears that DaVinci Resolve has support for Linux.

I had totally forgotten about Blender since I never really understood 3D animation. But great idea. Thanks for both suggestions.

Does anyone know of some modern blender tutorials aimed specifically at video editing?

if possible, including capturing footage form the screen (screen casts).

This us a great addon I discovered recently, but nevertheless they have an introductory video about the Blender editor: https://github.com/GDQuest/blender-power-sequencer/blob/1.5....

I'm showing my bias but my good friend Andrew runs Blender Guru on YouTube. Admittedly he covers everything from beginner to advanced so I don't know where your skill level sits, but Andrew has a lot of content.

DaVinci Resolve has pretty unfortunate restrictions on what graphics card drivers you can use on Linux

https://wiki.archlinux.org/index.php/DaVinci_Resolve

Brilliant! I found this as well https://www.fosslinux.com/24381/how-to-install-davinci-resol....

Great list!

> 5. Technically I've tried video editing with OpenShot, but do find myself back at Final Cut Pro X on my now 6 year old Macbook Pro for that for now

Have you tried Kdenlive lately?

Thank you for the suggestion. I'll have to try it out.

I second that suggestion :)

I'd be using LaTeX for professional looking documents.

LaTeX is great for professional-looking typesetting, and it has the power to do just about any layout tricks you want, but, in practice, you'll be fighting an uphill battle if you try to trick it into doing something that someone out there hasn't already written a package for.

(I say this as a professional mathematician, who lives his life in LaTeX. It's fantastic for writing math, and trusting that all the kerning etc. will be handled properly. However, when I want any formatting tricks, even after 25 years I still have to turn to my local TeX guru, who more often than not says "you don't really want to do that with TeX.")

Not sure what formatting tricks you mean, but that doesn't seem quite fair. When I want to do anything unusual, I google my problem and almost always there's a question on Tex stack exchange (which is blessed with the participation of most of the experts in the subject) with answers giving several easy ways to do it, using packages I already have on my computer! And it's easy to make your own commands when existing ones don't do the job.

The canonical one is forcing image placements. Sure you can Google and get a result, but all of them are prefixed with "if you _realy_ wany to do this here's my preferred workaround, but you should let Tex do the type setting".

> When I want to do anything unusual, I google my problem and almost always there's a question on Tex stack exchange (which is blessed with the participation of most of the experts in the subject) with answers giving several easy ways to do it, using packages I already have on my computer!

Yes, exactly! As I said:

> if you try to trick it into doing something that someone out there hasn't already written a package for.

There's an incredible package library out there, rivalling CPAN, and I love TeX and won't speak against it; but, if you try to step outside the package library (or even if you try to compose packages in sensible-seeming ways), as is very easy to do if you try to view TeX as a general-purpose typesetter, it rapidly becomes clear that there's a lot of magic going on that those packages hide away (more or less neatly, depending on their maturity), and that is hard to reproduce on your own, or add to.

I would say the vast majority of business documents, proposals, contracts, etc are collaborative editing with others. I didn't mention it but basically Google Docs and Sheets rules the world with these given we have a 50 user grandfathered account. For contracts, end result is exported to PDF and signed with Xournal and sent to the client for countersignature.

I meant to use LaTeX beamer to build these slides but couldn't figure an easy way to make the "Anatomy" figures eg. on slides 8,14,15 etc.

Tikz maybe?

I'm in a very similar position and have had much the same positive experience you've had. I use most of the tools you listed plus a few others like Gitkraken (git gui), Blender (3d effects, logo, video overlays) and ffmpeg (occasional video optimization or editing).

Great list additions! Thank you.

Instead of Gitkraken I recommend GitG. Gitkraken is both sluggish and absolutely guzzles memory, which isn’t unsurprising considering its another Electron application.

I prefer tig which is ncurses based

1. "Xournal" is amazing, small install footprint, simple interface and can do everything you need with pdf's.

5. You should try [1] "Kdenlive", it has it's quirks (as all linux tools do) but it's probably the best open-source Linux video editor out there.

[1] https://kdenlive.org/en/

"Quirks" is an understatement. Last I checked, you couldn't even have multiple timelines in the same project, every bit of editing had to be in the one same timeline.

For video editing, Blender is the best FOSS tool I've tried.

But you now get Davinci Resolve for free on Linux. It blows the FOSS competition out of the water, being a software with probably millions of dev hours funded by Hollywood studios behind it. Blackmagic decided to go the way of providing the basic tool for free in order to build user base amongst hobby video editors, and it's not a bad move IMO.

Wow, thank you for that, I didn't know Resolve. I do use KDEnlive but I can't say I'm satisfied with it.

There's also Olive Editor [1], still in Alpha but shows potential.

[1] https://www.olivevideoeditor.org/

Lightworks also runs under Linux.

Resolve has a quite steep learning curve, at least it had for me without experience with other NLE. For my simple workflow it was too much and I backed-off.

Happy to see the recommendations here for video editors on Linux, will give some of them a try.

A little out of topic, but which tool would you recommend for just the basic video editing on Linux? Nothing fancy, just cutting and pasting videos together, possibly with some simple transitions? Thanks!

Not OP, but I think OpenShot is made exactly for this kind of simpler editing. It's akin to Windows Movie Maker.

ffmpeg - you can do declarative editing combined with a Makefile. Clips can be previewed with `mpv --osd-fracitons` or equivalent.

My wife (a music teacher) was a Mac only video editing person. She now uses Open Shot almost exclusively, even though for $40 I bought her Sony Vega (she was complaining about Open Shot at the time). I think Open Shot is a good gateway to Blender or other video editing tools. She does all her stay at home videos on Open Shot now because she can do it so quickly. I would say it took about a week to get really comfortable making simple videos.

The basic version is also free on Windows & Mac.

Sorry for piggy-backing on this thread, but is there a simple tool which will let me do a screen capture and annotate it. Something like ScreenFlow on Mac. I use Peek for screen-capture currently.

I wanted to annotate some text and OpenShot doesn't really work well for that as I can't place the text freely and am limited to a few templates.

OBS will probably be able to do what you want[1].

1 - https://github.com/obsproject/obs-studio

I had too many headaches with Resolve 16.2 hanging on exports or UI stopping responding on Ubuntu 18.04, I hope it gets better. The camera stabilization filter is amazing.

Kdenlive does support MP4 as source assets, Resolve free only does ProRes MOVs and such. It's just an extra step with ffmpeg but still.

Avaliability of Davinci on Linux is impressive. I hope other productivity products do the same thing.

I have tried many video editors including OpenShot, Cinelerra and Blender to finally arrive to Kdenlive and stay with it! It is still in an active development and getting better and better! I also definitely recommend trying anyone Kdenlive.

Thank you for Xournal. I used LibreOffice for that but it added artifacts to my images (like a vertical black line to one if rendered as a PDF but not if rendered as a TIFF).

> 5. Technically I've tried video editing with OpenShot, but do find myself back at Final Cut Pro X on my now 6 year old Macbook Pro for that for now

I wish that there was a better video editing alternative for a non-mac... Openshot and the others I tried crash too much (Openshot crashed for me today while trying to do simple trim)

The new 2.5.1 version on OpenShot is a big stability improvement over 2.4 (which is the current repo version)

2.5.1 is the version of openshot-qt that I was using

Have you tried Davinci Resolve? It is one of the top 3 video editing suites, and it actually even runs on Linux native! It is even supported on redhat (but works on others too). Very stable as well (unlike Adobe's jokes).

The only one that I could always make work is Kdenlive. That said I never tried DaVinci Resolve as it's a bit beyond my use-cases, but I've heard good things about it.

I've not had an issue with openshot personally, though I haven't had to much more than trimming.

I use arch linux, so whatever the latest version released on that.

I agree with you regarding crashes. Kdenlive has been much more stable for me but I don't really like it. Can't say exactly why but it just feels clunky.

I've been using Flowblade for video editing lately, and have been very impressed (granted, my needs are pretty simple).

This sort of sounds like a vegan/vegetarian telling me that the veggie chicken tenders taste a lot like regular chicken tenders.

this analogy works on a deeper level than you think because you can make some really great vegan meat replacements although the majority of people just stubbornly refuses to believe we're there yet

vegetarian meat (usualy from industrial satay, made with dissolved soy in whatever chemical bath) is what drives me away from pretty much every vegetarian/vegan place.

this is why opensource that is just a cargo cult to copying even the bad decisions of commercial software harms more than help.

id be fine tasting a nice indian meal made with vegetables. but instead I get gnome changing the side of the window close buttons (while at the same time removing the options dialog to change it back) just because the designer du jour liked copying osx instead of windows. it's fake-meat all over again.

This makes no sense. Gnome has its buttons on the same side as Windows. You are probably thinking about ubuntu which patched gnome to move the buttons to the left.

And even ignoring that detail, the side the window buttons are on is entirely made up. Windows isn't the real OS with OSX as the fake windows clone. The gnome philosophy is to support a minimal number of configurations but to make sure they are all tested and work perfect. Other DEs allow full customizability but I have found them to be buggy.

How many desktop environments actually let you switch the window button side? I haven't seen one, and if you know one, why are you using gnome instead of it?

> How many desktop environments actually let you switch the window button side?

KDE does. And Gnome used to.

So use KDE then? Last time I tried it I found it to lack the polish that gnome has.

Give KDE a try. Gnome has steadily dumbed down and removed features; KDE is delightfully customizeable and very, very polished in current form.

This. After the KDE 4 debacle, I gave up on KDE because they cut all the functionality I relied on. However, I’ve been pleasantly surprised by how functional the newer KDE5/Plasma desktop is, especially once you change a handful of really ugly defaults (mouse cursors, window switcher and a couple other similar things)

That, imho, is the major boat anchor holding KDE down: the ugly defaults. If they would take a moment to apply tasteful default settings it would make a huge difference in the marketing value.

In the end, though, KDE is "just like Linux" in the philosophy of "you don't like it? change it!"

Out of the box, though, KDE Neon or Magneia are nice enough. The new Breeze theme is much better than Oxygen. What concerns me a lot more is that major features like Activities don’t “just work” on major Linux distributions (e.g. on Debian testing, on my desktop, trying to create a new activity just sort of hangs and causes a daemon’s cpu usage to spike to 100%

Interesting conundrum, with KDE I'm fighting the abundance of features not to break things but with GNOME I'm fighting the absence of features in order to make it productive.

Gnome still does:

Gnome tweaks -> Window titlebars

keep in mind that the tweaks app was a very voiced project against the gnome team. it kept fixing what they broke. it was mostly a f* you message ...that everyone must use daily, which say a lot about the message.

I can't remember the last time I clicked on a close/minimize/maximize button. May be when I was in middle school 20 years ago.

Not all meat substitutes are "industrial" after all they've been around about 2000 years: https://en.m.wikipedia.org/wiki/Mock_meat!

Maybe I should have just downvoted and left it at that, but it's late at night, so here goes.

How exactly do you think a comment like this contributes to the discussion? Someone posted a list of tools they use. You reply with this. The world would be better off without this type of information-free negativity.

Twenty-five-year Linux user and lover here, and heck, I got a good chuckle out of the guy's comment.

Chuckles don't meet The Bar. /s

eat chicken of the woods

For those asking about video editing, don't forget about Shotcut. I keep an eye on most gpl projects in github and I see consistent updates and communication from them, on top of it really advancing in features lately. Shout out to the shotcut team.

https://github.com/mltframework/shotcut

Can definitely vouch for Shotcut. I tried everything I could find a year or so ago, and Shotcut was by far the nicest to work with.

This feels off topic. I agree with these suggestions and appreciate the suggestion for Xournal, will look into that.

But TFA focused on fundamental command line tools and shells, not office productivity apps.

There are thousands of articles on the classical command line tools grep, awk etc. With a subject "Linux Productivity Tools" I think this was both on topic and valuable.

+1 I find the discussion here more appropriate to the "Linux Productivity Tools" title than the original article.

Seconded.

SolveSpace for engineering-oriented parametric CAD needs

Kinda like this guy except I use google apps for documents and sheets, VSCode for an IDE and...

I boot Win10 for Steam games, Fusion360 and AutoCAD.

There's so many games on Steam that run native on Steam or really well via Proton. I abandoned Windows a few years ago with video games being my last hold-out. If it doesn't work under Proton or isn't native, the game is a hard pass for me.

To each his own. Windows isn't that awful these days and wrestling with Nvidia drivers on Linux is a PITA and entirely Nvidia's fault.

Their drivers on Windows aren't that great, either.

Although it's difficult to use, I've found Natron for video editing to be not terrible, https://natrongithub.github.io/

It's like suggesting Nuke to do video editing, while possible, not the easiest and most likely you need previous experience with the software.

For 5, good alternatives: Shotcut and Kdenlive.

and 13. Krita for digital painting

I'll also add Olive[1] as an alternative video editor. I tried shotcut this weekend for just a simple clip/split mashup of videos and it was so slow and cumbersome. Olive on the other hand was extremely responsive.

[1] https://www.olivevideoeditor.org/

Thanks for this list! Wish I knew about Xournal earlier this year.

Care to share any details about pdfsandwich and tesseract? As in, do you have some scripts to glue it all together?

Xournal++ is also worth a look. It's a fork that includes a few features I missed in the original, otherwise it's almost exactly the same.

In your opinion, what's the best GUI for QEMU/KVM?

Virt-manager

+1 for virt-manager.

But you know what bothers me? Virt-manager is mature and feature-complete, and RedHat has discontinued it in favour of cockpit.

Cockpit is cool and I like it, but it's not as complete as virt-manager.

I'm a big fan of GNOME Boxes. The developers put a lot of care into the UI design and into choosing sensible defaults so most things work well out of the box. However, there are some kinds of advanced tasks that can only be done via virt-manager. I doesn't happen very often for me but when it happens it can be useful to know how to use virt-manager to configure the VMs that GNOME Boxes created.

https://blog.agchapman.com/configuring-gnome-boxes-vms-using...

See also xournal++ - https://github.com/xournalpp/xournalpp

Not affiliated in anyway. Just a very happy customer, but want to plug PDF Studio [1]. I've used Atril, Evince, Adobe, PDFXChange (through WINE) and this PDF Studio (not FOSS) is worth every penny. It even interfaces with docusign (but you don't need it since you can import real digital signature) their support is amazing, and linux is first class there.

[1] https://www.qoppa.com/pdfstudio/

As for PDF editing there is Scribus, which is fully free open-source (FLOSS).[0]

Scribus also available for Linux as AppImage.[1]

[0] https://www.scribus.net

[1] https://wiki.scribus.net/canvas/AppImages

I use Ledger for double entry accounting.

https://www.ledger-cli.org/

Ohh this seems SO much nicer than Gnucash! Thank you!

Just an addition to point 1: There's also Xournal++ [0]; some parts are based on Xournal but it provides some additional functionality [1].

I switched from Xournal to Xournal++ four weeks ago and it's been a blessing.

[0] https://github.com/xournalpp/xournalpp

[1] https://github.com/xournalpp/xournalpp#features

Nice list! Thanks

In certain forms, text need to be filled in a series of square boxes. If there is an option to adjust character spacing, you can in fact fill such fields too using a monsospace font. But I am yet to come across an open source (or even free) tool (for annotating PDFs) which can adjust character spacing too. Can Xournal++ do it? If not, are there any other open source or free tools that can manage it?

If all else fails, convert to SVG in Inkscape and edit it there. Export back to PDF when you're done.

Yes, only that it has to be done page by page and then you have to merge the pages back.

This is an excellent list. I just want to add that jetbrains products (intellij, webstorm, etc) all run perfectly well on Ubuntu (and probably other distros?), with regular releases using their toolbox app.

> 8. Tons and tons and tons of command line stuff that Linux is well known for

And this is what covered by "Linux Productivity Tools" slides.

> 5. Technically I've tried video editing with OpenShot, but do find myself back at Final Cut Pro X on my now 6 year old Macbook Pro for that for now

Checkout Olive video editor.[0]

[0] https://github.com/olive-editor/olive

1. Xournal to annotate PDFs (aka sign contracts without printing them and scanning them back).

Will check it out. For PDFs I mostly use Master PDF Editor

2. LibreOffice of course for most document related work

For me LibreOffice is a bad nightmare. I use the commercial Softmaker.

6. pdfsandwich and Tesseract OCR for OCR/turning PDFs into searchable files

Good luck with that. My results with Tesseract were always abysmal. I use ABBYY Finereader with wine. I would pay for a native Linux version. They have a Linux command line tool that has a biblical price tag.

12. Inkscape for messing around with vector graphics

Inkscape is good. I wish they would still develop Xara XL

What I am missing in your List:

Recoll. Find Stuff on your computer. One of my most important tools.

Try scantools instead of pdfsandwich for creating searchable OCR'd PDFs:

https://cplx.vm.uni-freiburg.de/scantools/

Easy to install and use and works great.

I use scantools for creating searchable PDFs:

https://cplx.vm.uni-freiburg.de/scantools/

For OCR I use OCRmyPDF[0] on my Mac (also available for Linux and Windows). It does a very good job, is pretty fast, can even OCR images by converting them to PDF and you can use different languages. It also reduces the file size significantly. I really like it.

[0]https://github.com/jbarlow83/OCRmyPDF

Thanks for the list. Btw LibreOffice Draw can also edit PDFs [1]

[1] https://www.ghacks.net/2018/02/13/using-libreoffice-as-a-pdf...

Its table of contents doesn't do it justice. this is a great deck! super dense and straightforward.

my tiny tip to contribute:

for those who could never be bothered to remember the ctrl-commands for traversing words, but are familiar with `vi`-style movements: you can go one step above the suggestion in this deck with `set -o vi`.

the great thing about this is you dont even need vim, its built into bash.

Thanks for the praise. The slides are kind of a summary of everything I learned here at HN over the years. Feels great to be on the front page. Definitely going to brag about it for a few years :).

> you can go one step above the suggestion in this deck with `set -o vi`

Highly recommended if you use Vim regularly (and you can define the same escape key(s) for your terminal and vim instance). For me that was one of the biggest productivity boosts when doing editing and command line work.

vi mode plus swapping my capslock and escape keys are by far the best thing I've ever done for terminal productivity. it kind of sucks on shared dev servers though.

I'm a heavy vim user, but I still prefer the emacs-style key bindings for bash. If I want to do vim-style editing on the command line, I sometimes use Ctrl-X Ctrl-E to edit the line in my default editor (which is vim).

YMMV, of course.

Same here, I'm more productive using directly the GNU readlines shortcuts, but it's maybe a matter of habit

I do the same. This might be overboard to some, but I went a step further and installed a vim plugin that brought many of the emacs-style key bindings into vim.

`.inputrc` is your friend. I have `v` (in normal mode) configured to open the line in an editor, because muscle memory sometimes makes me try to enter visual mode when editing a command.

fc is an interesting related bash-builtin command. It will open $EDITOR and put the last command in it to edit.

Another is C-x C-e to invoke $EDITOR (or $VISUAL, if defined), and go to twon in your preferred editor.

Like others, I use vi over emacs, but prefer emacs Readline bindings ... to a point. Full true editor at the shell is quite handy.

locate is not mentioned -- two orders of magnitude faster than find for searches from '/' on my laptop, e.g.

  find / -name \*bashrc\*

  locate bashrc

Faster because reading from a single binary cache file, updated nightly, rather than opening every directory inode under /. To force an immediate synchronous update of the cache:

  sudo updatedb

There is also `fd` [1] which is written in Rust and very fast alternative to `find` with a simpler syntax.

1. https://github.com/sharkdp/fd

curious, is there a rust based implementation of posix utils or at least stuff from gnu-coreutils etc. ? thanks !

https://github.com/uutils/coreutils

locate relies on indexing crons. This has killed several production servers until I disabled it.

Have you tried FZF? [1] Fuzzy matching. It's loaded as a vimplug but binds to bash. It even fuzzy matches your bash history (CTRL-R).

[1] https://github.com/junegunn/fzf

Relying on a database which gets updated nightly is problematic for some usage though in at least 2 ways: I don't leave my computer on if I'm not using it so there's a good chance the db will simply never update (I guess?) plus it's only usable for locating files with confidence if you know they're system files or else they were on the system yesterday. Or, you run updatedb manually all the time and weep because that's not something instant, at all.

All in all, the idea is good, but I'm almost sure there are better implementations (even Windows has tools like 'Everything' which use a database but which gets continuously updated).

Even running updatedb (except the initial run) and then running locate might be faster than running a recursive grep from the root directory. Definitely if you run that grep more than once.

There are cron implementations that handle the "turned off at night" problem. anacron is one of them.

>you run updatedb manually all the time and weep because that's not something instant, at all.

No, not instant, YMMV. On my mid-range laptop with SATA SSD:

  $ time sudo updatedb
  real 0m0.501s
  user 0m0.247s
  sys 0m0.218s

Is there any alternatives like "Everyting" on Windows? I really like it for its GUI and depending NTFS index which is immediately updated.

nothing beats recoll

Nice slides. I missed him covering ripgrep and friends, which is very useful to avoid slower find-grep queries.

Along with ripgrep, I think GNU Parallel (covered in the slides) and htop (briefly discussed) are great additions to any Unix development environment.

The rest of the standard utilities have stood the test of time surprisingly well. But top is a bit wonky, xargs has many pitfalls and, as I said, ripgrep is great for speeding up some find-grep workflows.

+1 for ripgrep (it really is a lot faster).

Also, I recommend using fd as a replacement for find. It’s API is a bit easier to grok imo, and it’s similarly blazing fast.

fd/rg/fzf are the core tools I end up using most frequently, working in a big monorepo.

Does fd actually cover all the same ground as find? I used to be intimidated by find, but these days I find it’s extremely useful for all sorts of otherwise complicated file location operations. For example, to find files newer than a certain date, just use ‘touch’ to set the date of a temp file and then ‘find -newer temp’. Or, I have a script that deduplicates all the regular files in a tree by hard linking them to a file named by the sha256 of the file’s contents ‘find -type f | ( while read -r fn; do ...; done )’ and then, after deleting files or whatever, I can “garbage collect” the shas by deleting files in my link farm that only have one hard link ‘find -links 1 -print -delete’

Much the same here, but I tried fd the other week, and for simple searches it definitely feels much quicker and "lighter", so worth adding to the toolkit imo.

Why not use `find -newermt` ?

I have a chapter on ripgrep [0], and I'm currently updating the book.

[0] https://github.com/learnbyexample/learn_gnugrep_ripgrep

Here's my non-cli list of to me usefuly software on linux (working as freelancer since 2010):

- Webstorm + Phpstorm: Amazing tools for web development

- Sublime text for simple text editing

- Meld for graphical diffs between files and directories. Very useful.

- Signal client: Using it more and more recently

- Spotify: That's not productivity though :)

- Joplin for markdown formatted note taking

- KeepassCX + FF extension: Works decently for password management. Though there's no solution for FF mobile as far as I know.

- Reaper for professional audio production (never liked audacity and ardour). There's also Bitwig which is more like Ableton live but quite expensive.

- Gimp: I use it but I don't like the user experience.

- Inkscape works pretty well for anything vector based

- Libreoffice: I tend to use calc regularly

- Darktable for raw image editing

- Arduino

- Figma for digital design (web-based, so not really linux)

- Nextcloud as file storage and for synching contacts and calendars.

- Zoom for video conferencing (need to try jitsi again though)

I too use joplin and darktable on MacOS.

> KeepassCX + FF extension: Works decently for password management. Though there's no solution for FF mobile as far as I know.

Not sure what mobile platform you use, but on Android I have been very happy with Keepass2Android (https://github.com/PhilippC/keepass2android). It uses the android password-manager API so you get autofill (or at least a quick-access button) on login fields in both apps and websites.

Edit: It also supports a lot of methods for syncing the keepass database between devices, I point it to a directory that's managed by Syncthing and it works pretty flawlessly.

Thank you, I will check that out!

I've tried Joplin but have stuck with standard notes for now.

Joplin is very good and has basic things like ctrl+f for finding withing a document that Standard Notes lack, but end to end encryption is baked in to SN where as it's somewhat trickier to set up in Joplin. Both are solid recommendations to check out though.

One useful thing when managing one's time is to know when a command will complete. I have a habit of using "pv" anywhere I wait for a result so that I know how long it takes:

http://www.ivarch.com/programs/pv.shtml

It's basically "cat", but with a progress bar. Also useful for measuring IO speed and volume (e.g. decompression: `pv -c largefile | gzip | pv -c > /dev/null` )

You can also use it with dd, or even just redirect it directly to a block device. I use it a lot for writing USB images. `pv image.iso > /dev/sdX`. Just gotta be careful not to redirect it to the wrong one...

Project idea: ML-based snafu-avoider that prompts for confirmation when past invocations by other users in the community of that command resulted in manifestations of regret[1].

[1]: screaming picked up by mic; detected maniacal CTRL-C'ing as immediate as it is futile; permanent loss of peer relationship to the project's cloud (indicating subsequent reformatting and/or booting-off-tall-building)

I use bmaptool for that. It shows progress and prevents writing to mounted volumes.

Latest versions of dd support the argument "status=progress" for displaying the average write speed while running.

I just use type-ahead.

This is an awesome deck. I'll be bookmarking this as a great resource to pass along.

Tangent: although the portability and consistency of PDFs is nice, I'd love a browser extension that gave me a nicer view of slides that were exported as a PDF. I find there's something more satisfying about discrete pagination vs continuous scroll for these types of things.

Are there any good parsers to convert pdf to html/epub? It would rely on the pdf being well structured.

Acrobat does very well in my experience.

Pandoc has worked for me.| it literally does everything.

Foxit does pagination.

Don't most PDF viewers? Apple's native PDF viewer (which isn't available on Linux, so not a solution here, but which isn't terribly feature-rich so that I assume that whatever it can do most any browser can do) switches at a keypress among continuous-scroll, one-page, and two-page views.

On Firefox and maybe on chrome, if you set the page height to fit, it will go in presentation mode and space bar goes to the next slide

Left/right arrow keys does it for me on Firefox

The python utils section missed the very cool json.tool that can pretty print JSON files (and exists on your OS if it includes Python):

cat /tmp/foo.json | python -m json.tool

https://docs.python.org/3/library/json.html#module-json.tool

I like jq (https://stedolan.github.io/jq/) for this purpose.

It's a very flexible tool for filtering, querying and manipulating JSON data, but when run with no arguments, it performs a no-op transformation that just pretty-prints its input.

jq doesn't come pre-installed with every OS, python's json.tool does. So it's always there when you need it, no installation required, which is why I put it the same category of all those classic command-line tools.

I don’t actually believe my last Linux install came with a system Python. And as a Python developer I always end up using some tool or another to manage installations of multiple Python versions anyway. Furthermore jq just works as a classic command-line tool.

> ... just works as a classic command-line tool.

impossible to understate the value in: not frequently fucking up your ability to run the critical/mundane/everyday/musclememory commands you've come to rely on after maybe 100+ invocations/week as far back as you can remember just because someone you've never met wanted to bump the .ruby-version in your java monorepo

What Linux distro do you use that it doesn't have Python on it?

Ubuntu but I think the same was true for Arch.

What? Half of Ubuntu's internal stuff is written in Python. They've switched to Python 3 but I've never heard of Ubuntu without Python.

I’m sure that’s true and maybe I’m misremembering but I still think I had to apt install a Python runtime I could actually invoke from the shell.

jq is yet another bloated syntax to learn and I'd definitely refrain from suggesting it to anyone whos not working with json everyday.

I tend to agree, regretfully.

I’ve unfortunately often seen coworkers spend half an hour painstakingly crafting and debugging a jq query, that could have been cobbled together in a few minutes with grep and sed.

I think the issue is that jq’s interface makes it really hard to “narrow in” on the right solution, whereas grep is much more forgiving.

I agree in the general case, but the syntax for pretty printing which is what he is doing is literally nothing.

Or 'jq', if you include the binary name.

Hardly "bloated".

Agreed about the bloated syntax bit. Whenever I need to work with jq , I tend to trial with jid first and then use the actual syntax with jq. Convoluted if you think about it....

i dont know if i'd call its syntax bloated at all.

jq is something i've hated learning but powerful beyond tying grep and awk together.

jq is a tool, and like every tool, requires some upfront learning. Once mastered its much more productive than the grep/awk/friends solution.

Take this example JSON

<pre> { "code": 42, "items": [ { "type":"color", "name":"red", "rgb": [120, 2, 2 ]}, { "type":"color", "name":"green", "rgb": [2, 120, 2]} ] } </pre>

for instance, to grab the rgb field for the color green... jq -r '.items | .[] | select(.name=="green") | .rgb

An alternative would be to use XPath 3.1.

I am implementing XPath in a tool called Xidel.

There the syntax for the example is much simpler: xidel - -e '?items ?* [?name="green"] ?rgb'

Depends what kabacha means with bloated. The XPath syntax is more verbose, but less cryptic

Firefox has quite a nice interactive JSON viewer, built in.

Which doesn't help on the command line.

Invocation from command line:

  $ firefox --new-window somefile.json

The viewer does however suffer from poor scaling with file size.

but you are out of command-line once you do that. The whole point of this guide is that you can combine all these tools together.

Don't abuse cats, use `python3 -m json.tool < test.json` instead.

Why go to all that extra effort to type a "<" character?

    $ python3 -m json.tool test.json

Bit of pointless contrarian opinion from a long time cat|'r:

> Characters saved with "<"? 3. > Hours lost with accidental ">"? unbounded.

Preface: I'm confident on the shell & with my keyboard: I've been using shell redirects for a couple decades, learned touch-typing up to around 80WPM, and used to note-take math lectures live in latex. Yes, I have noclobber enabled. But...

I can't promise myself I'm on a machine configured that way OR that I won't typo this one on a file that really matters.

I kinda use of `cat` as a `immutability-please` command: it _creates_ a barrier/seam between this file and whatever I'm doing with it, such that I'm _guaranteed_ nothing downstream will mutate it (as long as everything only works on stdio).

In a kinda similar way: by starting with `cat x |` (or even just `cat x`, frequently!), the rest of that compound command is _input source independent_ (and even _ignorant_!) -- if I want to hook it up later to a `curl example.com/xyz.json`, I don't need to split it into separate commands/introduce temp files/etc. This is especially because I find myself doing a _really_ quick scan of the entire file ("did it return the actual object this time or just `{}`?"), then piling on transforms from there.

Something about doing `cat x | head` and then kill-wording to fill in `next-thing` feels faster and more consistent than `head x` and then `next-thing x`, maybe because the filenames are often much longer than the commands? Maybe I need a "go to start of line, forward-kill one word" hotkey?

tl;dr: i'm playing psychological games with myself and TBH not sure I'm coming out ahead.

Wow, you described my workflow exactly. Using "<" might be shorter and faster-for-the-machine, but it's less readable and breaks the left-to-right order. I also frequently start with `cat x`, see what's in the file, move on to `cat x | some filter` etc.

'<file' can be used anywhere in the command. If it makes more human sense to have the file near the start, you can do that.

grep <file -opts 'regex'

This is why I walked back my claims a bit in my tldr — basically started trying stuff out and had a “whoa not used to the way that looks, but it does _work_...” moment.

I think I still have the “wrong” preference, but I appreciate learning some new stuff!

cat was just to demonstrate using it on streams. A more common use case would be to pipe some curl to it to see some API output, or grep some log file.

Thank you for this suggestion -- it is now included in the slides.

Some distros don't install lsb_release by default. It might be better to consult the /etc/os-release file instead.

https://www.freedesktop.org/software/systemd/man/os-release....

http://0pointer.de/blog/projects/os-release.html

These lists always delight and frustrate me at the same time. Every piece of software seems to have a lot of those useful, little tricks, shortcuts and functionalities but discovering them is always so difficult.

Biggest problem is that you don't even know they're there. You don't even know to look, what to look for. I seem to always stumble my way ass backwards into them, and then, be somewhat irritated that I didn't know them before.

I like to use systemd timers rather than cronjobs nowadays.

For those looking for a link https://wiki.archlinux.org/index.php/Systemd/Timers

feedgnuplot [1] is wonderful for quick visualizations of data from the shell. Also great for decoupling visualization from CLI programs.

[1]: https://github.com/dkogan/feedgnuplot

Hey that's my old roommate! Check out his vnlog project as well, it's kinda feedgnuplot 2.0

How in the world did I go this long without using the "at" command? (slide 69)

Thanks for the deck, OP.

Wow. Well, I wish I would have known abut GNU parallel before I wrote that massive bash script a few years ago. Thanks for posting this.

Its a very handy utility. Sometimes when I am lazy to write proper multiprocessing support in my codes, I get some work started anyway by (1) quickly providing a basic CLI version of my code (2) throwing in a bunch of code-invocation commands in a file and calling GNU parallel on it.

Learning more unix tools tends to lead to that feeling a lot. The last few decades of devs building tools generally means that most obvious problems have a solution, but of course one usually only learns about that solution after hacking together their own version:) One tip: Look up moreutils; it's a nice consolidation of things that make a lot of things easier. (One gotcha: moreutils ships its own different parallel tool.)

xargs can do most of the common tasks perl parallel does, see xars -P

Imho, the main advantage of Parallel is that it can execute the tasks on multiple hosts, straight from the man page:

> GNU parallel is a shell tool for executing jobs in parallel using one or more computers. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables.

This gives an (artificial) example of the power of GNU Parallel over xargs: https://unix.stackexchange.com/questions/405552/using-xargs-...

It's great for scaling text smashing and bashing imho.

It's an absolute godsend for plowing through data-parallel computations.

In native Mac text editors, you can use e.g. SHIFT + ALT + LEFT to select the whole word on the left of the cursor. Is there a way to get a Mac terminal that uses similar shortcuts? For copy/paste also?

Related, but I found https://micro-editor.github.io/ which tries to use shortcuts you'd expect in native Mac apps.

There’s a couple options here: most of the Mac terminals allow you to specify custom key-combo -> Terminal input mapping that can be used to get this. Also, on most shells you can customize the interpretation of the key bindings to attach this key combo to a specific editing command. Which works best depends on the functionality and the key combo you’re trying to bind.

slides 14 and 15 - terminals have emacs shortcuts??!

EDIT: I am god now

You can get vim shortcuts as well.

zsh: bindkey -v

bash: set -o vi

You can configure readline (which is what bash uses) to use vi-bindings via $HOME/.inputrc then everything that uses readline (which is pretty much everything) will also get vi-bindings.

This is because your shell uses readline. https://linux.die.net/man/3/readline

The main thing I miss from OS X is Droplr (pain free sharing of annotated screenshots) and Jumpcut (pastebuffer history tool). Can't find anything on Linux for either of those that works nearly as well. I can still go to an old Mac for occasional needs like Final Cut Pro but those are tools I used several/dozen of times every day before.

I'm on PopOS.

Have you tried Flameshot for screenshots?

I had not, just installed it though and it looks fantastic, thanks! The best I'd found previously was a no-annotation script that failed most of the time with a vague "something went wrong" error.

I hunted around for a while looking for something similar. I spent so much time in Slack working remote, that fast screenshots with arrows and notes are critical to communication.

Coming from a Mac environment, I can't remember what the tool was that I used previously but I think it was purchased by Evernote. Flameshot has been excellent as a replacement.

You're thinking of Skitch, I used that too before Droplr.

I have been using shutter and kazam for screen recording... but will check this out...

flameshot is excellent, but I think X11 only?

What about wayland?

> pastebuffer history tool Have you tried clipped? https://github.com/davidmhewitt/clipped

Not exactly linux but the Speedcrunch calculator is really good.

Another good productivity hack is having a good clipboard manager.

I highly recommend Green Clip: https://github.com/erebe/greenclip

You can assign custom shortcut keys to pull it up and it can fuzzy search through your clipboard items.

For the macOS users lurking here, I would like to suggest flycut https://github.com/TermiT/Flycut

Clipboard managers are probably the thing I miss the most on Wayland.

Haven't tried this yet but worth looking at? https://github.com/yory8/clipman

This is great! I might add a slide or two about Makefiles. Probably the topic is too big and out of scope, but inexperienced Unix users may not realize that they are useful for so much more than compiling multi-file C programs.

OP here -- thanks for the comment. I actually had 2 slides dedicated to "program development tools" that included gcc tools and makefile. Had to remove them due to time constraints -- had only 90 minutes to talk.

Please lets not recommend anyone new to Makefiles. It's such a dated system that you need to use tabs for functionality.

Just one of million better alternatives like https://www.pyinvoke.org/

Makefiles need to be allowed to die so we can clear ourself of this absurd legacy cruft.

Make is for tracking dependencies between targets, and rebuilding stuff. Looks like pyinvoke does some glorified scripting, or something. I will never understand why so many people have no idea how to use Make, but are convinced that it is terrible, and then try to fix it by reimplementing some tiny subset of its functionality.

Make is great. Learn it an use it. Its limitations are well-known, but easy-enough to work around. Fixing them properly requires lots and lots of complexity, and you get bazel when you try. And it's way overkill for all but the biggest projects, which is why Make has been so successful.

It's was great but then we improved our work flow patterns. Make is really limited and ugly by today's standards. I mean you have to use tabs for Pete's sake and if you include a single space the program will break. It's dated software that should be abandoned with either evolution or revolution.

Other useful tools include mosh or EternalTerminal to handle either low bandwidth or spotty network when you need to remote into a server, as well as VNC for when you need a GUI on the server but X-forwarding is too slow.

I expected something more of a traditional set of "productivity tools".

Eg. a tool like "PDF Mod" is a gem I struggle to remember the name of when I set up a new computer — quick and easy way to assemble/extract PDF pages :) It's long unmaintained it seems, though.

For those expecting the same, this is mostly "GNU terminal tools" (+ python, tmux and a few other tidbits), so applies to pretty much any modern GNU system (including non-Linux systems, and I am pretty sure homebrew on Mac has all of these).

Thanks, these look great. Is there a recorded talk for these slides?

It doesn't look like it. Here's the talk info from the conference: https://www.usenix.org/conference/lisa19/presentation/mahesh... Other talks have links to their recordings.

It also doesn't seem to be in the full video playlist from the conference: https://www.youtube.com/playlist?list=PLbRoZ5Rrl5ldJR-XU4xQx...

Unfortunately no. The conference recorded only regular track talks. This was on Tutorial/Training track.

> cmd #tag to tag a hard to remember command

Command tagging wasn't really explained in the deck. I had to read this archived web page for an explanation:

https://web.archive.org/web/20191216131445/https://www.ostec...

Thanks for the comment. An example of #tag in action is shown in slide 39 (In the talk, I mentioned we will see an example later).

This is an awesome intro, thanks! Will definitely share.

I can't recommend fish-shell highly enough for anyone spending significant time in the CLI. It can't completely replace bash et al, but it's not really meant to. Just install it and give it a try and you'll get addicted pretty quickly.

The only thing that I miss in linux is a pdf viewer that can show inline animations (like those produced by LaTeX's "animate" package). I am still forced to run adobe's 32 bit pdf viewer because it is the only viewer available in linux that supports this kind of pdf.

I never know about pdf inline animation. Do you have some sample of pdf files that contain inline animation? I would like to try them on various pdf viewer.

The documentation of the "animate" package contains an example on the first page:

http://tug.ctan.org/macros/latex/contrib/animate/animate.pdf

Notice that, while this is just a silly example, animations are very useful in serious technical presentations. For example, in research about image or video processing, you often want to alternate automatically (without user interaction) between the "before" and "after" images, or along a few frames of a video sequence.

Wow, that seems like a rather questionable feature for a "portable document format".

If you use pdf for making slides for a presentation, it is a legitimate use case to want some automatic animations or videos. It is actually possible, and adobe's viewer certainly supports it, but no program on linux does.

You could argue that we shouln't be using pdf for making slides in the first place, and maybe you are true. But if I want beautiful math typography (which is even more important than animations), all other options are sadly not up to par.

I understand; I used Beamer back in college. It just seems like that's not really a good fit for the original purpose of PDF files. If they wanted to support presentations, maybe they should have created a new file type based on PDF but with extra capabilities.

I just tried it in Okular (KDE's own) and it works. You have to let Okular "Show Forms" first, but it very conveniently shows the button at the top of the window.

Oh, man, you just saved my life! Now I can do everything I need with free software (notwithstanding that Okular's interface is quite wonky, but whatever).

I guess this is not the place to discuss about how to change all of computing and how it should be done better and that Linux is a distraction?

https://www.youtube.com/watch?v=rmsIZUuBoQs

I absolutely love this sort of thing. I am a sysadmin with something like 30 years experience. I am as passionate about the subject as anyone but I lack some presentation skills and possibly imagination. I'm also old enough to know what I lack!

Why not give open source stuff a go?

If you want to know which cli tools you can expect on every *nix machine, I can recommend this page [1] which gives a nice overview of the POSIX shell standard.

[1] https://shellhaters.org

Some essential stuff lacking here (unless I missed it): pushd, popd, dirs, jobs, and fg.

I did have jobs, fg, nohup in an expanded version but had to trim to fit the slot. I considered pushd, popd but decided against it as it requires a bit of explaining -- figured I won't have much time.

I see. I tend to recommend zsh, anyway, which has navigational features that superannuates pushd, popd, and dirs.

Awesome! Thank you for sharing this. I've been trying to get better with my Linux CLI knowledge, and this will really help.

Is the presentation this was delivered to recorded? I'd watch whatever lecture this was from.

As a wsl user I miss all this tools.

Is there any link to the actual video presentation of this?!

my list visual stuff:

- xournal - pdf editor

- calibre - epub

- keybase - slack replacement

- cherrytree - notes

- anki - study

- shutter - screenshots

- zoiper - voip calls

- kazam - video snippets (screen recording)

- remmina

- vsc

command line

grep -nr, locate, ncdu, htop, ipython, vim, ssh, tsp, tail, less, youtube-dl ;-)

Great compilation

tmux ftw!

screen!

My poor brain corrected the url to “unisex.org” on first glance.

Otoh facilities like Oak Ridge can be expected to be conservative in their tech, otoh sed and awk should be declared as a threat to national security.

How much of these is really still relevant with now Python quickly becoming new scripting language across all platforms? I now tend to prefer 3 lines of Python vs 1 line of cryptic unmaintanable Linux-only bash. With new great subprocess module in Python, I feel bash scripting is really unnecessary except in few situations like .bashrc or really quick onliners.

> How much of these is really still relevant

All of it? A very small chunk of these slides are even about bash scripts. Most of it is about standard unix commands and pipe and redirection operators.

I don’t think Python and bash solve the same problems at all. I don’t see how they could be seen as comparable, much less interchangeable.

If you're doing any filesystem based manipulation, then it's probably easier to use the shell and tradition Unix utilities. If you're manipulating files, the sort of thing where awk and sed start becoming important (particularly complex manipulations with awk and sed), you're probably better off with Python. Those use cases are pretty much why languages like Perl and Python came to be.

But even more powerful when working together....

You're comparing apples and oranges.

And did you read the slides? It's not about bash scripting, it's about using the command line competently and proficiently.

Python is not suitable for that at all.

That was what Perl was created for, but pretty much failed.

I feel that most of the tools mentioned here are quite outdated already and in 2019 alternatives should be recommended.

find -> fd

grep -> ripgrep

wget/curl -> httpie

awk/sed are just horrible to learn for beginners, would advise to just pick up python instead via xonsh[1] or pyp[2]

How about we don't teach new users these legacy tools and we can finally progress a little?

1 - http://xon.sh

2 - https://github.com/hauntsaninja/pyp

As much as I like httpie, in my view it's not a strict improvement like fd or ripgrep. This is unfortunately due to it being written in python.

Could you elaborate more why python is bad? If anything http ecosystem in python is exceptionally good.

I never said python is bad. What I mean is that httpie is not a strict improvement because it's written in python. Ripgrep for example is a strict improvement because it's safer, more performant, and has equivalent and extended features, without missing anything. The only advantage of grep over ripgrep is that it's much older and prevalent. With the exception of this advantage of age, one can say that X is a strict improvement over Y if it can do everything Y does but better, without doing anything worse.

Because HTTPie is written in python it has some disadvantages compared to curl or wget: it's not as performant, it's dependent on the Python interpreter, and less portable. If these don't matter to you, than you should use HTTPie. But it's not a strict improvement, so it's not something I can recommend to everyone, because some people will care about these things. Ripgrep and fd on the other hand I can recommend to everyone without any worry.

At what point you need computing performance on an I/O bound task?

The only performance that matters in http requests is async support. You can have 1000 pararel curl processes and will still be dusted by a single async python process.

That may be true, that has never matter to me. What has mattered however is portability. But it doesn't really matter what weaknesses there are, what matters is that they exist. Because they exist, HTTPie is not a strong of recommendation as ripgrep or fd.

You can think of it as a spectrum ranging from incomparable to strictly better. So you can't say that one should use Python over grep for example; you can't even compare the two because that doesn't make sense. They aren't even in the same space. Ripgrep however is a strict improvement over grep, it is faster, more featureful, safer, and has no downsides (with the exception of age, as I've mentioned).

I think HTTPie is very close to the end of the spectrum of being a strict improvement, but it's held back by various little things that don't matter to many but still matter.

Well I use it all the time and it's a huge improvement to my life. The elegant and simple syntax makes testing endpoints so much easier than the verbose curl and wget alternatives...

curl

   curl --header "Content-Type: application/json" -d '{"some": "content"}' <host>/<resource>

httpie

   http <host>/<resource> some=content

awk is much better than a full programming language for doing simple text manipulation. It needs no preliminaries like loading libraries, setting up the environment, etc. It is just a single purpose tool to manipulate text with minimum fuss.

> awk/sed are just horrible to learn for beginners

That doesn't seem fair. The two are entirely different. I found Awk very friendly and easy to learn, and use it every day for all kinds of things, in one-liners and in scripts. The AWK Programming Language is masterfully written, a joy to read, as are Arnold Robbins' books. Even the man page is short and very readable.

Reply on news.ycombinator.com