Unix commands I need often

1 Modernization

Why do we stick to the classic 80s Unix tools? mhoye recommends updating standard CLI utilities, as does Benjamin Pollack. Edited highlights from those posts dominate this page.

Improvements on “classic” tools and utilities:

• ripgrep, a line-oriented search tool that recursively searches the current directory for a regex pattern described as a better grep.

• sd, “Intuitive find & replace CLI (sed alternative)”

• fd, a “A simple, fast and user-friendly alternative to ‘find’”

  • Intuitive syntax: fd PATTERN instead of find -iname '*PATTERN*'.
  • Regular expression (default) and glob-based patterns.
  • Very fast due to parallelized directory traversal.
  • Uses colours to highlight different file types (same as ls).
  • Supports parallel command execution
  • Smart case: the search is case-insensitive by default. It switches to case-sensitive if the pattern contains an uppercase character*.
  • Ignores hidden directories and files, by default.
  • Ignores patterns from your .gitignore, by default.

• sharkdp/bat: A cat(1) clone with wings.

• The moreutils collection. Edited highlights:

  • chronic: runs a command quietly unless it fails
  • combine: combine the lines in two files using boolean operations
  • errno: look up errno names and descriptions
  • ifdata: get network interface info without parsing ifconfig output
  • ifne: run a program if the standard input is not empty
  • isutf8: check if a file or standard input is utf-8
  • lckdo: execute a program with a lock held
  • mispipe: pipe two commands, returning the exit status of the first
  • parallel: run multiple jobs at once
  • pee: tee standard input to pipes
  • sponge: soak up standard input and write to a file (allows us to pipe to and from the same file)
  • ts: timestamp standard input
  • vidir: edit a directory in your text editor
  • vipe: insert a text editor into a pipe
  • zrun: automatically uncompress arguments to command

• ripgrep / angle-grinder etc search through files

• atool, a set of scripts that wrap common compressed-file-format handlers.

• bat, a “better cat”.

• lsd and exa, both new takes on the venerable ls.

• There’s also zoxide: an interesting update to, of all things, cd.

• Broot: better navigation of directory trees.

• mcfly: McFly replaces your default ctrl-r shell history search with an intelligent search engine that takes into account your working directory and the context of recently executed commands. McFly’s suggestions are prioritised in real time with a small neural network.

There are many more under text data processing.

2 Resource monitoring

• htop, “a cross-platform interactive process viewer.” An htop-like utility called bottom also got some votes.

  As an aside, about htop: one commenter noted that they run HTOP on a non-interactive TTY, something like control-alt-F11; so do I, and it’s great, but you must not do this on sec-critical systems. You can kill processes through htop, and that gives you a choice of signals to issue, and on most machines running systemd systemd init responds to SIGTRMIN+1 by dropping back into rescue mode, and that’s a backstage pass to a root shell. I have used this to recover a personal device from an interrupted upgrade that broke PAM. You must never do this on a machine that matters.

• ncdu, friend of htop and a nice disk usage display for the terminal.

• bpytop: Linux/OSX/FreeBSD resource monitor

• duf a better df

• dust: “du on steroids.”

• duc, also a nice drive-use visualiser.

• procs: A modern replacement for ps written in Rust

• Tree: show you the tree structure of directories, a bit like microdosing on Midnight Commander from back in the day.

• Not really a new thing but a quality of life improvement: the “ducks” alias., “ducks: linux command for the 10 largest files in current directory”

3 Nice prompts

Try Starship: Cross-Shell Prompt.

4 Who owns this PID?

ps -o user -p $PID

5 WTF ^M?

Return produces ^M instead of a newline? Try:

stty sane^J

or possibly

stty icrn^J

For other miscellaneous screen garbage, maybe reset will help?

reset

It is more complicated if I made a mess in tmux.

As far as I am concerned this is all dark magic. If this nonsense wastes 2 hours of my life in total it will not justify the rather lengthier process of getting a better mental model of how terminals work.

6 File system watching

In Watch That Filesystem, Al Williams discusses how to trigger things based on filesystem events via inotifywait and incron.

Thanks Yohans Bastian for pointing out this handy trick to find the PID of the process using the inotify watchers.

find /proc/*/fd \
    -lname anon_inode:inotify \
    -printf '%hinfo/%f\n' 2>/dev/null \
    | xargs grep -c '^inotify' \
    | sort -n -t: -k2 -r

Cross platform filewatcher: emcrisostomo/fswatch:

A cross-platform file change monitor with multiple backends: Apple OS X File System Events, *BSD kqueue, Solaris/Illumos File Events Notification, Linux inotify, Microsoft Windows and a stat()-based backend.

7 Awk

pement.org awk one-liners

8 Rebuild on change

entr watches for changed files and recompiles the whatsit when it gets edited. This is incredibly useful for automating life.

9 Which file is crashing/hanging $PID?

lsof -r -p $PID | grep /path/to/file

10 Vars, expansions, file names, white space hell

See bash or fish depending on choice of shell.

Or avoid both by using some other utility, such as…

11 rename

rename is a script that makes renaming work how I imagine it should, avoiding the mysterious punctuation stew at least somewhat.

rename -s html txt *.html

NB depending on distro you may get some other systutil rename which is much less powerful than the one I linked to, albeit still often powerful enough.

12 Text processing

#!/bin/bash

# macOS version
sed -i '' -e’s/[[:space:]]*$//' "$1"

#!/bin/bash

# GNU version
sed -i -e’s/[ \t]*$//' "$1"

find . -type f -print0 | xargs -0 -I {} trimspace.sh \{\}

13 Incoming

find . -maxdepth 4 -depth -type d | shuf | rsync --delete -zHax --inplace --rsync-path='mkdir -p {} && rsync' ~/{}/ remote.host:/some/dir/{}/

for file in (find . -maxdepth 4 -depth -type d | shuf)
  mkdir -p "$file" && rsync --delete -Hax --inplace - ./{}/ /some/dir/{}/
end

curl --tlsv1.2 --proto=https --location \
  --remote-name-all --remote-header-name

find . -iname "*[üñàáèéöäçã]*"

14 Which process is bound to port $PORT?

This one!

lsof -nP -i4TCP:$PORT | grep LISTEN

test -d /Volumes/syncdrive/ && rsync --delete -avz \
    192.168.0.1:/path/to/stuff/ /Volumes/syncdrive/

for foo in *.zip
    unzip $foo -d (basename $foo .zip)
end

15 Meta command lines

Command lines to command your command lines.

I suppose if I were a good person I’d check if any of the above commands are novel and submit them to tldr or version my .cheats folder in a dotfile repo, or mention them in commandlinefu but I’m now exhausted and just want to go home.

## install
npm install -g tldr
# or
brew install tldr
# invoke
tldr tar

## install
brew install tealdeer
# invoke is still
tldr tar

## install
pip install cheat
# or
brew install cheat
# invoke
cheat tar

pip install howdoi
howdoi tar

16 Parallel execution

• Parallel shells with xargs: Utilize all your CPU cores on UNIX and Windows
• xargs options (GNU Findutils 4.9.0)
• Execute Shell loop in parallel, but only N workers