Linux

Useful commands

User management

Add/remove user: adduser [USER] / deluser [USER] & rm -r /home/[USER]
Add/remove group: adduser [USER] [GROUP] / groupdel [GROUP]
Migrate user:
- usermod -l [NEWUSER] [OLDUSER]
- usermod -d /home/[NEWUSER] -m [NEWUSER]
- groupmod -n [NEWUSER] [OLDUSER]
Change hostname: hostnamectl set-hostname [HOSTNAME]
Change owner of directory: sudo chown -R [USER]:[GROUP] [DIRECTORY]
Change permissions: sudo chmod [ugo+-rwx] / [664] [FILE/DIRECTORY]

File/Process management

Check SHA256 of file: sha256sum [FILE]
Set symbolic link: ln -s [SOURCE] [DEST]
See running processes: sudo ps aux
Kill processes: sudo kill -9 [PROCESS_ID] / sudo killall -9 [PROCESS_NAME]
Purge package + orphaned libraries: sudo apt remove --auto-remove [PACKAGE]
Check free disk space: df -h --total
Check disk usage: du -sh / du -h --max-depth=2 [DIRECTORY] / du -ha | grep -v "/sys/" | grep M
Run process in background: nohup [COMMAND] </dev/null >/dev/null 2>&1 & + Ctrl-C
Create ed25519 key: ssh-keygen -t ed25519 -C [COMMENT] -f [KEY_DEST]
Change key passphrase: ssh-keygen -o -p -f [KEY_FILE]
Generate public key: ssh-keygen -f [KEY_FILE] -y > [PUB_KEY_DEST]
Check disk partition type: df -Th
Check for bad blocks: badblocks -wsv -b 4096
Suspend + resume process in background: Ctrl-Z + jobs -l / bg [JOB_ID]
Create zip file: zip -r [ZIPFILENAME] [DIRECTORY]
Copy over SSH (with space): scp -rp user@example.com:"'web/tmp/Master File 13.xls'" ./
Run SSH agent in SSH shell: eval $(ssh-agent)
Change all files(f) / directories(d) to permissions: find ./ -type f -exec sudo chmod 644 {} \;
Send SIGKILL to process name: -kill -9 $$(ps aux | grep "[PROCESSNAME]" | grep -v "grep" | grep -oP '\d+' | head -n1)
Batch renaming using regex: rename -v -n "s/[SEARCH](.*)/[REPLACE]\1/" [WILDCARD]
Kill SSH sessions: who -a, ps aux | egrep "sshd: [a-zA-Z]+@", kill [PID]
Peek at file descriptors of process: sudo strace -p <PID> -s9999 -e write
Search for text: grep -RInwe <TEXT>
Check last journalctl logs: journalctl -b -1 -e

File copy hacks that track progress, from most GNU-compatible to least (i.e. need to install new libraries):

user:~$ curl -o [DST] file://[SRC]               # does not copy permissions
user:~$ dd if=[SRC] of=[DST] status=progress     # does not copy permissions
user:~$ rsync -ah --info=progress2 [SRC] [DST]
user:~$ pv [SRC] > [DST]                         # does not copy permissions
user:~$ cp [SRC] [DST]; progress -w              # monitors existing, but needs tty

BTRFS

On BTRFS partitions, need to be careful to avoid double-counting of disk space, since there is shared memory. Typically interface disk commands through btrfs utility.

List snapshot filenames: sudo du --max-depth=1 -h /.snapshots/
Check total and shared memory for snapshots: sudo btrfs filesystem du -s /.snapshots/*
Check disk btrfs-allocation and free space: sudo btrfs fi usage /

For rescuing disk full problems, see here.

Networking

Find open ports: netstat -tuplen
Find public IP address: curl --interface [INTERFACE] ifconfig.me
- icanhazip.com seems unstable as of Mar 24.
Check local ports: lsof -nPi
Check current IP address: ip addr
Download file from URL: wget -o [FILENAME] [URL]
Bring network interface down: sudo ip link set dev [INTERFACE] down
Change MAC address: sudo ip link set dev [INTERFACE] address [MAC]
Verify firewall hole-punched:
- Server: nc -kls 10.10.10.3 -p 4444
- Client: echo "test" | nc 10.10.10.3 4444

Webservice

Check Nginx syntax: sudo nginx -t
Reload Nginx webserver: sudo systemctl reload nginx
Check Nginx logs: cat /var/log/nginx/error.log | less / tail -n10
Edit PHP initialization: sudo vim /etc/php/[VER]/fpm/php.ini
Restart PHP server: sudo systemctl restart php[VER]-fpm
Check available PHP modules: php -m | grep [MODULE_NAME]
Start MySQL/MariaDB server: sudo /etc/init.d/mysql start
Login MySQL/MariaDB server: sudo mysql -uroot -p (no password as root)
Create certificates with Let's Encrypt: sudo certbot --nginx
Create P12 certificate: openssl pkcs12 -export -out [P12_NAME] -inkey [KEY_FILE] -in [CA_CHAIN_FILE]
Check cert and key aligns: openssl s_server -key key.pem -cert cert.pem

Docker

Start docker service: sudo systemctl start docker
Check docker service is up: sudo systemctl status docker
Check docker services: docker ps
Build docker image: docker build --network host --no-cache --rm -t [AUTHOR]/[NAME]:[TAG] .
Check docker statistics: docker info
Enter container: docker exec -it [CONTAINER] sh
Print logs: docker logs -f [CONTAINER]
Find dangling images: docker images -f "dangling=true" -q
Remove dangling images: docker rmi $(docker images -f "dangling=true" -q )
Rename image: docker image tag [IMAGE_ID] [ORG]/[NAME]:[TAG]
Prune everything: docker system prune --volumes
Prune volumes: docker volume prune -f
Find location that compose was started from: docker inspect [NAME] | grep "working_dir"

Docker Compose

Start docker compose (in detached mode): docker-compose up -d [CONTAINER...]
Stop docker compose + remove images: docker-compose down --rmi all

Git

Initial setup

Change local username / email: git config user.email [EMAIL] or user.name [NAME]
Change default editor: git config --global core.editor vim

Small editing and cleanups

Reset author: git commit --amend --reset-author --no-edit
Set author: git commit --amend --author "[NAME] <[EMAIL]>" --no-edit
Clean all: git reset --hard && git clean -dfx

Patching

Create a patch: git diff > [FILE]
Apply a patch: git apply --whitespace=fix --reject [FILE]
Apply rejected patches: wiggle --replace [FILE] [FILE.rej]

Changing history

Interactive add: git add -p .
Rebase:
- git rebase -i [COMMIT/--root]
- Change pick to edit, modify commit
- git rebase --continue
Insert changes to previous commits:
- git commit --fixup=[COMMIT]
- git stash
- git rebase -i --autosquash [COMMIT]^
- git stash pop
Enable autosquashing by default: git config --global rebase.autoSquash true

Signing

By default, GPG keys are used for signing commits (manually using -S flag). As of Git 2.34 (Nov 2021), SSH keys can also be used to sign commits.

Enable commit signing: git config --global commit.gpgsign true
Use SSH key for signing: git config --global gpg.format ssh
Specify signing key (GPG/SSH): git config --global user.signingkey [PUB_KEY]
Verify commit has been signed: git log --show-signature ("No signature" for SSH)

Misc

Beautify git log: git config --global alias.adog "log --all --decorate --oneline --graph"
Check branches with latest commits: git branch -a --sort=-committerdate
Manually delete detached commits (warning! stashes will be deleted): git reflog expire --expire-unreachable=now --all && git gc --prune=now
Subversion integration: apt install git-svn
History: gitk

Others

Create random string: openssl rand -hex 12
Test fileshare write speed: time dd if=/dev/zero of=/path/to/nfs/testfile bs=16k count=64k
Edit user sudoer rule file: sudo visudo -f [USER]
- Change sudo access timeout in mins: Defaults timestamp_timeout 15
Change console font:
- Ubuntu: sudo dpkg-reconfigure console-setup
- OpenSUSE, bashrc: setfont latarcyrheb-sun32, listed in /usr/share/kbd/consolefonts/
Trim bytes: head -c [NUM_BYTES] [FILENAME]
Show manually installed packages: sudo apt-mark showmanual
Show existing ufw rules while inactive: ufw show added

Create PDF pages:

# Create blank page using Ghostscript
# '/etc/ImageMagick-6/policy.xml' contains a policy setting rights
# to "none" for pattern "PDF" under the coder domain. This should be disabled
# (by commenting out) in order to use ghostscript for pdf. Note security concerns.
user:~$ convert xc:none -page A4 blank.pdf
user:~$ convert xc:none -page 842x595 blank.pdf  # horizontal
 
# Create blank page using 'ps2pdf'
user:~$ echo "" | ps2pdf -sPAPERSIZE=a4 - blank.pdf
 
# Append blank page, uses 'pdftk'
# Converts 'file.pdf' -> 'notes_file.pdf'
user:~$ export FILE=file.pdf
user:~$ pdftk A=$FILE B=blank.pdf cat A1-end B output notes_$FILE.pdf
 
# Interleave blank pages, uses 'pdftk'
user:~$ export FILE=file.pdf
user:~$ export NUM=`pdftk $FILE dump_data | grep NumberOfPages | awk '{print $2}'`
user:~$ pdftk blank.pdf cat $(printf '1 %.0s' $(seq $NUM)) output multiblank.pdf
user:~$ pdftk A=$FILE B=multiblank.pdf shuffle A B output notes_$FILE

> time dd if=/dev/zero of=/path/to/nfs/testfile bs=16k count=64k
65536+0 records in
65536+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 91.903 s, 11.7 MB/s

real	1m31.907s
user	0m0.035s
sys	0m1.307s


> time dd if=/path/to/nfs/testfile of=/dev/null bs=16k count=64k
65536+0 records in
65536+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 91.6337 s, 11.7 MB/s

real	1m31.639s
user	0m0.075s
sys	0m1.196s

Basic systemctl service file:

/etc/systemd/system/{{NAME}}.service

[Unit]
Description={{DESCRIPTION HERE}}
Wants=network-online.target
After=network-online.target
 
[Service]
User=root
Group=root
Type=simple
ExecStart={{PROGRAM HERE}}
 
[Install]
WantedBy=multi-user.target

Keeping pipes open while writing:

mkfifo mypipe
exec 3>mypipe  # open file descriptor with writes piped to mypipe
echo "stuff" >> mypipe
echo "morestuff" >> mypipe
exec 3>&-  # close file descriptor, which closes mypipe

Bash

Use Ctrl-R to invoke command searching (as opposed to the bang-expansion method that immediately invokes the command). Other controls:

Ctrl-A and Ctrl-E to move to front / back
Ctrl-K to delete from cursor onwards

~/.bashrc

# ~/.bashrc: executed by bash(1) for non-login shells.
 
# If not running interactively, don't do anything
case $- in
    *i*) ;;
      *) return;;
esac
 
# don't put duplicate lines or lines starting with space in the history.
# See bash(1) for more options
HISTCONTROL=ignoreboth
 
# append to the history file, don't overwrite it
shopt -s histappend
 
# for setting history length see HISTSIZE and HISTFILESIZE in bash(1)
HISTSIZE=1000
HISTFILESIZE=-1
 
# enable color support of ls and also add handy aliases
if [ -x /usr/bin/dircolors ]; then
    test -r ~/.dircolors && eval "$(dircolors -b ~/.dircolors)" || eval "$(dircolors -b)"
    alias ls='ls --color=auto'
    #alias dir='dir --color=auto'
    #alias vdir='vdir --color=auto'
 
    alias grep='grep --color=auto'
    alias fgrep='fgrep --color=auto'
    alias egrep='egrep --color=auto'
fi
 
# colored GCC warnings and errors
export GCC_COLORS='error=01;31:warning=01;35:note=01;36:caret=01;32:locus=01:quote=01'
 
# some more ls aliases
alias ll='ls -alF'
alias la='ls -A'
alias l='ls -CF'
alias ..='cd ..'
 
# Add timestamps to 'history' command
HISTTIMEFORMAT="%F %T  "
 
# Allow alias execution in sudo, e.g. "sudo ll"
# see https://askubuntu.com/a/22043
#   If the last character of the alias value is a space or tab character,
#   then the next command word following the alias is also checked for
#   alias expansion.
alias sudo='sudo '
 
# Go to latest directory
cdl() {
    cd  "$(\ls -1dt ./*/ | head -n 1)" && ls -rt
}

~/.inputrc

# Enables better *-wildcard completion
set show-all-if-ambiguous on
"\t": glob-complete-word

Vim

~/.vimrc

" Load internal configuration as well, if exists
" To see what was loaded, use :script
runtime defaults.vim
 
" Enable the FileType plugin
filetype plugin indent on
 
" Defaults for unknown files
set tabstop=4 " ts: show existing tab with 4 spaces width
set softtabstop=4 " sts: Sets the number of columns for a TAB
set shiftwidth=4 " sw: when indenting with '>', use 4 spaces width
set expandtab " On pressing tab, insert 4 spaces
" Use :retab to convert existing tabs to space
 
" EXCEPTIONS
" To see list of filetypes available, type ':setfiletype ' and Ctrl-D
" yaml, especially for docker-compose
autocmd FileType yaml setlocal ts=2 sts=2 sw=2 expandtab
 
" Enable relative line numbers
set relativenumber " rnu: add left-margin with line numbers relative to cursor
set number " replace cursor line number from 0 to absolute number
 
" Optional: Set vim colorscheme
colorscheme codedark
 
" Highlight git merge conflicts
" Adapted from: <https://vi.stackexchange.com/a/19760>
function! ConflictsHighlight() abort
    syn region conflictStart start=/^<<<<<<< .*$/ end=/^\ze\(=======$\||||||||\)/
    syn region conflictMiddle start=/^||||||| .*$/ end=/^\ze=======$/
    syn region conflictEnd start=/^\(=======$\||||||| |\)/ end=/^>>>>>>> .*$/
 
    highlight conflictStart ctermbg=red ctermfg=black
    highlight conflictMiddle ctermbg=blue ctermfg=black
    highlight conflictEnd ctermbg=green cterm=bold ctermfg=black
endfunction
 
augroup MyColors
    autocmd!
    autocmd BufEnter * call ConflictsHighlight()
augroup END

Clean trailing whitespaces: :%s/\s\+$//e
Change to script directory: cd "$(dirname "$0")"

Others

For VSCode dark mode color scheme:

# User colorscheme
mkdir -p ~/.vim/pack/themes/start
git clone https://github.com/tomasiser/vim-code-dark ~/.vim/pack/themes/start/vim-code-dark

# System colorscheme
# Use the target location: "/usr/share/vim/vim{{VERSION}}/pack/themes/start"
sudo mkdir -p /usr/share/vim/vim82/pack/themes/start
git clone https://github.com/tomasiser/vim-code-dark /usr/share/vim/vim82/pack/themes/start/vim-code-dark

Other color schemes.

Rerun script

rerun.sh

#!/usr/bin/env bash
# See documentation below.
#
# Changelog:
#   2023-10-01 Justin: Init from some SO/Gist answer
#   2023-11-14 Justin: Added recursive option
 
# Events that occur within this time from an initial one are ignored
cooldown_s='0.250'
clear=''
verbose=''
recursive=''
 
# check dependencies
if ! command -v inotifywait >/dev/null
then
    echo "error: inotifywait: command not found. hint: install inotify-tools"
    exit 1
fi
 
function usage {
    echo "Rerun a given command every time filesystem changes are detected."
    echo ""
    echo "Usage: $(basename $0) [OPTIONS] COMMAND"
    echo ""
    echo "  -c, --clear     Clear the screen before each execution of COMMAND."
    echo "  -r, --recursive Watch directories for changes recursively."
    echo "  -v, --verbose   Print the name of the files that changed to cause"
    echo "                  each execution of COMMAND."
    echo "  -h, --help      Display this help and exit."
    echo ""
    echo "Run the given COMMAND, and then every time filesystem changes are"
    echo "detected in or below the current directory, run COMMAND again."
    echo "Changes within $cooldown_s seconds are grouped into one."
    echo ""
    echo "This is useful for running commands to regenerate visual output every"
    echo "time you hit [save] in your editor. For example, re-run tests, or"
    echo "refresh markdown or graphviz rendering."
    echo ""
    echo "COMMAND can only be a simple command, ie. \"executable arg arg...\"."
    echo "For compound commands, use:"
    echo ""
    echo "    rerun bash -c \"ls -l | grep ^d\""
    echo ""
    echo "Using this it's pretty easy to rig up ad-hoc GUI apps on the fly."
    echo "For example, every time you save a .dot file from the comfort of"
    echo "your favourite editor, rerun can execute GraphViz to render it to"
    echo "SVG, and refresh whatever GUI program you use to view that SVG."
    echo ""
    echo "COMMAND can't be a shell alias, and I don't understand why not."
}
 
while [ $# -gt 0 ]; do
    case "$1" in
      -c|--clear) clear='1';;
      -v|--verbose) verbose='1' ;;
      -r|--recursive) recursive='--recursive';;
      -h|--help) usage; exit;;
      *) break;;
    esac
    shift
done
 
function execute() {
    [ -n "$clear" ] && clear
    [ -n "$verbose" ] && echo "$@"
    "$@"
}
 
execute "$@"
end_of_cooldown="$( date +%s%N )"
cooldown_ns="$( LC_ALL=C printf '%0.9f' "$cooldown_s" | sed -r -e 's|\.||' -e 's|^0+||' )"
 
inotifywait --quiet $recursive --monitor --format '%e %w%f' \
    --event='modify,close_write,move,create,delete' \
    --exclude='\.git|\..*\.swp|__pycache__|\.cache|\.pytest_cache' \
    . | while read changed
do
 
    now="$( date +%s%N )"
    if (( now > end_of_cooldown )); then
        end_of_cooldown="$(( now + cooldown_ns ))"
        ( sleep "$cooldown_s" && execute "$@" ) &
    fi
 
    if [ -n "$verbose" ]; then
        echo "$changed"
    fi
 
done