The Convenience Store

Ever get tired of constantly Googling every time you have to install or set up something? I do.

Linux packages or Windows programs or script-based installations, remember what to do is a pain in the ass and an overall crappy DX, as you are constantly ripped out of your flow to get the tools necessary to work. And let’s say you have just bought a brand spanking new computer or setting up a new VM, the momentous urge of diving right into creating things as fast as possible is fleeting every second you spend gathering the dev requirements and dependencies.

That is why I started this convenience store. To add some nitrous 🚀 to [y]our DX.

Security Acknowledgement

While I have high confidence in the platform and networking security of the host for this book, it does not guarantee any type of attack occurring by any means that will harm your system and expose your information.

An example of an attack scenario may be: man-in-the-middle attack while using curl to download and execute the script provided. The attacker might redirect your client to a unaffiliated and malicious copy of this website that download and execute instead a contaminated script.

⚠️ It is crucial that you inspect the script first via the provided link to the static resource.

⚠️⚠️ While I try my best to vet and securely configure the backend, some attacks may happen at your layer, so my disclaimer here is that I will not be held accountable for injuries or damages caused by running the scripts haphazardly.

Contributing

The documentation is not only for me, it’s also for the community!

If you know or believe there is a better way to do things, like a faster and more secure script, fork the repository https://github.com/aaanh/script-convenience-store, make changes, and open a pull request.

CI/CD

The Rust book is compiled with Github Actions, committed to a production branch, and deployed (with that branch) on Vercel as a static HTML site.

About the Author

Anh (@aaanh).

Homepage: https://aaanh.com

CNAME: Only https://script.aaanh.app as of September 2023

Quick links

Script files

Nerd/Powerline Fonts

Ubuntu Out-of-the-Box Experience

Convenience Script

Static file: /static/scripts/ubuntu-oobe.sh

sudo apt install -y curl && curl -O https://script.aaanh.app/static/scripts/ubuntu-oobe.sh && bash ./ubuntu-oobe.sh

If curl is not installed, install it with sudo apt install curl.

What this script does?
  • Use apt to update and upgrade
  • Install packages: git, zsh, net-tools, build-essential, powerline, fonts-powerline, vim, openssh-server, tmux, python3, python-is-python3 python3-pip.
  • Set up and configure zsh, oh-my-zsh, zsh-highlighting, zsh theme.

Dev Tools

sudo apt install -y curl && curl https://script.aaanh.app/static/scripts/ubuntu-dev-tools.sh | bash

Manual

  1. Update and upgrade apt

    sudo apt update && sudo apt upgrade

  2. Install commonly-used packages

    sudo apt install git zsh net-tools build-essential powerline \
  fonts-powerline vim openssh-server \
  tmux python3 python-is-python3 python3-pip curl

  3. Additional optional desktop packages

    sudo apt install gnome-tweaks grub-customizer nodejs npm

  4. Install and power up vim

    git clone https://github.com/aaanh/vimrc ~/.vim_runtime && cd ~/.vim_runtime && ./install_awesome_vimrc.sh

  5. Use local time (optional, for dual boot with Windows)

    timedatectl set-local-rtc 1 --adjust-system-clock

  6. Github CLI

    curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg | sudo dd of=/usr/share/keyrings/githubcli-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | sudo tee /etc/apt/sources.list.d/github-cli.list > /dev/null
sudo apt update
sudo apt install gh

    After installation, login with gh auth login

  7. Additional user applications

    1. Visual Studio Code: https://code.visualstudio.com/download

    2. Discord: https://discord.com/download

    3. Anaconda: https://www.anaconda.com/products/individual

    4. Zoom: https://zoom.us/download

    5. Slack (snap or rpm): https://slack.com/downloads/linux

  8. Solid black 4K resolution wallpaper 👌

    curl -o ~/Pictures/black-solid-4k.png "https://script.aaanh.app/static/images/black-solid-4k.png"

Kernel-based Virtual Machine

Partially adapted from https://ubuntu.com/blog/kvm-hypervisor, https://developer.hashicorp.com. And with references to Red Hat article(s).

Introduction

Refer to Red Hat’s article.

Prerequisites

sudo apt install -y qemu qemu-kvm libvirt-daemon bridge-utils virt-manager libvirt-clients

Start a VM

  1. Create empty image

    Synopsis:

    qemu-img create -f <format> <vm-disk-name>.img <size><unit>

    Example:

    qemu-img create -f qcow2 some-vm.img 60G

  2. Boot from CDROM/ISO image

    Synopsis:

    qemu-system-x86_64 -m <mem_size> -boot d -enable-kvm -smp 3 \
                     -net nic -net user -hda <vm-disk-name>.img -cdrom /path/to/downloaded/iso.iso

    Example:

    qemu-system-x86_64 -m 2048 -boot d -enable-kvm -smp 3 \
                     -net nic -net user -hda some-vm.img -cdrom ubuntu-22.04-amd64.iso

  3. Go through the installation process of the respected distro onto the created VM image.

  4. Shut down the VM.

  5. Boot from the image with the installed OS

    Synopsis:

    qemu-system-x86_64 -m <mem_size> -boot d -enable-kvm -smp 3 \
                    -net nic -net user -hda <vm-disk-name>.img

    Example:

    qemu-system-x86_64 -m 2048 -boot d -enable-kvm -smp 3 -net nic -net user -hda some-vm.img

Management

Virtual Machine Manager

sudo apt install virt-manager

GUI for easier management.

Vagrant

Infrastructure-as-Code implementation of VM and container management by HashiCorp.

wget -O- https://apt.releases.hashicorp.com/gpg | gpg --dearmor | sudo tee /usr/share/keyrings/hashicorp-archive-keyring.gpg && echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list && sudo apt update && sudo apt install vagrant

Terraform

Infrastructure-as-Code implementation of VM, container, network stackc management also by HashiCorp.

  • Installation
wget -O- https://apt.releases.hashicorp.com/gpg | \
    gpg --dearmor | \
    sudo tee /usr/share/keyrings/hashicorp-archive-keyring.gpg && gpg --no-default-keyring \
    --keyring /usr/share/keyrings/hashicorp-archive-keyring.gpg \
    --fingerprint && echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] \
    https://apt.releases.hashicorp.com $(lsb_release -cs) main" | \
    sudo tee /etc/apt/sources.list.d/hashicorp.list && sudo apt update && sudo apt install terraform

RHEL/Fedora/CentOS Out-of-the-Box Experience

Note: Cool thing about Fedora is that most dev-centric packages are already installed :)

Another one: These commands are exclusively tested out on Fedora, but probably are transferable to RHEL and CentOS as well.

Convenience script

Out of the box experience

Static file: /static/scripts/fedora-oobe.sh

⚠️ This OOBE is now highly opinionated since I am daily driving it. Please refer to the list of tasks that will be executed when running this script.

sudo dnf install -y curl && curl -O https://script.aaanh.app/static/scripts/fedora-oobe.sh && bash ./fedora-oobe.sh

Actions performed

  1. Run dnf update
  2. Install zsh, wget, git, neovim, tmux
  3. Change default shell to zsh and install oh-my-zsh
  4. Install node version manager (nvm)
  5. Install Golang using 3rd party script
  6. Install Rust using official rustup toolchain
  7. Install Github CLI gh
  8. Install Docker Engine
  9. Install Homebrew package manager
  10. Install Visual Studio Code
  11. Install a Powerline font called Fantasque Sans Mono

Development tools

It is recommended to run the OOBE script first to get the dependencies.

Static file: /static/scripts/fedora-dev-tools.sh

curl https://script.aaanh.app/static/scripts/fedora-dev-tools.sh | bash

Update

sudo dnf update

Install common applications

For general user

sudo dnf -y install zsh wget curl neovim

For developers

sudo dnf -y install nodejs gcc-c++

Github CLI

sudo dnf install 'dnf-command(config-manager)'
sudo dnf config-manager --add-repo https://cli.github.com/packages/rpm/gh-cli.repo
sudo dnf install gh
gh auth login

Docker

sudo dnf -y install dnf-plugins-core

sudo dnf install 'dnf-command(config-manager)'
sudo dnf config-manager --add-repo https://cli.github.com/packages/rpm/gh-cli.repo
sudo dnf install gh

Set up zsh

Change default shell using usermod instead because chsh is not installed by default

sudo usermod -s /usr/bin/zsh "$(whoami)"
git clone https://github.com/robbyrussell/oh-my-zsh.git ~/.oh-my-zsh
cp ~/.oh-my-zsh/templates/zshrc.zsh-template ~/.zshrc
git clone https://github.com/zsh-users/zsh-syntax-highlighting.git "$HOME/.zsh-syntax-highlighting" --depth 1
echo "source $HOME/.zsh-syntax-highlighting/zsh-syntax-highlighting.zsh" >> "$HOME/.zshrc"
sed -i 's/robbyrussell/flazz/g' ~/.zshrc

Set up vim

git clone https://github.com/aaanh/vimrc ~/.vim_runtime && cd ~/.vim_runtime && ./install_awesome_vimrc.sh

Ansible

TBA

// this is an example typescript snippet to demo syntax highlighting
import { type Uri } from "uri";

type ExampleParams = {
	paramOne: string | undefined;
	paramTwo: number | 69;
	paramThree: Uri | "https://example.com";
};

function toString(obj: any): string {
	throw new Error("Function not implemented.");
}

const Example = (params: ExampleParams[]): string => {
	let concat = params.paramOne + toString(params.paramTwo) + toString(params.paramThree);
	return concat;
};

export default Example;

Arch Linux Bare Installation

Notes

This is an abridged version based on https://wiki.archlinux.org/title/installation_guide and https://linuxiac.com/arch-linux-install/. The purpose is to create a dumbed down guide that blasts past whatever precautions and warnings on the wiki. I. AM. SPEED. (Though is not gonna be as fast as this guy https://www.youtube.com/watch?v=8utpbbdj0LQ)

Warning

Make sure you have backed up all data before proceeding. This includes: your USB devices and your system disks.

Disclaimer: Proceed with the setup steps with extreme cautions. You must know what you are doing. I am not held responsible for your negligence.

Download .iso image

https://archlinux.org/download/

Example download script (using University of Waterloo repository mirror)

cd ~/Downloads && curl -O http://mirror.csclub.uwaterloo.ca/archlinux/iso/2022.02.01/archlinux-2022.02.01-x86_64.iso

Create the USB installation medium with dd

dd if=/path/to/archlinux.iso of=/dev/somedisk bs=4M conv=fsync oflag=direct status=progress

Example:

dd if=~/Downloads/archlinux-2022.02.01-x86_64.iso of=/dev/sdb bs=4M conv=fsync oflag=direct status=progress

Note: Use lsblk to list the disks mounted on your system. A USB device should typically be in the form of sdx.

Restart and Boot into the USB

  • Lenovo: Enter → F12
  • Dell: F12
  • ASUS motherboard: Delete or F2

After a bit of initializations, you should now be staring at the live boot TTY command line as root, denoted by #.

Networking

Check the interfaces

ip link

If interface, wlan0 for example, is shown as DOWN, do:

sudo ip link set wlan0 up

Wireless (from Arch Linux Wiki)

iwctl

device list

station device scan

station device get-networks

station device connect SSID

Wired

Should run without much configurations. Typically, installation occurs in a DHCP environment.

For static IP configurations

  • IP assignment

    ip address show

    ip address add address/prefix_len broadcast + dev interfaces

  • Routing table

    ip route show

    ip route add PREFIX via address dev interface

Update pacman repository listing

```sh
pacman -Syy
```

Create new partition table

Make sure you have backed up all data before proceeding. These steps assume that you are using a whole disk as installation target.

  • List available disks

    fdisk -l

  • Create partition table

    cfdisk /dev/sdx

  • If prompted for label type, select gpt.

  • Delete all partitions.

Partition table should be:

PartitionTypeSizeFormat
/dev/sda1EFI System Partition512MFAT32
/dev/sda2Linux Swap4G
/dev/sda3Linux PartitionEnterEXT4
  • Write changes to disk.

Formatting the partitions

  • EFI System:

    mkfs.fat -F32 /dev/sda1

  • Linux Swap:

    mkswap /dev/sda2
swapon /dev/sda2

  • Linux Partition /:

    mkfs.ext4 /dev/sda3

Install Arch Linux

  • Mount root partition on system disk to live boot environment

    mount /dev/sda3 /mnt

  • pacstrap-ing

    pacstrap /mnt base linux linux-firmware sudo vim

If PGP signature is not trusted, keyring needs to be updated.

```
pacman -Sy archlinux-keyring
```

System Configurations

  • fstab

    genfstab -U /mnt >> /mnt/etc/fstab

  • Change root into the installed system

    arch-chroot /mnt

  • Timezone

    ln -sf /usr/share/zoneinfo/America/New_York /etc/localtime

  • Locale

This sed script needs to be verified!!

```sh
sed -i 's/#en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen
sed -i 's/#en_US ISO-8859-1/en_US ISO-8859-1/' /etc/locale.gen
```

Or manually:

```
vim /etc/locale.gen
```

  • Locale (cont.)

    locale-gen
echo LANG=en_US.UTF-8 > /etc/locale.conf
export LANG=en_US.UTF-8

  • Hostname

    echo "your-pc-name-of-choice" > /etc/hostname

    echo -e "127.0.0.1\tlocalhost\n::1\t\tlocalhost\n127.0.1.1\tyour_pc_name" > /etc/hosts

Should look something like this

# /etc/hosts
127.0.0.1   localhost
::1         localhost
127.0.1.1   your-pc-name

  • Set root password

    passwd

Install grub bootloader

pacman -S grub efibootmgr os-prober mtools

mkdir /boot/efi
mount /dev/sda1/ /boot/efi
grub-install --target=x86_64-efi --bootloader-id=grub_uefi
grub-mkconfig -o /boot/grub/grub.cfg

Install Desktop Environment of choice

Prerequisites

  • Install Xorg

    pacman -S xorg-server xorg-apps

  • Install GPU drivers

    # NVIDIA
pacman -S nvidia-utils

# AMD
pacman -S xf86-video-ati

# Intel
pacman -S xf86-video-intel

Desktop Environments

Not all DE’s are covered here. Only the 2 most used: gnome and kde. For other DE’s, the Arch wiki covers them all.

  • GNOME

    pacman -S gnome gnome-extra networkmanager
systemctl enable gdm
systemctl enable NetworkManager # THIS SHIT IS CASE-SENSITIVE

  • KDE

    pacman -S xorg plasma plasma-wayland-session kde-applications networkmanager
systemctl enable sddm.service
systemctl enable NetworkManager # THIS SHIT IS CASE-SENSITIVE

Create a user account

useradd -m -G wheel account_name
passwd account_name
EDITOR=vim visudo
  • Find and uncomment the line # %wheel ALL=(ALL) ALL

In vim Normal Mode, type /%wheel ALL=(ALL) to search for that line. If you feel lazy, in Normal Mode, type :set ignorecase to ignore case for search query.

Finalization

exit
umount -R /mnt
reboot

After the reboot, it should boot straight into the Arch installation on your system (assuming you have only 1 disk, 1 OS installed, lol).

Manjaro OOBE

Convenience Script

sudo pacman -Sy curl && curl -O https://script.aaanh.app/static/scripts/manjaro-oobe.sh && bash ./manjaro-oobe.sh

Manual

Manual steps

  1. Update and choose optimal mirror for pac(kage)man(ager)

    sudo pacman -Syyu

  2. Install build tools for AUR

    sudo pacman -Sy base-devel git

  3. Install commonly-used packages

    sudo pacman -Sy ttf-fira-code ibus-unikey zsh discord

  4. Install packages from AUR

  • Browse AUR here https://aur.archlinux.org/

  • Clone repository

    git clone <repository>

    cd <repository>

  • Compile, install with dev dependencies, remove or clean up build and dev dependencies.

    makepkg -sri

  1. Clone AUR repos and Install Script

    cd ~/Downloads\
(git clone https://aur.archlinux.org/google-chrome.git && cd google-chrome && makepkg -sri ./google-chrome/)\
(git clone https://aur.archlinux.org/visual-studio-code-bin.git && cd visual-studio-code-bin && makepkg -sri ./visual-studio-code-bin/)

System Operations

Disable app armor

Note: Applications like Docker Desktop would not start if app armor is enabled

echo 'kernel.apparmor_restrict_unprivileged_userns = 0' |
  sudo tee /etc/sysctl.d/20-apparmor-donotrestrict.conf
kernel.apparmor_restrict_unprivileged_userns = 0

Font installation

Convenient script. I know I wrote Fedora but actually works for Debian-based as well, lol. Source: Trust me bro.

curl -O https://gist.githubusercontent.com/aaanh/2477c68314a144ee76de6858a57bed36/raw/fcd6a2dbbdc034f5f22c99c8085a1d6e575d293d/install-fonts-fedora.sh

System Information

ls*

  • lspci
  • lscpu
  • lsblk

neofetch

Resource Monitoring

top or htop

For continuously querying resource usage. Simply open a terminal and run top.

htop has pretty formatting, not installed by default. Can be installed through common package managers.

Logs

journalctl

For systemd-based Linux operating system and/or distributions.

dmesg

Kernel-level logs.

/var/log/messages

This is where system log messages are stored in plain text.

File Operations

File Transfer

curl

  • Download and Save file with name extracted from the URL

    curl https://example.com/my-file.sh -O

  • Download and Save file with specified name

    curl https://example.com/my-file.sh -O my-downloaded-file.sh

rsync

  • Commonly used command

    rsync -avzP /PATH/TO/LOCAL/MY_FILE_FOLDER/ user@destination_machine:~/SOME_PARENT_FOLDER/

Explanation: (a)rchive, (v)erbose, (z)ip, (P)artial/(P)rogress. SOURCE is the first argument, DESTINATION comes second. So, to copy from a remote machine, simply swap the arguments above.

  • Example 
    rsync -avzP ~/Documents/assination_orders/ account_payables@[email protected]:~/Documents/orders_to_execute/

Archival, compression, extraction

tar

  • Create

    tar zcvf target.tar.gz sourcefile

  • Unpack

    tar zxvf sourcefile

  • List

    tar ztvf sourcefile

(un)zip

  • Create zip archive from files

    zip -r <target> <source_folder>

  • Extract/expand zip archive

    unzip <source_archive>

Finding and Searching

grep

find

Network Operations

SSH

  • Simple ssh connection

    ssh username@hostname

  • Tunneling

    ssh username@server_1 -N -f -L local_port:server_2:remote_port

  • Relay with port-forwarding

    Synopsis:

    ssh -L port_a:localhost:port_b username@server_1 sshpass -p password_for_server_2 ssh -L port_b:localhost:port_c -N username@server_2

    Example:

    ssh -L 3333:localhost:4444 [email protected] sshpass -p s0mePa$$w0rd ssh -L 4444:localhost:3389 -N [email protected]@server-2.example.com

    Note: This scenario was tested with a 3 machines. The physical Windows machine with ssh (client), a Debian server (server_1) on Linode VPS, and a Windows RDP host (server_2). server_1 and server_2 have VPN tunneling with Tailscale.

    In order for this to work, server_2 must have OpenSSH Server service enabled and running. server_1 must have sshpass installed, the in and out ports whitelisted with ufw.

    This is just a makeshift method in desperate times. Looking for better ways in the future.

Firewall

ufw

Might need to install first on some distributions or barebone server images.

It is basically a convenient wrapper for iptables and netfilter. Hence, (U)ncomplicated(F)ire(w)all.

⚠️ Proceed with caution. Incorrectly configuring your system firewall might result in massive failures on dependent systems and network nodes.

  • Show status

    sudo ufw status

  • Enable/Disable

    sudo ufw enable # or disable

Swap allow with deny for the opposite effect.

  • Allow port for both tcp and udp.

    ufw allow 6969

  • Allow common protocol by name for only tcp.

    sudo ufw allow http/tcp

  • Allow source and destination IP subnet range with specified protocol and port range.

    sudo ufw allow proto tcp from 10.0.0.1/24 to 192.0.0.2/28 port 69:420

Information Query

traceroute/tracert (Trace packets route)

  • Default and simple

    traceroute <IP | FQDN>

  • Specify max number of hops

    traceroute -m <int> <IP | FQDN>

  • Example

    traceroute -m 42069 google.com

dig (DNS lookup)

dig <IP | FQDN>

nslookup (Nameserver lookup)

Most effective when used to look into a certain web address, e.g. google.com, rather than an IP address.

nslookup <FQDN | IP>

ip/ipconfig/ifconfig

  • List network interfaces with status, Physical MAC address, IP address, and subnet.

    ip address

Virtual Private Network (VPN)

Tailscale

Tailscale is a VPN service that uses WireGuard protocol. If you are looking for a fully OSS alternative to Tailscale, you can check out Headscale instead.

Tailscale can be virtually installed and used across all popular platforms. Maybe except for z/OS?

  • Create Tailscale account: https://login.tailscale.com/start (I personally use Github SSO provider)

  • Installation

    This script is provided on Tailscale official web documentation.

    curl -fsSL https://tailscale.com/install.sh | sh

  • Start the service

    sudo tailscale up

  • Follow the instructions and complete the setup.

  • Some use cases I have experimented with:

macOS Out-of-the-Box Experience

Convenience Script

Static file: /static/scripts/macbook-oobe.sh

(curl -fSsl https://script.aaanh.app/static/scripts/macbook-oobe.sh >> macbook-oobe.sh && chmod 700 macbook-oobe.sh && bash macbook-oobe.sh && rm macbook-oobe.sh)

Manual

Set up homebrew

This also installs xcode command line tools

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Set up oh-my-zsh

  1. Install via official script

    sh -c "$(curl -fsSL https://raw.github.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"

  2. Configure theme

    sed -i 's/robbyrussell/apple/g' ~/.zshrc

  3. Set up zsh-syntax-highlighting

    git clone https://github.com/zsh-users/zsh-syntax-highlighting.git "$HOME/.zsh-syntax-highlighting" --depth 1
echo "source $HOME/.zsh-syntax-highlighting/zsh-syntax-highlighting.zsh" >> "$HOME/.zshrc"

Install VS Code

  1. Download via curl

    curl -L "https://code.visualstudio.com/sha/download?build=stable&os=darwin-universal" -o ~/Downloads/vscode.zip

  2. Extract

    unzip ~/Downloads/vscode.zip

  3. Install to Applications folder

    mv "~/Downloads/Visual\ Studio\ Code.app" /Applications/

  4. Add it to path

    echo export PATH="'$PATH:/Applications/Visual\ Studio\ Code.app/Contents/Resources/app/bin'" >> ~/.zshrc

System Operations

TBA

File Operations

File Transfer

curl

  • Download and Save file with name extracted from the URL

    curl https://example.com/my-file.sh -O

  • Download and Save file with specified name

    curl https://example.com/my-file.sh -O my-downloaded-file.sh

rsync

  • Commonly used command

    rsync -avzP /PATH/TO/LOCAL/MY_FILE_FOLDER/ user@destination_machine:~/SOME_PARENT_FOLDER/

Explanation: (a)rchive, (v)erbose, (z)ip, (P)artial/(P)rogress. SOURCE is the first argument, DESTINATION comes second. So, to copy from a remote machine, simply swap the arguments above.

  • Example 
    rsync -avzP ~/Documents/assination_orders/ account_payables@[email protected]:~/Documents/orders_to_execute/

Archival, compression, extraction

tar

  • Create

    tar zcvf target.tar.gz sourcefile

  • Unpack

    tar zxvf sourcefile

  • List

    tar ztvf sourcefile

(un)zip

  • Create zip archive from files

    zip -r <target> <source_folder>

  • Extract/expand zip archive

    unzip <source_archive>

Finding and Searching

grep

find

Network Operations

Information Query

traceroute (Trace packets route)

  • Default and simple

    traceroute/tracert <IP | FQDN>

  • Specify max number of hops

    traceroute -m <int> <IP | FQDN>

  • Example

    traceroute -m 42069 google.com

dig (DNS lookup)

dig <IP | FQDN>

nslookup (Nameserver lookup)

Most effective when used to look into a certain web address, e.g. google.com, rather than an IP address.

nslookup <FQDN | IP>

ip/ipconfig/ifconfig

  • List interfaces

    ipconfig getiflist

  • Get IP address of the Mac machine on LAN

    ipconfig getifaddr en0

  • Summary of the interface

    ipconfig getsummary en0

Convenience Scripts for Windows

ℹ️ Active development of an OOBE PWSH script project over at https://github.com/aaanh/autowin. This project does most of the below and more.

Shut down a remote PC

shutdown -r -m \\MACHINE-NAME.<domain>.<tld>

Execution Policy ⚠️

Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope LocalMachine
ExplanationBy default, the policy is set to Restricted: commands can be typed and run in the shell, but cannot run a script file. RemoteSigned is the minimal policy that allows it.

Virtualization Features ⚠️

Required for: Windows Subsytems for Linux, Hyper-V, Docker

$win_features = "HypervisorPlatform", "VirtualMachinePlatform", "Microsoft-Windows-Subsystem-Linux", "Microsoft-Hyper-V-All", "Microsoft-Hyper-V", "Microsoft-Hyper-V-Tools-All", "Microsoft-Hyper-V-Management-Powershell", "Microsoft-Hyper-V-Hypervisor", "Microsoft-Hyper-V-Services", "Microsoft-Hyper-V-Management-Clients"

ForEach ($0 in $win_features) {
  Write-Host "Now installing $0..."
  dism /online /enable-feature /featurename:$0 /All /NoRestart
}

Disable 256-character path limit ⚠️

a.k.a. Enable long path

New-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem" `
-Name "LongPathsEnabled" -Value 1 -PropertyType DWORD -Force
ExplanationThe command is taken from Microsoft's Docs. It adds an entry to the registry (GUI accessible through regedit) which allows the long path.

System Operations

TBA

File Operations

Create

  • Single file

    New-Item -Path . -Name "my_file.txt" -ItemType "file" -Value "lorem ipsum"

  • Directory

    New-Item -Path ../ -Name "my_directory" -ItemType "directory"

    Omit -Name and use a relative path, e.g. path/to/my_new_dir instead for a quicker operation, result in a directory named my_new_dir.

Delete

TBA

Copy

Robocopy

TBA

File Info

No native equivalent to Linux file.

File Content

  • With Powershell
Get-Content file.txt
  • With Command Prompt (pipe into MORE for scrolling)
TYPE file.txt | MORE

Network Operations

Information Query

ipconfig

  • Detailed list of network interfaces, much like Linux ip address

    ipconfig /all

tracert (Trace route)

tracert <IP | FQDN>

dig (DNS lookup)

dig <IP | FQDN>

nslookup (Nameserver lookup)

Most effective when used to look into a certain web address, e.g. google.com, rather than an IP address.

nslookup <FQDN | IP>

Docker 🐋

Docker Installation

Linux

curl -fsSL https://get.docker.com -o get.docker.sh && chmod +x get.docker.sh && sh ./get.docker.sh && rm ./get.docker.sh

macOS

  • Apple Silicon

    curl -fsSL https://desktop.docker.com/mac/main/arm64/Docker.dmg -o ~/Docker.dmg && \
  sudo hdiutil attach Docker.dmg && \
  sudo /Volumes/Docker/Docker.app/Contents/MacOS/install --accept-license --user=$(whoami) && \
  sudo /hdiutil detach /Volumes/Docker

  • AMDx64

    curl -fsSL https://desktop.docker.com/mac/main/amd64/Docker.dmg -o ~/Docker.dmg && \
  sudo hdiutil attach Docker.dmg && \
  sudo /Volumes/Docker/Docker.app/Contents/MacOS/install --accept-license --user=$(whoami) && \
  sudo /hdiutil detach /Volumes/Docker

Windows

Enable virtualization features

  1. In your BIOS/UEFI firmware settings.

  2. Windows additional DISM features (src: github repo)

    # 4. Enable optional features
$win_features = "HypervisorPlatform","VirtualMachinePlatform","Microsoft-Windows-Subsystem-Linux","Microsoft-Hyper-V-All","Microsoft-Hyper-V","Microsoft-Hyper-V-Tools-All","Microsoft-Hyper-V-Management-Powershell","Microsoft-Hyper-V-Hypervisor","Microsoft-Hyper-V-Services","Microsoft-Hyper-V-Management-Clients"

ForEach ($0 in $win_features) {
  Write-Host "Now installing $0..."
  dism /online /enable-feature /featurename:$0 /All /NoRestart
}

⚠️ Restart your PC to apply the changes!

Install (with PowerShell)

Invoke-WebRequest -Uri "https://desktop.docker.com/win/main/amd64/Docker%20Desktop%20Installer.exe" -OutFile DockerDesktopInstaller.exe

Start-Process DockerDesktopInstaller.exe -Wait install --quiet --accept-license --backend="wsl-2"

⚠️ Make sure to re-login for Docker Desktop to work!

Docker Operations

The commands written below are examples, all options, --<option> or -<o>, are optional.

  • Build

    docker build </path/to/context/folder> -t <my-docker-image>:<tag-version>

  • Start a container from an image

    docker run -dp <host_port>:<container_port> <my-docker-image>:<tag-version> -n <container-name>

    Note:

    • -d: Run in detached mode. You won’t be presented with the container stdout in the terminal.
    • -n: If not specified, Docker will randomly generate one.

  • Remote into a running container

    docker exec -it <host_port>:<container_port> <my-docker-image>:<tag-version> --tty --stdin -- /bin/bash

    Note:

    • For some minimal containers, the entrypoint should be /bin/sh
    • --it: Run in interactive mode

  • List

    • Images 
      docker image ls
    • Containers 
      docker container ls

  • Stop

    docker stop <container-name>

  • Delete

    • Images 
      docker rm image-tag:tag-version # e.g. nginx:latest
    • Containers 
      docker rm container-name
    • Prune 
      docker prune -a
      Note: -a option will prune images not being used by a container (stopped also qualifies). By default, it deletes images that are not tagged.

Dockerfile

… is a declarative file for building a container from a base image. Simply put, analogous to the commands you’d run when setting up a virtual machine.

Example Dockerfile to build an image that can run Nodejs applications using Microsoft’s base Linux image.

# base image
FROM mcr.microsoft.com/cbl-mariner/base/core:2.0

# Copy from the working directory where `docker build . -f Dockerfile` is executed
COPY . .

# Specify workdir in the container, abitrary dir name
WORKDIR /app

# Install dependencies
RUN yum install -y nodejs npm ca-certificates

# Export some environment variables to the container environment
ENV NODE_EXTRA_CA_CRT="/path/to/those/ca/cert.crt"

# ---- Creating an example nodejs application ----
RUN npm init -y
RUN echo "console.log(\"Hello world!\"); console.log(process.env)" >> index.js

# Set the entrypoint: executed when the container is started.
ENTRYPOINT ["node", "index.js"]

  • Try it out

  • Create the Dockerfile locally, copy and paste the content above

  • Build and run

    docker build . -t test && docker run test

  • Expect

    Hello world!
{
  PATH: '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin',
  HOSTNAME: '5b1d8f97d2ce', # yours could be anything else
  NODE_EXTRA_CA_CRT: '/path/to/those/ca/cert.crt',
  HOME: '/root'
}

Portainer

… is a web-based graphical interface to manage your containers. In all honesty, better than Docker Desktop. Should be ideal for managing in a headless server environment. However, Docker Desktop on Linux does come with that handy kubernetes switch which auto-deploys a cluster with all the necessary components for dev purposes.

Adapted from Portainer’s official documentation

  • Convenience one-liner

    docker volume create portainer_data && docker run -d -p 8000:8000 -p 9443:9443 --name portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ee:latest

  • Create volume

    docker volume create portainer_data

  • Start the Portainer Server container

    docker run -d -p 8000:8000 -p 9443:9443 --name portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ee:latest

    Note: If you are deploying the Community Edition instead, change portainer/portainer-ee to portainer/portainer-ce

  • Server is live at: https://localhost:9443

Kubernetes ☸️

Explaining kubernetes is a pain, so I’d encourage you to seek out better and more structured crash course on YouTube or something.

Installation

Pre-requisites

  • Container runtime:
    • CRI-O (recommended)
    • Docker Engine (for beginners)
      • Docker Desktop (for a right-click deploy simplicity)

Self-hosted cluster

Step-by-step

  • Install a container runtime

Make Life Easier

Tools

  • kubectx: Quickly switch kubernetes context defined in ~/.kube/config

    • Debian/Ubuntu

      sudo apt install kubectx

    • RHEL/Fedora

      Download the appropriate precompiled kubectx library at https://github.com/ahmetb/kubectx/releases/latest

  • Merge kubeconfig files

    ⚠️ This is a destructive command that might nuke your ~/.kube/config file.

    export KUBECONFIG=~/.kube/config:/path/to/config/1:/path/to/config/2 && \
kubectl config view --flatten > ~/.kube/config

Aliases

echo "alias k=kubectl" >> ~/.$(echo $0 | tr -d '-')rc
echo "alias ktx=kubectx" >> ~/.$(echo $0 | tr -d '-')rc

Operations

Namespace

  • GET: All resources in the default namespace

    Note: Use the -n option followed by the specific namespace to scope to it

    kubectl get all -n default

    Example output:

    NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   3m34s

  • GET: Existing namespaces

    kubectl get ns

    Example output:

    NAME              STATUS   AGE
default           Active   4m16s
kube-node-lease   Active   4m17s
kube-public       Active   4m17s
kube-system       Active   4m18s

  • CREATE: A new namespace

    kubectl create ns <namespace>

    Example output:

    namespace/<namespace> created

  • DELETE: A namespace

    kubectl d ns/<namespace>

Custom Resource Definitions (CRDs)

For brevity, the following examples will use a secretproviderclass CRD called azure-secrets.

  • GET: All resources of a CRD

    kubectl get secretproviderclass # optionally, use -o wide or -o yaml

    Example output:

    NAME           AGE
azure-secrets  3m
azure-certs    3m

  • GET: Manifest of the CRD

    kubectl get secretproviderclass/azure-secrets

    Example output:

    apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: azure-secrets
spec:
  provider: azure
  parameters:
    usePodIdentity: "false"
    keyvaultName: <keyvault-name>
    cloudName: "" # [OPTIONAL for Azure] if not provided, azure environment will default to AzurePublicCloud
    objects:  |
      array:
        - |
          objectName: <secret-name>
          objectType: secret
          objectVersion: "" # [OPTIONAL] object versions can be specified here, current version will be used if empty
        - |
          objectName: <cert-name>
          objectType: cert
          objectVersion: "" # [OPTIONAL] object versions can be specified here, current version will be used if empty
    tenantId: <tenant-id>
    subscriptionId: <subscription-id>
    resourceGroup: <resource-group>
    vmType: vmss # [OPTIONAL] vmss or standard. If not provided, it will default to standard
    useVMManagedIdentity: "false" # [OPTIONAL for Azure] if not provided, it will default to "false"
    userAssignedIdentityID: "" # [OPTIONAL for Azure] if provided, use this Virtual Machine's Managed Identity to access keyvault
    keyvaultSecretName: <keyvault-secret-name> # [OPTIONAL for Azure] if provided, will store the keyvault secret name in the pod annotation "secrets-store.csi.k8s.io/azure-secrets/<secret-name>=<keyvault-secret-name>"
    keyvaultSecretVersion: "" # [OPTIONAL for Azure] if provided, will store the keyvault secret version in the pod annotation "secrets-store.csi.k8s.io/azure-secrets/<secret-name>-ver=<keyvault-secret-version>"
    objects:  |
      array:
        - |
          objectName: <secret-name>
          objectType: secret
          objectVersion: "" # [OPTIONAL] object versions can be specified here, current version will be used if empty
        - |
          objectName: <cert-name>
          objectType: cert
          objectVersion: "" # [OPTIONAL] object versions can be specified here, current version will be used if empty
    tenantId: <tenant-id>
    subscriptionId: <subscription-id>
    resourceGroup: <resource-group>
    vmType: vmss # [OPTIONAL] vmss or standard. If not provided, it will default to standard

...

  • CREATE: A new CRD

    • For a non-existing resource, will error if CRD already exists

      kubectl create -f <crd-manifest>.yaml

    • For both existing and non-existing, this will also update the resource

      kubectl apply -f <crd-manifest>.yaml

  • DELETE: A CRD

    kubectl delete secretproviderclass/azure-secrets

  • DESCRIBE: A CRD

    kubectl describe secretproviderclass/azure-secrets

    Example output:

    Name:         azure-secrets
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  secrets-store.csi.x-k8s.io/v1alpha1
Kind:         SecretProviderClass
Metadata:
  Creation Timestamp:  2020-10-14T18:50:00Z
  Generation:          1
  Resource Version:    123456
  Self Link:           /apis/secrets-store.csi.x-k8s.io/v1alpha1/namespaces/default/secretproviderclasses/azure-secrets
  UID:                 123456
Spec:
  Objects:
    Array:
      Object Name:      <secret-name>
      Object Type:      secret
      Object Version:
      Object Name:      <cert-name>
      Object Type:      cert
      Object Version:
  Provider:            azure
  Parameters:
    Cloud Name:
    Keyvault Name:     <keyvault-name>
    Resource Group:    <resource-group>
    Subscription Id:   <subscription-id>
    Tenant Id:         <tenant-id>
    Use Pod Identity:  false
    Use VM Managed Identity:  false
    User Assigned Identity ID:
    VM Type:           vmss
Events:                <none>

Pods

  • GET: All pods

    kubectl get pods

    Example output:

    NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-5f6f6d65c7-4j4q4   1/1     Running   0          3m
nginx-deployment-5f6f6d65c7-5q2q2   1/1     Running   0          3m
nginx-deployment-5f6f6d65c7-6q2q2   1/1     Running   0          3m

  • EXEC: Run a command in a pod

    Only really works if the pod is not built from a scratch image with no shell.

    In other words, you could basically “ssh” into the pod.

    kubectl exec -it <pod-name> -- <command>

    Example output:

    root@nginx-deployment-5f6f6d65c7-4j4q4:/# ls
bin   dev  home  lib64  mnt  proc  run   srv  tmp  var
boot  etc  lib   media  opt  root  sbin  sys  usr

ffmpeg

I know… There are a bunch of GUI wrapper FOSS’s out there. But I want to look cool converting my anime OST UwU.

Head over to installation.md if you’ve not installed ffmpeg.

  • flac to alac
ffmpeg -i /path/to/input.flac -vcodec copy -acodec alac /path/to/output.m4a

Installation

  • Mac: brew install ffmpeg
  • Windows: Download the .exe binary from https://ffmpeg.org/download.html and add it to your PATH
  • Linux:
    • Ubuntu/Debian: sudo apt install ffmpeg or snap install ffmpeg
    • Arch: sudo pacman -S ffmpeg

Troubleshooting

[Solved] Could not load some mysterious libraries

Sometimes brew messes up somehow. In my case, ffmpeg suddenly was not able to load libtesseract and traced the stack to liblepto*. I was able to fix the error by reinstalling both.

brew reinstall tesseract && brew reinstall leptonica

Bottom line is, if it misses some lib$1, just try to reinstall the $1. If the name is incomplete and brew cannot find the name, just Bing or Google it.

[Unsolved] Transcoding rebooted my PC

Happens on Windows. This is on the weirder side, I did not find any logs from Windows Events. It might have to do with the hardware encoders like NVENC or RAM timing. I did some A/B testings which narrowed down to those 2 factors with 70-80% confidence :).

This bug completely shuts down my PC. No BSOD, no nothing.

ytb-dl

I recommend ytp-dl instead, which is a wrapper that provides better download performance.

Note: You will need to install missing codecs or ffmpeg to perform certain operations, for example, Audio Only mode.

  • Download with the highest quality possible for both video and audio
yt-dlp $uri -f bestvideo*+bestaudio/best
  • Download with only audio (requires ffmpeg on PATH)
yt-dlp -x $uri