My personal toolbox

Shell, Python, and Perl scripts I wrote/use on a daily basis; i.e. eating my own dog food.

Quick overview by category

The summary/index below was generated by local AI (DSv4 Flash); keep in mind that many utilities below are probably WAY out of date... Use your own judgement about them - that is, check whether they fit your own needs before using them.

Mostly, this repo establishes a common baseline in any machine I work in; a baseline that I've muscle-memoried over the years.

Even if none of the utilities below are useful to you, the approach itself - i.e. building a publicly accessible arsenal you can 'git clone' at a moment's notice... that is definitely recommended :-)

Sandboxing & container isolation

Script	What it does
isolate.sh	Firejail-based sandbox: network allowlisting, read-only $HOME, per-path rw/hide. The core isolation engine
vimisolated.sh	Launch vim inside isolate.sh, auto-detecting writable paths
pi.isolated.sh	Run pi (AI coding agent) inside isolate.sh, tunneling through a Unix socket to reach a local LLM. Symlinked as pi.sh
desert_island_container_execution.sh	Run untrusted TCP server code in a Docker container that the host can reach, but the container can't reach outside the LAN
dockerme.sh	Launch a Docker container with the current folder mapped, optional X11/PulseAudio passthrough, network toggle, root toggle
dockerClearRunningContainersAndNoneImages.sh	Nuke all containers and <none> images
parse-isolation-options-common.sh	Shared argument parser for isolate.sh / pi.isolated.sh (sourced, not standalone)

Example usage of isolation (vimisolated.sh is much better, but this works as a basic example:

# Read-only $HOME, no network
isolate.sh -- vim ~/bin/pi.sh

# Allow specific servers, make some paths writable, hide SSH keys from any vim-spawned subprocesses (Language servers, etc)
isolate.sh --rw ~/bin --rw ~/.viminfo --hide ~/.ssh \
           --servers servers.txt --dns 1.1.1.1 -- vim ~/bin/pi.sh

Example usage of isolated pi.dev harness:

# Assumes a local LLM at localhost:8081
pi.isolated.sh [--port PORT] [isolate.sh options] [-- pi options]

File indexing & syncing

Script	What it does
indexer.py	Scan folders, store MD5s in SQLite, detect changes. Parallel MD5 with ProcessPoolExecutor
syncer.py	Cross-reference a new folder against an existing indexer.db: finds already-existing files (by MD5) and similar filenames (Levenshtein distance)
vidir	Make the moreutils/vidir work properly with my isolated vim)
cmpDir1andDir2andRemoveIdenticalFromDir1	Remove from dir1 files that are identical (size+MD5) in dir2
cmpDir1andDir2andRemoveIdenticalFromDir1BasedOnSizeAndTimestamp	Same, but uses size + timestamp (faster, no MD5)

Example usage:

# Scan folders, store MD5s
indexer.py /mnt/photos

Duplicate detection

Script	What it does
find_dup_videos.py	Detect duplicate videos by duration + perceptual hash (imagehash)
find_dup_images.py	Find duplicate images via perceptual hashing

# Requires ffmpeg/ffprobe + python3 -m pip install ImageHash pillow
find_dup_videos.py /path/to/videos/

Video related

Script	What it does
minimalVideo.py	Transcode any video to x264 at ~0.06 b/p, mux with original audio via mkvmerge
minimalVideo265.py	Same concept, but using HEVC (x265)
minimalVideoCRF_265.sh	HEVC encode with CRF mode (quality-based), copy audio
minimalVideoCRF_265_copy_audio.sh	HEVC CRF encode, copy audio track directly
minimalVideoCRF_265_forQuest.sh	HEVC CRF encode variant for VR headsets (Quest)
vaapi_encode.sh	VAAPI hardware-accelerated HEVC encode
vaapi_encode.just_encode.sh	Minimal VAAPI HEVC encode (no rescaling)
cuda_encode_hevc.sh	NVENC HEVC encode (NVIDIA GPU)
cuda_encode_av1.sh	NVENC AV1 encode (NVIDIA GPU)
fps_2x.sh	Double frame-rate via VAAPI motion interpolation (60fps)
fps_2x_vapoursynth/	Vapoursynth-based 2x FPS scripts
any2mp3.sh	Extract audio track to LAME mp3 from any media file
transcode.sh	Generic ffmpeg transcode wrapper
show_video_bpp.py	Show bits-per-pixel stats for a video
show_video_details.py	Dump codec/res/fps/bitrate of a video file
m4a2aac.sh	Extract AAC from M4A container
mpvYoutubeNoVideo.sh	Play YouTube audio-only (no video, saves CPU)
jpg2lep.sh	Lossless JPEG recompression via Dropbox Lepton
png2webp.sh	Convert PNG to WebP via ffmpeg
recordX11.sh	Losslessly compressed X11 screen recording
flashVideo.sh	Extract flash video from browser plugin's pipe
misc.py	Shared utility module for video scripts

Pi (AI coding agent) helpers

Script	What it does
pi.isolated.sh	Primary launcher: sandboxed pi session with local LLM via Unix socket tunnel. Symlinked as pi.sh
pi.google.sh	Pi launcher using Google AI Studio (Gemini)
pi.google_run.sh	Quick one-shot Google AI Studio run
pi.dockerized.vllm.sh	Run vLLM model server in Docker, then launch pi connected to it
pi_parse_stream.py	Shared parser for pi's streaming JSON responses
cclog.sh	Run cclog Docker image to convert a JSONL conversation log to Markdown
localAI/	Dockerfiles, configs, and launch scripts for local LLM serving (vLLM, llama.cpp, etc.)
Dockerfiles/	Dockerfiles for cclog and yt-dlp binary images

System monitoring & statistics

Script	What it does
dstat	Enhanced dstat binary (system resource monitor)
dstat.sh	Wrapper: dstat -clnv --fs --vm
stats.py	Pipe in numbers → colored statistics (mean, stddev, median, min/max)
statsLive.py	Real-time mean+stddev over stdin stream (Welford's online algorithm)
histogram.py	Generate histogram + percentiles from piped data
histogram.sh	Bash wrapper for histogram.py with outlier detection
_histogram.py	Internal histogram module
asciigraph.py	ASCII line graph from piped data (matplotlib)
crystaldiskmark.sh	CrystalDiskMark clone for Linux using fio
benchmark.nvme.via.io.uring.sh	NVMe sequential read benchmark via io_uring
percentile.sh	Quick percentile calculator
barChartTimes.sh	Horizontal bar chart of execution times (from time output)
performance	Performance tuning wrapper
ondemand	CPU governor toggle (ondemand vs performance)

# Colored stats
for i in {1..100}; do echo $i; done | stats.py

# Histogram
cat numbers.txt | histogram.py

Networking

Script	What it does
block_user_via_iptables.sh	Block a Unix user's outgoing traffic via iptables
unblock_user_via_iptables.sh	Remove the iptables block for a user
sync_NIC_HW_timestamping_clock_from_host.py	Sync NIC hardware timestamp clock (phc2sys) for accurate tcpdump timestamps
tcpdump2binary.py	Extract raw Ethernet frames from a pcap into binary files (uses scapy)
myip.sh	Show my public IP
mydu.sh	Better du: sums file sizes (filesystem-agnostic) per directory
nointernet.for.genymotion.sh	Block Genymotion VM from internet access
youtubeFromClipboard.sh	Download YouTube video from clipboard URL
yt-dlp.sh	Wrapper around Dockerfiles/Dockerfile.yt-dlp container (sandbox the madness)

Subtitle tools

Script	What it does
get_subs_tmux.sh	Download YouTube subtitles, convert to text, launch interactive pi session to summarize. Symlinked as zz
vtt2text.py	Convert VTT subtitle files to clean text
subrip.sh	Subtitle RIP/conversion utility
sub_auto_fixer.py	Match subtitle files to video files by Dice coefficient + LCS

File/text utilities

Script	What it does
latest.py	List recently modified files in a directory, sorted by timestamp
latest.sh	Bash-only poor-mans-version of latest.py
exclude.sh	Interactively filter lines from a file by regex (used with latest.py)
excludeFilter.sh	Same, but via stdin pipe
greedy.py	Pack files to maximally fit a target size
lost_my_space.py	Find what's consuming disk space in a filesystem
lost_my_space.sh	Bash wrapper
logDurations.py	Parse timestamped logs and show duration of each run
sortXML.py	Sort XML elements while preserving structure
csvToHTML_bootstrap.py	CSV → styled HTML table (Bootstrap)
csvToHTML_barebones.py	CSV → minimal HTML table
excelToCsv.py	Convert Excel (.xls/.xlsx) to CSV
countMoviesLength.py	Sum total duration of video files in a folder
rmdirRecursive.sh	Remove all empty directories recursively (post-cleanup)
inlineDataInHTML.sh	Generate data: URI for embedding files in HTML
nocolor.sh	Strip ANSI color codes from pipe
quote.sh	Wrap each line in double-quotes, escaping inner quotes

Clipboard & automation

Script	What it does
clipboardDaemon.py	Clipboard monitoring daemon
execOnChange.sh	Run a command whenever files matching a pattern change (like make for file changes)
batch.SMP.processing.with.bash	Simple parallel processing with bash
lock.sh	Lock screen (i3lock)
setTitle.sh	Set XTerm title
tmux-attach.sh	Attach to a tmux session
tmux-restore.sh	Restore tmux sessions from saved state
tmux_colors.sh	Show tmux color palette for config
fzf-preview.sh	File preview while browsing with fzf + bat
timestamped_logger.sh	Run a command, log stdout+stderr with timestamps

Developer tools

Script	What it does
cpptypefmt	C++ type formatting helper
clean-template-mess.py	Pretty-print C++ template instantiations with indentation
findClassInJars.sh	Find a class name inside a collection of JAR files
findIn	Flexible file content search
git-lg	Git log with graph (one-line)
diffIgnoreCaseAndWS.sh	diff -u -i -b wrapper
diffu	Unified diff shortcut
diffu.meld	Diff with meld
xmlprettyprint.sh	XML pretty-printer (symlink to tidyXML.sh)
tidyXML.sh	XML tidy/format
html5check.py	HTML5 validation
make_ada_tags.py	Generate ctags for Ada sources (gnatxref)
pylintAll.sh	Run pylint on all Python scripts in current dir
showBashCmdline.sh	Show the currently executing command line inside a bash shell
vimcat.sh	Cat a file with vim-based ANSI syntax highlighting
svndiff	SVN diff wrapper
svndiffMELD	SVN diff via Meld
svndiffWS	SVN diff (whitespace-aware)
svndiffXXDIFF	SVN diff via xxdiff

Format conversion & encoding

Script	What it does
ansi2html.sh	Convert ANSI-colored terminal output to HTML
htmlEntities.pl	HTML-encode text (Perl)
unicodeUnescape.py	Decode \uXXXX escape sequences
hex2utf.py	Hex → UTF-8 converter
mime_decoder_inplace_pipe.sh	Decode MIME-encoded files in-place
epochToHuman.pl	Convert Unix epoch to human date
tohex	Decimal → hex
todec	Hex → decimal
tozero.sh	feed safely from a pipe to xargs -0
256color.pl	Display 256-color terminal palette
href.pl	Extract href links from HTML
img.pl	Extract img src links from HTML
unescape.pl	Decode URL-encoded (%XX) and HTML entities
speed.pl	Monitor download speed/progress of a file

System administration

Script	What it does
swap.usage.sh	Show swap usage per process
zswapStatus.sh	Check zswap status
hyperthreading	Toggle hyperthreading
brightScreen.sh	Set backlight brightness to max
waitForCoolCPU.sh	Wait until CPU temperature drops
waitForProcToDie.pl	Wait for a process to exit, then run a command
dumpMemoryOfPID.sh	Dump all memory pages of a PID (as root)
dump_all_thread_callstacks_for_pid.sh	Dump all thread callstacks of a PID via GDB
callstacks_of_threads.sh	Symlink to above
thread_callstacks.sh	Symlink to above
snap_remove_old_versions.sh	Clean old snap package versions
archLinux.orphans.ordered.by.size.sh	List orphaned pacman packages sorted by size
x11.off.sh	Turn off display (DPMS)
x11.off.saver	Screen saver variant
fixClock.sh	Sync system clock via NTP (pool.ntp.org)
7z-ttsiod-compress	7z compression wrapper
makeSelfSignedCertificate.sh	Generate self-signed SSL certificate (legacy, use Let's Encrypt instead)
bcmp.sh	Binary compare two files (shows offset + hex bytes)
setSubtract.pl	Subtract one list of strings from another
setUnversionedToSVNignore.sh	Mark all ? files as svn:ignore
USB_drives_need_no_sleep.py	Keep USB drives awake (prevent spin-down on ZFS server)
loopaesmount.py	Mount loopback AES-encrypted containers
pulseaudio_record.sh	Record PulseAudio output to OGG
sync_music_to_android_with_m3u_for_folder_symlinks.sh	Sync music to Android with .m3u playlist files
mergefs_show_realpath.sh	Show mergerfs real path for files
pdfCountPages.sh	Sum page counts of PDF files
yq_outliers.sh	Flatten YAML to dot-notation for grepping
x16.py	GDB helper: dump 16-column memory (vs default 8)
unchroot.pl	Break out of a chroot jail (security testing)

Dependencies

• Most scripts: bash, standard POSIX tools
• Python scripts: Python 3, various modules (see docstrings)
• Video scripts: ffmpeg / ffprobe, mkvmerge, x264/x265, VAAPI or NVENC drivers
• isolate.sh: firejail, ip, getent
• dockerme.sh: Docker
• pi.isolated.sh: A locally running LLM endpoint (llama.cpp, vLLM, etc.)
• cclog.sh: Docker image cclog (build from localAI/)