linux
"Linux is a family of free and open-source software operating systems built around the Linux kernel." - https://en.wikipedia.org/wiki/Linux
Most linux distros are built on GNU tools, and this article is relevant in distinguishing the importance GNU plays in the linux ecosystem: https://www.gnu.org/gnu/why-gnu-linux.en.html
Linux is part of the Unix family tree.
Performance monitoring
- Linux Load Averages: Solving the Mystery
- Brendan Gregg's Linux Performance page
- Notes from the Linux Performance Monitoring talk at Velocity 2015
Tricks
Best way to see mounts
There are a few ways to see mounts, but most of them will leave out little details in some cases. The best view of mounts is the /proc/self/mountinfo
file.
Determine if running kernel is 32 or 64 bit
Works on x86 or ARM.
getconf LONG_BIT
Configure a system to reboot on kernel panic
These lines should be added to sysctl.conf
## Reboot after 10 seconds if kernel panics
kernel.panic = 10
## Treat all oopses as panics
kernel.panic_on_oops = 1
Force reboot on corrupt system
For times that commands like reboot
and shutdown
are not available.
echo 1 > /proc/sys/kernel/sysrq
echo b > /proc/sysrq-trigger
Show process signals
This should work on other unixes too.
trap -l
Kernel namespaces
"A namespace wraps a global system resource in an abstraction that makes it appear to the processes within the namespace that they have their own isolated instance of the global resource. Changes to the global resource are visible to other processes that are members of the namespace, but are invisible to other processes. One use of namespaces is to implement containers." - man namespaces
"Control cgroups, usually referred to as cgroups, are a Linux kernel feature which allow processes to be organized into hierarchical groups whose usage of various types of resources can then be limited and monitored." - man cgroups
cgroup
is one of the linux namespaces. (see man namespaces
for more info.)
Tools and stuff
lsns
- list namespacescgcreate
- create new cgroupcgexec
- run the task in given control groupcgclassify
- move running task(s) to given cgroupnsenter
- Run a command in a referenced process cgroup configsystemd-cgls
- systemd-cgls - Recursively show control group contentssystemd-cgtop
- Show top control groups by their resource usage/proc/self/cgroup
- cgroup introspection
Various namespace-aware tool examples
ps cgroup output
ps -o pid,ppid,user,comm,flags,%cpu,sz,%mem,cgname
Run a process in another namespace
With nsenter
you specify a target pid to reference, and then specify which namespaces of its you want to enter.
On Ubuntu 18.04, udev
mounts devices in a non-global namespace, which prevents normal users from viewing those mounts. You must use nsenter
to enter the udevd namespaces to view the mounts, using either --all
to get all namespaces of udevd, or --mount
for just that one required namespace:
root@bionic:~# lsblk -o NAME,MOUNTPOINT /dev/sdc
NAME MOUNTPOINT
sdc
└─sdc1
root@bionic:~# nsenter --all -t $(pgrep systemd-udevd) lsblk -o NAME,MOUNTPOINT /dev/sdc
NAME MOUNTPOINT
sdc
└─sdc1 /mnt/adea64ca-e340-4961-8a4d-75d8a5970664
root@bionic:~# nsenter --mount -t $(pgrep systemd-udevd) lsblk -o NAME,MOUNTPOINT /dev/sdc
NAME MOUNTPOINT
sdc
└─sdc1 /mnt/adea64ca-e340-4961-8a4d-75d8a5970664
See udev for one permanent fix for this.
Find the path to a namespace
The path to a namespace can be used in some instances instead of the pid. We can discover the path to a namespace by using lsns
.
root@bionic:~# lsns -p $(pgrep udevd) -o +PATH
NS TYPE NPROCS PID USER COMMAND PATH
4026531835 cgroup 173 1 root /sbin/init /proc/1/ns/cgroup
4026531836 pid 173 1 root /sbin/init /proc/1/ns/pid
4026531837 user 173 1 root /sbin/init /proc/1/ns/user
4026531838 uts 173 1 root /sbin/init /proc/1/ns/uts
4026531839 ipc 173 1 root /sbin/init /proc/1/ns/ipc
4026532009 net 173 1 root /sbin/init /proc/1/ns/net
4026532286 mnt 1 5480 root /lib/systemd/systemd-udevd /proc/5480/ns/mnt
Access network sockets from the command line
This is a poor man's netcat
, useful for when there is no netcat
:
echo asdf > /dev/tcp/${REMOTE_IP_ADDRESS}/${REMOTE_PORT}
See also
Distros
Init systems
Filesystems and block devices
- Filesystem Hierarchy Standards: http://refspecs.linuxfoundation.org/FHS_3.0/fhs/index.html
- LVM
- ZFS
Links
- https://blog.quarkslab.com/digging-into-linux-namespaces-part-1.html
- https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html
- https://wiki.archlinux.org/index.php/Cgroups
- https://poor.dev/blog/terminal-anatomy
- https://www.linusakesson.net/programming/tty
- https://www.sobyte.net/post/2022-05/tty/
- https://www.linuxcommand.org/tlcl.php
- https://unix.stackexchange.com/a/367012: Linux sockets full names are limited to 107 characters
- https://www.linuxatemyram.com
- https://syscalls.mebeim.net: "Linux kernel syscall tables"
- https://specifications.freedesktop.org/basedir-spec/: "Various specifications specify files and file formats. This specification defines where these files should be looked for by defining one or more base directories relative to which files should be located."
- https://kevinboone.me/systemd_embedded.html: "Why systemd is a problem for embedded Linux"