sed

is a stream editor used to perform basic text transformations on an input stream (file or pipe). sed reads one line at a time, chops off the terminating newline, puts what is left into the pattern space (buffer) where ~ script can process it. Then, if there is anything to print, it appends a newline and prints out the result (usually to stdout).

sed [options] 'editing_cmds' [file_list]

For backward compatibility with old scripts sed uses BRE by default. If you want ERE, specify -r option. See regex (regular expressions).

If no -e, --expression, -f, --file options are given, then the first non-option arg is taken as ~ script to interpret. All remaining args are the names of the input files. If no input files are specified, stdin is read.

sed -n '/^[^#]/p' squid.conf > squid2.conf

remove comments from squid.conf (i.e. all lines beginning with #), and create a compact version (squid2.conf) of this config file (the original file is not changed);

sed 's/<prog>/<code>/g' man01.html > man02.html

replace all <prog> tags with <code> tags in man01.html, save modified version to man02.html (the initial file is not changed); mind g after the last slash: without it only first match in each line is processed;

sed 's/<\/prog>/<\/code>/g' man02.html > man03.html

replace all </prog> tags with </code> tags in man02.html, save modified version to man03.html (the initial file is not changed);

The following big example converts a file of strings (summary.txt) that look like (each line is exactly 15 fields separated with TABs)

a322   1   A   163.840   1983.140   18080.360   2135.080   15.840   0.000   0.000   0.000   0.000   3174.630   60014.230   685.040

a327   3   N   0.000   0.000   0.000   0.000   0.000   0.000   0.000   0.000   0.000   0.000   0.000   0.000

a518   5   A   NULL   NULL   NULL   NULL   NULL   NULL   NULL   NULL   147100.730   144224.250   134254.820   9785.050

into an SQL script (summ_ins.sql) containing a sequence of SQL stmts:

INSERT INTO summary VALUES (
 'a518',5,'A',NULL,NULL,NULL,NULL,NULL,
  NULL,NULL,NULL,147100.730,144224.250,
  134254.820,9785.050);
  ...

One simple way to perform this conversion may look like this:

sed "s/^[a-z]*[0-9]*/'&'/g" summary.txt > summ2.txt

put the first field in quotes (it starts with one or more letters followed by one or more digits);

sed "s/[A-Z]/'&'/" summ2.txt > summ3.txt

put the third field in quotes (it can only have a single letter); note that due to missing g, processing stops after the first match, and any possible NULL words are not touched;

sed 's/\t/,/g' summ3.txt > summ4.txt

replace TABs with commas;

sed 's/.*/INSERT INTO summary VALUES (&);/' summ4.txt > summ_ins.sql

add all required SQL words and make it a valid SQL script; note that & in this and in the following example specifies where to put the whole content of the source line;

Other examples:

sed 's/.*<td>.*$/&@home.net<\/td><\/tr>/' file1.html > file2.html

add text @home.net</td></tr> to the end of each line containing HTML tag <td>; save modified file as file2.html;

sed '2 d' manual.txt

delete the 2nd line from manual.txt and send a modified version of this file to stdout (the initial file is not changed, all modifications are applied to the output stream only);

sed '3 s/any/some' manual.txt

replace any with some in the 3rd line of manual.txt and send a modified version of this file to stdout;

sed -e '1,10d' test.sh

delete the first 10 lines of test.sh and send the rest to stdout;

sed -e '11,$d' test.sh

print the first 10 lines of test.sh, delete the rest;

sed -n -e '1,7p' messages.txt

print the first 7 lines of messages.txt;

sed -e '7q'

print the first 7 lines of messages.txt;

sed -e '1,4d' -e '22,36' history.txt

delete lines 1-4 and 22-36 from history.txt and send the modified version to stdout;

sed -e '/debug/d' < log.txt

delete all lines containing word debug from the file log.txt and send the modified vesion to stdout;

sed -f prog1.sed < mesg1

use script prog1.sed to process the file mesg1;

prog1.sed:
----------
10i\
This is 9th  (insert a str before line #10)
10a\
This is 11th (insert a str after line #10)
$a\
This is eof; (append the str to the eof)

sed -r 's/\s+$//' fc_upd.lst > fc_upd2.lst

remove trailing spaces and save edited lines to a new file;

sed 's/r\.php?\([0-9]*\)/mfchlp\1.html/'
mfchelp2.html > mfc02.html

for each line in mfchelp2.html substitute the part of the string that looks like <a href="r.php?XX"> with <a href="mfchelpXX.html"> and save the edited lines to a new file (mfc02.html);

sed 's/H:.*OG32[\]/\/\/SRV3\/USER\//' list.txt

for each line in the file list.txt substitute the substring that looks like H:\PROG32\ with //SRV3/USER/;

sed 's/.*/CREATE ROLE & IDENTIFIED BY pass;/'
roles.txt > roles.sql

convert a list of words (roles.txt) to an SQL script that can be used to create Oracle database roles with the corresponding names;

echo 1234567 | sed -n 's/^\(..\)\(.*\)/\1/p'

cut out the first two digits from '1234567' (the result is '12');

echo 1234567 | sed -n 's/^\(...\)\(.*\)/\2\1/p'

move the first three digits to the end (the result is '4567123');

count=`ifconfig eth1 |
sed -n 's/.*RX bytes://p' |
sed -n 's/\s.*//p'`

echo $count

fetch the number of the received bytes (RX) from ifconfig;

Apply substitution to the lines matching regexp:

the black cat was chased by the brown dog
the black cat was often chased by the brown dog

sed -e '/often/s/black/white/g' file1.txt

the black cat was chased by the brown dog
the white cat was often chased by the brown dog

Options

--help    --version

-n, --quiet

suppress automatic printing of pattern space;

-e script, --expression=script

add the specified script to the commands to be executed;

-f script_file, --file=script_file

add contents of the script file to the cmds to be executed;

-i [suffix], --in-place[=suffix]

edit files in place (make backup if extension is supplied);

-l length, --line-length=length

set the desired line-wrap length for the 'l' cmd;

-r, --regexp-extended

use extended regexp in the script;

-s, --separate

consider files as separate, rather than as a single continuous long stream;

-u, --unbuffered

load the minimal amounts of data from the input files and flush the output buffers more often;

Commands

Simple command = pattern + action. If no pattern is given, the action is applied to all lines, otherwise it is applied only to lines matching the pattern. General syntax for a command is:

[addr1[,addr2]] function [args]

If no address is given, a cmd is applied to all lines. If 1 address is given, then it is applied to all pattern spaces that match that address. If 2 addresses are given, then it is applied to all from addr1 to addr2 (including addr1 and addr2 themselves). Addresses may be line numbers or patterns. If patterns, then the substitution is applied to groups of lines from addr1 to the first match of addr2. If there are several groups like that in one file, they all will be affected. There are following cmds (first column is the number of args):

~ processes input line by line and does not keep large file in memory, so expressions like $-10 have no sense, because $ is unknown until the EOF is reached.

sfdisk

displays or manipulates the disk partition tables. It has 4 main uses:

• List the size of a partition.
• List the partitions on a device.
• Check the partitions on a device.
• Repartition a device (this is unsafe and must be done with care).

sfdisk is supposed to be used in scripts. Starting with version 2.26 it supports GPT (GUID Partition Table), but does not provide functionality for CHS (Cylinder-Head-Sector) addressing anymore.

sfdisk -s

show the sizes of all disks;

sfdisk -s /dev/sda2

show the size (in blocks) of the specified disk partition (/dev/sda2);

sfdisk -l /dev/sda

show (list) partition table of the first IDE drive (/dev/sda);

sfdisk -l

list partitions (show partition tables) of all HDDs;

sfdisk -g /dev/hda

show the geometry of /dev/hda;

sfdisk -V /dev/sdc

perform consistency checks to the partition tables on /dev/sdc;

sfdisk -d /dev/sda > sda.pt

save the partition table of /dev/sda to the file sda.pt;

sfdisk /dev/sda < sda.pt

create the partition table based on the data from sda.pt file (see man page for the format description);

sfdisk --print-id /dev/sda 2

show the ID (the type) of the partition 2 on /dev/sda;

sfdisk --change-id /dev/sda 2 83

change the ID of the partition 2 on /dev/sda to 83;

sfdisk [options] [device]

sfdisk -s [partition]

Options

-?    --help    -v    --version

-c n, --id n

show the partition Id (the partition type);

-c n id, --id n id

change the partition Id;

-D, --DOS

(for DOS-compatibility) if a partition cannot contain sector 0, e.g. because that is MBR, or contains the partition table of an extended partition, then ~ would make it start the next sector; however, when this option is given it skips to the start of the next track, wasting some space;

-E, --DOS-extended

take the starting sector numbers of "inner" extended partitions to be relative to the starting cylinder boundary of the outer one (like some versions of DOS do), rather than to the starting sector (like Linux does); it means that extended partitions should start at cylinder boundaries if DOS and Linux should interpret partition table in the same way;

-f, --force

do what is requested;

-g, --show-geometry

show the kernel’s idea of the disk's geometry;

-IBM, --leave-last

do not allocate the last cylinder (some IBM diagnostic programs use the last cylinder for disk testing; in some cases it contains a bad sector table);

-I file

restore the old (initial) disk state, if you earlier preserved it using -O option;

-L, --Linux

don't complain about things irrelevant for Linux;

-l, --list

list the partitions;

-O file

before writing the new partition, save the sectors that are going to be overwritten in the specified file (that resides on another disk);

-q, --quiet

suppress warning messages;

-s, --show-size

list the size of a partition;

-T, --list-types

print the recognized types (system Id’s);

-V, --verify

test whether the partitions seem correct;

-x, --show-extended

list also non-primary extended partitions on output, and expect descriptors for them on input;

--no-reread

when starting a repartitioning, ~ checks that disk is not mounted, or used as a swap device, and refuses to continue if the check fails; this option suppresses the test;

showmount

queries the mount daemon on a remote host for info about the state of the NFS server on that machine.

/usr/sbin/showmount [options] [host]

Without options ~ lists the set of clients who are mounting from that host. ~ sorts output and removes duplicate entries.

/usr/sbin/showmount -e srv2

display export list of srv2 (assuming it's a NFS server);

Options

-h    --help    -v    --version

-a, --all

list both client hostname and mounted dir;

-d, --directories

list only dirs mounted by some client;

-e, --exports

show the NFS server’s export list;

--no-headers

suppress the descriptive headings;

shutdown

(halt, reboot, poweroff, suspend, hibernate)

With introduction of systemd, the old cmds were replaced by systemctl. They are still available [for backward compatibility] but not recommended.

The old shutdown description is left FYI.

The [new] shutdown cmd calls the systemctl utility to perform the shutdown. However, it still accepts a time argument and allow to cancel the shutdown.

You need superuser privs to exec the following cmds!

systemctl reboot

restart the system (no delay);

systemctl --no-wall reboot

restart the system (no delay), do not send warning messages to users (logged in);

systemctl halt

shut down the system but do not power off the computer (starting immediately);

systemctl poweroff

shut down the system and power off the computer (starting immediately);

systemctl --no-wall poweroff

shut down the system and power off the computer, do not send warning messages to users (logged in);

shutdown --halt +5

shut down the system in 5 minutes from now, but do not power off the computer;

shutdown --poweroff 17:30

shut down the system and power off the computer at the specified time;

shutdown -c

cancel a pending shutdown (assuming the timeout is not over yet);

Suspend / hibernate

There is suspend aka Suspend To RAM (STR), aka sleep mode. And there is hibernate aka Suspend To Disk (STD).

Shortly speaking, suspend is good, hibernate is bad! (especially if your system is on SSD). I'm gonna skip the discussion, there is a lot on Internet. And, to be honest, I never use hibernate, and I don't need it at all!

As to suspend, for laptop it's simple: you close the lid - suspend; you open the lid - resume (wake up from suspend). Desktops in GUI mode usually have menu item with "Suspend" (see power off menu at the top/bottom right side of the bar). And it (desktop) wakes up from suspend (at least, it must!) when you press some key on the keyboard or move your mouse. Alternatively, you can suspend your system by executing (as superuser):

systemctl suspend

And, at your risk, considering above warning, you can put your system into STD mode, i.e. hibernate, with the following cmd:

systemctl hibernate

There is also a hybrid sleep (or hybrid suspend) mode when your computer does both STR and STD at the same time. In most cases you will get the normal fast resume from sleep, but if there was a power loss (or battery depletion) during sleep, you will get not so fast resume from STD. It's convenient, and it's supposed to be more reliable than STR, but it's still not good for SSD. Anyway, if you want it:

systemctl hybrid-sleep

Shutdown (old, deprecated)

brings the system down safely; all logged-in users are notified, and within the last 5 minutes of time new logins are forbidden.

/sbin/shutdown [options] time [msg]

The time arg can be an absolute time (hh:mm, 24h clock format), relative time (+m, which means m minutes from now), now, which is equivalent to +0. By default it's +1 (1 minute).

Pay attention to subtle differences in terminology: shutdown means termination of running apps and system services (daemons), halt means executing CPU halt cmd, power off means turning off power supply.

shutdown -h now

shutdown system immediately;

shutdown -h +5

start shutdown procedure in 5 min from now;

shutdown -h 20:00 "System stops at 20:00."

shutdown system at 20:00;

shutdown -r now

reboot system immediately;

shutdown -P now

shutdown and power off immediately;

shutdown -H +15

shutdown and halt system in 15 minutes from now;;

shutdown -c

cancel shutdown in progress;

Options

--help

--no-wall    no messages before halt, power-off, reboot;

sleep

delays for a specified amount of time. By default, n is the number of seconds (can be non-integer), but suffix can change the measure to minutes (m), hours (h), days (d).

sleep 5

delay for 5 seconds;

sleep 5s

delay for 5 seconds;

sleep 0.25m

delay for 15 seconds;

sleep n[suffix]

Options

--help    --version

slocate

provides a secure way to index/search for files on the system.

slocate [options] search_string

~ is a security enhanced version of GNU locate and uses incremental encoding to compress its database to make searching faster. However, unlike locate, it will also store file permissions and ownership so that users cannot see files they are not allowed to see.

slocate -l 1 -U /usr --output=sloc-usr.db

create a secure slocate database for /usr directory;

slocate -i -d /usr mplayer

run the case insensitive search for mplayer using slocate database for /usr dir;

Options

-h    --help    -V    --version    -v    --verbose

-d path, --database=path

the path of db to search in;

-e dir1,dir2,...

exclude the specified dirs from db;

-f fstype1,...

exclude files on the filesystems of the specified types from db;

-o file, --output=file

the database to create;

-r exp, --regexp=exp

search db using a basic POSIX regexp;

-U dir

create ~ database starting at dir;

smartctl

is the part of the smartmontools package which also includes smartd (SMART Disk Monitoring Daemon). S.M.A.R.T. means the Self-Monitoring, Analysis and Reporting Technology, and it is invisibly present in the majority of HDDs and SSDs.

You cannot get SMART data without daemon! Handle it as usually:

sudo systemctl status smartd

sudo systemctl start smartd

sudo systemctl stop smartd

To force this daemon to start auto at system startup, open config file /etc/defaults/smartmontools and uncomment the start_smartd line:

#start_smartd=yes
#smartd_opts="--interval=1800"

The --interval (one of the options that can be passed to the daemon at startup) specifies the interval (in seconds) between checks. By default it is 1800, i.e. every 30 minutes.

Actually, smartd has its own config file /etc/smartd.conf.

Frontend apps and usage examples

(to understand SMART data see SMART IDs)

Most of the time SMART works silently and the majority of users never heard about it. However, if SMART detects a serious problem, your computer will pause during boot to display a warning. And probably some info will be written to logs.

But it can happen that you will need to dive deep. And this is why you should be familiar with smartctl, a text-mode frontend for smartd. There is also a GUI frontend gsmartcontrol, but it does not come with the package and must be installed separately.

These frontends allow to control [mostly, it means "gather info"] S.M.A.R.T. system built into storage devices. The purpose of S.M.A.R.T. is to monitor the reliability of the hard (or solid state) drive, to predict drive failures, and to carry out different types of drive self-tests.

Note!

The majority of the following cmds require superuser privilege! Device name /dev/sda is used just for demo purposes!

To find if the target drive supports SMART:

smartctl -i /dev/sda

The following cmd turns SMART on (or does nothing if it's already enabled):

smartctl -s on /dev/sda

Most likely it's a waste of time because all modern drives have built-in SMART support, and usually SMART daemon turns it ON. So, people ignore the above cmds and go directly to health status (option -H):

smartctl -H /dev/sda

If it fails then either the SMART daemon is not running, or your device really does not support SMART (what a shock!).

But this is not a big deal. The real problem (let's hope it will never happen!) appears when your device reports failing health status. It means either that the device has already failed or predicts its own failure within the next 24 hours. In this situation you can use -a option to get the details, or you can immediately start the rescue operation, i.e., moving/copying the important stuff to a safe place.

To get all SMART info about drive, use option -a (or --all):

smartctl -a /dev/sda

smartctl -a -d ata /dev/sda

A tricky -d option specifies the type of the drive. In most cases it's simple: ata (directly attached SATA HDD/SSD), scsi (directly attached SCSI), nvme for NVMe cards (marked as experimental in Ubuntu 19.xx). There are also convenient types like auto (self-explaining; default) and test which prints the guessed type, opens device and prints the (possibly changed) type and then exits without performing any further cmds.

The difficulties with the drive type appear when the drive (like SAS/SCSI) is the part of the RAID. And then you have to use non-trivial cmds like:

smartctl -a -d megaraid,0 /dev/sda

smartctl -a -d megaraid,1 /dev/sda

The number after comma specifies the disk number in the MegaRAID controller (they say it works with Dell PERC too).

To slightly clear the situation with the drive types and names, you can use:

smartctl --scan

Some information can be found with the help of lspci (it outputs too much if you fail to specify the right search pattern):

lspci | egrep -i 'controller'

lspci | egrep -i 'SATA'

lspci | egrep -i 'raid|adaptec'

Understanding SMART info

There is no strict standard, and manufacturers have their own opinions about SMART data representation. That is, some values of some attributes may have different meaning depending on the model and the manufacturer. Experts advice to focus on critical attributes and deltas, i.e. relatively fast or sudden changes of attrib values that were stable before.

The following table shows some interesting SMART attributes.

Red color means critical, H - higher value is better, L - lower value is better.

SMART Self-tests

Usually we limit our disk tests to filesystem checks. But sometimes you really want to test you storage device. And smartmontools is exactly what you need: SMART provides tests that are best suited for your specific device model.

The bad news: if test ends with errors it is usually bad!

There are several types of SMART self-tests. To find what tests are available for a specific drive, and how much time they are supposed to take, use

smartctl -c /dev/sda

In fact, -c (or --capabilities) shows what SMART features are implemented and how the device responds to some cmds. For example, it shows if device logs errors, if it supports offline surface scanning, etc. If the device can perform self-tests, this option shows the estimated time required to run a specific test.

The type of test to run is specified by -t option. The most basic is short (SMART Short Self Test):

smartctl -t short /dev/sda

Usually it takes no more than 10 minutes, runs in the background, checks the electrical and mechanical performance, the read performance of the disk.

The long (SMART Extended Self Test) can take several hours or, at least, tens of minutes:

smartctl -t long /dev/sda

This is a thorough version of the Short Self Test.

The conveyance [ATA only] runs a SMART Conveyance Self Test which can take several minutes. This routine is intended to identify damage incurred during transportation:

smartctl -t coveyance /dev/sda

All tests can be run during normal system operation unless you add -C option, captive mode (or "foreground mode" for SCSI). However, running self-test can [temporarily] degrade drive performance. On the other hand, intensive use of device by the operating system can increase test time.

Important note! Do not use captive mode on drives with mounted partition/filesystem!

Since these tests run in the background, you cannot see the progress and must wait until the end. Then you can get your test results with

smartctl -l selftest /dev/sda

Note that some drives break the test procedure if you issue the above cmd while the test is still running. Usually the appropriate warning is present in the drive capabilities report (-c).

sort

sorts lines of the specified text files and writes the sorted concatenation of all files to stdout.

sort -b report04.txt

sort a text file ignoring leading blanks;

sort [options] [file ...]

Ordering options

-b, --ignore-leading-blanks

ignore leading blanks;

-d, --dictionary-order

consider only blanks and alphanum chars;

-f, --ignore-case

fold lower case to upper case;

-g, --general-numeric-sort

compare according to general numerical value;

-h, --human-numeric-sort

compare human readable numbers (e.g., 2K 1G);

-i, --ignore-nonprinting

consider only printable chars;

-M, --month-sort

compare (unknown) < 'JAN' < ... < 'DEC';

-n, --numeric-sort

compare according to string numerical value;

-R, --random-sort

sort by random hash of keys;

--random-source=file

get random bytes from file;

-r, --reverse

reverse the results of the comparisions;

--sort=word

sort according to word: general-numeric -g, human-numeric -h, month -M, numeric -n, random -R, version -V;

-V, --version-sort

natural sort of (version) numbers within text;

General options

--help    --version

--batch-size=n

merge at most n inputs at once; for more use temp files;

-c, --check

don't sort, just check whether input is sorted;

-C, --check=quiet, --check=silent

like prev, but do not report first bad line;

--compress-program=prog

compress temporaries with prog; decompress with prog -d;

--files0-from=file

read input from the files specified by NULL-terminated names in file (- means stdin);

-k pos1[,pos2], --key=pos1[,pos2]

start a key at pos1 (by default it is position 1), end it at pos2 (by default it is EOL);

-m, --merge

don't sort, just merge already sorted files;

-o file, --output=file

write to a file instead of stdout;

-S n, --buffer-size=n

set the size of mem buffer; n can be followed by % (perc of mem to use), b (bytes), K (kilobytes, default), M (megabytes), etc;

-s, --stable

stabilize sort by disabling last-resort comparison;

-T dir, --temporary-directory=dir

use dir for temporaries (instead of $TMPDIR or /tmp);

-t sep, --field-separator=sep

use the specified separator instead of non-blank to blank transition;

-u, --unique

with -c, check for strict ordering; without -c, output only the first of an equal run;

-z, --zero-terminated

end lines with a zero byte ('\0'), instead of a newline;

pos is f[.c][opts], where f is the field number and c is the char position in the field. opts is one or more single-letter ordering options, which override global ordering options for that key. If no key is given, use the entire line as the key.

The locale specified by env affects sort order, e.g., LC_ALL=C provides the traditional sort order that uses native byte values.

split

splits a file into fixed-size pieces.

split -b 1440k arc04.tar.gz

split arc04.tar.gz into separate files with the size no more than 1440 Kbytes each;

split [options] [file [prefix]]

If file is missing, or '-', the input data is taken from stdin. By default, ~ puts 1000 lines of the input file into each output file. The output names consist of prefix ('x' is default) followed by a group of chars ('aa', 'ab', ...). There is also csplit, which cuts file into sections determined by patterns.

Options

--help    --version    --verbose

-a n, --suffix-length=n

use suffixes of length n (default is 2); the splitting process will be aborted if all possible char combinations of the specified length are used but end of the input file is not reached yet;

-b n, --bytes=n

put n bytes into each output file;

-C n, --line-bytes=n

put into each output file as many complete lines as possible without exceeding size of n bytes;

-d, --numeric-suffixes

use numeric suffixes;

-l n, --lines=n

put n lines into each output file;

Size spec for -b, -C options may have a multiplier: b for 512 (block), k for 1024 (1K), m for 1048576 (1M).

ss

displays socket statistics and other useful network-related info. You must prepend it with sudo to get full info.

ss -s

show socket summary stat;

ss -tulpn

similar to netstat, i.e.: show listening sockets for tcp and udp protocols, do not resolve names, show related programs;

ss -lp

show listening sockets and related processes;

ss -nlt

show listening TCP sockets only, and do not perform the name resolution;

ss -t -a

show all TCP sockets;

ss -u -a

show all UDP sockets;

ss -t state established

list all established TCP connections;

ss -t4 state established

list all established TCP IPv4 connections;

ss -t4 state closed

list all closed TCP IPv4 connections;

ss -o state established
'( dport = :http or sport = :http )'

display all established HTTP connections;

ss -o state established
'( dport = :ssh or sport = :ssh )'

display all established SSH connections;

ss -o state fin-wait-1
'( sport = :http or sport = :https )'
dst 192.168.0/24

list all TCP sockets in state FIN-WAIT-1 for httpd to the specified network and look at their timers;

ss [options] [filter]

Where

filter := [ state tcp-state ] [ expression ]

Options

-h    --help    -V    --version

-A query, --query=query

a comma-separated list of socket tables to dump: all, inet, tcp, udp, raw, unix, packet, netlink, unix_dgram, unix_stream, packet_raw, packet_dgram;

-a, --all

show all sockets;

-D file

apply filters and dump raw TCP sockets info to a file;

-d, --dccp

show all DCCP sockets (Datagram Congestion Control Protocol);

-e, --extended

show detailed socket info;

-f family, --family=family

show sockets of type family; there are now: unix, inet, inet6, link, netlink;

-i, --info

show internal TCP info;

-l, --listening

show listening sockets only;

-m, --memory

show socket memory usage;

-n, --numeric

do not resolve service names;

-o, --options

show timer info;

-p, --processes

show processes using sockets;

-r, --resolve

try to resolve numeric addr / port;

-s, --summary

dump statistics summary;

-t, --tcp

show TCP sockets only;

-u, --udp

show UDP sockets only;

-w, --raw

show RAW sockets only;

-x, --unix

show UNIX domain sockets only;

ssh

(Secure Shell client program) provides remote access (tcp/22) to another host (remote login, cmd execution) using secure encrypted communication between two untrusted hosts over an insecure network.

ssh root@srv4

connect as user root to the host srv4;

ssh -v root@srv4

connect as user root to the host srv4, show debug messages during connection;

ssh -l root -C ws4.home.net

connect as user root to the host ws4.home.net and switch on the data compression;

ssh srv2 vmstat 3 5 > vmstat.txt

redirect the output of 'vmstat 3 5' executed on the remote host (srv2) to the file vmstat.txt on the local host;

df | ssh srv2 cat \> hdd_info.txt

redirect the output of the locally executed cmd (df) to a file (hdd_info.txt) on the remote host (srv2);

fdisk -l /dev/sda | ssh srv2 cat \>\> hdd_info.txt

redirect the output of the locally executed cmd (fdisk -l /dev/sda) and append it to an existing file (hdd_info.txt) on the remote host (srv2);

tar zcf - ./doc | ssh srv4 tar zxft -

transfer the specified dir (./doc) to the remote host using compression and retaining the access rights;

ssh [-l user] [user@]host [cmd]

~ can also forward over the secure channel X11 connections and arbitrary TCP ports.

ssh -f -L 1234:localhost:6667 srv.acme.com sleep 10

irc -p 1234 ...

tunnel connection to IRC server srv.acme.com using tcp/1234; connection is forwarded to port 6667 on the remote server; remote cmd 'sleep 10' is specified to allow to start service which is to be tunnelled;

Other related programs are: scp (provides secure file transfer between two hosts), ssh-keygen (auth key gen, management, conversion), ssh-agent (authentication agent), sshd (SSH daemon, tcp/22).

scp app.tar.gz root@srv2:app2.tar.gz

upload app.tar.gz as app2.tar.gz to root's home on srv2;

scp -p error.log root@srv5:error.log

upload error.log to the remote host srv5, preserving modif / access time and mode of the original file;

scp -r root@srv7:config/ config

download /root/config dir from the remote host srv7;

Options

-b bind_addr

specify the interface to transmit from;

-c blowfish | 3des | des

select the cipher to encrypt this session; 3des is default;

-c cipher_spec

like previous, but use comma-separated list of ciphers in order of preference (version 2);

-e ch | ^ch | none

set the escape char for session with pty;

-F config

ignore system-wide /etc/ssh/ssh_config, use an alternative per-user config file (the default per-user config is $HOME/.ssh/config);

-i identity_file

select a file from which the identity (private key) for RSA/DSA auth is read; the defaults are:

$HOME/.ssh/identity (v.1)

$HOME/.ssh/id_rsa (v.2)

$HOME/.ssh/id_dsa (v.2);

-I smartcard_device

use smart card device specified;

-l user

use the specified user name to log in on the remote host; this also may be specified on a per-host basis in the config file;

-L [bind_addr:]port:host:port

forward given port on the local host to the specified host and port on the remote side;

-p port

port to connect to on the remote machine;

-R [bind_addr:]port:host:port

forward given port on the remote side to the specified host and port on the local side;

Escape characters

Authentication methods

During connection user must prove his / her identity to the remote machine using one of the following methods (in the specified order):

• GSSAPI-based authentication;
• host-based authentication;
• public key authentication;
• challenge-response authentication;
• password authentication;

For host-based authentication, file /etc/hosts.equiv on the server (or /etc/ssh/shosts.equiv) must contain the name of the client computer, and user names on both sides must be the same. Also, server must be able to verify the client's host key. Host keys are stored in ~/.ssh/known_hosts in user's home dir at user's computer. A new key is added to this file automatically when you connect to a new server for the first time. There is also system-wide file /etc/ssh/ssh_known_hosts that in general should contain the public host keys of all machines in the organization.

To use public key authentication, you should run ssh-keygen to create necessary authentication files (all in $HOME/.ssh/):

Files *.pub are not secret, their contents should be added to $HOME/.ssh/authorized_keys on all remote machines where user wants to log in using public key authentication. For example, you generate DSA public / private key pair [on your computer]:

ssh-keygen -t dsa

Then you upload DSA public key to your home dir at the remote machine and add it to authorized_keys file:

cat id_dsa.pub >> ./ssh/authorized_keys

Check rights for .ssh dir and authorized_keys, because OpenSSH does not accept key if access rights are incorrect (unsafe):

chmod 700 .ssh

chmod 600 ./.ssh/authorized_keys

rm id_dsa.pub

Now you can login to the remote machine from your computer without password, provided usernames here and there are the same.

Password authentication is the simplest of all. It requires no manual config: you just initiate connection, ~ prompts you for password and sends it to the remote server; all communications are encrypted.

stat

displays file or filesystem status.

stat -f /var

show the status of the filesystem /var;

stat test.sh

show the status of test.sh in the default format;

stat -c '%A dev: %d type: %F %U/%G' test.sh

show the status of test.sh in the specified format;

stat [options] file ...

Options

--help    --version

-f, --filesystem

display filesystem status;

-c frm, --format=frm

use the specified format;

-L, --dereference

follow links;

-t, --terse

display info in terse form;

-Z, --context

display security context info for SELinux if available;

Format sequences (for files)

Format sequences (for filesystems)

strace

traces system calls and signals. It's a diagnostic and debugging tool.

strace [options] [cmd [arg ...]]

~ runs the specified cmd until it exits, intercepts and records system calls and signals which are received by a process. The name of each system call, its args and return value are send to stderr (or to a file).

strace -e trace=network -o net.trc -p 2818

trace network-related system calls for the earlier started process with pid=2818; send the trace data to net.trc;

strace -T -o apptest.trc apptest conf.dat

trace apptest, send the trace output to apptest.trc; conf.dat is an argument to apptest executable;

strace -p 17055 2>&1 | grep -i io

trace the process with pid = 17055, search for lines containing 'io', redirect stderr to stdout;

Options

-ttt

like prev, but the leading portion will be printed as the number of seconds since the epoch;

-e expr

a qualifying expression that modifies which events to trace or how to trace them; the format of expr is:

[qualifier=][!]value1[,value2]...

where qualifier is one of trace, abbrev, verbose, raw, signal, read, or write and value is a qualifier-dependent symbol or number (default qualifier is trace);

-e trace=set

trace only the specified set of system calls, for example trace=open,close,read,write (by default trace=all);

-e trace=file

trace all system calls which take a filename as arg;

-e trace=process

trace all system calls which involve process management;

-e trace=network

trace all network-related system calls;

-e trace=signal

trace all signal-related system calls;

-e trace=ipc

trace all IPC-related system calls;

-o file

send the trace output to file instead of stderr;

-p pid

attach to the process with the process ID pid and begin tracing; the trace (not the process!) may be terminated at any time by a keyboard interrupt signal (CTRL+C);

-S sortby

sort the output of the histogram printed when the -c option is set, by the specified criterion: time (default), calls, name, nothing;

-u username

run cmd with the user ID, group ID, and supplementary groups of username;

-E var=value

run cmd with env variable var set to value;

-E var

remove env variable var from the inherited list of env variables before passing it on to the cmd;

su

changes the effective user and group IDs to that of the specified user.

su -

become root (if you know the password); start a new [default] shell as root and work within root’s environment;

su - oracle

start a new [default] shell on behalf of the user oracle and work within the oracle’s environment;

su -c "make install"

perform a system-wide installation of a new application (regular users can only install software to their home dir);

su [options] ...  [-] [user [arg] ... ]

By default, user is root, '-' is equivalent to -l (--login).

Options

--help    --version

-c cmd, --command=cmd

pass a single cmd to the shell;

-f, --fast

pass -f to csh or tcsh;

-l, --login

make the shell a login shell;

-m, -p, --preserve-environment

do not reset env vars;

-s shell, --shell=shell

run the specified shell if /etc/shells allows it;

sudo

is a limited and restricted version of su. It allows some users to exec cmds that are usually supposed to be executed by a superuser (or some other users).

sudo cmd

sudo [options] cmd

sudo apt install mc

sudo adduser gamer2

sudo shutdown -h +5

To work properly, ~ requires correct config file /etc/sudoers. You should never edit this file with any editor except visudo (a "special version" of vi editor):

sudo visudo

Actually, you probably should not edit it at all! If you want to add something, create a file in /etc/sudoers.d/ dir and put your stuff in that file. Make sure this file has no extention, belongs to root/root, has 0440 mode, follows sudoers rules and is edited like this (file name can be different):

sudo visudo -f /etc/sudoers.d/appsrv

Note that the directive

#includedir /etc/sudoers.d

in /etc/sudoers must be exactly like this, i.e., prefixed with #, it's a directive, not a comment.

If, nevertheless, you messed up with your /etc/sudoers and cannot run sudo anymore, and you cannot login as superuser to correct your foolish mistakes, then, maybe, the following command will save you (not all Linux distros support it, but Ubuntu does):

pkexec visudo

The alternative is to reboot in single user mode, mount/remount root fs in r/w mode, etc .. obviously it will be longer and not so simple.

Note!

When ~ prompts you for password, it wants your account's pass, not superuser's pass.

On success the return value is the ret val of the program that was executed. Otherwise ~ quits with an exit value 1 (sometimes an err msg is sent to stderr).

In some Linux distributions (e.g. Ubuntu) root account is disabled after installation, i.e. you cannot login directly or use su to become a superuser. But the account still exists, and it is possible to exec cmds with superuser privs. Some of the following cmds are marked BAD, because they defeat the purpose of ~, and because working in superuser mode is quite dangerous (for your system).

sudo !!

if the last command has failed due to unsifficient permissions, run it again, this time as sudo cmd;

sudo -s

start a root shell, but keep the current shell's env (bad!);

sudo -i

start a root shell, root's env and root's login scripts (bad!);

sudo gnome-terminal

start a new GNOME terminal with root prompt and privs (bad!);

Note!

For GUI apps you should use gksudo (or kdesudo for KDE env). These cmds set HOME=~root, and copy .Xauthority to a tmp dir. This prevents files in your home dir becoming owned by root.

gksudo nautilus

start a GUI app (nautilus file browser) with root privs (to enter this cmd in GUI mode, you can use ALT+F2);

gksudo gedit /etc/fstab

edit system file that cannot be modified by a regular user;

Who can run sudo

The priv to exec ~ is given to members of group sudo. In Ubuntu versions before 12.04 the group was admin. To create a new user with the priv to run ~:

adduser username sudo

To give the priv to an existing user:

usermod -a -G sudo username

Some notes on the subj

The real and effective uid and gid are set to match those of the target user as specified in the /etc/passwd.

By default you must authenticate yourself with your own (not root's) password. After this you can use ~ for a limited period of time (5 .. 15 min), then you're once again prompted for pass. To cut this timeout short:

sudo -k

For security reasons ~ removes form env some variables that are passed to the program to be executed (sudo -V on behalf of root shows the list). Though in general ~ is much safer than su, there is risk that some user gets full root privileges due to superuser mistake. This happens when user is allowed to run some program with shell escape (some editors, for example), or is granted permission to execute sudo su or sudo sh.

Files

/etc/sudoers

defines authorized users; use visudo (special version of vi editor) to edit this file, otherwise there'll be problems;

/var/run/sudo

this is a dir containing timestamps;

Some options

-r role

set SELinux security context to have the specified role;

-t type

set SELinux security context to have the specified type (domain); if no type is given, the default type is derived from the specified role;

-u usr

run cmd as the specified user (not root); user can be given by the username or uid prefixed with '#';

You should not do this!

In Linux systems where root login is disabled (e.g. Ubuntu), you can return to old-style behaviour by setting root's pass:

sudo passwd root

GUI login must be configured separately:

System -> Administration -> Login screen, ...

But logging in to X as root may cause serious problems!

To re-disable root account:

sudo passwd -dl root

swapon, swapoff

enables / disables devices (or files) for paging and swapping.

swapon /dev/sda6

enable swap area on the disk partition /dev/sda6;

swapoff /dev/sdb2

disable swap area on the disk partition /dev/sdb2;

swapon -a

enable all swap devices specified in /etc/fstab;

swapoff -a

disable all swap devices specified in /etc/fstab;

swapon -s

show swap usage summary for all available devices (this is equivalent to cat proc/swaps);

swapon [options] special_file

Options

-h    -V    -v

-L label

use the partition that has the specified label;

-U uuid

use the partition that has the specified uuid;

sysctl

(do not confuse with systemctl ) is used to modify kernel parameters at runtime. The params available are those listed under /proc/sys/.

sysctl options

sysctl [options] var ...

sysctl [options] -w var=value ...

var is the name of a key to read from (like kernel.ostype); note that '/' can be used instead of '.');

/sbin/sysctl -a

show all available values;

/sbin/sysctl -n kernel.hostname

show the current value of kernel.hostname;

/sbin/sysctl -w kernel.domainname="home.net"

set kernel.domainname to home.net;

sysctl -a | grep disable_ipv6

check if IPv6 support is disabled (there are usually several lines, pay attention to net.ipv6.conf.all.disable_ipv6,
0 means "enabled", 1 - "disabled");

Options

-p file

load settings from the specified file (by default it is /etc/sysctl.conf);

-w var=value

set a key to the specified value; value may be enclosed in double quotes, if it contains qoutes or some other chars parsed by shell;

systemctl

is a front-end utility for systemd, and it is like it sounds: system control program, allowing to manage services, sockets, mount points, etc. In most cases it requires superuser priv, though cmds requesting status are usually available to regular users.

systemctl status

systemctl start $unit

systemctl stop $unit

systemctl restart $unit

systemctl show-environment

systemctl set-environment var=value

systemctl unset-environment var

systemctl list-dependencies --all

The last cmd shows relationships between services (pstree-style).

Usage examples for apache2 (former httpd) service:

systemctl start apache2.service

systemctl restart apache2.service

systemctl stop apache2.service

systemctl reload apache2.service

systemctl status apache2.service

systemctl status apache2.service

Of the above cmds only status prints the output. reload asks to reload service-specific config.

systemctl kill apache2

sends a signal to one or more processes of the unit; to specify the signal use --signal=sig option;

The following cmds are used to configure service auto start at boot time: enable creates the required symlinks, but it does not start service immediately (you should do it manually, if necessary); disable removes symlinks, but does not stop the running service.

systemctl is-active apache2.service

systemctl enable apache2.service

systemctl disable apache2.service

Note that active means "running". If some service is disabled it does not start at boot. However, you still can start it manually. The following cmds control the service availability: if service is masked, it cannot be started neither manually nor auto. Usually it's better to mask with --runtime option (to be cleared after reboot).

systemctl mask apache2.service

systemctl --runtime mask apache2.service

systemctl unmask apache2.service

Other cmds:

systemctl list-unit-files

list all the available units;

systemctl list-units

list all running units;

systemctl list-unit-files --type=service

list all services (including enabled and disabled);

systemctl list-unit-files --type=mount

list all system mount points;

systemctl list-unit-files --type=socket

list all available system sockets;

See other usage examples:

Shutdown // Network // PostgreSQL // MySQL