Boot procedure

can be roughly divided into the following four steps:

Hardware boot

After power-on, control is given to a program stored in BIOS. This program performs basic POST (Power-On Self-Tests), then reads params from CMOS. As a minimum, it needs to know what is the boot device, or which devices to probe as possible boot devices. If boot device is found, the program tries to load the OS Loader, which is located on a fixed position on the boot device, and, in case of success, transfers control to it (see MBR).

The successful completion of the hardware boot stage is usually signaled by one short beep. More than one beep means that POST failed due to some hardware problem:

UEFI

UEFI (Unified Extensible Firmware Interface) is a new "advanced BIOS" more sutable for modern hardware and modern 64-bit operating systems.

EFI (Extensible Firmware Interface) is the Intel's original name of the project.

The Hardware Boot stage and OS Loader stage in systems with classical BIOS firmware and in systems with UEFI firmware are not exactly the same, but those differences are not essential until you face some boot problem. And then you will find that GPT is far more complicated than good old MBR.

A typical PC with UEFI firmware usually supports two boot modes:

• Legacy boot mode (old-style, classical BIOS);
• EFI boot mode (new, advanced);

If your PC has a UEFI firmware, it does not mean that Linux must be installed to use EFI boot mode (though it may be the better choice). However, if you plan dual boot (e.g., Linux + Windows), EFI boot mode may be the only option. Also, if your HDD already has GPT or you plan to have GPT, then EFI mode is the only choice.

EFI boot requires a special EFI partition. It's assumed to be shared by all bootable OSs (if you're going to use more than on), and in ideal case Ubuntu Linux (since 12.04) can use EFI partition created by Windows 7/8 setup. However, in non-trivial cases (like manual Ubuntu installation) you may need to create this partition manually (GParted is one of the appropriate tools). An EFI partition must comply to the following requirements:

• Mount point: /boot/efi (no need to set this mount point when using the manual partitioning, the Ubuntu installer will detect it automatically);
• Size: 100MB .. 250MB;
• Type: FAT32;
• Must be located at the start of a GPT disk, and must have a "boot" flag;

OS Loader (boot loader)

GRUB menu // GRUB shell cmds // Recovery mode // Rescue mode // GRUB restore // GRUB legacy restore

The OS loader (aka Boot Loader) is located in the first sector of the boot device (MBR, cylinder 0, head 0, sector 1). This primary loader is limited due to the size of MBR (446 bytes; the rest is occupied by partition table), therefore, it usually calls a secondary OS loader which may be located on a specified disk partition. The main job of the OS loader is to locate, load and run the kernel. The most popular Linux OS loaders are lilo (Linux Loader, old) and grub (Grand Unified Bootloader). Most distros nowdays use grub. Currently, there are two branches: grub 0.9x (aka GRUB Legacy) and grub 1.xx (GRUB2). To find your version (this cmd requires root priv):

grub-install -v

GRUB Legacy main config file is /boot/grub/menu.lst (usually linked to /boot/grub/grub.conf) that looks like:

default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title Linux Fedora Core (2.6.17.11)
  root (hd0,0)
  kernel /vmlinuz-2.6.17.11 ro root=LABEL=/ rhgb quiet
  initrd /initrd-2.6.17.11.img
...
title Other
  rootnoverify (hd0,1)
  chainloader +1

where default=0 means that the first kernel in the list will be loaded by default, timeout=5 sets the timeout in seconds (if you do nothing during this timeout, GRUB waits for 5 sec, then loads default kernel; to cut timeout, press ENTER).

GRUB2 uses following files/dirs:

/boot/grub/grub.cfg

main config (DO NOT EDIT!);

/etc/grub.d/

this dir contains GRUB scripts, building blocks from which the grub.cfg file is built;

/etc/default/grub

the customization part of GRUB (like old menu.lst, except the actual boot entries; it includes options similar to "default", "timeout", etc);

Some useful (editable) options in /etc/default/grub are:

GRUB_DEFAULT=0
#GRUB_HIDDEN_TIMEOUT=0
GRUB_HIDDEN_TIMEOUT_QUIET=true
GRUB_TIMEOUT="3"

Note, that GRUB_DEFAULT=saved sets the default menu entry with whatever was selected last. Custom entries to GRUB menu can be added by editing /etc/grub.d/40_custom file. To activate changes in these and other files, you should run update-grub (it looks for new kernels, operating systems and re-creates grub.cfg). If you install or remove kernels using rpm or apt-get, GRUB menu is edited automatically.

GRUB menu

GRUB menu allows you to load different versions of kernel, to run memtest, etc. By default it's usually configured not to be displayed if there is only one OS installed. Still you can get it by pressing and holding the SHIFT key (or ESC in systems with old GRUB) when BIOS outputs misc info about devices. When menu appears, press ↓ or ↑ (Down/Up arrow keys) to select option, then ENTER to continue the boot process.

To force GRUB menu always to be displayed, edit /etc/default/grub (the root priv is required!):

...
#GRUB_HIDDEN_TIMEOUT=0
...
GRUB_TIMEOUT=7

By default Ubuntu starts quite silently. If you want to see the boot process details, edit /etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT=""

To run Ubuntu in text mode, set:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash text"

If you need the text mode just to solve some system problems, then may be you should select a recovery boot option in the GRUB menu!

Note that GRUB config changes are ineffective until you run

sudo update-grub

Also note that if you do something wrong, reboot may fail. You can then try recovery, or boot from USB or CD (whichever suits better), but in first case, root fs would be mounted read-only (i.e., you cannot edit config), in second case, update-grub cmd is not as simple as shown above. All these problems can be solved, but it requires some knowledge and manual work.

To run custom tasks at the end of the system start sequence, add your code to

/etc/rc.local

There are also some options in /etc/default/rcS, e.g.:

TMPTIME=0
...
VERBOSE=no

The first option requires /tmp dir to be cleaned at each reboot, non-zero values set the number of days to keep stuff (except -1 which means "never clean"). The second option requires to minimize the output during startup (see man 5 rcS for details).

Some GRUB shell cmds

(when GRUB menu is displayed, press 'c')

boot

boot OS or chain-loader;

chainloader file

prepare to boot other bootloader;

configfile file

load an alternative config file;

halt

shutdown computer;

help [cmd]

show help;

initrd file

load the specified initramfs;

insmod file

insert (load) grub2 module (GRUB2);

kernel file [args]

load a kernel (GRUB Legacy);

linux file [args]

load a kernel (GRUB2);

ls [-alh] [file]

list files (all, long, human-readable) (GRUB2);

lsmod

show loaded modules (GRUB2);

quit

exit from GRUB shell (GRUB Legacy);

reboot

reboot computer;

root

set GRUB's root device (GRUB Legacy);

set [var=value]

set/show env variables (GRUB2);

Note!

GRUB Legacy uses partition notation that starts with 0; GRUB2 uses partition notation that starts with 1; devices are still numbered from 0, so /dev/sda1 is now (hd0,1), NOT (hd0,0) as before!

Recovery mode (single user mode)

• Select kernel and press 'e';
• Go to the EOL, remove quiet and/or splash, type single (be sure there is a space before single), press ENTER;
• Back at the GRUB menu press 'b';

Some systems have recovery mode menu item added as the part of the new kernel installation procedure. Recovery mode is not the same as rescue mode.

Rescue mode

GRUB's functionality is provided by loadable modules residing in /boot/grub. If there are some problems with disks, partitions, files, several kinds of GRUB prompt may pop up. Normal GRUB prompt instead of menu means that there are config problems (like grub.cfg is bad or missing). In worse cases (bad partition/filesystem/files) you may fall back to the rescue prompt (grub rescue>). This usually means that GRUB cannot load the required modules and only a limited subset of the normal GRUB cmds is available (ls, insmod, set, unset). The first move in rescue mode is to try to load the appropriate modules and become normal GRUB shell (rescue shell cannot even load a kernel). Assuming that GRUB resides in a separate boot partition /dev/sda1 (otherwise use /boot/grub instead of just /grub):

grub rescue> set prefix=(hd0,1)/grub

grub rescue> insmod (hd0,1)/grub/normal.mod

rescue:grub> normal

If the above fails, try:

grub rescue> set prefix=(hd0,1)/grub

grub rescue> insmod (hd0,1)/grub/linux.mod

grub> set root=(hd0,2)

grub> linux /vmlinuz-2.6.31 root=/dev/sda2 ro

grub> initrd /initrd-2.6.31.img

grub> boot

Note, that you can use ls to see known devices and partitions:

ls -al

show all devices available;

ls -alh (hd0,1)/boot/

show files in (hd0,1)/boot dir;

If all attempts have failured, or if there are no GRUB prompt at all, reinstall GRUB. Total disapperance of GRUB may be the signal of a wide-scale disaster. However, in some cases this may be just the result of Windows installation procedure.

How to restore (reinstall) GRUB2

Some modern Linux distro DVDs can function as LiveCD, or have some kind of rescue mode. The best way to reinstall GRUB is to use the same distro DVD you used to install Linux. However, if it doesn't provide the required function (or is not available), use some other LiveCDs (except old stuff). Alternatively, you can create a special USB flash drive with GRUB software (stored files won't be erased or damaged).

One of possible algorithms:

• Boot system from LiveCD (distro DVD);
• Open a terminal;
• Mount boot partition:

mount /dev/sda1 /mnt/boot

Note!

If there is no separate boot partition, mount system partition containing /boot directory like this:

mount /dev/sda1 /mnt

• Reinstall GRUB2:

grub-install --root-directory=/mnt /dev/sda

• Unmount device:

umount /dev/sda1

• Reboot.

How to restore GRUB Legacy

This is a variation of the above procedure: LiveCD is only used to get GRUB prompt, then the existing system is loaded from HDD. Note, that some commands and device notation are slightly different:

grub> root (hd0,0)

grub> kernel /vmlinuz-2.6.17.11 ro root=LABEL=/ rhgb quiet

grub> initrd /initrd-2.6.17.11.img

grub> boot

When the system is loaded, there are two methods to reinstall GRUB. The first method is considered a little bit dangerous:

cd /

grub-install /dev/sda

Second method gives you more control.

grub

grub> find /grub/stage1

or

grub> find /boot/grub/stage1

This is not required, but helps to find root device as GRUB understands it in the current situation. Most likely it is (hd0,0) which means first partition on Primary IDE Master or SATA0 or SCSI 0.

grub> root (hd0,0)

grub> setup (hd0)

grub> quit

Note, that similar cmd setup (hd0,0) installs GRUB into the boot sector of the first partition (most likely you don't need this).

See also "How to..":

restore DOS MBR         create GRUB boot floppy

Kernel startup

When the kernel is loaded, it initializes the devices (via their drivers), starts the swapper (kernel process kswapd), and mounts the root file system. Then it creates first user land process init (/sbin/init; pid = 1), passing to it any params that weren’t handled by the kernel already.

In the old Linux (Unix) systems there was a /dev dir with a static set of device nodes (special files). It was large, because for every device, that might possibly appear in the system, there was a device node created beforehand. However, since Linux kernel 2.6.13 (?) device manager udev dynamically creates only the nodes for the devices actually present on a system. udev runs as a daemon and listens to uevents the kernel sends out via netlink socket when a new device is initialized or a device is removed from the system. The main components of udev are:

• libudev, library (provides access to device info);
• udevd, user space daemon (manages virtual /dev);
• udevadm, admin utility (provides info and diagnostics);

For some device classes (e.g., storage devices) udev supports persistent device naming, that does not depend on the order in which devices are plugged into the system. You can affect the process of device node creation by adding (modifying) rule files in /etc/udev/rules.d

sysfs is a filesystem managed by the kernel and mounted at /sys dir. It exports basic info about the devices currently plugged into your system. udev can use this info to create device nodes corresponding to your hardware.

System Initialization

Cmds // Startup // Timers // Unit files // Unit file struct  Unit types

Since 2015 (approx) systemd is the standard Linux OS initializer in the main Linux distributions. It tries to make system startup faster using fewer scripts and running tasks in parallel when possible.

Technically, systemd is a software suite that provides important system components for Linux OS. Its purpose was/is to unify service config and behavior across misc Linux distributions. The primary component is a system and service manager - a new init system used to bootstrap user space and manage user processes. It also provides replacements for some classical daemons and utilities that were used in the past to manage devices, login proc, network connections, event logging.

The term systemd can denote

• init system as a whole,
• Linux daemon systemd (/lib/systemd/systemd, linked to /usr/lib/systemd),
• software package containing the above daemon and other daemons, utilities, libs, files, e.g.
  • systemctl - frontend utility, the central management tool for systemd;
  • journald - log manager (daemon), a replacement of classical Unix log infrastructure;
  • journalctl - frontend utility for log management;
  • multiple executables like systemd-analyze, systemd-run, systemd-mount, etc

Systemd-related directories:

• /etc/systemd - global config;
• /etc/systemd/system - user-created unit files;
• /usr/lib/systemd/system - unit files supplied by package maintainer;
• /run/systemd/system - runtime units;

Note! If you want to modify one of the default unit files in /usr/lib/systemd/system, leave it as is, and put your own file (same name) in /etc/systemd/system dir which has higher priority. Your file will shadow the original one.

Some systemd concepts:

• Unit - a resource managed by systemd; units represent different types of resources (12 types); the units are defined in files known as unit files.
• Target - a unit configuration file with .target extension which encodes info about a target unit of systemd, and is used for grouping units and as well-known synchronization points during startup.

Startup

systemd is the first process to start when Linux boots up, and it controls everything that runs on your computer.

The systemd's model for starting processes (units) is

lazy dependency-based

That is, a unit will only start if and when some other starting unit depends on it. During boot, systemd starts a root unit (default.target, can be overridden in grub config), which then transitively expands and starts its dependencies. If you want to see all those sequencies and relations, try

pstree -p

You can find some details about the boot process with following cmds:

systemd-analize

some details and total duration of the boot process;

systemd-analize blame

what takes so much time during boot?

systemd-analize critical-chain

show the list of the critical chain of tasks during the boot process;

systemd-analize critical-chain firewalld.service

show the critical chain for a particular service;

Unit files

Unit files are config files used by systemd to perform its job. Technically, they are short text files following some rules. There are two types of unit files that can be interesting to users and admins:

*.service

files are used by systemd to handle misc services. Note that "service file" is not the service itself. Service can be something big and complicated and not related to systemd at all, like, for example, Apache Web-Server or PostgreSQL.

*.timer

files are used by systemd to run scheduled tasks (see timers).

A new service which is not activated through sockets, D-BUS, or in a similar way, but wants to be auto started during boot, must become a dependency of an existing boot target which is usually multi-user.target. Here's an example of a simple service file:

[Unit]
Description=Some useful service doing this and that
After=systemd-user-sessions.service
[Service]
Type=forking
ExecStart=/usr/local/bin/my_svc

It's supposed that my_svc is some Linux daemon. See explanation in Unit file structure.

To start this service:

systemctl start my_svc

If service is good, it starts to run in the background, and all output is intercepted by journald daemon (which is also the part of the Systemd pkg). You can find what's going on with your service using

systemctl status my_svc

It shows the last few log lines and info about the processes running. If you need more, you can view the entire log:

journalctl -u my_svc -e

The -u option specifies the unit to show the output of, and the -e means "start from the end of the file and move backwards".

Most distros still send the output from systemd services to /var/log/syslog, but this is an option rather than a feature, and it may disappear (?) in the future, i.e. journalctl dominates, old logfiles should be considered deprecated.

Note that by default journald stores its logs in /run/log/journal dir [which disappears after a reboot]. See below How to provide a permanent log storage.

Systemd Timers

Scheduled tasks are those we want to run periodically, or once at the specified point in time. In the past there were crond and atd for this job, but now it looks like systemd is going to replace both.

There are two types of timers:

• Realtime timers activate on a calendar event, i.e. at the specified date and time; they all have OnCalendar option;
• Monotonic timers activate after a time span relative to some starting point; they usually have an option like OnBootSec or OnActiveSec or similar, and they stop if the computer is temporarily suspended or shut down;

To view all started timers,

systemctl list-timers

To list all timers (including inactive),

systemctl list-timers --all

The status of a service started by a timer is usually inactive unless it is currently being triggered. If a timer gets out of sync, it may help to delete its stamp-* file in /var/lib/systemd/timers (or ~/.local/share/systemd/ in case of user timers).

To schedule a task you need two files:

• task_name.service - service config file specifying program/utility/application to be run;
• task_name.timer - timer config file specifying schedule details;

Each .timer file requires a matching .service file. The timer activates and controls the service. The service file does not require an [Install] section. It's possible to control a differently-named unit using the Unit= option in the timer's [Timer] section. Below you can see service and timer files that are supposed to run some task once a day.

This is proj_bkp.service file:

[Unit]
Description=This service archives my current project

[Service]
Type=simple
ExecStart=/usr/local/bin/arc02.sh

[Install]
WantedBy=multi-user.target

And this is proj_bkp.timer file:

[Unit]
Description=Runs proj_bkp every day at midnight

[Timer]
OnCalendar=*-*-* 00:00:00
Unit=proj_bkp.service

[Install]
WantedBy=timers.target

Both files can be created in /usr/lib/systemd/system or /etc/systemd/system directory. Of course, it requires superuser priv.

About date/time specification

The OnCalendar param in .timer file has the following format:

DayOfWeek Year-Month-Day Hour:Minute:Second

Some typical examples:

You can verify the correctness of the date/time spec using one of the following cmds:

systemd-analize calendar weekly

systemd-analize calendar "2024-02-29"

systemd-analize calendar "Mon-Fri 9:5"

If several relatively heavy tasks have exactly the same start time, you should add the RandomizedDelaySec option in the [Timer] sections of these tasks to avoid an essential surge of resource consumption:

...
[Timer]
OnCalendar=daily
RandomizedDelaySec=5m
Persistant=true

The Persistant=true triggers the service immediately if it missed the last start time due to some reasons, e.g. the system was powered off.

By default the accuracy of the time spec is 1 minute. If you need better, then add AccuracySec=1us option to the [Timer] section.

Here is an example of a monotonic timer which triggers N hours or minutes after some event. In this case it is 15 min after system boot, and then each week at the same time:

[Unit]
Description=Copy some log files after boot and then weekly

[Timer]
OnBootSec=15min
OnUnitActiveSec=1w 

[Install]
WantedBy=timers.target

Time options for monotonic timers:

Numbers are assumed to be seconds, i.e.

OnUnitActiveSec=30

means 30s after ... Other units must be specified explicitly, e.g. 20m (minutes), 4h (hours).

Transient timer units

It is possible to create a transient timer unit using systemd-run. Transient in this case means that you can schedule a command to run at a specified time without creating .timer file. For example, the following cmd allows you to "touch" the specified file after 30 sec:

systemd-run --on-active=30 /bin/touch /tmp/last.txt

You can also schedule execution of an existing service file that does not have the corresponding .timer file. Let's suppose there is a service file named my_proj_backup.service, and it should be executed in 6 hours 30 minutes from now:

systemd-run --on-active="6h 30m" --unit my_proj_backup.service

Miscell Systemd commands

To get the list of the critical chain for a particular target (e.g., basic.target):

systemd-analize critical-chain basic.target | grep target

To get the list of the dependencies, type:

systemctl list-dependencies

To get the list of dependencies for a particular service (e.g., sshd):

systemctl list-dependencies sshd.service

To get the list of dependencies for a particular target (e.g., graphical.target):

systemctl list-dependencies graphical.target | grep target

To get the list of targets that need a particular target (e.g., multi-user.target):

systemctl list-dependencies --reverse multi-user.target

Also, you can get other info about dependencies using following cmds:

systemctl show sshd | grep Requires

systemctl show sshd | grep Wants

To get the content of the Systemd journal:

journalctl

To get all the events related to the crond process in the journal:

journalctl /sbin/crond

or

journalctl `which crond`

All events related to a specific service:

journalctl --unit=service_name

All the events since the last boot:

journalctl -b

All events that appeared today:

journalctl --since=today

All events with a syslog priority of err:

journalctl -p err

All events concerning the kernel:

journalctl -k

The lass 10 events, continously (like tail -f /var/log/messages):

journalctl -f

To provide a permanent log storage:

Unit types

Unit file structure

[Unit]
Description=Job that runs the service_name daemon
Documentation=man:service_name(1)
After=some_target

[Service]
Type=service_type
ExecStart=command
PIDFile=/var/run/service_name.pid

[Install]
WantedBy=some_other_target