"

GNU OS

1 GNU OS

  • Notes about GNU operating systems. So far Linux kernel based.

2 command-line / terminal

2.1 miscellaneous tricks and tips

export editor=vi                set default editor

queue <command>                 add command to queue
    fg                          execute queue

CTRL+D                          same as exit
CTRL+R                          reverse history search

rm $(find | grep class)         delete all files containing 'class' in their name.
                                Uses command substitution.
env                             list all defined variables
export                          list all defined variables

unset   <variable>              remove environment variable

rehash                          reloads users and groups id-s, passwords, environment variables,
                                and other user related data.



su - <user>                     preserves defined environment when switching user
sux <user>                      switches user while retaining X session

!<command beginning>            replace shortcut with original command and executes (history)

CTRL+L                          clear console screen

<binary>  --help <command>      sometimes gives detailed command help

CTRL + ALT                      select rectangular area with the mouse

2.2 terminal control codes

echo -e "\033[44;37;5m ME \033[0m COOL"

0 reset all attributes to their defaults
1 set bold
2 set half-bright (simulated with color on a color display)
4 set underscore (simulated with color on a color display)
5 set blink
7 set reverse video
22 set normal intensity
24 underline off
25 blink off
27 reverse video off
   
30 set black foreground
31 set red foreground
32 set green foreground
33 set brown foreground
34 set blue foreground
35 set magenta foreground
36 set cyan foreground
37 set white foreground
   
38 set underscore on, set default foreground color
39 set underscore off, set default foreground color
   
40 set black background
41 set red background
42 set green background
43 set brown background
44 set blue background
45 set magenta background
46 set cyan background
47 set white background
49 set default background color
   
\033[2J clear screen
\033[0q clear all keyboard LEDs (won't work from Xterm)
\033[1q set "Scroll Lock" LED
\033[2q set "Num Lock" LED
\033[3q set Caps Lock LED
\033[15;40H move the cursor to line 15, column 40
\007 bell (beep)

3 GNU distributions

3.1 CentOS / Redhat / Fedora

3.1.1 package management

yum install <package name>
  1. RPM - software management tool
    rpm -qa                 show installed packages
    rpm -e <pack>           remove package
    rpm -i <pack>           install package
    rpm -ivh <pack>         install package while being very verbose
    
    rpm --replacepkgs
                            install even if package already exists
            --replacefiles  replace conflicting files
            --force         force install
            --nodeps        no dependency check
    
    locate package providing specified file
        urpmf --provides libwx_baseu_xml-2.6.so.0
    
    
    extract files from RPM
        mkdir tmp
        cd tmp
        rpm2cpio /path/to/package.rpm | cpio -d -i
    

3.2 Debian / Ubuntu / Linux Mint

3.2.1 Debian bug tracking systems

Every tracked bug has email address:

<bugNumber>@bugs.debian.org

To close the bug, send email to:

<bugNumber>-done@bugs.debian.org

3.2.2 package management

  1. apt-pinning (combine packages from stable and unstable)
    1. setting up
      1. add following lines to /etc/apt/sources.list
        # packages in Testing
        deb http://ftp.us.debian.org/debian testing main contrib non-free
        
        # packages in Unstable
        deb http://ftp.us.debian.org/debian unstable main contrib non-free
        
      2. insert following lines into /etc/apt/preferences file:
        Package: *
        Pin: release a=stable
        Pin-Priority: 700
        
        Package: *
        Pin: release a=testing
        Pin-Priority: 650
        
        Package: *
        Pin: release a=unstable
        Pin-Priority: 600
        
      3. update available packages
        sudo apt-get update
        
    2. example usage:

      Attempt installation of enlightment package from unstable repository. Will not attempt to upgrade any packages on your system, so if specific dependencies are not met, the install will fail.

      apt-get install enlightenment/unstable
      

      Install enlightment package and also installs/upgrades any needed dependencies. (This can easilly break system)

      apt-get -t unstable install enlightenment
      
  2. creating Debian package (deb)
    1. package naming convention

      pattern:

      packagename_version_arch.deb
      
      version package version with major, minor, and revision numbers
      packagename can contain lowercase letters, numbers, and the "-" and "+" characters
    2. package content


      debian-binary Text file. Contains the version number of the binary package.
                          Should be: 2.0
      control.tar.gz
          md5sums MD5 checksums for the package

          control
              sample content:

                  Package: acme
                  Version: 1.0
                  Section: web
                  Priority: optional
                  Architecture: all
                  Essential: no
                  Depends: libwww-perl, acme-base (>= 1.2)

                      this package requires the package acme-base version equal to or higher than 1.2.

                          earlier (<<)
                          earlier or equal (<=)
                          equal to only ()
                          equal to or later than (>
      )
                          strictly later than (>>)

                  Pre-Depends: perl

                      forces the system to make sure that the named packages are completely installed before attempting to install your package

                  Recommends: mozilla | netscape
                              This field tells that eighter mozilla or netscape would be nice to have,
                              both are not required.

                  Suggests: docbook
                      Packages that are likely to be useful but aren't in any way required

                  Enhances: xxxx
                      States that your package may make another package more useful

                  Installed-Size: 1024
                  Maintainer: Joe Brockmeier <jzb@dissociatedpress.net>
                  Conflicts: wile-e-coyote
                  Replaces: sam-sheepdog
                  Provides: acme
                  Description: The description can contain free-form text
                               describing the function of the program, what
                               kind of features it has, and so on.
                  .
                  More descriptive text.




          postinst instructions on what to do after installing the package

          prerm removal instructions

          conffiles configuration files for the package


      data.tar.gz actual payload for the package

    3. development statuses
      1 Planning
      2 Pre-Alpha
      3 Alpha
      4 Beta
      5 Production/Stable
      6 Mature
      7 Inactive
  3. install *.deb package from filesystem
    dpkg -i <pack.deb>
    
  4. install package by name
    apt-get install <package name>
    
  5. search package by name/pattern
    apt-cache search <pattern>
    
  6. update available packages list
    sudo apt-get update
    
  7. update/upgrade packages
    apt-get dist-upgrade
    
  8. use backported packages repository
    1. setup
      • Add backports to your sources.list. For Jessie add this line
        deb http://http.debian.net/debian jessie-backports main
        
      • Run
        sudo apt-get update
        
    2. install backported package

      All backports are deactivated by default (i.e. the packages are pinned to 100 by using ButAutomaticUpgrades: yes in the Release files. If you want to install something from backports run:

      apt-get -t jessie-backports install "package"
      

      of course you can use aptitude as well:

      aptitude -t jessie-backports install "package"
      
  9. list installed packages

    list all packages:

    dpkg-query -l
    

    filter list by pattern:

    dpkg-query -l 'gpart*'
    

4 Linux

  • Operating system kernel that supports multitasking. Is used in many GNU distributions.
  • Project started by Linus Torvalds.

4.1 Linux 4.8 compatibility issues

From Linux 4.8, several changes have been made in the kernel configuration to 'harden' the system, i.e. to mitigate security bugs. Some changes may cause legitimate applications to fail, and can be reverted by run-time configuration:

  • On 64-bit PCs (amd64), the old 'virtual syscall' interface is disabled. This breaks (e)glibc 2.13 and earlier. To re-enable it, set the kernel parameter: vsyscall=emulate
  • On most architectures, the /dev/mem device can no longer be used to access devices that also have a kernel driver. This breaks dosemu and some old user-space graphics drivers. To allow this, set the kernel parameter: iomem=relaxed
  • The kernel log is no longer readable by unprivileged users. To allow this, set the sysctl: kernel.dmesg_restrict=0

4.2 compiling the kernel

4.2.1 compile kernel into installable package

fakeroot make-kpkg --append-to-version "-mykernel" --revision "1" --initrd kernel_image kernel_headers

4.2.2 copmile kernel into bzip compressed image

make bzImage

4.2.3 configure build options

Text based color menus, radiolists & dialogs.

make menuconfig

X windows based configuration tool.

make xconfig

Kernel build parameters should be stored in file

.config

4.2.4 define number of threads for compiling kernel

Example: compile kernel using 4 threads

export CONCURRENCY_LEVEL=4

4.2.5 clean build directory

  • method 1
    • Makes sure you have no stale .o files and dependencies lying around.
    • This will also delete ".config" file
      make mrproper
      
  • method 2
    • cleans the source tree and reset the kernel-package parameters.
      make-kpkg clean
      

4.2.6 install prerequisites

sudo apt-get install libncurses5-dev fakeroot kernel-package --yes

4.3 LKM - Loadable Kernel Modules

located at /lib/modules/&lt;kernel version&gt;/

  • /etc/modules
  • /etc/modules.conf
  • load module
    modprobe <module>
    
  • remove module
    rmmod <module>
    
  • list currently loaded modules
    lsmod
    
  • displays quick inforamtion about the module
    modinfo <options> <module>
    
    • -p list parameters that module supports
    • -d breif description of the module

4.3.1 most common modules

mousedrv mouse driver
sound sound system
soundcore sound system
hid USB human interface device
usbmouse USB mouse driver

4.3.2 load module on boot

add module name into /etc/sysconfig/kernel under MODULES_LOADED_ON_BOOT variable

4.3.3 modprobe.conf syntax

  • set module alias this allows referencing a module under dirrefent name
    alias <module alias> <module name>
    
  • set module options
    options <module name> <module options>
    

    example

    options mcd     mcd=0x340,11
    

4.3.4 modconf - Linux kernel module manager

select kernel modules (drivers)

4.4 see Linux messages

sudo dmesg

Monitor log

sudo dmesg -wH

5 SystemD

5.1 service management

5.1.1 system service

  1. enable service
    systemctl enable <serviceName>
    
  2. disable service
    systemctl disable <serviceName>
    
  3. start service
    systemctl start <serviceName>
    
  4. stop service
    systemctl stop <serviceName>
    
  5. view service status
    systemctl -l status <serviceName>
    

5.1.2 user service

  1. enable service
    systemctl --user enable <serviceName>
    
  2. disable service
    systemctl --user disable <serviceName>
    
  3. start service
    systemctl --user start <serviceName>
    
  4. stop service
    systemctl --user stop <serviceName>
    

5.2 view system logs

journalctl -xb

5.3 reboot system

systemctl reboot

Try to reboot into default mode.

systemctl default

5.4 disable/enable GUI

In order to make text boot the default under systemd (regardless of which distro):

systemctl set-default multi-user.target

To change back to booting to the GUI,

systemctl set-default graphical.target

6 networking

6.1 network setup

See available network interfaces

ip a

Start network on interface

sudo ifup <deviceName>

See my ip address

sudo ip addr show

6.2 maximum allowed concurrent connections limit

View the current maximum configured connections

cat /proc/sys/net/netfilter/nf_conntrack_max

To see the current used connections

cat /proc/sys/net/netfilter/nf_conntrack_count

When maximum allowed connections limit is exceeded, similar messages appear in the kernel log:

nf_conntrack: table full, dropping packet

[404969.000157] net_ratelimit: 123 callbacks suppressed

6.2.1 increase maximum configured connections limit

  • Temporarily Solution
    echo 500000 > /proc/sys/net/netfilter/nf_conntrack_max
    
  • Permanent solution
    • add a line to file
      /etc/sysctl.conf
      

      like this:

      net.netfilter.nf_conntrack_max = 500000
      
    • then enter:
      sysctl -p
      

6.3 backup hard drive/partition over network

cat /dev/<partition> | bzip | ssh <userName>@<remoteHost> 'cat > /<backupFilesystemLocation>/<archiveName>.bz2'

6.4 determine process using network port

See PID and names of programs along with the port they are using:

netstat -nlp

see which process is using the port:

fuser -n tcp <port id>

6.5 DSL connection

6.5.1 setup

sudo pppoeconf

6.5.2 enable

sudo pon dsl-provider

This command also re-enables DSL connection after it has been dropped for some reason.

6.5.3 disable

sudo poff

6.5.4 check status

See DSL connection status:

plog
ifconfig ppp0

6.6 IP address management

6.6.1 fix IP to host MAC address in DHCP server

Edit

/etc/ethers

Include one line with syntax:

<MAC address> <IP address>

Example:

00:1E:E5:8C:F9:EF 192.168.0.250

6.6.2 manually set computer IP

ifconfig eth0 192.168.0.1 netmask 255.255.255.0 up

6.6.3 secondary IP address management

  1. assign multiple IP's for eth0 network device
    • approach using ifconfig command
      sudo ifconfig eth0:1 <ipAddress> netmask 255.255.255.0 up
      sudo ifconfig eth0:2 <ipAddress> netmask 255.255.255.0 up
      ...
      
    • alternative approach using ip command
      sudo ip address add <ipAddress> dev lo
      
  2. delete locally assigned IP address
    sudo ip addr del <ipAddress> dev lo
    
  3. list all IP addresses
    sudo ip addr list
    

6.7 bluetooth

Check that your Bluetooth adaptor is functional

hciconfig

scan for devices

hcitool scan

ping remote device

l2ping -c 2 00:17:E3:F2:DE:1A

Use sdptool (Service Discovery Protocol tool)

sdptool search DUN

Bind /dev/rfcomm0 to channel 2

rfcomm bind 0 00:17:E3:F2:DE:1A 2

Check that the bind was successfull.

rfcomm show 0

7 permanent storage

7.1 file / directory

7.1.1 archiving / compressing

  1. bzip2
    -z compress
    -d decompress
  2. cpio
    cpio -idmv < <filename>
    
  3. gzip

    uncompress

    gunzip -d <archive>
    
  4. rar

    uncompress

    rar x <archive>
    
  5. tar
    1. extension: tar

      extract

      tar -xvf mystuff.tar
      

      compress

      tar -cvf filename.tar <directoryToCompress>
      
    2. extension: tgz / tar.gz

      extract

      tar -xzvf mystuff.tgz
      
    3. extension: tbz2 / tar.bz2

      extract

      tar -xjvf mystuff.tbz2
      

      compress

      tar -cjvf mystuff.tbz2 <directoryToCompress>
      
  6. zip
    • Has 2 Gb output file limit.
    1. compress
      zip -r <zipFile> <file(s)/foler(s) to compress>
      
    2. uncompress
      • unzip to current directory
      unzip -x <zipFile>
      
      • unzip to specified directory
      unzip -x <zipFile> -d <destinationDir>
      
      • uncompress all zip files in the directory
      unzip \*.zip
      
  7. 7z
    7z --help
    
    Usage: 7z <command> [<switches>...] <archive_name> [<file_names>...]
           [<@listfiles...>]
    
    <Commands>
      a: Add files to archive
      b: Benchmark
      d: Delete files from archive
      e: Extract files from archive (without using directory names)
      l: List contents of archive
      t: Test integrity of archive
      u: Update files to archive
      x: eXtract files with full paths
    <Switches>
      -ai[r[-|0]]{@listfile|!wildcard}: Include archives
      -ax[r[-|0]]{@listfile|!wildcard}: eXclude archives
      -bd: Disable percentage indicator
      -i[r[-|0]]{@listfile|!wildcard}: Include filenames
      -m{Parameters}: set compression Method
      -o{Directory}: set Output directory
      -p{Password}: set Password
      -r[-|0]: Recurse subdirectories
      -scs{UTF-8 | WIN | DOS}: set charset for list files
      -sfx[{name}]: Create SFX archive
      -si[{name}]: read data from stdin
      -slt: show technical information for l (List) command
      -so: write data to stdout
      -ssc[-]: set sensitive case mode
      -t{Type}: Set type of archive
      -u[-][p#][q#][r#][x#][y#][z#][!newArchiveName]: Update options
      -v{Size}[b|k|m|g]: Create volumes
      -w[{path}]: assign Work directory. Empty path means a temporary directory
      -x[r[-|0]]]{@listfile|!wildcard}: eXclude filenames
      -y: assume Yes on all queries
    

7.1.2 chattr - change file attributes on a Linux file system

The format of a symbolic mode is

+-=[aAcCdDeijsStTu].

The operator '+' causes the selected attributes to be added to the existing attributes of the files; '-' causes them to be removed; and '=' causes them to be the only attributes that the files have.

The letters 'aAcCdDeijsStTu' select the new attributes for the files:

  • append only (a),
  • no atime updates (A),
  • compressed (c),
  • no copy on write (C),
  • no dump (d),
  • synchronous directory updates (D),
  • extent format (e),
  • immutable (i),
  • data journalling (j),
  • project hierarchy (P),
  • secure deletion (s),
  • synchronous updates (S),
  • no tail-merging (t),
  • top of directory hierarchy (T),
  • and undeletable (u).
-R Recursively change attributes of directories and thqeir contents.

7.1.3 chmod - change file mode bits

set UID bit

chmod +s <filename>

7.1.4 create delta files

usage:

xdelta COMMAND [OPTIONS] [ARG1 ...]

Commnads

delta Produce a delta from ARG1 to ARG2 producing ARG3
info List details about delta ARG1
patch Apply patch ARG1 using file ARG2 producing ARG3

7.1.5 delete

  1. recursively delete files matching certain pattern

    delete all *.avi files

    find . -name *.avi -exec rm {} \;
    
  2. recursively delete directories matching certain pattern

    delete all .svn directories with their contents

    find . -name .svn -exec rm -rf {} \;
    
  3. delete directory recursively
    rm -rf <directory>
    

7.1.6 diff - file comparison

diff <file to update> <take changes from>

-b Ignore changes in amount of white space.
-B Ignore changes that just insert or delete blank lines.
-I regexp Ignore changes that just insert or delete lines that match regexp.

7.1.7 display file content on terminal

cat <fileName>

7.1.8 file ownership

recursive uid change example

find -user myUser -exec chown 1114 {} \;

7.1.9 less - utility to view contents of text file

  1. frequently used options
    -g Highlights just the current match of any searched string.
    -I Case-insensitive searches.
    -M Shows more detailed prompt, including file position.
    -N Shows line numbers (useful for source code viewing).
    -S Disables line wrap ("chop long lines"). Long lines can be seen by side scrolling.
    -? Shows help.
    +F Follow mode for log.
  2. frequently used commands
    Key Command
    Space bar Next Page
    b Previous Page
    v Edit Content
    j or ↵ Enter Next Line
    k Previous Line
    F Follow Mode (for logs). Interrupt to abort.
    g or < First Line
    G or > Last Line
    <n>G Line <n>
    /<text> Forward Search for <text>. Text is interpreted as a regex.
    ?<text> Backward Search like /
    n Next Search Match
    N Previous Search Match
    Escu Turn off Match Highlighting (see -g command line option)
    -<c> Toggle option <c>, e.g., -i toggles option to match case in searches
    m<c> Set Mark <c>
    '<c> Go to Mark <c>
    = or Ctrl+G File information
    h Help. This is presented with less, q to quit.
    q Quit
  3. examples
    less -M readme.txt                     # Read "readme.txt."
    less +F /var/log/mail.log              # Follow mode for log
    file * | less                          # Easier file analysis.
    less -I -p void *.c                    # Case insensitive search for "void" in all .c files
    

7.1.10 link

create symbolic link

ln -s <existing file> <link name>

remove specified symbolic link

unlink <link name>

7.1.11 lsattr - list file attributes on a Linux second extended file system

The following attributes are read-only, and may be listed by lsattr but not modified by chattr:

  • compression error (E),
  • huge file (h),
  • indexed directory (I),
  • inline data (N),
  • compression raw access (X),
  • and compressed dirty file (Z).

7.1.12 tail - monitor log file for updates

tail -f <filename>

7.1.13 patch

patch files in current directory

patch < myPatch.tapch

7.1.14 search

  1. search file(s) contents for pattern(s)

    seek for string in files specified by mask

    grep "string" <filemask>
    

    grep directory files recursively for string

    grep -R <pattern> <filemask>
    

    another example

    find <path> -name <filenamepattern> | xargs grep <searchstring>
    

    recursively search all .properties for string

    grep -Ri common.concepts.issues --include="*.properties" .
    
  2. recursively find recently modified files
    find $1 -type f -exec stat --format '%Y :%y %n' {} \; | sort -nr | cut -d: -f2- | head
    
  3. locate - instantly find file by name using precompiled database

    Find file usig:

    locate <partOfTheFileName>
    

    Index is kept up to date by dedicated cron task. To manually update files index use:

    updatedb
    
    1. exclude paths of filesystems from scan

      edit file

      /etc/updatedb.conf
      

      Example:

      PRUNEPATHS="/tmp /var/spool /media"
      PRUNEFS="NFS nfs nfs4 rpc_pipefs afs binfmt_misc proc smbfs autofs iso9660 ncpfs coda devpts ftpfs devfs mfs shfs sysfs cifs lustre tmpfs usbfs udf fuse.glusterfs fuse.sshfs curlftpfs"
      

7.1.15 split file into smaller pieces

split -b 1500m -d images.tgz.gpg images.tgz.gpg
         Size     Input          Prefix

7.2 HDD / SSD

7.2.1 backup / restore MBR

backup

dd if=/dev/sda of=~/mbr.img bs=512 count=1

restore

dd of=/dev/sda if=mbr.img bs=512 count=1

7.2.2 check for bad blocks

Non destructive mode (slow):

sudo badblocks -nvs /dev/<drive>

Destructive mode (OVERWRITES ENTIRE DISK!!!):

sudo badblocks -wvs /dev/<drive>

Where <drive> stands for block device: sda, sdb, ….

7.2.3 SSD cache for HDD disk

  1. Flashcache

    Flashcache is implemented as loadable kernel module and user space utilities to manage cache device through device mapper.

  2. Bcache

    Data partition has to be prepared i.e. "formatted" beforehand. This invalidates bcache as viable solution for plugging to existing file systems with data.

7.2.4 hdparm - hard drive configuration utility

turn on DMA on that drive

hdparm -d1 /dev/hda

test drive speed

hdparm -t /dev/hdd

display drive configuration info

hdparm -v /dev/hdd

7.2.5 CD/DVD handling

  1. record CD
    1. under console
      1. blank RW disc
        cdrecord -v blank=fast dev=0,0,0 speed=10
        
      2. make ISO
        mkisofs -f -R -r -l -J -odest.iso src
        

        where

        • dest.iso
          • is the destination filename of the newly created ISO image;
        • src
          • is the temporary ISO directory containing the files and file structure you wish to have included in the ISO image.
      3. setup CD drive

        see loaded modules

        lsmod
        

        load module: ide-scsi

        modprobe ide-scsi
        

        see detected optical storage devices

        cdrecord -scanbus
        
      4. write audio CD
        1. Create your audio tracks and store them as uncompressed, 16-bit stereo .wav files.
        2. Name the audio files in a manner that will cause them to be listed in the desired track order when listed alphabetically, such as 01.wav, 02.wav, 03.wav, etc.
        3. Change into the directory containing the wave files and make sure there are not any wave files you do not want included in the CD.
        4. With a blank CD in your burner, issue the following command:
          cdrecord -v -pad speed=8 dev=0,0,0 -dao -audio -swab *.wav
          
      5. write data CD
        cdrecord -v speed=8 dev=0,0,0 <ISO file>
        
    2. under KDE

      Use k3b application. It has nice GUI.

  2. record DVD
    growisofs -Z /dev/scd0 -V "DVD label" -J -R /path/to/files/to/be/burned
    
  3. UDF - Universald Disk Format

    Allows copying and deleting of files on rewritable CDs and DVDs in random order.

    UDF Versions

    1.00 (24.10.1995) for CD-ROM
    1.01 (03.11.1995) for DVD-ROM
    1.02 (30.08.1996) for DVD-Video
    1.50 (04.02.1997) for Data-DVDs (DVD-R, DVD+R, DVD-RAM)
    2.00 (03.04.1998) for Videorecording
    2.01 (15.03.2000) Bugfix for 2.00
    2.50 (30.04.2003) for BD-RE
    2.60 (01.03.2005) for BD-R

7.2.6 list drives by UUID

ls -l /dev/disk/by-uuid

Another way to get the uuid by usage of the tool blkid:

sudo blkid /dev/sda1

result:

/dev/sda1: LABEL="/" UUID="ee7cf0a0-1922-401b-a1ae-6ec9261484c0" SEC_TYPE="ext2" TYPE="ext3"

7.3 filesystem

7.3.1 create new filesystem

mkfs

7.3.2 check filesystem for errors

fsck
sudo touch /forcefsck

7.3.3 monitor disk usage

Use software package

gt5

7.3.4 BTRFS

Btrfs is a new copy on write (CoW) filesystem for Linux aimed at implementing advanced features while focusing on fault tolerance, repair and easy administration.

  1. btrfs-scrub - scrub btrfs filesystem

    Start background scrub process or root filesystem:

    sudo btrfs scrub start /
    

    View background scrub process status on root filesystem:

    sudo btrfs scrub status /
    
  2. balance

    balance root filesystem

    sudo btrfs balance start /
    

    monitor balance status on root filesystem

    btrfs balance status /
    

7.3.5 types

  1. union fs

    http://en.wikipedia.org/wiki/Aufs

    package: aufs-util

    example:

  2. swap - virtual memory

    /etc/fstab entry

    /dev/sdb2    none   swap   sw,pri=3   0   0
    

    format linux swap partition/file

    mkswap <partition/file>
    

    Turn swapping on. Priority 3. Higher priority number => higher usage priority.

    swapon -p 3 <device>
    

    turn swapping off

    swapoff
    
    1. set up a swap to filesystem file

      create swap file in size of 512 MB

      sudo dd if=/dev/zero of=/extraswap bs=1M count=512
      

      format file as swap file

      sudo mkswap /extraswap
      

      enable swap file usage

      sudo swapon /extraswap
      

      Make swap file permament accross reboots. Eedit /etc/fstab file, add line:

      /extraswap   none   swap   sw   0   0
      

7.3.6 /etc/fstab format

<file system> <mount point>   <type>  <options>       <dump>  <pass>

example */etc/fstab* entry to mound Windows partition

/dev/sda1     /mnt/mydata     auto    umask=000       0       0
  1. dump

    used for these filesystems by the dump(8) command to determine which filesystems need to be dumped. If the fifth field is not present, a value of zero is returned and dump will assume that the filesystem does not need to be dumped.

  2. pass

    used by the fsck(8) program to determine the order in which filesystem checks are done at reboot time. The root filesystem should be specified with a fs_passno of 1, and other filesystems should have a fs_passno of 2. Filesystems within a drive will be checked sequentially, but filesystems on different drives will be checked at the same time to utilize parallelism available in the hardware. If the sixth field is not present or zero, a value of zero is returned and fsck will assume that the filesystem does not need to be checked.

7.3.7 find out processes who use specific moutpoint

fuser -m /mnt/<myMountPoint>

7.3.8 mount

  1. define owner group and owner user of the mounted filesystem
    -o uid=some_user
       gid=some_group
    
  2. mount /tmp to RAM

    Open your */etc/fstab* as root in your favourite text editor

    Add this line to the end:

    none /tmp tmpfs defaults,size=424m 0 0
    
  3. mount filesystem directory

    mount –bind /var/ftp/incoming /home/dave/incoming

    makes content of the first folder accessible at the second folder location

  4. mount floppy image

    Creating the image: $ dd bs=512 count=2880 if=/dev/zero of=imagefile.img $ mkfs.msdos imagefile.img

    Mounting it: $ sudo mkdir /media/floppy1/ $ sudo mount -o loop floppy.img /media/floppy1/

  5. mount ISO image
    mount -o loop -t iso9660 filename.iso /mnt/iso
    
  6. mount samba share
    smbmount //192.168.0.14/MP3 /mnt/mp3 -o username=mp3,password=listenmp3,codepage=cp866
    mount -t smbfs //192.168.0.153/install asdf
    
  7. mount USB memory stick

    mount /dev/sda1 /mnt/removable/ -o exec,rw,umask=000

    On Mandriva Linux inserting memory stick causes 'kded' (kdeinit) process to hang, consuming all CPU esources. Solution remove files /usr/lib/kde3/kded_mountwatcher.la /usr/lib/kde3/kded_mountwatcher.so

7.3.9 show free space on filesystems

df

7.4 partition

7.4.1 list available partitions

cat /proc/partitions

Example output:

major minor #blocks name
8 0 97685784 sda
8 1 21454776 sda1
8 2 1 sda2
8 5 2112516 sda5
8 6 10273536 sda6
8 7 10321731 sda7
8 8 53520516 sda8
8 16 97685784 sdb
8 17 97683201 sdb1

7.4.2 show mounted partitions

mount

7.5 IO tuning

7.5.1 monitor permanent storage load

watch iostat

7.5.2 set process IO priority

example:

ionice -c 3 -p 21409

7.5.3 IO schedulers

  1. IO schedulers list
    1. deadline
      • deadline based, eliminates starvation
    2. anticipatory
      • deadline based, eliminates starvation
      • anticipates additional requests
      • uses huristics table to determine process behaviour
    3. cfq - Completely Fair Queuing

      CFQ works by placing synchronous requests submitted by processes into a number of per-process queues and then allocating timeslices for each of the queues to access the disk.

    4. NOOP - noop

      no optimization, just queues the requests

    5. bfq - Budget Fair Queueing

      BFQ is a proportional-share storage-I/O scheduler that also supports hierarchical scheduling with a cgroups interface. Here are the main nice features of BFQ.

      1. Low latency for interactive applications

        According to our results, whatever the background load is, for interactive tasks the storage device is virtually as responsive as if it was idle. For example, even if one or more of the following background workloads are being served in parallel:

        • one or more large files are being read or written,
        • a tree of source files is being compiled,
        • one or more virtual machines are performing I/O,
        • a software update is in progress,
        • indexing daemons are scanning the filesystems and updating their databases,

        Starting a command/application or loading a file from within an application takes about the same time as if the storage device was idle. As a comparison, with CFQ, NOOP, DEADLINE or SIO, and under the same conditions, applications experience high latencies, or even become unresponsive until the background workload terminates (especially on SSDs).

      2. Low latency for soft real-time applications

        Also soft real-time applications, such as audio and video players or audio audio- and video-streaming applications, enjoy about the same latencies regardless of the device load. As a consequence, these applications do not suffer from almost any glitch due to the background workload.

      3. High throughput

        BFQ achieves up to 30% higher throughput than CFQ on hard disks with most parallel workloads, and about the same throughput with the rest of the workloads we have considered. BFQ achieves the same throughput as CFQ, NOOP, DEADLINE and SIO on SSDs.

      4. Strong fairness guarantees

        As for long-term guarantees, BFQ distributes the throughput as desired to I/O-bound applications (and not just the device time), with any workload and independently of the device parameters..

  2. change IO scheduler

    Can be changed during runtime using:

    echo cfq > /sys/block/sda/queue/scheduler
    
  3. IO scheduler can be determined for any device using
    cat /sys/block/sda/queue/scheduler
    

7.5.4 tune writeback behavior

Linux can lock for minutes on big writes on slow devices like for example SD cards or USB sticks.

See http://lwn.net/Articles/572911/

Workaround: as root issue:

echo $((16*1024*1024)) > /proc/sys/vm/dirty_background_bytes
echo $((48*1024*1024)) > /proc/sys/vm/dirty_bytes
  1. /proc/sys/vm/dirty_background_bytes

    Contains the amount of dirty memory at which the background kernel flusher threads will start writeback.

    Note: dirty_background_bytes is the counterpart of dirty_background_ratio. Only one of them may be specified at a time. When one sysctl is written it is immediately taken into account to evaluate the dirty memory limits and the other appears as 0 when read.

  2. /proc/sys/vm/dirty_bytes

    Contains the amount of dirty memory at which a process generating disk writes will itself start writeback.

    Note: dirty_bytes is the counterpart of dirty_ratio. Only one of them may be specified at a time. When one sysctl is written it is immediately taken into account to evaluate the dirty memory limits and the other appears as 0 when read.

    Note: the minimum value allowed for dirty_bytes is two pages (in bytes); any value lower than this limit will be ignored and the old configuration will be retained.

7.5.5 drop disk cache

echo 3 | sudo tee /proc/sys/vm/drop_caches

8 software

8.1 how to write man pages

8.1.1 chapters convention

NAME
Name section, the name of the function or command.
SYNOPSIS
Usage.
DESCRIPTION
General description
OPTIONS
Should include options and parameters.
RETURN VALUES
Sections two and three function calls.
ENVIRONMENT
Describe environment variables.
FILES
Files associated with the subject.
EXAMPLES
Examples and suggestions.
DIAGNOSTICS
Normally used for section 4 device interface diagnostics.
ERRORS
Sections two and three error and signal handling.
SEE ALSO
Cross references and citations.
STANDARDS
Conformance to standards if applicable.
BUGS
Gotchas and caveats.
SECURITY CONSIDERATIONS
Security issues to be aware of.
other
Customized headers may be added at the authors discretion.

8.1.2 formatting keywords

Title/header

    .TH [name of program] [section number] [center footer] [left footer] [center header]


section heading

    .SH text for a heading

new paragraph

    .PP

a comment line

    ."

Indent the text that comes 2 lines after this macro

    .TP


pre formatted text
    .nf
    _your_pre_fromatted_
    _text_goes_here_____
    .fi



.B Bold
.BI Bold alternating with italics
.BR Bold alternating with Roman
.I Italics
.IB Italics alternating with bold
.IR Italics alternating with Roman
.RB Roman alternating with bold
.RI Roman alternating with italics
.SM Small (scaled 9/10 of the regular size)
.SB Small bold (not small alternating with bold)

8.1.3 man sections

section purpose
1 User commands
2 System calls, that is, functions provided by the kernel.
3 Subroutines, that is, library functions.
4 Devices, that is, special files in the /dev directory.
5 File format descriptions, e.g. /etc/passwd.
6 Games, self-explanatory.
7 Miscellaneous, e.g. macro packages, conventions.
8 System administration tools that only root can execute.
9 Another
n New documentation, that may be moved to a more appropriate section.
l Local documentation referring to this particular system.

8.1.4 sample man page

.TH cdspeed 1  "September 10, 2003" "version 0.3" "USER COMMANDS"
.SH NAME
cdspeed \- decrease the speed of you cdrom to get faster access time
.SH SYNOPSIS
.B cdspeed
[\-h] [\-d device] \-s speed
.SH DESCRIPTION
Modern cdrom drives are too fast. It can take several seconds
on a 60x speed cdrom drive to spin it up and read data from
the drive.  The result is that these drives are just a lot slower
than a 8x or 24x drive.  This is especially true if you are only
occasionally (e.g every 5 seconds) reading a small file. This
utility limits the speed and makes the drive more responsive
when accessing small files.
.PP
cdspeed makes the drive also less noisy and is very useful if
you want to listen to music on your computer.
.SH OPTIONS
.TP
\-h
display a short help text
.TP
\-d
use the given device instead of /dev/cdrom
.TP
\-s
set the speed. The argument is a integer. Zero means restore maximum
speed.
.SH EXAMPLES
.TP
Set the maximum speed to 8 speed cdrom:
.B cdspeed
\-s 8
.PP
.TP
Restore maximum speed:
.B cdspeed
\-s 0
.PP
.SH EXIT STATUS
cdspeed returns a zero exist status if it succeeds to change to set the
maximum speed of the cdrom drive. Non zero is returned in case of failure.
.SH AUTHOR
Guido Socher (guido (at) linuxfocus.org)
.SH SEE ALSO
eject(1)

8.2 install software from source

  1. configure
    • generates makefile for current platform
  2. make
    • compiles binaries
  3. make install
    • installs binaries to appropriate directories
  4. make clean
    • removes compiled binaries, useful for recompiling

9 processes and services

9.1 create a system service

9.1.1 old initv

create new service descriptor file using /etc/init.d/skeleton as a sample

finally register new service using

update-rc.d <serviceName> defaults

When specifying hard dependencies or ordering requirements, you can use names of services (contents of their Provides: section) or pseudo names starting with a $. The following ones are available according to LSB (1.1):

$local_fs
all local file systems are mounted (most services should need this!)
$remote_fs
all remote file systems are mounted (note that /usr may be remote, so many services should Require this!)
$syslog
system logging facility up
$network
low level networking (eth card, …)
$named
hostname resolution available
$time
the system time has been set correctly
$portmap
SunRPC portmapping service available
$ALL
indicates that a script should be inserted at the end
  1. return values acc. to LSB for all commands but status:
    0 success
    1 generic or unspecified error
    2 invalid or excess argument(s)
    3 unimplemented feature (e.g. "reload")
    4 user had insufficient privileges
    5 program is not installed
    6 program is not configured
    7 program is not running
    8–199 reserved (8–99 LSB, 100–149 distrib, 150–199 appl)
  2. runlevels
    0 halt/poweroff
    1 single user
    2 multiuser without network exported
    3 multiuser w/ network (text mode)
    5 multiuser w/ network and X11 (xdm)
    6 reboot

9.2 cron - task scheduler

allows commands to be scheduled for execution

list cron jobs for specified user

crontab -l -u <user>

edit cron jobs for specified user

crontab -e -u <user>

9.2.1 crontab entry format

MIN HOUR DOM MON DOW CMD
Field Description Allowed Value
MIN minute field 0 to 59
HOUR hour field 0 to 23
DOM day of month 1-31
MON month field 1-12
DOW day of week 0-6
CMD command Any command to be executed

Example:

1,6,11,16,56 1-23 * * * /usr/local/scripts/wb_robot.sh
  1. matching of values
    *
    an asterisk that matches all possible values
    3
    a single integer that matches that exact value
    2,3,7
    a list of integers separated by commas (no spaces) used to match any one of the values
    1-3
    two integers separated by a dash (a range) used to match any value within the range.
  2. day of week
    • weekday names can be used:
      • SUN, MON, TUE, WED, THU, FRI, SAT
    • numerical references can be used too:
      • 0 and 7 stand for sunday
      • 6 stands for saturday
      • 1-5 means from monday to friday
    numerical named
    0 SUN
    1 MON
    2 TUE
    3 WED
    4 THU
    5 FRI
    6 SAT
    7 SUN
  3. month field
    • month names can be used:
      • JAN, FEB, MAR, APR, MAY, JUN, JUL, AUG, SEP, OCT, NOV, DEC
  4. minute field

    schedule a background cron job for every 10 minutes

    */10 * * * * /home/user/myTask
    
  5. run command on system reboot
    @reboot <command>
    

9.3 grub - boot loader

9.3.1 setup

grub-install --root-directory=/boot /dev/hda

arguments:

/dev/hda
drive to install to

9.3.2 partition naming

hd0 first hard drive
0 first partition

9.4 log handling

9.4.1 delete old logs via cron task

Following root user crontab entry will delete Tomcat logs that are older than 3 days:

01 02 *   * * find /var/log/tomcat8/ -name "*log*" -type f -mtime +3 -exec rm -f {} \;

9.4.2 logrotate - utility to rotate system log files

  1. Postrotate

    Logrotate runs the postrotate script each time it rotates a log specified in a configuration block. You usually want to use this script to restart an application after the log rotation so that the app can switch to a new log.

    postrotate
        /usr/sbin/apachectl restart > /dev/null
    endscript
    
  2. Compression

    If you want archived log files to be compressed (in gzip format), you can include the following command, usually in /etc/logrotate.conf:

    compress
    

    Compression is normally a good idea, because log files are usually all text and text compresses well. If, however, you have some archived logs that you don’t want to compress, but you still want compression to be on by default, you can include the following command in an application-specific configuration:

    nocompress
    

    Another command of note in regard to compression is as follows:

    delaycompress
    

    This command is useful if you want to compress the archived logs, but want to delay the compression. When delaycompress is active, an archived log is compressed the next time that the log is rotated. This can be important when you have a program that might still write to its old log file for a time after a fresh one is rotated in. Note that delaycompress works only if you have compress in your configuration.

    An example of a good time to use delaycompress would be when logrotate is told to restart Apache with the “graceful” or “reload” directive. Because old Apache processes do not end until their connections are finished, they could potentially try to log more items to the old file for some time after the restart. Delaying the compression ensures that you won’t lose those extra log entries when the logs are rotated.

  3. Size

    You can use the size command to specify a file size for logrotate to check when determining whether to perform a rotation. The format of the command tells logrotate what units you’re using to specify the size:

    • size 100k
    • size 100M
    • size 100G
  4. Rotation interval
    • daily
    • weekly
    • monthly
    • yearly

9.5 monitor system status

9.6 process scheduler

9.6.1 BFS - Brain Fuck Scheduler

  1. Isochronous SchedulingEdit

    Isochronous scheduling is a unique scheduling policy designed to provide near-real-time performance to unprivileged (ie non-root) users without the ability to starve the machine indefinitely. Isochronous tasks (which means "same time") are set using, for example, the schedtool application like so:

    schedtool -I -e amarok
    

    This will start the audio application "amarok" as SCHED_ISO.

  2. Idleprio SchedulingEdit

    Idleprio scheduling is a scheduling policy designed to give out CPU to a task _only_ when the CPU would be otherwise idle. The idea behind this is to allow ultra low priority tasks to be run in the background that have virtually no effect on the foreground tasks. This is ideally suited to distributed computing clients (like setiathome, folding, mprime etc) but can also be used to start a video encode or so on without any slowdown of other tasks. To avoid this policy from grabbing shared resources and holding them indefinitely, if it detects a state where the task is waiting on I/O, the machine is about to suspend to ram and so on, it will transiently schedule them as SCHED_NORMAL. As per the Isochronous task management, once a task has been scheduled as IDLEPRIO, it cannot be put back to SCHED_NORMAL without superuser privileges. Tasks can be set to start as SCHED_IDLEPRIO with the schedtool command like so:

    schedtool -D -e ./mprime
    

9.6.2 CFS - Completely Fair Sgheduler

9.7 processes

9.7.1 find process that have opened files, devices or network sockets

lsof

9.7.2 process states

state code explanation
R running or runnable (on run queue)
D uninterruptible sleep (usually IO)
S interruptible sleep (waiting for an event to complete)
Z defunct/zombie, terminated but not reaped by its parent
T stopped, either by a job control signal or because it is being traced

9.7.3 determine process owner

ps -aux

9.7.4 get running processes ID

pidof <processname>

9.7.5 kill all processes returned by grep

Example below kills all processes containing "sample-process" in their name:

kill `ps -ef | grep <sampleProcess> | grep -v grep | awk '{print $2}'`

9.7.6 killall - kills all processes specified by name

-I, --ignore-case
-w, --wait          Wait for all killed processes to die.

example:

killall -I -w mozilla-firefox-bin

9.7.7 run command even after terminal session has been terminated

nohup <command> &

example:

nohup pullftp.sh &

9.8 shared libraries

9.8.1 print shared library dependencies

ldd <pid>

9.8.2 shared library

lists all directories from where shared libraries can be loaded.

/etc/ld.so.conf

After changing libraries configuration execute '# ldconfig' to reload lib setting.

9.9 SysRQ key

9.9.1 shortcuts

  • Alt+SysRq+r takes keyboard and mouse control from the X server.
    • This can be useful if the X-Server crashed, you can change
    • to a console and kill the X-Server or check the error log.
  • Alt+SysRQ+k kills all processes on the current terminal.
    • Its a bad idea to do this on a console where X is running.
    • The graphic will stop and you cant see what you type.
  • Alt+SysRQ+b is like a reset: a reboot without umounting or sync.
  • Alt+SysRQ+o shuts down via APM.
  • Alt+SysRQ+s writes all data from the disc cache to the harddiscs, its a sync.
  • Alt+SysRQ+u remounts all mounted filesystem readonly. After using this key,
    • you can reboot the system with Alt+SysRQ+b without harming the system.
  • Alt+SysRQ+m prints memory information to the console.
  • Alt+SysRQ+e sends SIGTERM to all processes except init.
  • Alt+SysRQ+i sends SIGKILL to all processes except init.
  • Alt+SysRQ+l sends SIGKILL to all processes, inclusive init.
    • (The system is not working after using this.)…

9.9.2 To shut down the system after a really bad kernel panic, do the following:

Alt+SysRQ+e
sends TERM-signal, processes can shutdown properly (e.g. save data)
Alt+SysRQ+u
a sync will be done when unmounting anyway
Alt+SysRQ+i
for the processes that didn't listen for the TERM signal, this is a kill -9 process
Alt+SysRQ+b
reboot

9.9.3 To be able to use the SysRq feature, you need to do

echo "1" > /proc/sys/kernel/sysrq

or add an entry to /etc/sysctl.conf:

kernel.sysrq = 1

10 hardware

  • HDD / SSD
  • list hardware found on PCI interface
    lspci
    

10.1 CPU

list CPU info:

lscpu

10.1.1 CPU frequency management

Monitor current CPU frequency:

watch grep \"cpu MHz\" /proc/cpuinfo
  1. list available CPU frequency governors
    cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors
    

    usually they are:

    • userspace
    • powersave
    • conservative
    • ondemand
    • performance
  2. see current CPU frequency governor
    cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
    
  3. set CPU frequency governor

    install package

    sudo apt-get install cpufrequtils
    

    create file

    /etc/default/cpufrequtils
    

    with content like

    GOVERNOR="conservative"
    

10.2 view system device information by bus, class, and topology

systool

10.3 keyboard

10.3.1 enable keyboard layout on HDD encryption password entry

edit

/etc/initramfs-tools/initramfs.conf

ensure that

KEYMAP=y

then issue

sudo update-initramfs -k all -u

10.3.2 choose keyboard layout via commandline

  1. Estonian
    setxkbmap -model armada -layout ee,us
    
  2. Russian
    setxkbmap -model armada -layout ru,us
    
  3. USA
    setxkbmap -model armada -layout us
    

10.4 RAM

10.4.1 determine what type of RAM is installed

sudo dmidecode --type 17

10.4.2 KSM - kernel same page merging

Is a memory-saving de-duplication feature.

  1. enable KSM at boot

    insert info file

    /etc/rc.local
    

    following line

    echo 1 > /sys/kernel/mm/ksm/run
    
  2. KSM interface:
    /sys/kernel/mm/ksm/
    
    1. control
      file purpose default
      pages_to_scan how many present pages to scan before ksmd goes to sleep 100
      sleep_millisecs how many milliseconds ksmd should sleep before next scan 20
    2. monitor
      file indicator
      pages_shared how many shared pages are being used
      pages_sharing how many more sites are sharing them i.e. how much saved
      pages_unshared how many pages unique but repeatedly checked for merging
      pages_volatile how many pages changing too fast to be placed in a tree
      full_scans how many times all mergeable areas have been scanned
      • A high ratio of pages_sharing to pages_shared indicates good sharing.
      • High ratio of pages_unshared to pages_sharing indicates wasted effort.

10.4.3 kswapd - The Kernel Swap Daemon

swap daemon tries three ways to reduce the number of physical pages being used by the system:

  • Reducing the size of the buffer and page caches,
  • Swapping out shared pages,
  • Swapping out or discarding pages.

10.4.4 /proc/sys/vm/

  1. min_free_kbytes

    This is used to force the Linux VM to keep a minimum number of kilobytes free. The VM uses this number to compute a pages_min value for each lowmem zone in the system. Each lowmem zone gets a number of reserved free pages based proportionally on its size.

    min_free_kbytes changes the page reclaim thresholds. When this number is increased the system starts reclaiming memory earlier, when its lowered it starts reclaiming memory later. In adition, this controls the number of free pages the system maintains.

    You don't want to make this value too low: the kernel frequently requires memory to be readily available, so a low value would force applications to sleep while kswapd tries to free up some memory, effectively causing some contention issues. On the other hand, a high value would limit the effectiveness of the page cache and would also wake up the kswapd daemon more frequently. This could hurt system performance.

    1. Defaults

      The default min_free_kbytes is calculated at boot time based on the number of pages of physical RAM available on the system. Default values under 2.6.18 (used for Red Hat Enterprise Linux 5) are:

      RAM min_free_kbytes
      3171 * 16MB: 512k
      3172 * 32MB: 724k
      3173 * 64MB: 1024k
      3174 * 128MB: 1448k
      3175 * 256MB: 2048k
      3176 * 512MB: 2896k
      3177 * 1024MB: 4096k
      3178 * 2048MB: 5792k
      3179 * 4096MB: 8192k
      3180 * 8192MB: 11584k
      3181 * 16384MB: 16384k

      The algorithm that computes those values is essentially:

      min_free_kbytes = sqrt(lowmem_kbytes * 16)

      For some systems, this algorithm could produce default values that are too low compared to the stress which will be imposed on the memory subsystem. It might thus be important to multiply the values by a factor of two or four, depending on how much memory we can afford to keep aside for the kernel.

      Note that these values used to be smaller under kernel 2.6.9 and earlier. Four times smaller, more precisely.

      In order to determine when kswapd should start freeing unused pages of memory, min_free_kbytes affects three different kernel variables:

    2. pages_high

      sets the point after which kswapd will stop trying to free up unused memory and go back to sleep.

    3. pages_low

      is used to wake up kswapd in a situation where we are running out of memory. Whenever the system reaches this watermark, kswapd starts freeing up unused pages of memory in order to let the kernel allocate more memory.

    4. pages_min

      is the lowest threshold. Bellow that value, one page of RAM has to be freed before an application can allocate another page. The kernel, however, can make use of the memory available bellow this limit. This is essentially the value that min_free_kbytes corresponds to.

    5. To make change permanent across reboots

      add this line to /etc/sysctl.conf:

      vm.min_free_kbytes=50000
      
  2. lowmem_reserve_ratio

    Ratio of total pages to free pages for each memory zone.

  3. overcommit_memory

    This file contains the kernel virtual memory accounting mode. Values are:

    0 heuristic overcommit (this is the default)
    1 always overcommit, never check
    2 always check, never overcommit

    In mode 0, calls of mmap(2) with MAP_NORESERVE set are not checked, and the default check is very weak, leading to the risk of getting a process "OOM-killed". Under Linux 2.4 any non-zero value implies mode 1.

    In mode 2 (available since Linux 2.6), the total virtual address space on the system is limited to (SS + RAM*(r/100)), where SS is the size of the swap space, and RAM is the size of the physical memory, and r is the contents of the file /proc/sys/vm/overcommit_ratio.

    In sysctl config file corresponding keys are:

    vm.overcommit_memory = 2
    vm.overcommit_ratio = 80
    
  4. laptop_mode

    laptop_mode is a knob that controls "laptop mode". When the knob is set, any physical disk I/O (that might have caused the hard disk to spin up, see /proc/sys/vm/block_dump) causes Linux to flush all dirty blocks. The result of this is that after a disk has spun down, it will not be spun up anymore to write dirty blocks, because those blocks had already been written immediately after the most recent read operation. The value of the laptop_mode knob determines the time between the occurrence of disk I/O and when the flush is triggered. A sensible value for the knob is 5 seconds. Setting the knob to 0 disables laptop mode.

  5. nr_pdflush_threads

    The count of currently-running pdflush threads. This is a read-only value.

  6. page-cluster

    page-cluster controls the number of pages which are written to swap in a single attempt. The swap I/O size. It is a logarithmic value - setting it to zero means "1 page", setting it to 1 means "2 pages", setting it to 2 means "4 pages", etc. The default value is three (eight pages at a time). There may be some small benefits in tuning this to a different value if your workload is swap-intensive.

  7. panic_on_oom

    This enables or disables panic on out-of-memory feature. If this is set to 1, the kernel panics when out-of-memory happens. If this is set to 0, the kernel will kill some rogue process, by calling oom_kill(). Usually, oom_killer can kill rogue processes and system will survive. If you want to panic the system rather than killing rogue processes, set this to 1. The default value is 0.

  8. vfs_cache_pressure

    Controls the tendency of the kernel to reclaim the memory which is used for caching of directory and inode objects.

    At the default value of vfs_cache_pressure = 100 the kernel will attempt to reclaim dentries and inodes at a "fair" rate with respect to pagecache and swapcache reclaim. Decreasing vfs_cache_pressure causes the kernel to prefer to retain dentry and inode caches. Increasing vfs_cache_pressure beyond 100 causes the kernel to prefer to reclaim dentries and inodes.

10.4.5 RAM overcommit

Linux can allow applications to allocate more RAM than is actually available in hope that it will actually not be used

  1. /proc/sys/vm/overcommit_memory
    0
    Heuristic overcommit handling. Obvious overcommits of address space are refused. Used for a typical system. It ensures a seriously wild allocation fails while allowing overcommit to reduce swap usage. root is allowed to allocate slighly more memory in this mode. This is the default.
    1
    Always overcommit. Appropriate for some scientific applications.
    2
    Don't overcommit. The total address space commit for the system is not permitted to exceed swap plus a configurable percentage (default is 50) of physical RAM. Depending on the percentage you use, in most situations this means a process will not be killed while attempting to use already-allocated memory but will receive errors on memory allocation as appropriate.
  2. /proc/sys/vm/overcommit_ratio

    Percentage of physical memory size to include in overcommit calculations.

    Memory allocation limit = swapspace + physmem * (overcommit_ratio / 100)

    • swapspace = total size of all swap areas
    • physmem = size of physical memory in system

10.4.6 shared memory

list shared memory areas

ipcs

remove shared memory

ipcrm -m

10.4.7 system RAM usage info

free

10.4.8 UKSM - Ultra KSM

  1. intro

    Linux kernel has a feature named KSM(Kernel SamePage Merging). it lets the hypervisor system share identical memory pages amongst different processes or virtualized guests. However, it has its limitation such like high CPU usage and slow responses to workload change. So here comes UKSM. With a revolutionary algorithm redesign, UKSM has many advanced features:

    1. Full system scan.

      It automatically scans all user processes’ anonymous VMAs. Before UKSM, a process need to call KSM kernel API to submit its memory areas to KSM for scan. This makes many legacy software other than KVM can NOT benefit from KSM. Now, UKSM scans whole system applications including KVM. All programs benefit from UKSM without even knowing how it works.

    2. Super quiet CPU usage.

      It automatically detects rich areas containing abundant duplicated pages. Rich areas are given a full scan speed. Poor areas are sampled at a reasonable speed with very low CPU consumption usually under 1%. Our benchmarks show that even for CPU intensive workloads, it has a negligible performance impact.

    3. Ultra scan speed and CPU efficiency.

      A new hash algorithm is proposed. As a result, on a machine withCore(TM)2 Quad Q9300 CPU in 32-bit mode and 800MHZ DDR2 main memory, it can scan memory areas that does not contain duplicated pages at speed of 627MB/sec ~ 2445MB/sec and can merge duplicated areas at speed of 477MB/sec ~ 923MB/sec. For a bunch of busy workloads creating lots of duplicated pages, this means, with UKSM, you no long suffer from deadly swapping which is observed in KSM enabled setting.

    4. Thrashing area avoidance.

      If a VM is constantly writing the same data to the duplicated pages, KSM suffer a problem of thrashing, i.e. the pages it merges will soon get copied again. In this situation, the CPU is purely wasted without having much memory saving. UKSM can perfectly avoid this situation by detecting the thrashing areas.

    5. conclusion

      In short, if you are managing a Linux server with lots of KVM virtual machines or containers(e.g. OpenVZ, LXC), UKSM is a must have feature! If you are desktop user, you also benefit a lot from it.

      (via:http://kerneldedup.org/en/projects/uksm/introduction/).

  2. usage

    After you reboot with our uksm enabled kernel, uksm works as a system daemon — uksmd. The main interfaces for uksmd are under

    /sys/kernel/mm/uksm/
    
    1. user controllable interfaces:
      1. run

        The main control interface to stop/run. e.g. echo 1 > run.

      2. sleep_millisecs

        how long will uksmd sleep before it wakes up and does its work.

      3. cpu_governor:

        The macro governors for controlling how aggressively the uksmd uses CPU. You can do cat cpu_governor to find out the available governors and which is currently active. e.g. [full] medium low quiet shows that currently it works at full speed. These governors roughly stand for 90%, 50%, 20%, <1% of max CPU consumption and some pre-setting of micro-control parameters. (NOTE: for max it means only when uksmd is working at memory areas with many many redundancy pages. For most of the time uksmd is very quiet)

      4. max_cpu_percentage:

        The max CPU consumption you can further specify under some cpu governor. It’s used when you are not happy with the preset values. read-only interfaces: full_scans: how many times uksmd sampling has covered all of your memory areas. hash_strength: the current hash strength for adaptive-random-sampling hashing algorithm. The lower the faster uksmd gonna merge pages. Its range is 1 ~ 1034. pages_scanned: how many pages has been scanned. pages_shared: how many pages has been mapped by different page table entries, i.e. shared by different processes. pages_sharing: this is roughly the number of redundant pages you have saved, with uksm. sleep_times: how many times uksmd has waken up and then got sleep. Since version of 0.1.1,the pages with full zeros will considered specially and seperately. You can get the information for uksmd merged zero pages by “cat /proc/meminfo | grep KsmZeroPages”. (via:<a href="http://kerneldedup.org/en/projects/uksm/uksmdoc/usage/usage0-1-2">http://kerneldedup.org/en/projects/uksm/uksmdoc/usage/usage0-1-2</a>/).

    2. read-only interfaces:
      1. full_scans

        how many times uksmd sampling has covered all of your memory areas.

      2. hash_strength

        the current hash strength for adaptive-random-sampling hashing algorithm. The lower the faster uksmd gonna merge pages. Its range is 1 ~ 1034.

      3. pages_scanned

        how many pages has been scanned.

      4. pages_shared

        how many pages has been mapped by different page table entries, i.e. shared by different processes.

      5. pages_sharing

        this is roughly the number of redundant pages you have saved, with uksm.

      6. sleep_times

        how many times uksmd has waken up and then got sleep.

10.4.9 zswap - lightweight compressed cache for swap pages

enable

echo 1 > /sys/module/zswap/parameters/enabled

check

cat /sys/module/zswap/parameters/enabled

10.5 power management

10.5.1 turn off computer

turn off computer, now

shutdown now -h

turn off after 30 min.

shutdown +30

10.5.2 disable suspending when laptop lid is closed

Edit

/etc/systemd/logind.conf

and make sure you have,

HandleLidSwitch=ignore

10.5.3 disable USB suspend

  1. using modprobe
    sudo echo "options usbcore autosuspend=-1" | sudo tee /etc/modprobe.d/disable-usb-autosuspend.conf > /dev/null
    
  2. using Linux parameter

    In file

    /etc/default/grub
    

    change the line

    GRUB_CMDLINE_LINUX_DEFAULT
    

    to add the option

    usbcore.autosuspend=-1
    

    After saving the file, update grub

    sudo update-grub
    

11 users and groups

11.1 sudo - gives one user privileges of another one

Example: gives current user superuser privileges:

sudo -s

11.2 show currently logged in user name

whoami

11.3 user

11.3.1 create user

Friendly way:

sudo adduser <username>

Low level way: create user and add it to the group

useradd <username> -g <group>

11.3.2 remove user

Friendly way:

deluser <username>

Low level way:

userdel <username>

11.3.3 set user password

passwd <username>

11.4 group

11.4.1 create group

friendly way

addgroup <groupName>

low level way

groupadd <groupName>

11.4.2 add user to group

example:

sudo adduser <user> <group>

alternative command:

sudo usermod -a -G <group> <user>

11.4.3 remove user from group

edit file */etc/group*

11.4.4 what groups user belong to ?

id -nG <username>

11.4.5 set user main group

sudo usermod -G <group> <user>

11.4.6 system groups

audio members of this group has permission to access audio subsystem
floppy members of this group has access to floppy drive
wheel members of this group have administrative rights