Linux Processes:Administration and Monitoring tutorial

The brain of any operating system is its kernel. But a computer with only an operating system(kernel) is of no use, because nobody really uses an operating system. The end user of a computer is always concerned about programs, and not the operating system.

What an operating system really does is to provide an environment, where these programs can run. So the only job of an operating system is to provide resources to program's, so that the program can complete its task. People forget the difference between programs and operating systems, simply because, modern day operating system installs a good number of programs along with the operating system itself.

When you run a program, the program will start one or multiple processes, to complete the task it was designed to do. Processes which get's started, will execute and die on its own without any user's intervention. This wonderful task of flawlessly handling the complete life of a process is done by the kernel itself. For a Linux system administrator, its important to understand methods used to analyze, monitor and modify the running processes. In this post we will be going through different aspects of a process in Linux to gain some better understanding about the topic.

Running any command in a Linux system will result in a process being created, which will die after it has finished the job it was created to do. Let's begin this by getting some basics clear. As i told before, each and every program will create one or more process. At any point of time, there are multiple processes that are running on the system.

Each and every process in a Linux operating system is identified with the help of a number called as process id.

What is process id or pid in Linux?

Process ID is nothing but a unique identification number that helps to identify a particular process in Linux. PID's are 16 bit number's that are sequentially assigned to different processes one after the other. Process ID is also referred to as Thread Group Identifier (TGID). If you want to explore few more details about threads, i would recommend reading the below article.

Read: How is Thread different from a Process in Linux?

After the execution of a process, the process id number is released from the process table, for reuse.

PID itself stands for process identifier. Any modification targeted to a process must be done, with the help of the Process Identifier, for that process. Addressing a process is not at all possible without a character, that is unique to that process. PID is unique to each and every processes running on the system. The process id number starts from 1 (which is always reserved for a process called init.)

INIT is the only process that gets the same PID number each and every time. INIT is the first program that's run by a Linux system, due to which a process id number of 1 is given to it(process id number's in Linux are always assigned starting from 1 sequentially). As INIT is the first process that get's started, it is also the parent of all other processes in Linux. Let me pin point some of the noteworthy things, to make the concept more clear.

• Init is the parent and the first process, which always gets the process ID number of 1.
• There is a limit to the maximum number of processes that can be run on a Linux operating system. This was introduced from kernel version 2.5 onwards.

And the maximum number of PID of a Linux system can be found from the file /proc/sys/kernel/pid_max.

[root@myvm1 ~]# cat /proc/sys/kernel/pid_max
32768
[root@myvm1 ~]#

The above output shows that, the maximum number of process id's are 32768. Which means 32768 number of processes can be run simultaniously at one time in the system.

• Higher PID number of 30000, on a system does not mean 30000 processes are running on the system. A process ID number of 1000 does not always mean that it was started before the PID number 30000. This is because a process id number is always reused once a process releases it.
• If you have large amount of RAM available, increasing the total number of PID is a nice method, to increase the number of processes that a server can handle.
• You can increase the maximum number of process ID on a Linux system with the help of the below command.

You can modify sysctl.conf file to modify the this limit of max PID in Linux.

4194303 is the maximum number of PID that a 64 bit Linux machine can have. So you can modify your pid_max, with a higher value like 4194303, by modifying sysctl.conf file in Linux.

kernel.pid_max=<maximum value>

How to list processes running on a Linux machine?

There are different tools available in Linux, that provides you with a list, of all the processes that are currently running on the system. One such commonly used tool is ps

Let's see what are the contents of its output and understand each of the filed. Understanding each and every field of ps command, will give a more in depth idea of processes and their different parameter's.

[root@myvm1 ~]# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.1  0.1   2064   624 ?        Ss   08:16   0:00 init [3]
root         2  0.0  0.0      0     0 ?        S<   08:16   0:00 [migration/0]
root         3  0.0  0.0      0     0 ?        SN   08:16   0:00 [ksoftirqd/0]
root         4  0.0  0.0      0     0 ?        S<   08:16   0:00 [watchdog/0]
root         5  0.0  0.0      0     0 ?        S<   08:16   0:00 [events/0]
root         6  0.0  0.0      0     0 ?        S<   08:16   0:00 [khelper]
root         7  0.0  0.0      0     0 ?        S<   08:16   0:00 [kthread]
root        10  0.0  0.0      0     0 ?        S<   08:16   0:00 [kblockd/0]
root        11  0.0  0.0      0     0 ?        S<   08:16   0:00 [kacpid]
root       175  0.0  0.0      0     0 ?        S<   08:16   0:00 [cqueue/0]
root       178  0.0  0.0      0     0 ?        S<   08:16   0:00 [khubd]
root       180  0.0  0.0      0     0 ?        S<   08:16   0:00 [kseriod]

If you see the ps command man page, the option aux which i have used in the above shown example mentioned as "To see every process on the system using BSD syntax:"

Let's understand each and every column of the output. The first column is the USER column, which gives you the name of the user, who started the process. In other words, it tells you the owner of the process.

PID(Second Column) gives you the PID number of the process.(which we have already discussed in the previous section of this article.). If you notice the ps command output, the INIT process is having a process ID number of 1(which testifies the that its the first process that was started in the system.)

%CPU(Third Column) is the amount of CPU in percentage, the process is currently using. This column can be very helpful during crisis to identify the process, which are CPU intensive.

%MEM(Fourth Column) is the amount of memory utilized by the process in percentage.

VSZ and RSS tell's you the amount of memory used by the process. VSZ is used to show the amount of virtual memory used by the process, however RSS(Resident Set Size), is the portion of RAM(Physical memory) that a process is using.

TTY shows you the terminal name from where the process was started. Whenever you execute a command, to start a process. The terminal, from where the process was started will be shown under TTY column in ps output.

[root@myvm1 ~]# ps aux | grep ls
root     13594  0.0  0.1   3912   672 pts/1    R+   09:46   0:00 grep ls
[root@myvm1 ~]# tty
/dev/pts/1
[root@myvm1 ~]#

The above shown example, servers two purposes. Firstly it shows that, whatever command you type, it will start a process with that command name. In the above case a process got started with the command name of "grep ls" as soon as the command was run(in our case this process is very much momentary, as the output is already shown and the process also got exited.)

In the ouput, tty column contains a pts/1 which is the tty name from where the command was issued, to confirm, we have also checked the tty name with the help of tty command.

But in some cases, there will be a question mark (?) instead of the tty name in that column. Question mark in tty field appears, when the process was not started from a TTY. This means the process is either a daemon, or process started by the system, or a process related to a cron job.

Process state in PS Command Output

A process can be classified into different state's, depending upon the current status. For example, a process can be idle, waiting for a user input, a process can be waiting for an IO request to be completed, A process can be zombie, or even orphan. Let's go through different states of a process in Linux.

Let's first understand interruptible and uninterruptible processes in Linux.

Processes which are not currently doing any task, and are waiting for any interrupt, or say for example a process which is waiting for an input from the user is normally in the interruptible sleep state. This state of a process is denoted by "S". You will find a large number of processes in the system with an S state. This is because, most of the processes are in the started state, and are waiting for something to execute as required.

Processes that are waiting for something to get completed are said to be in uninterruptible state. For example, a process that is waiting for an IO request to be completed goes into an Uninterruptible state. Such a state is denoted by D  in the ps command output. You cannot send any message, to a process that is in uninterruptible state, because it will not accept any signal or message. You will normally not see, too many processes in D state.

Related: IO monitoring in Linux

Process status column also tells the priority of the process over other processes. There are two symbols that indicate the priority of the process.

"<" indicate's that the process is having a higher priority than other processes. In other words, you can also say that the process with < status symbol indicates the fact that its not nice to other processes in terms of priority in execution.

On the other hand N, indicates that the process is nice to other's. Which means a process with the status of N  has a lower priority compared to other processes running on the system.

If you want to understand, methods to change priority of processes in linux, then the below post will be very much helpful.

Read: How to change priority of a process

what is zombie and orphan processes in linux?

Two other widely discussed status are called as zombie and orphan. Let's discuss both zombie and orphan process one by one.

We had discussed a process called INIT during the beginning of this article. INIT is the parent process of all other processes in Linux. It is first process that runs on a Linux system. All processes that run on a Linux system has a parent process to it. For example, you open a shell, and run a process called elinks(which is a text based console browser.)

In this case, the elinks process is the child of a parent process called as bash(the shell from which it was started.). Similarly, if you have httpd web server running on the system, all other httpd processes will be spawned by a single master httpd process(which is started by a user, using the init script.). Let's see the parent process of an already running Apache process with the help of ps command.

[root@myvm1 ~]# ps aux | grep 16975

apache   16975  0.0  0.7  23620  3812 ?        S    13:47   0:00 /usr/sbin/httpd

In the above example, we can see that the PID no 16975 is an Apache process. Let's see the parent PID of this process, with the help of ps command.

[root@myvm1 ~]# ps -p 16975 -o ppid=
16969

Now from the above output we know that 16969 is the parent process id of the process 16975. Let's see what process is in fact this parent process.

[root@myvm1 ~]# ps aux | grep 16969
root     16969  0.0  1.4  23620  7520 ?        Ss   13:47   0:00 /usr/sbin/httpd

Did you see that, the parent process is the /usr/sbin/httpd script started by root user. This way each and every process has a relationship with their parent process. At the end of the day, init is the parent of all processes.

Let's get back to our discussion of zombie and orphan process. Zombie process has all the characteristics of a real zombie(A dead body without a soul is called as a zombie). A zombie process is called a zombie, because, its a dead process without any task left to do(without utilizing any system resource) just sitting in the process table to be reaped by the parent.

when a process completes its execution, it informs its parent process that it has completed its execution. Now after being informed, its the job of the parent to get the complete status of the process that has completed execution, and then finally remove it from the process table, so that the pid is available for reuse. There are some irresponsible parent processes that takes a lot of time to collect the complete status of its children, even after being informed. Due to this reason, the children becomes zombies till the parent does its job of collecting information and removing them from the list.

As i said, its the irresponsible parents, who are responsible for the existence of zombie processes. Generally this happens, due to programming inefficiencies and bugs. But generally a zombie cannot cause any sort of harm, because they are dead bodies without soul(They does not consume any system resources. They are just an entry in the process table, that needs to be removed by the parent). But yeah if you have a large number of zombie processes on the system, then you will eventually run our of PID number's. Because until and unless a process releases the PID, it cannot be reused, and zombie does not release the PID. Zombie processes are denoted by the status symbol of Z.  You cant kill a zombie, because its a dead body, and how will you kill a dead body. But yeah you can surely ask the parent process to reap their child processes that have completed execution.

This can be done by sending SIGCHLD, signal to the parent. This SIGCHLD signal will ask the irresponsible parent to reap their children, who are zombies. Suppose 2343 is the parent process id of a zombie process(you can easily find out the parent process of any process in Linux, by the same ps command i have shown previously in this tutorial), you can send a sigchld signal to it as shown below.

[root@myvm1 ~]# kill -s SIGCHLD 2343

Now let's discuss what is an orphan process in Linux. Orphan processes are those unfortunate processes whose parents die leaving the children behind. Parents can die due to accident (which means can crash). However some powerful processes immediately adopts the children, whose parents are no more, and are orphan. So for example, INIT adopts almost all orphan process immediately.

In such cases, you will see PID no 1 as their parent process.

In Unix and Linux system's, whenever you kill the parent process, all children get's destroyed. This normally happens, until and unless, a process is intentionally made orphan, so that init adopts it as its child. This process of intentionally making a process orphan, is useful when you want to run a program from a shell, for a very long period of time, and it also does not require any user intervention in between. Normally when you run a program from a command line shell, the program becomes the child of the shell process. So when you logout from that shell, all child gets killed. But in some cases, you want the process to run until it gets completed by itself.

This can be achieved by purposely making a process orphan, so that the INIT process immediately adopts it. This is done with the help of a command called as nohup in Linux

[root@myvm1 ~]# nohup sh sample.sh &
[2] 22416
[root@myvm1 ~]# nohup: appending output to `nohup.out'

The above command starts the sample.sh process with nohup, so that it does not get killed even after the shell is exited. And the final & indicates that the process should be run in background(Don't worry i will be explaining background and foreground process in some time).

What is foreground and background processes in linux?

Previously when we were running our example script using & at the end of the command, it makes that process go in the background. Now let's understand what's a background and foreground process.

To understand what is background and foreground, let's see an example. If you are working on a linux terminal, then each and every command you execute will first need to complete, before you get the shell prompt. For example, if you want to run a command like yum update, from terminal, you need to first wait for the command to complete, before you can run another command on the same terminal.

What if you want to run multiple commands one after the other at the same time, without waiting for each of them to complete. In order to do this, you need to send a command(the process), in the background, before you can run another command. Whenever you run a command, its by default run on the foreground. But there are methods that can be used to send a foreground process to background.

[root@myvm1 ~]# yum update

When you run the above command, it will run the foreground by default, and you will have to wait till the command completes, to get the shell prompt back again to run another command. But you can pause the process or say suspend the process with the help of a CTRL + Z. Let's do that.

[root@myvm1 ~]# yum update
[1]+  Stopped                 yum update
[root@myvm1 ~]#

As soon as i press CTRL + Z, the process was suspended/stopped. Which is clear from the above message [1]+ stopped yum update. Pressing CTRL+Z, is not a good solution to run another command or to get the shell prompt back, because it suspends the already running process. We need the system to run this process, as well as allow us to run another commands, by releasing the shell prompt.

[1] indicates the job number id. As we only have one job here, which is suspended it is given the job id number of 1. Now let's resume this process by bringing it to the foreground.

[root@myvm1 ~]# fg %1
yum update

But even if we resume the process, with the help of fg %1, it will once again come to the foreground, and you still need to wait till the process gets completed, to run another commands. A good solution to this problem is to send the process to background, so that the process will continue to run, as well you will get the shell prompt back, to run another commands.

[root@myvm1 ~]# bg %1
[1]+ yum update &

The above command, will resume the yum update process and will also put that process in the background, so that you can run other commands. If you have multiple processes, running in the background, then you can view their status, with the help of jobs command, as shown below.

[root@myvm1 ~]# jobs
[1]+  Running                 yum update &
[root@myvm1 ~]#

You can anytime switch between background and foreground with the help of fg and bg command in linux, by simply giving the exact job number as the argument(%1,%2 etc)

Alternatively you can always use the symbol &, immediately after the command, to send it to the background. So for example, let's send the same yum update command to background during run time.

[root@myvm1 ~]# yum update &
[1] 31281

As you can see, using the &, symbol will also do the same thing of assigning a job id number and sending it to background.

Some process monitoring commands

The PS command we saw previously does a handsome job for getting the complete details, of processes running on the system. However there is one another tool available in linux, that will help you to monitor current and active states of all the processes, live. The most useful feature of this process monitoring tool is the ability to interact with the user and sort the output according to the requirement.

This is none other than the top command utility.

top - 05:23:08 up  7:09,  4 users,  load average: 0.00, 0.11, 0.10
Tasks:  94 total,   1 running,  93 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.7%us,  0.7%sy,  0.0%ni, 97.0%id,  0.0%wa,  0.3%hi,  0.3%si,  0.0%st
Mem:    515444k total,   441964k used,    73480k free,    73364k buffers
Swap:        0k total,        0k used,        0k free,   276772k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
31571 root      15   0  2196 1008  804 R  0.7  0.2   0:00.02 top
 4535 root      15   0  1968  644  564 S  0.3  0.1   0:23.77 hald-addon-stor
    1 root      15   0  2064  624  536 S  0.0  0.1   0:00.94 init
    2 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 migration/0
    3 root      34  19     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/0

top commands will keep on updating its stats, each and every 3 seconds by default. However you can change this value with a command line option

[root@myvm1 ~]# top -d 1

Although top command, is a pretty nice tool to monitor process details on a linux system, it also provides you the complete overview of the system in one interface. If you see the top area of the command output, you will be able to see the complete details of a running system.

It shows you the below things.

• No of user's logged in
• System Load average
• Uptime
• Total number of process, with how many are sleeping, running, stopped, or zombie.
• CPU statistics, memory statistics

You can sort the output of the top command, based on memory usage, cpu usage, swap usage etc. With the help of sorting option's available in the command. You can get the options by pressing SHIFT + O, which will display the list of sorting options.

  j: P          = Last used cpu (SMP)
* K: %CPU       = CPU usage
  l: TIME       = CPU Time
  m: TIME+      = CPU Time, hundredths
  n: %MEM       = Memory usage (RES)
  o: VIRT       = Virtual Image (kb)
  p: SWAP       = Swapped size (kb)
  q: RES        = Resident size (kb)

From the sorting options available, you can sort by typing the letter corresponding to the required sorting and press ENTER. So for example, if you want to sort the top output for memory usage, then you can do that by first getting inside this sorting menu by pressing SHIFT + O and then pressing n followed by an ENTER.

Let's see another interesting command, that will display the parent child relationship of all the processes in the system.

[root@myvm1 ~]# pstree
init-+-acpid
     |-atd
     |-auditd-+-audispd---{audispd}
     |        `-{auditd}
     |-automount---4*[{automount}]
     |-avahi-daemon---avahi-daemon
     |-crond
     |-cupsd
     |-dbus-daemon
     |-dhclient
     |-events/0
     |-gam_server
     |-gpm
     |-hald---hald-runner-+-hald-addon-acpi
     |                    |-hald-addon-keyb
     |                    `-hald-addon-stor

Wow that's nice, the above pstree command, shows you the complete process tree of the system. From the beginning till the end. If you see INIT is at the top, as its the parent of all processes.

Most of the command line utilities that shows you the process details, fetch their information from the /proc, file system. If you go inside the /proc file system, you will be able to see seperate directories for seperate processes.

[root@myvm1 proc]# ll
total 0
dr-xr-xr-x  5 root      root              0 May 11 07:59 1
dr-xr-xr-x  5 root      root              0 May 11 07:59 10
dr-xr-xr-x  5 root      root              0 May 11 07:59 11
dr-xr-xr-x  5 root      root              0 May 12 06:10 1443
dr-xr-xr-x  5 root      root              0 May 12 06:10 1446
dr-xr-xr-x  5 root      root              0 May 11 07:59 1684
dr-xr-xr-x  5 apache    apache            0 May 11 13:47 16971
dr-xr-xr-x  5 apache    apache            0 May 11 13:47 16972
dr-xr-xr-x  5 apache    apache            0 May 11 13:47 16973

These seperate PID directories contains the complete process information. Let's see the contents of a PID directory under /proc. Let's get inside a PID directory called 4282 to see its contents.

[root@myvm1 4282]# ls
attr             cpuset   fd        mem         oom_score  stat    wchan
auxv             cwd      limits    mounts      root       statm
cmdline          environ  loginuid  mountstats  schedstat  status
coredump_filter  exe      maps      oom_adj     smaps      task

Although i will not be able to explain each and every file and directory contents inside, let's understand some of them.

• cmdline file tells about the command line arguments that were used while running this process.

for example, the pid 4282 is a mysql process, lets see the contents of the cmdline file.

[root@myvm1 4282]# cat cmdline
/usr/libexec/mysqld--basedir=/usr--datadir=/var/lib/mysql--user=mysql--federated--log-error=/var/log/mysqld.log--pid-file=/var/run/mysqld/mysqld.pid--socket=/var/lib/mysql/mysql.sock

• CWD links the process to the working directory of the process
• exe is the executable of the process.
• environ file contains the environment variables of the process.

[root@myvm1 4282]# cat environ
CONSOLE=/dev/consoleSELINUX_INIT=YESTERM=linuxINIT_VERSION=sysvinit-2.86PATH=/sbin:/usr/sbin:/bin:/usr/binrunlevel=3RUNLEVEL=3PWD=/LANG=en_US.UTF-8previous=NPREVLEVEL=NSHLVL=3HOME=/MYSQL_HOME=/usr_=/usr/bin/nohup

The file that contains a complete information about the process is the status file, let's see the content of that file.

[root@myvm1 4282]# cat status
Name: mysqld
State: S (sleeping)
SleepAVG: 96%
Tgid: 4282
Pid: 4282
PPid: 4169
TracerPid: 0
Uid: 27 27 27 27
Gid: 27 27 27 27
FDSize: 256
Groups: 27
VmPeak:   149852 kB
VmSize:   132704 kB
VmLck:        0 kB
VmHWM:    22248 kB
VmRSS:    22060 kB
VmData:   118972 kB
VmStk:       84 kB
VmExe:     6764 kB
VmLib:     5580 kB
VmPTE:      104 kB
StaBrk: 092ce000 kB
Brk: 09719000 kB
StaStk: bfa2fcd0 kB
ExecLim: 086e3000
Threads: 10
SigQ: 0/8192
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000087007
SigIgn: 0000000000001006
SigCgt: 00000001800066e9
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
Cpus_allowed: 00000001
Mems_allowed: 1

Oh yeah that file contains the complete details of the process. Most of the system monitoring tools, fetch process information from this location only. Lets see what all information is provided by this file.

• Process Name
• PID
• Parent Process ID, indicated bu PPID
• uid and gid of the process
• state of the process
• Virtual memory and real memory statistics etc.

Sending Signal's to process in Linux

Most of the people who use Linux in day to day lives are aware of a command called kill, although the command is commonly used to kill a process, there are other things that can be achieved using KILL.

One major task done by kill command is to send signal's to a running process. There are various types of signal's that can be send to a process. The way the process responds to a particular signal, depends upon how that process was programmed. Let's discuss some commonly used kill signal's.

We have discussed that programs run from a terminal get's exited, when you exit the terminal. Basically when you exit the terminal, the program's run from that terminal receives a SIGHUP signal.

SIGHUP signal tell's the process to hang up. Basically this signal will terminate the process.

What happens when you type CTRL + C or CTRL + Z, to a running process. CTRL + C will send a SIGINT, signal to the running process(Most of the times, it will terminate the running application.)

We have previously seen that a running program can be suspended using a CTRL + Z. The shell will send a SIGSTOP signal to a running program when you press CTRL + Z.

Kill command is used with different signal level's as an argument. Let's see some example's of KILL command in Linux.

Example 1)

#kill -1 <PID NO>

The above command will send a SIGHUP signal to a running process. It all depends upon how a program is configured to behave on receiving a SIGHUP signal.  For example, if you send a SIGHUP signal by using kill -1 to an Apache process, Apache will reread its configuration file. If you want to confirm this, you can always open the /etc/init.d/httpd script and search for "reload".

Example 2)

#Kill -2 <PID NO>

The above shown command will send a SIGINT, signal to a running process. Its very much the same as pressing a CTRl + C

Example 3)

#kill -9 <PID NO>

The above shown signal will send a SIGKILL, signal to a running process. This signal asks the process to exit immediately. This is a harsh method of killing a process. But the process will never ignore this signal.

A main disadvantage of this signal is that the process, will get killed ungracefully.

Example 4)

#Kill -15 <PID NO>

This is the most cultured form of KILL signal. This command will send a SIGTERM signal to a running process. This will ask the process to terminate itself gracefully. This is the best method that's advisable to use against any process in Linux.

SIGTERM is a safe kind of signal, and will cause no harm to the process.

Explaining all the available forms of signal's in Linux is beyond the scope of this article. You can get the complete list of signal's that the KILL command supports by running the below command.

[root@myvm1 ~]# kill -l
 1) SIGHUP       2) SIGINT       3) SIGQUIT      4) SIGILL
 5) SIGTRAP      6) SIGABRT      7) SIGBUS       8) SIGFPE
 9) SIGKILL     10) SIGUSR1     11) SIGSEGV     12) SIGUSR2
13) SIGPIPE     14) SIGALRM     15) SIGTERM     16) SIGSTKFLT
17) SIGCHLD     18) SIGCONT     19) SIGSTOP     20) SIGTSTP
21) SIGTTIN     22) SIGTTOU     23) SIGURG      24) SIGXCPU
25) SIGXFSZ     26) SIGVTALRM   27) SIGPROF     28) SIGWINCH
29) SIGIO       30) SIGPWR      31) SIGSYS      34) SIGRTMIN
35) SIGRTMIN+1  36) SIGRTMIN+2  37) SIGRTMIN+3  38) SIGRTMIN+4
39) SIGRTMIN+5  40) SIGRTMIN+6  41) SIGRTMIN+7  42) SIGRTMIN+8
43) SIGRTMIN+9  44) SIGRTMIN+10 45) SIGRTMIN+11 46) SIGRTMIN+12
47) SIGRTMIN+13 48) SIGRTMIN+14 49) SIGRTMIN+15 50) SIGRTMAX-14
51) SIGRTMAX-13 52) SIGRTMAX-12 53) SIGRTMAX-11 54) SIGRTMAX-10
55) SIGRTMAX-9  56) SIGRTMAX-8  57) SIGRTMAX-7  58) SIGRTMAX-6
59) SIGRTMAX-5  60) SIGRTMAX-4  61) SIGRTMAX-3  62) SIGRTMAX-2
63) SIGRTMAX-1  64) SIGRTMAX

Hope this tutorial was helpful in understanding and grasping several concepts related to processes in Linux.