Nginx Performance Tuning: How to do it

Nginx is a well known web server, and is adopted by many major players in the industry. The main reason for its fast adoption was that its so fast compared to other web servers (like Apache). Basically nginx was made to solve a problem that is known as c10k. It performs much better than any other web server's in the market out of the box for many. In this article we will see how you can modify your nginx configuration to give it a boost or say performance tune nginx.

We will get inside the configuration part a little later, coz there are quite a few concepts that needs to be understood first.

C10k refers to the method of optimizing network connections so that it can handle connections in the range of ten thousand simultaneously. There is a very famous article explaining this in the below link.

Read: What is c10k Problem

Apache works in a blocking I/O model. In layman terms it means that when a read/write request is issued it blocks other requests till that request is completed. A best solution to this problem is to create separate threads/processes for each connections. This is what Apache does.

Apache works by using a dedicated thread per client with blocking I/O.

Although dedicated thread per connections/request by Apache is a good method to serve clients, its a memory consuming as well as processor consuming method.

Its processor consuming, because for each request that hits the Apache server, the server processor has to switch between different processes (because each http requests connections creates a new process/thread)

Related: Process administration in Linux

Now if performance and resource utilization is not at all a concern for you, then you can go ahead with the thread per connection model of Apache by using an Apache web server to serve requests. But if you are concerned about both resource utilization and performance, then nginx model of event driven architecture is the best to go ahead with.

Nginx uses a single threaded non-blocking I/O mechnism to serve requests. As it uses non-blocking I/O, one single process can server too many connection requests.

 

Related: Nginx and Apache difference

As everything is a file in linux, even network connections are files internaly. Each and every process has its own set of file descriptors that needs to be polled frequently to identify which is one is ready to be read from or write to (Even connections that an nginx server receives are maintained by the nginx process in the form of file descriptors for the life time of a connection).

As we discussed before, nginx uses a non-blocking I/O with a single thread. Hence the single process must identify which connection file is ready to be read or written to. For this, the operating system has three different methods. They are mentioned below.

• Select Method
• Poll Method
• Epoll Method

Using both select and poll is not an efficient way to identify which file descriptor is ready. Because both of these method are identical in what they does (am not capable enough to explain you the difference between select and epoll, if you are a programmer i will recommend to go through the below link to learn more about select and poll.)

Read: Select and Poll

So nginx can use either select or poll method to identify which file(file descriptor) is ready to be read or written to. But both these methods are not efficient enough when there are too many number of connections. For example, if you want to serve say 10000 connections at a time. And only one of the them is ready to be read. To identify that one file descriptor to which is ready to be read, the process has to still scan through rest of the 9999 file descriptors (which is waste of resource).

Another method other than the select and Poll is called as epoll comes to rescue here. This is only available in Linux kernels which are later than 2.6. Epoll uses an efficient mechanism compared to poll and select.

If you have 10000 connections made to your server, and 8000 among them are idle connections. Using poll and select is not that efficient because epoll will only give importance to connections that are active. Please note the fact that all three select, poll and epoll basically does the same thing, but using epoll is less cpu intensive when you have to serve thousands of connections. 

You can modify your nginx configuration file to use epoll as shown below. If you are on centos this nginx configuration file will be located at /etc/nginx/nginx.conf. This is done inside an events block in the configuration file.

events {
    use epoll;
}

Now as discussed earlier, nginx works on a non-blocking single process model. Although we say single process, we can ask our nginx to start multiple worker process. Basically its like if you have two worker process, each one will handle all the requests. This setting is normally configured depending upon the number of CPU cores you have on your system. You can find the number of CPU cores on your system by using either of the below commands.

[root@www ~]# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                8

cat /proc/cpuinfo

Am quite sure that if you are on a production server, you will be having multiple cpu cores. As seen from the lscpu command output, i have 8 processors, so i will ask nginx to launch 8 worker process that will be serving requests.

worker_processes  8;

Also you can alternatively configure nginx to do the job of identifying the number of cores on your server and launch worker processes accordingly. This can be done by modifying the worker_processes to auto as shown below.

worker_processes  auto;

Another important factor that we can modify in nginx is the maximum number of files that can be opened by our worker processes. As discussed before, each and every connection is handled by creating files to identify them. Hence more the number of files that are allowed to be be created, workers can work with more connections.

There is a configuration value that can be modified in nginx to get this done. Its always good to set this value to a higher one if your server is a very high traffic one.

worker_rlimit_nofile 100000;

Even if i set worker_process in my nginx.conf to 8 or auto, it will end up launching 8 worker processes. All these 8 processes are ready to serve connections. But you can further optimize these worker process to make them server as many connections as possible.

We can define the maximum number of connections which can be simultaniously served by a single worker process. The default is 512 connections if you dont specify it. But you can modify this to accept more connections per worker. This can be done as shown below.

events {
    use epoll;
    worker_connections 1024;
}

We also need to configure our worker processes to accept multiple connections at one time. This can be done by modifying the events block as shown below.

events {
    use epoll;
    worker_connections 2048;
    multi_accept on
}

Our final configuration after making the above changes must look something like the below.

worker_processes auto;
worker_rlimit_nofile 100000;

events {
    worker_connections 1024;
    use epoll;
    multi_accept on;
}

So now my nginx web server is ready to serve around 8192 connections(8 worker process * 1024 connections ) at one time. 

Please note the fact that we have used events module in nginx to modify worker connections, use epoll method, and multi_accept. This is the reason why they are inside events block (events {}) in the configuration file. There is another module called http, which is the primary module that you will be using to define http related stuff.

We wil be now modifying and adding few parameters inside the http module to speed up our nginx a little bit more.

One of the major and required feature of any webserver is logging. Logging will give the server administrator an indepth detail of the kind of requests that are being served, and also it helps for troubleshooting. A web server log can also be used to make a web analytics because log files contain the complete details of the requests. These details include the source address of the request, user agent (the browser used by the client), the resource requested in the request (or the url accessed), destination address, referral url etc etc.

Related: Central Logging Server in Linux

Although logging is an important and required feature, it does a lot of I/O operation on the disk. Please dont forget the fact that each and every request is logged as soon as it hits the server and served.

This means for every request the web server has to process the logs and add the relevant details to the log file (which adds up to processing as well as I/O usage.). If the website that you host is getting too much traffic, then am sure your web server is having a realy bad time logging it.

Hence turning off the logs completely on the web server can save a lot of I/O as well as CPU power. This becomes very handy when you are having a high traffic website. In nginx you can completely turn of logging by the following option.

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    access_log off;
}

access_log off option shown in the above http block tells not to log requests that are being processed by the server.
 

When you access a website with a web browser, what really happens is that your browser establishes an HTTP connection with the target server. The real overhead is the beginning. Which means the main time consuming (although in seconds, it matters when performance is considered) thing is to establish the connection.

Let's take an example, where you are accessing a news related web site, and you are flipping through different url's on the website to have a quick overview of different news contents. Imagine a situation where your client browser has to make new http connections to the server for each URL you are accessing on the site. That will be a little slow for the site visitors. Because accessing each url has another overhead of establishing an HTTP connection first.

HTTP version 1.0 implemented a feature called as keep alive connections to solve this overhead of creating a connection for each request. However there is a timeout period for this. In short the web server's allow you to specify the timeout period for which to keep the connections alive, so that a connection made during that interval does not require to create a connection.

In short connections are kept alive till the keep alive timeout you specify

And believe me nginx is much better in handling large number of active connections, compared to any other web servers. It claims that it can handle upto 10000 connections alive by only utilizing 2.5 MB of memory. So if you are using nginx, go ahead and raise your keep alive value to something higher like 65.

keepalive_timeout  65;

If you are a Linux user, then am sure you must have heard about gzip compression. And you might already know the amount of compression level that can be achieved with gzip for compressing text content. You get compression rate in the range of 90's. Which is a very awesome compression level.

A major portion of a web page is always text content. The components that make a web page look good as well as enhance its functionalities are also text content. Although they are all code in different languages like html, CSS and even JavaScript (which the browser knows to interpret) at the end of the day they are text content. 

Sending all data as it is over the wire to clients is always a bit slow. So each and every web server supports compression mechanism to compress the data that are being sent out to the clients. Also all modern web browsers knows how to decompress it.

Enabling compression can be very helpful to people who are on slow networks accessing your website. Enabling compression is not a tough task. Similar to the previously shown options, compression and gzip option is enabled inside the http block section in nginx.conf file.

If you have done compression with gzip tool in linux, then you must be knowing that it takes a little bit of time to compress the content depending upon the size. And you might also know that if you increase the compression level, the end result does not make any large substancial difference, but it will take a little more time along with a little more CPU usage.

Similarly while configuring compression on our nginx web server, we need to take an extra care of what to compress, when to compress, and the level of compression. Also we need to have a mechanism where we can disable compression for older web browsers that does not support this feature.

So the first step is to enable compression. This can be done by adding the below parameter inside the http block.

gzip on;

The second step is to tell nginx, to only compress, when the size of the data being sent is above our specified limit. This can be done as shown below.

gzip_min_length 256;

Official nginx web site describes the above min length parameter in a nice way. Below is the description of the min length parameter from the official nginx http module section.

Responses shorter than this byte-length will not be compressed. Length is determined from the "Content-Length" header.

Now as we discussed earlier, too much compression does not make a substantial difference, but will only add up to CPU usage. Compression level is mentioned from a scale of 1 to 9. We do need compression, but not at a higher level. Because even the basic gzip compression does a pretty nice job. So lets keep the value of compression level somewhere between from 3 to 5 (Depending upon your wish)

gzip_comp_level 3;

Now we need to tell our nginx web server about the types of data that needs to be compressed. This is generally text content ranging from rss, css, html, javascript etc. This can be done by adding the following line.

gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

So our final http block section will look something like the one shown below.

http {

include       /etc/nginx/mime.types;
default_type  application/octet-stream;
access_log off;
keepalive_timeout  65;
gzip on;

gzip_min_length 256;
    
gzip_comp_level 3;

gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

}

Another aspect of nginx that can substantially improve performance is to enable caching for metadata. Please note the fact that this does not enable caching for the content, but its caching for metadata. This feature of nginx (which is again part of the http block section), is called as Open_file_cache. The first thing to do is to enable it.

open_file_cache max=10000 inactive=30s;

Please note the fact that there is no ON switch for this, but there is an off switch. Before we understand the above configuration parameter, let's see what are the contents that will be cached with this.

• Open File Descriptors
• File Descriptor modification time, their size etc.
• Error status like no permission, no file etc

The first parameter, max=10000 tells nginx web server to only cache this many number of entries. Old inactive entries are automatically flushed out. Please note the fact that the inactive switch tells the time after which a cache entry will be removed if inactive. 

Other open_file_cache options that needs to be configured are mentioned below.

1. Validity of a cache entry (open_file_cache_valid).
2. Minimum number of times the cache entry has to be accessed before the inactive number of seconds, so that it stays in cache (open_file_cache_min_uses)
3. Cache errors while searching a file (open_file_cache_errors)

So our open file cache configuration will look something like the below.

open_file_cache max=10000 inactive=30s;
open_file_cache_valid    60s;
open_file_cache_min_uses 2;
open_file_cache_errors   on;

If you have a very high traffic web site with nginx, setting the above open file cache parameters properly can really boost the performance.

Although we did see the keep alive stuff in nginx, there are yet few more parameters that is very helpful in specifing the number of connections to be kept alive from a client. Also we have few more options to close connections when the client does not respond.

The first one is to limit the total number of requests that can be served through a keep alive connection. The default value if not mentioned is 100 (which means a client can make 100 successfull request inside one keep alive connection.). Even that is too high for one client, if you want you can increase this value to something like 200 or something. This can be done as shown below.

keepalive_requests 200;

Keeping connections open for a client which is not responding is not a good idea, its always better to close that connection and free up memory associated with it. This can be done as shown below.

reset_timedout_connection on;

Another nice option that needs to enabled for faster tcp data transfer is sendfile. There is an excellent article from techrepublic, that explains the entire stuff in detail.

Read: What is sendfile() and how is it useful

This option can be enabled in nginx as shown below.

sendfile on;
tcp_nopush on;

tcp_nopush option will make nginx to send all header files in a single packet rather than seperate packets.

So our final nginx.conf file will look something like the below shown one.

worker_processes  auto;
worker_rlimit_nofile 100000;
events {
    use epoll;
    worker_connections 1024;
    multi_accept on    
}
http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    access_log off;
    keepalive_timeout  65;
    keepalive_requests 200;
    reset_timedout_connection on;
    sendfile on;
    tcp_nopush on;
    gzip on;
    gzip_min_length 256;
    gzip_comp_level 3;
    gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
    open_file_cache max=10000 inactive=30s;
    open_file_cache_valid    60s;
    open_file_cache_min_uses 2;
    open_file_cache_errors   on;
}

Once you have configured the required changes to improve the speed, i would suggest to do a web server performance test, with AB tool or any other tool to confirm its serving the purpose.

Read: How to do web server performance test

Please let me know if this article was helpful. There are more number of options in nginx, i will surely update this article from time to time, as i find or learn something new about this. Also please do not forget to point out any wrong or incorrect information if any is present in this article through comments (as this will help us as well as our readers)