What is IPSEC and how IPSEC does the job of securing data communication

From the past couple of days i was getting my hands dirty with a technology called as IPSEC, which is not at all a new technology, but yeah it was new for me to configure it in the correct manner. I have seen people in the field of System Administration doing blind troubleshooting by simply changing some configuration, to get the problem fixed somehow.

Although such a kind of approach does work most of the times, but is not at all a good way to go forward. Because you are surely going to stuck again with some issue down the line, which requires proper understanding of the technology to resolve it. I can’t even blame system administrator’s for taking  such an approach, because most of the times, the configuration tasks are assigned with a deadline attached to it, which makes the duration worked on it more important, rather than the configuration you did.

So i was working on IPSEC, due to which i did read some of the official protocol documentation as well as white papers from the major players in the field. In this tutorial, similar to other tutorials in slashroot, i will try my level best to put things in a simple manner, so that even a novice can grasp it without much effort. Your suggestions and recommendations are always welcome through comments.

What is IPSEC?

IPSEC stands for Internet Protocol Security. To understand this a little bit of networking knowledge is required. Our TCP/IP model of networking works in layers. This means there are different layers that do different modifications to the data.

Each and every layer in the entire architecture plays a major role in making the networking communication work.  There is a specific assigned task to each layer that does its job and hands over the result to the layer beneath. I will not be explaining networking layers in depth here.

Let’s take an example to understand this layered architecture.

Layer 5- Application Layer:

This layer is where our day to day applications exist. Like for example, when an FTP client request s data from an FTP server, you can say that the client FTP process is in direct contact with the server FTP process because of the application layer in both server and client.

Layer 4- Transport Layer:

Transport layer is the layer which establishes a reliable connection between both the server and the client. Protocols like TCP and UDP exist in this layer of networking. So lets take the same example of FTP connection between a server and a client. The application layer encapsulates data that needs to be send and hands over to Transport layer.

Now transport layer will add its own headers of TCP or UDP and other required flags. Now the resulting thing is handed over to the layer beneath.

Layer 3: Network Layer

When data grams from the transport layer with all TCP details arrives network layer it adds its own headers to the data gram. This layer is very much important in understanding IPSEC technology. We will be coming back to this in some time.

So when a data gram (containing both the application layer content and transport layer content) arrives at network layer, it adds IP information to it. The information added is mentioned below.

• Host addressing (IP address of the host sending data and also IP address of the machine to which the data is being sent). In short To and FROM address is added in the IP address form in the networking layer. Data in this layer is called as a packet.
• Length of the packet, IP protocol version and number of packets that will be send to complete a data.
• A checksum to detect modification and error. And also a value called as TTL. TTL in networking is the unit used to specify when the packet should be dropped if it has not reached the required destination (basically it’s the number of routers and devices that comes in between).

Do not forget the fact that the data given by the upper layers is encapsulated (kept inside as a payload) inside the ip packet. IPSEC is security mechanism devised to secure insecure communication between two parties or two networks.

You might argue that there are already security protocols available to do the task of securing data with the help of encryption, like SSH, SSL etc. You are in fact correct when you say that. But IPSec is entirely different from those types of security.

Related: How does SSL Work

Related: How does SSH work

If you see in the above explanation of the three networking layers (application, transport, and network) i have not mentioned any mechanism for encryption inside them. This means that SSH and other protocols like SSL encrypts and provides security in the application layer (or sometimes even before the application layer, which means the program itself). Like for example when we send an encrypted email with the help of GPG, the entire encryption and decryption is done by applications that are not part of the network layer architecture.

Related: Understanding encryption and decryption with GPG

This makes application programmers to either program their application to support some form of encryption by themselves, or will need to make them compatible with other forms of security like SSL (but the application requires that capability.)

Imagine a technology which will encrypt the entire content you are sending over the network to your desired host, no matter which application you are using to send it. Such a technology requires to be implemented in the layered architecture of networking, as you want to encrypt the complete communication between two parties over wire.

IPSEC is such an implementation, which encrypts the entire ip packet itself. Do not forget the fact that any application layer data (any data send by any application) is already encapsulated inside the IP packet, along with the data and headers added by transport layer. So if an IP packet is encrypted, everything is encrypted. Key features of IPSEC are mentioned below.

• IPSEC provides confidentiality in communication with the help of encryption
• IPSEC provides integrity in communication by checking data modification, which is done through message authentication codes (MAC)
• IPSEC provides authentication of both the parties in the communication
• IPSEC protects the ip packets from analysis, which means you can’t say who is communication with whom by simply looking at the packet(people do get confused when this is said, i will come back to this point later)

The most common use of IPSEC is to provide secure VPN services (Virtual Private Networks). Which means you can use this technology to connect to a remote network very securely through the internet itself. I will show you how this is done.

So in the above shown figure, there are two different subnets one is 172.16.200.0 and the other is 192.168.0.0. Imagine that both of these networks are part of the same company, but at remote locations. A better way is to consider it as two branch offices. Now IPSEC can be used to create a secure tunnel between these two networks through the internet.

Internet is an insecure medium for critical communications. So to secure the communication an IPSEC tunnel is made between two machines having public ip addresses, which will act as a gateway to reach the other network. This terminology is called as Gateway-to-Gateway VPN. Whenever security comes we need to discuss cryptography, IPSEC is capable of using symmetric (where one single key is used for both encryption and decryption) and asymmetric cryptography to secure the communication (where different keys are used for encrypting and decrypting.).

The main problem with asymmetric cryptography is that it requires more processing power. And due to performance reasons, most of the time, asymmetric cryptography is only used to share secret key used to do symmetric encryption.

Fundamentals of IPSec

Let’s understand three primary components of IPSEC implementation. Please note the fact that i will not be including any configuration part in this tutorial, will make a separate post for configuring IPSEC VPN in different modes in Linux.

The main components of IPSec are mentioned below.

• Authentication Header or sometimes called as AH
• Encapsulating Security Payload or ESP
• Internet Key Exchange Protocols or IKE

AH (Authentication Header)

Authentication header is used for authentication, and also for data integrity protection. Authentication Header cannot encrypt the data inside the packet. However ESP could do both authentication as well as encryption together, due to which some of the later implementation of IPSec does not use AH at all. But still a lot number of IPSec packages are using AH headers only. Let’s understand the two main modes of IPSec operation.

1. Transport Mode
2. Tunnel Mode

There are many blogs out there in the internet that discuss the difference between these two modes of operation. Let’s understand their major difference through diagrams.

Let’s begin with the Tunnel mode. As the name suggests tunnel mode must be used when you require an ipsec gateway to gateway connection, as shown previously in the diagram. Let’s look at what in fact happens when you use tunnel mode. In tunnel mode ipsec encrypts the entire ip packet along with its headers and then generates a new header to stick on top of the encrypted ip packet.

Understand the encryption and header part very clearly, because this can cause confusion. When i was searching over the internet, i found it very difficult to understand how ipsec works in the tunnel mode. The main confusing part is that many documents and articles out there say that in tunnel mode an entire ip packet is encrypted(along with its headers). Now imagine if an entire ip packet with its headers is encrypted, then how can you send that packet over the internet (how will the routers in between know where to send the packet and from where it has come, because the headers are also encrypted.)

They never say that IPSec creates a brand new header and attaches it in front of the completely encrypted ip packet. Let’s see that. Also remember the fact that the encryption part is not done by AH. AH simply creates a new ip header on top of the previous one, as shown in the figure below.

Now let's understand what is the purpose of this new header after encrypting the already present header along with the data. This tunnel mode of operation is mostly used when you are configuring a gateway to gateway ipsec VPN.  Let's understand the purpose behind this by taking an example.

For this example, we will take the same diagram shown previously where two internal networks are connected together with the help of two gateways with public ips, through ipsec VPN. Now if a client say for example 192.168.0.2 wants to communicate with 172.16.200.3, the packet along with the communication data from 192.168.0.2 first reaches the gateway. The gateway has ipsec in tunnel mode configured to connect with the gateway on the other network(which also has a public IP). One receiving the packet from 192.168.0.2, the gateway, with the help of IPSEC first encrypts both the header and the data and adds another new IP header, which contains its own ip information instead of 192.168.0.2.

Keep in mind that the original entire ip packet originated from 192.168.0.2 is now completely encrypted, along with its headers, and a new header containing the ip information of the gateway and the target gateway is added. On receiving this ip packet the gateway on the other side will remove the first header and decrypts the original packet along with its headers, and finally sends the decrypted packet to 172.16.200.3.

The main purpose behind doing this is that, even if any eavesdropper captures the packet in between (in the internet), he will only know that the communication is happening between the two gateways and the data is encrypted. He will never come to know that the communication is in fact happening between 192.168.0.2 and 172.16.200.3(because the original packet is encrypted inside a new ip header containing the gateway ip information). This will be true for the entire communication between all the hosts in the two networks.

IPSec Transport Mode 

IPSec transport mode is very much similar to the tunnel mode with two major differences.

• It does not change the original IP header
• Also it does not create a new IP header

As ip header modification is not done, transport mode is most commonly used in host-to-host communication. This means where you can communicate with a remote host from your computer itself, without any IPSec gateway in between.

If you remember, i have said in the beginning of this AH header section that it provides integrity protection for the entire packet(along with the data and headers.)

Let's see how this integrity protection is done. Integrity protection is done with the help of message authentication codes.

Related: Integrity Protection With the help of Message Authentication Codes

IPSec works on a technology called as keyed hash algorithm for MAC. A simple hash algorithm creates a Message authentication hash, only based upon the data. But a keyed hashing algorithm generates a MAC code, based on both the data as well as the shared key(which is used for encrypting as well as decrypting ipsec data)

If you remember, have previously told that AH header will completely safeguard the integrity of the entire IP header, which is not at all true. Let's see why its not true. An IP header consists of some fields that are very much dynamic in nature. This means the values on some header fields at the origin will be very much different from the final value that the destination gets.

These dynamic values include the below fields.

• TTL value
• Checksum

TTL value is a very important field in the ip header that helps in successful communication between two parties in the internet. I will not be explaining TTL in ip headers here, however i did explain a little bit of what TTL actually does in the below post.

Read: Find out remote operating system with the help of TTL values

TTL value is very much dynamic in nature, while a packet travels through the internet, till it reaches the destination(each and every device that comes in between goes on altering the TTL value). MAC codes are very much unique to the data you process. Hence if you process the MAC hash value of an IP header in the beginning before being transmitted, the MAC value calculated at the destination will be completely different due to the legitimate changes in the TTL filed.

Due to this reason, ipsec calculates its MAC codes after excluding certain IP fields which are dynamic in nature. Hence it does not provide complete integrity protection of the headers.

This is the main reason why it becomes complex to configure IPSec behind NAT(because in NAT the ip headers containing the source and destination address changes before leaving the NAT box). Although its not an impossible task, it requires a little bit tweaking in the operating system level. We will be discussing IPSec behind NAT in the next part of this tutorial, when we discuss the configuration of IPSec in Linux. Some key points to keep in mind about the AH header before we conclude the AH section are mentioned below.

• Authentication header provides integrity protection for both data and headers(of course it excludes some dynamic header fields)
• A little bit of complexity is involved when AH header is used with NAT
• AH headers are a little bit outdated when compared to the new standards of IPSec implementation.
• Latest implementations of IPSec uses something called as Encapsulating Security Payload (ESP), which provides both authentication as well as data integrity protection.

Encapsulating Security Payload or ESP

ESP is the second major component of IPSec. If you remember i have told previously that encryption of the data is not done by AH header, but is done by ESP.

I must say that during the initial versions of ipsec, ESP only provided the encryption part, and authentication as well as integrity protection was entirely provided by AH header.

But later advancements in ESP made it capable of doing both the authentication, integrity checks, as well as encryption.

As ESP is capable of single handedly doing both encryption as well as integrity protection along with authentication, it also works on the similar two modes as AH header.

1. Tunnel Mode
2. Transport Mode

Similar to the AH headers ESP in tunnel mode adds a new ip header, after encrypting the original ip header and the payload. And in transport mode, ESP does not create a new ip header but simply encrypts the original data payload.

In both the modes, ESP adds a ESP header field to the IP packet.

Now let's discuss the encryption part. ESP uses a symmetric key encryption where the same key is used for both the encryption as well as decryption. ESP divides data to smaller chunks and encrypts it with AES(Advanced Encryption Standard) algorithm.

Internet Key Exchange Protocol

IKE is the main part that secure the entire communication. IKE is used to discuss the security algorithms, that will be used for securing the communication.

There are different types of IKE that are combinely used in IPSec. Lets discuss them to get a clear idea of how the whole security through encryption works

Main Mode

In the main mode, ipsec makes a secure channel by exchanging three pairs of messages. The first pair of messages consists of the following things.

• Encryption algorithms that will be used in the secure session. For example, DES, RC5, Blowfish and AES are some of the commonly used encryption algorithms.
• Message authentication code algorithm to be used. For example HMAC-MD5 and HMAC-SHA1
• Authentication Method that will be used in this secure session. For example, secret key authentication, Public key authentication, Digital Signatures etc.
• Diffie- Hellman to share a shared secret to be used for encrypting the data throughout the session.

The second pair of messages are used to complete the Diffie Hellman key exchange.

Agressive Mode

Agressive mode is always used to establish the same IKE with lesser number of messages compared to Main mode. It establishes the same thing with only three messages instead of the three pair used in Main mode.

Now finally before concluding this article, let's go through the steps involved in establishing a secure IPSec connection in tunnel mode, in a gateway to gateway structure(similar to the diagram shown in the article).

We will take the same example of the previously shown diagram where two networks 192.168.0.0/24 and 172.16.200.0/24 communicates with each other through a secure ipsec tunnel with the help of two interconnecting gateway.

192.168.0.2 wants to communicate with 172.16.200.3

Step 1: 192.168.0.2 creates a packet with the destination address of 172.16.200.3. This packet is not an ipsec packet

Step 2: The network packet reaches the gateway of 192.168.0.0/24

Step 3: The gateway of 192.168.0.0/24 will do a Network Address Translation and will change the source IP address to its own public IP address

Step 4: On looking at the packet, the gateway at the 192.168.0.0/24 comes to know that this packets need to be protected by encryption and integrity protection.

Step 5: Gateway at 192.168.0.0/24 network will start to negotiate encryption and MAC algorithms and other secret keys with the target gateway. This negotiation is done by either the aggressive or the main mode.

Step 6: Now the packet is encrypted and a new ip header is attached as its the tunnel mode, and is send to the gateway at 172.16.200.0/24 network.

Step 7: On recieving this packet the gateway at 172.16.200.0/24 network will look at the headers to determine the association, and will remove the additional header and decrypt the remaining header and payload.

Step 8:  Finally the unencrypted packet is send to the destination of 172.16.200.3 by the gateway at 172.16.200.0/24 network.

Hope the above article was helpful in understanding and getting a basic idea regarding the working of IPSec. Keep the fact in mind that the most complex part in any security protocol is the key exchange, and encryption. Understanding it in detail is beyond the scope of this article.

I will be publishing another article in some time, which will contain the entire configuration of IPSec in tunnel and transport mode, under Linux. As well as methods to overcome where ipsec headers are blocked.