What is a Jumbo Frame in Ethernet
Ethernet is one of the core technology that enables most of our communications and data transfer that we do over the internet possible. Its an inexpensive technology for high performance networking.
When your application (no matter what..can be a web browser, a download client etc etc) initiates a connection to a remote server, the application’s job is to transfer the data/request it wants to send to the network protocol stack.
The job of the protocol stack is to break that data down to smaller chunks and wrap it with required information, to send it through the wire. This information wrapped around the data chunks enables the data to be received by the correct receiver, and reassemble the data chunk in the correct order.
Once the data is passed by the application to the operating system’s networking stack, the networking stack will pass each chunk of data to the Ethernet hardware. This Ethernet hardware is most of the times a Network Interface Card attached to the sender’s machine.
The headers added by the hardware to each data chunk is called as Ethernet headers. These Ethernet headers helps the packet to reach the correct destination.
Say for example, if you are transmitting data to a remote machine, your NIC card will add its own headers to each data chunk and sent it to the gateway. The gateway (no matter whether its a router or a Linux machine or anything), will strip of the headers it got, and read the proper destination in the IP packet, it will then add its own Ethernet headers, and then transmit the Ethernet packet to the next hop(no matter whether its a Linux system, or a switch or a router or anything)
Each hop(a machine/device that comes in between the sender and the receiver) will strip off the headers it gets, to read the proper destination, and adds its own headers, before forwarding. I have a diagram of an Ethernet Frame, which i used for another article regarding VLAN's (shown below).
Read: What is VLAN, and how to configure it in Linux?
You can safely ignore the VLAN INFO part, in the figure above(that is only used for VLAN).
Each Ethernet frame starts with Destination address, Source Address, And a Frame Check sequence at the end of the frame (referred to as FCS)
As you all know IEEE is the group who defines protocol standards. The initial Ethernet standard called as IEEE 802.3, had an Ethernet frame size ranging from 64 to 1518 bytes.
Its 1518 because of the 18 byte Ethernet headers(like source mac, destination mac, etc). The main payload (the data given by other layers in the protocol stack) for a standard Ethernet frame will range from 46 to 1500 bytes. This is called as Maximum Transmission Unit(MTU). If the payload is more than the maximum, the network can support, then that packet needs to be broken down to smaller fragments by layer 3 (Network Layer), so that the MTU can accommodate it.
But as requirement and use cases increased, modifications to the original Ethernet standard became a necessity. Some of those changes to Ethernet standards, which caused to increase the frame size are mentioned below
- VLAN information in an Ethernet header increased the frame size from 1518 to 1522. This is because VLAN headers required an additional 4 bytes in the header(as shown in the figure shown previously). This standard of Ethernet, that supports VLAN information is named as IEEE 802.1Q
- Another specification called 802.1ad adds an additional 8 bytes to the original frame size of 1518. Hence it becomes 1526 bytes. 802.1ad is a specification for including multiple vlan information in a single frame (Read: Wikipedia article on 802.1ad)
- Now as the frame size due to addition of multiple new headers went on increasing, IEEE decided to increase the overall frame size to 2000 bytes. Read:IEEE 802.3as specification article
- Then FCOE (Fiber Channel Over Ethernet) required higher transfer rates, hence they came up with an MTU size of 2500 bytes.
As i said before, MTU is nothing but the maxmimum amount of data, an IP packet can contain, that can be sent over a frame (supported by your network) without fragmenting. If your IP packet (data given by network layer) is above the maximum frame size supported by your network, then it needs to be fragmented.
Recently i was reading a document from CISCO, about Jumbo Frames support on their Catalyst series(Read: http://www.cisco.com/image/gif/paws/29805/175.pdf). And According to that document...
Ethernet frames upto the size of 1600 bytes are referred to as Baby giant frames, and frames that range between anything above 1600 till 9216 bytes can be called as Jumbo Frames.
The reason i discussed about that cisco article is because different vendors have different views regarding what jumbo frame is.. And currently jumbo frames are not defined in any IEEE specifications for Ethernet(i believe something is on their draft, but not yet under any specification).
Due to this, different vendors might support different frame sizes(And jumbo frames to different vendors will be different sizes). But yeah we can have a middle ground and think of jumbo frames as any frames between 1600 to 9216 bytes, as Cisco article says.
The advantages of using Jumbo Frames
The first and foremost thing that comes to my mind when discussing about jumbo frames is greater efficiency in data transfer.
- 1. NFS(Network File System). As this protocol is mainly used to share files over the network, larger frames can significantly improve its performance.
- ISCSI SAN or FCOE: These protocol's deal with storage over Ethernet and requires higher transfer rates over the wire. Having jumbo frame support for them can improve their performance.
Let's take an example to understand how jumbo frames can improve network performance significantly. Let's say i want to send a file of size 6Kb (6144 bytes) to one of my destination server. The standard MTU size is 1500 Bytes (excluding the frame headers).
Now that 6kb of data needs to be sliced to smaller chunks of 1500 bytes (as my MTU is 1500 bytes). Which means i need to send around 4 to 5 frames to my destination server, to complete the transfer.
However if my MTU size is around 9000(if my network card and network devices support jumbo frames), i can send that 6Kb file in a single frame to my destination. See the difference? So without the jumbo frame support, and a very low latency network, we can say we are not fully utilizing our network capacity for our transfer. So without jumbo frame support the sending machine has to fragment the data to smaller chunks, so that it can fit inside the supported MTU size.
Do not forget the fact that the receiving machine, also has to reassemble those smaller chunks of 1500 bytes to form data i send.
So If you are using jumbo frames, the sending machine and the receiving machine has to go through less number of CPU cycles, as it only needs to send one frame without fragmenting, to the destination. Less CPU usage means, higher throughput. So as a foot note we can say the below.
One single jumbo frame of 9000 bytes is capable of replacing 6 standard Ethernet frames of 1500 bytes. And is also capable of saving CPU to an extent of 50 percent.
Apart from the above mentioned advantages of less CPU usage, and higher transfer rates because of single frame sent over the wire instead of 6, jumbo frame can in fact reduce the number of bytes that you sent over the wire. To understand this claim, let's take our 6kb file example once again. So if you want to send that 6kb file over the standard Ethernet MTU size, then layer 3 has to fragment it to 4 different chunks. And all those different 4 chunks of ip packet will have their own ip headers inside. Which means we are sending the same TCP/IP header 4 times, to send that single 6kb file to our destination. In case of a 9000 byte MTU, we are only sending those TCP/IP headers one single time. Hence we are sending less bytes over the wire.
The real data that any operating system sends over the wire in a single frame is most of the times 1460 bytes. This is because 20 bytes is used for IP headers, and another 20 bytes is used for TCP headers. Hence the IP data payload in a single frame is around 1460 bytes (1500 - 20 bytes - 20 bytes)
Ideal Use cases of Jumbo Frames
Jumbo frames should not be used, just because it can improve data transfer rates. Sometimes it can have weird effects on your network communication. The initial MTU size of 1500 was standardized when our requirement was not that huge as of today. At that time it was all 10 Megabit or a maximum of 100 Megabit Ethernet.
But later came Gigabit Ethernet, and now we have 10 gigabit Ethernet on most of our servers. But standards group (IEEE) needed backward compatibility, due to which the standard MTU size is kept 1500 bytes.
You should configure Jumbo frames only if you have the following basic requirements met.
- You have a full Gigabit Network
- All network equipment that comes on the way (from source to destination) supports jumbo frames
- All your server hosts are installed with a NIC card and its driver, that has jumbo frame support.
TCP can manage to work in a varying MTU size network, because of the initial TCP 3 way handshake. The initial 3 way handshake negotiates the MSS (Maximum Segment Size).
maximum segment size is nothing but the maximum amount of data a TCP/IP packet can carry (this does not include tcp/ip headers). This is negotiated when a tcp connection is established between a source and destination. The 1460 bytes of real data that we send over each frame (excluding 20 bytes tcp & 20 bytes ip headers) is the MSS.
As UDP does not have a initial handshake, UDP can exhibit weird issues, while on a varying MTU size network. Running multiple hosts on a single network with different MTU sizes is not recommended.
So if you want your network to support 9000 byte MTU size, all switches and hosts on the way should be configured to accept atleast 9000 bytes MTU.
How to configure MTU 9000 or Jumbo frames in Linux
Before configuring MTU of 9000 on your linux system, double check whether you have it configured already. See the output below, you should see the MTU value from ifconfig command output.
eth0 Link encap:Ethernet HWaddr 0A:65:38:CA:63:1E inet addr:10.1.130.195 Bcast:10.1.130.255 Mask:255.255.255.128 inet6 addr: fe80::865:30ff:feca:631e/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:9001 Metric:1 RX packets:3501576178 errors:0 dropped:0 overruns:0 frame:0 TX packets:3621233432 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:4833044650833 (4.3 TiB) TX bytes:4930864397824 (4.4 TiB) Interrupt:44
If you do not have value 9000 in your ifconfig output, then you can configure it using the below command.
# ifconfig eth0 mtu 9000
Or you can alteratively edit the interface configuration file and add the MTU size value over there to make it permanent, and restart network service. In red-hat based system's edit the file /etc/sysconfig/network-script/ifcfg-eth0 with the value(append) MTU=9000. In ubuntu/debian based system's edit the file /etc/network/interfaces and append MTU 9000 at the end.
How to check whether Jumbo Frames are working or not?
You can check whether your configured jumbo frames are working or not, with a simple ping command. PING command has an option to specify the packet size to be sent. Try sending 8000 bytes over ping and see if you are getting a reply from your target host, as shown below.
[ec2-user@myserver ~]$ ping -M do -s 8000 10.1.130.195 PING 10.1.130.195 (10.1.130.195) 8000(8028) bytes of data. 8008 bytes from 10.1.130.195: icmp_seq=1 ttl=64 time=0.832 ms 8008 bytes from 10.1.130.195: icmp_seq=2 ttl=64 time=0.805 ms 8008 bytes from 10.1.130.195: icmp_seq=3 ttl=64 time=0.816 ms
If your target server does not support jumbo frames, then you should get a reply as shown below.
[ec2-user@myserver ~]$ ping -M do -s 8000 10.2.0.71 PING 10.2.0.71 (10.2.0.71) 8000(8028) bytes of data. From 10.1.130.25 icmp_seq=1 Frag needed and DF set (mtu = 1500) From 10.1.130.25 icmp_seq=1 Frag needed and DF set (mtu = 1500) From 10.1.130.25 icmp_seq=1 Frag needed and DF set (mtu = 1500) From 10.1.130.25 icmp_seq=1 Frag needed and DF set (mtu = 1500) From 10.1.130.25 icmp_seq=1 Frag needed and DF set (mtu = 1500) From 10.1.130.25 icmp_seq=1 Frag needed and DF set (mtu = 1500) From 10.1.130.25 icmp_seq=1 Frag needed and DF set (mtu = 1500)
The output clearly says that fragmentation is needed.
The cons or Disadvantages of Jumbo Frames
- Larger frames over the wire tend to increase delay hence its not suitable for applications that require very less delay. Like VOiP or chat applications etc.
- In order to effectively use jumbo frames, you need to make sure that all the hopes that comes in between the source and destination supports your required MTU size. Basically use it only on a known network and devices.
- At times, switches that do not have support for jumbo frames do drop packets.
So as a footnote, jumbo frames do increase the overall transfer rate and network performance, but only enable it in a local network where it is really required.
Today's Most Popular
Most Commented
Sarath Pillai
Satish Tiwary
Add new comment