CGroups

Introduction:

                Recently I received a call from my colleague to suggest a solution for a serious performance issue that he has been experiencing in his infrastructure environment right after deploying a new application.

Thinking for a solution for this problem, I have brought the details of all the native tools available in Linux to limit resource usage of process groups. Tools like “nice, renice, ulimit are in my mind, but the challenge was to have a fine grained control at process level.

“Cgroups” is a feature available from RHEL6 onwards not only to limit resource usage by process but also to monitor the same. The following chapters discuss about “cgroups- the most useful but least used feature “in Linux

Control Groups

                “Control Groups” well known in its shorter name as “cgroups” is a new kernel feature available from RHEL6 onwards.  Cgroups helps the system administrator to have fine grained control over the process in its system resource utilization. Cgroups allows us to allocate as well as deny processes from using system resources.  It also helps us in managing, prioritizing and monitoring system resources and thus to increase the overall efficiency of the system.

Though we have the command “ulimit” to control the resource utilization, the benefit of “cgroups” over “ulimit” is its hierarchical nature which helps us to manage altogether a process and its child process.  This feature helps us to avoid certain situations that may lead to “fork bomb”

Advantages of Cgroups

The biggest advantage of Cgroups is to impose limits on different workloads without impacting other workloads running on the same server.  In this era of virtualization, there is lots of usage for this feature. One of the real world example is   to put different priorities for different virtual machines running on a Linux server.

How “cgroups” Organized

                “Cgroups” has been organized hierarchically like processes in Linux.  In Linux all processes are child processes of a common process named “init” and all child process inherits certain environment from its parent.

                Same way “cgroups” also have child cgroups and it inherits certain attributes from its parent cgroups.  But the major difference for cgroups hierarchical model from the process model is that we can have different hierarchies of cgroups simultaneously on a system.

Setting up cgroups

                The method to set up the cgroups is different from distribution to distribution. The general method is to mount the virtual file system “cgroup “in to the existing file system hierarchy

 Ex:         #mount -t  cgroup  cgroup  /cgroup

                #mount -> will display the cgroup virtual file system as mounted under /cgroup

Here the “cgroup” virtual file system is the window to Linux kernel just like proc file system.  Also we will be using different parameters in /proc file system to tune the system, just like this cgroup also have attached with different controller or subsystem which we are used to control process from utilizing resources. We will be discussing about the different controller available in Linux later.

Here  “/cgroup”  is known as a hierarchy and it is the root cgroup.

 #lssubsys – will display the default subsystems/controller attached to this hierarchy.

cpuset, ns, cpu, cpuacct, memory, devices, freezer, net _cls, blkio, perf_event, net_prio  are the controller  available with RHEL. 

In RHEL, we have a service called “cgconfig”, which will mount the cgroup virtual file system as well as attach the same with different subsystem/controller according to the configuration done in /etc/cgconfig.conf.

Hierarchy-Subsystem-cgroup-Tasks- Relation ship

There are certain rules to follow when we are managing cgroups and the same are described below

 

Using Cgroups

There is two way to manage cgroups

1)      mount the hierarchies and  set cgroup  parameters manually using  shell commands

2)      Install libcgroup-rpm and use the tools provided by that rpm to create, delete, add tasks to cgroup …etc.

The libcgroup rpm provides the cgconfig service and it refers the /etc/cgconfig.conf file while it starts. The /etc/cgconfig.conf file contains two types of entries

1.       mount

2.       group

The mount entries create and mount hierarchies and attach subsystem with it. The group entry creates cgroups and set subsystem parameters.

The syntax of the cgconfig.conf file is shown below.  If you want to create a new hierarchy for the cpuset subsystem, edit the /etc/cgconfig.conf file and add the entry in the mount block as follows, 

                                mount  {

                                subsystem = hierarchy    ### Syntax of the /etc/cgconfig.conf file

}

mount    {

                                All the other entries …..

                                cpuset = /cgroup/test  ### This will  form a new hierarchy

}

This entry will create a new hierarchy for the subsystem cpuset after restarting the cgconfig service. The same can be achieved using shell commands as follows,

                #mkdir /cgroup/test

                #mount –t cgroup –o cpuset   test  /cgroup/test

                               

To create a new cgroup, add the entries as follows in /etc/cgconfig.conf and restart the cgconfig service.

                group <name> {

                                <controller  >  {

                                                Controller parameters = value

                                                …………

                                                …………

                                                                }

                                                }

So to create a cgroup named “http” under the memory hierarchy , add the entries as follows,

group http {

Memory {

Controller parameter  = value;

}

                                                                                }             

                #lscgroup -> will display the cgroups available in the system. You  can see the newly added cgoup http under the hierarchy memory.

               

Available Subsystems

The following section gives the summary of the different subsystems available and what is it used for. The details of the different parameters that can be used with each subsystem is available in the Reference section added at the end of this arcticle.

blkio:     This subsystem put limit on input/output access to and from block devices.       cpu:       Uses the scheduler to  provide cgroup tasks access to  the cpu       cpuacct: Generate automatic reports on CPU resources used by tasks       cpuset:  Assigns individual cpus and memory  group to  tasks in cgroup       memory:  Sets  limits on memory use by tasks       devices:  Allows or denies access to devices by tasks       freezer: Suspends or resumes tasks in a group       net_cls : Tags network packets with a class identifier       net_prio:  Use to  set  priority  dynamically on network traffic per network interfaces       ns:              namespace subsystem

Managing cgroups                     

Managing cgroups means how we can use this feature to set limit or deny process when they are trying to access resources. I would like to introduce the tools/commands available with libcgroup to manage the cgroups. I am not going to give the complete usage or syntax of each command, for that you can refer the topics mentioned in the Reference section added in the end of this article.

Adding hierarchy                                 We have already learned how to add new hierarchies and how to attach subsystem to that hierarchy. ü  Using cgconfig service ü  Manually mount a cgroup virtual file system and attach subsystem to that hierarchy Deleting hierarchy ü  Un mount the hierarchy after deleting all the cgroups in that hierarchy Attach and detach subsystem to a hierarchy ü  Remounting hierarchy  by adding or omitting the subsystem to  be attached or detached a.        mount –t cgroup –o remount, subsystem1,subsystem2  cpu  /cgroup/cpu Creating control group ü  By adding the appropriate entries in /etc/cgconfig.conf file ü  By using “cgcreate”  command Removing control group ü  By removing the entries from /etc/cgconfig.con file ü  By using the “cgdelete” command To set or get parameter value in to a cgroup ü  By adding appropriate entries  in  /etc/cgconfig.conf file ü  By using “cgset” command ü  By  echoing values  as follows ü  “echo  “value” > /cgroup/hierarchy/”cgroup name”/parameter” ü  “cgget”  to  get the value for particular parameter Moving process to a particular cgroup ü  By using “cgclassify” command ü  By echoing the “pid” of the process to  be moved to the tasks file ü  echo “pid” > /cgroup/hierarchy/”cgroup name”/tasks ü  By using “cgred” service ü  Edit /etc/cgrules.conf  - cgred configuration file and add appropriate entries ü  Restart the “cgred” service to bring the changes in effect. Starting a process in a control group ü  By using “cgexec” command ü  By moving  the shell  process in to the cgroup and start process from that shell ü  echo “$$” > /cgroup/hierarchy/”cgroup name” /tasks         Starting a service in a cgroup ü  Edit the /etc/sysconfig/service name file and add the entry as follows and restart the service o    CGROUP_DAEMON = subsystem:cgroup ü  To start httpd daemon in  a control group named “test” under memory hierarchy, o    Add the entries in /etc/sysconfig/httpd o    CGROUP_DAEMON=memory:/test cgclear                                                                 The command “cgclear” is used to clear all the control groups defined in the hierarchies

 

 

 

 

 

o   CGROUP_DAEMON=subsystem:/cgroup             

Test Case

                We learned how to configure and use the feature “cgroups” in RHEL servers to limit resource usage by processes. In the following topic we are going to see how cgroups can be implemented to prioritizing Database I/O

In this environment, two databases are running on two individual virtual machines hosted on KVM hypervisor. Among the two DB one is a high priority DB and other one is low priority DB. When both the DB are running simultaneously the I/O throughput is decreased to accommodate requests from both the servers equally.

To prioritize high priority DB, it can be assigned to a cgroup with high number of reserved I/O operations.

1.       Create a hierarchy and attach “blkio”  subsystem to  that a.       mkdir  /cgroup/db b.      mount –t cgroup –o blkio  db  /cgroup/db 2.       Create high and low priority cgroups a.       mkdir /cgroup/db/low-prio b.      mkdir /cgroup/high-prio 3.       Get the pids of the process that represent both the VMs. For this example , the VM names are A-high & B-low a.       ps  –eLf  | grep  qemu |  grep  A-high | awk ‘{print $4}’  | while read  pid;  do echo  $pid >> /cgroup/blkio/high-prio/tasks ; done b.      ps  –eLf  | grep  qemu |  grep  B-low | awk ‘{print $4}’  | while read  pid;  do echo  $pid >> /cgroup/blkio/low-prio/tasks; done 4.       Set the ratio of 10:1 for the high-prio and low-prio  cgroups . a.       echo “1000” > /cgroup/blkio/high-prio/blkio.weight b.      echo “100” > /cgroup/blkio/low-prio/blkio.weight 5.       Process in the respective cgroups will immediately start to use the resources assigned to them. In this example, the high-prio cgroup permits, A-high database server to use 90% of the I/O operations whereas the low-prio cgroup will allow the B-low DB server to use only 10 % of the I/O operations.

How to achieve this

                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               

Any Overheads

                Experienced Administrator:

 Though there are command line tools available in Linux to configure “Cgroups” easily, only experienced Linux administrators can only configure this.

Conclusion

                In summary, we were learning “Cgroups” the new feature available in Linux. The feature cgroups helps us to solve the difficulty which we faced earlier to put control on a group of processes together.

References

1.       Red Hat Enterprise Linux 6- Resource Management Guide

2.       /usr/share/doc/kernel-doc/cgroups

3.       Articles related to cgroups from Google.