[VM] The Practice of VMware vSphere 4 PerformanceTuning

rfcvg3

ABSTRACT
　　As we all know, VMware vSphere has become the most popularvirtualization infrastructure in the world. VMware vSphere virtualizes andaggregates the underlying physical hardware resources across multiple systemsand provides pools of virtual resources for thedatacenter. The current VMware vSphere4 includes two critical subsystems: ESX and vCenter Server. The article mainly discussesVMware vSphere 4 practice for performance tuning.
1.   Introduction
　　As a cloudvirtualization infrastructure, VMware vSphere manages large collections ofinfrastructures (such as CPUs, storages, and networking ) as a seamless anddynamic operating environment, and also manages the complexity of a datacenter.The following componentlayers make up VMware vSphere.
　　
　　Figure 1 Component layers of VMware vSphere.


　　VMware vSphere includes the following main components:
　　·       ESX and ESXi: A virtualization-customized OS running on physical servers/hosts that abstracts processors,memories, storages, and resources into multiple virtual machines.
　　·       vCenterServer: A central point for configuring, provisioning, and managing virtualized IT environments.
　　·       vSphere Client:An interface that allows users to connect remotely to vCenter Server or ESX/ESXi from any Windows PC. This component is the main configuration tool for the vSphere infrastructure.
　　·       vSphere Web Access: A Web interface that enables you to manage virtual machines and access remote consoles. The VMWare Lab Manger product is designed by vSphere Web Access.
　　·       VirtualMachine File System(VMFS): A high performance cluster file system for ESX/ESXi virtual machines.
　　·       VMotion and Storage VMotion: VMware VMotion enables the live migration of running virtual machines from one physical server to another with zero down time,continuous service availability, and complete transaction integrity. StorageVMotion enables the migration of virtual machine files from one datastore to another without service interruption.
　　[E1] SinceVMware vCenter Server is the main vSphere management component, we also need tointroduce the structure of vCenter Server in general. The vCenterServermanages ESX/ESXi servers on which virtual machines (VMs) are running. ThevCenter Server uses a database to store and organize inventory data. Clientsare connected to vCenter Server to manage and view the inventory. vCenter Server, the database,and theclients can either run on physical machines or inside virtual machines.Our best practice is that the vCenter products mentioned above run inside virtualmachines because vCenter itself provides VM Backup/Disaster Recovery/High Availability/FaultTolerance advancefeatures.
　　Figure 2: High level architecture of VMwarevCenter


2.   Hardware consideration with VMware vSphere
　　Firstly, the hardware needs to be validated before we install ESX or vCenter Server on it.
　　All hardware in the system is on the hardware compatibility list for the VMware ESX version
　　you will be running. We also need to make sure that your hardware meets the minimum configuration supported by  the VMware ESX version you will berunning.
　　
　　CPU considerations: Many recent processors from both Intel and AMD include hardware features to assist virtualization. You can enable the hardware-assisted virtualization features on BIOS. Dell PowerEdge series servers adopt the hardware-assistedvirtualization CPU.
　　
　　Storage Considerations: Storage performance issues are most often the result of configuration issues with underlying storage devices and are not specific to vSphere. It depends on workload, hardware, vendor, RAID level,cache size, stripesize, and so on. According to my experience, many VM workloads are very sensitive to the latency of I/O operations between storage and ESX.
　　
　　Network Considerations: The server-class/physical network interface cards (NICs) should be considered for the best performance. The network infrastructure between the source and destination NICs couldn’t be bottlenecks such as the same networkspeed in all cables and switches. VMware vSphere adopts virtual switches (vSwitch) to configure virtual networks. You could see VMware Virtual Networking Concepts for more information.
　　
3.   ESX and Virtual Machines Consideration
　　The general consideration is based on the proper configuration of ESX and Virtual Machine physicsand virtual hardware. The following two points are very important for ESX andVirtual Machine.
　　·       Allocate only as much virtual hardwareas required for each virtual machine.
　　·       Disconnect or disable unused orunnecessary physical/virtual hardware devices.
　　Same as the hardwaresection above, ESX and Virtual Machine best practice relates to CPU, Memory, Storageand Network.
　　
　　ESX CPU considerations
　　When configuring virtualmachines, the total CPU resources needed by the virtual machines running on thesystem should not exceed the CPU capacity of the host. We suggest using as fewervirtual CPUs (vCPUs) as possible when creating VM. But we could monitorall ofthe behaviors on every VM users. For the best practice, we recommend that you limitthe CPU resources for them in order to prevent the host CPU capacity from beingoverloaded.
　　For the VM vSphereAdministration, it is a good idea to periodically monitor the CPU usage of thehost. This can be done through the vSphere Client or by usingesxtop orresextop.
　　Hyper-threading technology (called symmetric multithreading, or SMT)is enabled on ESX server for best performance if both the processors and theBIOS support this feature. It allows a single physical processor core to behavelike two logicalprocessors, essentially allowing two independent threads torun simultaneously.
　　
　　ESX Memory considerations
　　Virtualization causes anincrease in the amount of physical memory required due to the extra memoryneeded by ESX for its own code and for data structures. This additional memoryrequirement can be separated into two components:
　　·       A system-wide memory space overhead for the serviceconsole (typically 272MB) and forthe VMkernel (typically about 100MB).
　　·       An additional memory space overhead for each virtualmachine.
　　We recommend that you carefullyselect the amount of memory you allocate to your virtual machines, so that youcan minimize swapping, and avoid over-allocating memory. Allocating more memorythan needed increases the virtual machine memory overhead.
　　ESX uses three memorymanagement mechanisms — page sharing,ballooning, and swapping —to dynamically reduce the amount of machine physical memory required for eachvirtual machine. The three[E2] mechanismsabove are enabled on ESX automatically. The VMware administratorcould investigate memory overcommit issues according to these three values inthe vSphere Client Performance Chart if it affects the performance of a virtualmachine.
　　In addition to the usual4KB memory pages, ESX also makes 2MB memory pages available as the largepage. The use of large pages results in reduced memory management overhead, andthus it increases hypervisor performance.
　　
　　ESX Storage Considerations
　　The ESX storageconsideration relates to so many system-level designs so that the common userdoesn’t have to know them deeply. In here, we just introduce the main featureson ESX storage.
　　ESX supports threevirtual disk modes: Independent persistent, Independent nonpersistent, and Snapshot.
　　·       Independentpersistent – Changes areimmediately written to the disk, so this mode provides the best performance.         
　　·       Independentnonpersistent – Changesto the disk are discarded when you power off or revert to a snapshot.
　　·       Snapshot – A snapshot captures the entire stateof the virtual machine. This includes the memory and disk states as well as thevirtual machine settings.
　　ESX supports multipledisk types
　　·       Thick – Thick disks, which have all their space allocated atcreation time, are further divided into two types: eager zeroed and lazyzeroed.
　　·       Thin – Space required for a thin-provisioned virtual disk isallocated and zeroed upon demand, as opposed to upon creation.
　　The alignment of thefile system partitions can impact performance. So we recommend that you createVMFS partitions to avoid this problem since it automaticallyaligns the partitions along the 64KB boundary.
　　Multiple heavily-usedvirtual machines concurrently accessing the same VMFS volume, or multiple VMFSvolumes backed by the same LUNs, can result in decreased storage performance.
　　I/O latency statisticscan be monitored using esxtop (or resxtop), which reports device latency,time spent in the kernel, and latency seen by the guest.
　　Disk I/O bandwidth canbe unequally allocated to virtual machines by using the vSphere Client.
　　
　　ESX Networking Considerations
　　Because insufficient CPUresources will reduce maximum throughput, it is important to monitor the CPUutilization of high-throughput workloads for networking. We suggest observingthe status of CPU resources through the vSphere client regularly.
　　If the virtual machinesare connected to different virtual switches, traffic will go through wires andincur unnecessary CPU and network overhead.
　　If the switch is configuredwith a specific speed and duplex setting, however, the administrator must forcethe network driver to use the same speed and duplex setting.
　　
4.   VirtualInfrastructure Management
　　The followingsuggestions included in this section can be implemented using the vSphereClient connected to a VMware vCenter server or an individual ESX host. Resourcemanagement configurations can have a significant impact on virtual machineperformance.
　　·       If you expect frequent changes to the total availableresources, use Shares, instead of Reservation, to allocate resourcesfairly across virtual machines.
　　·       Use Reservationto specify the minimum acceptable amount of CPU or memory, instead of the amountyou would like to have.
　　·       Use resource pools for delegated resource management.To fully isolate a resource pool, set the resource pool type toFixed and useReservation andLimit.
　　·       Group virtual machines for a multi-tier service into a resourcepool. This allows resources to be assigned for the service as a whole.
　　
　　VMware vCenter Best Practices
　　Large numbers of managedhosts, managed virtual machines, and connected VMware vSphere Clients can impactthe performance of a vCenter server. Exceeding the supported maximums, thoughit might work, is even more likely to impact vCenter performance. Currently,vCenterServer in IIG Platform team manages over 40 ESX hosts and over 1500VMs. We should migrate the vCenter Server on more powerful server for the bestperformance. Besides that, the following tips could help improve vCenterperformance.
　　·       Disconnect VMware vSphere Clients from the vCenterserver when they are no longer needed.
　　·       Avoid overly-aggressive vCenter alarm settings. Eachtime an alarm condition is met the vCenter server must take appropriate actions.
　　·       The VMWare user should consider running VMware vCenter UpdateManager and vCenter Converter on its own system and providing it with adedicated database.
　　·       We recommend that the user select Microsoft WindowsServer 2008 64-bit operating system for vCenter installations with no more than300 hosts or3000 virtual machines.
　　
5.   vSphere Case Study
　　The following solutions are shown for vSphere Systemmanagement. They include the most common issues that occurred in the practicalwork for IIG Platform team.
　　·      How to upgradeESX Server to 4.0.4 (the latest version) properly
　　1.    Download the Update package“update-from-esx4.0-4.0_update04.zip”.
　　2.    Transfer the update package file(update-from-esx4.0-4.0_update04.zip) onto the ESX server (make sure the ESXserver has a few gigabytes of available space, or the upgrade will fail).
　　3.    In Lab Manager, undeploy all the VMs that arerunning on the host.
　　4.    In Lab Manager, go to the Resources link ->Hosts tab -> Highlight the host -> Disable (this will prevent any newconfigurations from deploying onto the host).
　　5.    In Lab Manager, go to the Resources link ->Hosts tab and unprepare Host (It could cut off the connection between LabMangerand vCenter Server).
　　6.    In the Virtual Center, put the host intomaintenance mode.
　　7.    Ssh into the ESX host and run this command inthe same directory where the zip file locates :
　　esxupdateupdate --bundle update-from-esx4.0-4.0_update04.zip
　　8.    After the command finishes, it will prompt youto reboot the server.  After rebooting, it will be upgraded to ESX 4.0Update 4.
　　9.    In Virtual Center, go back to the host and exitmaintenance mode
　　10.  InLab Manager, prepare Host and enable the host again.
　　11.  InLab Manager, reset Physical Network if necessary.
　　
　　·      How to resolvethe full disk space issue and purge SqlServer database for vCenter Server
　　1.    Shut down the vCenter Server services.
　　2.    Shrink the Database of vCenter Server.
　　(Please refer to http://msdn.microsoft.com/en-us/library/ms189035.aspx)
　　3.    Run the purge scripts from VMWare to purge theperformance data.
　　(Please refer to http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&externalId=1025914)
　　4.    Reducethe size of the VirtualCenter database manually when the rollup scripts take along time to run. You must run the specific delete sql statementsmanually on database.
　　(Please refer to http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&externalId=1007453)
　　5.    If the above operation still couldn’t reduce thesize of database effectively. You have to detach the data files of database inthe current disk and move the data to a larger disk. After that, you could attachthe data files of database in a new disk.
　　
　　·      How to allocatethe resources on Lab Manager to improve performance
　　1.     Createa workspace for each Lab Manager User.
　　2.     Allocatethe specific resource to the specific user as required.
　　3.     Restrictthe access privileges in case of performance limitation.
　　For more information about howto migrate VMs and allocate the resource effectively, refer to the following IIGPlatform xPlore team sample:
　　
　　·      How to resolveNIC's connected status modified issue on Lab Manager
　　The followingerrors will be thrown when you deploy a configuration of 20 or more virtualmachines in Lab Manager, if the physical network is configured to use aDistributed Virtual Switch.
　　NIC's connected status modified
<date stamp> <time stamp>
Network Interface: NIC <n>
NIC's connected status was modified in vCenter.
　　The resolution isto set Default Local vNetwork Distributed Switch Ports to X&#43;1where X is the number of virtual machines in the configuration. Make sure theversion of Lab Manager is larger than 4.0.2. The change takes effect only inconfigurationsthat are deployed after the change is made or after you redeploythe current configurations that generated a related error.
　　(Please refer to http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1020450)
　　
　　·      How toresolve the issue Powering off a virtual machine fails with the error
　　An error message that resembles the following may occur inthe console of the vSphere client when you power off a virtual machine. Forexample,Cannot power Off: Another task isalready in progress.
　　You could resolve this issue using the following steps.
　　1.     Openthe .vmx file of the virtual machine using a text editor.
　　2.     Commentthis line:
#log.fileName = &quot;/vmfs/volumes/4b8bd18f-c1f89b5a-1914-002219c8e7a3/vmware.log&quot;
　　3.    Restart the virtual machine for the changes totake effect.
# vmware-vim-cmd vmsvc/getallvms
This command returns the VMID.
# vmware-vim-cmd vmsvc/reload <VMID>
　　(Please refer to http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1027040&sliceId=1&docTypeID=DT_KB_1_1&dialogID=286943832&stateId=0%200%20286949092)
　　·      How to troubleshootvirtual machine performance issues
　　As the teamdevelops, the virtual machine performance issues will become the bottleneck ofwork efficiency. The following tips from VMWare Knowledge Base website couldhelp improve the performance of VM.
　　http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1008360
　　The performancetuning of VMs is the main task for every VMWare System Administrator. Actuallyso many factors could impact the performance of VM and we are still seeking forthe better performance of VMs so far.
　　
　　·      How to troubleshootvirtual machine performance issues
　　Symptom:
　　The configurationyou are trying to deploy requires consolidation of one or more virtualmachines. You can consolidate either the configuration or the virtual machine&quot;CS70C18DFS&quot;.
　　This is due to thelimit on the number of hosts that can simultaneously access the same storage.
　　
　　·      How to delete virtual machine file on SAN storage
　　Symptom:
　　Cannot delete file[SANFS] chnservices-lm/-17162
　　
　　
　　
　　·      VMWareLab Manager Best Practice
　　1.     InstallvCenter Server/Lab Manager on VM env for DRS/FailOver
　　2.     VMName Definition and improve Reusability
　　3.     Poweroff VMs in idle time
　　4.     Usingvirtual networks in case of network conflict
　　5.     UsingOrganization/Workspace to manage resource
　　6.     InstallVM tool
　　7.     Don’tsnapshot in Lab Manager and using Library on Configuration.
　　8.     UsingLinked Clone not Full Clone for VM
　　9.     LimitVM numbers/Configuration numbers
　　10.  LimitCPU/Memory sizes
　　11.  Use StorageLeases
　　12.  Theshared storage/vCenter are the main bottlenecks not LM
　　13.  UpgradevSphere(ESX/vCenter/LabManager) timely
　　14.  BackupvCenter/Lab Manager database periodically
　　
6.   Conclusion
　　The VMware vSphere performance tuning is very important andcomplicated for every VMware administrator. The performance considerationincludes CPUs, Memories, Storages and Networking configurations on multiplelayers (Hardware/ESX/Virtual Machine). So the understandingof vSphere designand concept is essential. And vSphere client as the common and useful vSpheremanagement tool should be mastered proficiently for the best performance.VMware community on VMware official website also could help the VMware usersfind out thesolution to resolve all kinds of VMware vSphere issues.
7.   Reference
　　[1] Performance BestPractices for VMware vSphere 4.0
　　http://www.vmware.com/pdf/Perf_Best_Practices_vSphere4.0.pdf
　　[2] PerformanceTroubleshooting for VMware vSphere 4
　　http://www.vmware.com/resources/techresources/10066
　　[3] Understanding MemoryResource Management in ESX.pdf
　　http://www.vmware.com/files/pdf/perf-vsphere-memory_management.pdf
　　[E1]Wedo not use “due to” to start a clause. Usually, “due to” is followed by agerund or noun, for example, “Due to the heavy rain, he did not come” (We donot say “Due to it was rainy, he did notcome.”)
　　[E2]Fornumbers smaller than ten, better to spell them out.
         版权声明：本文为博主原创文章，未经博主允许不得转载。