Error:

Power Measurement Configuration:

Physical Configuration:

Virtual Configuration:

Notes:

Physical System Notes

• BIOS settings used:
• Boot Mode = UEFI
• System Profile = Performance

Storage Notes

• The System Under Test used 16 x 400 GB SSDs & 2 x 120 GB SSDs. Detailed configuration is as follows:
• Create 2 x RAID10 arrays, each with 8 x 400 GB SSD drives
• Create 8 x 135 GB logical disks for each RAID array for each tile's VMs (except DB server VM)
• Create 2 x 40 GB logical disks for each RAID array for DB server VMs
• Create 1 x RAID10 array with 2 x 120 GB SSD drives for the OS. The other logical disks were partitioned for the VMs of 1 tile

Virtualization Software Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline clocksource=tsc pci=realloc cgroup_disable=memory,cpu,cpuacct,blkio,hugetlb nosoftlockup mce=ignore_ce ipv6.disable=1 intel_iommu=on intel_idle.max_cstate=0
• The SUT booted RHEL 7.2 using kernel-3.10.0-327.36.3.el7.x86_64
• The following services were disabled:
• abrt-oops abrt-pstoreoops abrtd arp-ethers atd autofs autovt@ avahi-daemon blk-availability cgconfig cgdcbxd cgred chrony-wait chronyd console-getty console-shell cpupower crond debug-shell dm-event dnsmasq ebtables fcoe firewalld gssproxy ip6tables ipmievd iprdump iprinit iprupdate iptables irqbalance iscsid iscsiuio kdump ksm ksmtuned libstoragemgmt libvirt-guests lldpad lvm2-lvmetad lvm2-lvmpolld NetworkManager-dispatcher NetworkManager-wait-online NetworkManager nfs-blkmap nfs-server nfs ntpdate numad plymouth-halt plymouth-kexec plymouth-poweroff plymouth-quit-wait plymouth-quit plymouth-read-write plymouth-reboot plymouth-start postfix psacct radvd rdisc rdma rhel-dmesg rhel-domainname rhsmcertd rngd rsyncd saslauthd serial-getty@ smartd svnserve systemd-bootchart systemd-nspawn@ systemd-ask-password-plymouth target tcsd virt-who wpa_supplicant
• /usr/lib/sysctl.d/specvirt.conf was updated with the following lines:
• kernel.sched_min_granularity_ns = 10000000
• kernel.sched_wakeup_granularity_ns = 15000000
• kernel.sched_latency_ns = 20000000
• kernel.sched_migration_cost_ns = 500000
• kernel.numa_balancing = 0
• vm.dirty_background_ratio = 5
• vm.dirty_ratio = 10
• vm.swappiness = 0
• Network configuration is as follows:
• The X520-DA2 SFP+ were configured in /etc/modprobe.d/ixgbe.conf as follows,
• options ixgbe FCoE=0,0,0,0
• blacklist ixgbevf
• The X520-DA2 SFP+ were configured in /etc/rc.d/rc.local as follows,
• echo '63' > /sys/bus/pci/devices/0000:05:00.0/sriov_numvfs
• echo '63' > /sys/bus/pci/devices/0000:05:00.1/sriov_numvfs
• echo '63' > /sys/bus/pci/devices/0000:82:00.0/sriov_numvfs
• echo '63' > /sys/bus/pci/devices/0000:82:00.1/sriov_numvfs
• All the webservers, infraservers, mailservers, appservers, batchservers and dbservers were assigned with a VF
• VM configurations were as follows:
• All VMs were managed with libvirt
• All VMs were created with the following XML template:
• <domain type='kvm'>
• <name>$vmname</name>
• <memory>$vmem</memory>
• <currentMemory>$vmem</currentMemory>
• <vcpu placement='static' cpuset='$host_cpu_list'>$vcpus</vcpu>
• <os>
• <type arch='x86_64' machine='pc'>hvm</type>
• <boot dev='hd'/>
• </os>
• <features>
• <acpi/>
• <apic/>
• <pae/>
• </features>
• <cpu mode='host-passthrough'>
• <topology sockets='1' cores='$vcpus' threads='1'/>
• </cpu>
• <clock offset='utc'>
• <timer name='pit' tickpolicy='delay'/>
• </clock>
• <on_poweroff>destroy</on_poweroff>
• <on_reboot>restart</on_reboot>
• <on_crash>restart</on_crash>
• <devices>
• <emulator>/usr/libexec/qemu-kvm</emulator>
• <disk type='block' device='disk'>
• <driver name='qemu' type='raw' cache='none' io='native'/>
• <source dev='$vdisk'/>
• <target dev='vda' bus='virtio'/>
• </disk>
• <interface type='network'/>
• <mac address='$vmac'/>
• <source network='$vnet'/>
• </interface>
• <memballoon model='none'/>
• </devices>
• </domain>
• VMs' nice value was set after startup via the renice command to the following settings:
• appservers = -19
• dbserver = -19
• webserver = 19
• infraserver = 19
• mailserver = 19
• batchserver = 19
• All VMs of tiles 1,3,5 and 7 were bound to NUMA node 0
• All VMs of tiles 2,4,6 and 8 were bound to NUMA node 1
• infraserver9 mailserver9 webserver9 were bound to NUMA node 0
• appserver9 batchserver9 dbserver5 dbserver9 were bound to NUMA node 1

Web Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline cgroup_disable=memory,cpu,blkio console=ttyS0,115200
• ixgbevf module option used: InterruptThrottleRate=2000
• The following services were disabled:
• abrt-ccpp, abrt-oops, abrtd, atd, auditd, autofs, cgconfig, cgred, cups, dovecot, haldaemon, htcacheclean, httpd, ip6tables, iptables, irqbalance, kdump, messagebus, netconsole, nfs, nfslock, ntpd, ntpdate, php-fpm, portreserve, postfix, psacct, qpidd, quota_nld, rdisc, restorecond, rhnsd, rhsmcertd, rngd, rpcgssd, rpcsvcgssd, saslauthd, smartd, winbind
• tuned virtual-guest profile applied to VM
• Following items were added in /etc/sysctl.conf
• fs.file-max = 500000
• net.core.somaxconn = 81920
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_tw_reuse=1
• net.ipv4.tcp_keepalive_time = 120
• net.ipv4.tcp_keepalive_intvl = 30
• net.ipv4.tcp_timestamps = 0
• vm.swappiness=0
• root file-system options: barrier=0,noatime
• Contents of nginx configuration file /etc/nginx/nginx.conf
• user nginx;
• worker_processes 4;
• worker_cpu_affinity 00000001 00000001 00000010 00000010;
• worker_rlimit_nofile 65535;
• pid /var/run/nginx.pid;
• events {
• use epoll;
• worker_connections 65535;
• multi_accept on;
• accept_mutex off;
• }
• http {
• include /etc/nginx/mime.types;
• default_type application/octet-stream;
• log_format common '$remote_addr - $remote_user [$time_local] $request'
• '"$status" $body_bytes_sent ';
• access_log /var/log/access.log common gzip buffer=4m flush=1m;
• sendfile on;
• sendfile_max_chunk 1m;
• tcp_nopush on;
• tcp_nodelay on;
• keepalive_timeout 120;
• open_file_cache max=204800 inactive=20s;
• open_file_cache_min_uses 1;
• open_file_cache_valid 30s;
• charset utf-8;
• server_names_hash_bucket_size 128;
• client_header_buffer_size 2k;
• large_client_header_buffers 4 4k;
• include /etc/nginx/conf.d/*.conf;
• }
• Contents of nginx configuration file /etc/nginx/conf.d/default.conf
• server {
• listen 80 backlog=8192;
• server_name localhost;
• location / {
• root /var/www/html;
• index index.html index.htm index.php;
• }
• error_page 500 502 503 504 /50x.html;
• location = /50x.html {
• root /usr/share/nginx/html;
• }
• location ~ .*\.(html|htm|gif|jpg|jpeg|bmp|png|ico|txt|js|css)$
• {
• root /var/www/html;
• expires 100d;
• }
• location ~ ^/images/ {
• root /var/www/html/support;
• expires 100d;
• }
• location ~ \.php$ {
• root html;
• fastcgi_pass unix:/dev/shm/php-cgi.sock;
• fastcgi_index index.php;
• fastcgi_param SCRIPT_FILENAME /var/www/html$fastcgi_script_name;
• include fastcgi_params;
• }
• }
• Contents of nginx configuration file /etc/nginx/conf.d/example_ssl.conf
• server {
• listen 443 ssl;
• server_name localhost;
• ssl on;
• ssl_certificate /etc/pki/tls/certs/localhost.crt;
• ssl_certificate_key /etc/pki/tls/private/localhost.key;
• ssl_session_cache shared:SSL:1m;
• ssl_session_timeout 5m;
• ssl_ciphers ALL:!kEDH!ADH:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP;
• ssl_prefer_server_ciphers on;
• location / {
• root /var/www/html;
• index index.html index.htm index.php;
• }
• location ~ .*\.(html|htm|gif|jpg|jpeg|bmp|png|ico|txt|js|css)$
• {
• root /var/www/html;
• expires 100d;
• }
• location ~ ^/images/ {
• root /var/www/html/support;
• expires 100d;
• }
• location ~ \.php$ {
• root html;
• fastcgi_pass unix:/dev/shm/php-cgi.sock;
• fastcgi_index index.php;
• fastcgi_param SCRIPT_FILENAME /var/www/html$fastcgi_script_name;
• include fastcgi_params;
• }
• }
• The following lines were added to /etc/security/limits.conf:
• * soft nofile 800000
• * hard nofile 800000
• * soft nproc 102400
• * hard nproc 102400
• The following lines were changed in /etc/php-fpm.d/www.conf:
• listen = /dev/shm/php-cgi.sock
• listen.owner = nginx
• listen.group = nginx
• user = nginx
• group = nginx
• NFS mount options used: defaults,noatime,noacl,actimeo=14400,nocto

Application Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline cgroup_disable=memory,cpu,blkio console=ttyS0,115200
• The following services were disabled:
• abrt-ccpp, abrt-oops, abrtd, atd, auditd, autofs, cgconfig, cgred, crond, cups, dovecot, haldaemon, htcacheclean, httpd, ip6tables, iptables, irqbalance, kdump, mdmonitor, messagebus, netconsole, nfs, nfslock, ntpd, ntpdate, portreserve, postfix, psacct, qpidd, quota_nld, rdisc, restorecond, rhnsd, rhsmcertd, rngd, rpcgssd, rpcsvcgssd, saslauthd, smartd, winbind
• tuned virtual-guest profile applied to VM
• ixgbevf module option used: InterruptThrottleRate=7000
• Following items were added in /etc/sysctl.conf
• fs.file-max = 500000
• net.ipv4.tcp_timestamps = 0
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.tcp_rmem = 1048576 1048576 41943040
• net.ipv4.tcp_wmem = 1048576 1048576 41943040
• net.ipv4.tcp_mem = 1048576 1048576 41943040
• net.ipv4.tcp_window_scaling = 0
• net.ipv4.tcp_tso_win_divisor = 8
• net.ipv4.tcp_max_syn_backlog = 8192
• net.ipv4.conf.all.arp_filter=1
• net.ipv4.ip_local_port_range = 10000 65000
• net.ipv4.tcp_tw_reuse=1
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.core.rmem_max = 41943040
• net.core.wmem_max = 41943040
• net.core.rmem_default = 1048576
• net.core.wmem_default = 1048576
• net.core.optmem_max = 41943040
• net.core.netdev_max_backlog = 81920
• net.core.somaxconn = 81920
• vm.nr_hugepages = 3072
• vm.swappiness=0
• vm.min_free_kbytes = 65536
• vm.nr_hugepages = 3072
• root file-system options: defaults,barrier=0
• Following items were set in /etc/rc.d/rc.local
• echo tsc > /sys/devices/system/clocksource/clocksource0/current_clocksource
• The following lines were added to glassfish domain.xml:
• To jdbc-connection-pool line parameter 'max-pool-size = 1000' was set
• To request-processing line parameter 'initial-thread-count = 90' was set
• To ejb-container line, parameter 'max-pool-size = 512' was set
• To mdb-container line, parameter 'max-pool-size = 512' was set
• In keep-alive section: set timeout-in-seconds=300
• In http-thread-pool section: set max-thread-pool-size=32
• In jvm-options section: set -Xmn1024m, -Xms4800m, -Xmx4800m, -XX:PermSize=256m, -XX:MaxPermSize=384m
• set -XX:+UseCompressedOops, -XX:+AggressiveOpts, -XX:+UseLargePages, -XX:CompileThreshold=1000, -XX:-UseBiasedLocking,
• set -XX:+DisableExplicitGC, -XX:+UseParallelOldGC, -XX:ParallelGCThreads=4, -XX:+UseFastAccessorMethods, -XX:+UseStringCache,
• set -XX:+OptimizeStringConcat, -Djava.nio.channels.spi.SelectorProvider=sun.nio.ch.EPollSelectorProvider
• The following lines were added to /etc/security/limits.conf:
• root hard memlock 50000000
• root soft memlock 50000000
• root hard nofile 500000
• root soft nofile 500000
• root hard nproc 100000
• root soft nproc 100000

Mail Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline cgroup_disable=memory,cpu,blkio console=ttyS0,115200
• ixgbevf module option used: InterruptThrottleRate=1000
• The following services were disabled:
• abrt-ccpp, abrt-oops, abrtd, atd, auditd, autofs, cgconfig, cgred, cups, dovecot, haldaemon, htcacheclean, httpd, ip6tables, iptables, irqbalance, kdump, messagebus, netconsole, nfs, nfslock, ntpd, ntpdate, portreserve, postfix, psacct, qpidd, quota_nld, rdisc, restorecond, rhnsd, rhsmcertd, rngd, rpcgssd, rpcsvcgssd, saslauthd, smartd, winbind
• tuned virtual-guest profile applied to VM
• Following items were added in /etc/sysctl.conf
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• root file-system options: barrier=0
• mail store file-system options: noatime,barrier=0,data=writeback
• Contents of dovecot configuration file /etc/dovecot/dovecot.conf:
• dict {
• }
• !include conf.d/*.conf
• base_dir = /var/run/dovecot/
• disable_plaintext_auth = no
• log_path = /var/log/dovecot.log
• mail_location = dbox:/mailstore/dovecot/maildir/%u
• mbox_write_locks = fcntl
• mail_fsync=never
• maildir_very_dirty_syncs = yes
• mbox_very_dirty_syncs = yes
• passdb {
• args = scheme=plain /etc/dovecot/passwd.dovecot
• driver = passwd-file
• }
• postmaster_address = postmaster@example.com
• service auth {
• user = root
• }
• service imap-login {
• inet_listener imap {
• port = 143
• }
• process_min_avail = 10
• service_count = 0
• }
• service imap {
• process_limit = 1024
• }
• ssl = no
• ssl_cert = </etc/pki/dovecot/certs/dovecot.pem
• ssl_key = </etc/pki/dovecot/private/dovecot.pem
• userdb {
• driver = passwd
• }
• The following lines were added to /etc/security/limits.conf:
• * soft nofile 102400
• * hard nofile 102400
• * soft nproc 102400
• * hard nproc 102400

Database Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline cgroup_disable=memory,cpu,blkio console=ttyS0,115200
• The following services were disabled:
• abrt-ccpp, abrt-oops, abrtd, atd, auditd, autofs, cgconfig, cgred, cups, dovecot, haldaemon, htcacheclean, httpd, ip6tables, iptables, irqbalance, kdump, messagebus, netconsole, nfs, nfslock, ntpd, ntpdate, portreserve, postfix, postgresql-9.2, psacct, qpidd, quota_nld, rdisc, restorecond, rhnsd, rhsmcertd, rngd, rpcgssd, rpcsvcgssd, saslauthd, smartd, winbind
• tuned virtual-guest profile applied to VM
• ixgbevf module option used: InterruptThrottleRate=7000
• Following items were added in /etc/sysctl.conf
• kernel.sem = 500 64000 400 512
• kernel.shmmax = 137438953472
• kernel.shmmni = 11118096
• kernel.shmall = 8294967296
• net.ipv4.tcp_timestamps = 0
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.tcp_rmem = 1048576 1048576 41943040
• net.ipv4.tcp_wmem = 1048576 1048576 41943040
• net.ipv4.tcp_mem = 1048576 1048576 41943040
• net.ipv4.tcp_window_scaling = 0
• net.ipv4.tcp_tso_win_divisor = 8
• net.ipv4.tcp_max_syn_backlog = 8192
• net.ipv4.ip_local_port_range = 10000 65000
• net.core.rmem_max = 41943040
• net.core.wmem_max = 41943040
• net.core.rmem_default = 1048576
• net.core.wmem_default = 1048576
• net.core.optmem_max = 41943040
• net.core.netdev_max_backlog = 81920
• net.core.somaxconn = 81920
• fs.file-max = 500000
• vm.min_free_kbytes = 65536
• net.ipv4.tcp_tw_reuse=1
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• vm.swappiness = 0
• root file-system options: defaults,barrier=0
• database store file-system options: defaults,noatime,barrier=0,data=writeback
• Following items were set in /etc/rc.d/rc.local
• echo tsc > /sys/devices/system/clocksource/clocksource0/current_clocksource
• Database tuning options changed based on the sample configuration:
• synchronous_commit = off
• wal_sync_method = open_datasync
• bgwriter_delay = 200ms
• listen_addresses = '*'
• commit_siblings = 3
• ssl = off
• full_page_writes = off
• autovacuum = off
• autovacuum_naptime = 6h
• debug_assertions = false
• log_rotation_size = 16MB
• log_destination = syslog
• logging_collector = off
• log_truncate_on_rotation = off
• log_rotation_age = 0
• log_rotation_size = 0
• track_activities = off
• track_counts = off
• default_statistics_target = 100
• shared_buffers=7000MB
• max_connections = 2000
• max_prepared_transactions = 1000
• work_mem=200MB
• maintenance_work_mem = 1025MB
• fsync = on
• commit_delay = 10
• checkpoint_segments = 256
• checkpoint_timeout = 300
• random_page_cost = 1.5
• effective_cache_size = 40GB
• temp_buffers = 2000
• vacuum_cost_delay = 50
• bgwriter_lru_maxpages=100
• full_page_writes = off
• wal_buffers = 2500
• checkpoint_warning = 99
• cpu_tuple_cost = 0.001
• cpu_index_tuple_cost = 0.0005
• cpu_operator_cost = 0.00025
• log_min_duration_statement = 1500
• The following lines were added to /etc/security/limits.conf:
• * soft nproc 578528
• * hard nproc 578528
• * soft memlock 33554432
• * hard memlock 33554432
• postgres soft nproc 578528
• postgres hard nproc 578528

Infraserver VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline cgroup_disable=memory,cpu,blkio console=ttyS0,115200
• The following services were disabled:
• abrt-ccpp, abrt-oops, abrtd, atd, auditd, autofs, cgconfig, cgred, cups, dovecot, haldaemon, htcacheclean, httpd, ip6tables, iptables, irqbalance, kdump, messagebus, netconsole, nfslock, ntpd, ntpdate, portreserve, postfix, psacct, qpidd, quota_nld, rdisc, restorecond, rhnsd, rhsmcertd, rngd, rpcgssd, rpcsvcgssd, saslauthd, smartd, winbind
• tuned virtual-guest profile applied to VM
• Following items were added in /etc/sysctl.conf
• fs.file-max = 500000
• net.core.somaxconn = 81920
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_tw_reuse=1
• net.ipv4.tcp_keepalive_time = 120
• net.ipv4.tcp_keepalive_intvl = 30
• net.ipv4.tcp_timestamps = 0
• net.ipv4.tcp_syn_retries = 4
• net.ipv4.tcp_retries2 = 7
• vm.swappiness=0
• root file-system options: barrier=0,noatime
• backend store file-system options: noatime,barrier=0,data=writeback
• httpd.conf changes:
• Timeout 20
• #KeepAliveTimeout 15
• #Include conf.d/*.conf
• #MaxKeepAliveRequests 100
• CustomLog logs/access_log common
• Listen 81
• ServerName besim:80
• <IfModule prefork.c>
• StartServers 10
• MinSpareServers 10
• MaxSpareServers 10
• ServerLimit 4096
• MaxClients 4096
• MaxRequestsPerChild 640000
• </IfModule>
• #IndexOptions FancyIndexing VersionSort NameWidth=* HTMLTable
• ScriptAlias /fcgi-bin/ "/var/www/fcgi-bin/"
• FastCgiIpcDir /etc/httpd/fastcgi
• <Directory "/var/www/fcgi-bin ">
• AllowOverride None
• Options +ExecCGI -Includes
• SetHandler fastcgi-script
• Order allow,deny
• Allow from all
• </Directory>
• AddHandler fastcgi-script fcgi
• FastCgiServer /var/www/fcgi-bin/besim_fcgi.fcgi -processes 50
• NFS export options:
• rw,async,no_subtree_check,no_root_squash
• ixgbevf module option used: InterruptThrottleRate=3000
• The following lines were added to /etc/security/limits.conf:
• * soft nofile 102400
• * hard nofile 102400
• * soft nproc 102400
• * hard nproc 102400

Batch Server VM Notes

• Boot parameters set: selinux=0 audit=0 elevator=deadline cgroup_disable=memory,cpu,blkio console=ttyS0,115200
• The following services were disabled:
• abrt-ccpp, abrt-oops, abrtd, atd, auditd, autofs, cgconfig, cgred, cups, dovecot, haldaemon, htcacheclean, httpd, ip6tables, iptables, irqbalance, kdump, messagebus, netconsole, nfs, nfslock, ntpd, ntpdate, portreserve, postfix, psacct, qpidd, quota_nld, rdisc, restorecond, rhnsd, rhsmcertd, rngd, rpcgssd, rpcsvcgssd, saslauthd, smartd, winbind
• tuned virtual-guest profile applied to VM
• root file-system options: barrier=0
• Following items were added in /etc/sysctl.conf
• fs.file-max = 500000
• net.core.somaxconn = 8192
• net.ipv4.tcp_max_tw_buckets = 500000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• The following lines were added to /etc/security/limits.conf:
• * soft nofile 102400
• * hard nofile 102400
• * soft nproc 102400
• * hard nproc 102400

Client Driver Notes

• All prime clients were VMs running on Red Hat Enterprise Linux Server 6.8 KVM
• The prime controller ran on the first vClient
• Two vClients were used for each tile. One vClient per tile ran the webserver workload, the other vClient per tile ran the other workloads
• Each vClient has 4 vCPUs, 8GB memory, 9.8GB disk, 1 virtio_blk device and 1 ixgbevf device
• The physical clients have 4 x 10 Gbps ports and were configured to use virtual functions for the vClients
• Create one raid10 array with 2 x 400GB SSDs. It contains one logical volume with all VM images deployed on it

Power Configuration Notes

• The Yokogawa WT210 power meter was connected to the SUT only.
• The Digi Watchport/H temperature sensor was connected to the SUT only.

Other Notes

• 1 x Dell Force10 S4810 switch was used to connect the clients and the SUT