Scale Up or Scale Out?

Well? 

You know the answer to this one.  It depends.  🙂  Of course it depends as every environment is different.  However, today I was looking into the scalability of a XenApp 6.5 farm for a customer to see if we wanted to scale up or out.

Tradition says to scale out but tradition is for ceremonies, Christmas and getting tourists to part with their money 🙂  Tradition doesn’t have a place in today’s modern virtualised world of IT.  So, in order to work out if we need to scale up and out, I needed data!  “It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts”.

That data came in two formats – knowledge of what governs why we would scale up or out in the form this article by Citrix Architect Nick Rintalan – very useful.  Also, in order to understand the current hardware used and see what the CPU and NUMA configuration was I found this article and this article on the Xen Wiki very useful indeed!

The second part of the data came from EdgeSight.  Real life actual statistical data from the client’s XenApp farm. Treasure!

To give a little bit of background, the customer is running XenApp 6.5 HRP2 on XenServer 6.0.2 to publish a single, business critical application, which turns out is a bit more CPU hungry than originally envisaged.  The XenApp servers are provisioned via a Provisioning Services 6.1 vDisk.  All basic stuff.

The underlying hardware uses 4 Intel Xeon E7540 2GHz CPUs with 6 cores each which gives us a grand total of 48 logical cores (4 x 6 x 2 ;-p).  Plenty to be getting on with!  The current XenApp VM configuration is 6 vCPU and 32GB RAM.

EdgeSight was showing that the servers were getting their CPU maxed out with around 52 users, each running the same published application.  Rough maths shows that this means each instance of the process was using about 11% of a CPU.  In comparison, we were seeing peak memory usage of 28GB so about 87-88% meaning we did have some memory to spare.

So, do we add more vCPU, say to take it to 8 or add more XenApp servers?  The decision is going to be based on if the XenApp servers will scale linearly or not, i.e if we add an additional 33.3% of CPU, will we get an additional 33.3% of users?  In this case would we max out at 70 users?  If we maxed out at less than 70 users then we’re not scaling linearly so it would be better to scale out – savvy?

In order to try and work this out prior to any testing, I wanted to understand the NUMA topology of the Intel E7540’s to see what the multiples would be (again, reference here for more details).

On the XenServer host, using the command xl info –n, you will see the following:

CPU configuration

CPU configuration

This confirms that we’re running 48 cpus and have 4 nodes.  These are the NUMA nodes.  As we have 4 sockets, that means 1 node per socket.  Scrolling down, we see the following to confirm that and also show we’re getting 12 CPUs per NUMA node.

Numa configuration

Numa configuration 1

Numa configuration 2

Numa configuration 2

Numa configuration 3

Numa configuration 3

So, to summarise, based on Mr Rintalan’s insight, we want to be running CPU multiples of the NUMA configuration, in this instance 12.  We don’t want to use less CPUs than we currently have (4), which means the next one up is 12 .  This is not practical as to have 12 vCPU we’ll need 64GB RAM which isn’t an option at present so with all this in mind we ARE running the OPTIMAL configuration at present so the answer is…

Scale out 🙂

Thanks for reading and I hope you find my experiences and the links useful.

Advertisements

2 thoughts on “Scale Up or Scale Out?

  1. JP says:

    Are you saying that in order to get 52 users on a server you would need 12 vcpu’s and 64gb of ram? How many users could I expect to get on a server with 4vcpu and 8gb of RAM? Thanks.

    • Hi JP,

      I think you have misunderstood some of my article. At the beginning I state the answer to scale out or up is “It depends. Of course it depends as

        every environment is different

      “. In the environment I was working in I had a particular hardware configuration and a single application that was CPU intensive, not memory intensive. The figures I was working with would be completely different to the figures you will have for your environment. However, the approach to working out these figures will be the same.

      If you look at the links I’ve included in the article, they will give you more information into

        NUMA

      and how to spec your XenApp VMs for optimum load. Also, you need to take into account the load itself as all applications will behave differently and if you are publishing desktops instead of applications, you will again see a difference in load. Using something similar to Citrix EdgeSight or Lakeside SysTrack will give you the metrics to work this out so unfortunately I cannot comment on your circumstances as there is no “one size fits all” for VM sizing. Also be aware that not all environments will scale linearly – i.e. doubling your CPU/RAM will not necessarily double your user load.

      Thanks for reading my article.

      Stephen

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s