0

I'm trying to simplify a monitoring system we have.

It has a variety of views for looking at the CPU usage of a server including:

  • Average CPU usage (overall, this accounts for all cores).

  • Low and high core occupancy (the # of cores used over 20% or 70% at a given time)

We also have specific metrics on the usage of each individual core.

Core occupancy is useful because you might have 4 cores, 1 core at 100% and 3 cores at 0%. In this case, you can go down to having 1 or 2 cores without affecting your workload, where as the average CPU usage would just show 25% (useless).

Keeping all of these separate metrics is very taxing as we have thousands of servers reporting metrics multiple times a minute.

Is there a standard way people use to measure CPU usage that accounts for both total power and # of cores used (how well you're parallelizing your work)?

Edit: We're getting some awesome/helpful responses in terms of engineering an actual system. But let's focus on the general problem of "How can you score/make a single or small set of metrics to represent a computer's usage of it's CPU resources including its use parallelism?".

Improve this question
8
  • 3
    It also depends on what you have running on the servers. 7 idle cores do you no good if your webapp only runs on one and it's maxed out. Commented Jul 5, 2017 at 18:47
  • You're right, that was why I left the core occupancy example in there :). It handles that kind of detection. Commented Jul 5, 2017 at 18:51
  • 1
    Ah, gotcha. There's no unified solution that I'm aware of...as much as monitoring teams tend prefer the one size fits all model, if it's a mission critical app the monitoring for that server has to be designed with the app behavior in mind. Especially once you have to start taking things like processor licensing (ugh) into account. Commented Jul 5, 2017 at 18:57
  • The question is what you planning on doing with the data collected. Are you trying to extract efficiency out of your environment or are you like a lot of us that have more compute than the apps know what to do with? Do you have the time/money to analyze these reports and implement findings? Do you want low level results or application layer results? Could the application owners shoulder most of this responsibility and determine for themselves if the apps are having CPU perf issues and then come to you for resource assignment or addition? Commented Jul 5, 2017 at 20:57
  • The use case is more like "profile thousands of servers and reduce their footprint as much as possible without sacrificing performance". So, some individual apps/databases may have extra constraints to review. But we're more interested in the larger common case where servers just have idling cores or little used cores that can be reduced. If an app pins processes to 2 cores and hits them at 40% checking core occupancy should catch it, making stuff like that generally useful. Commented Jul 5, 2017 at 20:57

2 Answers 2

Reset to default
0

Measuring the amount of idle time on your cpus is not a good way to measure performance or capacity. High cpu usage may indicate that more cpu could make the service go faster, but a better indicator is the load average. Firstly, this will tell you if work is waiting to be processed but is stuck on the run queue waiting for a timeslice, also, on linux (and some other operating systems) the scheduler will start pre-empting tasks when there is a backlog of tasks waiting to run (you have a backlog when the load average is greater than the number of cpus). This injection of context switches results in a decrease in throughput as the OS tries to complete a tssk and get it off the run queue.

Your description of 1 core at 100% and 3 idle cores may have lots of stories to explain it, and lots of solutions to make more efficent use of your "thousands of servers". You need to dig deeper to find out if it is a configuration issue, a lack of task sharding by the application, poor irq distribution or several other causes.

Improve this answer
0

There was always a LA out there serving for exactly the same cause. LA (Load Average) by itself shows a number of processes that were ready to run on the last check. This makes LA somewhat difficult to understand but also makes LA very useful to detect wrong server utilization or potential room for improvement.

Low LA tells us that the server is feeling well under current load so even 100% CPU usage is not a problem for it as it is responsive and prompt.

High LA tells us that server experiences some heavy load or can't get along with current tasks so even a 5% CPU usage may feel like a server is totally stuck and not responding. This can happen in various situations like waiting for data from disk (disk overloaded), too much data crossing the kernel boundary (bad coding/space for improvement), bad CPU behavior (hardware problems) and so on.

Margin between low and high LA is imaginary and can slightly differ from OS to OS. Back in the old days Linux was quite sloppy on LA 20 while BSD's could reach 100 without severe effects. This probably changed a lot today.

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.