0

I'll preface this by saying I'm fairly new to working with AWS.

Yesterday I deployed a new version of one of our services to our test environment.
Most people are on holidays at the moment, so there isn't a lot of usage of this particular container.
The new version I deployed had some changes to logger configuration, the logger uses a new encoder that outputs logstash json format logs. So no business logic changes.
I deployed the service and it timed out because the health check failed twice (trying to run two instances).
I added 30 seconds to the wait time to see if it would deploy, it hung for a long time with CPU at 100%, but eventually stabilised.
However, once it stabilised, the CPU was running between 0% and 10% and seemed quite erratic. I let it run for a while then decided to revert to the previous image and investigate what the issue might be.
When I reverted to the older image the same thing happened. Spiked at 100% for a while, then stabilised, with the CPU active with around 10% load.
I left it over night, but it's still the same this morning.
What can the issue be? The old image worked fine before I deployed the new one.

We're using ECS, running Tasks.
I should note that the service is working as expected, I can query the API's and I get results quickly. This is what the health graphs looked like this morning:
container health graphs CPU and Memory

The flat part of the graph is using the original task, the first spike is deploying the new task, and the second spike is re-deploying the original task.
What could be the issue here?

Here is the same graph over a longer time period: graph CPU and Memory 1 week

Improve this question

1 Answer 1

Reset to default
0

We are facing similar issue with our ECS service where average CPU spikes up to 20-30% for around 1 hour after the deployment succeeds and same trend for memory as you have shown, and then it stabilises around 5% post ~1 hour. Were you able to figure this out?

Improve this answer
4
  • We weren't I'm afraid, just like you said, it seemed to stabilise after an hour or so. Never got to the bottom of it. It did stop happening after a while, and hasn't happened much recently. So I'm wondering if it's something going on with the 'magic' in the background. In any case, I haven't given it much thought since I've had other priorities I needed to focus on. Commented Feb 9, 2024 at 12:58
  • Also, FYI - this should probably be a comment on my question, rather than an answer. :-) Commented Feb 9, 2024 at 12:59
  • Hasn't stopped happening for us apparently, will need to go to the bottom of it..Alas! PS: don't have enough repo points to comment so had to resort to answer :/ Commented Feb 9, 2024 at 13:27
  • This does not really answer the question. If you have a different question, you can ask it by clicking Ask Question. To get notified when this question gets new answers, you can follow this question. Once you have enough reputation, you can also add a bounty to draw more attention to this question. - From Review Commented Feb 9, 2024 at 15:54

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.