8

edit: this seems to be a problem specifically with the AMI that I am trying to shut down. Other AMIs politely shut down with the "sudo poweroff" command.

The goal is for an EC2 instance to shut itself down. All my instances are Ubuntu 10.10.

I used to be able to execute the command "sudo poweroff -n -f" and the instance would shut down. However, now the instances stop responding and peg their CPUs at 50%.

I've tried every possible combination of shutdown/halt/poweroff and they all either:

  • Do nothing, leaving the instance still running
  • Put the instance into this halted/buys but still "on" state

Additional details / exact scenarios:

  • Run sudo poweroff: I get the normal "The system is going down for power off NOW!" message. Then am returned to the shell. The system does not power off. If I execute "sudo poweroff" again, the system seems to shut down (ssh connections terminated), however the AMI does not shut down and remains in a "running" state with constant 50% CPU usage (as monitored from the AWS Management console.

  • Run sudo shutdown now: I get the normal "The system is going down for maintenance NOW!" message but nothing happens. The system keeps running normally. Tailing the syslog I see that the following messages:

-

Dec 21 16:24:36 ip-10-212-177-30 init: ssh main process (599) terminated with status 255 Dec 21 16:24:36 ip-10-212-177-30 init: rc main process (677) killed by TERM signal Dec 21 16:24:36 ip-10-212-177-30 init: tty4 main process (678) killed by TERM signal Dec 21 16:24:36 ip-10-212-177-30 init: tty5 main process (682) killed by TERM signal Dec 21 16:24:36 ip-10-212-177-30 init: tty2 main process (685) killed by TERM signal Dec 21 16:24:36 ip-10-212-177-30 init: tty6 main process (688) killed by TERM signal Dec 21 16:24:36 ip-10-212-177-30 init: cron main process (692) killed by TERM signal Dec 21 16:24:36 ip-10-212-177-30 init: irqbalance main process (701) killed by TERM signal Dec 21 16:24:36 ip-10-212-177-30 init: tty3 main process (686) killed by TERM signal

All log messages after calling "sudo poweroff"

sudo poweroff

==> auth.log <== Dec 21 21:39:37 ip-10-244-74-146 sudo: ubuntu : TTY=pts/1 ; PWD=/home/ubuntu ; USER=root ; COMMAND=/sbin/poweroff ==> wtmp <== 02~~~runlevel2.6.35-28-virtualR?N?? Broadcast message from ubuntu@ip-10-244-74-146 (/dev/pts/1) at 21:39 ... The system is going down for power off NOW! ==> boot.log <== init: Disconnected from system bus init: dbus main process (663) killed by TERM signal hostname: the specified hostname is invalid

system is still running, no change

sudo poweroff

==> wtmp <== ~~~shutdown2.6.35-28-virtual)R?N^?

no more log files, system hangs, can't SSH in, CPU spins at 50%

sudo halt -p --verbose

ubuntu@ip-10-2-22-35:~$ /sbin/runlevel N 2 ubuntu@ip-10-2-22-35:~$ sudo halt -p --verbose Calling shutdown Broadcast message from ubuntu@ip-10-2-22-35 (/dev/pts/1) at 17:31 ... The system is going down for power off NOW! ubuntu@ip-10-2-22-35:~$ /sbin/runlevel 2 0 ubuntu@ip-10-2-22-35:~$ ps aux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.1 0.1 2892 1732 ? Ss 17:21 0:00 /sbin/init root 2 0.0 0.0 0 0 ? S 17:21 0:00 [kthreadd] root 3 0.0 0.0 0 0 ? S 17:21 0:00 [ksoftirqd/0] root 4 0.0 0.0 0 0 ? S 17:21 0:00 [migration/0] root 5 0.0 0.0 0 0 ? S 17:21 0:00 [watchdog/0] root 6 0.0 0.0 0 0 ? S 17:21 0:00 [events/0] root 7 0.0 0.0 0 0 ? S 17:21 0:00 [cpuset] root 8 0.0 0.0 0 0 ? S 17:21 0:00 [khelper] root 9 0.0 0.0 0 0 ? S 17:21 0:00 [netns] root 10 0.0 0.0 0 0 ? S 17:21 0:00 [async/mgr] root 11 0.0 0.0 0 0 ? S 17:21 0:00 [pm] root 12 0.0 0.0 0 0 ? S 17:21 0:00 [xenwatch] root 13 0.0 0.0 0 0 ? S 17:21 0:00 [xenbus] root 14 0.0 0.0 0 0 ? S 17:21 0:00 [sync_supers] root 15 0.0 0.0 0 0 ? S 17:21 0:00 [bdi-default] root 16 0.0 0.0 0 0 ? S 17:21 0:00 [kintegrityd/0] root 17 0.0 0.0 0 0 ? S 17:21 0:00 [kblockd/0] root 18 0.0 0.0 0 0 ? S 17:21 0:00 [ata_aux] root 19 0.0 0.0 0 0 ? S 17:21 0:00 [ata_sff/0] root 20 0.0 0.0 0 0 ? S 17:21 0:00 [khubd] root 21 0.0 0.0 0 0 ? S 17:21 0:00 [kseriod] root 22 0.0 0.0 0 0 ? S 17:21 0:00 [kmmcd] root 24 0.0 0.0 0 0 ? S 17:21 0:00 [khungtaskd] root 25 0.0 0.0 0 0 ? S 17:21 0:00 [kswapd0] root 26 0.0 0.0 0 0 ? SN 17:21 0:00 [ksmd] root 27 0.0 0.0 0 0 ? S 17:21 0:00 [aio/0] root 28 0.0 0.0 0 0 ? S 17:21 0:00 [ecryptfs-kthrea] root 29 0.0 0.0 0 0 ? S 17:21 0:00 [crypto/0] root 35 0.0 0.0 0 0 ? S 17:21 0:00 [khvcd] root 36 0.0 0.0 0 0 ? S 17:21 0:00 [kstriped] root 37 0.0 0.0 0 0 ? S 17:21 0:00 [kmpathd/0] root 38 0.0 0.0 0 0 ? S 17:21 0:00 [kmpath_handlerd] root 39 0.0 0.0 0 0 ? S 17:21 0:00 [ksnapd] root 40 0.0 0.0 0 0 ? S 17:21 0:00 [kondemand/0] root 41 0.0 0.0 0 0 ? S 17:21 0:00 [kconservative/0] root 143 0.0 0.0 0 0 ? S 17:21 0:00 [xfs_mru_cache] root 144 0.0 0.0 0 0 ? S 17:21 0:00 [xfslogd/0] root 145 0.0 0.0 0 0 ? S 17:21 0:00 [xfsdatad/0] root 146 0.0 0.0 0 0 ? S 17:21 0:00 [xfsconvertd/0] root 148 0.0 0.0 0 0 ? S 17:21 0:00 [xfsbufd/sda1] root 149 0.0 0.0 0 0 ? S 17:21 0:00 [xfsaild/sda1] root 150 0.0 0.0 0 0 ? S 17:21 0:00 [xfssyncd/sda1] root 177 0.0 0.0 0 0 ? S 17:21 0:00 [flush-202:1] root 183 0.0 0.0 2560 988 ? S 17:21 0:00 /sbin/plymouthd --mode=boot --attach-to-session root 187 0.0 0.0 4484 1260 ? S 17:21 0:00 mountall --daemon daemon 382 0.0 0.0 1968 536 ? Ss 17:22 0:00 portmap root 413 0.0 0.0 2304 252 ? Ss 17:22 0:00 dhclient3 -e IF_METRIC=100 -pf /var/run/dhclient.eth0.pid -lf /var/lib/dhcp3/dhclient.eth0.leases eth0 statd 593 0.0 0.0 2104 824 ? Ss 17:22 0:00 rpc.statd -L root 607 0.0 0.0 0 0 ? S 17:22 0:00 [kjournald] root 624 0.0 0.0 0 0 ? S 17:22 0:00 [rpciod/0] root 627 0.0 0.0 0 0 ? S< 17:22 0:00 [kslowd000] root 628 0.0 0.0 0 0 ? S< 17:22 0:00 [kslowd001] root 639 0.0 0.0 0 0 ? S 17:22 0:00 [nfsiod] ganglia 684 0.0 0.0 4260 1644 ? Ss 17:22 0:00 /usr/sbin/gmond root 723 0.0 0.1 52988 3168 ? Ss 17:22 0:00 php-fpm: master process (/etc/php5/fpm/main.conf) www-data 724 0.0 0.1 52972 2536 ? S 17:22 0:00 php-fpm: pool www www-data 725 0.0 0.1 52972 2540 ? S 17:22 0:00 php-fpm: pool www www-data 726 0.0 0.1 52972 2540 ? S 17:22 0:00 php-fpm: pool www www-data 727 0.0 0.1 52972 2540 ? S 17:22 0:00 php-fpm: pool www www-data 728 0.0 0.1 52972 2540 ? S 17:22 0:00 php-fpm: pool www www-data 729 0.0 0.1 52972 2540 ? S 17:22 0:00 php-fpm: pool www www-data 730 0.0 0.1 52972 2540 ? S 17:22 0:00 php-fpm: pool www www-data 731 0.0 0.1 52972 2540 ? S 17:22 0:00 php-fpm: pool www www-data 732 0.0 0.1 52972 2540 ? S 17:22 0:00 php-fpm: pool www www-data 733 0.0 0.1 52972 2652 ? S 17:22 0:00 php-fpm: pool www www-data 734 0.0 0.1 52972 2652 ? S 17:22 0:00 php-fpm: pool www www-data 735 0.0 0.1 52972 2652 ? S 17:22 0:00 php-fpm: pool www www-data 736 0.0 0.1 52972 2652 ? S 17:22 0:00 php-fpm: pool www www-data 737 0.0 0.1 52972 2652 ? S 17:22 0:00 php-fpm: pool www www-data 738 0.0 0.1 52972 2652 ? S 17:22 0:00 php-fpm: pool www www-data 739 0.0 0.1 52972 2652 ? S 17:22 0:00 php-fpm: pool www www-data 740 0.0 0.1 52972 2652 ? S 17:22 0:00 php-fpm: pool www www-data 741 0.0 0.1 52972 2652 ? S 17:22 0:00 php-fpm: pool www www-data 742 0.0 0.1 52972 2652 ? S 17:22 0:00 php-fpm: pool www www-data 743 0.0 0.1 52972 2652 ? S 17:22 0:00 php-fpm: pool www www-data 744 0.0 0.1 52972 2652 ? S 17:22 0:00 php-fpm: pool www www-data 746 0.0 0.1 52972 2652 ? S 17:22 0:00 php-fpm: pool www www-data 747 0.0 0.1 52972 2652 ? S 17:22 0:00 php-fpm: pool www www-data 748 0.0 0.1 52972 2652 ? S 17:22 0:00 php-fpm: pool www www-data 750 0.0 0.1 52972 2652 ? S 17:22 0:00 php-fpm: pool www www-data 751 0.0 0.1 52972 2652 ? S 17:22 0:00 php-fpm: pool www www-data 753 0.0 0.1 52972 2652 ? S 17:22 0:00 php-fpm: pool www www-data 754 0.0 0.1 52972 2652 ? S 17:22 0:00 php-fpm: pool www www-data 755 0.0 0.1 52972 2652 ? S 17:22 0:00 php-fpm: pool www www-data 756 0.0 0.1 52972 2656 ? S 17:22 0:00 php-fpm: pool www www-data 757 0.0 0.1 52972 2656 ? S 17:22 0:00 php-fpm: pool www www-data 758 0.0 0.1 52972 2656 ? S 17:22 0:00 php-fpm: pool www www-data 759 0.0 0.1 52972 2656 ? S 17:22 0:00 php-fpm: pool www www-data 760 0.0 0.1 52972 2656 ? S 17:22 0:00 php-fpm: pool www www-data 761 0.0 0.1 52972 2656 ? S 17:22 0:00 php-fpm: pool www www-data 762 0.0 0.1 52972 2656 ? S 17:22 0:00 php-fpm: pool www www-data 763 0.0 0.1 52972 2656 ? S 17:22 0:00 php-fpm: pool www www-data 764 0.0 0.1 52972 2656 ? S 17:22 0:00 php-fpm: pool www www-data 765 0.0 0.1 52972 2656 ? S 17:22 0:00 php-fpm: pool www www-data 766 0.0 0.1 52972 2656 ? S 17:22 0:00 php-fpm: pool www root 1240 0.0 0.1 8848 3116 ? Ss 17:31 0:00 sshd: ubuntu [priv] ubuntu 1334 0.0 0.0 8848 1368 ? S 17:31 0:00 sshd: ubuntu@pts/1 ubuntu 1335 0.5 0.3 9016 6048 pts/1 Ss 17:31 0:01 -bash root 1373 0.0 0.0 1904 512 ? Ss 17:31 0:00 /bin/sh /etc/init.d/rc 0 root 1375 0.0 0.0 3000 1304 ? S 17:31 0:00 /bin/bash /etc/rc0.d/K01unregister stop root 1394 0.0 0.1 5144 1952 ? S 17:31 0:00 ssh -i /XXXX [email protected] sudo sh -c 'sed -e "/10.2.22.35 fb?"-//W3C//DTD??lang="e ubuntu 1400 0.0 0.0 4280 1088 pts/1 R+ 17:34 0:00 ps aux

last VMS console output after two calls to halt -p (in wierd 50% cpu hung state)

init: tty2 main process (679) killed by TERM signal init: tty3 main process (680) killed by TERM signal init: tty6 main process (682) killed by TERM signal init: hwclock-save main process (881) terminated with status 1 init: Disconnected from system bus init: dbus main process (618) killed by TERM signal hostname: the specified hostname is invalid [ 52.346590] System halted.

Thanks!

Improve this question
11
  • I am not using a public AMI, however the AMI that I am using was at one time based on a public one. Is there a way to trace it's origin? Commented Dec 21, 2011 at 3:14
  • Poweroff does still work without issue on Amazon instances - including Ubuntu (just used it 2 days ago). I might suggest checking if you have enabled termination protection (although, it does normally display a message). Trying poweroff without the flags may help to reveal any errors that are logged. Are you able to reboot? (Also, any success with calling shutdown directly)? Commented Dec 21, 2011 at 12:43
  • Checked and termination protection is off. Can you clarify what you mean by "shutdown directly"? Just "sudo shutdown now"? Commented Dec 21, 2011 at 16:12
  • Yes, that is what I meant - I believe poweroff calls halt which is a wrapper to shutdown. (For some reason this reminds me of a disk problem - full disk, drive not unmounting, etc - can you reboot from SSH?). Anything in logs after the first failed poweroff (dmesg, messages?) Commented Dec 21, 2011 at 16:45
  • 1
    Mike: Did you ever track down what process was hanging the shutdown? Commented Jan 2, 2012 at 8:35

3 Answers 3

Reset to default
5

By default, when you shutdown/halt an EBS boot instance it is simply moved to the "stopped" state. This leave the instance available for restarting. You are no longer paying for instance hours, but you will continue paying a minimal charge for the EBS volume(s).

You can change this behavior so that a shutdown/halt will terminate the instance by setting the instance-initiated-shutdown-behavior to "terminate".

I've written a bit about this here:

Three Ways to Protect EC2 Instances from Accidental Termination and Loss of Data
http://alestic.com/2010/01/ec2-instance-locking

You'll want to run the instance with the option:

--instance-initiated-shutdown-behavior terminate

With this flag set, either of the following commands will terminate the EC2 instance:

sudo shutdown -h now # or sudo halt

You can also "Change Shutdown Behavior" through the AWS console for running instances (right mouse click).

Improve this answer
1
  • 3
    Actually the instances aren't even getting to the "stopped" state. They are left in a "started" state but cannot be connected to and the CPUs are pegged at 50%. Already am setting the terminate shutdown behavior and have tried both "sudo shutdown -h now" and "sudo halt" Commented Dec 22, 2011 at 0:31
3

Look into ec2stop / ec2-stop-instances commands that are a part of the AWS tools

Improve this answer
3
  • Those seem to be for stopping a remote instance and require require a Private Key / etc. I want a way to be able to stop the local EC2 instance. Commented Dec 21, 2011 at 3:38
  • Well, yes and no - it is the interface with AWS via their API to turn off instance - it's somewhat inconvenient that each instance isn't automatically setup to do it for itself, but you can set it up to behave like that by using the appropriate environment variables. Commented Dec 21, 2011 at 5:55
  • can you expand a bit on how I could use that locally on an instance? My main concern is that I don't want to put any .pem or AWS Identity pairs on my AMIs. Commented Dec 21, 2011 at 16:03
-1

Have you tried:

echo o > /proc/sysrq-trigger

Depending on the kernel that should cause an immediate poweroff. There is no graceful shutdown in this case but if you dont need the image again and start from a fresh one each time, that wont matter.

Some refernce material here: http://en.wikipedia.org/wiki/Magic_SysRq_key#Command_line_access

Improve this answer
9
  • This gives me a "Permission denied" error: $ sudo echo o > /proc/sysrq-trigger -bash: /proc/sysrq-trigger: Permission denied Commented Dec 21, 2011 at 16:23
  • You may need to first enable sysrq: sudo echo 1 > /proc/sys/kernel/sysrq Commented Dec 21, 2011 at 16:46
  • Same permission denied Commented Dec 21, 2011 at 21:35
  • 2
    Mike: The "Permission denied" was because the redirect (>) happens before the "sudo". If this command did what you wanted, the correct syntax on Ubuntu would be: echo o | sudo tee /proc/sysrq-trigger Commented Dec 22, 2011 at 0:08
  • 1
    How do you know cpu if system is unreachable? Commented Dec 22, 2011 at 8:31

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.