Thursday, April 22, 2010

Upgrading an EBS Instance

Update 20110323: If you are reading this article, you almost certainly should be reading my Migrating to pv-grub kernels for kernel upgrades. The process described here will still work for 10.04 images, but the process described there is ultimately much easier. If you are using 10.10, or 11.04 images, you do not need to do anything, simply 'apt-get update && apt-get dist-upgrade && reboot' to get a new kernel.

For the majority of the existence of EC2 there was no way to change the kernel that an instance was using. With the addition of EBS instances, that changed. I wanted to explain how you can take advantage of that EBS feature by upgrading a Ubuntu 10.04 LTS (Lucid Lynx) instance launched from a Beta-2 AMI to the Release Candidate. This same basic process should also allow you to upgrade across a release, perhaps from a 9.04 Alestic instance to Ubuntu 10.04 LTS.

In reality, if you're hoping to upgrade you're kernel and EBS instance, its because you already have one running and need to upgrade. But for the sake of this excercise, we'll launch a new instance based on the Beta-2 image in the us-east-1 region and then connect to it.

$ ec2-run-instances --key mykey ami-4be50b22
# wait a bit
$ ec2-describe-instances | awk '-F\t' '$1 == "INSTANCE" { print $4 }'
ec2-184-73-101-171.compute-1.amazonaws.com
$ ssh -i mykey.pem ubuntu@ec2-184-73-101-171.compute-1.amazonaws.com

Now, on the instance we'll go ahead and do the upgrade. Whenever I'm working on ec2, I like to use GNU screen to protect against lost network.
% screen -S upgrade

This is the same basic process as upgrading any Ubuntu system. First update and then upgrade. Here, I've run '--dry-run' to point out that there would be kernel upgrades.

Also, the ec2 images suffer from a bug where grub will be installed and prompt you for some information even though its of no value. Because I know what those prompts will be, I'm go ahead and set them here so you're not interrupted during the dist-upgrade.

% sudo apt-get update
% sudo apt-get dist-upgrade --dry-run | grep "linux.*ec2"
  libparted0debian1 linux-image-2.6.32-21-virtual linux-image-2.6.32-305-ec2
  libpolkit-gobject-1-0 libpython2.6 libss2 libudev0 linux-ec2 linux-firmware
  linux-image-ec2 linux-image-virtual linux-virtual locales module-init-tools
Inst linux-image-2.6.32-305-ec2 (2.6.32-305.9 Ubuntu:10.04/lucid)
Inst linux-ec2 [2.6.32.304.5] (2.6.32.305.6 Ubuntu:10.04/lucid) []
Inst linux-image-ec2 [2.6.32.304.5] (2.6.32.305.6 Ubuntu:10.04/lucid)
Conf linux-image-2.6.32-305-ec2 (2.6.32-305.9 Ubuntu:10.04/lucid)
Conf linux-image-ec2 (2.6.32.305.6 Ubuntu:10.04/lucid)
Conf linux-ec2 (2.6.32.305.6 Ubuntu:10.04/lucid)

% echo grub-pc grub2/linux_cmdline string | sudo debconf-set-selections
% echo grub-pc grub-pc/install_devices_empty boolean true | sudo debconf-set-selections

% sudo apt-get dist-upgrade
..
94 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
Need to get 84.1MB of archives.
..

Above, If you were upgrading from a previous release to 10.04, then you would use 'do-release-upgrade' from the 'update-manager-core' package.

At this point we've got 2 kernels installed, the new one and the old one. Unsurprisingly, we're booted into the old one.

% dpkg-query --show | grep "linux.*ec2"
linux-ec2       2.6.32.305.6
linux-image-2.6.32-304-ec2      2.6.32-304.8
linux-image-2.6.32-305-ec2      2.6.32-305.9
linux-image-ec2 2.6.32.305.6
% uname -r
2.6.32-304-ec2

Above, we can see that the 2.6.32-305.9 version of the kernel is the newest one. Its installed locally, but to boot it we have to find the aki of the version that is published by Ubuntu to ec2. The Ubuntu kernels are registered in ec2 such that you can correlate the dpkg version to the registered aki. We're going to query all the images, save that output to a file and then search for results that are owned by the Canonical user, and match our version string and arch.

$ owner=099720109477; # this is the canonical user's id
$ ver=2.6.32-305.9; arch=i386
$ ec2-describe-images --all > /tmp/images.list
$ awk '-F\t' '$4 == o && $3 ~ v && $8 == a { print $2, $3 }' \
   a=${arch} "o=${owner}" "v=${ver}" /tmp/images.list
aki-1f02ec76 099720109477/ubuntu-kernels-milestone/ubuntu-lucid-i386-linux-image-2.6.32-305-ec2-v-2.6.32-305.9-kernel
aki-d324caba 099720109477/ubuntu-kernels-testing/ubuntu-lucid-i386-linux-image-2.6.32-305-ec2-v-2.6.32-305.9-kernel

That shows us that we have 2 kernels available matching that explicit version. One is "testing", and one is "milestone". These are actually the same thing. We label the kernels differently so the user easily knows what is testing and what is "released". The released kernel version will be labeled with 'ubuntu-kernels'. The RC kernel version gets labeled "ubuntu-kernels-milestone".

In order to change the kernel, we have to stop the instance, modify the 'kernel' attribute, and then start it up again.
$ ec2-stop-instances i-23453048
$ ec2-modify-instance-attribute --kernel aki-1f02ec76
kernel   i-23453048  aki-1f02ec76

$ ec2-start-instances i-23453048
# wait a bit
$ ec2-describe-instances | awk '-F\t' '$1 == "INSTANCE" { print $4 }'
ec2-184-73-116-205.compute-1.amazonaws.com

So, in theory, we should have booted into our nice and shiny-new kernel. Lets test that theory:
$ ssh ubuntu@ec2-184-73-116-205.compute-1.amazonaws.com 'uname -r'
2.6.32-305-ec2

There you have it! This process can be applied to upgrading the RC to Release (which likely won't have a kernel change), or, eventually to upgrading your 10.04 LTS instance to a Maverick one.

21 comments:

  1. This comment has been removed by the author.

    ReplyDelete
  2. Thank you for this article. :)

    ReplyDelete
  3. Would you happen to have any idea why an updated instance would not be accessible after starting it back up or rebooting?
    (http://developer.amazonwebservices.com/connect/thread.jspa?threadID=46483&tstart=0)

    ReplyDelete
  4. Hi! Very usefull article. Thank you so much!

    Regarding the issue Enis and myself found, I managed to overcome it commenting the line that says /dev/sdb /mnt in fstab as described in some link in the forums Enis posted.

    Hope it helps others with the same problem.

    ReplyDelete
  5. Thanks for your clear explanation, but after following through i get stuck at this point:
    ec2-modify-instance-attribute i-7844da0f --kernel aki-1f02ec76 --region eu-west-1

    I get this error:
    Client.InvalidAMIID.NotFound: The AMI ID 'ami-1f02ec76' does not exist

    Any idea?

    Regards Ruben

    ReplyDelete
  6. Once again, I found the answer on your site. I've been very confused by the grub-pc part of my apt-get upgrade on ec2. I had put apt-get remove in my setup script, but I replaced that with the debconf-set-selections line like what you used here, since it seems much safer.

    If I understand correctly from this article, grub on ec2 instances does absolutely nothing, so I could safely remove it. Is that correct? Or could removing grub break something else if I need to reboot the instance?

    Thanks for writing about this stuff. I'll be back for more!

    ReplyDelete
  7. @mnebuerquo,
    grub on ec2 does nothing. That is true for both Lucid and Maverick (10.10). However, in 10.10 we changed a few things.
    a.) grub-pc still does nothing, but you won't be annoyed by it
    b.) we've taken advantage of the amazon 'pv-grub' feature [1] so that we can reboot into an upgraded kernel, making this post mostly obsolete. The package inside the images that manage /boot/grub/menu.lst on ec2 is grub-legacy-ec2.

    --
    [1] http://aws.typepad.com/aws/2010/07/use-your-own-kernel-with-amazon-ec2.html

    ReplyDelete
  8. I was using the original 10.4 release, and started getting this message on login:

    A newer build of the Ubuntu lucid server image is available.
    It is named 'release' and has build serial '20100923'.

    I did a dist-upgrade and followed the above instructions to update the kernel and that all went well, but the message is still displayed at login. Any ideas why? Thanks.

    ReplyDelete
  9. @Tom,
    You can safely ignore that message now. You've definitely applied all available upgrades. The message comes from /etc/cron.d/cloudinit-updates . It is simply checking http://uec-images.ubuntu.com/query and comparing that to information in /etc/cloud/build.info.
    I've opened bug to address fixing such issues in the future.
    https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/653220

    ReplyDelete
  10. Scott - I also upgraded from a previous version of 10.4 like Tom and also get the message. However, if I do the dpkg-query that you suggested, I get back:

    linux-ec2 2.6.32.308.9
    linux-image-2.6.32-305-ec2 2.6.32-305.9
    linux-image-2.6.32-308-ec2 2.6.32-308.16
    linux-image-ec2 2.6.32.308.9

    I'm guessing this indicates I'm still running the older version...?

    ReplyDelete
  11. Gary,
    kernel binaries are not removed on dist-upgrade, so the fact that you have them doesn't tell you anything.
    To see what kernel you're running *now*, use 'uname -r'.

    Scott

    ReplyDelete
  12. @Tom,
    Remove /var/lib/cloud/data/available.build on the server and that message will go away.

    ReplyDelete
  13. Scott,

    I'm running a 10.04 AMI from Canonical (EBS, small instance). If I understand correctly, a simple "sudo apt-get update && sudo apt-get dist-upgrade" will get me a newer kernel (when available), but it will not be used at boot without following your instructions above. I also understand that this is not the case with 10.10?

    Since I plan to stick with LTS, I'd like to know whether it's recommended to use dist-upgrade and always try to stick with the newest kernel (for the 10.04 release), or if I'm perfectly safe (or better off) just using "upgrade" instead of "dist-upgrade" and allowing newer kernels to be held back?

    Or, is this just one of those "it depends" situations, where I should be reviewing what changes with each new kernel?

    Thanks!

    ReplyDelete
  14. @Jamie,
    right. On 10.04 images, apt-get update && apt-get dist-upgrade will get you a new kernel, and leave the old in place. On reboot, you'll boot into the old.
    On 10.10, 'apt-get update && apt-get dist-upgrade && reboot' will get you into the new kernel.
    On 10.04, what I would recommend at the moment is moving to the pv-grub kernels. If you use one of the most recent images (20101228), then you can launch the new instance with a pv-grub kernel and everything will "just work".
    If you're on an old ebs-root image, you'll need to
    * sudo apt-get install cloud-init-legacy-ec2 && sudo apt-get dist-upgrade
    * sudo poweroff
    * ec2-modify-instance-attributes --kernel
    * ec2-start-instances

    Then, from there out, you should be able to apt-get dist-upgrade and reboot into a new kernel.
    In case the start-instances failed, set it back to the original aki and troubleshoot.

    I realize it might seem like a big change for an LTS. However, it is a HUGE feature that is almost a necessity if you have a long lived instance.

    ReplyDelete
  15. @Scott,
    Thanks a bunch for the quick response. Once I launch a new 10.04 image with the pv-grub kernel, you feel this is a long-term solution for 10.04? Meaning I can happily use dist-upgrade on my 10.04 instances and be safe and stable right on through the next LTS release? Thanks again.

    ReplyDelete
  16. @Jamie,
    Yes, I really suggest using the 10.04 images with pv-grub. I just (yesterday) released updated images that use pv-grub by default. pv-grub represents the best possible upgrade path we have for 10.04 LTS -> 12.04 LTS.

    ReplyDelete
  17. @Scott,
    Fantastic, I have an opportunity to just use the new images you just released (2011-02-01). Thanks!

    ReplyDelete
  18. Hi Scott,

    can I upgrade an existing 10.04 with 2.6.32-312-ec2 to pv-grub kernels?

    With standard sources.list I do not see anything similar to 'pv-grub' in packages.

    ReplyDelete
  19. @chronos,
    Please see http://ubuntu-smoser.blogspot.com/2011/02/migrating-to-pv-grub-kernels-for-kernel.html . It should explain what you need to do.

    ReplyDelete
  20. Just a comment for those on 10.10 - they use '-virtual' instead of '-ec2' now, so change the greps above.

    Also, as stated in a comment, if your on 10.10 simply run:
    apt-get update && apt-get dist-upgrade && reboot

    and you should be golden

    ReplyDelete
  21. @ryan, thanks for pointing this out. Also, I've update the post to include a blog entry indicating how you can use the pv-grub kernels in 10.04, so you'll only have to go through the stop/modify-instance-attribute/start sequence one time. After doing so, you can 'apt-get update && apt-get dist-upgrade && reboot' your way to new kernels in 10.04 images as well as 10.10 (and beyond).

    ReplyDelete