Create a server image with Packer

Packer is a free, open source application from Hashicorp. It can generate a server image based on an existing one, and configure it for your special needs. You can use the generated image when you launch a server instance in the cloud or on your local workstation.

Install Packer

Generate the server image with Packer

  1. Open a Bash window,
  2. Navigate to the folder of the Packer JSON script,
  3. Execute the following command. Get the AWS access key and secret key from the ~/.aws/credentials file on your Macintosh or Linux workstation. On Windows, the file is at C:\Users\YOUR_USER_NAME\.aws\credentials.
    packer build -var 'aws_access_key=MY_ACCESS_KEY' -var 'aws_secret_key=MY_SECRET_KEY' ./MY_PACKER_SCRIPT.json
  4. The command window will display the ID of the generated image, or you can find it by name in the EC2 section of the AWS console under AMIs.

Share the generated server image with other cloud accounts

If you work in multiple cloud accounts you need to share the generated server image with other accounts

AWS

  1. Log into the AWS account you have used to generate the server image,
  2. On the left side of the EC2 section select AMI and find the new image by name of ID,
  3. On the Permissions tab click the Edit button,
  4. Make sure the Private radio button is selected if you don’t want to share the image publicly,
  5. Enter the account number of the account you want to share the image with,
  6. Check the Add “create volume” permissions… checkbox,
  7. Click the Add Permission button,
  8. When you have added all accounts to share with, click the Save button.

 

Prevent the auto-termination of stranded instances in RighScale

When you launch an instance with RightScale Self Service, and the Chef cookbook execution fails, the instance goes into “stranded” mode. By default RightScale Self Service terminates the stranded instances, so there is no way to remote into them and read log files to find the cause of the problem.

To keep stranded instances running in RightScale

  1. Find the booting instance in Cloud Management and click the instance name,
  2. Click the lock icon on the top of the screen

RightScale Self Service cannot terminate locked instances. To terminate the instance after the troubleshooting process, unlock the instance and terminate the instance by hand.

Upgrade a server on an AWS EC2 instance with minimum risk and downtime

When you need to upgrade an application on an AWS EC2 instance with minimum downtime, there are many options.

Upgrade an EC2 instance

  1. Stop the application, so users don’t make more changes,
  2. Create a backup of the database (snapshot of the RDS instance),
  3. Create a backup of the server (backup image of the EC2 instance),
  4. Upgrade the application on the server,
  5. Start the application,
  6. Test the new version of the application.

In case the upgrade fails, or the post-upgrade test fails

  1. Stop the application,
  2. If you use RightScale to launch servers:
    1. stop the RightLink service on the failed server to prevent RightScale auto terminating the restored server
      (When RightScale finds a new identical server instance, it automatically shuts it down to avoid multiple instances with the same identifier.)
    2. Terminate the failed server
  3. Stop the failed server,
  4. Restore the database from the pre-upgrade backup with a new name,
  5. Restore the server instance from the pre-upgrade backup,
  6. Start the restored server,
  7. Change the database address in the application’s config file to point to the restored database,
  8. Start the application,
  9. Test the restored version of the application.

To make the backup image of the AWS EC2 instance

  1. Open the AWS console and navigate to EC2, Running instances
  2. Find the server instance you want to backup
  3. Right-click the row of the instance and select Image, Create Image

  4. Name the image and click the Create Image button
  5. Save the image Id from the popup

Test your RightScale script without creating unnecessary revisions

There are certain functions in RightScale scripts (RightScripts) that you cannot test without launching a new instance. To test your script:

  1. Edit the Head revision of the RightScale script and save it,
  2. Attach the Head revision of the RightScale script to the Head revision of the RightScale Server Template,
  3. In your Self Service CAT file reference revision 0 (zero) of the Server Template (Head revision),
    ( “server_template_revision”  => “0”, )
  4. Upload the Self Service CAT file to Self Service, but do not publish it,
  5. Launch a server instance from Self Service using the RightScale Self-Service Designer page, not the Catalog.

If the script has some errors, modify the Head revision of the RightScale script and launch a new server instance using the RightScale Self-Service Designer page.

When the script works perfectly:

  1. Commit the RightScale script, creating a new RightScale script revision,
  2. Attach the committed revision of the RightScale script to the Server Template’s Head revision,
  3. Commit the Server Template, creating a new Server Template revision,
  4. Publish the new revision of the Server Template,
  5. Reference the new revision of the Server Template in your Self Service CAT file.

Find the AWS AMI that a RightScale server template is using

When you use RightScale to launch servers in the cloud, you want to use the same base image to test your Chef cookbooks in Test Kitchen.

Packer by Hashicorp is a utility to create custom server images based on cloud images. You supply the image ID and other configuration parameters to create a new custom image.

To find the AMI ID of the base image of the RightScale Server Template

  1. Open the RightScale user interface in your browser,
  2. Select the RightScale account where the server template was created,
  3. In RightScale Cloud Management select  Design / ServerTemplates,
  4. Select the server template,
  5. Select Images,
  6. Select the MultiCloud image,
  7. Select Clouds,
  8. Select the image in the region you are working in,
  9. The Resource UID is the AMI ID of the image.

Migrate databases to and from AWS RDS instances

Amazon Web Services do not provide tools to migrate data to, from and between AWS RDS instances.

There is a free utility that can do the job.

SQL Server Management Studio (SSMS)

Install Microsoft SQL Server Management Studio (SSMS), because the utility uses libraries from it.

  1. Download the installer for free from https://msdn.microsoft.com/en-us/library/mt238290.aspx

SQL Azure Migration Wizard

Installation

  1. Download the SQL Azure Migration Wizard from http://sqlazuremw.codeplex.com/
  2. Unzip the downloaded ZIP file into a folder on your hard drive
  3. Create a shortcut in your Start Menu for SQLAzureMW.exe

Migrate a database

  1. Start the SQL Azure Migration Wizard by executing the SQLAzureMW.exe file
  2. Select Database in the Analyze / Migrate section and click Next

Connect to the source database

  1. On the Connect to Server page
    1. Enter the address of the database server
    2. Select SQL Server Authentication
    3. Enter the database owner login name and password
    4. If you keep Master DB selected, the next screen will show the list of databases to select from
    5. To save the username and password select Save Login Information
    6. Click Connect

  2. Select the database to connect to and click Next
  3. Leave Script all database objects selected and click Next
  4. On the Script Wizard Summary page click Next
  5. Click Yes to start the read process
  6. On the Results Summary page select Auto Scroll Display to see the scrolling results
  7. When the read is done click Next

Connect to the target database

  1. Enter the connection properties of the target database server
  2. Select the target database
    1. To migrate the data into a new database click the Create Database button
    2. On mirrored RDS instances we cannot drop and recreate databases. To migrate data into an existing database
      1. Use SQL Management Studio to drop the existing tables to avoid key conflicts
      2. Select the target database on this page
  3. Click Next
  4. Click Yes to start the write process
  5. When the write process is done click Exit

 

Launch Windows instances locally with Chef Test Kitchen

Most Linux distributions are free, and do not require product keys to launch them.

The steps below are based on the great article at http://kitchen.ci/blog/test-kitchen-windows-test-flight-with-vagrant/

I have summarized the steps below to create a free Virtual Box Windows Server 2012R2 image on your workstation, so Test Kitchen can use Vagrant and Virtual Box to launch Windows instances and test cookbooks locally fast and free.


Install the Vagrant WinRm plugin

vagrant plugin install vagrant-winrm

Get BoxCutter

Create the Windows image with BoxCutter. This process will take 30 minutes or more to fully configure the Windows image. The download step may time out on slower VPN connections. In that case disconnect the VPN connection and try it again without it. In case of any error, just execute the last command again and if it can, the process will continue from the point of error.

cd ~/
mkdir boxcutter
cd boxcutter
git clone https://github.com/boxcutter/windows.git
cd windows

Modify the BoxCutter JSON file

Change the ~/boxcutter/windows/eval-win2012r2-standard.json file to avoid the error message:

virtualbox-iso: Removing floppy drive…
==> virtualbox-iso: Error removing floppy controller: VBoxManage error: VBoxManage: error: The machine ‘eval-win2012r2-standard’ is already locked for a session (or being unlocked)
==> virtualbox-iso: VBoxManage: error: Details: code VBOX_E_INVALID_OBJECT_STATE (0x80bb0007), component MachineWrap, interface IMachine, callee nsISupports
==> virtualbox-iso: VBoxManage: error: Context: “LockMachine(a->session, LockType_Write)” at line 1038 of file VBoxManageStorageController.cpp

  • Change “headless”: “false”, to “headless”: “true”,
  • Add under “headless”: “true”,
"shutdown_timeout": "60m",
"post_shutdown_delay": "120s",

 

  • Add under every occurrence of “headless”: “{{ user `headless` }}”,
"shutdown_timeout": "{{ user `shutdown_timeout` }}",
"post_shutdown_delay": "{{ user `post_shutdown_delay` }}",

Create the virtual machine

make virtualbox/eval-win2012r2-standard

Add the image to Vagrant

vagrant box add windows-2012r2 ./box/virtualbox/eval-win2012r2-standard-nocm-1.0.4.box

Test the virtual machine

Your .kitchen.yml file should look like this

---
driver:
  name: vagrant

provisioner:
  name: chef_zero

platforms:
  - name: windows-2012r2

suites:
  - name: MY_SUITE_NAME
  run_list:
  - recipe[MY_COOKBOOK::MY_RECIPE]

 

You may need to execute the kitchen converge commands as sudo to be able to launch the Windows instance.

Add an RSA private key to a Chef Data Bag

To send RSA private keys to instances, store them in encrypted data bags. The data bag item is a JSON file that contains keys and values.

To place the multi-line RSA key into the value part of the JSON file, we need to replace the new line characters with “\n”

On a Windows  workstation we can use Notepad++ to do the replacement:

  • Open the RSA key file in Notepad++,
  • In the Search menu select Replace…,
  • Select Extended mode in the Search Mode section,
  • Enter \r\n to the Find what text box
  • Enter * (star) to the Replace with text box 
  • Press the Replace All button

rsa-key-in-data-bag-01

  • Select Normal in the Search Mode section,
  • Enter *  (star) to the Find what text box
  • Enter \n to the Replace with text box
  • Press the Replace All button

rsa-key-in-data-bag-02

You can place the single line key into any encrypted Data Bag file. See Data Bags on Data Bag encryption.

HTTP Request Returned 409 Conflict: Client already exists

The Chef server maintains the list of registered nodes and clients in its database.  When you launch a new instance with Chef you may encounter the following error message:

*** Input CHEF_CLIENT_NODE_NAME is undefined, using: IP-0AFE6965
...
*** Starting chef-client
*** Finished chef-client
Printing Log
...
[...] INFO: Client key C:\chef\client.pem is not present - registering
[...] INFO: HTTP Request Returned 409 Conflict: Client already exists
[...] INFO: HTTP Request Returned 403 Forbidden: error
[...] ERROR: Running exception handlers
[...] ERROR: Exception handlers complete
[...] FATAL: Stacktrace dumped to C:/chef/cache/chef-stacktrace.out
[...] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[...] FATAL: Net::HTTPServerException: 403 "Forbidden"

In this case the CHEF_CLIENT_NODE_NAME already exists in the client database. You can search for it from a Bash window.

To find the node in the Chef server database

knife search node '*:IP-0AFE6965'

To find the client in the Chef server database

knife search client '*:IP-0AFE6965'

If any of them found, those are leftovers from a previous launch. AWS reuse the IDs, so you have to remove the nodes from the Chef server database when you terminate the instances. To delete the unnecessary entries use the following commands:

knife node delete 'IP-0AFE6965'
knife client delete 'IP-0AFE6965'

To regularly remove the terminated instances from the Chef server database you can also use the Chef web interface.

  • On the Nodes tab enter the IP address of the instance into the search box and hit Enter
  • In the Actions column click the down arrow next to the instance and select Delete

DevOps Engineering part 7. – Launching production instances in the cloud

In the previous parts of this tutorial we have launched instances (servers) in the cloud, but those were created by Test Kitchen, running on our workstation. Those instances are as good as they can be, but the cookbook did not reside on the Chef server.

To launch pre-production and production instances in the cloud, first we need to upload the cookbook to the Chef server. In Beginner’s Guide to DevOps Engineering part 4. Connect to the Chef server we have already connected our workstation to the Chef server.

Upload the cookbook to the Chef server

To make your cookbook available for all the servers of the organization we will upload the cookbook to the Chef server.

  • In the folder of the cookbook open a Bash window
  • Execute the command to upload the cookbook to the Chef server
    knife cookbook upload COOKBOOK_NAME --freeze

    Don’t forget to add the –freeze option to the end to protect the version of the cookbook from accidental changes.

Launch the server in the Cloud

There are many ways to launch servers in a scripted way, we are going to review a few options. One is to use RightScale, a Cloud Management Platform, that can manage servers in multiple cloud environments.

RightScale

Create a new Deployment

  • Using your web browser log into the RightScale Cloud Management user interface,
  • In the Manage menu select Deployment and click the New button,
  • Enter the name of the new deployment,
  • To add the new deployment to your bookmarks,
    • On the left side in the Bookmarks section click the Add link,
    • If necessary shorten the name of the bookmark and click OK,
    • To reorder the bookmarks
      • Click the Edit link and drag the bookmark to the desired position,
      • Click the Done link to save your changes.

Create the CAT file

CAT files are Ruby like scripts that describe the configuration of the instance. It contain the name of the deployment the server should be placed it, the cloud attributes to assign security groups, regions, availability zones, set the drive size, specify security keys. The easiest way to configure a server is to copy an existing RightScale Self Service CAT file and update it with the new values.

If this is the first CAT file in your organization, get help from RightScale support and get a sample file from them.

If your organization already has a library of CAT files

  • Make a copy of an existing CAT file
  • Update the values to customize it for your server
    • Application name
    • Chef runlist
    • Security group
    • Deployment name
    • Load balancer name
    • Instance type

 

Upload the CAT file to RightScale

  • Open the RightScale Self Service user interface,
  • Select Designer on the left,
  • Click the Upload CAT button and select the CAT file,
  • Click the Upload button in the lower right corner,
  • In the green popup in the upper right corner click the Publish to catalog link,
  • In the lower right corner click the Yes, Publish button.

Launch the server in RightScale Self Service

  • In the RightScale Self Service interface select Catalog on the left side,
  • Select the catalog item you want to launch,
  • Enter a name for the instance to be able to identify it later and select the instance in the dropdown list,
  • Click the Launch CloudApp button.

To monitor the server launch

  • In your web browser open the RightScale user interface,
  • In the design menu select Deployments,
  • Select the deployment of the instance,
  • Click the instance to see in formation on it.

Back:

to the Tutorials page