Create a server image with Packer

Packer is a free, open source application from Hashicorp. It can generate a server image based on an existing one, and configure it for your special needs. You can use the generated image when you launch a server instance in the cloud or on your local workstation.

Install Packer

Generate the server image with Packer

  1. Open a Bash window,
  2. Navigate to the folder of the Packer JSON script,
  3. Execute the following command. Get the AWS access key and secret key from the ~/.aws/credentials file on your Macintosh or Linux workstation. On Windows, the file is at C:\Users\YOUR_USER_NAME\.aws\credentials.
    packer build -var 'aws_access_key=MY_ACCESS_KEY' -var 'aws_secret_key=MY_SECRET_KEY' ./MY_PACKER_SCRIPT.json
  4. The command window will display the ID of the generated image, or you can find it by name in the EC2 section of the AWS console under AMIs.

Share the generated server image with other cloud accounts

If you work in multiple cloud accounts you need to share the generated server image with other accounts

AWS

  1. Log into the AWS account you have used to generate the server image,
  2. On the left side of the EC2 section select AMI and find the new image by name of ID,
  3. On the Permissions tab click the Edit button,
  4. Make sure the Private radio button is selected if you don’t want to share the image publicly,
  5. Enter the account number of the account you want to share the image with,
  6. Check the Add “create volume” permissions… checkbox,
  7. Click the Add Permission button,
  8. When you have added all accounts to share with, click the Save button.

 

Prevent the auto-termination of stranded instances in RighScale

When you launch an instance with RightScale Self Service, and the Chef cookbook execution fails, the instance goes into “stranded” mode. By default RightScale Self Service terminates the stranded instances, so there is no way to remote into them and read log files to find the cause of the problem.

To keep stranded instances running in RightScale

  1. Find the booting instance in Cloud Management and click the instance name,
  2. Click the lock icon on the top of the screen

RightScale Self Service cannot terminate locked instances. To terminate the instance after the troubleshooting process, unlock the instance and terminate the instance by hand.

Getting started with InSpec

InSpec is an open-source testing framework to verify your infrastructure satisfies the design requirements.

In this article, we will learn to install and use InSpec with Chef.

Install InSpec

  1. Navigate to https://downloads.chef.io/inspec, and download the installer for the operating system of your workstation.
  2. Execute the downloaded installer.

Start to use InSpec

To use InSpec as the default integration testing tool in Chef Test Kitchen

  1. Open the .kitchen.yml file of the cookbook,
  2. Add the following lines to the file:
    verifier:
      name: inspec
  3. Add this to every suite, so InSpec will search for the test files in the test/recipes directory. Otherwise, the test file needs to be in the test/recipes/SUITE_NAME directory
     verifier:
       inspec_tests:
         - test/recipes
  4. Create a folder structure in the cookbook folder for the InSpec integration tests,
    test
    |--recipes
  5. Create an integration test for your recipe. Create a new file in the test/recipes folder and name it RECIPE_NAME_test.rb. For the default recipe call it default_test.rb,
  6. The following is a simple example of an InSpec integration test:
    # # encoding: utf-8
    
    # Inspec test for recipe my_cookbook::default
    
    # The Inspec reference, with examples and extensive documentation, can be
    # found at https://docs.chef.io/inspec_reference.html
    
    unless os.windows?
     describe user('root') do
     it { should exist }
     skip 'This is an example test, replace with your own test.'
     end
    end
    
    describe port(80) do
     it { should_not be_listening }
     skip 'This is an example test, replace with your own test.'
    end

As you can see, the syntax of InSpec is (intentionally) very similar to ServerSpec, that it replaces. It is very easy to convert existing ServerSpec integration tests to InSpec compliance tests.

Differences between ServerSpec and InSpec

ServerSpec “process”

changed from

 describe process('PROCESS_NAME') do
   it { should be_running }
 end

to

describe processes('PROCESS_NAME') do
  its('states') { should eq ['R<'] }
end

Undefined method or attribute error in a Chef recipe

There are multiple reasons Chef can display the following error message

NoMethodError
 -------------
 Undefined method or attribute `...' on `node'

 

There are multiple ways to reference an attribute in a Chef recipe:

node['ATTRIBUTE_NAME'] (the recommended way)
node["ATTRIBUTE_NAME"] 
node[:ATTRIBUTE_NAME]  ( use it only if the single or double quotes (' or ") would cause a problem in the expression)

node.ATTRIBUTE_NAME

To check if the attribute value is nil, use the following format:

if ( !node['ATTRIBUTE_NAME'].nil? )
   ...
end

If you use the node.ATTRIBUTE_NAME format and the value is nil Chef throws the above error message.

How to restore a Microsoft SQL database from backup with the Microsoft SQL Server Management Studio (MSSMS) user interface

There are multiple reasons to restore a database from backup. One of them can be disaster recovery, the other is to bring the production database to the developer machine. In both cases, the computer already has the old version of the database. In the new version of Microsoft SQL Server Management Studio (MSSMS) we cannot find the “Close connections” checkbox anymore, so we have to make sure all connections are closed, and we specify a unique database file name to restore the database to.

  1. On the developer machine close all instances of Visual Studio to close the open database connections,
  2. Close all Microsoft SQL Server Management Studio tabs that are connected to the database,
  3. Move the backup file to the C:\Temp folder. Microsoft SQL Server Management Studio cannot see it in your Downloads folder.
  4. Make sure your user account does not automatically connect to the database you want to restore:
    1. In Security, Logins right click your username and select Properties,
    2. Set the Default database to master
  5.  Disconnect from the database server:
    1.  Right-click the database server and select Disconnect,

  6. Connect to the database engine, but do not open the database you want to restore:
    1. Execute,
      USE Master
  7. Import the database from production with the user interface:
    1. Right-click the database and select Tasks, Restore, Files and Filegroups
    2. Select the From device radio button
    3. Click the button to select the backup file
    4. In the Select backup devices window click the Add button to select the file
    5. Navigate to the C:\Temp folder and select the database backup file
    6. Click the OK button
    7. In the Select the backup sets section select the Restore check box in the file row, and click Options on the left side,
    8. On the Options tab select the Overwrite existing database (WITH REPLACE) check box, and click the button to set the new unique database file name,
    9. Click OK to ignore the error message on the Locate Database Files – … message box
    10. Navigate to the current database file, click it, and append the date of the backup to the end of the file name to specify a unique data file name, and click OK,
    11. On the Restore Files and FileGroups window click the OK button to restore the database

Update the database to make it work at the new location

  1. On a development machine set the databases to Simple recovery mode, so the log files do not grow out of bound.
    ALTER DATABASE my_database SET RECOVERY SIMPLE;
  2. Delete and recreate the users in the restored database, because those have different internal IDs if the database was migrated from a different server
    USE my_database;
    GO
    spDropUser 'my_user', 'dbo'
    GO
    DROP SCHEMA my_user
    GO

    For some reason If I type DECLARE below, WordPress crashes. Please add the letter E to the end of the first word below.

    DECLAR @user_name varchar(50)
    SET @user_name = 'my_user'
    USE my_database; EXEC sp_grantdbaccess @user_name; EXEC sp_addrolemember @rolename = 'db_datareader', @membername = @user_name; EXEC sp_addrolemember @rolename = 'db_datawriter', @membername = @user_name; 
    GO

Error waiting for instance (i-…) to become ready: unexpected state ‘terminated’, wanted target ‘running’

When you launch a server instance with Terraform, sometimes the error message does not contain the underlying cause. When the cloud provider cannot complete the request, many times Terraform displays a generic error message:

Error waiting for instance (i-...) to become ready: unexpected state 'terminated', wanted target 'running'

To find the root cause of the error in AWS

  1. Log into the AWS console and navigate to the EC2 section,
  2. Search for the instance by the instance Id,
  3. You can find the error message at the bottom of the Description tab

In our specific case, it was Client.VolumeLimitExceeded: Volume limit exceeded

We had to increase the volume limit to be able to launch more large EC2 instances.

Upgrade a server on an AWS EC2 instance with minimum risk and downtime

When you need to upgrade an application on an AWS EC2 instance with minimum downtime, there are many options.

Upgrade an EC2 instance

  1. Stop the application, so users don’t make more changes,
  2. Create a backup of the database (snapshot of the RDS instance),
  3. Create a backup of the server (backup image of the EC2 instance),
  4. Upgrade the application on the server,
  5. Start the application,
  6. Test the new version of the application.

In case the upgrade fails, or the post-upgrade test fails

  1. Stop the application,
  2. If you use RightScale to launch servers:
    1. stop the RightLink service on the failed server to prevent RightScale auto terminating the restored server
      (When RightScale finds a new identical server instance, it automatically shuts it down to avoid multiple instances with the same identifier.)
    2. Terminate the failed server
  3. Stop the failed server,
  4. Restore the database from the pre-upgrade backup with a new name,
  5. Restore the server instance from the pre-upgrade backup,
  6. Start the restored server,
  7. Change the database address in the application’s config file to point to the restored database,
  8. Start the application,
  9. Test the restored version of the application.

To make the backup image of the AWS EC2 instance

  1. Open the AWS console and navigate to EC2, Running instances
  2. Find the server instance you want to backup
  3. Right-click the row of the instance and select Image, Create Image

  4. Name the image and click the Create Image button
  5. Save the image Id from the popup

Test your RightScale script without creating unnecessary revisions

There are certain functions in RightScale scripts (RightScripts) that you cannot test without launching a new instance. To test your script:

  1. Edit the Head revision of the RightScale script and save it,
  2. Attach the Head revision of the RightScale script to the Head revision of the RightScale Server Template,
  3. In your Self Service CAT file reference revision 0 (zero) of the Server Template (Head revision),
    ( “server_template_revision”  => “0”, )
  4. Upload the Self Service CAT file to Self Service, but do not publish it,
  5. Launch a server instance from Self Service using the RightScale Self-Service Designer page, not the Catalog.

If the script has some errors, modify the Head revision of the RightScale script and launch a new server instance using the RightScale Self-Service Designer page.

When the script works perfectly:

  1. Commit the RightScale script, creating a new RightScale script revision,
  2. Attach the committed revision of the RightScale script to the Server Template’s Head revision,
  3. Commit the Server Template, creating a new Server Template revision,
  4. Publish the new revision of the Server Template,
  5. Reference the new revision of the Server Template in your Self Service CAT file.

Custom Git Status and Git Pull commands to work with all of your repositories

If you work with many Git repositories it is easy to miss a repository when you commit your changes, and very time consuming to pull from all repositories.

It is easy to automate both processes.

In this example, we will place all files in the ~/Git/devops-scripts folder. If you want to place your files at another location change it in the files below.


\/  \/  \/  \/  \/  \/  \/  \/  \/  \/

In Windows

To find your home directory on a Windows workstation, open a Bash window (Git Bash) and execute

echo ~

/\  /\  /\  /\  /\  /\  /\  /\  /\  /\


Create the Bash script files

  • Create a file with the list of Git repositories and save it as ~/Git/devops-scripts/git-repositories.txt
    You can separate groups of repositories with empty lines.
~/Git/MY_REPO1
~/Git/MY_REPO2

~/Git/MY_REPO3
~/Git/MY_REPO4
  • Create a file to check the status of all Git repositories and save it as ~/Git/devops-scripts/git-status.sh
#!/bin/bash
input="$HOME/Git/devops-scripts/git-repositories.txt" # Use $HOME instead of ~ to refer to the home directory. The Bash loop cannot interpret ~ as the home directory

RED='\033[0;31m'
NC='\033[0m' # No Color

while IFS= read -r folder
do

 # Get the length of the path
 # On Windows the empty line manifests itself as a 1 character long not empty string 
 path_length=${#folder}

 if [[ ( -n "$folder" ) ]] && [[ ( 1 < $path_length ) ]] ; then
 # The path is not empty and the length is greater than 1

 eval cd $folder # Need EVAL, otherwise CD does not work if the path contains ~
 pwd | tr -d '\n'

 status=$(git status)

 # --------------------------------------------------------------------------
 # To handle

 # On branch master
 # Your branch is up-to-date with 'origin/master'.
 # Untracked files:
 # (use "git add <file>..." to include in what will be committed)

 if [[ $status == *"Untracked files"* ]]; then
 echo -e " ---- ${RED}New files added, please ADD${NC}" | tr -d '\n' # -e tells echo to enable backslash escapes
 fi

 # --------------------------------------------------------------------------
 # To handle

 # On branch master
 # Your branch is up-to-date with 'origin/master'.
 # Changes not staged for commit:
 # (use "git add <file>..." to update what will be committed)
 # (use "git checkout -- <file>..." to discard changes in working directory)

 if [[ $status == *"Changes not staged for commit"* ]]; then
 echo -e " ---- ${RED}Files changed, please ADD${NC}" | tr -d '\n' # -e tells echo to enable backslash escapes
 fi

 # --------------------------------------------------------------------------
 # To handle

 # On branch master
 # Your branch is up-to-date with 'origin/master'.
 # Changes to be committed:
 # (use "git reset HEAD <file>..." to unstage)

 if [[ $status == *"Changes to be committed"* ]]; then
 echo -e " ---- ${RED}Please COMMIT${NC}" | tr -d '\n' # -e tells echo to enable backslash escapes
 fi

 # --------------------------------------------------------------------------
 # To handle

 # On branch master
 # Your branch is ahead of 'origin/master' by 1 commit.
 # (use "git push" to publish your local commits)
 # nothing to commit, working tree clean

 if [[ $status == *"git push"* ]]; then
 echo -e " ---- ${RED}Please PUSH${NC}" | tr -d '\n' # -e tells echo to enable backslash escapes
 fi

 # --------------------------------------------------------------------------
 # To handle

 # On branch master
 # Your branch and 'origin/master' have diverged,
 # and have 1 and 1 different commits each, respectively.
 # (use "git pull" to merge the remote branch into yours)
 # nothing to commit, working tree clean

 if [[ $status == *"git pull"* ]]; then
 echo -e " ---- ${RED}Repos diverged, PULL to merge${NC}" | tr -d '\n' # -e tells echo to enable backslash escapes
 fi

 # --------------------------------------------------------------------------

 echo

 fi
done < "$input"

  • Create a file to pull from all Git repositories and save it as ~/Git/devops-scripts/git-pull.sh
#!/bin/bash
input="$HOME/Git/devops-scripts/git-repositories.txt" # Use $HOME instead of ~ to refer to the home directory. The Bash loop cannot interpret ~ as the home directory

while IFS= read -r folder
do

 # Get the length of the path
 # On Windows the empty line manifests itself as a 1 character long not empty string 
 path_length=${#folder}

 if [[ ( -n "$folder" ) ]] && [[ ( 1 < $path_length ) ]] ; then
 # The path is not empty and the length is greater than 1

 eval cd $folder # Need EVAL, otherwise CD does not work if the path contains ~
 pwd
 git pull

 fi

done < "$input"

Enable the script execution


\/  \/  \/  \/  \/  \/  \/  \/  \/  \/

 In Mac OS

  • Open a terminal window in the ~/Git/devops-scripts folder
  • Run these commands to enable the execution of the Bash files
chmod u+x git-status.sh
chmod u+x git-pull.sh

/\  /\  /\  /\  /\  /\  /\  /\  /\  /\


Create an alias

Aliases allow you to execute commands from any folder in the command line and hide the complexity of the entire command.


\/  \/  \/  \/  \/  \/  \/  \/  \/  \/

In Mac OS

If you use the iTerm terminal window

iTerm reads the configuration settings from the~/.zshrc file. If you use iTerm add the alias definitions to the end of the ~/.zshrc file.

  • Open the~/.zshrc file and add the lines below to the end of it
If you use the built-in OSX Terminal

The built-in Terminal windows reads the configuration settings from the~/.bash_profile file.

  • Open the ~/.bash_profile file and add the lines below to it

In Windows

If you use Git Bash
  • Navigate to your home directory ( cd ~ )
  • Create the Bash config file if does not exist
    cp > ~/.bashrc

    Bash will show an error message, but it will create the file that you can edit with any text editor

  • Add the lines below to the file

/\  /\  /\  /\  /\  /\  /\  /\  /\  /\


 

alias status='~/Git/devops-scripts/git-status.sh'
alias pull='~/Git/devops-scripts/git-pull.sh'

Refresh the settings in the Bash shell environment


\/  \/  \/  \/  \/  \/  \/  \/  \/  \/

In MacOS

source ~/.bash_profile

In Windows

Close the Bash window and open it again

/\  /\  /\  /\  /\  /\  /\  /\  /\  /\


How to use it

Open a terminal window in any folder

To check if you need to add, commit or push files in a Git repository:

status

If you have added new files to the repository you get:

/Users/YOUR_USER_NAME/Git/devops-scripts —- New files added, please ADD

If you have changed files in a Git repository you get:

/Users/YOUR_USER_NAME/Git/devops-scripts —- Files changed, please ADD

If you have already executed the git add . command, but have not yet committed the changes:

/Users/YOUR_USER_NAME/Git/devops-scripts—- Please COMMIT

If you have already committed the changes, but have not pushed the to the remote repository:

/Users/YOUR_USER_NAME/Git/devops-scripts —- Please PUSH

To pull from every Git repository before you start to work

pull