Error waiting for instance (i-…) to become ready: unexpected state ‘terminated’, wanted target ‘running’

When you launch a server instance with Terraform, sometimes the error message does not contain the underlying cause. When the cloud provider cannot complete the request, many times Terraform displays a generic error message:

Error waiting for instance (i-...) to become ready: unexpected state 'terminated', wanted target 'running'

To find the root cause of the error in AWS

  1. Log into the AWS console and navigate to the EC2 section,
  2. Search for the instance by the instance Id,
  3. You can find the error message at the bottom of the Description tab

In our specific case, it was Client.VolumeLimitExceeded: Volume limit exceeded

We had to increase the volume limit to be able to launch more large EC2 instances.

Keep multiple versions of applications on Macintosh

Most of the DevOps tools are still in beta versions, many times the new version is not compatible with your existing scripts or have an error that stops your scripts working. To be able to keep multiple versions of the applications and easily switch between them, create symbolic links and point to the version you want to execute.

Create version specific locations

Create a folder for your optional applications.

mkdir /opt

 

Set you as the owner of the “opt” directory and it’s children

sudo chown -R YOUR_USER_NAME:wheel /opt

Set the security of the folder.

sudo chmod 755 /opt

We will create a directory structure to place each version of the application into its own folder.

Create a symbolic link to point to the application in the appropriate version folder. These are the examples for Packer, Terraform, and Vagrant. Notice, that the Vagrant application is in the “bin” directory, so the symbolic link has to point there.

cd /opt/packer
ln -s /opt/packer/packer_1.0.0/packer packer

cd /opt/terraform
ln -s /opt/terraform/terraform_0.9.7/terraform terraform

cd /opt/vagrant
ln -s /opt/vagrant/vagrant_1.9.1/bin/vagrant vagrant #The vagrant application is in the bin folder

Add the main application folder to the path in the config file of your terminal

  • for iTerm2 with zterminal: ~/.zshrc
  • for other terminals: ~/.bash_profile
PATH=/opt/packer:$PATH
PATH=/opt/terraform:$PATH
PATH=/opt/vagrant:$PATH

Vagrant

Vagrant has an installer, that places the bin and embedded folders to “/opt/vagrant”. To be able to keep multiple versions of Vagrant execute the following:

cd /opt/vagrant
mkdir vagrant_MY_VERSION                                 # Create a folder for the new version
sudo chown -R YOUR_USER_NAME:wheel bin                   # Set the owner, so you can move it
sudo chown -R YOUR_USER_NAME:wheel embedded              # Set the owner, so you can move it
mv bin vagrant_MY_VERSION                                # Move the folder to the version specific location
mv embedded vagrant_MY_VERSION                           # Move the folder to the version specific location
rm vagrant                                               # Delete the old version of the symbolic link
ln -s /opt/vagrant/vagrant_MY_VERSION/bin/vagrant vagrant # Create the symbolic link

 


To change the version you want to execute

  • Delete the symbolic link,
    cd /opt/packer
    rm packer
  • Create a new symbolic link pointing to the other version of the application as above.

Variable ‘…’: duplicate found. Variable names must be unique.

Starting with Terraform version 0.7.3 you can only define a variable once within a directory or a module. Before that release you could copy variable definition files from other modules and did not throw an error if you had the same variable defined in multiple files within the module.

In the new version of Terraform if you define the same variable name multiple times within the module you will get the following error message,

Variable '...': duplicate found. Variable names must be unique.

Make sure a variable is only defined once in the module or directory.

How to migrate the DevOps development environment to another workstation

Git

Move the Git repositories to a new workstation

If you want to move Git repositories to your new workstation

  • Commit and push all repositories to GitHub on the old workstation,
  • Copy the folders from your old workstation to the new,
  • Execute the following command in all Git repositories on the new workstation.
    git reset --hard

Terraform

Copying Terraform scripts from a Windows workstation

If the Terraform scripts reference modules, terraform creates symbolic links to those modules in the .terraform/modules directories. When you try to copy those symbolic links on a Windows machine the copy process stops. To copy the Terraform scripts

  • Start Windows explorer in the folder above the Terraform script folders,
  • Search for .terraform,
  • In the search result list delete the .terraform folders.

Test Kitchen

If you migrate between Windows and Macintosh you need to update a few paths, because the two operating systems store user specific files at different places.

The ssh_key location in the .kitchen.yml file should start with

  • On Macintosh: ~/.aws/
  • On Windows: C:\Users\YOUR_USER_NAME\.aws\