Terraform provider.aws: no suitable version installed

The new versions of Terraform do not contain all plugins after the application installation. When you try to access a provider the first time, Terraform may not be able to communicate with it.

* provider.aws: no suitable version installed
version requirements: “(any version)”
versions installed: none

To download the necessary plugins, initialize the Terraform script directory

terraform init

The command will instruct Terraform to get all referenced modules and download all necessary plugins.

 

 

Failed to complete #create action: [undefined method `version’ for nil:NilClass] on default…

When you execute kitchen converge to launch an EC2 instance in AWS with Chef Test Kitchen, you get the error message:

>>>>> ——Exception——-
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: 1 actions failed.
>>>>>> Failed to complete #create action: [undefined method `version’ for nil:NilClass] on default-windows-2012r2
>>>>>> ———————-
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose –all` for configuration

The kitchen-ec2 driver tries to read the operating system version from the name of the image, but it cannot find it.

When you create your own AMI (Amazon Machine Image), make sure the version of the operating system is in the name.

The Test Kitchen Git repository has the following at https://github.com/test-kitchen/kitchen-ec2

Note that the image_search method requires that the AMI image names be in a specific format. Some examples are:

  • Windows-2012
  • Windows-2012r2
  • Windows-2012r2sp1
  • RHEL-7.2

It is safest to use the same naming convention as used by the public images published by the OS vendors on the AWS marketplace.

An acceptable name is my_windows-2012r2_base-1

Could not load the ‘ec2’ driver from the load path

When you execute kitchen list and the driver in your .kitchen file is “ec2“, the following error message appears:

>>>> ——Exception——-
>>>>>> Class: Kitchen::ClientError
>>>>>> Message: Could not load the ‘ec2’ driver from the load path. Please ensure that your driver is installed as a gem or included in your Gemfile if using Bundler.
>>>>>> ———————-
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose –all` for configuration

If you have recently installed the Chef Development Kit, you need to install the Kitchen EC2 driver to be able to launch servers in AWS.

To install the Kitchen EC2 driver, execute

chef gem install kitchen-ec2

 

CloudExceptions::CloudException – 400: VPCIdNotSpecified: No default VPC for this user

When you launch a new EC2 instance in the AWS cloud from the command line or with other cloud management platforms, you may get the error message:

CloudExceptions::CloudException – 400: VPCIdNotSpecified: No default VPC for this user (RequestID: …)

This can happen when the specified Subnet Id is not a valid subnet in the selected availability zone (datacenter).

Create the AWS credentials file from a Chef Data Bag

When a process on a server instance needs access to an AWS account, the user who will execute the AWS CLI commands needs to be able to automatically authenticate in AWS.

For automatic AWS authentication, the AWS CLI creates two files in the .aws directory:

  • config and
  • credentials.

The location of this directory depends on the operating system and the type of user.

  • On Linux, the location is ~/.aws ( the user’s home directory )
  • On Windows, it is located at C:\Users\USER_NAME\.aws
  • On Windows, if the file was created by SYSTEM, the location is C:\Windows\System32\config\systemprofile\.aws

Store the AWS key values

To create these files, you need to store the AWS Access Key and Secret Key. The safest place for these values is an encrypted data bag. To automatically generate the AWS files, create a data bag file and name it the same as the “id” in the following structure:

{
  "id": "MY_DATA_BAG_ITEM_NAME",
  "MY_PROFiLE_1": {
    "region": "MY_REGION_1",
    "aws_access_key_id": "MY_ACCESSKEY_1",
    "aws_secret_access_key": "MY_SECRET_KEY_1"
  },
  "MY_PROFiLE_2": {
    "region": "MY_REGION_2",
    "aws_access_key_id": "MY_ACCESSKEY_2",
    "aws_secret_access_key": "MY_SECRET_KEY_2"
  }
}

To create and encrypt the data bag see my post on Chef Data Bags

Create the AWS authentication files

  1. In your Chef recipe, first install the AWS CLI and reboot the server, so the new path entry will be available for the Chef process.
  2. The following Chef code will create the AWS config and credential files. The script
    1. opens and decrypts the data bag,
    2. loads it into a hash table,
    3. iterates through the hash items,
    4. skips the “id” item,
    5. stores the AWS key values in a temporary file,
    6. executes the “aws configure” command to generate the AWS config and credential files.
  # Iterate through the data bag and create the credentials file

  puts "***** Creating the AWS credentials file"

  # Load the encrypted data bag into a hash
  aws_credentials = Chef::EncryptedDataBagItem.load('MY_DATA_BAG_NAME', 'MY_DATA_BAG_ITEM_NAME').to_hash

  # Iterate through the items, skip the "id"
  aws_credentials.each_pair do |key, value|

    # skip the "id"
    next if key == "id"

    # Add the credentials to the .aws/credentials file
    puts "Account #{key}, Region #{value['region']}"

    batch "add_aws_credentials_#{key}" do
      code <<-EOF echo #{value["aws_access_key_id"]}> input.txt
        echo #{value["aws_secret_access_key"]}>> input.txt
        echo #{value["region"]}>> input.txt
        echo.>> input.txt
        aws configure --profile #{key} < input.txt
      EOF
    end

  end

 

Error waiting for instance (i-…) to become ready: unexpected state ‘terminated’, wanted target ‘running’

When you launch a server instance with Terraform, sometimes the error message does not contain the underlying cause. When the cloud provider cannot complete the request, many times Terraform displays a generic error message:

Error waiting for instance (i-...) to become ready: unexpected state 'terminated', wanted target 'running'

To find the root cause of the error in AWS

  1. Log into the AWS console and navigate to the EC2 section,
  2. Search for the instance by the instance Id,
  3. You can find the error message at the bottom of the Description tab

In our specific case, it was Client.VolumeLimitExceeded: Volume limit exceeded

We had to increase the volume limit to be able to launch more large EC2 instances.

Find the AWS AMI that a RightScale server template is using

When you use RightScale to launch servers in the cloud, you want to use the same base image to test your Chef cookbooks in Test Kitchen.

Packer by Hashicorp is a utility to create custom server images based on cloud images. You supply the image ID and other configuration parameters to create a new custom image.

To find the AMI ID of the base image of the RightScale Server Template

  1. Open the RightScale user interface in your browser,
  2. Select the RightScale account where the server template was created,
  3. In RightScale Cloud Management select  Design / ServerTemplates,
  4. Select the server template,
  5. Select Images,
  6. Select the MultiCloud image,
  7. Select Clouds,
  8. Select the image in the region you are working in,
  9. The Resource UID is the AMI ID of the image.

Migrate databases to and from AWS RDS instances

Amazon Web Services do not provide tools to migrate data to, from and between AWS RDS instances.

There is a free utility that can do the job.

SQL Server Management Studio (SSMS)

Install Microsoft SQL Server Management Studio (SSMS), because the utility uses libraries from it.

  1. Download the installer for free from https://msdn.microsoft.com/en-us/library/mt238290.aspx

SQL Azure Migration Wizard

Installation

  1. Download the SQL Azure Migration Wizard from http://sqlazuremw.codeplex.com/
  2. Unzip the downloaded ZIP file into a folder on your hard drive
  3. Create a shortcut in your Start Menu for SQLAzureMW.exe

Migrate a database

  1. Start the SQL Azure Migration Wizard by executing the SQLAzureMW.exe file
  2. Select Database in the Analyze / Migrate section and click Next

Connect to the source database

  1. On the Connect to Server page
    1. Enter the address of the database server
    2. Select SQL Server Authentication
    3. Enter the database owner login name and password
    4. If you keep Master DB selected, the next screen will show the list of databases to select from
    5. To save the username and password select Save Login Information
    6. Click Connect

  2. Select the database to connect to and click Next
  3. Leave Script all database objects selected and click Next
  4. On the Script Wizard Summary page click Next
  5. Click Yes to start the read process
  6. On the Results Summary page select Auto Scroll Display to see the scrolling results
  7. When the read is done click Next

Connect to the target database

  1. Enter the connection properties of the target database server
  2. Select the target database
    1. To migrate the data into a new database click the Create Database button
    2. On mirrored RDS instances we cannot drop and recreate databases. To migrate data into an existing database
      1. Use SQL Management Studio to drop the existing tables to avoid key conflicts
      2. Select the target database on this page
  3. Click Next
  4. Click Yes to start the write process
  5. When the write process is done click Exit

 

DevOps Engineering part 1. (Mac) – Install the DevOps development tools on Macintosh

Enable the right-click

To be able the right-click on the mouse

  1. Open settings.
  2. Enable Secondary click for the mouse and the trackpad.

Lock the screen

To be able to lock the screen from the menu bar:

  1. In Finder search for Keychain Access and start it,
  2. In the Keychain Access menu select Preferences,
  3. Check the Show keychain status in menu bar checkbox,
    This will place a lock icon in the menu bar.

To lock your screen

  1. Click the lock icon in the menu bar
  2. Select Lock Screen

Show the full path in Finder

  1. Open Finder
  2. In the View menu select Show Path Bar
    Finder will show the full path of the current folder at the bottom of the window. 

To jump to a folder shown in the path bar just double click it.

Show hidden files and folders

  1. Open the terminal and execute the following line
    defaults write com.apple.finder AppleShowAllFiles YES
  2. For the change to take effect relaunch Finder.
    • Press the ‘Option/Alt’ key, right click on the Finder icon in the dock and click Relaunch

Show the battery charge percentage

  1. Click the battery icon in the toolbar
  2. Select Show Percentage

Apple Id without credit card

If you don’t have an Apple Id, create one. To obtain one without a credit card, you need to create the account in an older version of iTunes. In version 12.7 Apple removed the access to the App Store.

  1. Downgrade iTunes on your Mac to version 12.6. Follow the instructions at http://osxdaily.com/2017/09/17/downgrade-itunes-12-7-to-itunes-12-6/
  2. Start iTunes 12.6 on your Macintosh,
  3. Click the drop-down menu in the upper left corner,
    apple-id-01
  4. If Apps is visible, select it, otherwise click Edit Menu
    • Select Apps, and click Done to display it in the drop down.
      apple-id-02
  5. Search for a free app in the App Store and start to download it,
    apple-id-03
  6. Create a new Apple Id and select None for credit card type.

Show the user home directory

  1. Open Finder
  2. In the Finder menu select Preferences
  3. Select the Sidebar tab
  4. In the Favorites section select the checkbox next to your user name
    show-home-directory

Create a directory for optional applications

Create the opt folder in the root of the hard disk.

Memory usage monitor

  1. Download  Dr. Cleaner Elite from the App Store to be able to monitor memory, CPU and network usage from the Menu Bar.


UTC Clock in the menu bar

BitBar is a plugin manager that can display plugins in the menu bar. One of them is World Clock that can display multiple clocks in a drop down of the menu.

Install BitBar

  1. Navigate to https://getbitbar.com/
  2. Click the Get BitBar button
  3. Click the downloaded ZIP file to extract the application
  4. Move the BitBar application to the Applications folder
  5. Launch BitBar form the Launcher
    • The first time BitBar launches, opens the Finder to point to the BitBar plugins folder
      • Click the New Folder button to create a folder for plugins in the Documents folder: BitBar Plugins
      • Click the Use As Plugins Directory to select the new folder as the plugins folder

Add World Clock to BitBar

  1. Navigate to https://getbitbar.com/plugins/Time/worldclock.1s.sh
  2. Click the + Add to BitBar button

Customize World Clock

To customize World Clock edit the worldclock.1s.sh file in the BitBar Plugins folder

To change the displayed timezones, edit the ZONES variable inthe file:

ZONES="US/Pacific UTC Europe/London Europe/Berlin Asia/Kolkata Asia/Tokyo Australia/Sydney"

To change the display format, edit the data formats.

  1. To remove the seconds from the menu bar, delete the :%S from the format line
date -u +'%H:%M UTC'

To add the day of the week and date to the clocks in the World Clock drop down

add

  • %a for the day of the week
  • %Y for the four digit year
  • %b for the name of the month
  • %d for the date
  • %z for the time zone offset
echo "$(TZ=$zone date +'%H:%M - %a, %Y %b %d %z') $zone"

To refresh the dropdown formats after making changes in the config file

  1. Click the World Clock in the menu bar
  2. Select Preferences, Refresh all

 

Homebrew

Homebrew can install packages on your Macintosh that apple does not provide.

Installation instructions are at http://brew.sh/


NPM

NPM is another package manager. Install it to be able to install other applications later on this page, including the SQL Command utility

brew install node

Remote Desktop Client

Download the Microsoft Remote Desktop app from the App Store

  1. Open the App Store
  2. Search for “microsoft remote desktop”
  3. Click the Microsoft Remote Desktop icon
  4. Click the blue Get button
  5. Click the green Install App button

Text Editor

Atom

  1. Download it from https://atom.io/
  2. Double click the downloaded ZIP file to extract the application,
  3. Drag the Atom application into Applications

Configure Atom

See Atom text editor configuration

 


Terminal Window

iTerm2

Install iTerm2, a smart terminal emulator to issue Bash commands and log into Linux servers.

  1. Download iTerm2 from https://www.iterm2.com/downloads.html
  2. Double click the downloaded ZIP file to extract the application,
  3. Move the iTerm application to Applications.

iTerm2 configuration

Enable unlimited scroll back

  1. Start iTerm2 and open the preferences window by pressing ⌘, (command-comma)
  2. On the Terminal tab click the Unlimited scrollback check box.

Start iTerm2 from Finder

To be able to open iTerm2 in any folder from Finder follow the steps below based on https://gist.github.com/shuntaroy/e5ce3ad61c0deb7e27f0

  1. Start Automator,
  2. Select Service and click Choose,

  3. On the top of the screen set Service receives selected to files and folders,
  4. Set in to Finder,

  5. On the second side bar from the left double click Run AppleScript,

  6. Paste the code from the website referenced above into the editor window replacing the sample script in the editor window,
  7. In the File menu save the script,
  8. Do not run it, it does not execute in the Automator.

To use it

  1. Open Finder and navigate to the location you want to work in,
  2. Right click a folder to work in that folder or a file to work in the current folder
  3. Select Services, iTerm2 in Finder


 

“Solarized” color scheme

  1. Download the color scheme to the Desktop with the following Bash command:
    curl -o ~/Desktop/solarized.itermcolors https://raw.githubusercontent.com/altercation/solarized/master/iterm2-colors-solarized/Solarized%20Dark.itermcolors
  2. Import the color scheme to iTerm2
    • Press ⌘, (command-comma) to open the Preferences window
    • Select Profiles
    • In the lower right corner of the Colors tab click the Color Presets… drop down
    • Select Import
    • On the Desktop select the downloaded solarized.itermcolors file
    • In the  Color Presets… drop down select Solarized Dark

 Meslo Powerline font

  • Import the Meslo Powerline font with the following bash command:
    git clone https://github.com/powerline/fonts.git && cd fonts && ./install.sh
  • Select the font in iTerm2
    • Press ⌘, (command-comma) to open the Preferences window
    • Select Profiles
    • On the Font section of the Text tab click the Change Font button
    • Select the 12pt Meslo LG L Regular for Powerline

Make the prompt shorter

to remove your username@computername from the prompt

  • Edit the ~/.zshrc file
  • Add the following line
    DEFAULT_USER="YOUR_USER_NAME"

Colored prompt

To color the prompt:

  • Download the agnoster theme
sh -c "$(curl -fsSL https://raw.github.com/robbyrussell/oh-my-zsh/master/tools/install.sh)"
  • Edit the ~/.zshrc file and set the ZSH_THEME to ZSH_THEME=”agnoster”

More configuration suggestions are at

https://ruigomes.me/blog/perfect-iterm-osx-terminal-installation/
https://gist.github.com/kevin-smets/8568070

Information on how terminals read settings at https://shreevatsa.wordpress.com/2008/03/30/zshbash-startup-files-loading-order-bashrc-zshrc-etc/


Git

  • Navigate to https://git-scm.com/download/mac to download Git for the Macintosh. The page automatically downloads the installer for the operating system you use.
  • This app is not trusted by Apple, so to install it
    • Control-click (right-click) the downloaded file and select Open
    • Click the Open button to confirm the action

Configure Git

To configure Git see Git configuration.


VirtualBox

For the VirtualBox installation on your operating system see: VirtualBox


Vagrant

For Vagrant installation see Vagrant.


Ruby

Ruby is already a part of the operating system.


Chef Development Kit


Terraform by Hashicorp

 Terraform Installation

  • Download Terraform from https://www.terraform.io/downloads.html
  • Double click the downloaded ZIP file to extract the application
  • Create a directory, terraform for the Terraform application in the /opt folder
  • Move the Terraform application into the terraform directoryin the /opt folder

Add the location to the path

Depending on the terminal window you use, it may open different configuration files. If you use iTerm2 you need to update the path in the~/.zshrc file.  Other terminal windows usually read the configuration from the  ~/.bash_profile file. To be safe you can add the following to both configuration files:

# PATH Export
PATH=/opt/terraform:$PATH
export PATH
unset DYLD_LIBRARY_PATH

Graphviz

Graphviz is a Dependency Graph Visualization Software. We will use this utility to display the Terraform graphs.

To install execute the following in the terminal

brew install graphviz

AWS CLI

The Amazon Web Services command line interface installation will set up your workstation to launch instances in AWS from Test Kitchen. If you know you will work with AWS, see DevOps Engineering part 3. – Working with AWS for the AWS CLI installation.


SQL Command

To be able to execute Microsoft SQL commands from the command line or through Terraform, install the MS SQL utility

npm install -g sql-cli

Version information

mssql --version

More information on the tool is at https://www.npmjs.com/package/sql-cli


Packer

We use Packer to create custom AWS AMIs that contain the fundamental configuration and applications that are common in every instance we launch.

Install Packer

  • Add the Packer installation directory to the system path

Working with Windows servers from the Mac

If you want to work with Windows servers, you may need to run Windows applications on your workstation.

See Minimum Windows workstation setup to work with Windows servers



Next:

Set up the DevOps development environment in Beginner’s Guide to DevOps Engineering part 2.

Back:

to the Tutorials page

DevOps Engineering part 5. – Create an enterprise cookbook

In this exercise we will create a Chef cookbook for a corporation. It will be robust and include all the necessary elements to be used in a large enterprise. If your company already has established standards, as you progress with this tutorial, copy the appropriate file from an existing cookbook. If this is the first cookbook in your organization, or you don’t want to follow the current standards, copy the sample files from this page.

Create the cookbook

  • Navigate to the C:\Chef\…\cookbooks folder and open a Bash window
  • Create an empty cookbook
    chef generate cookbook COOKBOOK_NAME

    Chef creates a new sub-folder with the name of the cookbook.

  • Rename the new folder to cookbook-COOKBOOK_NAME to distinguish it in version control from other repository types.

Update the .gitignore file

  • Chef DK has generated a .gitignore file that contains most of the important entries, but we need to add a few important lines:
    # Certificates
    *.pem
    
    # Unencrypted Data Bags
    data_bags_unencrypted/
    
    # Macintosh folder custom attribute file
    .DS_Store
    
    # If working in a team, personalized Test Kitchen config file 
    .kitchen.yml

Update the metadata.rb file

Add your name, email address, and the version of the cookbook to the top of the metadata.rb file

name 'THE_NAME_OF_THE_COOKBOOK'
maintainer 'YOUR_NAME'
maintainer_email 'YOUR_EMAIL_ADDRESS'
license 'All rights reserved'
description 'Installs/Configures THE_NAME_OF_THE_COOKBOOK'
long_description IO.read(File.join(File.dirname(__FILE__), 'README.md'))
version '1.0.0'

The expression in the long_description line refers to the README.md file that GitHub can create when you set up a new repository. Place the cookbook related information into that file, so others who want to use your cookbook can easily find it.

Update the Berksfile

Add the list of cookbooks this cookbook depends on to the Berksfile file. To download all necessary cookbooks execute

berks install

Update the .kitchen.yml file to be able to test your cookbook.

If you want to test your cookbook in the Cloud, copy the following sections from an existing .kitchen.yml file of your organization. To learn Chef and launch instances with Vagrant on your workstation, the automatically created .kitchen.yml file is perfect.

  • driver
  • provisioner
  • platforms
  • a suite as an example

If your organization use tags to track instances in the Cloud,  update the tags section to reflect the values of the cookbook.

To add multiple recipes to the Chef run list use this syntax:

run_list:
  - recipe[COOKBOOK_NAME1::RECIPE_NAME]
  - recipe[COOKBOOK_NAME2::RECIPE_NAME]

Encrypted Data Bags

If you want to use encrypted data bags, ask your Chef administrator to send you the key file to encrypt and decrypt data bags. Create the data_bags_unencrypted folder for the data bag secret key on the same level where the cookbook and environment folders are.

This is a chicken and egg paradox. We don’t want to commit secrets into version control, so we need to encrypt them. But how can we place the secret encryption key on the server to decrypt the secrets? We will use Packer to create our own server images that will contain the secret key, so when Chef starts to run on the server, the key is going to be there.

Default recipe

We will place code in the default.rb file that is common to all recipes in the cookbook. All custom recipes will call the default recipe as the first step.

Update the header comments of the default.rb recipe with your name and company information

#
# Cookbook Name:: COOKBOOK_NAME
# Recipe:: default
#
# Author:: YOUR_NAME (<YOUR_EMAIL_ADDRESS>)
# Copyright (c) 2015-2016 COMPANY_NAME, All Rights Reserved.

Attribute file

Create the default attribute file with

chef generate attribute default

Add the necessary attributes to the attribute file to store the AWS tag and other cookbook specific values.

Custom recipe

Use the Chef generate command to create the new recipes. It creates all test files in the corect location.

  • Create a new recipe and the test files with
    chef generate recipe MY_RECIPE_NAME
  • Add the following line under the header to call the default recipe, even if it is currently empty
    include_recipe 'COOKBOOK_NAME::default'

Add the cookbook to GitHub

Create the local repository

  • In the Bash window execute the following commands
git init
git add .
git commit -m "Initial commit"

Add the repository to GitHub

In your web browser log into your GitHub account and create a new repository

  • Click the New Repository button
  • Name the repository the same as the name of the folder of the cookbook (cookbook-…)
  • Execute the lines in the section …or push an existing repository….  If you work on a Windows workstation make sure HTTPS is selected
    git remote add origin https://github.com/....git
    git push -u origin master
  • In the Collaborators & teams section of Settings select the group who will have access to the new repository

Test the cookbook

In the Bash window launch the instance with Test Kitchen

List the available instances

kitchen list

Launch the instance

kitchen converge STRING_UNIQUE_TO_THE_INSTANCE

If there are multiple suites or platforms in the .kitchen.yml file you need to type the unique part of the name of the instance to identify it.

Start a Remote Desktop connection to a Windows instance

kitchen login STRING_UNIQUE_TO_THE_INSTANCE

If you work on a Macintosh workstation and testing a Windows server, the best way to remote into the server is

  1. Install Microsoft Remote Desktop for free from the Apple App Store (See the Remote Desktop Client section in Install the DevOps development tools on Macintosh,
  2. Execute the kitchen login STRING_UNIQUE_TO_THE_INSTANCE command in the terminal window,
  3. The Microsoft Remote Desktop window will pop up with the User account name and the IP address. If you click “Connect”, the remote connection opens with the default settings, that are may not optimal for your display. The “login” command already created a new entry in the Microsoft Remote Desktop app with the IP address of the instance. To use custom settings, click Cancel,
  4. Open the Microsoft Remote Desktop app, right click the last entry that contains the IP address, select Edit, and copy the IP address to the clipboard,
  5. Create a new connection with custom display settings and paste the IP address there. You can keep this generic connection to access the Test Kitchen instances, just update the IP address.
  6. Delete the automatically created connection

SSH into a Linux instance

ssh USER_NAME@IP_ADDRESS -i PATH_TO_THE SSH_KEY_FILE

Where

  • USER_NAME is the value of username: in the .kitchen.yml file.
  • IP_ADDRESS is the IP of the instance. Get it from the “Waiting for SSH service on…” line of the Test Kitchen bash or command window, or from the .yml file in the .kitchen/logs folder of the cookbook.
  • PATH_TO_THE_SSH_KEY_FILE is the value of ssh_key: in the .kitchen.yml file.

Terminate the instance

kitchen destroy STRING_UNIQUE_TO_THE_INSTANCE

Upload the cookbook to the Chef server

See Connect to the Chef server in Beginner’s Guide to DevOps Engineering part 4.


Next:

Infrastructure as code in Beginner’s Guide to DevOps Engineering part 6.

Back:

to the Tutorials page