CloudExceptions::CloudException – 400: VPCIdNotSpecified: No default VPC for this user

When you launch a new EC2 instance in the AWS cloud from the command line or with other cloud management platforms, you may get the error message:

CloudExceptions::CloudException – 400: VPCIdNotSpecified: No default VPC for this user (RequestID: …)

This can happen when the specified Subnet Id is not a valid subnet in the selected availability zone (datacenter).

Create the AWS credentials file from a Chef Data Bag

When a process on a server instance needs access to an AWS account, the user who will execute the AWS CLI commands needs to be able to automatically authenticate in AWS.

For automatic AWS authentication, the AWS CLI creates two files in the .aws directory:

  • config and
  • credentials.

The location of this directory depends on the operating system and the type of user.

  • On Linux, the location is ~/.aws ( the user’s home directory )
  • On Windows, it is located at C:\Users\USER_NAME\.aws
  • On Windows, if the file was created by SYSTEM, the location is C:\Windows\System32\config\systemprofile\.aws

Store the AWS key values

To create these files, you need to store the AWS Access Key and Secret Key. The safest place for these values is an encrypted data bag. To automatically generate the AWS files, create a data bag file and name it the same as the “id” in the following structure:

{
  "id": "MY_DATA_BAG_ITEM_NAME",
  "MY_PROFiLE_1": {
    "region": "MY_REGION_1",
    "aws_access_key_id": "MY_ACCESSKEY_1",
    "aws_secret_access_key": "MY_SECRET_KEY_1"
  },
  "MY_PROFiLE_2": {
    "region": "MY_REGION_2",
    "aws_access_key_id": "MY_ACCESSKEY_2",
    "aws_secret_access_key": "MY_SECRET_KEY_2"
  }
}

To create and encrypt the data bag see my post on Chef Data Bags

Create the AWS authentication files

  1. In your Chef recipe, first install the AWS CLI and reboot the server, so the new path entry will be available for the Chef process.
  2. The following Chef code will create the AWS config and credential files. The script
    1. opens and decrypts the data bag,
    2. loads it into a hash table,
    3. iterates through the hash items,
    4. skips the “id” item,
    5. stores the AWS key values in a temporary file,
    6. executes the “aws configure” command to generate the AWS config and credential files.
  # Iterate through the data bag and create the credentials file

  puts "***** Creating the AWS credentials file"

  # Load the encrypted data bag into a hash
  aws_credentials = Chef::EncryptedDataBagItem.load('MY_DATA_BAG_NAME', 'MY_DATA_BAG_ITEM_NAME').to_hash

  # Iterate through the items, skip the "id"
  aws_credentials.each_pair do |key, value|

    # skip the "id"
    next if key == "id"

    # Add the credentials to the .aws/credentials file
    puts "Account #{key}, Region #{value['region']}"

    batch "add_aws_credentials_#{key}" do
      code <<-EOF echo #{value["aws_access_key_id"]}> input.txt
        echo #{value["aws_secret_access_key"]}>> input.txt
        echo #{value["region"]}>> input.txt
        echo.>> input.txt
        aws configure --profile #{key} < input.txt
      EOF
    end

  end

 

Ruby Gem Management

Ruby gems are Ruby programs and libraries with a name, version and the platform that can execute them.

List the installed gems on your system

gem list

Detailed list that includes the author, homepage, license, install location and a short description

gem list -d

Install the latest version of the gem

gem install GEM_NAME

Install a specific version of the Gem

gem install GEM_NAME -v GEM_VERSION

Uninstall the gem from your system

gem uninstall GEM_NAME

Uninstall a  gem from a specific location

gem uninstall GEM_NAME -i LOCATION_FROM_GEM_LIST_-D

Update the Gem list in your system after Gem uninstallation

gem update --system

 

 

 

 

Bootstrap Chef nodes to connect them to the Chef server

A Chef node is a physical or virtual machine with an operating system that is connected to the Chef server. Once the node has made the connection to the Chef server, the installed Chef Client can execute Chef cookbooks to configure the machine.

Bootstrapping is the process to connect the node the first time to the Chef server, or to attach it again if the node lost the connectivity to the Chef server. To be able to bootstrap a node, your workstation needs to have the Chef Development Kit installed. The kit includes the ‘knife’ command that communicates with the Chef server. Your workstation also has to be able to connect to the Chef server with the YOUR_USERNAME.pem file you store in the .chef directory just above your cookbooks.

Bootstrap a Linux node

To bootstrap a Linux node, open a terminal window on your workstation and execute the command:

knife bootstrap MY_NODE_IP -x MY_USERNAME -P MY_PASSWORD --sudo --node-name THE_NODE_NAME --environment THE_ENVIRONMENT --run-list 'recipe[MY_COOKBOOK1::default],recipe[MY_COOKBOOK2::default]'

Bootstrap a Windows node

knife bootstrap windows winrm MY_NODE_IP -x MY_USERNAME -P MY_PASSWORD --node-name THE_NODE_NAME --environment THE_ENVIRONMENT --run-list 'recipe[MY_COOKBOOK1::default],recipe[MY_COOKBOOK2::default]' -V

where

  • MY_NODE_IP is the IP address of the node you want to attach to the Chef server,
  • MY_USERNAME and MY_PASSWORD are the credentials to connect to the node,
  • THE_NODE_NAME is the unique name you want the node to use in the Chef server database. If you are bootstrapping a server that lost connectivity to the Chef server, find the node name in the node list.
  • THE_ENVIRONMENT is the name of the environment the node will run the cookbook in,
  • the run list is a list of cookbooks and roles

Dynamically set Chef resource attributes

When you need to set a Chef resource attribute based on the current state of the environment, there is a way to dynamically provide the value.

  1. Set the value of a boolean variable with a test,
  2. Declare the Chef resource and assign a reference to it to a variable,
  3. Set the resource attribute based on the value of the boolean variable.
# Set a boolean variable with a test 
def MY_BOOLEAN_VARIABLE?
  "#{node['domain']}" != ""
end 

# Execute a resource and get a reference to it into a variable
t = MY_RESOURCE 'MY_RESOURCE_NAME' do
  ...
end
# Set the attribute value based on the boolean variable
t.MY_RESOURCE_ATTRIBUTE MY_ATTRIBUTE_VALUE if MY_BOOLEAN_VARIABLE?

 

 

Setup failed: Failed to copy slug dir: lstat /Users: no such file or directory error in Terraform Enterprise

When you try to execute a Terraform configuration in Terraform Enterprise (Atlas) you may get the error message:

Setup failed: Failed to copy slug dir: lstat /Users: no such file or directory

Cause:

The Git repository contains the .terraform/modules directory, and the Terraform Enterprise server cannot get the latest modules from GitHub.

Solution:

  1. Create a .gitignore file in the repository and add these lines:
    */.terraform/modules/
    
    # Ignore DS_Store if working on a mac
    .DS_Store
  2. Delete the .terraform directory and push the changes to GitHub

 

 

Create a Terraform Enterprise environment

If you just start to work with Terraform Enterprise, you need to create a Terraform environment.

Preparation

GitHub account

To access GitHub from Terraform Enterprise, create a GitHub team and account with admin access to the GitHub repository that will store the Terraform scripts.

  1. Create a GitHub team who will have admin access to the Terraform script repository,
  2. Create a GitHub user what Terraform Enterprise will use to access the Terraform script repository,
  3. Add the new GitHub user to your GitHub organization,
  4. Log into GitHub with the new user account credentials and wait for the email verification email from GitHub,
  5. Open the verification email from GitHub and click the button in the email to verify your email address,
  6. Stay logged into GitHub with the new user account and wait for the invitation email from GitHub to join the organization,
  7. Open the invitation email from GitHub and click the button to accept the invitation,
  8. In GitHub add the new GitHub user to the new team,
  9. Create a GitHub repository to store the Terraform scripts,
  10. Add the new team to the Terraform script repository with admin rights.

If the Terraform modules are in a separate GitHub repository, add the new team to that repository with Read rights.

AWS account

Create a new user account that Terraform Enterprise will use to access AWS

  1. Create a new AWS user,
  2. Add user rights
    1. AmazonEC2FullAccess
    2. AmazonS3FullAccess

Connect your Atlas account to GitHub

We will authorize Terraform Enterprise to access the Terraform script GitHub repository. We set this connection up in our personal profile in Terraform enterprise, but when we create new Terraform environments Terraform Enterprise will use this connection to access the GitHub repository the environment will be connected to. Do not use your personal GitHub account to make the connection, because if your personal account loses access to the GitHub repository, the Terraform environment will not be able to connect to it. For this connection, we have already created a new user in GitHub above.

  1. In a web browser log into GitHub with the user account you want Terraform Enterprise use to access the GitHub repository,
  2. In the same web browser open a new tab and navigate to https://atlas.hashicorp.com/settings/connections and log into Atlas with your Hashicorp account,
  3. Click Connect GitHub to Atlas
  4. On the GitHub authorization page click the Authorize hashicorp button,
  5. Later if you want to unlink Atlas from your GitHub account, in the Personal section select Connections, and click the Unlink button.

Create the GitHub repository

Create a repository in GitHub to store the Terraform config files. You can specify subdirectories for each Terraform environment to watch, so one repository can serve the security group creation, and EC2 instance creation for the same developer group.

  1. Create the GitHub repository,
  2. Create a folder that the Terraform Enterprise environment will monitor for changes.

Create a new environment

    1. In a web browser navigate to https://atlas.hashicorp.com/configurations/import,
    2. Log in with your Atlas user account,
    3. In the dropdown list of the New Environment page (click the text of the dropdown, not the arrow) select Link to GitHub.
      Name the environment to include the type of the server you want to launch and the server environment, for example, test_linux_sandbox.
      To save the environment click the Create & Continue button,
    4. On the left side select Variables,
    5. In the Environment section click the Edit button,
  1. If the environment is linked to the Terraform Enterprise Server

    1. The environment has been created. Click the your account settings page to create a personal token. It will allow you to access your account without using your username and password.
    2. Enter a description for the token and click the Generate Token button,
    3. Save the token value at a safe place
    4. To return back to the Runs page select Terraform Enterprise in the upper left corner
    5. Click the name of the environment you want to work with
    6. Configure the environment. On the left side click Settings
      1. Uncheck Plan on artifact uploads
    7. Store your personal access token on your workstation to be able to access Terraform Enterprise. Open a Bash window and execute the export command:

      export ATLAS_TOKEN=[YOUR AUTHENTICATION TOKEN]
    8. Add the terraform backend configuration to the .tf file on your workstation

      terraform {
        backend "atlas" {
          name    = "YOUR_ORGANIZATION/YOUR_ENVIRONMENT"
          address = "https://atlas.hashicorp.com"
        }
      }
    9. Navigate to the Terraform configuration folder and initialize it with the backend configuration you have entered to the .tf file.
      terraform init
    10. Push the Terraform configuration to the Enterprise server

      terraform push -name="YOUR_ORGANIZATION/YOUR_ENVIRONMENT" -atlas-address="https://atlas.hashicorp.com"
    11. On the left side click Configurations to see the list of configs your organization have pushed to the server.
    12. Run a Terraform Plan to see if your code works
      terraform plan
    13. Execute your Terraform config on your workstation
      terraform apply
    14. Push the result of the run to the Terraform Enterprise server
      terraform push -name="YOUR_ORGANIZATION/YOUR_ENVIRONMENT" -atlas-address="https://atlas.hashicorp.com"
    15. To view the result of the run, on the left side click Runs.
    16. To view the created infrastructure states on the left side click States

Create the Terraform scripts

 

More resources

For a tutorial on Multiple Environments in Terraform see https://github.com/hashicorp/multiple-envs

No instances for regex `’, try running `kitchen list’ in Chef Test Kitchen

When I tried to execute the “kitchen list” command in Chef Test Kitchen the following error came up:

No instances for regex `’, try running `kitchen list’

I could not find the reason for the error, but when I opened the .kitchen.yml file in the Atom editor, made an insignificant change and saved the file, the error went away.

GitKraken installation and configuration

GitKraken is a Git user interface to manage Git repositories.

Installation

  1. Download the GitKraken installer from https://www.gitkraken.com/download

Configuration

  1. Start the GitKraken application
  2. Login with your GitHub account, or create a new account

    or
  3. Connect to a Git repository
  4. Click the Open a Repository button
  5. Select a repository folder and click the Open button