Create the AWS credentials file from a Chef Data Bag

When a process on a server instance needs access to an AWS account, the user who will execute the AWS CLI commands needs to be able to automatically authenticate in AWS.

For automatic AWS authentication, the AWS CLI creates two files in the .aws directory:

  • config and
  • credentials.

The location of this directory depends on the operating system and the type of user.

  • On Linux, the location is ~/.aws ( the user’s home directory )
  • On Windows, it is located at C:\Users\USER_NAME\.aws
  • On Windows, if the file was created by SYSTEM, the location is C:\Windows\System32\config\systemprofile\.aws

Store the AWS key values

To create these files, you need to store the AWS Access Key and Secret Key. The safest place for these values is an encrypted data bag. To automatically generate the AWS files, create a data bag file and name it the same as the “id” in the following structure:

{
  "id": "MY_DATA_BAG_ITEM_NAME",
  "MY_PROFiLE_1": {
    "region": "MY_REGION_1",
    "aws_access_key_id": "MY_ACCESSKEY_1",
    "aws_secret_access_key": "MY_SECRET_KEY_1"
  },
  "MY_PROFiLE_2": {
    "region": "MY_REGION_2",
    "aws_access_key_id": "MY_ACCESSKEY_2",
    "aws_secret_access_key": "MY_SECRET_KEY_2"
  }
}

To create and encrypt the data bag see my post on Chef Data Bags

Create the AWS authentication files

  1. In your Chef recipe, first install the AWS CLI and reboot the server, so the new path entry will be available for the Chef process.
  2. The following Chef code will create the AWS config and credential files. The script
    1. opens and decrypts the data bag,
    2. loads it into a hash table,
    3. iterates through the hash items,
    4. skips the “id” item,
    5. stores the AWS key values in a temporary file,
    6. executes the “aws configure” command to generate the AWS config and credential files.
  # Iterate through the data bag and create the credentials file

  puts "***** Creating the AWS credentials file"

  # Load the encrypted data bag into a hash
  aws_credentials = Chef::EncryptedDataBagItem.load('MY_DATA_BAG_NAME', 'MY_DATA_BAG_ITEM_NAME').to_hash

  # Iterate through the items, skip the "id"
  aws_credentials.each_pair do |key, value|

    # skip the "id"
    next if key == "id"

    # Add the credentials to the .aws/credentials file
    puts "Account #{key}, Region #{value['region']}"

    batch "add_aws_credentials_#{key}" do
      code <<-EOF echo #{value["aws_access_key_id"]}> input.txt
        echo #{value["aws_secret_access_key"]}>> input.txt
        echo #{value["region"]}>> input.txt
        echo.>> input.txt
        aws configure --profile #{key} < input.txt
      EOF
    end

  end

 

Dynamically set Chef resource attributes

When you need to set a Chef resource attribute based on the current state of the environment, there is a way to dynamically provide the value.

  1. Set the value of a boolean variable with a test,
  2. Declare the Chef resource and assign a reference to it to a variable,
  3. Set the resource attribute based on the value of the boolean variable.
# Set a boolean variable with a test 
def MY_BOOLEAN_VARIABLE?
  "#{node['domain']}" != ""
end 

# Execute a resource and get a reference to it into a variable
t = MY_RESOURCE 'MY_RESOURCE_NAME' do
  ...
end
# Set the attribute value based on the boolean variable
t.MY_RESOURCE_ATTRIBUTE MY_ATTRIBUTE_VALUE if MY_BOOLEAN_VARIABLE?

 

 

No instances for regex `’, try running `kitchen list’ in Chef Test Kitchen

When I tried to execute the “kitchen list” command in Chef Test Kitchen the following error came up:

No instances for regex `’, try running `kitchen list’

I could not find the reason for the error, but when I opened the .kitchen.yml file in the Atom editor, made an insignificant change and saved the file, the error went away.

Chef Attributes

Chef attributes are global variables that are available for every cookbook on the node. There are multiple formats to declare and use an attribute. For important notes on the syntax, please see Undefined method or attribute error in a Chef recipe.

To override the value of an attribute that is defined in another cookbook, use the following syntax

node.override['ATTRIBUTE_NAME'] = 'NEW_VALUE'

During compilation, this line will replace the default value of the attribute with the NEW_VALUE.

 

Getting started with InSpec

InSpec is an open-source testing framework to verify your infrastructure satisfies the design requirements.

In this article, we will learn to install and use InSpec with Chef.

Install InSpec

  1. Navigate to https://downloads.chef.io/inspec, and download the installer for the operating system of your workstation.
  2. Execute the downloaded installer.

Allow InSpec to verify Red Hat Enterprise Linux instances

InSpec needs “sudo” access to execute the tests, but Red Hat Enterprise Linux prevents that access. Execute the following code on every instance when it runs in Test Kitchen:

if (node.chef_environment == "_default")
  # Running in Test Kitchen

  # Ensure sudo is installed
  package 'Install sudo' do
    package_name 'sudo'
    action :install
  end

  file '/etc/sudoers' do
    mode 0440
    owner 'root'
    group 'root'
    action :create
  end

  delete_lines 'remove hash-comments from /some/file' do
    path '/etc/sudoers'
    pattern '^.*requiretty'
  end

end

 

Start to use InSpec

To use InSpec as the default integration testing tool in Chef Test Kitchen

  1. Open the .kitchen.yml file of the cookbook,
  2. Delete the following lines from the platform section if exist:
    busser:
      sudo: true
  3. Add the following lines to the file:
    verifier:
      name: inspec
  4. Place the test files into the default location of the InSpec integration test. The “verify” command executes all files in the directory.
    test
    |--recipes
       |--THE_SUITE_NAME
          |--MY_TESTFILE1.rb
          |--MY_TESTFILE2.rb
    
  5. To use the test file of another suite, add this to the suite. With a relative path, you can use tests from other cookbooks (../ANOTHER_COOKBOOK/test/recipes/ANOTHER_SUITE_NAME).
    verifier:
      inspec_tests:
        - path: test/recipes/ANOTHER_SUITE_NAME
  6. Create an integration test for your recipe. Create a new file in the test/recipes/THE_SUITE_NAME folder. The name does not matter, if you are planning to create only one test file for the suite, name the file after the suite: default_test.rb,
  7. The following is a simple example of an InSpec integration test:
    # # encoding: utf-8
    
    # Inspec test for recipe my_cookbook::default
    
    # The Inspec reference, with examples and extensive documentation, can be
    # found at https://docs.chef.io/inspec_reference.html
    
    unless os.windows?
     describe user('root') do
     it { should exist }
     skip 'This is an example test, replace with your own test.'
     end
    end
    
    describe port(80) do
     it { should_not be_listening }
     skip 'This is an example test, replace with your own test.'
    end

As you can see, the syntax of InSpec is (intentionally) very similar to ServerSpec, that it replaces. It is very easy to convert existing ServerSpec integration tests to InSpec compliance tests.

Differences between ServerSpec and InSpec

ServerSpec “process”

changed from

 describe process('PROCESS_NAME') do
   it { should be_running }
 end

to

describe processes('PROCESS_NAME') do
  its('states') { should eq ['R<'] }
end

For more information

For more information on the Kitchen InSpec verifier visit https://github.com/chef/kitchen-inspec

Undefined method or attribute error in a Chef recipe

There are multiple reasons Chef can display the following error message

NoMethodError
 -------------
 Undefined method or attribute `...' on `node'

 

There are multiple ways to reference an attribute in a Chef recipe:

node['ATTRIBUTE_NAME'] (the recommended way)
node["ATTRIBUTE_NAME"] 
node[:ATTRIBUTE_NAME]  ( use it only if the single or double quotes (' or ") would cause a problem in the expression)

node.ATTRIBUTE_NAME

To check if the attribute value is nil, use the following format:

if ( !node['ATTRIBUTE_NAME'].nil? )
   ...
end

If you use the node.ATTRIBUTE_NAME format and the value is nil Chef throws the above error message.

Add SSH key to a Jenkins Git step

To access a Git repository Jenkins can use an SSH key.

To add the SSH key to the Jenkins server use the following Chef script

Store the SSH key in an encrypted data bag called “keys”.

{
 "id": "ci_private_keys",
 "ci_github_key": "-----BEGIN RSA PRIVATE KEY-----\n...\n-----END RSA PRIVATE KEY-----",
}

 

Add the following to the Jenkins Chef recipe

  • Install Git
package 'git'
  • Install the Git and Credentials Jenkins plugins
jenkins_plugin 'git'
jenkins_plugin 'credentials'
  • Copy the SSH key to the Jenkins server
rsa_key = data_bag_item('keys', 'ci_private_keys')
file '/var/lib/jenkins/.ssh/id_rsa' do
  content "#{rsa_key['ci_github_key']}"
  owner 'jenkins'
  group 'jenkins'
  mode '0600'
end
  • Add github.com to the known hosts
bash 'provide github.com RSA fingerprint' do
  code <<-EOF
   ssh-keyscan github.com >> /var/lib/jenkins/.ssh/known_hosts
   chown jenkins.jenkins /var/lib/jenkins/.ssh/known_hosts
  EOF
  not_if{system('grep github.com /var/lib/jenkins/.ssh/known_hosts')}
end

 

To specify the SSH key in the Git step

  1. When the Jenkins server is operational, navigate to the Web interface
  2. Create a new Jenkins project
  3. In the Source Code Management section
    1. Select Git
    2. Enter the SSH URL of the repository
    3. When you are adding the first project, click the Add button to create the credential

      1. Click Jenkins to select the credentials provider
      2. Select SSH Username with private key as the Kind
      3. Enter the username you used when you created the SSH key for the Git repository
      4. Select From the Jenkins master ~/.ssh as the Private Key
      5. Click the Add button
    4. In the Credentials drop down select the credential you have created (the Git user name)

 

Chef custom resource is using the same property name as the called resource

When you create a Chef custom resource, you can call other resources including custom resources you have created. For ease of use it can be convenient to use the same property name as the called resource use.

property :delay_mins,          Fixnum, default: 3

reboot 'Hostname was changed' do
 reason reboot_reason
 delay_mins delay_mins
 action :request_reboot
end

When you you execute the code, chef will display the following error message:

property delay_mins is declared in both reboot[Hostname was changed] and utils_reboot[hostname_reboot] action :request_reboot. Use new_resource.delay_mins instead.

To tell Chef that you want to use the property you have created in this custom resource, add new_resource. in front of your property:

property :delay_mins,          Fixnum, default: 3

reboot 'Hostname was changed' do
 reason reboot_reason
 delay_mins new_resource.delay_mins
 action :request_reboot
end

NoMethodError: undefined method `exists?’ for Chef::Resource::File:Class

When you create a Chef custom resource and use the File class, you need to make slight a change in the syntax you use.

In a Chef recipe you can use

if File.exists?("#{FileName}")

to check for the existence of a file. If we use the same line in a Chef custom resource, we get the error message:

NoMethodError: undefined method `exists?’ for Chef::Resource::File:Class

In a Chef custom resource, you need to add :: in front of the File class to make it work

The leading :: tells Chef not to look for the “File” class in the Chef namespace.

if ::File.exists?("#{FileName}")