Change the AWS account to launch instances with Chef Test Kitchen

The Chef SDK contains Test Kitchen, that can launch server instances to test your Chef cookbooks. Test Kitchen uses the “chef_zero” provisioner to use your workstation as the virtual Chef server.

To switch Test Kitchen to launch instances in another AWS account

  1. In the .kitchen.yml file update the
    1. availability_zone
    2. subnet_id
    3. aws_ssh_key_id
    4. security_group_ids. Make sure the management ports are open:
      1. for Linux: port 22 for SSH, or
      2. for Windows: port 5985 – 5986 for WinRM and port 3389 for Remote Desktop
    5. ssh_key
  2. In the default.rb attribute file update the
    1. …[‘aws_profile’]
  3. In the ~/.aws/credentials file update the
    1. [default] section

Test your cookbook in Chef Test Kitchen against multiple version of the Chef Client

In large environments, during the Chef Client version change, some older servers still run the prior version of the Chef Client, the newly created servers launch with the new version of the Chef Client.

It is very important to test your cookbooks with the old and the new versions of Chef Client.

To specify the Chef Client version for the Test Kitchen “converge” run, add the highlighted three lines to the provisioner section of the .kitchen.yml file:

provisioner:
  name: chef_zero
  product_name: chef          # Install the Chef Client
  product_version: 14.0.0   # Set the Chef Client version, default=latest
  install_strategy: always    # Forces the installation of the specified Chef Client version even if another version is installed

During “converge”, test kitchen will download and install the specified version of Chef Client.

Warning:

If the installed Chef Client version is newer than the specified version, Test Kitchen downloads the specified version but executes the already installed newer version.

 

Chef exit codes

Chef uses the standard RFC 062 exit codes.

In your .kitchen.yml file, you can supply an array of exit codes in the “retry_on_exit_code” option to retry the operation in case the Chef script execution is interrupted. The usual values are

 retry_on_exit_code: # An array of exit codes that can indicate that kitchen should retry the converge command. Defaults to an empty array.
   - 35              # Reboot is scheduled
   - 20              # The Windows system cannot find the device specified.
   - 1               # Generic failure

Chef exit codes

  0: SUCCESS            - Successful run     - Any successful execution of a Chef utility should return this exit code
  1: GENERIC_FAILURE    - Failed execution   - Generic error during Chef execution.
  2: SIGINT_RECEIVED    - SIGINT received    - Received an interrupt signal
  3: SIGTERM_RECEIVED   - SIGTERM received   - Received an terminate signal
 35: REBOOT_SCHEDULED   - Reboot Scheduled   - Reboot has been scheduled in the run state
 37: REBOOT_NEEDED      - Reboot Needed      - Reboot needs to be completed
 41: REBOOT_FAILED      - Reboot Failed      - Initiated Reboot failed - due to permissions or any other reason
 42: AUDIT_MODE_FAILURE - Audit Mode Failure - Audit mode failed, but chef converged successfully.
213: CLIENT_UPGRADED    - Chef upgrade       - Chef has exited during a client upgrade

Relevant Windows exit codes

 20: ERROR_BAD_UNIT     - The system cannot find the device specified.

The list of all Windows error codes is located at https://msdn.microsoft.com/en-us/library/windows/desktop/ms681381(v=vs.85).aspx

The Linux exit codes for

FATAL: Chef::Exceptions::Reboot: Rebooting server at a recipe’s request.

When your Chef recipe requests a reboot using the “reboot” Chef resource, the output window shows an error message:

Chef Client finished, …/… resources updated in … seconds
[…] WARN: Rebooting server at a recipe’s request. Details: {:delay_mins=>1, :reason=>”…”, :timestamp=>…, :requested_by=>”…”}

Running handlers:
[…] ERROR: Running exception handlers
Running handlers complete
[…] ERROR: Exception handlers complete
Chef Client failed. … resources updated in … seconds
[…] FATAL: Stacktrace dumped to C:/Users/…/AppData/Local/Temp/kitchen/cache/chef-stacktrace.out
[…] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[…] FATAL: Chef::Exceptions::Reboot: Rebooting server at a recipe’s request. Details: {:delay_mins=>1, :reason=>”…”, :timestamp=>…, :requested_by=>”…”}

Chef also saves the error message in the stacktrace.out file at C:/Users/…/AppData/Local/Temp/kitchen/cache

As coderanger wrote it, replying to a StackOverflow post, it is not an error, it is a necessity to be able to stop the Chef cookbook execution when a reboot is requested. See https://stackoverflow.com/questions/39057254/why-does-chef-throw-fatal-error-on-restart-request

“Because we want to avoid running anything further in the Chef converge and the easiest way to do that is to raise an exception.”

“It’s a weird special case because Ruby doesn’t have any other non-local code execution flow construct that would work well in this case. You’ll just have to add a filter for it. Such are the realities of building software in an imperfect world :) – coderanger Aug 21 ’16 at 5:24″

 

 

Unable to satisfy constraints on package

When you make a change to the version restrictions in the metadata.rb file and execute berks install in your Chef cookbook directory, you may get the error message:

Unable to satisfy constraints on package … due to solution constraint (…). Solution constraints that may result in a constraint on …: [(…)], [(…) -> (…)]
Demand that cannot be met: (…)
Artifacts for which there are conflicting dependencies: … = … -> [(…), (… ~> …)]Unable to find a solution for demands:…

 

Execute

berks update

to

  • find the latest allowed versions of all referenced cookbooks,
  • update the Berksfile.lock file and
  • download the missing cookbooks to the cookbook cache folder at ~/.berkshelf/cookbooks
  • This command executes berks install too behind the scenes.

 

 

Skip steps in an InSpec integration test based on Chef attribute values

To speed up the test of some complex recipes, I use an attribute to skip certain long-running installations when I only need to test the rest of the recipe.

We will pass Chef Attribute values into an InSpec test using environment variables.

Declare an attribute in the atrributes.rb file with a safe default value, to allow the execution of all steps if the value is not overridden

# Set this to true in the .kitchen.yml file to skip long-running tasks during local test
default['quick_launch_for_test'] = false

Set the value in the suites: section of the kitchen.yml file to true to skip the long-running installations

  attributes:
    quick_launch_for_test: true

Save the attribute value in an environment variable in the recipe to pass it to InSpec

# Store the value in an environment variable for the integration test
env 'quick_launch_for_test' do
  value "#{node['quick_launch_for_test']}"
end

Make a decision in the recipe based on the value of the attribute

if ( node['quick_launch_for_test'] != true )
  #########################################
  #Microsoft Visual Studio Community 2017 #
  #########################################
  ...
end

Read the value of the environment variable into a local variable in the InSpec test file. If the environment variable does not exist, or the value is not “true”, the value of the local_execution variable will be “false”.

local_execution = ( os_env('quick_launch_for_test').content() == 'true')

Use the if statement to make the decision in your InSpec test file

if !local_execution
  describe package('Microsoft Visual Studio Community 2017') do
    it { should be_installed }
  end
end

 

Waiting for SSH service on …:22, retrying in 3 seconds

When you try to launch a new EC2 instance in AWS using Chef Test Kitchen and the process times out with the message:

  Waiting for SSH service on …:22, retrying in 3 seconds

$$$$$$ [SSH] connection failed, terminating (#<Net::SSH::AuthenticationFailed: Authentication failed for user …@…>)
>>>>>> ——Exception——-
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: 1 actions failed.
>>>>>> Failed to complete #create action: [SSH session could not be established] on default-rhel7
>>>>>> ———————-
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose –all` for configuration

Check the access to the “ssh_key” which is specified in the platforms: driver: transport: section of the .kitchen.yml file.

The .pem file should be accessible from the cookbook directory with the absolute or relative path.

Failed to complete #create action: [undefined method `version’ for nil:NilClass] on default…

When you execute kitchen converge to launch an EC2 instance in AWS with Chef Test Kitchen, you get the error message:

>>>>> ——Exception——-
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: 1 actions failed.
>>>>>> Failed to complete #create action: [undefined method `version’ for nil:NilClass] on default-windows-2012r2
>>>>>> ———————-
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose –all` for configuration

The kitchen-ec2 driver tries to read the operating system version from the name of the image, but it cannot find it.

When you create your own AMI (Amazon Machine Image), make sure the version of the operating system is in the name.

The Test Kitchen Git repository has the following at https://github.com/test-kitchen/kitchen-ec2

Note that the image_search method requires that the AMI image names be in a specific format. Some examples are:

  • Windows-2012
  • Windows-2012r2
  • Windows-2012r2sp1
  • RHEL-7.2

It is safest to use the same naming convention as used by the public images published by the OS vendors on the AWS marketplace.

An acceptable name is my_windows-2012r2_base-1

Could not load the ‘ec2’ driver from the load path

When you execute kitchen list and the driver in your .kitchen file is “ec2“, the following error message appears:

>>>> ——Exception——-
>>>>>> Class: Kitchen::ClientError
>>>>>> Message: Could not load the ‘ec2’ driver from the load path. Please ensure that your driver is installed as a gem or included in your Gemfile if using Bundler.
>>>>>> ———————-
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose –all` for configuration

If you have recently installed the Chef Development Kit, you need to install the Kitchen EC2 driver to be able to launch servers in AWS.

To install the Kitchen EC2 driver, execute

chef gem install kitchen-ec2

 

No instances for regex `’, try running `kitchen list’ in Chef Test Kitchen

When I tried to execute the “kitchen list” command in Chef Test Kitchen the following error came up:

No instances for regex `’, try running `kitchen list’

I could not find the reason for the error, but when I opened the .kitchen.yml file in the Atom editor, made an insignificant change and saved the file, the error went away.