No instances for regex `’, try running `kitchen list’ in Chef Test Kitchen

When I tried to execute the “kitchen list” command in Chef Test Kitchen the following error came up:

No instances for regex `’, try running `kitchen list’

I could not find the reason for the error, but when I opened the .kitchen.yml file in the Atom editor, made an insignificant change and saved the file, the error went away.

Berks update fails with ‘Missing artifacts’ error message

When you add cookbooks as dependencies with the “depends” statement to the metadata.rb file of your Chef cookbook, to be able to test your cookbooks in Chef Test Kitchen, you also have to specify the location of those cookbooks in the Berksfile file.

For all the cookbooks that are available on the Chef Supermarket, one line

source "https://supermarket.chef.io"

is sufficient to specify their location. If a cookbook is only available at GitHub, specify the location with

cookbook 'COOKBOOK_NAME', git: 'git@github.com:PATH_TO_COOKBOOK.git'

If the cookbook is available on the local drive of the workstation, specify the path with

cookbook 'COOKBOOK_NAME', path: '../COOKBOOK_FOLDER_NAME'

Use the above relative path if all of your cookbooks are under the same cookbooks directory.

If a reference to a Chef cookbook is missing from the Berksfile file, the following message appears when you execute berks update.

Unable to satisfy constraints on package …, which does not exist, due to solution constraint (… = …). Solution constraints that may result in a constraint on …: [(… = …) -> (… >= …)]
Missing artifacts: ...
Demand that cannot be met: (… = …)
Unable to find a solution for demands: … (…)

Chef file locations

The Chef file and folder locations are different on Linux and Windows machines. This article explains the purpose of each file and the location.

Summary

Linux Windows
Cookbook location /var/chef/cache/cookbooks  C:\chef\cache\cookbooks
Chef Client run log /var/log/chef.log First run only
C:\chef\chef-client.log
Subsequent Chef client runs
C:\chef\log\client.log
Error log /var/chef/cache/chef-stacktrace.out C:\chef\cache\chef-stacktrace.out
Ohai output /var/chef/cache/failed-run-data.json C:\chef\cache\failed-run-data.json
Recommended location for custom log files /tmp/cheflog.log C:\Logs\Chef\cheflog.log
Chef Client configuration /etc/chef/client.rb C:\chef\client.rb

When you test your cookbook in Test Kitchen

The .kitchen.yml file contains the username to execute the Chef cookbook. It is specified under platforms:, transport:, username:

Use that value in place of USER-NAME-FROM-KITCHEN-YML below.

Linux Windows
Cookbook location /tmp/kitchen/cookbooks
/tmp/kitchen/cache/cookbooks
 C:\Users\USER-NAME-FROM-KITCHEN-YML\AppData\Local\Temp\kitchen\cookbooks
Error log /tmp/kitchen/cache/chef-stacktrace.out C:\Users\USER-NAME-FROM-KITCHEN-YML\AppData\Local\Temp\kitchen\cache\chef-stacktrace.out
Ohai output /tmp/kitchen/cache/failed-run-data.json C:\Users\USER-NAME-FROM-KITCHEN-YML\AppData\Local\Temp\kitchen\cache\failed-run-data.json
Data bags /tmp/kitchen/data_bags C:\Users\USER-NAME-FROM-KITCHEN-YML\AppData\Local\Temp\kitchen\data_bags

Cookbook location

When the Chef recipes are executed, all cookbooks are stored on the node. You can examine the code to make sure your latest changes are reflected on the machine.

The log of the Chef client run

The output of the Chef cookbook execution is in the chef.log or chef-client.log file

On Windows

The log of the first Chef Client run and subsequent runs are stored in different log files. After the initial Chef Client run, the rest of the log entries are collected in the second file.

Stacktrace

Chef saves information on the hard drive when scripts are executed. If there is a failure, the stack trace of the last error is saved in the chef-stacktrace.out file.

Ohai output

All the information that Ohai collects on the instance, is saved in the failed-run-data.json file, even if there is no error. It is a great resource to get the server specific values.

Cloud info

  • cloud-specific information under “cloud
  • cloud instance information under “ec2

Computer info

  • CPU and memory configuration under “cpu“, “cs_info“, “memory
  • drive sizes and network settings under “filesystem” and “network

Operating system info

  • operating system information under “os_info
  • list of enabled Windows features under “dism_features
  • the list of installed applications under “packages

Chef info

  • Chef client configuration values under “chef_client
  • the Chef node information under “chef_type”: “node
  • the Chef run list under “run_list
  • list of Chef cookbooks and their versions under “cookbooks
  • list of the executed recipes under “recipes
  • the value of the passed in Chef attributes under “normal
  • the value of the Chef cookbook attributes under “chef_client” and the cookbook name
  • all the information on the Chef resources under “json_class
  • the stack trace of the last error under “exception

Launch Windows instances locally with Chef Test Kitchen

Most Linux distributions are free, and do not require product keys to launch them.

The steps below are based on the great article at http://kitchen.ci/blog/test-kitchen-windows-test-flight-with-vagrant/

I have summarized the steps below to create a free Virtual Box Windows Server 2012R2 image on your workstation, so Test Kitchen can use Vagrant and Virtual Box to launch Windows instances and test cookbooks locally fast and free.


Install the Vagrant WinRm plugin

vagrant plugin install vagrant-winrm

Get BoxCutter

Create the Windows image with BoxCutter. This process will take 30 minutes or more to fully configure the Windows image. The download step may time out on slower VPN connections. In that case disconnect the VPN connection and try it again without it. In case of any error, just execute the last command again and if it can, the process will continue from the point of error.

cd ~/
mkdir boxcutter
cd boxcutter
git clone https://github.com/boxcutter/windows.git
cd windows

Modify the BoxCutter JSON file

Change the ~/boxcutter/windows/eval-win2012r2-standard.json file to avoid the error message:

virtualbox-iso: Removing floppy drive…
==> virtualbox-iso: Error removing floppy controller: VBoxManage error: VBoxManage: error: The machine ‘eval-win2012r2-standard’ is already locked for a session (or being unlocked)
==> virtualbox-iso: VBoxManage: error: Details: code VBOX_E_INVALID_OBJECT_STATE (0x80bb0007), component MachineWrap, interface IMachine, callee nsISupports
==> virtualbox-iso: VBoxManage: error: Context: “LockMachine(a->session, LockType_Write)” at line 1038 of file VBoxManageStorageController.cpp

  • Change “headless”: “false”, to “headless”: “true”,
  • Add under “headless”: “true”,
"shutdown_timeout": "60m",
"post_shutdown_delay": "120s",

 

  • Add under every occurrence of “headless”: “{{ user `headless` }}”,
"shutdown_timeout": "{{ user `shutdown_timeout` }}",
"post_shutdown_delay": "{{ user `post_shutdown_delay` }}",

Create the virtual machine

make virtualbox/eval-win2012r2-standard

Add the image to Vagrant

vagrant box add windows-2012r2 ./box/virtualbox/eval-win2012r2-standard-nocm-1.0.4.box

Test the virtual machine

Your .kitchen.yml file should look like this

---
driver:
  name: vagrant

provisioner:
  name: chef_zero

platforms:
  - name: windows-2012r2

suites:
  - name: MY_SUITE_NAME
  run_list:
  - recipe[MY_COOKBOOK::MY_RECIPE]

 

You may need to execute the kitchen converge commands as sudo to be able to launch the Windows instance.

`block in filter_instances’: undefined method `empty?’ for nil:NilClass (NoMethodError)

When you try to execute any of the “kitchen” commands in Chef Test Kitchen you may get the following error message.

lib/kitchen/config.rb:182:in `block in filter_instances': undefined method `empty?' for nil:NilClass (NoMethodError)

This happened in my cookbook, when I have removed the value from the “excludes:” option in the .kitchen.yml file, but left the “excludes:” option there alone.

To make your cookbook work in Chef Test Kitchen again, remove the empty “excludes:” option form the .kitchen.yml file

DevOps Engineering part 5. – Create an enterprise cookbook

In this exercise we will create a Chef cookbook for a corporation. It will be robust and include all the necessary elements to be used in a large enterprise. If your company already has established standards, as you progress with this tutorial, copy the appropriate file from an existing cookbook. If this is the first cookbook in your organization, or you don’t want to follow the current standards, copy the sample files from this page.

Create the cookbook

  • Navigate to the C:\Chef\…\cookbooks folder and open a Bash window
  • Create an empty cookbook
    chef generate cookbook COOKBOOK_NAME

    Chef creates a new sub-folder with the name of the cookbook.

  • Rename the new folder to cookbook-COOKBOOK_NAME to distinguish it in version control from other repository types.

Update the .gitignore file

The last line is the most important! We want to exclude sensitive information from source control.

.vagrant
Berksfile.lock
*~
*#
.#*
\#*#
.*.sw[a-z]
*.un~
*.pem

# Bundler
Gemfile.lock
bin/*
.bundle/*

.kitchen/
.kitchen.local.yml
.kitchen.yml

# Ignore the unencrypted Data Bags
data_bags_unencrypted/

Update the metadata.rb file

Add your name, email address, and the version of the cookbook to the top of the metadata.rb file

name 'THE_NAME_OF_THE_COOKBOOK'
maintainer 'YOUR_NAME'
maintainer_email 'YOUR_EMAIL_ADDRESS'
license 'All rights reserved'
description 'Installs/Configures THE_NAME_OF_THE_COOKBOOK'
long_description IO.read(File.join(File.dirname(__FILE__), 'README.md'))
version '1.0.0'

The expression in the long_description line refers to the README.md file that GitHub can create when you set up a new repository. Place the cookbook related information into that file, so others who want to use your cookbook can easily find it.

Update the .kitchen.yml file to be able to test your cookbook.

If you want to test your cookbook in the Cloud, copy the following sections from an existing .kitchen.yml file of your organization. To learn Chef and launch instances with Vagrant on your workstation, the automatically created .kitchen.yml file is perfect.

  • driver
  • provisioner
  • platforms
  • a suite as an example

If your organization use tags to track instances in the Cloud,  update the tags section to reflect the values of the cookbook.

The following is a sample .kitchen.yml file with Windows and Linux platforms

driver:
name: ec2
aws_ssh_key_id: YOUR_AWS_SSH_KEY_NAME
region: THE_AWS_REGION # us-east-1
availability_zone: THE_AWS_AVAILIBILITY_ZONE # e
subnet_id: THE_SUBNET_ID
require_chef_omnibus: true
tags:
Name: TK_test
Environment: DEV
created-by: test-kitchen
associate_public_ip: false #This can be false if test instance is accessible by VPN or LAN
retryable_tries: 120
retryable_sleep: 5
# interface: dns #uncomment if test instance should be registered with DNS

provisioner:
name: chef_zero

platforms:
- name: windows-2012r2
driver:
image_id: "A_WINDOWS_AMI_ID"
instance_type: INSTANCE_TYPE # t2.large #Can be changed to any AWS instance size
security_group_ids: [ "SECURITY_GROUP_ID_1", "SECURITY_GROUP_ID_2" ]
transport:
name: winrm
elevated: true # Required for the installation of some Windows features
connection_timeout: 10
connection_retries: 5
username: ADMIN_USERNAME
password: ADMIN_PASSWORD

- name: rhel7
driver:
image_id: "A_LINUX_AMI_ID"
instance_type: INSTANCE_TYPE # t2.medium #Can be changed to any AWS instance size
block_device_mappings:
- device_name: /dev/sda1
ebs:
volume_type: standard
volume_size: 100
delete_on_termination: true
security_group_ids: [ "SECURITY_GROUP_ID_1", "SECURITY_GROUP_ID_2" ]
transport:
ssh_key: C:\Users\YOUR_USERNAME\.aws\YOUR_KEY_FILE_NAME.pem
connection_timeout: 10
connection_retries: 5
username: ADMIN_USERNAME
busser:
sudo: true

suites:
- name: windows_server
data_bags_path: "data_bags"
encrypted_data_bag_secret_key_path: "../../data_bags/DATABAG_SECRET_FILE_NAME"
run_list:
- recipe[COOKBOOK_NAME::RECIPE_NAME]
attributes:
excludes:
- rhel7

Encrypted Data Bags

If you want to use encrypted data bags, ask your Chef administrator to send you the key file to encrypt and decrypt data bags. In the sample .kitchen.yml file above the DATABAG_SECRET_FILE is the placeholder for the key file name. Create a folder for the key at

C:\Chef\...\data_bags

on the same level where the cookbook and environment folders are.

data-bags-04-secret-file-location

This is a chicken and egg paradox. We don’t want to commit secrets into version control, so we need to encrypt them. But how can we place the secret encryption key on the server to decrypt the secrets? We will use Packer to create our own server images that will contain the secret key, so when Chef starts to run on the server, the key is going to be there.

Default recipe

We will place code in the default.rb file that is common to all recipes in the cookbook. All custom recipes will call the default recipe as the first step.

Update the header comments of the default.rb recipe with your name and company information

#
# Cookbook Name:: COOKBOOK_NAME
# Recipe:: default
#
# Author:: YOUR_NAME (<YOUR_EMAIL_ADDRESS>)
# Copyright (c) 2015-2016 COMPANY_NAME, All Rights Reserved.

Custom recipe

Make a copy of the default.rb file and name it to describe the purpose of the instance (server.rb)

  • Update the name of the recipe in the header
  • Add the following line under the header to call the default recipe, even if it is currently empty
    include_recipe 'COOKBOOK_NAME::default'

Add the cookbook to GitHub

Create the local repository

  • In the Bash window execute the following commands
git init
git add .
git commit -m "Initial commit"

Add the repository to GitHub

In your web browser log into your GitHub account and create a new repository

  • Click the New Repository button
  • Name the repository the same as the name of the folder of the cookbook (cookbook-…)
  • Execute the lines in the section …or push an existing repository….  If you work on a Windows workstation make sure HTTPS is selected
    git remote add origin https://github.com/....git
    git push -u origin master
  • In the Collaborators & teams section of Settings select the group who will have access to the new repository

Test the cookbook

In the Bash window launch the instance with Test Kitchen

List the available instances

kitchen list

Launch the instance

kitchen converge STRING_UNIQUE_TO_THE_INSTANCE

If there are multiple suites or platforms in the .kitchen.yml file you need to type the unique part of the name of the instance to identify it.

Start a Remote Desktop connection to a Windows instance

kitchen login STRING_UNIQUE_TO_THE_INSTANCE

If you work on a Macintosh workstation and testing a Windows server, the best way to remote into the server is

  1. Install Microsoft Remote Desktop for free from the Apple App Store (See the Remote Desktop Client section in Install the DevOps development tools on Macintosh,
  2. Execute the kitchen login STRING_UNIQUE_TO_THE_INSTANCE command in the terminal window,
  3. The Microsoft Remote Desktop window will pop up with the User account name and the IP address. If you click “Connect”, the remote connection opens with the default settings, that are may not optimal for your display. The “login” command already created a new entry in the Microsoft Remote Desktop app with the IP address of the instance. To use custom settings, click Cancel,
  4. Open the Microsoft Remote Desktop app, right click the last entry that contains the IP address, select Edit, and copy the IP address to the clipboard,
  5. Create a new connection with custom display settings and paste the IP address there. You can keep this generic connection to access the Test Kitchen instances, just update the IP address.
  6. Delete the automatically created connection

SSH into a Linux instance

ssh USER_NAME@IP_ADDRESS -i PATH_TO_THE SSH_KEY_FILE

Where

  • USER_NAME is the value of username: in the .kitchen.yml file.
  • IP_ADDRESS is the IP of the instance. Get it from the “Waiting for SSH service on…” line of the Test Kitchen bash or command window, or from the .yml file in the .kitchen/logs folder of the cookbook.
  • PATH_TO_THE_SSH_KEY_FILE is the value of ssh_key: in the .kitchen.yml file.

Terminate the instance

kitchen destroy STRING_UNIQUE_TO_THE_INSTANCE

Upload the cookbook to the Chef server

See Connect to the Chef server in Beginner’s Guide to DevOps Engineering part 4.


Next:

Infrastructure as code in Beginner’s Guide to DevOps Engineering part 6.

Back:

to the Tutorials page

Remove Policyfile.rb from your Chef cookbook

When you test your cookbook in Chef Test Kitchen and get the following error, delete the “Policyfile.rb” from your Chef cookbook directory.

$$$$$$ You must set your run_list in your policyfile instead of kitchen config. The run_list your config will be ignored.
$$$$$$ Ignored run_list: ["recipe[...::...]"]
       Preparing dna.json
       Exporting cookbook dependencies from Policyfile /tmp/...
       Error: Invalid lockfile data
       Reason: (ChefDK::DependencyConflict) Cookbook ... (...) has dependency constraints that cannot be met by the existing cookbook set:
       Cookbook ... isn't included in the existing cookbook set.

Unable to satisfy the following requirements error message in Chef Test Kitchen

When Berkshelf in the Chef Test Kitchen cannot resolve the cookbook dependencies, it displays the following error message during converge:

Unable to satisfy the following requirements

To help Berks to start a fresh calculation, delete the Berksfile.lock file and enter the following into the command window:

berks install

DevOps Engineering part 2. – Set up the DevOps development environment

In the first part of the series, Beginner’s Guide to DevOps Engineering Part 1. we have already installed the DevOps development tools.

Create and test your first cookbook in 5 minutes

 Set up the Chef working folder

  • Create a folder for the Chef development(C:\Chef)
  • In the Chef folder create a sub-folder cookbooks
  • Double click the cookbooks folder to navigate into it
  • In Windows Explorer right click the white area and select Git Bash Here

Create your first cookbook

  • Enter the following command into the Bash window to create a new cookbook
    chef generate cookbook test
  • In Windows Explorer navigate to the test\recipes folder
  • Double click the default.rb file and select Select a program from a list of installed programs.
    associate-01-rb-with-notepad
  • Click the Browse… button and navigate to C:\Program Files (x86)\Notepad++\notepad++.exe
    associate-02-rb-with-notepad
  • Enter the following to create your first recipe
    file 'C:\\Windows\\Temp\helloworld.txt' do
     content 'Hello world'
    end

     

  • Save the file
  • In the bash Window navigate to the test cookbook directory
    cd test
  • Execute your first recipe with the following in the Bash window
    chef-client --local-mode recipes/default.rb
  • Navigate to the C:\Windows\Temp directory and open the helloworld.txt file to see the result of your script.

Test your cookbook on a virtual machine

We will use Test Kitchen and Vagrant to launch virtual machines. Currently only Linux images are available, so we will modify our recipe to distinguish between Linux and Windows and act accordingly.

Open the default.rb recipe and update it to look like this

case node['os']
when 'linux'
  file "/tmp/helloworld.txt" do
    content 'This file was created by Chef!'
  end
when 'windows'
  file "C:\\Chef\\helloworld.txt" do
    content 'This file was created by Chef!'
  end
end

Enter the following commands into the Bash window

  • Display the available test configurations ( suites )
    kitchen list
  • Launch a virtual Linux machine and test your recipe
    kitchen converge ubuntu

    The first time you launch a virtual machine with an operating system you have never used on your workstation, Vagrant has to download the machine image from the Internet. It can take 5-10 minutes. When the virtual machine is created and launched, Test Kitchen will copy your cookbook to it and execute it there.
  • Test kitchen will display the —–> Kitchen is finished message when the cookbook has successfully executed.
  • Log into the virtual machine
    kichen login ubuntu
  • The root prompt appears in the Bash windows. Let’s check if the file has been created or not.
  • To destroy the virtual machine
    kitchen destroy ubuntu

Windows in Vagrant

To test your cookbook on a Windows virtual machine locally, create one for Vagrant. See Launch Windows instances locally with Chef Test Kitchen for the details.

Learn the basics, so you can ask questions

Chef has a steep learning curve. Chef is not just scripting or programming, but you have to understand how Chef works to be able to use it to configure servers. There are many ways to do the same thing and there is not much documentation to recommend the best way. If you search Google, the problems usually have multiple solutions, and many times the “best” answer is selected based on personal preference. To get started, you should familiarize yourself with the tools, because you will use most of them during the development process.

In this guide I will use a Windows computer as a workstation, but all tools work on Mac and Linux computers.

Vagrant

Learn Vagrant to understand how Test Kitchen manages the test servers on your local machine or at AWS. You will not use vagrant directly, but Test Kitchen uses it to launch servers.

Chef

Test Kitchen

Ruby

Terraform


Next:

Working with AWS in Beginner’s Guide to DevOps Engineering part 3.

Back:

to the Tutorials page

SSL certificate is missing after installing ChefDK

When you install a new version of ChefDK and try to run Test Kitchen converge you may get the following error message
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: Failed to complete #create action: [SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed

To solve the problem add the following to the Windows environment variables and open a new Command or Bash window for the change to take effect:
SSL_CERT_FILE=C:\opscode\chefdk\embedded\ssl\certs\cacert.pem