Berks update fails with ‘Missing artifacts’ error message

When you add cookbooks as dependencies with the “depends” statement to the metadata.rb file of your Chef cookbook, to be able to test your cookbooks in Chef Test Kitchen, you also have to specify the location of those cookbooks in the Berksfile file.

For all the cookbooks that are available on the Chef Supermarket, one line

source "https://supermarket.chef.io"

is sufficient to specify their location. If a cookbook is only available at GitHub, specify the location with

cookbook 'COOKBOOK_NAME', git: 'git@github.com:PATH_TO_COOKBOOK.git'

If the cookbook is available on the local drive of the workstation, specify the path with

cookbook 'COOKBOOK_NAME', path: '../COOKBOOK_FOLDER_NAME'

Use the above relative path if all of your cookbooks are under the same cookbooks directory.

If a reference to a Chef cookbook is missing from the Berksfile file, the following message appears when you execute berks update.

Unable to satisfy constraints on package …, which does not exist, due to solution constraint (… = …). Solution constraints that may result in a constraint on …: [(… = …) -> (… >= …)]
Missing artifacts: ...
Demand that cannot be met: (… = …)
Unable to find a solution for demands: … (…)

Chef Attributes

Chef attributes are global variables that are available for every cookbook on the node. There are multiple formats to declare and use an attribute. For important notes on the syntax, please see Undefined method or attribute error in a Chef recipe.

To override the value of an attribute that is defined in another cookbook, use the following syntax

node.override['ATTRIBUTE_NAME'] = 'NEW_VALUE'

During compilation, this line will replace the default value of the attribute with the NEW_VALUE.

 

Getting started with InSpec

InSpec is an open-source testing framework to verify your infrastructure satisfies the design requirements.

In this article, we will learn to install and use InSpec with Chef.

Install InSpec

  1. Navigate to https://downloads.chef.io/inspec, and download the installer for the operating system of your workstation.
  2. Execute the downloaded installer.

Start to use InSpec

To use InSpec as the default integration testing tool in Chef Test Kitchen

  1. Open the .kitchen.yml file of the cookbook,
  2. Add the following lines to the file:
    verifier:
      name: inspec
  3. Add this to every suite, so InSpec will search for the test files in the test/recipes directory. Otherwise, the test file needs to be in the test/recipes/SUITE_NAME directory
     verifier:
       inspec_tests:
         - test/recipes
  4. Create a folder structure in the cookbook folder for the InSpec integration tests,
    test
    |--recipes
  5. Create an integration test for your recipe. Create a new file in the test/recipes folder and name it RECIPE_NAME_test.rb. For the default recipe call it default_test.rb,
  6. The following is a simple example of an InSpec integration test:
    # # encoding: utf-8
    
    # Inspec test for recipe my_cookbook::default
    
    # The Inspec reference, with examples and extensive documentation, can be
    # found at https://docs.chef.io/inspec_reference.html
    
    unless os.windows?
     describe user('root') do
     it { should exist }
     skip 'This is an example test, replace with your own test.'
     end
    end
    
    describe port(80) do
     it { should_not be_listening }
     skip 'This is an example test, replace with your own test.'
    end

As you can see, the syntax of InSpec is (intentionally) very similar to ServerSpec, that it replaces. It is very easy to convert existing ServerSpec integration tests to InSpec compliance tests.

Differences between ServerSpec and InSpec

ServerSpec “process”

changed from

 describe process('PROCESS_NAME') do
   it { should be_running }
 end

to

describe processes('PROCESS_NAME') do
  its('states') { should eq ['R<'] }
end

Undefined method or attribute error in a Chef recipe

There are multiple reasons Chef can display the following error message

NoMethodError
 -------------
 Undefined method or attribute `...' on `node'

 

There are multiple ways to reference an attribute in a Chef recipe:

node['ATTRIBUTE_NAME'] (the recommended way)
node["ATTRIBUTE_NAME"] 
node[:ATTRIBUTE_NAME]  ( use it only if the single or double quotes (' or ") would cause a problem in the expression)

node.ATTRIBUTE_NAME

To check if the attribute value is nil, use the following format:

if ( !node['ATTRIBUTE_NAME'].nil? )
   ...
end

If you use the node.ATTRIBUTE_NAME format and the value is nil Chef throws the above error message.

Add SSH key to a Jenkins Git step

To access a Git repository Jenkins can use an SSH key.

To add the SSH key to the Jenkins server use the following Chef script

Store the SSH key in an encrypted data bag called “keys”.

{
 "id": "ci_private_keys",
 "ci_github_key": "-----BEGIN RSA PRIVATE KEY-----\n...\n-----END RSA PRIVATE KEY-----",
}

 

Add the following to the Jenkins Chef recipe

  • Install Git
package 'git'
  • Install the Git and Credentials Jenkins plugins
jenkins_plugin 'git'
jenkins_plugin 'credentials'
  • Copy the SSH key to the Jenkins server
rsa_key = data_bag_item('keys', 'ci_private_keys')
file '/var/lib/jenkins/.ssh/id_rsa' do
  content "#{rsa_key['ci_github_key']}"
  owner 'jenkins'
  group 'jenkins'
  mode '0600'
end
  • Add github.com to the known hosts
bash 'provide github.com RSA fingerprint' do
  code <<-EOF
   ssh-keyscan github.com >> /var/lib/jenkins/.ssh/known_hosts
   chown jenkins.jenkins /var/lib/jenkins/.ssh/known_hosts
  EOF
  not_if{system('grep github.com /var/lib/jenkins/.ssh/known_hosts')}
end

 

To specify the SSH key in the Git step

  1. When the Jenkins server is operational, navigate to the Web interface
  2. Create a new Jenkins project
  3. In the Source Code Management section
    1. Select Git
    2. Enter the SSH URL of the repository
    3. When you are adding the first project, click the Add button to create the credential

      1. Click Jenkins to select the credentials provider
      2. Select SSH Username with private key as the Kind
      3. Enter the username you used when you created the SSH key for the Git repository
      4. Select From the Jenkins master ~/.ssh as the Private Key
      5. Click the Add button
    4. In the Credentials drop down select the credential you have created (the Git user name)

 

Chef custom resource is using the same property name as the called resource

When you create a Chef custom resource, you can call other resources including custom resources you have created. For ease of use it can be convenient to use the same property name as the called resource use.

property :delay_mins,          Fixnum, default: 3

reboot 'Hostname was changed' do
 reason reboot_reason
 delay_mins delay_mins
 action :request_reboot
end

When you you execute the code, chef will display the following error message:

property delay_mins is declared in both reboot[Hostname was changed] and utils_reboot[hostname_reboot] action :request_reboot. Use new_resource.delay_mins instead.

To tell Chef that you want to use the property you have created in this custom resource, add new_resource. in front of your property:

property :delay_mins,          Fixnum, default: 3

reboot 'Hostname was changed' do
 reason reboot_reason
 delay_mins new_resource.delay_mins
 action :request_reboot
end

NoMethodError: undefined method `exists?’ for Chef::Resource::File:Class

When you create a Chef custom resource and use the File class, you need to make slight a change in the syntax you use.

In a Chef recipe you can use

if File.exists?("#{FileName}")

to check for the existence of a file. If we use the same line in a Chef custom resource, we get the error message:

NoMethodError: undefined method `exists?’ for Chef::Resource::File:Class

In a Chef custom resource, you need to add :: in front of the File class to make it work

The leading :: tells Chef not to look for the “File” class in the Chef namespace.

if ::File.exists?("#{FileName}")

 

Chef file locations

When the Chef cookbook is tested in Test Kitchen

The .kitchen.yml file contains the username to run the Chef cookbook. It is specified under platforms:, transport:, username:

Use that value in place of USER_NAME_FROM_KITCHEN_YML below

Cookbook location

When the Chef recipes are executed, all cookbooks are stored on the node. You can examine the code to make sure your latest changes are reflected on the machine.

On Linux

/var/chef/cache/cookbooks

On Linux under Test Kitchen

/tmp/kitchen/cookbooks and
/tmp/kitchen/cache/cookbooks

On Windows

C:\chef\cache\cookbooks

On Windows under Test Kitchen

C:\Users\USER_NAME_FROM_KITCHEN_YML\AppData\Local\Temp\kitchen\cookbooks

 

The log of the Chef client run

The output of the Chef cookbook execution is in the chef.log or chef-client.log file

On Linux

/var/log/chef.log

On Windows

The log of the first Chef Client run. The log entries of the initial Chef Client run are saved here.

C:\chef\chef-client.log

The log of subsequent Chef Client runs. After the initial Chef Client run, the rest of the log entries are collected in this file.

C:\chef\log\client.log

Stacktrace

Chef saves information on the hard drive when scripts are executed. If there is a failure, the stack trace of the last error is saved in the chef-stacktrace.out file.

On Linux

/var/chef/cache/chef-stacktrace.out

On Linux under Test Kitchen

/tmp/kitchen/cache/chef-stacktrace.out

On Windows

C:\chef\cache\chef-stacktrace.out

On Windows under Test Kitchen

C:\Users\USER_NAME_FROM_KITCHEN_YML\AppData\Local\Temp\kitchen\cache\chef-stacktrace.out

Ohai output

All the information that Ohai collects on the instance, is saved in the failed-run-data.json file, even if there is no error. It is a great resource to find get

Cloud info

  • cloud-specific information under “cloud
  • cloud instance information under “ec2

Computer info

  • CPU and memory configuration under “cpu“, “cs_info“, “memory
  • drive sizes and network settings under “filesystem” and “network

Operating system info

  • operating system information under “os_info
  • list of enabled Windows features under “dism_features
  • the list of installed applications under “packages

Chef info

  • Chef client configuration values under “chef_client
  • the Chef node information under “chef_type”: “node
  • the Chef run list under “run_list
  • list of Chef cookbooks and their versions under “cookbooks
  • list of the executed recipes under “recipes
  • the value of the passed in Chef attributes under “normal
  • the value of the Chef cookbook attributes under “chef_client” and the cookbook name
  • all the information on the Chef resources under “json_class
  • the stack trace of the last error under “exception

On Linux

/var/chef/cache/failed-run-data.json

On Linux under Test Kitchen

/tmp/kitchen/cache/failed-run-data.json

On Windows

C:\chef\cache\failed-run-data.json

On Windows under Test Kitchen

C:\Users\USER_NAME_FROM_KITCHEN_YML\AppData\Local\Temp\kitchen\cache\failed-run-data.json

Data Bags

On Linux under Test Kitchen

/tmp/kitchen/data_bags

On Windows under Test Kitchen

C:\Users\USER_NAME_FROM_KITCHEN_YML\AppData\Local\Temp\kitchen\data_bags

 

Internalize Chocolatey packages

To access Chocolatey packages in your private network you need to download them from the Internet and store them at a location where all servers can access them. You can internalize Chocolatey packages if you have a Chocolatey Business subscription.

You can use an Artifactory server to host the internalized packages.

List of available packages

To get the list of the available Chocolatey packages on an Artifactory server

choco list -s http://ARTIFACTORY_SERVER_URL/artifactory/api/nuget/ARTIFACTORY_REPOSITORY_NAME

Non copyrighted applications

If the Chocolatey package does not contain copyrighted components, Chocolatey can download and repackage the entire package, including the application.

Download a Chocolatey package from the Internet and store it on your local drive

choco download PACKAGE_NAME --internalize

To store the Chocolatey packages on an Artifactory server

  • The Artifactory server has to have a Pro license
  • Create a NuGet type local repository

Upload the package from your local drive to an Artifactory server

curl -u ARTIFACTORY_USERNAME:ARTIFACTORY_PASSWORD -X PUT "http://ARTIFACTORY_SERVER_URL/artifactory/ARTIFACTORY_REPOSITORY_NAME/PACKAGE_NAME/PACKAGE_FILE_NAME.nupkg" -T ./PACKAGE_FILE_NAME.nupkg

Copyrighted applications

If the Chocolatey package installs an application that is copyrighted, the author of the package cannot publish the copyrighted source, but can place the download location and installation process into the package.

Internalize the package

To download the necessary files and create a custom Chocolatey package with copyrighted application source:

    1. Download the Chocolatey package without the internalize option
      choco download PACKAGE_NAME

      Example:

      choco download javaruntime
    2. Delete the small .nupkg files. These are too small to contain the installer files. We will recreate them in a subfolder with the downloaded installer files.
    3. Find the package folder in the download sub-folder that has a tools subfolder. In the case of “javaruntime”, the real package is the “jre8”, not “javaruntime”.
    4. Open the tools\chocolateyInstall.ps1 file
    5. Find the download urls of the installer files.

       Some scripts contain variables for the version of the application, so you have to manually assemble the final download URL.
    6. Assemble the actual download URL and download the application source with your web browser,
    7. Place the downloaded installer files into the tools folder,
    8. Update the chocolateyInstall.ps1 to point to the installer files in the package:
      • Add the following before the $url =… and $url64 =… statements
        $toolsDir = "$(Split-Path -parent $MyInvocation.MyCommand.Definition)"
      • Replace the the $url =… and $url64 =… statements. Use the actual file names, and make sure to use double quotes () for the string interpolation to work!
        $url = "$toolsDir\32_BIT_INSTALLER_FILE"
        $url64 = "$toolsDir\64_BIT_INSTALLER_FILE"

      • Add
        -UseOriginalLocation

        to the Install-ChocolateyPackage  line.

    9. Right click the .nuspec file in the package folder and select Compile Chocolatey Package

      (or execute choco pack PATH\TO\NUSPEC.nuspec)
    10. The package is created in the folder of the .nuspec file and the file name is the composite of id + “.” + version tag values in the .nuspec file.

Upload the package to the local repository

  • In the command line execute
    curl -u ARTIFACTORY_USERNAME:ARTIFACTORY_PASSWORD -X PUT "http://ARTIFACTORY_SERVER_URL/artifactory/ARTIFACTORY_REPOSITORY_NAME/PACKAGE_NAME/PACKAGE_FILE_NAME.nupkg" -T PATH/TO/PACKAGE_FILE_NAME.nupkg

    Video tutorials

  • Package Synchronizer