Create the AWS credentials file from a Chef Data Bag

When a process on a server instance needs access to an AWS account, the user who will execute the AWS CLI commands needs to be able to automatically authenticate in AWS.

For automatic AWS authentication, the AWS CLI creates two files in the .aws directory:

  • config and
  • credentials.

The location of this directory depends on the operating system and the type of user.

  • On Linux, the location is ~/.aws ( the user’s home directory )
  • On Windows, it is located at C:\Users\USER_NAME\.aws
  • On Windows, if the file was created by SYSTEM, the location is C:\Windows\System32\config\systemprofile\.aws

Store the AWS key values

To create these files, you need to store the AWS Access Key and Secret Key. The safest place for these values is an encrypted data bag. To automatically generate the AWS files, create a data bag file and name it the same as the “id” in the following structure:

{
  "id": "MY_DATA_BAG_ITEM_NAME",
  "MY_PROFiLE_1": {
    "region": "MY_REGION_1",
    "aws_access_key_id": "MY_ACCESSKEY_1",
    "aws_secret_access_key": "MY_SECRET_KEY_1"
  },
  "MY_PROFiLE_2": {
    "region": "MY_REGION_2",
    "aws_access_key_id": "MY_ACCESSKEY_2",
    "aws_secret_access_key": "MY_SECRET_KEY_2"
  }
}

To create and encrypt the data bag see my post on Chef Data Bags

Create the AWS authentication files

  1. In your Chef recipe, first install the AWS CLI and reboot the server, so the new path entry will be available for the Chef process.
  2. The following Chef code will create the AWS config and credential files. The script
    1. opens and decrypts the data bag,
    2. loads it into a hash table,
    3. iterates through the hash items,
    4. skips the “id” item,
    5. stores the AWS key values in a temporary file,
    6. executes the “aws configure” command to generate the AWS config and credential files.
  # Iterate through the data bag and create the credentials file

  puts "***** Creating the AWS credentials file"

  # Load the encrypted data bag into a hash
  aws_credentials = Chef::EncryptedDataBagItem.load('MY_DATA_BAG_NAME', 'MY_DATA_BAG_ITEM_NAME').to_hash

  # Iterate through the items, skip the "id"
  aws_credentials.each_pair do |key, value|

    # skip the "id"
    next if key == "id"

    # Add the credentials to the .aws/credentials file
    puts "Account #{key}, Region #{value['region']}"

    batch "add_aws_credentials_#{key}" do
      code <<-EOF echo #{value["aws_access_key_id"]}> input.txt
        echo #{value["aws_secret_access_key"]}>> input.txt
        echo #{value["region"]}>> input.txt
        echo.>> input.txt
        aws configure --profile #{key} < input.txt
      EOF
    end

  end

 

Bootstrap Chef nodes to connect them to the Chef server

A Chef node is a physical or virtual machine with an operating system that is connected to the Chef server. Once the node has made the connection to the Chef server, the installed Chef Client can execute Chef cookbooks to configure the machine.

Bootstrapping is the process to connect the node the first time to the Chef server, or to attach it again if the node lost the connectivity to the Chef server. To be able to bootstrap a node, your workstation needs to have the Chef Development Kit installed. The kit includes the ‘knife’ command that communicates with the Chef server. Your workstation also has to be able to connect to the Chef server with the YOUR_USERNAME.pem file you store in the .chef directory just above your cookbooks.

Bootstrap a Linux node

To bootstrap a Linux node, open a terminal window on your workstation and execute the command:

knife bootstrap MY_NODE_IP -x MY_USERNAME -P MY_PASSWORD --sudo --node-name THE_NODE_NAME --environment THE_ENVIRONMENT --run-list 'recipe[MY_COOKBOOK1::default],recipe[MY_COOKBOOK2::default]'

Bootstrap a Windows node

knife bootstrap windows winrm MY_NODE_IP -x MY_USERNAME -P MY_PASSWORD --node-name THE_NODE_NAME --environment THE_ENVIRONMENT --run-list 'recipe[MY_COOKBOOK1::default],recipe[MY_COOKBOOK2::default]' -V

where

  • MY_NODE_IP is the IP address of the node you want to attach to the Chef server,
  • MY_USERNAME and MY_PASSWORD are the credentials to connect to the node,
  • THE_NODE_NAME is the unique name you want the node to use in the Chef server database. If you are bootstrapping a server that lost connectivity to the Chef server, find the node name in the node list.
  • THE_ENVIRONMENT is the name of the environment the node will run the cookbook in,
  • the run list is a list of cookbooks and roles

How to create a bootable USB drive to install Windows

If the computer you want to install Microsoft WIndows on, does not have a DVD drive, you can install Windows from a USB drive. To start the computer from the USB drive, you need to prepare the drive to make it bootable.

Microsoft has a free tool that can download the edition of the WIndows operating system you need, format the USB drive, make it bootable, and place the installer file on it.

  1. Using a web browser navigate to https://www.microsoft.com/en-us/software-download/windows10,
  2. Click the Download tool now button to install the Microsoft Media Creation Tool,
  3. Start the downloaded MediaCreationTool.exe program and follow the prompts.

Cannot restart the Atlassian Confluence service on Windows

When the Atlassian Confluence wiki is installed on a Windows server, it frequently becomes unavailable. Sometimes it is possible to restart the Atlassian Confluence Windows service, but most of the time the Stop phase times out with:

Windows could not stop the Atlassian Confluence service on Local Computer.
Error 1053: The service did not respond to the start or control request in a timely fashion.

To make Atlassian Confluence work again

  1. Open Task Manager,
  2. End the tomcat…exe process,
  3. Start the Atlassian Confluence Windows service.

Send CTRL-ALT-DELETE a Windows Virtual Box computer from a Macintosh

When you start a Windows 7 or equivalent server machine, to log in, you need to press CTRL-ALT-DELETE on the keyboard to get the login page. To send Control-Alt-Delete to a Windows virtual machine in Virtual Box from a Macintosh

on a Macintosh laptop press the fn – Command  – Delete back  keys

on a Macintosh desktop press the Command  – Delete forward  keys.

Test your RightScale script without creating unnecessary revisions

There are certain functions in RightScale scripts (RightScripts) that you cannot test without launching a new instance. To test your script:

  1. Edit the Head revision of the RightScale script and save it,
  2. Attach the Head revision of the RightScale script to the Head revision of the RightScale Server Template,
  3. In your Self Service CAT file reference revision 0 (zero) of the Server Template (Head revision),
    ( “server_template_revision”  => “0”, )
  4. Upload the Self Service CAT file to Self Service, but do not publish it,
  5. Launch a server instance from Self Service using the RightScale Self-Service Designer page, not the Catalog.

If the script has some errors, modify the Head revision of the RightScale script and launch a new server instance using the RightScale Self-Service Designer page.

When the script works perfectly:

  1. Commit the RightScale script, creating a new RightScale script revision,
  2. Attach the committed revision of the RightScale script to the Server Template’s Head revision,
  3. Commit the Server Template, creating a new Server Template revision,
  4. Publish the new revision of the Server Template,
  5. Reference the new revision of the Server Template in your Self Service CAT file.

Custom Git Status and Git Pull commands to work with all of your repositories

If you work with many Git repositories it is easy to miss a repository when you commit your changes, and very time consuming to pull from all repositories.

It is easy to automate both processes.

In this example, we will place all files in the ~/Git/devops-scripts folder. If you want to place your files at another location change it in the files below.


\/  \/  \/  \/  \/  \/  \/  \/  \/  \/

In Windows

To find your home directory on a Windows workstation, open a Bash window (Git Bash) and execute

echo ~

/\  /\  /\  /\  /\  /\  /\  /\  /\  /\


Create the Bash script files

  • Create a file with the list of Git repositories and save it as ~/Git/devops-scripts/git-repositories.txt
    You can separate groups of repositories with empty lines.
# Local_Path+Remote_Path+Rename_the_folder_to(optional)
~/Git/MY_REPO1+https://github.com/MY_REPO1.git+RENAME_TO1
~/Git/MY_REPO2+https://github.com/MY_REPO2.git

~/Git/MY_REPO3+https://github.com/MY_REPO3.git
  • Create a file to check the status of all Git repositories and save it as ~/Git/devops-scripts/git-status.sh
#!/bin/bash

# git-status.sh
# Shows the GIT STATUS of the Git repositories and shows the recommended actions
# The list is in git-repositories.txt

eval input="~/Git/devops-scripts/git-repositories.txt" # Need EVAL, otherwise the Bash loop cannot interpret ~ as the home directory if the path contains ~

RED='\033[0;31m'
NC='\033[0m' # No Color

while IFS= read -r folder
do

 # Get the length of the path
 # On Windows the empty line manifests itself as a 1 character long not empty string
 path_length=${#folder}

 if [[ ( -n "$folder" ) ]] && [[ ( 1 < $path_length ) ]] ; then
 # The path is not empty and the length is greater than 1

 # Ignore the lines starting with #
 if [[ ! $folder == \#* ]] ; then
 # The line is not a comment, process it

 # Split the local and remote path
 eval local_path=$(echo "$folder" | cut -f1 -d+) # Need EVAL, otherwise the IF statement below does not work if the path contains ~
 remote_path=$(echo "$folder" | cut -f2 -d+)

 if [ ! -d "$local_path" ]
 then

 # The local directory doesn't exist.
 echo -e "${RED}$local_path does not exist, PULL to get $remote_path${NC}"

 else

 eval cd $local_path # Need EVAL, otherwise CD does not work if the path contains ~
 pwd | tr -d '\n'

 status=$(git status)

 # --------------------------------------------------------------------------
 # To handle

 # On branch master
 # Your branch is up-to-date with 'origin/master'.
 # Untracked files:
 # (use "git add <file>..." to include in what will be committed)

 if [[ $status == *"Untracked files"* ]]; then
 echo -e " ---- ${RED}New files added, please ADD${NC}" | tr -d '\n' # -e tells echo to enable backslash escapes
 fi

 # --------------------------------------------------------------------------
 # To handle

 # On branch master
 # Your branch is up-to-date with 'origin/master'.
 # Changes not staged for commit:
 # (use "git add <file>..." to update what will be committed)
 # (use "git checkout -- <file>..." to discard changes in working directory)

 if [[ $status == *"Changes not staged for commit"* ]]; then
 echo -e " ---- ${RED}Files changed, please ADD${NC}" | tr -d '\n' # -e tells echo to enable backslash escapes
 fi

 # --------------------------------------------------------------------------
 # To handle

 # On branch master
 # Your branch is up-to-date with 'origin/master'.
 # Changes to be committed:
 # (use "git reset HEAD <file>..." to unstage)

 if [[ $status == *"Changes to be committed"* ]]; then
 echo -e " ---- ${RED}Please COMMIT${NC}" | tr -d '\n' # -e tells echo to enable backslash escapes
 fi

 # --------------------------------------------------------------------------
 # To handle

 # On branch master
 # Your branch is ahead of 'origin/master' by 1 commit.
 # (use "git push" to publish your local commits)
 # nothing to commit, working tree clean

 if [[ $status == *"git push"* ]]; then
 echo -e " ---- ${RED}Please PUSH${NC}" | tr -d '\n' # -e tells echo to enable backslash escapes
 fi

 # --------------------------------------------------------------------------
 # To handle

 # On branch master
 # Your branch and 'origin/master' have diverged,
 # and have 1 and 1 different commits each, respectively.
 # (use "git pull" to merge the remote branch into yours)
 # nothing to commit, working tree clean

 if [[ $status == *"git pull"* ]]; then
 echo -e " ---- ${RED}Repos diverged, PULL to merge${NC}" | tr -d '\n' # -e tells echo to enable backslash escapes
 fi

 # --------------------------------------------------------------------------

 echo

 fi

 fi

 fi
done < "$input"
  • Create a file to pull from all Git repositories and save it as ~/Git/devops-scripts/git-pull.sh
#!/bin/bash

# git-pull.sh
# Pulls Git repositories and clones the missing ones
# The list is in git-repositories.txt

eval input="~/Git/devops-scripts/git-repositories.txt" # Need EVAL, otherwise the Bash loop cannot interpret ~ as the home directory if the path contains ~

RED='\033[0;31m'
NC='\033[0m' # No Color

while IFS= read -r folder
do

 # Get the length of the path
 # On Windows the empty line manifests itself as a 1 character long not empty string
 path_length=${#folder}

 if [[ ( -n "$folder" ) ]] && [[ ( 1 < $path_length ) ]] ; then
 # The path is not empty and the length is greater than 1

 # Ignore the lines starting with #
 if [[ ! $folder == \#* ]] ; then
 # The line is not a comment, process it

 # Split the local and remote path
 eval local_path=$(echo "$folder" | cut -f1 -d+) # Need EVAL, otherwise the IF statement below does not work if the path contains ~
 remote_path=$(echo "$folder" | cut -f2 -d+)
 rename_folder_to=$(echo "$folder" | cut -f3 -d+)

 # Check if the local path exists
 if [ ! -d "$local_path" ]; then
 # The local directory doesn't exist.
 echo
 echo -e "${RED}$local_path does not exist, cloning $remote_path${NC}"

 # Get the parent directory name
 parent_dir="${local_path%/*}"

 # Create the parent directory
 mkdir -p $parent_dir

 # Go to the parent_dir
 cd $parent_dir

 # Clone the Git repository
 git clone $remote_path

 if [[ ( -n "$rename_folder_to" ) ]] ; then

 # Rename the folder if necessary
 end_of_url="${remote_path##*/}"
 filename="${end_of_url%.*}"

 echo -e "${RED}Renaming $filename to $rename_folder_to${NC}"

 mv $filename $rename_folder_to

 fi


 else
 echo
 eval cd $local_path # Need EVAL, otherwise CD does not work if the path contains ~
 pwd
 git pull

 fi

 fi

 fi

done < "$input"

Enable the script execution


\/  \/  \/  \/  \/  \/  \/  \/  \/  \/

 In Mac OS

  • Open a terminal window in the ~/Git/devops-scripts folder
  • Run these commands to enable the execution of the Bash files
chmod u+x git-status.sh
chmod u+x git-pull.sh

/\  /\  /\  /\  /\  /\  /\  /\  /\  /\


Create an alias

Aliases allow you to execute commands from any folder in the command line and hide the complexity of the entire command.


\/  \/  \/  \/  \/  \/  \/  \/  \/  \/

In Mac OS

If you use the iTerm terminal window

iTerm reads the configuration settings from the~/.zshrc file. If you use iTerm add the alias definitions to the end of the ~/.zshrc file.

  • Open the~/.zshrc file and add the lines below to the end of it
If you use the built-in OSX Terminal

The built-in Terminal windows reads the configuration settings from the~/.bash_profile file.

  • Open the ~/.bash_profile file and add the lines below to it

In Windows

If you use Git Bash
  • Navigate to your home directory ( cd ~ )
  • Create the Bash config file if does not exist
    cp > ~/.bashrc

    Bash will show the “cp: missing file operand” an error message, but it will create the file that you can edit with any text editor

  • Add the lines below to the file

/\  /\  /\  /\  /\  /\  /\  /\  /\  /\


 

alias status='~/Git/devops-scripts/git-status.sh'
alias pull='~/Git/devops-scripts/git-pull.sh'

Refresh the settings in the Bash shell environment


\/  \/  \/  \/  \/  \/  \/  \/  \/  \/

In MacOS

source ~/.bash_profile

In Windows

Close the Bash window and open it again

/\  /\  /\  /\  /\  /\  /\  /\  /\  /\


How to use it

Open a terminal window in any folder

To check if you need to add, commit or push files in a Git repository:

status

If you have added new files to the repository you get:

/Users/YOUR_USER_NAME/Git/devops-scripts —- New files added, please ADD

If you have changed files in a Git repository you get:

/Users/YOUR_USER_NAME/Git/devops-scripts —- Files changed, please ADD

If you have already executed the git add . command, but have not yet committed the changes:

/Users/YOUR_USER_NAME/Git/devops-scripts—- Please COMMIT

If you have already committed the changes, but have not pushed the to the remote repository:

/Users/YOUR_USER_NAME/Git/devops-scripts —- Please PUSH

To pull from every Git repository before you start to work and clone the missing ones

pull

 

 

Windows 10 can’t connect to this network

If your Windows 10 computer can successfully connect to some wireless networks, but cannot connect to certain wireless networks, it is possible, that the stored credentials are out of date. To force Windows 10 to ask you for the updated username and password

  1. In the Start Menu select Settings,
     
  2. On the Windows Settings page select Network & Internet,
  3. On the left side select Wi-Fi,
  4. On the Wi-Fi page select Manage known networks,
  5. Select the network you are not able to connect and click the Forget button,
  6. In the lower right corner of the screen select the Wi-Fi network icon, select the network, and click the Connect button.

 

Turn off the built in touchpad when a mouse is attached

All laptops today feature a touchpad, that enables you to move the pointer on the screen. It can be very annoying when your thumb accidentally moves the cursor when you type, even if an external mouse is connected to the computer.

Windows 10 provides a checkbox to automatically disable the internal pointing device when you connect a mouse.

  • Open the Control Panel,
  • Select Mouse,
  • On the Device Settings tab check the Disable internal pointing device when external USB pointing device is attached checkbox.