Tag: DevOps

Using tmux terminal multiplexer

Posted on by Laszlo Pinter

The tmux terminal multiplexer allows us to open multiple terminal windows in the same SSH session and continue the command execution even when we log out of the SSH session. This way we can execute long-running copy commands overnight without keeping the SSH session open.

Install tmux

On CentOS family Linux

yum install tmux

To start tmux and attach to the last session

tmux a

To keep the current session active

To continue the session execution even when you log out of the server, detach from the session before closing the connection. See the commands below.

Commands

To switch to binding mode

ctrl-b

Commands in binding mode

d detach
% vertical split
“ horizontal split
arrows move between panes
c new window
p previous window
n next window
l last used pane
q display pane numbers
[ enable scroll with arrow keys or pgup, pgdown
q quit scroll mode
x close the pane (will ask you to press “y” to confirm)

Zooming

cmd-shift + zoom in
cmd-shift – zoom out
cmd-shift 0 (zero) 100% zoom

Add SSH key access to a GitHub repository

Posted on by Laszlo Pinter

Generate an SSH key pair

  1. In a terminal execute
ssh-keygen

Generating public/private rsa key pair.
Enter file in which to save the key : /Users/MY_USERNAME/Git/_Keys/MY_PROJECT/MY_PROJECT_rsa_ci
Enter passphrase (empty for no passphrase):
Enter same passphrase again:

Leave the passphrase empty, many systems cannot work with password protected key pairs.

This process will save the key-pair in two files. The private key with no extension, and the public key with the .pub extension.

Create a new service GitHub user account which will have access to the repository

  1. Create a new email address (GitHub requires unique email addresses for every user)
  2. Register a new user which will have access to the repository

Upload the public key to the service GitHub account

    1. Log into the GitHub account and select Settings in the drop-down in the upper right corner
    2. On the left side select SSH and GPG keys
    3. In the upper right corner select the New SSH key button
    4. Copy the above generated public key to your clipboard.
      On a Mac copy the public key to your clipboard with the command

      pbcopy < /Users/MY_USERNAME/Git/_Keys/MY_PROJECT/MY_PROJECT_rsa_ci.pub
    5. Paste the public key into the textbox

Add the service user as a collaborator to the GitHub repository

  1. Log into the GitHub account that has admin access to the repository
  2. Navigate to the repository and select Settings
  3. Select COllaborators & teams
  4. Enter the username into the search box and click Add collaborator

If you use two-factor authentication in Github

If you use two-factor authentication in your GitHub account, and you need automated access to it, create a Personal Access Token and use it instead of your password. GitHub will not ask you to verify your identity.

Generate a Personal Access Token

    1. Log into your GitHub account
    2. On the left side select Developer settings
    3. Select Personal access tokens
    4. Click the Generate new token button
    5. Select the repo checkbox
    6. Copy the token to your clipboard. This is the last time you are able to see the token.
    7. Save the token at a secure location, and use it instead of your password when you need automated access to your GitHub account.

Creating CentOS server images with Packer

Posted on by Laszlo Pinter

The CentOS images are not available on the AWS Quick Start tab.

CentOS publishes official images on the AWS Marketplace, but you need to subscribe to the image to be able to launch it with an automation software, like Terraform.

Find the latest available CentOS image in the AWS Marketplace

  1. Execute this command to display the list of available images 
    aws --region us-east-1 ec2 describe-images --owners aws-marketplace --filters Name=product-code,Values=aw0evgkw8e5c1q413zgy5pjce
  2. Select the latest AMI from the list. The images are NOT ordered by date!
  3. If you try to launch the image before you subscribe to it, the error message is displayed

    Error launching source instance: OptInRequired: In order to use this AWS Marketplace product you need to accept terms and subscribe. To do so please visit http://aws.amazon.com/marketplace/pp?sku=…
    amazon-ebs: status code: 401, request id: []

  4. Follow the link in the error message to http://aws.amazon.com/marketplace/pp?sku=…
  5. Click the Continue to Subscribe button
  6. Once the new image has been created, share it with all the accounts you use.

SocketError: Error connecting to … Name or service not known

Posted on by Laszlo Pinter

In an enterprise environment, the company usually operates its own DNS servers.

When a Linux instance launches in AWS, the DNS settings only contain the AWS DNS server. If the company DNS server settings are applied with Chef, during the first Chef Client run those settings do not take effect.

When we reference an internal DNS entry during the first Chef Client run we get the error message:

 

SocketError: Error connecting to https://INTERNAL_ADDRESS.net/… – Failed to open TCP connection to INTERNAL_ADDRESS.net:443 (getaddrinfo: Name or service not known)

There two solutions to handle this:

  1. Use the IP address of the internal server or load balancer in the URL ( switch from HTTPS to HTTP, because the SSL certificate will not work with the IP address),
  2. Set the DNS servers of the Linux server with Packer when you generate the base image.

The specified version of the NuGet package is not found in Artifactory

Posted on by Laszlo Pinter

The “chocolatey_package” Chef resource can install NuGet packages from Artifactory.

Artifactory is inconsistent in case sensitivity when an application is searching for a NuGet package. When we specify the package ID only, the search is not case sensitive. If the package is called “GoogleChrome” and we search for “googlechrome” the NuGet package is found in Artifactory.

chocolatey_package 'googlechrome' do
  source 'devops-chocolatey'
  options "--allow-empty-checksums --ignore-package-exit-codes"
end

When we specify the version of the package, the search becomes case sensitive in Artifactory. The “chocolatey_package” Chef resource automatically converts the package IDs to lowercase, even if we spell the package ID the same as it is in Artifactory, so “GoogleChrome” can never be found in Artifactory when we specify the NuGet package version.

chocolatey_package 'GoogleChrome' do
  version '66.0.3359.18100'
  source 'devops-chocolatey'
  options "--allow-empty-checksums --ignore-package-exit-codes"
end

The following error message is misleading. In this case, the version was in Artifactory, but the spelling of the package ID is not all lower case in Artifactory.

googlechrome not installed. The package was not found with the source(s) listed.

If you specified a particular version and are receiving this message, it is possible that the package name exists but the version does not.
Version: “66.0.3359.18100”
Source(s): “http://artifactory….”

When we internalize the “googlechrome” Chocolatey package, the package ID in the NuGet package will be spelled as “GoogleChrome”. The “internalize” process downloads the NuGet package from the Chocolatey server, downloads the necessary installer files from the source URL specified in the NuGet package, and creates a new NuGet package that contains all necessary files for the software installation. To have access to these packages any time, even when the Chocolatey package is no longer available at Chocolatey.org, we can upload them to Artifactory or other NuGet repository.

To solve the case sensitivity issue, currently, the only solution is to unzip the package, change the spelling of the ID to lower case and re-create the package again.

  1. Install Chocolatey on your Windows workstation. See Install Chocolatey
  2. Open a Windows command prompt as Administrator
  3. Internalize the package 
    choco download googlechrome --internalize
  4. Unzip the NuGet package
    1. Right-click the .nupkg file and select 7-Zip, Extract to “…”
    2. Open the .nuspec file in a text editor
    3. Change the ID to all lower case 
      <id>googlechrome</id>
    4. Save the .nuspec file
  5. Re-create the NuGet package
    1. Delete the existing _rels folder, the package creation will recreate it with updated information
    2. Right-click the .nuspec file and select Compile Chocolatey Package…
  6. Upload the NuGet package to Artifactory.

 

Copy files between Windows and Linux computers

Posted on by Laszlo Pinter

There are many tools to copy files between Windows and Linux computers, I have found this method the simplest.

It does not require any software installation on the Windows machine, and only one package installation on the Linux machine.

Share a folder on the Windows machine

Share a folder on the Windows machine and allow access to it for a user. If the Windows computer is in the Windows domain, the domain user does not have to be a member of any security group on the Windows machine.

If you copy files from Windows to Linux, make the folder read-only for the user. If you copy files to the Windows machine, allow write access to the folder for the user.

Set up the Linux machine

  1. Install the cifs-utils on the Linux machine
    1. On Red Hat, CentOS, and Amazon Linux 
      sudo yum install cifs-utils
    1. On Ubuntu 
      sudo apt-get install cifs-utils

Mount the shared Windows folder on the Linux machine

  1. On the Linux machine create a directory to mount the Windows folder to 
    mkdir /tmp/windows
  2. Mount the Windows share 
    sudo mount.cifs '\\WINDOWS_SERVER_IP\attachments' /tmp/windows -o domain=MY_DOMAIN,username=MY_USERNAME,password=MY_PASSWORD,vers=1.0

    First, you will be asked for the root password on the Linux machine.
    If you do not specify your password in the line above, you will be also asked to enter your password on the Windows machine.

Access the Windows share

  1. On the Linux machine navigate to the mount directory 
    cd /tmp/windows
  2. List the files of the Windows share 
    ls -al

Troubleshooting

If you get the error message when you issue the mount command

mount error(16): Device or resource busy

try to unmount (umount !) the share first and try the mount again

umount /tmp/windows

 

Edit the HKEY_CURRENT_USER Windows Registry keys of another user

Posted on by Laszlo Pinter

The user-specific settings in the Windows registry are stored under the HKEY_CURRENT_USER key. If you open the Regedit.exe application the HKEY_CURRENT_USER key contains the settings for your user account.

To access the registry keys of another user we need to

Find the Security ID of the user

  1. In Regedit navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\hivelist
  2. The key lists the Security IDs and usernames
  3. Save the Security ID of the user.

Another Security ID list location:

The partial list of the Security IDs is also available at
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileList

Not all user profiles are listed here.

Click the Security ID folders on the left to see the username in the ProfileImagePath.

Open the user-specific registry keys

  1. In Regedit navigate to the HKEY_USERS key
  2. Select the Security ID of the user you are looking for
  3. The user-specific values are under that key

 

“Ran out of time waiting for the server with id” with Windows Server 2016 in Chef Test Kitchen

Posted on by Laszlo Pinter

AWS changed how Windows Server EC2 instances send messages during boot.

Windows Server 2012 R2 AWS EC2 instances sent the “Windows is ready” message every time those became available after boot.

When a Windows Server 2016 AWS EC2 instance launches, it only sends the “Windows is ready” message during the first boot. If you create your custom AMI with Packer, the first boot happens during the Packer AMI creation, so Chef Test Kitchen does not receive the expected message and the process times out with the message

>>>>>> Ran out of time waiting for the server with id [i-… to become ready, attempting to destroy it
EC2 instance <i-…> destroyed.

To configure Windows Server 2016 instances to send the expected “Windows is ready” message during every boot, add a PowerShell line to your Packer template that creates your custom AMI:

{
  "type": "powershell",
  "pause_before":"2s",
  "elevated_user": "MY_ADMIN_USER",
  "elevated_password": "MY_ADMIN_PASSWORD",
  "inline": [
    "C:\\ProgramData\\Amazon\\EC2-Windows\\Launch\\Scripts\\SendWindowsIsReady.ps1 -Schedule"
  ]
},

This code creates a scheduled task to execute the configuration script on the instance.

More information is at https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/ec2launch.html#ec2launch-sendwinisready