Copy files between Linux machines

The rsync command allows you to copy files using SSH connection, between your workstation and another Linux machine. You have to be logged into one of the machines, this command cannot copy files between two remote machines.

To copy a file from your local machine to a remote server

rsync -avz -e "ssh -i SSH_KEY_FILE -o 'StrictHostKeyChecking=no'" --rsync-path="sudo rsync" SOURCE_FILE USER_NAME@SERVER_IP:TARGET_DIRECTORY/

To copy a file from a remote server to your local workstation

rsync -avz -e "ssh -i SSH_KEY_FILE -o 'StrictHostKeyChecking=no'" --rsync-path="sudo rsync" USER_NAME@SERVER_IP:SOURCE_FILE TARGET_DIRECTORY/

Using tmux terminal multiplexer

The tmux terminal multiplexer allows us to open multiple terminal windows in the same SSH session and continue the command execution even when we log out of the SSH session. This way we can execute long-running copy commands overnight without keeping the SSH session open.

Install tmux

On CentOS family Linux

yum install tmux

To start tmux and attach to the last session

tmux a

To keep the current session active

To continue the session execution even when you log out of the server, detach from the session before closing the connection. See the commands below.

Commands

To switch to binding mode

ctrl-b

Commands in binding mode

d Detach
% Vertical split
“ Horizontal split
arrows Move between panes
c Create new pane
p Previous window
n Next window
l Last used window
q Display window numbers
[ Enable scroll with arrow keys or pgup, pgdown. Press Esc or q to return normal mode
x Close the pane (will ask you to press “y” to confirm)

Zooming

cmd-shift + zoom in
cmd-shift – zoom out
cmd-shift 0 (zero) 100% zoom

Shared sessions

tmux -S /tmp/shared_session Start a new shared session
chmod 777 /tmp/shared_session Allow anyone to join your session
tmux -S /tmp/shared_session attach Attach to a shared session

Add SSH key access to a GitHub repository

Generate an SSH key pair

  1. In a terminal execute
ssh-keygen

Generating public/private rsa key pair.
Enter file in which to save the key : /Users/MY_USERNAME/Git/_Keys/MY_PROJECT/MY_PROJECT_rsa_ci
Enter passphrase (empty for no passphrase):
Enter same passphrase again:

Leave the passphrase empty, many systems cannot work with password protected key pairs.

This process will save the key-pair in two files. The private key with no extension, and the public key with the .pub extension.

Create a new service GitHub user account which will have access to the repository

  1. Create a new email address (GitHub requires unique email addresses for every user)
  2. Register a new user which will have access to the repository

Upload the public key to the service GitHub account

    1. Log into the GitHub account and select Settings in the drop-down in the upper right corner
    2. On the left side select SSH and GPG keys
    3. In the upper right corner select the New SSH key button
    4. Copy the above generated public key to your clipboard.
      On a Mac copy the public key to your clipboard with the command

      pbcopy < /Users/MY_USERNAME/Git/_Keys/MY_PROJECT/MY_PROJECT_rsa_ci.pub
    5. Paste the public key into the textbox

Add the service user as a collaborator to the GitHub repository

  1. Log into the GitHub account that has admin access to the repository
  2. Navigate to the repository and select Settings
  3. Select COllaborators & teams
  4. Enter the username into the search box and click Add collaborator

If you use two-factor authentication in Github

If you use two-factor authentication in your GitHub account, and you need automated access to it, create a Personal Access Token and use it instead of your password. GitHub will not ask you to verify your identity.

Generate a Personal Access Token

    1. Log into your GitHub account
    2. On the left side select Developer settings
    3. Select Personal access tokens
    4. Click the Generate new token button
    5. Select the repo checkbox
    6. Copy the token to your clipboard. This is the last time you are able to see the token.
    7. Save the token at a secure location, and use it instead of your password when you need automated access to your GitHub account.

Creating CentOS server images with Packer

The CentOS images are not available on the AWS Quick Start tab.

CentOS publishes official images on the AWS Marketplace, but you need to subscribe to the image to be able to launch it with an automation software, like Terraform.

Find the latest available CentOS image in the AWS Marketplace

  1. Execute this command to display the list of available images 
    aws --region us-east-1 ec2 describe-images --owners aws-marketplace --filters Name=product-code,Values=aw0evgkw8e5c1q413zgy5pjce
  2. Select the latest AMI from the list. The images are NOT ordered by date!

Accept the terms

In every account, you want to launch the image, you have to accept the license agreement terms

If you try to launch the image before you subscribe to it, the error message is displayed

Error launching source instance: OptInRequired: In order to use this AWS Marketplace product you need to accept terms and subscribe. To do so please visit http://aws.amazon.com/marketplace/pp?sku=…
amazon-ebs: status code: 401, request id: []

  1. Follow the link in the error message to http://aws.amazon.com/marketplace/pp?sku=…
  2. Click the Continue to Subscribe button
  3. Click the Accept Terms button
  4. The next page should display
  5. Once the new image has been created, share it with all the accounts you use.

SocketError: Error connecting to … Name or service not known

In an enterprise environment, the company usually operates its own DNS servers.

When a Linux instance launches in AWS, the DNS settings only contain the AWS DNS server. If the company DNS server settings are applied with Chef, during the first Chef Client run those settings do not take effect.

When we reference an internal DNS entry during the first Chef Client run we get the error message:

 

SocketError: Error connecting to https://INTERNAL_ADDRESS.net/… – Failed to open TCP connection to INTERNAL_ADDRESS.net:443 (getaddrinfo: Name or service not known)

  1. There are two solutions to handle this:
  2. Use the IP address of the internal server or load balancer in the URL ( switch from HTTPS to HTTP, because the SSL certificate will not work with the IP address),
  3. Set the DNS servers of the Linux server with Packer when you generate the base image.

The specified version of the NuGet package is not found in Artifactory

The “chocolatey_package” Chef resource can install NuGet packages from Artifactory.

Artifactory is inconsistent in case sensitivity when an application is searching for a NuGet package. When we specify the package ID only, the search is not case sensitive. If the package is called “GoogleChrome” and we search for “googlechrome” the NuGet package is found in Artifactory.

chocolatey_package 'googlechrome' do
  source 'devops-chocolatey'
  options "--allow-empty-checksums --ignore-package-exit-codes"
end

When we specify the version of the package, the search becomes case sensitive in Artifactory. The “chocolatey_package” Chef resource automatically converts the package IDs to lowercase, even if we spell the package ID the same as it is in Artifactory, so “GoogleChrome” can never be found in Artifactory when we specify the NuGet package version.

chocolatey_package 'GoogleChrome' do
  version '66.0.3359.18100'
  source 'devops-chocolatey'
  options "--allow-empty-checksums --ignore-package-exit-codes"
end

The following error message is misleading. In this case, the version was in Artifactory, but the spelling of the package ID is not all lower case in Artifactory.

googlechrome not installed. The package was not found with the source(s) listed.

If you specified a particular version and are receiving this message, it is possible that the package name exists but the version does not.
Version: “66.0.3359.18100”
Source(s): “http://artifactory….”

When we internalize the “googlechrome” Chocolatey package, the package ID in the NuGet package will be spelled as “GoogleChrome”. The “internalize” process downloads the NuGet package from the Chocolatey server, downloads the necessary installer files from the source URL specified in the NuGet package, and creates a new NuGet package that contains all necessary files for the software installation. To have access to these packages any time, even when the Chocolatey package is no longer available at Chocolatey.org, we can upload them to Artifactory or other NuGet repository.

To solve the case sensitivity issue, currently, the only solution is to unzip the package, change the spelling of the ID to lower case and re-create the package again.

  1. Install Chocolatey on your Windows workstation. See Install Chocolatey
  2. Open a Windows command prompt as Administrator
  3. Internalize the package 
    choco download googlechrome --internalize
  4. Unzip the NuGet package
    1. Right-click the .nupkg file and select 7-Zip, Extract to “…”
    2. Open the .nuspec file in a text editor
    3. Change the ID to all lower case 
      <id>googlechrome</id>
    4. Save the .nuspec file
  5. Re-create the NuGet package
    1. Delete the existing _rels folder, the package creation will recreate it with updated information
    2. Right-click the .nuspec file and select Compile Chocolatey Package…
  6. Upload the NuGet package to Artifactory.

 

Scale inserted videos with the correct aspect ratio in Adobe Premiere CC

When we insert a video clip into our timeline that has the same height, but different width than our project, Adobe Premiere CC distorts the inserted clip.

My project is 1920 x 1080, the inserted clip is 1440 x 1080. When I insert the narrower clip into my timeline, Adobe Premiere stretches the inserted clip horizontally.

To set the correct aspect ratio

  1. Select the inserted clip in the timeline
  2. In the upper left region select Effect Controls
  3. Expand the Motion element
  4. Uncheck the Uniform Scale checkbox
  5. Set the Scale Width to the appropriate value, in my case it was 75% (1440 / 1920 )

Copy files between Windows and Linux computers

There are many tools to copy files between Windows and Linux computers, I have found this method the simplest.

It does not require any software installation on the Windows machine, and only one package installation on the Linux machine.

Share a folder on the Windows machine

Share a folder on the Windows machine and allow access to it for a user. If the Windows computer is in the Windows domain, the domain user does not have to be a member of any security group on the Windows machine.

If you copy files from Windows to Linux, make the folder read-only for the user. If you copy files to the Windows machine, allow write access to the folder for the user.

Set up the Linux machine

  1. Install the cifs-utils on the Linux machine
    1. On Red Hat, CentOS, and Amazon Linux 
      sudo yum install cifs-utils
    1. On Ubuntu 
      sudo apt-get install cifs-utils

Mount the shared Windows folder on the Linux machine

  1. On the Linux machine create a directory to mount the Windows folder to 
    mkdir /tmp/windows
  2. Mount the Windows share 
    sudo mount.cifs '\\WINDOWS_SERVER_IP\attachments' /tmp/windows -o domain=MY_DOMAIN,username=MY_USERNAME,password=MY_PASSWORD,vers=1.0

    First, you will be asked for the root password on the Linux machine.
    If you do not specify your password in the line above, you will be also asked to enter your password on the Windows machine.

Access the Windows share

  1. On the Linux machine navigate to the mount directory 
    cd /tmp/windows
  2. List the files of the Windows share 
    ls -al

Troubleshooting

If you get the error message when you issue the mount command

mount error(16): Device or resource busy

try to unmount (umount !) the share first and try the mount again

umount /tmp/windows