Custom Git Status and Git Pull commands to work with all of your repositories

If you work with many Git repositories it is easy to miss a repository when you commit your changes, and very time consuming to pull from all repositories.

It is easy to automate both processes.

In this example, we will place all files in the ~/Git/devops-scripts folder. If you want to place your files at another location change it in the files below.


\/  \/  \/  \/  \/  \/  \/  \/  \/  \/

In Windows

To find your home directory on a Windows workstation, open a Bash window (Git Bash) and execute

echo ~

/\  /\  /\  /\  /\  /\  /\  /\  /\  /\


Create the Bash script files

  • Create a file with the list of Git repositories and save it as ~/Git/devops-scripts/git-repositories.txt
    You can separate groups of repositories with empty lines.
~/Git/MY_REPO1
~/Git/MY_REPO2

~/Git/MY_REPO3
~/Git/MY_REPO4
  • Create a file to check the status of all Git repositories and save it as ~/Git/devops-scripts/git-status.sh
#!/bin/bash
input="$HOME/Git/devops-scripts/git-repositories.txt" # Use $HOME instead of ~ to refer to the home directory. The Bash loop cannot interpret ~ as the home directory

RED='\033[0;31m'
NC='\033[0m' # No Color

while IFS= read -r folder
do

 # Get the length of the path
 # On Windows the empty line manifests itself as a 1 character long not empty string 
 path_length=${#folder}

 if [[ ( -n "$folder" ) ]] && [[ ( 1 < $path_length ) ]] ; then
 # The path is not empty and the length is greater than 1

 eval cd $folder # Need EVAL, otherwise CD does not work if the path contains ~
 pwd | tr -d '\n'

 status=$(git status)

 # --------------------------------------------------------------------------
 # To handle

 # On branch master
 # Your branch is up-to-date with 'origin/master'.
 # Untracked files:
 # (use "git add <file>..." to include in what will be committed)

 if [[ $status == *"Untracked files"* ]]; then
 echo -e " ---- ${RED}New files added, please ADD${NC}" | tr -d '\n' # -e tells echo to enable backslash escapes
 fi

 # --------------------------------------------------------------------------
 # To handle

 # On branch master
 # Your branch is up-to-date with 'origin/master'.
 # Changes not staged for commit:
 # (use "git add <file>..." to update what will be committed)
 # (use "git checkout -- <file>..." to discard changes in working directory)

 if [[ $status == *"Changes not staged for commit"* ]]; then
 echo -e " ---- ${RED}Files changed, please ADD${NC}" | tr -d '\n' # -e tells echo to enable backslash escapes
 fi

 # --------------------------------------------------------------------------
 # To handle

 # On branch master
 # Your branch is up-to-date with 'origin/master'.
 # Changes to be committed:
 # (use "git reset HEAD <file>..." to unstage)

 if [[ $status == *"Changes to be committed"* ]]; then
 echo -e " ---- ${RED}Please COMMIT${NC}" | tr -d '\n' # -e tells echo to enable backslash escapes
 fi

 # --------------------------------------------------------------------------
 # To handle

 # On branch master
 # Your branch is ahead of 'origin/master' by 1 commit.
 # (use "git push" to publish your local commits)
 # nothing to commit, working tree clean

 if [[ $status == *"git push"* ]]; then
 echo -e " ---- ${RED}Please PUSH${NC}" | tr -d '\n' # -e tells echo to enable backslash escapes
 fi

 # --------------------------------------------------------------------------
 # To handle

 # On branch master
 # Your branch and 'origin/master' have diverged,
 # and have 1 and 1 different commits each, respectively.
 # (use "git pull" to merge the remote branch into yours)
 # nothing to commit, working tree clean

 if [[ $status == *"git pull"* ]]; then
 echo -e " ---- ${RED}Repos diverged, PULL to merge${NC}" | tr -d '\n' # -e tells echo to enable backslash escapes
 fi

 # --------------------------------------------------------------------------

 echo

 fi
done < "$input"

  • Create a file to pull from all Git repositories and save it as ~/Git/devops-scripts/git-pull.sh
#!/bin/bash
input="$HOME/Git/devops-scripts/git-repositories.txt" # Use $HOME instead of ~ to refer to the home directory. The Bash loop cannot interpret ~ as the home directory

while IFS= read -r folder
do

 # Get the length of the path
 # On Windows the empty line manifests itself as a 1 character long not empty string 
 path_length=${#folder}

 if [[ ( -n "$folder" ) ]] && [[ ( 1 < $path_length ) ]] ; then
 # The path is not empty and the length is greater than 1

 eval cd $folder # Need EVAL, otherwise CD does not work if the path contains ~
 pwd
 git pull

 fi

done < "$input"

Enable the script execution


\/  \/  \/  \/  \/  \/  \/  \/  \/  \/

 In Mac OS

  • Open a terminal window in the ~/Git/devops-scripts folder
  • Run these commands to enable the execution of the Bash files
chmod u+x git-status.sh
chmod u+x git-pull.sh

/\  /\  /\  /\  /\  /\  /\  /\  /\  /\


Create an alias

Aliases allow you to execute commands from any folder in the command line and hide the complexity of the entire command.


\/  \/  \/  \/  \/  \/  \/  \/  \/  \/

In Mac OS

If you use the iTerm terminal window

iTerm reads the configuration settings from the~/.zshrc file. If you use iTerm add the alias definitions to the end of the ~/.zshrc file.

  • Open the~/.zshrc file and add the lines below to the end of it
If you use the built-in OSX Terminal

The built-in Terminal windows reads the configuration settings from the~/.bash_profile file.

  • Open the ~/.bash_profile file and add the lines below to it

In Windows

If you use Git Bash
  • Navigate to your home directory ( cd ~ )
  • Create the Bash config file if does not exist
    cp > ~/.bashrc

    Bash will show an error message, but it will create the file that you can edit with any text editor

  • Add the lines below to the file

/\  /\  /\  /\  /\  /\  /\  /\  /\  /\


 

alias status='~/Git/devops-scripts/git-status.sh'
alias pull='~/Git/devops-scripts/git-pull.sh'

Refresh the settings in the Bash shell environment


\/  \/  \/  \/  \/  \/  \/  \/  \/  \/

In MacOS

source ~/.bash_profile

In Windows

Close the Bash window and open it again

/\  /\  /\  /\  /\  /\  /\  /\  /\  /\


How to use it

Open a terminal window in any folder

To check if you need to add, commit or push files in a Git repository:

status

If you have added new files to the repository you get:

/Users/YOUR_USER_NAME/Git/devops-scripts —- New files added, please ADD

If you have changed files in a Git repository you get:

/Users/YOUR_USER_NAME/Git/devops-scripts —- Files changed, please ADD

If you have already executed the git add . command, but have not yet committed the changes:

/Users/YOUR_USER_NAME/Git/devops-scripts—- Please COMMIT

If you have already committed the changes, but have not pushed the to the remote repository:

/Users/YOUR_USER_NAME/Git/devops-scripts —- Please PUSH

To pull from every Git repository before you start to work

pull

 

 

Add SSH key to a Jenkins Git step

To access a Git repository Jenkins can use an SSH key.

To add the SSH key to the Jenkins server use the following Chef script

Store the SSH key in an encrypted data bag called “keys”.

{
 "id": "ci_private_keys",
 "ci_github_key": "-----BEGIN RSA PRIVATE KEY-----\n...\n-----END RSA PRIVATE KEY-----",
}

 

Add the following to the Jenkins Chef recipe

  • Install Git
package 'git'
  • Install the Git and Credentials Jenkins plugins
jenkins_plugin 'git'
jenkins_plugin 'credentials'
  • Copy the SSH key to the Jenkins server
rsa_key = data_bag_item('keys', 'ci_private_keys')
file '/var/lib/jenkins/.ssh/id_rsa' do
  content "#{rsa_key['ci_github_key']}"
  owner 'jenkins'
  group 'jenkins'
  mode '0600'
end
  • Add github.com to the known hosts
bash 'provide github.com RSA fingerprint' do
  code <<-EOF
   ssh-keyscan github.com >> /var/lib/jenkins/.ssh/known_hosts
   chown jenkins.jenkins /var/lib/jenkins/.ssh/known_hosts
  EOF
  not_if{system('grep github.com /var/lib/jenkins/.ssh/known_hosts')}
end

 

To specify the SSH key in the Git step

  1. When the Jenkins server is operational, navigate to the Web interface
  2. Create a new Jenkins project
  3. In the Source Code Management section
    1. Select Git
    2. Enter the SSH URL of the repository
    3. When you are adding the first project, click the Add button to create the credential

      1. Click Jenkins to select the credentials provider
      2. Select SSH Username with private key as the Kind
      3. Enter the username you used when you created the SSH key for the Git repository
      4. Select From the Jenkins master ~/.ssh as the Private Key
      5. Click the Add button
    4. In the Credentials drop down select the credential you have created (the Git user name)

 

How to edit the GitHub README.md file

The GitHub repositories usually contain a README.md file to describe how to use the project. The GitHub web site has a simple editor, but it has a few limitations

  • If you accidentally refresh the page, you lose your changes,
  • The preview pane is wider than the page that will display your file, so the formatting can be different,
  • You have to switch between the editor and the the preview every time you want to see the result.

One of the simplest ways to edit the README.md is to use the Atom editor.

  • Open the README.md file with Atom,
  • In the Packages menu select Markdown Preview, or press Ctrl-Shitft-M to open the preview pane.

As you type in the editor window the preview pane will show the live review of the file.

For the Markdown syntax visit https://guides.github.com/pdfs/markdown-cheatsheet-online.pdf

Permission denied message when you try to upload your new repository to GitHub from a Windows computer

If you work on a Windows computer and create a new GitHub repository, you can copy the code from the GitHub page to set the remote address and push the existing code to the GitHub server. You may get the following error message:

$ git push -u origin master
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Make sure you have selected the HTTPS button to see the correct lines for Windows.
github-01-create-repo

To change to address of the remote server execute the following
git remote set-url origin https://github.com/.....

Switching to 2 factor authentication in GitHub

When you switch to 2 factor authentication GitHub generates a 40 character Personal Access Token that you can use as a password to access Git repositories.

Create a Personal Access Token to use it as password in the command line tools

  • Log into GitHub and in the pull down at the upper right select Settings
  • On the left select Personal access tokens and click the Generate new token button
  • Enter a description for your token, so you can revoke it later in case of a  security breach
  • Copy the 40 character long token and use it as your password on the command line.

Update your username or password in the Git credential helper

When you have already stored your password in the credential helper of Git and switching to 2 factor authentication, or just want to update your GitHub username or password

  • In the command line execute the following to instruct the Git client to ask for your username and password
    git config --global credential.helper wincred
  • The next time you will access the GitHub repository to fetch, clone or push changes the Git CLI will pup up a dialog box to ask your username and password.

Display the branch structure and other important events in the Git command line interface

To set up the display of the branch structure, tags and pulls in the command line interface first create the following “alias”

git config --global alias.lgb "log --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)<%an>%Creset%n' --abbrev-commit --date=relative --branches"

To display the branch structure, execute the following “alias”

git lgb

This will execute the above defined “alias” and display the most important events of the branch.

Press q to exit from the display mode to the $ prompt.

Branching and merging in Git

Branching and Merging are very powerful features in Git.

There are may branching strategies, my favorite is well explained at http://nvie.com/posts/a-successful-git-branching-model/

Pick any model you like and stick to it for a while to better understand and evaluate it. If you do not like that try another one.

In this model when we start to work on a new feature we

  • Create a new branch (let’s call it feature) based on the develop branch and switch to it.
    • git checkout develop
      git branch feature
      git checkout feature
  • Develop the new feature in the feature branch and add the changes to the repository.
    • git add .
      git commit -m "Feature is done in feature branch"
  • Before merging the changes of the feature branch back into the develop branch get the latest changes from the develop branch, apply your changes on top of them and test the application
    • git rebase develop
  • If there are merge conflicts, launch the Merge Tool. See Merging in Git with four panels in Windows  to set up and use a great merge tool.
    • git mergetool
  • It is possible that you will encounter merge conflicts in the same file multiple times. Just repeat the git mergetool command as many times as needed.
  • When you are done with merging, complete the rebase
    • git rebase --continue
  • Add and commit your changes in the feature branch
    • git add .
      git commit -m "Feature is tested in feature branch"
  • When you have tested the code of the new feature, merge it back to the develop branch.
    Switch back to the develop branch and merge the changes from the feature branch.

    • git checkout develop
      git merge feature
  • If there are any merge conflicts, Git will ask you to resolve them.

Merging in Git with four panels in Windows

Git is a great repository for small and large projects. It is easy to create and merge branches to separate code for the features you work on. To make merging easier you can use a free 4 panel merging tool, Perforce P4Merge.

Download

  • Download the Perforce P4Merge Visual Merge Tool from https://www.perforce.com/downloads/integrations
  • Select the operating system of your computer and click the Download button.

Install Perforce P4Merge

  • Double click the downloaded EXE file
  • Unselect the Administration tool
    P4Merge 01
  • Leave the server as is, we will not use it
    P4Merge 02
  • Just click OK, the address is not important for us
    P4Merge 03
  • Click Next again on the Client Configuration page
    P4Merge 04
  • On a Windows machine P4Merge will be installed at C:\Program Files\Perforce. Git cannot access P4Merge if the path contains a space character, so once the installation is done move the Perforce folder to the root of the C: drive.

Add the Perforce P4Merge settings to the Git config file

  • To use the same tool in every repository navigate to C:\Users\[YOUR USER NAME] and open the .gitconfig file.
  • If you already have [merge] and [mergetool “p4merge”] entries update them, if not, add the following lines. Even on Windows machines you have to use forward slashes (/) in the path for Git to understand it.
    [merge]
      tool = p4merge
    
    
    [mergetool "p4merge"]
      cmd = "C:/Perforce/p4merge" "$BASE" "$LOCAL" "$REMOTE" "$MERGED"
      keepTemporaries = false
      trustExitCode = false
      keepBackup = false

Solve merge conflicts

  • If you use the Git command line merge the branches. Git is case sensitive.
    • Change to the target branch
      • git checkout master
    • Merge the changes from the feature branch
      • git merge Feature
    • If there is a merge conflict type
      • git mergetool
    • Git will display the name of the file that caused the merge conflict
      • $ git mergetool
        Merging:
        Test1.txt
        Normal merge conflict for 'Test1.txt':
          {local}: modified file
          {remote}: modified file
        Hit return to start merge resolution tool (p4merge):
    • Hit Enter and the P4Merge opens with four panels
      • The left (Local) panel shows the changes in the current (target) branch,
      • The middle panel shows the original (Base) version of the file before both changes,
      • The right panel shows the coming changes (Remote) that were made in the branch you are merging from,
      • The bottom shows the result of the merge.
    • Change the file to resolve the conflict and click the Save button to save the result.

Merging is safe, because Git creates backup copies of all three version of the file in the same folder. If there are multiple files with merge conflicts Git will open the merge tool for each of them one-by-one.

  • Add the changes to the next commit
    • git add .
  • Commit the changes
    • git commit -m “Merged with P4Merge”

How to delete a folder from Git, but not from the local disk

I have accidentally included a folder in my Git repository, that I don’t want to include in source control. I wanted to remove it from the Git index, but not from the local hard drive.

The solution is the following command

git rm -r --cached myFolder

If the folder name contains spaces enclose it in quotes "my folder"

This will remove the folder from the Git repository, but leave it on the local drive.