Python basics

Great Python introductory video for C# developers
https://www.youtube.com/watch?v=jj60geqaP08

__ (double underscore) Starting the attribute name with double underscore attribute makes it private. To expose the private attributes and methods

print(dir(MY_CLASS))

Code hierarchy

In Python indentation denotes the hierarchy of the instructions, not curly braces, like in C# or Java.

Directories

When you create a directory in your project, create an empty __init__.py file in the new directory to be able to reference objects from that directory, and avoid

from MY_FOLDER.MY_FILE import MY_CLASS causes an error:
ImportError: No module named MY_FOLDER.MY_FILE

Classes

Predefined methods

__init__(self) Initializer method (constructor)

__repr__(self) Representation

__iter__(self) Method to define hot to iterate through the object

Decorators

@property Exposes a method as a property of the class

from _04_anonymous.anonobject import AnonObject
ImportError: No module named _04_anonymous.anonobject

{(“Not importing directory ‘/Users/pinterl/Git/devops-scripts/Python_Tutorial/Python/Python_for_C_sharp_developers/_04_anonymous’: missing __init__.py”, <type ‘exceptions.ImportWarning’>, 6): True}

Project management

Create a new project in PyCharm

In the File menu select New Project…,
On the left side select the Python framework you want to use,
Enter the name for your project to the end of the Location path,
If you have selected a framework that asks for more information
1. Click the More Settings link and enter the additional values,
2. Click the Create button,
3. PyCharm generates folders for your project and application,
4. When PyCharm asks you about compatibility inspection, enable it, to make sure your code is compatible with future versions of Python.
5. PyCharm will enable Code compatibility inspection. Click OK to save the setting.
6. PyCharm will still set your project interpreter to 2.6, so set it to 2.7. In the PyCharm menu select Preferences…,
7. On the left side select Project:…, Project Interpreter,
8. In the Project Interpreter drop-down select 2.7 and click the OK button.

Delete a project in PyCharm

The PyCharm project explorer does not have a delete option. To delete a PyCharm project

Open the project location in Finder,
Move the project folder to the Trash,
The PyCharm project explorer window will reflect the change.

Shortcuts, tips, and tricks

Auto generate

Auto generate functions, classes, attributes

Enter the name of the undefined element where you want to use it,
Click the light bulb, or press ⌥⏎ (Alt-Enter) to pop up the Intention Action alert,
Select the action to generate the definition.

Auto generate fields for classes

Enter the __init__ function of the class with the field names,
Place the cursor on the field you want to generate the definition for,
Click the light bulb, or press ⌥⏎ (Alt-Enter) to pop up the Intention Action alert,
Select Add field ‘…’ to class …,
Press Enter to select the option,
Press Enter at the end of the new line to move the cursor to the next line, otherwise, the Intention Action alert does not pop up when you press ⌥⏎ (Alt-Enter)

Code completion

Press Ctrl-Space to complete the name of the element.
Press Ctrl-Space twice to complete the name of any class, even if it is not yet imported. The import statement will be automatically added.

Quick documentation

To get a short documentation on an element under the cursor, press F1. (By default on a Macintosh you need to press fn-F1, because the F1 button is programmed to adjust the brightness.)

For more information

Auto completing code, see https://www.jetbrains.com/help/pycharm/auto-completing-code.html

Keyboard shortcuts
https://www.jetbrains.com/help/pycharm/keyboard-shortcuts-and-mouse-reference.html

Create a Terraform Enterprise environment

If you just start to work with Terraform Enterprise, you need to create a Terraform environment.

Preparation

GitHub account

To access GitHub from Terraform Enterprise, create a GitHub team and account with admin access to the GitHub repository that will store the Terraform scripts.

Create a GitHub team who will have admin access to the Terraform script repository,
Create a GitHub user what Terraform Enterprise will use to access the Terraform script repository,
Add the new GitHub user to your GitHub organization,
Log into GitHub with the new user account credentials and wait for the email verification email from GitHub,
Open the verification email from GitHub and click the button in the email to verify your email address,
Stay logged into GitHub with the new user account and wait for the invitation email from GitHub to join the organization,
Open the invitation email from GitHub and click the button to accept the invitation,
In GitHub add the new GitHub user to the new team,
Create a GitHub repository to store the Terraform scripts,
Add the new team to the Terraform script repository with admin rights.

If the Terraform modules are in a separate GitHub repository, add the new team to that repository with Read rights.

AWS account

Create a new user account that Terraform Enterprise will use to access AWS

Create a new AWS user,
Add user rights
1. AmazonEC2FullAccess
2. AmazonS3FullAccess

Connect your Atlas account to GitHub

We will authorize Terraform Enterprise to access the Terraform script GitHub repository. We set this connection up in our personal profile in Terraform enterprise, but when we create new Terraform environments Terraform Enterprise will use this connection to access the GitHub repository the environment will be connected to. Do not use your personal GitHub account to make the connection, because if your personal account loses access to the GitHub repository, the Terraform environment will not be able to connect to it. For this connection, we have already created a new user in GitHub above.

In a web browser log into GitHub with the user account you want Terraform Enterprise use to access the GitHub repository,
In the same web browser open a new tab and navigate to https://atlas.hashicorp.com/settings/connections and log into Atlas with your Hashicorp account,
Click Connect GitHub to Atlas
On the GitHub authorization page click the Authorize hashicorp button,
Later if you want to unlink Atlas from your GitHub account, in the Personal section select Connections, and click the Unlink button.

Create the GitHub repository

Create a repository in GitHub to store the Terraform config files. You can specify subdirectories for each Terraform environment to watch, so one repository can serve the security group creation, and EC2 instance creation for the same developer group.

Create the GitHub repository,
Create a folder that the Terraform Enterprise environment will monitor for changes.

Create a new environment

1. In a web browser navigate to https://atlas.hashicorp.com/configurations/import,
2. Log in with your Atlas user account,
3. In the dropdown list of the New Environment page (click the text of the dropdown, not the arrow) select Link to GitHub.
  Name the environment to include the type of the server you want to launch and the server environment, for example, test_linux_sandbox.
  To save the environment click the Create & Continue button,
4. On the left side select Variables,
5. In the Environment section click the Edit button,
If the environment is linked to the Terraform Enterprise Server
1. The environment has been created. Click the your account settings page to create a personal token. It will allow you to access your account without using your username and password.
2. Enter a description for the token and click the Generate Token button,
3. Save the token value at a safe place
4. To return back to the Runs page select Terraform Enterprise in the upper left corner
5. Click the name of the environment you want to work with
6. Configure the environment. On the left side click Settings
  1. Uncheck Plan on artifact uploads
7. Store your personal access token on your workstation to be able to access Terraform Enterprise. Open a Bash window and execute the export command:
```
export ATLAS_TOKEN=[YOUR AUTHENTICATION TOKEN]
```
8. Add the terraform backend configuration to the .tf file on your workstation
```
terraform {
  backend "atlas" {
    name    = "YOUR_ORGANIZATION/YOUR_ENVIRONMENT"
    address = "https://atlas.hashicorp.com"
  }
}
```
9. Navigate to the Terraform configuration folder and initialize it with the backend configuration you have entered to the .tf file.
```
terraform init
```
10. Push the Terraform configuration to the Enterprise server
```
terraform push -name="YOUR_ORGANIZATION/YOUR_ENVIRONMENT" -atlas-address="https://atlas.hashicorp.com"
```
11. On the left side click Configurations to see the list of configs your organization have pushed to the server.
12. Run a Terraform Plan to see if your code works
```
terraform plan
```
13. Execute your Terraform config on your workstation
```
terraform apply
```
14. Push the result of the run to the Terraform Enterprise server
```
terraform push -name="YOUR_ORGANIZATION/YOUR_ENVIRONMENT" -atlas-address="https://atlas.hashicorp.com"
```
15. To view the result of the run, on the left side click Runs.
16. To view the created infrastructure states on the left side click States

Create the Terraform scripts

More resources

For a tutorial on Multiple Environments in Terraform see https://github.com/hashicorp/multiple-envs

No instances for regex `’, try running `kitchen list’ in Chef Test Kitchen

When I tried to execute the “kitchen list” command in Chef Test Kitchen the following error came up:

No instances for regex `’, try running `kitchen list’

I could not find the reason for the error, so I opened the .kitchen.yml file in the Atom editor, added a space to one of the comments, and saved the file. The error went away.

GitKraken installation and configuration

GitKraken is a Git user interface to manage Git repositories.

Installation

Download the GitKraken installer from https://www.gitkraken.com/download

Configuration

Start the GitKraken application
Login with your GitHub account, or create a new account

or
Connect to a Git repository
Click the Open a Repository button
Select a repository folder and click the Open button

3D Printer GCODE instructions

GCODE is a standard file type to control 3D printers. The 3D object is usually exported from the CAD program to an STL file, that fully describes the end product. 3D printers build the physical objects layer-by-layer, so we need to slice the object into thin layers.

Turn off the fan after 5 minutes

When the printer completed the job, the fan of the Monoprice Select Mini printer says on, making noise, and wearing out the bearing. To turn the fan off after 5 minutes, add this code to the end of every GCODE file.

G4 P300000 ;wait 5 minutes before turning off the fan
M106 S1    ;turn off fan

To automatically append the instructions to every GCODE file, in Cura add the lines to the end.code section:

In the Cura application select the Start/End-GCode tab,
Select end.gcode,
Enter the lines in the bottom window.

Terraform Enterprise Administration

Create a new organization

In your web browser navigate to https://atlas.hashicorp.com/help/organizations,
On the left menu bar under Organizations click Create,
In the middle of the screen click the new organization page link,
Enter the email address and user name for the organization owner and click the Create organization button.

Using Git

Frequently used Git commands

Git runs entirely on your workstation, and a copy of the entire repository is also on your local hard drive. GitHub, BitBucket, and other providers only give you a storage space to allow you to share your repository with others and provide a web user interface to manage it. You can use any provider’s application on your workstation to manage any of your Git repositories. Many developers use SourceTree, the great application written by Atlassian, the owner of BitBucket, to manage Git repositories on their workstations that are shared at GitHub.

Here are the most frequently used Git commands.

Create the local repository

Initialize a new Git repository in the current directory. This command creates the .git sub-directory to store your repository and its configuration file.

git init

Display the local repository status

Display the list of added, deleted and modified files in the local repository

git status

Display your changes in the files

Display the changes in files since the last git add

git diff

Display the changes in the local stage area after you have executed git add

git diff --staged

Stage your changes in the local repository

Add your changes to the local stage area

git add .

Save the changes to the local repository

Commit your changes from the local stage area to the local repository with a message

git commit -m "My message"

Edit the last commit message

You can edit the last commit message even after the push.

git commit --amend

When you execute git status you get the message

Your branch and ‘origin/master’ have diverged,
and have 1 and 1 different commits each, respectively.
(use “git pull” to merge the remote branch into yours)
nothing to commit, working tree clean

To synchronize your local repository with the remote server, execute

git pull

The merge window pops up. You can leave the default message, or type your explanation. To save your message and close the window

Press the ESC key on your keyboard
Press the keys (including the colon at the beginning) :wq

Send your changes to the remote repository

Push the changes to the remote repository at GitHub, Bitbucket, or others

git push

Get the latest from the remote repository

Pull the latest changes from the remote repository

git pull

Advanced topics

Push an existing repository from the command line

git remote add origin https://github.com/ORGANIZATION/REPOSITORY_NAME.git
git push -u origin master

Clone a private Git repository from the command line

git clone https://my_username:my_password_or_token@github.com/my_user_name/my_repo_name.git

or use an SSH key, see https://help.github.com/en/articles/connecting-to-github-with-ssh

Branching

Great article on a branching strategy that can serve small projects and large teams: A successful Git branching model

Forks and Pull Requests

Simple forking tutorial at https://help.github.com/articles/fork-a-repo/
A detailed explanation of the recommended steps to fork an existing project: GitHub Standard Fork & Pull Request Workflow
A Pull Request Tutorial is at https://yangsu.github.io/pull-request-tutorial/

Restoring and working with older versions of the code

Save the latest version in a branch and restore an old version into the Main branch

If your application is deployed from the Main branch, and you want to make changes to an older deployed version after changes have been made to the Main branch we will

create a feature branch for the new version
restore the old version into the Main branch

Follow the steps below:

Create a duplicate copy of the repository directory on your workstation to be able to restore the repository in case you accidentally overwrite a file.
Create a new branch for the new version and switch to it
git switch -c MY_NEW_BRANCH
Push the new branch to GitHub
git push --set-upstream origin MY_NEW_BRANCH
Switch back to the Main branch
git checkout main
Restore the Main branch to an earlier version
git reset --hard OLD_COMMIT_HASH
git will display the name of the selected commit, check if that is the correct repo version you want to restore.
Make your changes to fix the application and commit them into the Main branch
- To commit only the files already tracked by Git
  git commit -a
- To commit selected files
  git add [files]
  git commit
- To push the old version of the Main branch to GitHub and clear the history in the Main branch after the commit
  git push origin main -f
Deploy the fixed version from the Main branch,
Work in the feature branch until you are ready to deploy the new version:
- Merge the feature branch to the Main branch
- Deploy the new version from the Main branch

View an old version of the repository

To view the repository in the state of an old commit and return back to the current state use the “checkout” command

Save the state of the working directory

Temporarily save the current state of the working directory including untracked (new) and ignored files

 git stash -a

List the commits of the repository

git log

Search all commits of a file in a branch

First, select the branch to search in, if it is not the current branch

git checkout BRANCH_NAME
git log --all --full-history -- **/MY_FILE.*

View the old version of the repository

git checkout MY_OLD_COMMIT_SHA

To create a new branch for the restored version and switch to it

git switch -c <new-branch-name>

To switch back to the “main” branch to see the latest version of the repository

git switch main

To switch to the new branch for the restored version of the repository

git switch <new-branch-name>

Create a new branch and continue the work

If you realize, this is the last stable version of the application, create a new branch and develop your project from this point forward. This way you will not lose changes you made after this commit, but you will be able to create new commits based on the working version and merge the new branch back to the “master” branch later.

git checkout -b MY_APP_WORKS_AGAIN_BRANCH

Restore the state of the working directory before the “stash” command

git stash pop

If you want to restore the repository to the state before the “git stash” with the “git stash pop” command and you have created new files, to prevent the accidental deletion of those files, Git will display the error message:

MyPath/File already exists, no checkout
Could not restore untracked files from stash

You also get this message when an ignored file has been created, like the “.DS_Store” file on Mac OS.

Remove ignored files

To remove the ignored files from the work tree execute the following. Please note the upper case “X”.

Dry run to see what will be deleted:

git clean -n -f -X

Delete the files:

git clean -f -X

Remove ignored and untracked files

To remove the ignored files, and new (untracked) files and directories from the working tree, clean the repository with the following. Please note the lower case “x”.

Dry run to see what will be deleted:

git clean -n -f -d -x

Delete the files:

git clean -f -d -x

Rollback

If you have accidentally committed a change and want to roll back the changes you can use the “reset” command. It is very dangerous because it can rewrite history, remove commits, delete files in your working directory, so you can lose your work. The “–mixed” is the default option of the “reset” command, so if no option is specified, that will be executed.

If the change has NOT been pushed to the remote repository (GitHub)

Remove a file from the stage

The “add” command adds files to the “stage”. If you have “add”-ed multiple files and do not want to “commit” one of them together with the rest, remove a file from the “stage”, but keep it in your working directory.

git reset HEAD -- <file>

git reset HEAD -- <directoryName>

Remove the last commit

Move the history back before the last “commit” and all “add”s that are associated with it. You will not lose any changes in your working directory. Use this command if you realize you want to make more changes before the next commit.

git reset HEAD~1

Remove the last commit and lose all changes since that

Restore the files in the repository to the state of the prior “commit”. You will lose all changes you made since that. This command moves the HEAD back one commit, so it deletes the last commit from the history.

git reset --hard HEAD~1

Undo the rollback

If the reset was unnecessary you can undo it for a limited time. Git runs the garbage collector every 30 days, and it removes orphaned commits, so you have 0 to 30 days to undo the rollback. If the garbage collector runs a few minutes after the reset, the changes are lost forever.

To see the list of commit SHAs that the garbage collector not yet deleted

git reflog

To undo the reset of a commit while it is still available

git checkout -b aNewBranchName shaYouDestroyed

If the changes have already been pushed to the remote repository

The “reset” command can cause serious problems for others working in the same repository.

To undo a commit

Use the “revert” command to correct mistakes. The “revert” command will create a new commit with the state you want without rewriting the history of the repository.

git revert <bad-commit-sha1-id>
git push origin

To restore the repo to the state of a previous commit

To erase history, and restore the repository to the state of a previous commit use the “reset” command. “git push .. -f” will force the push to erase the history on the server, and also overwrite other user’s commits after the specified one, so coordinate with them, and they also need to “reset”their local repositories.

git reset --hard <commit-id>
git push origin main -f

Remove a file form the entire history of the repository

If a file is too big to be uploaded to the remote repository, you may get the error message when you push the repository to the remote:

remote: Resolving deltas: 100%, done.
remote: error: GH001: Large files detected. You may want to try Git Large File Storage – https://git-lfs.github.com.
remote: error: Trace: …
remote: error: See http://git.io/iEPt8g for more information.
remote: error: File … is … MB; this exceeds GitHub’s file size limit of 100.00 MB
To https://github.com/….git
! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to ‘https://github.com/….git’

You may also want to remove every trace of a file from the history for security reasons.

git filter-branch command

This command rewrites the history of the repository and removes every trace of a file

git filter-branch -f --index-filter 'git rm --cached --ignore-unmatch "MY_TOO_BIG_FILE_NAME"'

git-filter-repo utility

An alternative to remove files is the Python utility git-filter-repo

Clone the git filter repository from https://github.com/newren/git-filter-repo.git

git clone https://github.com/newren/git-filter-repo.git

Copy the git-filter-repo file to a directory in your path

Usage

Open a terminal window in the repository you want to clean

cd MY_REPOSITORY_TO_CLEAN

The git-filter-repo utility selects files from the repository, so to delete those files and keep the rest of the files in the repository, use the --invert-paths option.

To remove files or directories from the repositorygit filter-repo --path README.md --path guides/ --path tools/releases --invert-paths

To use wildcards include the --path-glob optiongit filter-repo --path-glob 'src/*/data' --invert-paths

To remove multiple files or directories, save the list in a text file and refer to it with the --paths-fom-file option

git-filter-repo --invert-paths --paths-from-file /tmp/delete-from-git-repo.txt --force

BFG Cleaner

An other alternative to git filter-branch to remove files from the repository is at https://rtyley.github.io/bfg-repo-cleaner/

error: package … does not exist in NetBeans

If your Java source code imports packages, you have to add the JAR files containing them to the Library.

When you try to compile your Java application in NetBeans, and you get the error message:

error: package … does not exist

In the NetBeans Project view right-click the Libraries folder
In the drop down menu select Add JAR/Folder
Select all the JAR files you want to add to the library, not just the folder,
Click the Open button.

Splunk configuration

Splunk stores the configuration values in files in the /opt/splunkforwarder directory structure.

Splunk client

Description	Location
Splunk Deployment server	/opt/splunkforwarder/etc/system/local/deploymentclient.conf
Example	targetUri = DEPLOYMENT_SERVER_URL:8089

Splunk Forwarder address	/opt/splunkforwarder/etc/apps/tcpout-aws/local/outputs.conf
Example	server = FORWARDER1_ADDRESS:9997,FORWARDER2_ADDRESS:9997

Linux event log. Splunk tails this file.	/var/log/messages
To log a message in the Linux event log	logger "My message"
To find a message in the Linux event log	grep "My message" /var/log/messages

Splunk server

Description	Location
Default data directory	/opt/splunk/var/lib/splunk/defaultdb/
Log location	/opt/splunk/var/log/splunk/splunkd.log

Useful Splunk UI searches

To list all indexes

| REST /services/data/indexes | dedup title | sort title | table title