Docker commands to run images

Maven

Navigate to the Maven project directory on your workstation and launch the Maven container.

To run a Maven project by using the Maven Docker image directly, passing a Maven command to docker run.

docker run -it --rm --name my-maven-project -v "$(pwd)":/usr/src/mymaven -w /usr/src/mymaven maven:3.3-jdk-8 mvn clean install

To launch the Maven Docker container and open a Bash terminal in the container for an interactive session.

docker run -it --name my-maven-project -v "$(pwd)":/usr/src/mymaven -w /usr/src/mymaven maven:3.3-jdk-8 bash

 

Set up a user to connect to a Linux server with a private-public key pair

To secure a Linux server, disable password authentication on it. This way only those users can connect to it, who have access to an authorized private key.

To enable users to connect to a Linux server with a private-public key pair

Generate an RSA key pair

  1. In a Bash terminal on your workstation execute
    ssh-keygen
  2. Follow the prompts to specify the name of the key file pair. In most of the cases, you don’t need to protect the key with a password.
    1. If you don’t specify the file name, the key will be saved as ~/.ssh/id_rsa
    2. If you specify a file name, the key files will be saved in the current directory
  3. The public key file will get the “.pub” extension, the private file has no extension

Upload the public key to the Linux server

  1. Log into the server with the “ssh” command using a username and password
    ssh MY_USER_NAME@SERVER_IP_ADDRESS
  2. Add the public part of the key to the user configuration
    1. Switch to sudo mode, this command will ask for the password again
      sudo -i
    2. Navigate to the user home directory
      cd /home/USER_NAME/
    3. Add the public key to the user’s authorized_keys file. Open the file with a text editor and copy the public key into a new line.
      vi authorized_keys
    4. To test the configuration, on your workstation navigate to the directory where the new key is located, and log into the server with
      ssh -i MY_KEY_NAME MY_USER_NAME@SERVER_IP_ADDRESS

Turn off password authentication

  1. Make sure you can log in with the new key !!!
  2. Execute the command
    sed -i 's/#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config

Install a ZIP web application package with Octopus Deploy

Install the Octopus Tentacle on the application server.

  1. Automated Tentacle installation instructions are at https://octopus.com/docs/infrastructure/windows-targets/automating-tentacle-installation
    (Note for our DevOps team: execute ~/Git/devops-scripts/OctopusDeploy/Tentacle_Installation/install_tentacle.ps1.)
  2. Make sure the box registered itself in the Octopus server. On the Deployment Targets tab of the Infrastructure menu enter the server name into the search box. The script already created the Environment and the Role and attached them to the target.

To manually create the application-specific environments

  1. On the Environments tab of the Infrastructure menu push the ADD ENVIRONMENT button

Create an application-specific lifecycle

  1. On the Lifecycles tab of the Library menu click the ADD LIFECYCLE button
  2. Click the ADD PHASE button to add a new phase for the environments
  3. Click the ADD ENVIRONMENT link to add the environments to the phase
  4. Create a phase for every environment. For automated deployment, select the Deploy automatically… radio button, and select the application-specific environments, and click the Ok button
  5. The list of phases is at the bottom of the page. Click the Save button to save the lifecycle.

Create a project for the application deployment

  1. Create a new project group
  2. Create a new project
  3. Set the project group and the lifecycle
  4. Click the DEFINE YOUR DEPLOYMENT PROCESS button
  5. Click the ADD STEP button
  6. Select the Deploy to IIS template
  7. Enter the name of the step and select the Role and Package ID


Trigger an Octopus Deploy application package installation from TeamCity

Push the package to Octopus Deploy from TeamCity

Create project level parameters

In TeamCity create the parameters to centralize the configuration of reused values

  1. On the project level create a system parameter for the Octopus package name including the build number interpolation

Push the artifact to Octopus Deploy

  1. Create a build configuration, and on the General Settings page specify a build number format that Octopus can understand
  2. Add the build step as a dependency
  3. Create a trigger on the same dependency to automatically execute the Octopus push on a successful build
  4. Create a Command Line build step to get the artifact from the built-in TeamCity repository. Use curl to download the file to the working directory, use the octopus_package_name parameter you created above.
    curl -o %system.MY_APP_api_octopus_package_name% http://%system.tc_server%/guestAuth/repository/downloadAll/MY_PROJECT_ID/.lastSuccessful/artifacts.zip

  5. Create an OctopusDeploy: Create and Push Packages build step to upload the package to the Octopus server. Use the system parameters to specify the Octopus server URL, and the package name, enter the API key that has enough rights to push a package to the Octopus server. The user needs the BuiltInFeedPush permission to push a package. Octopus Deploy provides a built-in role called Package Publisher that has been granted the BuiltInFeedPush permission.
  6. To check the push to Octopus Deploy, run the TeamCity configuration to upload the package to the OctopusDeploy built-in repository. in Octopus Deploy click the name of the package to see the versions.
  7. The next page shows the package versions

Create an Octopus Deploy release

  1. In the TeamCity configuration, add an OctopusDeploy: Create release type build step to create a release in Octopus Deploy. Use the %OctopusURL% parameter from the root project level, enter the Octopus API key, and use the %build_number% macro that you have specified in the Release number field on the General Settings tab. If you have not created a custom channel in the Octopus project, the default is “Default”.
    List the environments where the release can be deployed, and check the Show deployment process checkbox.
  2. To check the release creation in Octopus Deploy, navigate to the overview page of the Octopus Deploy project.

Deploy the Octopus Deploy release

To install the package add an OctopusDeploy: Deploy Release build step

  1. Create a project parameter for the Octopus Deploy project name
  2. Create a new build step using the project level parameters.

 

Connect to a database in a Docker container from a VirtualBox virtual machine

Docker is an emerging technology to launch applications on any guest operating system using any host operating system. The same host can run multiple Docker containers of any type, so one physical or virtual server can serve as a complete mini cloud environment. Most of the time Linux (including macOS) is used as the host operating system.

In this example, we will launch a PostgreSQL Docker container on macOS, and access the PostgreSQL database server from a Windows VirtualBox virtual machine.

Thanks to Robert Lestak for the Docker scripts.

Create a network between the guest VM and the host in Oracle VirtualBox

A VirtualBox virtual machine can share the networking with the host with the NAT setting, or it can be an independent computer on the network with the Bridged setting.

To access the host directly from the guest VM we can set up a host-only network, that only the host and the guest can access. This way we can launch a container on the host and access it from the guest via the host-only mini-network.

To set up the private host-only network

  1. Open the VirtualBox application and shut down all virtual machines
  2. In the File menu select Host Network Manager
  3. On the Host Network Manager window click the Create button
  4. Click the Enable checkbox and the Close button

Connect the virtual machine to the host-only network

The VirtualBox virtual machines can have four virtual network adapters. Always use the first adapter to connect to the host-only network.

  1. Select the virtual machine in the list and click the Settings icon
  2. On the Network tab click the arrow next to the Attached to drop-down
  3. Select the Host-only adapter
  4. The host-only network is automatically selected
  5. Select the second adapter, click the Enable Network Adapter checkbox, and select NAT
  6. Click the OK button to close the window.

Find the IP address of the host machine

  1. Open a terminal on the host machine.
  2. On Mac
    1. enter the following command. Use the IP address range of the host-only network above.
      ifconfig | grep 192.168.56

      inet 192.168.56.1 netmask 0xffffff00 broadcast 192.168.56.255

    2. Based on the output  above, the IP address of the host machine is 192.168.56.1

Launch a PostgreSQL Docker container

  1. Create a docker-compose.yml file. Containers are ephemeral, when we terminate them, all data stored inside the container is lost. We want to keep the database files on our local hard drive, so this configuration will map the /var/lib/postgresql/data directory of the Docker container to the pgdata subdirectory of the current working directory. The 5432 is the default port of the PostgreSQL database server, we will expose it on port 5433 of the host.
    version: '3'
    
    services:
      postgres:
        image: postgres:10
        env_file: .env
        ports:
          - "5433:5432"
        restart: always
        volumes:
          - $PWD/pgdata:/var/lib/postgresql/data
  2. Open a terminal window in the same directory and launch the Docker container
    docker-compose up -d

Connect to the host machine from the guest machine

Test the connection

  1. Open a terminal on the guest machine
  2. Test the connection to the host machine
    ping 192.168.56.1

Configure an ASP.NET Core 2.1 application to connect to the Docker PostgreSQL database server

  1. Change the connection string, and specify the exposed port on the host machine:
    "Server=192.168.56.1;Port=5433;Database=MY_DATABASE;Username=MY_USERNAME;Password=MY_PASSWORD"

 

 

The user account does not have permission to run this task

When a Scheduled task is created by another user (or SYSTEM) most of the time only that user can manually trigger the task execution.

When you try to execute the scheduled task from the user interface you can get the error message:

The user account does not have permission to run this task

This error is caused by the lack of permissions on the XML file that contains the task details. To allow all administrators to execute the scheduled task

  1. Remote into the Windows server
  2. Open the C:\Windows\System32\Tasks folder in Windows Explorer
  3. Right-click the file with the name of the scheduled task and select properties
  4. On the Security tab select Administrators and click the Edit button
  5. Check the Read & execute checkbox and click the OK button
  6. Click Yes on the Windows Security popup

Now you should be able to manually execute the scheduled task

Error loading table definition of a PostgreSQL table

When a PostgreSQL table name contains uppercase letters there is no known way to get the table definition SQL script.

When we right-click a table name in SQLPro for Postgres the has an uppercase letter and select Script as, CREATE to, New query window

we get the error message

/*
* Error loading table definition.
* ERROR: relation “public.users” does not exist
*/

As the error message tells us, PostgreSQL is looking for the table name with all lowercase letters.


The pg_dump utility acts the same way.

pg_dump -s -h MY_SERVER_ADDRESS -U M_USERNAME -t Users MY_DATABASE

The error message is

pg_dump: no matching tables were found


The same process works if the table name only contains lowercase letters.

Convert an existing PostgreSQL database to all lowercase table and column names

To avoid similar errors caused by upper case letters in table and column names, convert the existing database definition to all lowercase

    1. Backup your database from the command prompt
      PGPASSWORD=MY_PASSWORD pg_dump -h MY_DATABASE_ADDRESS -U MY_DATABSE_USER -f MY_BACKUP_FILENAME.bak MY_DATABASE_NAME
    2. Open the backup file in a text editor
    3. Add underscores to the table and column names to match the .NET entity framework user identity processes.
      1. The PostgreSQL driver converts the first letter of all lowercase table and column names to upper-case and the letter that follows an underscore. For example, asp_net_users will be converted to AspNetUsers.
        1. Replace the table names with TableName -> table_name ( leave __EFMigrationsHistory as is)
        2. Replace the column names with “ColumnName” -> “column_name”
        3. Replace the primary key names with PK_ -> pk_
        4. Replace the index names with IX_ -> ix_
        5. Replace the foreign key names with FK_ -> fk_
    4. Drop the database. In SQL Pro for Postgres
      1. Right-click the database and select Drop database
    5. Re-create the database. In SQL Pro for Postgres
      1. Rith-click the server name and select Create database
    6. Import the modified backup file from the command prompt
      PGPASSWORD=MY_PASSWORD psql -h MY_DATABASE_ADDRESS -U MY_DATABSE_USER -d MY_DATABASE_NAME -f MY_BACKUP_FILENAME.bak
    7. Reverse engineer the database back to the .NET data layer project
      1. Delete the class files that correspond to the database tables
      2. Delete the MY_DATABSEContext.cs file
      3. Import execute the command in the data layer project directory
        dotnet ef dbcontext scaffold --force "Host=MY_DATABASE_ADDRESS;Database=MY_DATABASE_NAME;Username=MY_DATABSE_USER;Password=MY_PASSWORD" Npgsql.EntityFrameworkCore.PostgreSQL

Use two GitHub accounts on the same computer

Many developers have multiple GitHub accounts. One for personal projects, and another one for their work or business. There is a way to access multiple accounts simultaneously from the same computer. Only one account at a time can be configured using the HTTP connection, but we can configure the rest of the accounts using SSH.

There are many great articles on this topic, but most of them only provide partial instructions on how to set up multiple GitHub accounts on the same computer. In this post, I will document all the necessary steps to set up two accounts.

Usually, you don’t have much control over what GitHub connectivity is used for the company account, it can be dictated by the customer or your workplace, but you can always control what to use for your personal account. With SSH you can specify all aspects of your personal account, so you can leave the default SSH or HTTP connection for the business or company account.

In this post, we will set up two accounts with SSH connection. Even if the business account currently uses HTTP, it makes sense to set up SSH for that account too, in case they will switch to SSH at a later time.

Create two keys

    1. Open a terminal window and start the authentication agent to avoid the “Could not open a connection to your authentication agent” error later
       eval `ssh-agent -s`
    2. Create the personal SSH key
      ssh-keygen -t rsa -C "MY_PERSONAL_EMAIL@ADDRESS"

      when asked for the file name by the “Enter file in which to save the key“prompt, save the key as id_rsa_pers

    3. Add the personal key to the authentication agent
      ssh-add id_rsa_pers
    4. Create the business SSH key
      ssh-keygen -t rsa -C "MY_COMPANY_EMAIL@ADDRESS"

      when asked for the file name by the “Enter file in which to save the key“prompt, save the key as id_rsa_COMPANY_NAME

    5. Add the business key to the authentication agent
      ssh-add id_rsa_COMPANY_NAME

If you want to use existing keys from another location

If you already have the keys at another location or at another user account, copy the ~/.ssh directory to the new home directory. Do the same if you want to use the same keys in sudo mode. In that case switch to sudo mode with

sudo -i

Copy the .ssh directory

cd ~
cp -r /home/OTHER_USER/.ssh .

Set the file permissions

If you have copied the keys from another location, the file permissions may not set correctly. If the key permissions are too open, Linux displays the message:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: UNPROTECTED PRIVATE KEY FILE! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0755 for ‘/home/MY_USER_NAME/.ssh/id_rsa_pers’ are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
Load key “/home/MY_USER_NAME/.ssh/id_rsa_pers”: bad permissions
git@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Set the file permissions. Execute the following statements in a terminal window

  1. Navigate to the .ssh directory
    cd ~/.ssh
  2. Set the key file permissions
    sudo chmod 600 ~/.ssh/id_rsa
    sudo chmod 600 ~/.ssh/id_rsa.pub
    sudo chmod 600 ~/.ssh/id_rsa_pers
    sudo chmod 600 ~/.ssh/id_rsa_pers.pub
    sudo chmod 600 ~/.ssh/id_rsa_COMPANY_NAME
    sudo chmod 600 ~/.ssh/id_rsa_COMPANY_NAME.pub
    
  3. Set the nown_host file permission
    sudo chmod 644 ~/.ssh/known_hosts
  4. Set the directory permission
    sudo chmod 755 ~/.ssh
  5. Start the authentication agent to avoid the “Could not open a connection to your authentication agent” error later
     eval `ssh-agent -s`
  6. Register the keys
    ssh-add id_rsa
    ssh-add id_rsa_pers
    ssh-add id_rsa_COMPANY_NAME

Configure SSH to use the two new keys

  1. Create the SSH config file
    touch ~/.ssh/config
  2. Open the ~/.ssh/config file with a text editor
  3. Add the following lines
    #Personal GitHub
    Host github-pers
      HostName github.com
      User MY_PERSONAL_GITHUB_USERNAME
      IdentityFile ~/.ssh/id_rsa_pers
    
    #Company GitHub
    Host github.com
      HostName github.com
      User MY_COMPANY_GITHUB_USERNAME
      IdentityFile ~/.ssh/id_rsa_COMPANY_NAME
    

Upload the keys to your GitHub accounts

  1. Log into the GitHub website with your personal credentials
  2. In the upper right corner click the down arrow next to your icon, and select Settings
  3. On the left side select SSH and GPG keys
  4. Click the New SSH key button
  5. Type a name for the key,  paste the contents of the ~/.ssh/id_rsa_pers.pub public key file into the text area, and click the Add SSH key button
  6. Log into Github with your company credentials
  7. Upload the id_rsa_COMPANY_NAME.pub public key file as explained above.

Create a personal repository

  1. Create a new folder, personal-test, and add a file to it on your computer
  2. Initialize the Git repository in the folder
    git init
    git add .
    git commit -m "Initial commit"
  3. Log into your personal GitHub account
  4. Create a new repository, and name it personal-test
  5. On the confirmation page click the SSH button
  6. Copy the …or push an existing… code to a text editor
  7. Edit the origin address to match the “Host” of the personal section in the ~/.ssh/config file, and execute the commands
    git remote add origin git@github-pers:MY_PERSONAL_GITHUB_USERNAME/personal-test.git
    git push -u origin master

Company repositories

For your company account use the default HTTP or SSH GitHub repository address without any modifications.

Change the AWS account to launch instances with Chef Test Kitchen

The Chef SDK contains Test Kitchen, that can launch server instances to test your Chef cookbooks. Test Kitchen uses the “chef_zero” provisioner to use your workstation as the virtual Chef server.

To switch Test Kitchen to launch instances in another AWS account

  1. In the .kitchen.yml file update the
    1. availability_zone
    2. subnet_id
    3. aws_ssh_key_id
    4. security_group_ids. Make sure the management ports are open:
      1. for Linux: port 22 for SSH, or
      2. for Windows: port 5985 – 5986 for WinRM and port 3389 for Remote Desktop
    5. ssh_key
  2. In the default.rb attribute file update the
    1. …[‘aws_profile’]
  3. In the ~/.aws/credentials file update the
    1. [default] section