How to duplicate a Windows server that is attached to the Windows Domain

When a Windows server is attached to a Domain only one instance of it can run at a time. To be able to duplicate the server and start a second instance of it

Create an image of the server

  1. Remote into the server
  2. Make sure there is a local administrator account that you can log in with
  3. Remove it from the domain
  4. Reboot the server
  5. In the AWS console create an image of it
  6. Add the server back to the domain with the same username and password that was used to originally add it to the domain

Launch a new instance

  1. In the AWS console launch a new instance with the saved image
  2. Log into the new instance with the local administrator account
  3. Change the server name (only one server can be added to the domain with the same name)
  4. Add the server to the domain
  5. Reboot the server

Bootstrap the new instance in the Chef server

If you configured the original server with Chef, attach the new instance to the Chef server with a new Node Name

  1. Open a terminal window in the cookbooks directory (below the .kitchen directory, so the Chef kitchen command can find the Chef server configuration)
  2. Execute the bootstrap command. See Bootstrap Chef nodes to connect them to the Chef server for the details on the bootstrapping process. Don’t forget to use a double backslash (\\) in front of the username.
knife bootstrap windows winrm MY_NODE_IP -x MY_USERNAME -P MY_PASSWORD --node-name THE_NODE_NAME --environment THE_ENVIRONMENT --run-list 'recipe[MY_COOKBOOK1::default],recipe[MY_COOKBOOK2::default]' --json-attributes '{"MY_ATTRIB1":"MY_VALUE1","MY_ATTRIB2":"MY_VALUE2"}' -V

Stop multiple untagged AWS EC2 instances with a Bash script

 List all EC2 instances without a specific tag

One day we have found 499 instances running in our account without any tags. Most likely someone accidentally started a process to launch those, so we needed a way to find them and stop them. Later we will terminate them with the same script below when we can make sure those are not needed.

For simplicity, place the appropriate aws_access_key_id and aws_secret_access_key into the [default] section of the “~/.aws/credentials” file or use the –profile option in every command below.

List all instances

To list all EC2 instances, execute

aws ec2 describe-instances

List all instances missing a specific tag

I have found the command to list those instances that are missing the “Name” tag at https://www.onica.com/blog/using-aws-cli-to-find-untagged-instances/

I have directed the output to a text file with the additional last line.

To get all info on the instances with no “Name” tag into a JSON file

aws ec2 describe-instances \
--query 'Reservations[].Instances[?!not_null(Tags[?Key == `Name`].Value)]' \
> instances-no-name-tag.json

To output multiple properties into a tab-separated file for reporting in Excel.

aws ec2 describe-instances \
--output text \
--filters Name=instance-state-name,Values=running \
--query 'Reservations[].Instances[?!not_null(Tags[?Key == `Name`].Value)] | [].[InstanceId,ImageId,InstanceType,Platform,LaunchTime,SubnetId,KeyName]' \
> instance-info-no-name-tag.csv

Get the list of instance IDs into a text file for batch processing

aws ec2 describe-instances \
--output text \
--filters Name=instance-state-name,Values=running \
--query 'Reservations[].Instances[?!not_null(Tags[?Key == `Name`].Value)] | [].[InstanceId]' \
> instance-ids-no-name-tag.txt

Stop an instance with the instance Id

aws ec2 stop-instances --instance-ids MY_INSTANCE_ID

Stop multiple instances

To stop all instances listed in the “instance-ids-no-name-tag.txt” file created above, create and execute this Bash script:

#!/bin/bash

# The file with the instance IDs
filname=instance-ids-no-name-tag.txt

# Iterate through the lines
while read p; do
  echo "Stopping $p"
  aws ec2 stop-instances --instance-ids $p
done <$filname

 

Creating CentOS server images with Packer

The CentOS images are not available on the AWS Quick Start tab.

CentOS publishes official images on the AWS Marketplace, but you need to subscribe to the image to be able to launch it with an automation software, like Terraform.

Find the latest available CentOS image in the AWS Marketplace

  1. Execute this command to display the list of available images
    aws --region us-east-1 ec2 describe-images --owners aws-marketplace --filters Name=product-code,Values=aw0evgkw8e5c1q413zgy5pjce
  2. Select the latest AMI from the list. The images are NOT ordered by date!
  3. If you try to launch the image before you subscribe to it, the error message is displayed

    Error launching source instance: OptInRequired: In order to use this AWS Marketplace product you need to accept terms and subscribe. To do so please visit http://aws.amazon.com/marketplace/pp?sku=…
    amazon-ebs: status code: 401, request id: []

  4. Follow the link in the error message to http://aws.amazon.com/marketplace/pp?sku=…
  5. Click the Continue to Subscribe button
  6. Once the new image has been created, share it with all the accounts you use.

SocketError: Error connecting to … Name or service not known

In an enterprise environment, the company usually operates its own DNS servers.

When a Linux instance launches in AWS, the DNS settings only contain the AWS DNS server. If the company DNS server settings are applied with Chef, during the first Chef Client run those settings do not take effect.

When we reference an internal DNS entry during the first Chef Client run we get the error message:

 

SocketError: Error connecting to https://INTERNAL_ADDRESS.net/… – Failed to open TCP connection to INTERNAL_ADDRESS.net:443 (getaddrinfo: Name or service not known)

There two solutions to handle this:

  1. Use the IP address of the internal server or load balancer in the URL ( switch from HTTPS to HTTP, because the SSL certificate will not work with the IP address),
  2. Set the DNS servers of the Linux server with Packer when you generate the base image.

“Ran out of time waiting for the server with id” with Windows Server 2016 in Chef Test Kitchen

AWS changed how Windows Server EC2 instances send messages during boot.

Windows Server 2012 R2 AWS EC2 instances sent the “Windows is ready” message every time those became available after boot.

When a Windows Server 2016 AWS EC2 instance launches, it only sends the “Windows is ready” message during the first boot. If you create your custom AMI with Packer, the first boot happens during the Packer AMI creation, so Chef Test Kitchen does not receive the expected message and the process times out with the message

>>>>>> Ran out of time waiting for the server with id [i-… to become ready, attempting to destroy it
EC2 instance <i-…> destroyed.

To configure Windows Server 2016 instances to send the expected “Windows is ready” message during every boot, add a PowerShell line to your Packer template that creates your custom AMI:

{
  "type": "powershell",
  "pause_before":"2s",
  "elevated_user": "MY_ADMIN_USER",
  "elevated_password": "MY_ADMIN_PASSWORD",
  "inline": [
    "C:\\ProgramData\\Amazon\\EC2-Windows\\Launch\\Scripts\\SendWindowsIsReady.ps1 -Schedule"
  ]
},

This code creates a scheduled task to execute the configuration script on the instance.

More information is at https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/ec2launch.html#ec2launch-sendwinisready

 

Find the AWS account number

The AWS account number uniquely identifies the AWs account you are working with. All AWS “arn” identifiers contain it, and you need to know it when you want to share AMIs with other accounts.

If there are no resources created yet in the account, you can find the account number in the “arn” of your user account.

  1. Log into the AWS Console
  2. In the upper right corner click your username and select My Security Credentials
  3. On the left side select Users, and click any of the usernames in the list
  4. The account number is part of the “User ARN”

Execute Terraform scripts with Octopus

Terraform is a very powerful, free command-line tool to launch servers in any cloud or virtual machine environment. Hashicorp, the creator of Terraform just introduced the paid Terraform Enterprise server, that orchestrates the execution of the Terraform scripts.

Octopus is another tool that added Terraform orchestration functionality in version 2018.3

In this example, we will set up a Multi-Tenant Octopus project to launch servers in AWS with Terraform scripts and configure them with Chef.

Terraform script location

Create a GitHub repository for the Terraform script

GitHub is the most flexible way to store the Terraform scripts. Octopus can download and execute the same script in multiple Octopus projects. This eliminates the code duplication and makes the scripts available for execution from the command line.

Create a GitHub user with read access to the repository

If the GitHub repository is private, Octopus will need a way to access it with a username and password.

Tag the source code version, Octopus needs a version to refer to the version of your script

git tag 1.0.0
git push --tags

Automation with Octopus

We will create a Multi-Tenant Octopus project to be able to use the same Octopus project for multiple server types in multiple AWS environments.

We will create

  • an Octopus project group to hold the AWS launch projects,
  • a generic Octopus project to launch an AWS EC2 instance,
  • an operating system specific Octopus project that calls the generic project
  • environments for each AWS account, application, application environment,
  • tenants for each server.

Create the Environments

  1. On the Infrastructure Octopus page select Environments
  2. On the Environments page click the ADD ENVIRONMENT button
  3. Create the environments you need (aws-dev, aws-qa, aws-uat, aws-prod)


Create a Lifecycle

The lifecycle governs how the project is promoted between environments.

  1. On the Library Octopus page select Lifecycles
  2. Click the ADD LIFECYCLE button
  3. Enter the name and description of the Lifecycle and click the ADD PHASE button

  4. Enter a name for the Phase and click the ADD ENVIRONMENT link to select the environment
  5. Select the environment and the Users will need to manually… radio button
  6. Keep the All environments must be deployed… and click the Save button.

Create a project group

The project group holds your launch projects together.

  1. On the Projects Octopus page select ADD GROUP
  2. Enter a name and description for the project group

Create a GitHub feed

This feed will allow Octopus to connect to GitHub

  1. On the Library Octopus page select External Feeds
  2. Click the ADD FEED button
  3. Select the GitHub Repository Feed type, enter a name and copy the recommended URL. Add the GitHub user’s name and password you have created above, and click the SAVE AND TEST link

  4. Enter the complete name of your Terraform repository and click the SEARCH button to make sure Octopus has access to the private repository. If you do not specify the full path of the repository, the search will display public repositories with similar names.

Store an AWS account reference in Octopus

AWS accounts store the credentials to connect to Amazon Web Sevices

  1. On the Infrastructure page select Accounts
  2. In the Amazon Web Services section click the ADD ACCOUNT button
  3. Enter a name, description, Access Key and Secret Key for the account. Enable the usage in tenanted and untenanted deployments, and click the SAVE AND TEST link.



Store SSH Keys in Octopus

Store your private keys in Octopus, so you can reference them in your variables.

  1. On the infrastructure page select Accounts
  2. In the SSH Key Pairs section click the ADD ACCOUNT button
  3. Enter the SSH key name, the username, and click the up arrow to upload the SSH private key file to the Octopus server.
    1. Select the Include in both tenanted and untenanted deployments radio button.
    2. In the Associated Tenants section select the tenant groups you want to use the key for.
  4. Click the SAVE button to save the certificate in Octopus

Create a Project

The project contains the reference to the Terraform script to launch the server, the connection to the GitHub repository where the Terraform script is stored, and credentials to access the AWS account. The Terraform script and GitHub connection are going to be the same for every server launch, but the AWS account credentials have to be different for each AWS account. If you work with multiple AWS accounts, you need to create separate projects for each AWS account.

  1. On the Projects Octopus page scroll down to the DevOps project group and click the ADD PROJECT button
  2. Enter a name and description for the new project, and click the ADVANCED SETTINGS link
  3. Select the lifecycle you have created earlier, and click the SAVE button.

Create the launch process

  1. On the left side select the Process menu item
  2. On the Process page click the ADD STEP button
  3. Type “terraform” into the filter field and select Apply a Terraform template step

Create the variables

  1. In the project click the Variables menu item
  2. Enter the name of the variable and click the value field
  3. Click the CHANGE TYPE link
  4. Select the Amazon Web Services Account type
  5. Enter a description and select the AWS account you have created above, and click the DONE button.
  6. Create the rest of the variables.
  7. Add prefixes to the variables that will be set in the OS-specific parent project, in the tenant, environment, and target.

Update the variable names in the Terraform script

When we added prefixes to the project variables to show which component responsible for setting it, we have changed the variable name. Update the Terraform script to reflect it.

ami = "#{os.instance_ami}"


Create an OS-specifc project that calls the generic project

There are values that are different for each operating system in each AWS account.

The AMI ID depends on the operating system and the application environment. We can test the new version of the operating system in the development environment, while we are still launching servers with the old, tested version for production.

The common, OS-specific security group ID is different in every AWS account. It depends on the AWS account and the operating system.

To be able to set the value of this variable based on the OS and the AWs account, or the OS and application environment, we need to create an OS-specific project that passes in the appropriate variable values to the generic project.

  1. On the Projects page scroll down to the DevOps project group and click the ADD PROJECT button
  2. Enter the name and description of the new project and click the ADVANCED SETTINGS link
  3. Select the DevOps-aws-lifecycle
  4. Click the Process menu item
  5. Click the ADD STEP button
  6. Select the Deploy a Release step
  7. Enter the name for the step and select the generic AWS EC2 Launch project

Create variables

Set the OS-specific variables in the OS-specific parent project. To avoid variable name collision later add the parent.prefix to the variables passed in from the parent to the child project.

Settings

  1. Click the Settings menu item
  2. Require tenant for the deployment, allow the deployment without target, and ask if there is an error. Chef raises an error when it reboots the server, so if the Chef cookbook contains the reboot resource, we need to accept the error for a successful deployment.
  3. Click the Variables menu item
  4. Add the OS-specific variables

Create Library Variable Templates

Library Variable Sets specify the variables that have to be set in a higher level component, like a tenant, environment, and target.

  1. On the Library Octopus page select Variable Sets
  2. On the Variable Sets page click the ADD NEW VARIABLE SET button
  3. Enter a name and description for the library variable set
  4. Click the VARIABLE TEMPLATES tab
  5. On the VARIABLE TEMPLATES tab click the ADD TEMPLATE link
  6. Add the variables to the Library Variable Template. For pre-defined choices use the Drop down control type. To avoid variable name collisions when we pass the variable values to the child project later, add the “parent.” prefix to every variable.

  7. Save the Library Variable Template

Add the Library Variable Templates to the OS-specific project

Add the Library Variable Sets to the OS-specific project, so the tenants can set them with the instance-specific values.

  1. Open the project and select the Variables menu item
  2. Select the Library Sets menu item
  3. Click the INCLUDE LIBRARY VARIABLE SETS button
  4. Select the newly created Library Variable Set

Pass the variables from the parent project to the child project

The parent project can pass variables into the child project. Select the variable to pass into the child project. You need to do this in every parent project that call the common child project.

  1. In every parent project select Process
  2. Select the process that calls the child project
  3. Click the Variables section
  4. Click the ADD VARIABLE link
  5. Add the OS-specific variables set in the parent project and the host-specific variables defined in the Library Variable Template and set in the tenant.

Create a Tenant

Tenants are server types that use the same Octopus project. A tenant can be an application or a software development project where the servers share common attributes and run on the same operating system type, like all Linux, or all Windows, so the same Terraform script can launch them.

  1. In Octopus select the Tenants page and click the ADD TENANT button
  2. Enter the name of the new tenant
  3. Click the Setting button to select a logo for the tenant
  4. Click the logo placeholder to open the region
  5. Click the file selection button to browse to a 100×100 image file

Tag the Tenant

Tags help you to organize your tenants.

  1. On the Library page select Tenant Tag Sets
  2. Click the ADD TAG SET button
  3. Enter a name and description for the tag set, and create the first tag, and click the ADD link
  4. Click the EDIT TAGS link to add tags to the tenant
  5. Click the small arrow to select the tags

Connect the Tenant to the project

  1. On the Tenants page select the tenant you want to add to the project
  2. Click the CONNECT PROJECT button
  3. Select the project and all environments you want to launch instances for this tenant (software project)

Set the tenant variable values

  1. On the Tenant page select Variables
  2. Select the COMMON VARIABLES tab
  3. Click the name of the template to open it
  4. Set the value and click the Bind icon
  5. When all required variables are set, the SAVE button becomes available.

Create a deployment target (server)

Deployment targets are the servers we are launching. In enterprise settings, all servers have a standardized name that conforms to the company standards.

  1. In the
  2. Click the ADD DEPLOYMENT TARGET button
  3. Select the Cloud Region radio button
  4.  Enter the server name, deployment and a role, and do not press the Save button yet
  5. In the Restrictions region select the Include in both tenanted and untenanted deployments radio button select the Tenant you have created.

Create project releases

If there was a change in the child project, we need to create a release for the child project and all parent projects that call it.

Create a release of the child project

  1. On the project page click the CREATE RELEASE button
  2. If you use GitHub as the Terraform script source, the no version specified message displays in the Packages section
  3. Tag the GitHub repository. Octopus needs a version to link to a GitHub package.
    git tag 1.0.0
    git push --tags
  4. Click the SEARCH link to find the latest version of the repository
  5. Select the latest version of the GitHub repository
  6. Click the SAVE button

Create releases for the parent projects

  1. Open the project
  2. Click the CREATE RELEASE button
  3. Enter the Release Notes, so later you can identify this release

Deploy the server

  1. Open the OS-specific project
  2. Click the Releases menu item
  3. Select the release
  4. Click the DEPLOY TO button and select the environment

    1. If the project was already deployed to an environment, that environment does not show up in the list, so click the ellipsis and select Delpoy To from the dropdown menu.
    2. Select the environment from the list
  5. Click the tenant section to select the tenant (server) to launch
  6. Click the DEPLOY button

Adding new servers, operating systems, environments, and AWS accounts

New Server

  1. Create a new Tenant

New AMI

  1. Update the AMI in the OS-specific parent project
  2. Open the Releases page of the parent project
  3. Select the latest release
  4. Click the UPDATE VARIABLES link

New Application

  1. Create a new Tenant Tag in the “DevOps Applications” Tenant Tag Set
  2. Add the new Tenants to the Tenant Tag of the application

New Operating System

  1. Clone one of the DevOps AWS EC2 OS… projects
  2. Update the project settings, because those are not cloned
    1. Tenants required for all deployments
    2. Deployments with no target allowed
  3. Update the project variables for the new operating system
  4. Create a release of the new project
  5. Create new Tenants and link them to the new OS project

New Environment or AWS account

  1. Add the new account if necessary. See Add AWS AccountStore an AWS account reference in Octopus above.
  2. Create the DevOps-env-[environment][account number] environment
  3. Add the new environment to the “DevOps-aws-lifecycle”
  4. Add the new environment to the variables in the
    1. Add to the base project: “DevOps AWS EC2 Launch”
      1. account.aws_key
      2. account.aws_key_name
      3. environment.aws_account_credentials
      4. environment.chef_environment
      5. environment.instance_tag_environment
    2. Add to all OS specific projects: “DevOps AWS EC2 OS…
      1. parent.os.account.instance_common_security_group_id

New Chef Attribute

To pass in a new Chef attribute from the Tenant

  1. Add the new attribute to the chef_attributes.json file with variable substitution
    {
     "set_hostname": "#{host.set_hostname}",
     "agent_name": "#{host.agent_name}",
     "add_to_domain": "#{host.add_to_domain}"
    }
  2. Add the new variable to the Library Variable Set’s Variable Template with the parent. prefix
  3. Add the new variable to the passed in variable list in every parent project process. See Pass the variables from the parent project to the child project

Troubleshooting

To save the variable values to the log add the following two variables to the project

OctopusPrintVariables True
OctopusPrintEvaluatedVariables True

For the change to take effect

Create a new release

or 

Update the variable snapshot of the existing release

  1. On the Project -> Releases  page select the latest release
  2. In the Variable Snapshot section click the SHOW link
  3. Click the UPDATE VARIABLES link

More info at https://octopus.com/docs/support/debug-problems-with-octopus-variables

 

 

 

Open the system drive of an AWS instance you cannot log into

If you cannot log into an AWS instance and want to inspect files on it, you can detach the volume from the lost instance and attach it to another instance as the secondary drive.

Create a new instance

  1. Create a new AWS instance and log into it,
  2. Make a note of the Instance ID of this new instance,
  3. In the AWS console find your lost instance, that you cannot log into, by IP address or server name,
  4. Using the AWS console stop the lost instance,
  5. Make a note of the Instance ID of the stopped instance.

Detach the root volume from the stopped lost instance

  1. On the Description tab click the name of the root device
  2.  In the popup click the EBS ID
  3. Make a note of the volume ID,
  4. Click the Actions button and select Detach Volume.

Attach the volume to the new instance

  1. On the same, detached root volume page, click the Actions button and select Attach Volume,
  2. Search for the new instance by Instance ID and enter a name for the device
  3. On the Description tab of the new instance, the second drive appears,

Assign a drive letter to the attached drive

  1. Log into the new instance,
  2. On the toolbar click the Server Manager icon,
  3. In the Tools menu select Computer Management,
  4. On the left side select Disk Management,
  5. Right-click the second disk and select Online,
  6. The partitions of the attached volume automatically get the drive letters. The primary partition received the letter E:
  7. The drives show up in Windows Explorer too.

Create a new server image for a RightScale server template

The RightScale server templates publish server images to launch. It is advisable to create your own server image because the cloud providers can remove their published images anytime. If you generate your own image, you control the lifecycle of those.

Create your own server image

  1. Use Packer to create a new server image.
    1. Install RightLink.
      1. On Windows
            1. Create a PowerShell script and save it as install-rightlink.ps1
              $wc = New-Object System.Net.Webclient
              $wc.DownloadFile("https://rightlink.rightscale.com/rll/10/rightlink.install.ps1",
                 "$pwd\rightlink.install.ps1")
              Powershell -ExecutionPolicy Unrestricted -File rightlink.install.ps1
            2. Execute it from the Packer template
              "provisioners": [
                {
                  "type": "powershell",
                  "pause_before":"10s",
                  "scripts": [
                  "install-rightlink.ps1"
                  ]
                }
              ]
    2. For more information see “Create Custom Images with RightLink” at  http://docs.rightscale.com/rl10/getting_started.html
  2. Make the image public, so RightScale can see it
    1. Open the AWS console
    2. On the EC2 Dashboard select AMIs
    3. Find the image by AMI resource ID
    4. On the Permission tab click the Edit button
    5. Keep the image private, but share it with the AWs account you create the server template in.

Attach the image to your RightScale server template

Make sure the image is visible in RightScale.

It can take 30 minutes for the new image to appear in the list.

  1. Open the RightScale Cloud Management console
  2. In the upper right corner select the account you want to work in,
  3. In the Clouds menu find the region where you created the image and click the Images link
  4. Search for the image

If you have an existing MultiCloud image and want to update the server image in it

  1. In the Design menu select the MultiCloud Images link
  2. Find the existing image by name
  3. Click the name of the image to open it
  4. On the Clouds tab select region, you are working in and click the pencil icon
  5. Click the Edit Image button
  6. Type a part of the name of the new server image you have just created to find it
  7. Click the name of the image to select it
  8. Click the Save button to add it to the MultiCloud image
  9. Click the Update button to save the MultiCloud image

Test the new MultiColud Image

Set the MultiCloud image version in the Server Template

Test the new MultiCloud image by TEMPORARILY using the HEAD revision of the Server Template and the MultiCloud image

To test the new image TEMPORARILY select the HEAD revision of the MultiCloud image to be used by the Server Template

  1. In the Design menu select the ServerTemplates link
  2. Find your Server Template
  3. Click the name of the template to open the info page
  4. Select the HEAD revision of the Server Template
  5. On the Images tab click the pencil to set the revision of the MultiCloud image to use
  6. TEMPORARILY select the HEAD revision for the MultiCloud Image and click the green check mark to save the selection.

In your CAT file set revision 0 to use the HEAD revision of the Server Template

Launch a new server using Self Service with the HEAD revision of the Server Template


Create the new version of the MultiCloud Image

When you have successfully tested the new MultiCloud Image create a new version of it.

  1. Find your MultiCloud Image in Desing, MultiCloud Images
  2. Click the Commit button and enter a description for the change

Create the new version of the Server Template

Save a new version of the Server Template

  1. Find your Server Template in Design, ServerTemplates
  2. Click the Commit button

Publish the Server Template

  1. Select the last revision and click the Publish to MultiCloud Marketplace button
  2. Keep it private
  3. Select the account group to share with and click the DESCRIPTIONS button
  4. Click the PREVIEW button
  5. Click the PUBLISH button

Import the Server Image to the other accounts you use

To be able to use the new version of the Server Image switch to the other accounts and import the template

  1. In the Design menu under MultiCloud Marketplace select the ServerTemplates link
  2. Enter a portion of the Server Template name in the Keyword box and click the GO button
  3. Click the name of the template
  4. Click the Import button

If you want to use a new MultiCloud image in the Server Template

To create a new MultiCloud image

 

Add the new MultiCloud image to the server template

  1. Open the RightScale Cloud Management console
  2. In the Design menu select ServerTemplates
  3. Find your server template
  4. On the Images tab click the Add MultiCloud Image button