Stop multiple untagged AWS EC2 instances with a Bash script

 List all EC2 instances without a specific tag

One day we have found 499 instances running in our account without any tags. Most likely someone accidentally started a process to launch those, so we needed a way to find them and stop them. Later we will terminate them with the same script below when we can make sure those are not needed.

For simplicity, place the appropriate aws_access_key_id and aws_secret_access_key into the [default] section of the “~/.aws/credentials” file or use the –profile option in every command below.

List all instances

To list all EC2 instances, execute

aws ec2 describe-instances

List all instances missing a specific tag

I have found the command to list those instances that are missing the “Name” tag at https://www.onica.com/blog/using-aws-cli-to-find-untagged-instances/

I have directed the output to a text file with the additional last line.

To get all info on the instances with no “Name” tag into a JSON file

aws ec2 describe-instances \
--query 'Reservations[].Instances[?!not_null(Tags[?Key == `Name`].Value)]' \
> instances-no-name-tag.json

To output multiple properties into a tab-separated file for reporting in Excel.

aws ec2 describe-instances \
--output text \
--filters Name=instance-state-name,Values=running \
--query 'Reservations[].Instances[?!not_null(Tags[?Key == `Name`].Value)] | [].[InstanceId,ImageId,InstanceType,Platform,LaunchTime,SubnetId,KeyName]' \
> instance-info-no-name-tag.csv

Get the list of instance IDs into a text file for batch processing

aws ec2 describe-instances \
--output text \
--filters Name=instance-state-name,Values=running \
--query 'Reservations[].Instances[?!not_null(Tags[?Key == `Name`].Value)] | [].[InstanceId]' \
> instance-ids-no-name-tag.txt

Stop an instance with the instance Id

aws ec2 stop-instances --instance-ids MY_INSTANCE_ID

Stop multiple instances

To stop all instances listed in the “instance-ids-no-name-tag.txt” file created above, create and execute this Bash script:

#!/bin/bash

# The file with the instance IDs
filname=instance-ids-no-name-tag.txt

# Iterate through the lines
while read p; do
  echo "Stopping $p"
  aws ec2 stop-instances --instance-ids $p
done <$filname

 

Creating CentOS server images with Packer

The CentOS images are not available on the AWS Quick Start tab.

CentOS publishes official images on the AWS Marketplace, but you need to subscribe to the image to be able to launch it with an automation software, like Terraform.

Find the latest available CentOS image in the AWS Marketplace

  1. Execute this command to display the list of available images
    aws --region us-east-1 ec2 describe-images --owners aws-marketplace --filters Name=product-code,Values=aw0evgkw8e5c1q413zgy5pjce
  2. Select the latest AMI from the list. The images are NOT ordered by date!
  3. If you try to launch the image before you subscribe to it, the error message is displayed

    Error launching source instance: OptInRequired: In order to use this AWS Marketplace product you need to accept terms and subscribe. To do so please visit http://aws.amazon.com/marketplace/pp?sku=…
    amazon-ebs: status code: 401, request id: []

  4. Follow the link in the error message to http://aws.amazon.com/marketplace/pp?sku=…
  5. Click the Continue to Subscribe button
  6. Once the new image has been created, share it with all the accounts you use.

SocketError: Error connecting to … Name or service not known

In an enterprise environment, the company usually operates its own DNS servers.

When a Linux instance launches in AWS, the DNS settings only contain the AWS DNS server. If the company DNS server settings are applied with Chef, during the first Chef Client run those settings do not take effect.

When we reference an internal DNS entry during the first Chef Client run we get the error message:

 

SocketError: Error connecting to https://INTERNAL_ADDRESS.net/… – Failed to open TCP connection to INTERNAL_ADDRESS.net:443 (getaddrinfo: Name or service not known)

There two solutions to handle this:

  1. Use the IP address of the internal server or load balancer in the URL ( switch from HTTPS to HTTP, because the SSL certificate will not work with the IP address),
  2. Set the DNS servers of the Linux server with Packer when you generate the base image.

“Ran out of time waiting for the server with id” with Windows Server 2016 in Chef Test Kitchen

AWS changed how Windows Server EC2 instances send messages during boot.

Windows Server 2012 R2 AWS EC2 instances sent the “Windows is ready” message every time those became available after boot.

When a Windows Server 2016 AWS EC2 instance launches, it only sends the “Windows is ready” message during the first boot. If you create your custom AMI with Packer, the first boot happens during the Packer AMI creation, so Chef Test Kitchen does not receive the expected message and the process times out with the message

>>>>>> Ran out of time waiting for the server with id [i-… to become ready, attempting to destroy it
EC2 instance <i-…> destroyed.

To configure Windows Server 2016 instances to send the expected “Windows is ready” message during every boot, add a PowerShell line to your Packer template that creates your custom AMI:

{
  "type": "powershell",
  "pause_before":"2s",
  "elevated_user": "MY_ADMIN_USER",
  "elevated_password": "MY_ADMIN_PASSWORD",
  "inline": [
    "C:\\ProgramData\\Amazon\\EC2-Windows\\Launch\\Scripts\\SendWindowsIsReady.ps1 -Schedule"
  ]
},

This code creates a scheduled task to execute the configuration script on the instance.

More information is at https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/ec2launch.html#ec2launch-sendwinisready

 

Find the AWS account number

The AWS account number uniquely identifies the AWs account you are working with. All AWS “arn” identifiers contain it, and you need to know it when you want to share AMIs with other accounts.

If there are no resources created yet in the account, you can find the account number in the “arn” of your user account.

  1. Log into the AWS Console
  2. In the upper right corner click your username and select My Security Credentials
  3. On the left side select Users, and click any of the usernames in the list
  4. The account number is part of the “User ARN”

Open the system drive of an AWS instance you cannot log into

If you cannot log into an AWS instance and want to inspect files on it, you can detach the volume from the lost instance and attach it to another instance as the secondary drive.

Create a new instance

  1. Create a new AWS instance and log into it,
  2. Make a note of the Instance ID of this new instance,
  3. In the AWS console find your lost instance, that you cannot log into, by IP address or server name,
  4. Using the AWS console stop the lost instance,
  5. Make a note of the Instance ID of the stopped instance.

Detach the root volume from the stopped lost instance

  1. On the Description tab click the name of the root device
  2.  In the popup click the EBS ID
  3. Make a note of the volume ID,
  4. Click the Actions button and select Detach Volume.

Attach the volume to the new instance

  1. On the same, detached root volume page, click the Actions button and select Attach Volume,
  2. Search for the new instance by Instance ID and enter a name for the device
  3. On the Description tab of the new instance, the second drive appears,

Assign a drive letter to the attached drive

  1. Log into the new instance,
  2. On the toolbar click the Server Manager icon,
  3. In the Tools menu select Computer Management,
  4. On the left side select Disk Management,
  5. Right-click the second disk and select Online,
  6. The partitions of the attached volume automatically get the drive letters. The primary partition received the letter E:
  7. The drives show up in Windows Explorer too.

Create a new server image for a RightScale server template

The RightScale server templates publish server images to launch. It is advisable to create your own server image because the cloud providers can remove their published images anytime. If you generate your own image, you control the lifecycle of those.

Create your own server image

  1. Use Packer to create a new server image.
    1. Install RightLink.
      1. On Windows
            1. Create a PowerShell script and save it as install-rightlink.ps1
              $wc = New-Object System.Net.Webclient
              $wc.DownloadFile("https://rightlink.rightscale.com/rll/10/rightlink.install.ps1",
                 "$pwd\rightlink.install.ps1")
              Powershell -ExecutionPolicy Unrestricted -File rightlink.install.ps1
            2. Execute it from the Packer template
              "provisioners": [
                {
                  "type": "powershell",
                  "pause_before":"10s",
                  "scripts": [
                  "install-rightlink.ps1"
                  ]
                }
              ]
    2. For more information see “Create Custom Images with RightLink” at  http://docs.rightscale.com/rl10/getting_started.html
  2. Make the image public, so RightScale can see it
    1. Open the AWS console
    2. On the EC2 Dashboard select AMIs
    3. Find the image by AMI resource ID
    4. On the Permission tab click the Edit button
    5. Keep the image private, but share it with the AWs account you create the server template in.

Attach the image to your RightScale server template

Make sure the image is visible in RightScale.

It can take 30 minutes for the new image to appear in the list.

  1. Open the RightScale Cloud Management console
  2. In the upper right corner select the account you want to work in,
  3. In the Clouds menu find the region where you created the image and click the Images link
  4. Search for the image

If you have an existing MultiCloud image and want to update the server image in it

  1. In the Design menu select the MultiCloud Images link
  2. Find the existing image by name
  3. Click the name of the image to open it
  4. On the Clouds tab select region, you are working in and click the pencil icon
  5. Click the Edit Image button
  6. Type a part of the name of the new server image you have just created to find it
  7. Click the name of the image to select it
  8. Click the Save button to add it to the MultiCloud image
  9. Click the Update button to save the MultiCloud image

Test the new MultiColud Image

Set the MultiCloud image version in the Server Template

Test the new MultiCloud image by TEMPORARILY using the HEAD revision of the Server Template and the MultiCloud image

To test the new image TEMPORARILY select the HEAD revision of the MultiCloud image to be used by the Server Template

  1. In the Design menu select the ServerTemplates link
  2. Find your Server Template
  3. Click the name of the template to open the info page
  4. Select the HEAD revision of the Server Template
  5. On the Images tab click the pencil to set the revision of the MultiCloud image to use
  6. TEMPORARILY select the HEAD revision for the MultiCloud Image and click the green check mark to save the selection.

In your CAT file set revision 0 to use the HEAD revision of the Server Template

Launch a new server using Self Service with the HEAD revision of the Server Template


Create the new version of the MultiCloud Image

When you have successfully tested the new MultiCloud Image create a new version of it.

  1. Find your MultiCloud Image in Desing, MultiCloud Images
  2. Click the Commit button and enter a description for the change

Create the new version of the Server Template

Save a new version of the Server Template

  1. Find your Server Template in Design, ServerTemplates
  2. Click the Commit button

Publish the Server Template

  1. Select the last revision and click the Publish to MultiCloud Marketplace button
  2. Keep it private
  3. Select the account group to share with and click the DESCRIPTIONS button
  4. Click the PREVIEW button
  5. Click the PUBLISH button

Import the Server Image to the other accounts you use

To be able to use the new version of the Server Image switch to the other accounts and import the template

  1. In the Design menu under MultiCloud Marketplace select the ServerTemplates link
  2. Enter a portion of the Server Template name in the Keyword box and click the GO button
  3. Click the name of the template
  4. Click the Import button

If you want to use a new MultiCloud image in the Server Template

To create a new MultiCloud image

 

Add the new MultiCloud image to the server template

  1. Open the RightScale Cloud Management console
  2. In the Design menu select ServerTemplates
  3. Find your server template
  4. On the Images tab click the Add MultiCloud Image button

amazon-ebs: Error waiting for SSH: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

When you launch a Linux AWS EC2 instance with Terraform or create a Linux AWS image with Packer, one of the following errors are displayed:

amazon-ebs: Error waiting for SSH: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

aws_instance.default: 1 error(s) occurred:
* ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

Possible causes

Incorrect username

In the Packer template the “ssh_username”, in the Terraform script the “user” attribute contains the username to connect to the instance. It is important to use the correct username for each operating system. For the list of correct usernames for each operating system, see Create a server image with Packer

Wrong SSH key

Make sure you use the correct SSH key to connect to the instance. In the Terraform file, the “key_name”  attribute of the “aws_instance resource” specifies the SSH key to launch the instance with, and the “private_key” attribute of the “connection” contains the key itself as a string.

 

Splunk App for AWS

To collect data from AWS install the Splunk App for AWS plugin.

The app currently can collect information from

  • AWS Config,
  • Config Rules,
  • CloudTrail,
  • Inspector,
  • CloudWatch,
  • CloudWatch Logs,
  • Billing,
  • S3,
  • Kinesis,
  • Metadata.

New data source

To set up a new data source, click the Set up button

Already set up data source

To add a new input to the data source, click the New Input button

To view the list of existing inputs, click the link in the section

To view the settings of an existing input, click the name of the input, to add a new input, click the Add Input button. There is no way to edit the settings of existing inputs, to make changes, create a new input. There is also no method to disable an existing input, to stop the data collection, delete the input in the Action column.

S3

To read and process files from S3, select the S3 data source.

To select the Source Type click the Source Type field. If the source type is in the list, select it.

To add a source type, not in the list, enter the name into the text box, and click the generated link under the text box.

To collect the data more frequently than the default 30-minute interval, click the Advanced Settings link and enter the interval in seconds.