The specified Security Group and Parameter Group are not set in the RDS instance

If the Terraform apply execution times out during the RDS instance creation, the specified Security Group and Parameter Group is not set in the RDS instance.

The solution is to set the timeout in the aws_db_instance resource. When a multi-az RDS instance is launched from a snapshot, the process can take more than 55 minutes. The default value is 40 minutes.

resource "aws_db_instance" "default" {
...
  timeouts {
    create = "120m"
    delete = "120m"
  }
...
}

SQL Server AWS RDS instance ALARM FreeableMemory <=... MB

The SQL database servers use the available memory for caching to speed up the database operation. If we do not restrict the SQL database server memory usage, the operating system will not have enough memory to run. This setting is also necessary for an AWS RDS instance, otherwise, you will get the alert

ALARM FreeableMemory <=… MB

In AWS we can specify the maximum SQL server memory dynamically, so every RDS instance type will leave enough memory for the operating system regardless of the size of the available memory size. in this example, we will leave 1.5 GB (1536 MB) memory for the operating system so the default 1024 MB free memory alarm will not sound.

DBInstanceClassMemory returns the total memory size in bytes, so we need to convert the value to MB, to be able to set the value of “max server memory (mb)” to the correct number.

If you use Terraform to create your RDS instance, create a script with the aws_db_parameter_group resource to create a Parameter Group in your AWS account. You need to execute it once, as all RDS instances will use the same group.

resource "aws_db_parameter_group" "default" {

  name = "max-server-memory"
  family = "sqlserver-se-12.0"
  description = "DBInstanceClassMemory"

  parameter {
    name = "custom-sqlserver-se-12-0"
    value = "SUM({DBInstanceClassMemory/1048576},-1536)"
    apply_method = "immediate"
  }
}

In the RDS instance creation script assign the Parameter Group to the RDS instance and increase the timeout of the create and delete operations to make sure Terraform waits during the creation and deletion process.

resource "aws_db_instance" "default" {
...
  # Add the Max Server Memory parameter group to the instance
  parameter_group_name = "custom-sqlserver-se-12-0"
...
  timeouts {
    create = "120m"
    delete = "120m"
  }
...
}

 

 

 

Failed to complete #create action: [undefined method `version’ for nil:NilClass] on default…

When you execute kitchen converge to launch an EC2 instance in AWS with Chef Test Kitchen, you get the error message:

>>>>> ——Exception——-
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: 1 actions failed.
>>>>>> Failed to complete #create action: [undefined method `version’ for nil:NilClass] on default-windows-2012r2
>>>>>> ———————-
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose –all` for configuration

The kitchen-ec2 driver tries to read the operating system version from the name of the image, but it cannot find it.

When you create your own AMI (Amazon Machine Image), make sure the version of the operating system is in the name:

Windows-2012
Windows-2012r2
Windows-2012r2sp1
RHEL-7.2

An acceptable name is my_windows-2012r2_base-1

Could not load the ‘ec2’ driver from the load path

When you execute kitchen list and the driver in your .kitchen file is “ec2“, the following error message appears:

>>>> ——Exception——-
>>>>>> Class: Kitchen::ClientError
>>>>>> Message: Could not load the ‘ec2’ driver from the load path. Please ensure that your driver is installed as a gem or included in your Gemfile if using Bundler.
>>>>>> ———————-
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose –all` for configuration

If you have recently installed the Chef Development Kit, you need to install the Kitchen EC2 driver to be able to launch servers in AWS.

To install the Kitchen EC2 driver, execute

chef gem install kitchen-ec2

 

CloudExceptions::CloudException – 400: VPCIdNotSpecified: No default VPC for this user

When you launch a new EC2 instance in the AWS cloud from the command line or with other cloud management platforms, you may get the error message:

CloudExceptions::CloudException – 400: VPCIdNotSpecified: No default VPC for this user (RequestID: …)

This can happen when the specified Subnet Id is not a valid subnet in the selected availability zone (datacenter).

Create the AWS credentials file from a Chef Data Bag

When a process on a server instance needs access to an AWS account, the user who will execute the AWS CLI commands needs to be able to automatically authenticate in AWS.

For automatic AWS authentication, the AWS CLI creates two files in the .aws directory:

  • config and
  • credentials.

The location of this directory depends on the operating system and the type of user.

  • On Linux, the location is ~/.aws ( the user’s home directory )
  • On Windows, it is located at C:\Users\USER_NAME\.aws
  • On Windows, if the file was created by SYSTEM, the location is C:\Windows\System32\config\systemprofile\.aws

Store the AWS key values

To create these files, you need to store the AWS Access Key and Secret Key. The safest place for these values is an encrypted data bag. To automatically generate the AWS files, create a data bag file and name it the same as the “id” in the following structure:

{
  "id": "MY_DATA_BAG_ITEM_NAME",
  "MY_PROFiLE_1": {
    "region": "MY_REGION_1",
    "aws_access_key_id": "MY_ACCESSKEY_1",
    "aws_secret_access_key": "MY_SECRET_KEY_1"
  },
  "MY_PROFiLE_2": {
    "region": "MY_REGION_2",
    "aws_access_key_id": "MY_ACCESSKEY_2",
    "aws_secret_access_key": "MY_SECRET_KEY_2"
  }
}

To create and encrypt the data bag see my post on Chef Data Bags

Create the AWS authentication files

  1. In your Chef recipe, first install the AWS CLI and reboot the server, so the new path entry will be available for the Chef process.
  2. The following Chef code will create the AWS config and credential files. The script
    1. opens and decrypts the data bag,
    2. loads it into a hash table,
    3. iterates through the hash items,
    4. skips the “id” item,
    5. stores the AWS key values in a temporary file,
    6. executes the “aws configure” command to generate the AWS config and credential files.
  # Iterate through the data bag and create the credentials file

  puts "***** Creating the AWS credentials file"

  # Load the encrypted data bag into a hash
  aws_credentials = Chef::EncryptedDataBagItem.load('MY_DATA_BAG_NAME', 'MY_DATA_BAG_ITEM_NAME').to_hash

  # Iterate through the items, skip the "id"
  aws_credentials.each_pair do |key, value|

    # skip the "id"
    next if key == "id"

    # Add the credentials to the .aws/credentials file
    puts "Account #{key}, Region #{value['region']}"

    batch "add_aws_credentials_#{key}" do
      code <<-EOF echo #{value["aws_access_key_id"]}> input.txt
        echo #{value["aws_secret_access_key"]}>> input.txt
        echo #{value["region"]}>> input.txt
        echo.>> input.txt
        aws configure --profile #{key} < input.txt
      EOF
    end

  end

 

Get AWS SSL Certificate resource ids from existing Load Balancers

To launch an Elastic Load Balancer ( ELB ) with an existing SSL certificate using Terraform, you need to specify the AWS certificate resource id. If you have already uploaded the certificate and attached it to an existing load balancer, the following AWS CLI command will display it in the command window. MY_PROFILE is the name of the profile in the square brackets [] in the ~/.aws/credentials file.

aws elb describe-load-balancers --region MY_AWS_REGION --profile MY_PROFILE |grep SSL

To get all information on the load balancers, just omit the grep command:

aws elb describe-load-balancers --region MY_AWS_REGION --profile MY_PROFILE

Convert PEM files to PPK to use them in PuTTY

When you create a key in AWS you can download it one time in PEM format. To use it in PuTTY, the free SSH and Telnet client, you have to convert it to PPK format.

To install PuTTY, see the Terminal Emulator section in Recommended utilities for your workstation

To convert a PEM file to PPK

  1. Open a terminal window in the folder of the PEM file
  2. Execute the following
    puttygen MYKEY.pem -o MYKEY.ppk

 

Error waiting for instance (i-…) to become ready: unexpected state ‘terminated’, wanted target ‘running’

When you launch a server instance with Terraform, sometimes the error message does not contain the underlying cause. When the cloud provider cannot complete the request, many times Terraform displays a generic error message:

Error waiting for instance (i-...) to become ready: unexpected state 'terminated', wanted target 'running'

To find the root cause of the error in AWS

  1. Log into the AWS console and navigate to the EC2 section,
  2. Search for the instance by the instance Id,
  3. You can find the error message at the bottom of the Description tab

In our specific case, it was Client.VolumeLimitExceeded: Volume limit exceeded

We had to increase the volume limit to be able to launch more large EC2 instances.

Upgrade a server on an AWS EC2 instance with minimum risk and downtime

When you need to upgrade an application on an AWS EC2 instance with minimum downtime, there are many options.

Upgrade an EC2 instance

  1. Stop the application, so users don’t make more changes,
  2. Create a backup of the database (snapshot of the RDS instance),
  3. Create a backup of the server (backup image of the EC2 instance),
  4. Upgrade the application on the server,
  5. Start the application,
  6. Test the new version of the application.

In case the upgrade fails, or the post-upgrade test fails

  1. Stop the application,
  2. If you use RightScale to launch servers:
    1. stop the RightLink service on the failed server to prevent RightScale auto terminating the restored server
      (When RightScale finds a new identical server instance, it automatically shuts it down to avoid multiple instances with the same identifier.)
    2. Terminate the failed server
  3. Stop the failed server,
  4. Restore the database from the pre-upgrade backup with a new name,
  5. Restore the server instance from the pre-upgrade backup,
  6. Start the restored server,
  7. Change the database address in the application’s config file to point to the restored database,
  8. Start the application,
  9. Test the restored version of the application.

To make the backup image of the AWS EC2 instance

  1. Open the AWS console and navigate to EC2, Running instances
  2. Find the server instance you want to backup
  3. Right-click the row of the instance and select Image, Create Image

  4. Name the image and click the Create Image button
  5. Save the image Id from the popup