Migrate databases to and from AWS RDS instances

Amazon Web Services do not provide tools to migrate data to, from and between AWS RDS instances.

There is a free utility that can do the job.

SQL Server Management Studio (SSMS)

Install Microsoft SQL Server Management Studio (SSMS), because the utility uses libraries from it.

Download the installer for free from https://msdn.microsoft.com/en-us/library/mt238290.aspx

SQL Azure Migration Wizard

Installation

Download the SQL Azure Migration Wizard from http://sqlazuremw.codeplex.com/
Unzip the downloaded ZIP file into a folder on your hard drive
Create a shortcut in your Start Menu for SQLAzureMW.exe

Migrate a database

Start the SQL Azure Migration Wizard by executing the SQLAzureMW.exe file
Select Database in the Analyze / Migrate section and click Next

Connect to the source database

On the Connect to Server page
1. Enter the address of the database server
2. Select SQL Server Authentication
3. Enter the database owner login name and password
4. If you keep Master DB selected, the next screen will show the list of databases to select from
5. To save the username and password select Save Login Information
6. Click Connect
Select the database to connect to and click Next
Leave Script all database objects selected and click Next
On the Script Wizard Summary page click Next
Click Yes to start the read process
On the Results Summary page select Auto Scroll Display to see the scrolling results
When the read is done click Next

Connect to the target database

Enter the connection properties of the target database server
Select the target database
1. To migrate the data into a new database click the Create Database button
2. On mirrored RDS instances we cannot drop and recreate databases. To migrate data into an existing database
  1. Use SQL Management Studio to drop the existing tables to avoid key conflicts
  2. Select the target database on this page
Click Next
Click Yes to start the write process
When the write process is done click Exit

Chef file locations

The Chef file and folder locations are different on Linux and Windows machines. This article explains the purpose of each file and the location.

Summary

	Linux	Windows
Cookbook location	/var/chef/cache/cookbooks	C:\chef\cache\cookbooks
Chef Client run log	/var/log/chef.log	First run only C:\chef\chef-client.log
		Subsequent Chef client runs C:\chef\log\client.log
Error log	/var/chef/cache/chef-stacktrace.out	C:\chef\cache\chef-stacktrace.out
Ohai output	/var/chef/cache/failed-run-data.json	C:\chef\cache\failed-run-data.json
Recommended location for custom log files	/var/log/cheflog.log	C:\Logs\Chef\cheflog.log
Chef Client configuration	/etc/chef/client.rb	C:\chef\client.rb

When you test your cookbook in Test Kitchen

The .kitchen.yml file contains the username to execute the Chef cookbook. It is specified under platforms:, transport:, username:

Use that value in place of USER-NAME-FROM-KITCHEN-YML below.

	Linux	Windows
Cookbook location	/tmp/kitchen/cookbooks /tmp/kitchen/cache/cookbooks	C:\Users\USER-NAME-FROM-KITCHEN-YML\AppData\Local\Temp\kitchen\cookbooks
Error log	/tmp/kitchen/cache/chef-stacktrace.out	C:\Users\USER-NAME-FROM-KITCHEN-YML\AppData\Local\Temp\kitchen\cache\chef-stacktrace.out
Ohai output	/tmp/kitchen/cache/failed-run-data.json	C:\Users\USER-NAME-FROM-KITCHEN-YML\AppData\Local\Temp\kitchen\cache\failed-run-data.json
Data bags	/tmp/kitchen/data_bags	C:\Users\USER-NAME-FROM-KITCHEN-YML\AppData\Local\Temp\kitchen\data_bags

Cookbook location

When the Chef recipes are executed, all cookbooks are stored on the node. You can examine the code to make sure your latest changes are reflected on the machine.

The log of the Chef client run

The output of the Chef cookbook execution is in the chef.log or chef-client.log file

On Windows

The log of the first Chef Client run and subsequent runs are stored in different log files. After the initial Chef Client run, the rest of the log entries are collected in the second file.

Stacktrace

Chef saves information on the hard drive when scripts are executed. If there is a failure, the stack trace of the last error is saved in the chef-stacktrace.out file.

Ohai output

All the information that Ohai collects on the instance, is saved in the failed-run-data.json file, even if there is no error. It is a great resource to get the server specific values.

Cloud info

cloud-specific information under “cloud“
cloud instance information under “ec2“

Computer info

CPU and memory configuration under “cpu“, “cs_info“, “memory“
drive sizes and network settings under “filesystem” and “network“

Operating system info

operating system information under “os_info“
list of enabled Windows features under “dism_features“
the list of installed applications under “packages“

Chef info

Chef client configuration values under “chef_client“
the Chef node information under “chef_type”: “node“
the Chef run list under “run_list“
list of Chef cookbooks and their versions under “cookbooks“
list of the executed recipes under “recipes“
the value of the passed in Chef attributes under “normal“
the value of the Chef cookbook attributes under “chef_client” and the cookbook name
all the information on the Chef resources under “json_class“
the stack trace of the last error under “exception“

Debug mode

To get detailed information on the Chef run, start the Chef Client in debug mode. Execute the command in sudo mode, because many Chef steps require elevated rights.

chef-client -l debug -L debug.txt

The debug information is saved in the debug.txt file in the current folder.

Internalize Chocolatey packages

To access Chocolatey packages in your private network you need to download them from the Internet and store them at a location where all servers can access them. You can internalize Chocolatey packages if you have a Chocolatey Business subscription.

You can use an Artifactory server to host the internalized packages.

List of available packages

To get the list of the available Chocolatey packages on an Artifactory server

choco list -s http://ARTIFACTORY_SERVER_URL/artifactory/api/nuget/ARTIFACTORY_REPOSITORY_NAME

Non copyrighted applications

If the Chocolatey package does not contain copyrighted components, Chocolatey can download and repackage the entire package, including the application.

Download a Chocolatey package from the Internet and store it on your local drive

choco download PACKAGE_NAME --internalize

To store the Chocolatey packages on an Artifactory server

The Artifactory server has to have a Pro license
Create a NuGet type local repository

Upload all packages from your local drive to an Artifactory server. Some of the large packages depend on other NuGet packages that have to be available on your Artifactory server.

curl -u ARTIFACTORY_USERNAME:ARTIFACTORY_PASSWORD -X PUT "http://ARTIFACTORY_SERVER_URL/artifactory/ARTIFACTORY_REPOSITORY_NAME/FOLDER_NAME/PACKAGE_FILE_NAME.nupkg" -T ./PACKAGE_FILE_NAME.nupkg

Copyrighted applications

If the Chocolatey package installs an application that is copyrighted, the author of the package cannot publish the copyrighted source, but can place the download location and installation process into the package.

Internalize the package

To download the necessary files and create a custom Chocolatey package with copyrighted application source:

1. Download the Chocolatey package without the internalize option
```
choco download PACKAGE_NAME
```
  Example:
```
choco download javaruntime
```
2. Delete the small .nupkg files. These are too small to contain the installer files. We will recreate them in a subfolder with the downloaded installer files.
3. Find the package folder in the download sub-folder that has a tools subfolder. In the case of “javaruntime”, the real package is the “jre8”, not “javaruntime”.
4. Open the tools\chocolateyInstall.ps1 file
5. Find the download urls of the installer files.
  
  Some scripts contain variables for the version of the application, so you have to manually assemble the final download URL.
6. Assemble the actual download URL and download the application source with your web browser,
7. Place the downloaded installer files into the tools folder,
8. Update the chocolateyInstall.ps1 to point to the installer files in the package:
  - Add the following before the $url =… and $url64 =… statements
```
$toolsDir = "$(Split-Path -parent $MyInvocation.MyCommand.Definition)"
```
  - Replace the the $url =… and $url64 =… statements. Use the actual file names, and make sure to use double quotes (“) for the string interpolation to work!
```
$url = "$toolsDir\32_BIT_INSTALLER_FILE"
$url64 = "$toolsDir\64_BIT_INSTALLER_FILE"
```
  - Add
```
-UseOriginalLocation
```
    to the Install-ChocolateyPackage line.
9. Right click the .nuspec file in the package folder and select Compile Chocolatey Package
  
  (or execute choco pack PATH\TO\NUSPEC.nuspec)
10. The package is created in the folder of the .nuspec file and the file name is the composite of id + “.” + version tag values in the .nuspec file.

Upload the package to the local repository

In the command line execute

curl -u ARTIFACTORY_USERNAME:ARTIFACTORY_PASSWORD -X PUT "http://ARTIFACTORY_SERVER_URL/artifactory/ARTIFACTORY_REPOSITORY_NAME/PACKAGE_NAME/PACKAGE_FILE_NAME.nupkg" -T PATH/TO/PACKAGE_FILE_NAME.nupkg

Video tutorials

Package Synchronizer

…refers to a location that is unavailable

When you try to copy a file from an external drive, or a network location, you may get the following error message:

On Windows 10

… is unavailable. If the location is on this PC, make sure the device or drive is connected or the disc is inserted, and then try again. If the location is on a network, make sure you’re connected to the network or Internet, and then try again. If the location still can’t be found, it might have been moved or deleted.

On Windows 7

… refers to a location that is unavailable. It could be on a hard drive on this computer, or on a network. Check to make sure that the disk is properly inserted, or that you are connected to the Internet or your network, and try again. If it still cannot be located, the information might have been moved to a different location.

This can happen when you try to copy a file from an external drive or network location to a folder on your computer, and the operating system wants to ask your permission to access the target location.

On Windows 10

On Windows 7

For some reason clicking the Continue button does not satisfy Windows when you copy from a network location.

To be able to copy the file, first copy the file to the desktop, and move it to the destination folder.

Recommended utilities for your workstation

Graphics

Snagit

Snagit is a great utility to capture screens and annotate them for documentation. The program is available for Mac and Windows.

Download the latest version at https://www.techsmith.com/

Get older versions at https://www.techsmith.com/download/oldversions

Terminal emulator

PuTTY

Install PuTTY

On Macintosh

Open a terminal window
Install PuTTY with Homebrew. For the Homebrew installation see the Homebrew section of Install the DevOps development tools on Macintosh
```
brew install putty
```

On Windows

Download the MSI (‘Windows Installer’) of PuTTY from https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html,
Execute the MSI installer.

How to get into the BIOS menu on a Windows 10 computer.

When you select Shut down on a Windows 10 computer, your machine is really not shut down. When you turn the computer on again it executes a fast start, so you cannot open the BIOS menu.

To get into the computer’s BIOS first shut down you computer as described in How to completely shut down a Windows 10 computer
Press and hold the keyboard button that opens the BIOS menu,

Manufacturer	Keyboard button
ASUS mother boards	Del

Keep the keyboard button pressed and turn on the computer.

How to completely shut down a Windows 10 computer

Windows 10 does not really shut down the computer when you select Shut down. This enables the operating system to start quickly, but if you want to cold start your computer to enter the BIOS menu you need to turn off your computer. To really shut down your Windows 10 computer:

Right click the Windows logo in the lower left corner of the screen
Click Shut down or sign out
Keep the Shift button pressed on your keyboard and click Shut down

ASUS motherboard does not boot after BIOS update

The motherboard BIOS update can cause problems, it is possible that after the update the computer does not boot anymore.

The ASUS motherboards come with the USB BIOS Flashback feature to load the BIOS even if the computer does not boot anymore.

The process below is for the ASUS P8Z77-V PRO/THUNDERBOLT motherboard, but all modern ASUS motherboards provide the same functionality.

Before BIOS update make sure you load the optimized BIOS configuration, otherwise after the BIOS update the computer may not boot anymore

Load the optimized BIOS configuration before the BIOS update

Restart the computer,
During the startup press the Del button to enter the BIOS setup,
Press F5 to load the optimized configuration,
Press F10 to save the changes and exit.

To reload the BIOS after a failed update

To fix a corrupted BIOS, you need access to a working computer that is connected to the Internet and a USB stick (pen drive) that is formatted with the FAT file system. Older motherboards cannot read the NTFS file system during the BIOS Flashback process.

Download the “Bios Renamer for USB BIOS Flashback” utility

On the asus.com web site select Support in the menu,
Select Consumer,
In the search box enter the model of your motherboard,
In the sub-menu of the product page select Support,
Click Driver & Tools,
In the Please select OS drop down menu select Others,
In the BIOS-Utilities section download the Bios Renamer for USB BIOS Flashback.

Download the BIOS for your motherboard

In the Please select OS drop down menu select the operating system you use,
In the BIOS section download the latest BIOS file.

Rename the downloaded BIOS file

The USB BIOS Flashback searches for a specific file on the USB device that you insert into the BIOS FlashBack USB port (marked with green border). The Bios Renamer for USB BIOS Flashback utility will rename the downloaded BIOS file to the correct name for your mother board

Create a folder for the BIOS file,
Extract the BIOS file from the downloaded BIOS zip file and place it in the folder,
Extract the BRenamerl.exe utility from the zip file and place it in the same folder,
Double click the BRenamerl.exe file to execute it. The program will rename the BIOS file, so the Flashback utility can find it.
Copy the renamed file to the FAT formatted USB stick.

The BIOS file names

Motherboard model	BIOS file name
P8Z77-V PRO/THUNDERBOLT	Z77VPTB.CAP

Flash the BIOS of the unresponsive ASUS motherboard

Turn off the computer,
Open the computer case to get access to the component side of the motherboard,
Insert the USB stick into the green framed USB Flashback port,
Plug the computer into the wall outlet,
If the power supply has a switch, turn it on, but do not start the computer,
Keep the BIOS Flashback button pressed on the motherboard for 3 seconds,
- If the BIOS Flachback LED keeps blinking, the update starts. When the LED goes dark, the update has completed.
- If the BIOS Flachback LED blinks a few more times and stays lit, the USB drive is not inserted to the correct port, not readable, the BIOS file name is not correct, or the file is corrupted. Turn on the computer for a few seconds to clear the BIOS Flashback LED light.
If the BIOS Flashback light goes out in a few minutes, the BIOS flashback has succeeded.

Load the optimized BIOS configuration as described at the top of this post.

Delay updates in Windows 10

On more occasions Microsoft released Windows 10 updates that disabled the network connection, the webcam or caused other inconveniences. Windows 10 automatically downloads and installs the updates without asking for permission from the owner of the computer. In Windows professional you can delay the installation of updates, so your work is not interrupted with frequent recommended updates, and Microsoft can fix the update before your computer installs it. The important security updates are still installed when those are released.

To defer Windows 10 updates

Click the Windows icon in the lower left corner and select the gear (Settings)

Inline image 2

On the main Settings page select Update & security

Inline image 3

Under Update settings select Advanced options

Inline image 4

On the Advanced options page select Defer feature updates

Inline image 1

Keep multiple versions of applications on Macintosh

Most of the DevOps tools are still in beta versions, many times the new version is not compatible with your existing scripts or have an error that stops your scripts working. To be able to keep multiple versions of the applications and easily switch between them, create symbolic links and point to the version you want to execute.

Create version-specific locations

Create a folder for your optional applications.

mkdir /opt

Set you as the owner of the “opt” directory and its children.

YOUR_GROUP should be on:

MacOS: wheel
Ubuntu: your username

sudo chown -R YOUR_USER_NAME:YOUR_GROUP /opt

Set the security of the folder.

sudo chmod 755 /opt

We will create a directory structure to place each version of the application into its own folder. This example uses Packer.

Create a symbolic link to point to the application in the appropriate version folder. These are the examples for Packer, Terraform, and Vagrant. Notice, that the Vagrant application is in the “bin” directory, so the symbolic link has to point there.

If there is an existing symbolic link that points to the latest version of the application, rename it to be able to quickly return to the prior version of the application. This can come handy when we need to destroy instances created with the prior version of Terraform.

cd /opt/terraform
mv terraform terraform_v_0.9.11

cd /opt/packer
mv packer packer_v_1.0.4

Create a new symbolic link that points to the latest version of the application.

# For Packer
cd /opt/packer
ln -s /opt/packer/packer_1.2.0/packer packer

# For Terraform
cd /opt/terraform
ln -s /opt/terraform/terraform_0.10.0/terraform terraform

# For Vagrant
cd /opt/vagrant
ln -s /opt/vagrant/vagrant_1.9.5/bin/vagrant vagrant #The vagrant application is in the bin folder

Add the main application folder to the path in the config file of your terminal

for iTerm2 with zterminal: ~/.zshrc
for other terminals: ~/.bash_profile

PATH=/opt/packer:$PATH
PATH=/opt/terraform:$PATH
PATH=/opt/vagrant:$PATH

Vagrant

For Vagrant installation see Vagrant

To change the version you want to execute

Delete the symbolic link,
```
cd /opt/packer
rm packer
```
Create a new symbolic link pointing to the other version of the application as above.