Cloud-based Cluster Setup

Anaconda for cluster management can launch and bootstrap clusters on a variety of cloud services. We currently support Amazon EC2. Other providers such as Microsoft Azure, Rackspace, Google Cloud Platform, and others are on our roadmap. If you are interested in using a cloud provider that is not listed here, please contact us at sales@anaconda.com.

Cloud configuration involves the use of profiles and providers. Splitting configurations allows users to easily share and distribute specific cluster configurations (profiles), while retaining private authentication credentials (providers).

1. Provider Setup

Authentication and cloud definitions are managed in a single provider file on the client machine: ~/.acluster/providers.yaml. A provider file can contain multiple providers with different settings and credentials.

A sample providers file is included with a new installation of Anaconda for cluster management and is located within the ~/.acluster/providers.yaml file on the client machine. You can edit the contents of this file to reflect the settings and credentials for your cloud provider.

An example ~/.acluster/providers.yaml file with a provider named aws_east that is configured for Amazon EC2 is shown below:

aws_east:
  cloud_provider: ec2
  keyname: my-private-key
  location: us-east-1
  private_key: ~/.ssh/my-private-key.pem
  secret_id: AKIAXXXXXX
  secret_key: XXXXXXXXXX

Note that you will need access to ports 22, 4505, and 4506 from the client machine to the cluster nodes to provision the cluster via SSH and Salt.

Refer to the Provider settings page for more details about provider settings, including security groups.

Linux or Mac: You should set the permissions of providers.yaml file to 0600.

2. Profile Setup

The settings for each cluster are managed in a profile file.

A sample profile is included with a new installation of Anaconda for cluster management and is located in the ~/.acluster/profiles.d/ directory on the client machine. You can edit the contents of this file to reflect the settings for your cluster.

An example profile located at ~/.acluster/profiles.d/profile_name.yaml and named profile_name that is configured to use the aws_east provider is shown below:

name: profile_name
provider: aws_east
num_nodes: 4
node_id: ami-d05e75b8  # Ubuntu 14.04, us-east-1 region
node_type: m3.large
user: ubuntu

Anaconda for cluster management supports and is tested with the following Linux-based AMIs on Amazon EC2 (us-east-1 region), which can be set as the node_id:

OS	AMI	User
Ubuntu 12.04	ami-08faa660	ubuntu
Ubuntu 14.04	ami-d05e75b8	ubuntu
CentOS 6.6	ami-d89fb7b0	root

For more information on AMIs that are available in other Amazon EC2 regions, refer to the Amazon EC2 documentation or the Ubuntu Amazon EC2 AMI Locator.

Refer to the Profile settings page for more details about profile settings.

3. Create Cluster

Use the following command to launch a cluster with the specified profile:

$ acluster create demo_cluster --profile profile_name

Note: Replace demo_cluster with the name of your cluster, and profile_name with the name of your profile.

After this command is executed, the cloud-based cluster will be created, and you will see updates as the tasks and initialization steps are completed.

A typical launch takes between 5 and 10 minutes, and has output similar to the following:

$ acluster create demo_cluster --profile profile_name
Creating cluster
Number of existing nodes: 0
Number of requested nodes: 4
Licensed nodes: 16
License is valid for the current number of nodes.
INFO: Creating new cluster "demo_cluster" with profile "profile_name"
INFO: Creating 4 instances
INFO: Instances configuration:
INFO:   Name: demo_cluster
INFO:   Security Group: ['all-open']
INFO:   Number of Nodes: 4
INFO:   Type: <NodeSize: id=m3.large, name=Large Instance, ram=7168
        disk=32000 bandwidth=None price=0.14 driver=Amazon EC2 ...>
INFO:   Location: us-east-1
INFO:   Additional Tags {'billingProject': 'anaconda-cluster'}
INFO: Successfully created instances
INFO: Cluster info: {'ips': ['54.81.228.35', '54.167.198.242',
      '54.145.107.208', '54.166.207.40'], 'user': 'ubuntu',
      'ids': ['i-29818d89', 'i-2c818d8c', 'i-2e818d8e', 'i-2f818d8f'],
      'name': u'demo_cluster'}
Saving cluster file
Cluster "demo_cluster": 4 nodes
Number of existing nodes: 4
Number of requested nodes: 0
Licensed nodes: 16
License is valid for the current number of nodes.
Checking ssh connection
INFO: Checking SSH connection
Checking sudo
Bootstraping conda
INFO: Installing miniconda
INFO: Anaconda URL: https://repo.anaconda.com/miniconda/Miniconda-latest-Linux-x86_64.sh
INFO: Checking conda installation
INFO: Checking conda installation
Bootstraping salt
INFO: Setting up Salt master and Minions
INFO: Creating conda env for salt
INFO: Installing salt
INFO: Settings grains/roles
INFO: Head roles: []
INFO: Compute roles: []
Starting salt
INFO: Flushing iptables rules
INFO: Setting SELinux to permissive
INFO: Starting salt-master daemon
INFO: Starting salt-minion daemon
Checking salt connection
Uploading formulas
INFO: Uploading formulas to head
INFO: Uploading profile
Syncing formulas
Done

The cluster is now ready for use.

View the Troubleshooting FAQ if you encounter errors while creating a cluster.

4. Destroy Cluster

If you want to destroy a cluster, use the following command:

$ acluster destroy demo_cluster

Note: Replace the name demo_cluster above with the actual name of your cluster.

After issuing the above command, you will be prompted to confirm termination of the specified cluster.