Cloud-based Cluster Setup¶
Anaconda for cluster management can launch and bootstrap clusters on a variety of cloud services. We currently support Amazon EC2. Other providers such as Microsoft Azure, Rackspace, Google Cloud Platform, and others are on our roadmap. If you are interested in using a cloud provider that is not listed here, please contact us at sales@anaconda.com.
Cloud configuration involves the use of profiles
and providers
.
Splitting configurations allows users to easily share and distribute specific
cluster configurations (profiles
), while retaining private authentication
credentials (providers
).
1. Provider Setup¶
Authentication and cloud definitions are managed in a single provider file on
the client machine: ~/.acluster/providers.yaml
. A provider file can contain
multiple providers with different settings and credentials.
A sample providers file is included with a new installation of Anaconda for
cluster management and is located within the ~/.acluster/providers.yaml
file on the client machine. You can edit the contents of this file to reflect
the settings and credentials for your cloud provider.
An example ~/.acluster/providers.yaml
file with a provider named aws_east
that is configured for Amazon EC2 is shown below:
aws_east:
cloud_provider: ec2
keyname: my-private-key
location: us-east-1
private_key: ~/.ssh/my-private-key.pem
secret_id: AKIAXXXXXX
secret_key: XXXXXXXXXX
Note that you will need access to ports 22, 4505, and 4506 from the client machine to the cluster nodes to provision the cluster via SSH and Salt.
Refer to the Provider settings page for more details about provider settings, including security groups.
Linux or Mac: You should set the permissions of providers.yaml
file to 0600
.
2. Profile Setup¶
The settings for each cluster are managed in a profile file.
A sample profile is included with a new installation of Anaconda for cluster
management and is located in the ~/.acluster/profiles.d/
directory on the
client machine. You can edit the contents of this file to reflect the settings
for your cluster.
An example profile located at ~/.acluster/profiles.d/profile_name.yaml
and named profile_name
that is configured to use the aws_east
provider
is shown below:
name: profile_name
provider: aws_east
num_nodes: 4
node_id: ami-d05e75b8 # Ubuntu 14.04, us-east-1 region
node_type: m3.large
user: ubuntu
Anaconda for cluster management supports and is tested with the following
Linux-based AMIs on Amazon EC2 (us-east-1 region), which can be set as the
node_id
:
OS | AMI | User |
---|---|---|
Ubuntu 12.04 | ami-08faa660 | ubuntu |
Ubuntu 14.04 | ami-d05e75b8 | ubuntu |
CentOS 6.6 | ami-d89fb7b0 | root |
For more information on AMIs that are available in other Amazon EC2 regions, refer to the Amazon EC2 documentation or the Ubuntu Amazon EC2 AMI Locator.
Refer to the Profile settings page for more details about profile settings.
3. Create Cluster¶
Use the following command to launch a cluster with the specified profile:
$ acluster create demo_cluster --profile profile_name
Note: Replace demo_cluster
with the name of your cluster, and
profile_name
with the name of your profile.
After this command is executed, the cloud-based cluster will be created, and you will see updates as the tasks and initialization steps are completed.
A typical launch takes between 5 and 10 minutes, and has output similar to the following:
$ acluster create demo_cluster --profile profile_name
Creating cluster
Number of existing nodes: 0
Number of requested nodes: 4
Licensed nodes: 16
License is valid for the current number of nodes.
INFO: Creating new cluster "demo_cluster" with profile "profile_name"
INFO: Creating 4 instances
INFO: Instances configuration:
INFO: Name: demo_cluster
INFO: Security Group: ['all-open']
INFO: Number of Nodes: 4
INFO: Type: <NodeSize: id=m3.large, name=Large Instance, ram=7168
disk=32000 bandwidth=None price=0.14 driver=Amazon EC2 ...>
INFO: Location: us-east-1
INFO: Additional Tags {'billingProject': 'anaconda-cluster'}
INFO: Successfully created instances
INFO: Cluster info: {'ips': ['54.81.228.35', '54.167.198.242',
'54.145.107.208', '54.166.207.40'], 'user': 'ubuntu',
'ids': ['i-29818d89', 'i-2c818d8c', 'i-2e818d8e', 'i-2f818d8f'],
'name': u'demo_cluster'}
Saving cluster file
Cluster "demo_cluster": 4 nodes
Number of existing nodes: 4
Number of requested nodes: 0
Licensed nodes: 16
License is valid for the current number of nodes.
Checking ssh connection
INFO: Checking SSH connection
Checking sudo
Bootstraping conda
INFO: Installing miniconda
INFO: Anaconda URL: https://repo.anaconda.com/miniconda/Miniconda-latest-Linux-x86_64.sh
INFO: Checking conda installation
INFO: Checking conda installation
Bootstraping salt
INFO: Setting up Salt master and Minions
INFO: Creating conda env for salt
INFO: Installing salt
INFO: Settings grains/roles
INFO: Head roles: []
INFO: Compute roles: []
Starting salt
INFO: Flushing iptables rules
INFO: Setting SELinux to permissive
INFO: Starting salt-master daemon
INFO: Starting salt-minion daemon
Checking salt connection
Uploading formulas
INFO: Uploading formulas to head
INFO: Uploading profile
Syncing formulas
Done
The cluster is now ready for use.
View the Troubleshooting FAQ if you encounter errors while creating a cluster.
4. Destroy Cluster¶
If you want to destroy a cluster, use the following command:
$ acluster destroy demo_cluster
Note: Replace the name demo_cluster
above with the actual name of your
cluster.
After issuing the above command, you will be prompted to confirm termination of the specified cluster.