Customizing a cluster profile

Although the default cluster profile described in the installation guide is appropriate for most environments and use cases, you can customize profile settings as needed.

Cluster profiles are used to define information about a cluster before it is created, including the cluster name, nodes in the cluster, and other platform component options. Cluster files are located in the ~/.continuum/adam/profile.d/ directory.

Cluster definitions are used to identify information about a running cluster, including the cluster name, nodes in the cluster, and other platform component options. Cluster definitions are located in the ~/.continuum/adam/cluster.d/ directory.

Sample profile showing all settings

A cluster profile located at ~/.continuum/adam/profile.d/cluster.yaml with all configurable settings is shown below:

name: cluster
provider: bare

bare:
  username: centos
  # password: anaconda  # Optional
  port: 22
  keypair: ~/.ssh/my-private-key
  nodes:
  - host: node1.company.com
  - host: node2.company.com
  - host: node3.company.com
  - host: node4.company.com

security:
  flush_iptables: false
  selinux_context: false
  selinux_permissive: false

network:
  http_proxy: http://server:port/
  # https_proxy: http://server:port/

system:
  tmp_dir: /tmp

plugins:
  conda:
    anaconda_hash: md5=d72add23bc937ccdfc7de4f47deff843
    anaconda_url: http://repo.anaconda.com/archive/Anaconda2-4.4.0-Linux-x86_64.sh
    channel_alias: https://conda.anaconda.org/
    channels:
    - defaults
    - anaconda-adam
    conda_canary: false
    enabled: true
    miniconda_hash: md5=7097150146dd3b83c805223663ebffcc
    miniconda_url: http://repo.anaconda.com/miniconda/Miniconda2-4.3.21-Linux-x86_64.sh
    rootdir: /opt/continuum
    ssl_verify: False

  dask:
    bokeh_port: 8787
    bokeh_whitelist: '''*'''
    enabled: false
    host: 0.0.0.0
    http_port: 9786
    nprocs: 1
    port: 8786

  enterprise-notebooks:
    admin_email: admin@yourdomain.com
    admin_password: anaconda
    admin_user: wakari
    directory: /opt/wakari
    elasticsearch_fn: elasticsearch-1.7.2.noarch.rpm
    enabled: false
    enterprise_notebooks_version: 4.0.0
    gateway_port: 8089
    java_fn: jre-8u65-linux-x64.rpm
    mongodb_fn: mongodb-org-2.6.8-1.x86_64.rpm
    mongodb_mongos_fn: mongodb-org-mongos-2.6.8-1.x86_64.rpm
    mongodb_server_fn: mongodb-org-server-2.6.8-1.x86_64.rpm
    mongodb_shell_fn: mongodb-org-shell-2.6.8-1.x86_64.rpm
    mongodb_tools_fn: mongodb-org-tools-2.6.8-1.x86_64.rpm
    mongodb_version: 2.6.8
    nginx_fn: nginx-1.6.2-1.el6.ngx.x86_64.rpm
    root_download_url: https://820451f3d8380952ce65-4cc6343b423784e82fd202bb87cf87cf.ssl.cf1.rackcdn.com/
    user: wakari

  jupyter:
    dashboards_server:
      enabled: false
      ip: 0.0.0.0
      port: 3000
      user: anaconda
    enabled: false
    kernel_gateway:
      enabled: false
      ip: 0.0.0.0
      port: 7000
      user: anaconda
    notebook:
      directory: ~/notebooks
      enabled: false
      ip: 0.0.0.0
      password: 'anaconda'
      port: 8888
      user: anaconda

  repository:
    binstar_server_version: 2.21.0
    channel: main
    email: youremail@anaconda.com
    enabled: false
    mongodb_fn: mongodb-org-2.6.8-1.x86_64.rpm
    mongodb_mongos_fn: mongodb-org-mongos-2.6.8-1.x86_64.rpm
    mongodb_server_fn: mongodb-org-server-2.6.8-1.x86_64.rpm
    mongodb_shell_fn: mongodb-org-shell-2.6.8-1.x86_64.rpm
    mongodb_tools_fn: mongodb-org-tools-2.6.8-1.x86_64.rpm
    mongodb_version: 2.6.9
    password: anaconda
    port: 8080
    root_download_url: https://820451f3d8380952ce65-4cc6343b423784e82fd202bb87cf87cf.ssl.cf1.rackcdn.com/
    server_user: anaconda-server
    superuser: superuser
    token: qu-a49e2e69-1047-4eab-a879-a2ee9c198381

  salt_settings:
    acl:
    - anaconda
    job_pub_port: 14505
    minion_pub_port: 14510
    minion_pull_port: 14511
    minion_ret_port: 14506
    rest_port: 18000
    salt_groupname: anaconda
    salt_password: anaconda
    salt_username: anaconda
    service_scripts: false

Cluster name

The name setting specifies the name of the cluster and is used by the -n option in Anaconda Adam commands.

EXAMPLE: To create a new cluster named “cluster” from a profile named “profile”:

$ adam up -n cluster profile

SSH authentication

To connect to remote machines via SSH, Adam requires a valid username, port–default is 22–and an authentication method. Adam supports the following SSH authentication methods:

  • Password.
  • Key pair–full path to your private key.
  • Encrypted key pair–full path to your public key and ssh-agent. The private key must first be added to a running ssh-agent. Use the agent_pubkey setting to specify the full path to the associated public key.

For more information about system and account requirements, see Account access and security.

EXAMPLE: To use SSH with a username and password:

bare:
  username: centos
  password: anaconda

EXAMPLE: To use SSH with a key pair:

bare:
  username: centos
  keypair: /full/path/to/key.rsa

EXAMPLE: To use SSH with an encrypted key pair:

bare:
  username: centos
  agent_pubkey: /full/path/to/key.pub

Node specification

The nodes setting defines the nodes that exist within the cluster. By default, the first node in the list of nodes is defined as the head node and the remaining nodes are defined as compute nodes.

Each host must be able to connect to the other hosts via the specified FQDN or IP address.

TIP: We recommended using the node’s FQDN that matches the hostname on each machine:

EXAMPLE:

bare:
  nodes:
  - host: node1.company.com
  - host: node2.company.com
  - host: node3.company.com
  - host: node4.company.com

Security settings

The security settings let you configure IPTables and SELinux.

To flush iptables, reset flush_iptables to true. Default value: false.

security:
  flush_iptables: false

To set SELinux to permissive, reset selinux_permissive to true. Default value: false.

security:
  selinux_permissive: false

To set SELinux contexts, reset selinux_context to true. Default value: false.

security:
  selinux_context: false

Network settings–proxy configuration

The network settings let you install Adam to nodes that access the internet via an HTTP or HTTPS proxy.

To specify an HTTP proxy:

network:
  http_proxy: http://server:port/

NOTE: Replace server:port with the actual server and port.

To specify an HTTPS proxy:

network:
  https_proxy: http://server:port/

NOTE: Replace server:port with the actual server and port.

Conda settings

These are the default settings for configuring conda:

conda:
    anaconda_hash: md5=d72add23bc937ccdfc7de4f47deff843
    anaconda_url: http://repo.anaconda.com/archive/Anaconda2-4.4.0-Linux-x86_64.sh
    channel_alias: https://conda.anaconda.org/
    channels:
    - defaults
    - anaconda-adam
    enabled: true
    miniconda_hash: md5=7097150146dd3b83c805223663ebffcc
    miniconda_url: http://repo.anaconda.com/miniconda/Miniconda2-4.3.21-Linux-x86_64.sh
    rootdir: /opt/continuum
    ssl_verify: False

You can set ssl_verify to:

  • False–no SSL verification–default.
  • True–SSL verification is used and conda verifies certificates for SSL connections.
  • [cert path]–the string path to a certificate to be used to verify SSL connections.

Salt settings

Salt is the configuration management system used by Adam. The salt_settings let you configure the network options, access control list and API user credentials used by Salt.

Use the settings below to specify the ports used by the Salt master, minions and REST API:

plugins:
  salt_settings:
    job_pub_port: 14505
    minion_pub_port: 14510
    minion_pull_port: 14511
    minion_ret_port: 14506
    rest_port: 18000

Use the acl setting to specify an access control list that defines non-root system users who can execute Salt commands:

plugins:
  salt_settings:
    acl:
    - anaconda

Use the settings below to specify the username, password and group that gets created across the cluster and owns the directory and files located in the root installation directory–default: /opt/continuum:

plugins:
  salt_settings:
    salt_username: anaconda
    salt_password: anaconda
    salt_groupname: anaconda

If you do not want to store a clear text password in the cluster profile or cluster definition file, set salt_password to an empty string:

plugins:
  salt_settings:
    salt_username: anaconda
    salt_password: ''
    salt_groupname: anaconda

In this case, Adam prompts for a password in the CLI and only stores the password in memory for the duration of the command or job.

System settings

The tmp_dir setting lets you specify the download directory where temporary installers are downloaded–default: /tmp:

system:
  tmp_dir: /tmp