Icinga2

I have just finished working out how to set up a distributed icinga2 environment with 2 zones. Zone1 is on the public internet, and Zone2 is behind a firewall, so Zone2 can connect to Zone1, but there is no connectivity the other way around. The setup steps for me were:

The things I would like to note are:

• I did not set up a HA cluster for any zone, so I do not have instructions on setting this up
• You only need to set up a single CA, which will need to be accessible on the API port (5665 by default) from all slave zones and endpoints that will get an agent
• You need to manually define the local and parent zone on each node

• Install on the central server.
  • Add the icinga2 config to the package manager (see below for different distro options)
  • Install the check plugins (see below for distro options)
  • Edit /etc/icinga2/icinga2.conf and include the api-users.conf file:
    
//include_recursive "conf.d"
include "conf.d/api-users.conf"

  • Edit /etc/icinga2/constants.conf and put a random string into the TicketSalt constant (use the shell command below to generate a random 32 character string)
    
< /dev/urandom tr -dc _A-Z-a-z-0-9 | head -c${1:-32};echo;

  • Using the command-line, run "icinga2 node wizard" on the master node and select "no" for the satellite setup, then run through the wizard.
  • Create a "director-global" zone manually in /etc/icinga2/zones.conf as follows
    
object Zone "director-global" {
global = true
}

  • Use the command-line "icinga2 feature list" and "icinga2 feature enable", make sure the features "api checker command compatlog ido-mysql mainlog notification statusdata" are enabled on the master
• Install icingaweb2 (apt-get install icingaweb2)
• Install the icinga director, plus a couple of patches
  • mkdir -p /usr/local/share/icingaweb2
  • cd /usr/local/share/icingaweb2
  • git clone https://github.com/eskyuu/icingaweb2-module-director director
  • ln -s /usr/local/share/icingaweb2/director/ /usr/share/icingaweb2/modules/director
• Run through the icingaweb2 setup:
  • On the command line run:
    • icingacli setup config directory --group icingaweb2
    • icingacli setup token create
    • ICINGAWEB2PASS=`< /dev/urandom tr -dc _A-Z-a-z-0-9 | head -c${1:-32};`
    • mysql -u root -p -e "create database icingaweb2 ; grant all on icingaweb2.* to 'icingaweb2'@'localhost' identified by '$ICINGAWEB2PASS'"
    • DIRPASS=`< /dev/urandom tr -dc _A-Z-a-z-0-9 | head -c${1:-32};`
    • mysql -u root -p -e "create database icinga2_director DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; grant all on icinga2_director.* to 'icinga2_director'@'localhost' identified by '$DIRPASS'"
    • echo "icinga2_director $DIRPASS"
  • Visit http://_servername_/icingaweb2/setup
    • Enter the security token
    • Fix and PHP config settings
    • Enter the database details for the icingaweb2 user that you created above using the database icingaweb2 when prompted
    • Look in /etc/icinga2/features-available/ido-mysql.conf for the IDO config when prompted
    • Log into the web interface
    • Go to Configuration->Application->Resources->Create a New Resource
    • Create a SQL Database resource with the following settings:
      • Resource Name: director_db
      • Database Name: icinga2_director
      • Username: icinga2_director
      • Password: _Copied from command output_
    • Go to Configuration->Modules->director->Configuration
    • Select the director_db resource
    • Enter the endpoint name (configured during the icinga2 node wizard step - usually the machines FQDN, which can be found by running "hostname --fqdn")
    • Get the API user and password from /etc/icinga2/conf.d/api-users.conf
• Run through the director setup
• Add a child icinga zone
  • Install icinga on the server that will run the checks
  • On the command line of the new server, run "icinga2 node wizard", and run through the wizard. When prompted, select:
    • "Yes" for the satellite setup
    • The parent server's hostname for the parent server's CN
    • "Yes" to connect to the master from this node
    • The master server's hostname or IP address for the master endpoint host
    • The first master server's hostname or IP address for the CSR auto-signing host
    • Yes to both accept config and accept commands
  • Reload icinga2 to apply the updates
  • In Director, add a new zone with the same zone name as the new child node, "no" for global zone and the correct parent as the parent zone
  • In Director, add a new endpoint template with the port set to 5665, and the API user selected
  • In Director, add a new endpoint to the new zone with the same hostname as the new server, and the endpoint address set to the IP or hostname of the endpoint
  • Deploy the director config
• Adding a host to run local commands (icinga2 agent)
  • In Director, add a new zone for all the agents to belong to if needed (e.g. zone2-agents) with the parent zone set to the local icinga2 zone (e.g. zone2) and "no" for global zone
  • In Director, add a new endpoint to the correct "agents" zone, making sure the endpoint name matches the hostname for the host object
  • In Director, create a new host object in the parent zone (e.g. zone2), and set the appropriate variables to have the correct services applied
  • Deploy the director config
  • Install icinga2 he new host that is being monitored along with ant plugins that are required to run locally
  • Run "icinga2 node wizard" and select
    • "Yes" for the satellite setup
    • The parent server's hostname for the parent server's CN
    • "Yes" to connect to the master from this node
    • The parent server's hostname or IP address for the master endpoint host
    • The first master server's hostname or IP address for the CSR auto-signing host
    • Yes to both accept config and accept commands
    • Create a "director-global" zone manually in /etc/icinga2/zones.conf as follows
      
object Zone "director-global" {
global = true
}

  • Reload icinga2 to apply the updates

Windows Client:

• Download and install icinga 2 MSI
• Run setup wizard
• Make sure Instance Name matches the hostname and endpoint name in the icing2 config (including capitalisation)
• On the master instance, run "icinga2 pki ticket --cn _instance_name_" to generate a ticket, and copy to the setup ticket field
• Add the master instance name (must match the common name used when setting up the PKI), tick "Connect to this endpoint", and type in the hostname/IP address to connect to
• Tick "Accept commands" and "Accept config updates" from master.
• Edit c:\programdata\icinga2\etc\icinga2\icinga2.conf and comment out "include_recursive conf.d"
• Edit c:\programdata\icinga2\etc\icinga2\zones.conf and add the following:
  
object Zone "director-global" {
global = true
}

• Re-start the icinga2 service

Distribution setup

Debian:
Add the following to /etc/apt/sources.list.d/icinga.list
deb http://packages.icinga.org/debian icinga-jessie main

apt-get install icinga2 monitoring-plugins-basic monitoring-plugins-common monitoring-plugins-standard

Edit /etc/icinga2/icinga2.conf and comment out (// at the beginning of the line):
include_recursive "conf.d"

Run:
systemctl enable icinga2.service
/etc/init.d/icinga2 restart

CentOS/RHEL:
yum install https://packages.icinga.org/epel/6/release/noarch/icinga-rpm-release-6-…
rpm -Uvh http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.no…

yum install icinga2 nagios-plugins-all
(or run "yum search nagios-plugins" and install only the required plugins)