EYE v5 Installation Guide, April 2010 (c) Glacier Consulting. This guide provides installation instructions for the EYE collector and utilites. ---------------------- Utilities Installation ---------------------- Minimum Requirements: --------------------- - Operating System: AIX v5.3 TL5 or higher (packages for systems on lower levels can be made available on demand). Linux (2.6 recommended). Packages for the following systems can be provided on special request: Sun Solaris (5.8/5.9/5.10). HP-UX. - Disk Space: Recommended: 250-500MB in /var/eye. At least 64MB of free space in /tmp (recommended: 96-128MB). - Configuration: The iconfig utility is provided as a menu-driven configuration tool. - Software: IDIST requires SSH (ssh/scp) to perform data collection (recommended: OpenSSH 5.1 and higher). The utilities is used on a central monitoring (health-check) server and consists of: IHEALTH - Performs system checks based on collected data. IDIST - Performs EYE tasks on groups of servers, such as data collection. IREAD - Displays the content and status of collected data. IGREP - Scans collected data for patterns, similar to the UNIX grep utility. IDIFF - Functions similar to the UNIX diff utility on collected data. ICONFIG - Text-based menu driven configuration utility for EYE. EYE Utilities Installation steps: --------------------------------- If you are upgrading from a previous release: 1. Create a backup of existing data, in case any unforeseen problem is encountered. If you are upgrading a system that is currently running EYE v5 or later: Make a backup of eye.db, which can normally be found in /usr/local/eye/db (this contains all host (nethost) and group definitions). Make a backup of all .rc files. This includes ihealth.rc and any ihealth-HOSTNAME.rc files that contains host-specific settings. If you are upgrading from EYE v4 or an older version: Save idist.rc and any custom ihealth.rc and ihealth-HOSTXXX.rc files you may have. A migration utility can be used to convert the idist.rc based host entries to the new eye.db SQL database format. 2. Software Installation Perform the steps in 2.1. if you are upgrading to a new major release of EYE, else move on to step 2.2. 2.1 Steps to upgrade to a new major release of EYE (for example, from EYE v4.3 to EYE v5.1) Note: the following steps are NOT for use during minor updates (i.e. upgrading from v5.1 to v5.2). 2.1.1. Uninstall the old version of the EYE utilities (eye.utils.rte). On AIX: smitty install -> Software Maintenance -> Remove Installed Software Select eye.utils.rte under Software Name. On Linux: Remove all files & sub-directories in the program installation directory, except files in the database directory. 2.1.2. Prepare the program installation directory. 1. Remove any remnant configuration files that may still be left in /usr/local/eye (program installation directory). 2. If you prefer to use a separate location for the EYE databases: 1. Create a separate file-system for the EYE database. 2. Either link the ./db sub-directory to this new location or use iconfig to configure the database directory location. For more information, refer to the ICONFIG.readme document. 2.1.3. Install the new version of the EYE utilities: On AIX: 1. Copy eye.utils-x.x.x-aix53.bff to /tmp/eye on the server that will perform the monitoring and analysis. 2. Run: inutoc /tmp/eye 3. Run: smit install_all and select and install 'eye.utils.rte'. Press Enter to install the software. On Linux: 1. Copy the eye.utils-x.x.x-linux.tgz file to /tmp/eye on the monitoring server. 2. Create a directory where the utilities should operate from, i.e.: mkdir /support/eye cd /support/eye 3. Run: tar xzvf /tmp/eye/eye.utils-x.x.x-linux.tgz 2.2. Steps to perform a minor update of EYE (for example, to upgrade from EYE v5.1 to EYE v5.2) On AIX: 1. Copy eye.utils-x.x.x-aix53.bff to /tmp/eye on the server that will perform the monitoring and analysis. 2. Run: inutoc /tmp/eye 3. Run: smit update_all and select and select 'eye.utils.rte'. Press Enter to update the existing software. 4. At this point, the software upgrade is complete. 5. Perform configuration steps only if not already done, or if there is a specific new feature that you wish to make use of. On Linux: 1. Copy the eye.utils-x.x.x-linux.tgz file to /tmp/eye on the monitoring server. 2. Change directory to the location where the utilities should operate from, i.e.: mkdir /support/eye cd /support/eye 3. Run: tar xzvf /tmp/eye/eye.utils-x.x.x-linux.tgz 4. At this point, the software upgrade is complete. 5. Perform configuration steps only if not already done, or if there is a specific new feature that you wish to make use of. 3. Configure the EYE utilities by running: ./iconfig See README.iconfig for detailed information on how to configure the EYE tools. At a minimum, the following settings should be configured in a new setup: - Global settings - set the HTML report and database directory locations. - Defining some NetHosts (hosts that should be checked). NOTE: If you are performing an upgrade from EYE v4 or older, you should select Maintenance -> Import NetHosts if you require your old hosts to be automatically imported into the EYE v5 database. 4. Configure clients for data collection. Configure trusted SSH keys from this monitoring server to each client machine (see "SSH trusted key setup"). NOTE: Ensure that an SSH login can automatically be established to each client machine before proceeding. The SSH login connection should be able to automatically reach the actual login prompt of the target server. Operations against the client machine will NOT work unless this is the case. ---------------------- Collector Installation ---------------------- -------------------------------------------------------------------- WARNING: USAGE NOTE FOR THE GLACIER ICOL DATA COLLECTOR -------------------------------------------------------------------- Never manually copy an EYE executable between different releases of AIX, for example from a system running on AIX v5.3 to AIX v5.2. A known AIX defect (IY79272 on AIX 5.2, IY84261 on AIX 5.3) may cause a system crash should this guideline not be adhered to. If the AIX SMIT utility is used to install the EYE utilities, this will not be an issue as it is not possible to install the incorrect version of the utilities on any system due to the built-in package pre-requisite checking. As with any software, it is highly recommended that you test each release of the EYE utilities and collector software before attempting wide-scale deployment. EYE Collector ------------- The icol collector gathers a snapshot of critical client system data which can be analysed by the EYE utilities, or manually inspected at a remote location. The collector is normally installed (as root) on each client server that should be analysed. On environments with restricted access such as pSeries HMC systems, it is still possible to collect information without the collector installed. This is referred to as agentless operation mode. Installation of the Collector on AIX: ------------------------------------- Simultaneous Multi-client Installation (install collector software in parallel on multiple client machines) 1. Install the EYE utilities, as per the above instructions. 2. Ensure SSH trusted key authentication is set up and functional between the server that contains the EYE utilities and each client (for more information, refer to "SSH trusted key setup"). Note: Do not continue unless you have manually tested an SSH connection to each client machine without a password. (The SSH software needs to have added the SSH key fingerprint to known_hosts, and a manual connection to the client machine should no longer display a messages like "Authenticity of host X not established. Continue (y/n)?"). 3. Use the iconfig utility to create a NetHost for each client machine that requires the collector: 1. Run: ./iconfig 2. Select NetHosts -> 3. Add NetHost 3. Enter a unique name for the NetHost. The name must be reachable via SSH. 4. Enter a primary group for the NetHost. This is required to group similar types of client machines (i.e. use group 'prod' for production client machines). 4. If the client machine is an HMC, or an environment that should not have the collector installed: 1. Select and set Communication Method to 'ssh'. 2. Select and set Connection Arguments to 'hscroot@hmchostname'. NOTE: Agentless data collection is not recommended for servers other than pSeries HMCs. 5. Press C to continue and save the new configuration. Repeat step 3 as required. 6. It should now be possible to use the IDIST utility to install the collector tool simultaneously on all configured client machines. Continue with "Manual Collector Installation" should you prefer to not use IDIST. IDIST based installation examples: To install the collector on each client that belongs to the group 'prod': ./idist.exe install -l ./eye.collector-4.x.x.x-aix53 prod To install the collector on all client machines: ./idist.exe install -l ./eye.collector-4.x.x.x-aix53 all To install the collector on a user-specified list of servers: ./idist.exe install -l ./eye.collector-4.x.x.x-aix53 -w server1,server2,serverN Note: It is not necessary to perform separate installations for each version of AIX, as long as all the relevant BFF installation files are present in the directory that was specified by the -l option. The IDIST utility will automatically determine and adjust the name of the BFF installation file to use on each target client machine, depending on the operating system level. 7. The collector should now have been installed to /var/eye on each client system, and data can now be collected and analysed. If any problems are encountered, see the -v and -D options as described in README.idist. Manual Collector Installation (AIX): 1. Upload the eye.collector-xxx-bff installation file to /tmp/eye on the target server. 2. Run "inutoc /tmp/eye". 3. Run smitty install_all, and install the eye.collector.rte fileset. 4. Configure /usr/local/eye/icol.rc (using icol.rc.sample). Manual Collector Installation (Other UNIX systems): 1. Upload eye.collector-4.x.x.x-platform.tgz to /tmp/eye on the target server. 2. Run "mkdir -p /usr/local/eye && cd /usr/local/eye" 3. Run "tar zxvf /tmp/eye/eye.collector-4.x.x.x-platform.tgz". 4. Configure /usr/local/eye/icol.rc (using icol.rc.sample). Agentless Collector Installation: In this mode of operation, the collector runs on the monitoring server itself, without the collector being installed on the remote client machine. This is only necessary for the monitoring of pSeries HMC systems which operate using a restricted shell and do not readily provide access to the 'root' user account. 1. Install the collector utility on the monitoring server itself as per the steps in "Manual Collector Installation". 2. Ensure that this server is SSH trusted-key enabled, and that passwordless logins can be made to each client. 3. Change the communication configuration of each agentless NetHost, as per step 4 of the Simultaneous Multi-client Installation instructions. 4. This steps will enable the IDIST utility to collect information from these NetHosts, normally as the 'hscroot' user. SSH Trusted Key Setup: 1. If the client machine is an HMC: Enable SSH access to the HMC using WebSM or the web-browser interface, as described in the IBM pSeries HMC documentation. 2. Copy the existing authorized_keys2 file from the client machine: For an HMC: scp hscroot@hmchostname:~/.ssh/authorized_keys2 /tmp For an AIX/Linux server: scp hscroot@hmchostname:~/.ssh/authorized_keys /tmp 3. Generate a SSH keypair for your local user unless it already exists in ~/.ssh/id_dsa.pub. ssh-keygen -t dsa 4. Add your public key to the authorized_keys file: On HMC systems: cat ~/.ssh/id_dsa.pub >> /tmp/authorized_keys2 On AIX/Linux systems: cat ~/.ssh/id_dsa.pub >> /tmp/authorized_keys 5. Copy the modified file back to the client machine. scp /tmp/authorized_keys2 hscroot@hmchostname:~/.ssh/authorized_keys2 6. Delete the local /tmp/authorized_keys file. rm /tmp/authorized_keys 7. Test that you are able to perform an SSH login without a password: On HMC systems: ssh hscroot@hmcname On AIX/Linux systems: ssh root@host 8. If step 7 fails, you should check if 'PermitRootLogin' is set to Yes. Usage instructions can be found in README.idist and README.icol. Non-root Installation --------------------- As of v4.15 it is possible to operate the collector without requiring root privileges. Note that it will still be necessary to install the EYE collector BFF package as the root user. Instructions: Perform instructions 1-4 on the remote host. Perform instructions 5 on the server that is used to perform the health-checks. 1. Follow the above "Installation of the Collector on AIX" instructions to install the collector on the desired host. After completion, the collector software will be installed in /usr/local/eye, and a collector file output directory would have been created in /var/eye. 2. Create the user account to use for data collection. This can be done with the AIX command 'smitty mkuser'. For the purpose of this installation guide the user will be considered to be 'eye', with a primary group of 'health'. 3. Change ownership of /usr/local/eye to the regular user. Run: chgrp health /usr/local/eye Run: chmod 750 /usr/local/eye The permissions of the directory should now allow access to the 'eye' user, e.g. drwxr-x--- 6 root health 1536 Apr 03 15:23 /usr/local/eye Notes: * Never change the permissions of /usr/local/eye to allow access to all users, to prevent unwanted information disclosure. 4. Change ownership of the collector output directory (/var/eye) to the regular user. Run: chown eye:health /var/eye Run: chmod 750 /var/eye The permissions of the directory should only allow access to the 'eye' user, e.g. drwxr-x--- 6 eye health 1536 Apr 03 15:23 /var/eye Notes: * Never change the permissions of /var/eye to allow access to any other user except 'eye', to prevent unwanted information disclosure. 5. Update the server connection information (if IDIST is used for data collection). 1. Follow the steps in "SSH Trusted Key Setup" so that you can connect as the desired user to the remote machine. At this point, entering 'ssh eye@host' should allow direct SSH login to the remote machine. 2. Change the connection type on the Central Monitoring (health-check) Server. run: ./iconfig 1. Pick NetHost -> Edit NetHost (enter name of host you configured in step 1) 2. Select option 2 (Communication - SSH connection string), and change it to: eye@hostname Notes: 1. To enable EYE collector installation via idist at a future time, change the connection string back to 'root@host'.