5. OpenNMS Administration

The OpenNMS server requires Postgres, Sun's J2SDK 1.4, and Apache Tomcat to run. The front-end and back-end operate independently, allowing a back-end (monitoring and database updating) restart without interruption to the front-end (HTML user interface and database querying). Conversely, the front-end can be disabled without stopping the back-end from monitoring. This is particularly useful when performing opennms package upgrades or making configuration changes. The Postgres database must be operational in order for either the front-end user interface or the back-end monitoring to function. For more resources on the inner workings of OpenNMS check out the section near the end of this document, Further Information.

5.1 Server Startup and Shutdown

OpenNMS must be configured at this point. If the OpenNMS server is started without a proper configuration, ERROR and WARN will show up in your log files and things will be a general mess. Read the HOWTOs. Basic (very) configuration must have ip addresses in the appropriate include lines in capsd-configuration.xml, discovery-configuration.xml, collectd-configuration.xml and poller-configuration.xml. Ensure appropriate modifications to the packages in the poller-configuration.xml and collectd-configuration.xml to control polling and data collection from specific groups of devices. Ensure that community strings are entered properly in snmp-config.xml. For quick node name resolution (i.e. no DNS delays to affect processing of events or DNS queries to generate network traffic) modify the /etc/hosts file with appropriate device names. SNMP names will be used if devices are configured for SNMP properly.

The Postgres server must be running prior to starting the back-end (OpenNMS pollers) or the front-end (web interface). The startup order (automatic with system startup) is:

1. bash#/etc/init.d/postgresql start (Postgres)
2. bash#/etc/init.d/tomcat4 start (Web UI)
3. bash#/etc/init.d/opennms start (OpenNMS engine)

Shutdown is the reverse except with the stop directive, rather than start . It is important that Postgres starts first and is shutdown last. Of course this happens automatically with the init scripts on a complete system startup or shutdown. Except for database maintenance, the Postgres service should never require manual starting or stopping, even after configuration changes.

5.2 Enabling Maps

1. Stop the tomcat4 and opennms services

   bash~#/etc/init.d/tomcat4 stop
   bash~#/etc/init.d/opennms stop
   
    
   

2. Enable the feature in opennms

   bash~#cp /etc/opennms/map.disable /etc/opennms/map.enable
   
    
   

3. In the default /etc/default/tomcat4 uncomment the following CATILINA_OPTS line:

   # Arguments to pass to the Java virtual machine (JVM)
   # "-Djava.awt.headless=true" is automatically set if CATALINA_OPTS is empty
   CATALINA_OPTS=""
   
    
   

4. Restart Tomcat4 and OpenNMS

   bash~#/etc/init.d/tomcat4 start
   bash~#/etc/init.d/opennms start
   
    
   

At the time of v1.5 of this document, Windows and Mac users could use the SVG maps if the the Adobe SVG plugin was installed. Windows users can get this plugin from Adobe SVG Downloads. Linux users may have to wait for lirsvg2 to be fully integrated with Moziila/Galeon/Safari, etc. The PNG format works exactly the same (in appearance). SVG is a standard maintained by the w3c.

5.3 Database Maintenance

This section covers very basic operations on the ONMS Postgres database.

How to Perform a database Vacuum

If it is suspected that the database has gained entropy, this is a good idea.

1. Stop OpenNMS and Tomcat (Though it could probably work while ONMS is running).

   bash~#/etc/init.d/tomcat4 stop
   bash~#/etc/init.d/opennms stop
   
    
   

2. From the root user, become the Postgres user.

   bash~#su - postgres
   
    
   

3. Perform a vacuum on the OpenNMS database.

   bash~$/usr/lib/postgresql/bin/vacuumdb -v -f -d opennms 
   
    
   

4. Restart OpenNMS and Tomcat.

   bash~#/etc/init.d/opennms start
   bash~#/etc/init.d/tomcat4 start
   
    
   

How to Purge and Recreate the OpenNMS Database

This process can be used to 'reset' the server's database while preserving the OpenNMS configuration data. A system admin that suspects that a problem or corruption in the database can use this process to clean out the entire Postgres database, and restart it using the current configuration. i.e. Current device nodes will be preserved, however events for those nodes will be purged. The RRD data will be preserved unless manually removed from /usr/share/opennms/share/.

1. Become the Postgres user from the root user.

   bash~#su - postgres
   
    
   

2. Purge the OpenNMS database.

   bash~$dropdb opennms
   
    
   

3. Become the root user again

   bash~$exit
   
    
   

4. Recreate the OpenNMS database (as root user).

   bash~#/usr/share/opennms/bin/install.pl -q /usr/share/opennms/etc/create.sql -l /usr/lib/postgresql/lib/opennms 
   
    
   

5.4 Backing Up and Restoring OpenNMS Configuration

Three backup files are created with this process. In our example they will be 062303_onms_sql.tar.gz, 062303_onms_etc.tar.gz and 062303_onms_rrd.tar.gz. These files should be created at the same time to ensure that the rrd data, the SQL database and the configuration files match. It is necessary to capture the configuration and the databases at the same time in order to start an ONMS server with restored data.

Stop OpenNMS and Tomcat

(Not tested, though this may work while ONMS is running with only the web UI shutdown).

bash~#/etc/init.d/tomcat4 stop
bash~#/etc/init.d/opennms stop

Dump and Backup the OpenNMS SQL database

1. Shutdown Tomcat4

   bash~#/etc/init.d/tomcat4 stop
   
    
   

2. Shutdown OpenNMS

   bash~#/etc/init.d/opennms stop
   
    
   

3. Become the Postgres user

   bash~#su - postgres
   
    
   

4. Perform a vacuum on the OpenNMS database.

   bash~$/usr/lib/postgresql/bin/vacuumdb -v -f -d opennms 
   
    
   

5. Dump all databases (This is OK for most users where OpenNMS is the only Postgres database)

   bash~$pg_dumpall > 062303_onms_sql
   
    
   

6. Tar the dump

   bash~$tar cvfz 062303_onms_sql.tar.gz 062303_onms_sql
   
    
   

7. Optionally move the tarball somewhere. A simple backup script could perform this task.

   bash~$scp 062303_onms_sql.tar.gz backupuser@backupserver:
   
    
   

8. Optionally remove the dump files once they have been transferred

   bash~$rm 062303_onms_sql* 
   
    
   

9. Logout of the postgres user

   bash~$exit
   
    
   

Backup the OpenNMS Configuration files

1. Tar up the configuration.

   bash~#tar cvfzP 062303_onms_etc.tar.gz /etc/opennms/* -R
   
    
   

2. Optionally move it somewhere.

   bash~#scp 062303_onms_etc.tar.gz backupuser@backupserver:
   
    
   

3. Optionally remove the tarball once it has bee transferred.

   bash~#rm 062303_onms_etc.tar.gz
   
    
   

Backup the OpenNMS RRD database files and reports

1. Tar up the configuration.

   bash~#tar cvfzP 062303_onms_rrd.tar.gz /usr/share/opennms/share/* -R
   
    
   

2. Optionally move it somewhere.

   bash~#scp 062303_onms_rrd.tar.gz backupuser@backupserver:
   
    
   

3. Optionally remove the tarball once it has been transferred.

   bash~#rm 062303_onms_rrd.tar.gz
   
    
   

Restore a Previous OpenNMS Backup

1. Install Debian and OpenNMS if they are not already setup on the destination server.
2. From the new OpenNMS server transfer the backup files as root

   bash~#scp backupuser@backupserver:062303* .
   
    
   

3. Become the database superuser from root

   bash~#su - postgres
   
    
   

4. Purge the current (perhaps empty) OpenNMS database.

   bash$dropdb opennms
   
    
   

5. Become the root user again

   bash~$exit
   
    
   

6. Recreate the OpenNMS database (as root user).

   bash~#/usr/share/opennms/bin/install.pl -q /usr/share/opennms/etc/create.sql -l /usr/lib/postgresql/lib/opennms 
   
    
   

7. Become the database superuser from root

   bash~#su - postgres
   
    
   

8. Untar and restore the OpenNMS SQL database

   bash~$tar xvfz /root/062303_onms_sql.tar.gz
   bash~$psql -f 062303_onms_sql opennms
   
    
   

   There will be some ERROR messages regarding structures that already exist, these can be safely ignored.

9. Become the root user again

   bash~$exit
   
    
   

10. Untar the OpenNMS configuration

    bash~#tar xvfzP 062303_onms_etc.tar.gz
    
     
    

    ---

    Note about using tar: If the backups were not created with the 'P' option to preserve their relative location, the files can be untarred appending the equivalent '--directory /' to make them relative to the root directory.

    ---

11. Delete the RRD database files and reports that currently exist. (left over if this ONMS server had other uses)

    bash~#rm /usr/share/opennms/share/* -rf
    
     
    

12. Restore the RRD database and report file backup

    bash~#tar xvfzP 062303_onms_rrd.tar.gz
    
     
    

13. Remove the local backup tarballs (optional)

    bash~#rm 062303*
    
     
    

Restart Tomcat4 and OpenNMS

1. Restart tomcat4

   bash#/etc/init.d/tomcat4 start
   
    
   

2. Restart OpenNMS

   bash#/etc/init.d/opennms start
   
    
   

5.5 OpenNMS Log Files

The ONMS system logs all activity to /var/log/opennms. Messages within the log files contain either FATAL, ERROR, WARN, INFO or DEBUG tags. The size and verbosity of the log files can be controlled by modifying /etc/opennms/log4j.properties. By default each file is set to INFO level (not as verbose as DEBUG) and will grow to 100MB. Each log file will "spawn" four times (i.e. 400MB total). All of these settings can be changed.

Viewing Error Messages

To view error messages that contain the ERROR tag, use the following command:

bash#grep "ERROR" /var/log/opennms/* | more

Replacing ERROR with another tag such as FATAL or WARN will display all log messages with that particular tag. Logs should contain no FATAL messages, minimal ERROR messages (ideally none), and possibly many WARN and INFO messages that should be at least be fully understood.

Related log messages will often share a common 'Pool-Fibre' as shown below. Time and Date can be used to correlate messages between different log files. Always try to determine the root cause, or initial error message as often the final error message is preceded by more meaningful log messages. Often related log messages are separated by other unimportant messages so creative use of grep and | is often required. Remember that there can be many situations that result in the same message. For help on troubleshooting messages please see the Log Troubleshooting section.

A messages example

capsd.log

A device has other private interfaces that are not reachable by ONMS (but automatically detected and added to the interface table)

INFO  [Capsd Rescan Pool-fiber1] FtpPlugin: FtpPlugin: Unable to test host 10.8.9.2, no route available...
WARN  [Capsd Rescan Pool-fiber1] IfCollector: IfCollector: No route to host 10.8.9.2, continuing protocol scans. 
INFO  [Capsd Rescan Pool-fiber1] TcpPlugin: TcpPlugin: Could not connect to host 10.8.9.2, no route to host 

Resetting the Log Files

Log files can grow very quickly making it difficult to find valuable information within. In order to minimize the effort required to troubleshoot problems, it is a good idea to backup the current logs and purge them before performing an OpenNMS restart. DO NOT DELETE /var/log/* as this will disable the Linux server. The procedure below outlines a typical restart scenario where the log files are backed up to the directory /root .

1. Stop OpenNMS

   bash#/etc/init.d/opennms stop
   
    
   

2. Tar up the current log files

   bash#tar cvfz /root/opennms_logs_061703.tar.gz /var/log/opennms/*
   
    
   

3. Reset the OpenNMS log directory

   bash#rm -rf /var/log/opennms/*
   
    
   

4. Start OpenNMS

   bash#/etc/init.d/opennms start