Little known IDS commands: ifxcollect

Abstract

In this series of articles, we are exploring some of the lesser known – but still useful – commands hidden away in your Informix bin directory. This article explores the ifxcollect utility.

Introduced in version 11.70, ifxcollect is a tool designed to gather diagnostic information from IDS to aid Technical Support should the issue need to be escalated. This article looks at the features of the latest version shipped with 14.10.

Content

The general form of the command most often requested by tech support is as follows:

ifxcollect -c performance -s general

This command creates a tar file in $INFORMIXDIR/isa/data with a filename ifxcollect.MMDDYYhhmm.tar. Note: With these default options, it will take over 10 mins to run (the reason for this is below). Inside the tar file, you will find three timestamped directories named run.MMDDYYhhmmss. There are three as this is the default run: 3 collection points 5 mins apart; this is why the default command takes this long to run (these settings can be configured – see below). Each of these directories contains the output from the individual diagnostic commands run, along with a manifest file listing each file and the command run to produce it.

The above command generates 57 files on a Linux machine: 10 OS based stats, with the rest being various onstat outputs. The OS diagnostics include output from netstat, nfsstat, vmstat, mpstat, iostat, swapon, free, ps, top and ipcs – so these cover a lot including network, memory, CPU and disk statistics.

A large number of onstats are also collected. It’s worth noting that with the -c performance -s general parameters on this default command, this does not include an onstat -a, although it does run many of the same commands as onstat -a (and many more of the -g global commands).

As you can probably guess from the example above, there are a few options to this tool that can gather different diagnostics for different support situations. The -c (Category) and -s (Subcategory) are mandatory parameters, and there are several options to these. If you wanted to run more or less collections than the standard three, this can be done with the ‘-r x’ option, and ‘-d x’ sets the delay between collections. You can also zero the profile stats after each collection with ‘-z’, although you would need a very specific reason to do this.

These are the categories and sub options you can run for:

IDS (-c ids)

There is only one possible sub category for IDS: –s general. This gathers only a few stats, mainly on the IDS version and environment (including the onconfig file, sqlhosts, and last 5000 lines of the online log), and Operating System and kernel parameters. It needs to be run as root in order to gather all of the kernel stats.

AF files (-c af)

There is only one possible sub category for AFs: –s general. This command gathers the last three af files and associated stack files, basic OS version information, and the IDS environment (including the onconfig file, sqlhosts, and last 5000 lines of the online log).

This command could be useful to gather minimal information to raise an new case where the engine has already crashed.

Enterprise Replication (-c er)

Two sub categories are available for ER:

-s general

This option collects general information about the database server and operating system, as well as any recent af files and specific ER related diagnostics.

Specific ER data includes ats and ris files, server and replicate definitions (from the cdr command), and output from onstat -g cdr.

-s init

Collects basic IDS and OS information and any recent af files. For the ER initialisation status, cdr list server is run, and output is captured from crdstate.sql run against the syscdr database.

Performance (-c performance)

As well as the general option shown above, there’s an additional CPU option available designed to gather diagnostics in a CPU bottleneck scenario.

-s general

As above, as well as the basic IDS and OS information gathered by the others, this option collects information from a large number of onstat and OS system information commands. The onstats do differ slightly from those gathered in an onstat -a run.

-s cpu

As well as the usual IDS and OS information, this version of the performance option takes output from onstat -a, plus several captures of ps -elf, onstat -g all and other specific subsets. This appears to be the comprehensive implementation of ifxcollect command, and will use significantly more disk space for the output.

Onbar (-c onbar)

Designed to be run specifically during or following an onbar issue, this option has specific sub categories for backup and restore problems:

-s archive_failure

This option collects recent af files and the bar_act log, as well as archive specific onstat and ps captures on each iteration.

-s restore_failure

Similar to the above, this command collects recent af files and the bar_act log as well as restore specific onstat captures.

Ontape (-c ontape)

The two commands for onbar are also available for ontape backups and restores:

-s archive_failure

Almost identical to the above onbar command version, but without the bar_act.log capture.

-s restore_failure

Almost identical to the above onbar command version, but without the bar_act.log capture.

Connection (-c connection)

Useful for gathering diagnostics during connection issues, there are two sub categories for this command:

-s failure

This sub-option grabs the hosts file, a netstat -an, recent af files, as well as the usual basic OS and IDS version information.

-s hang

A more comprehensive version of the above. This command runs several ip based commands, lists the informixtmp directory, and takes an onstat -a, onstat -g all, onstat -g stk all, and any latest af files on each iteration.

Customer (-c cust)

There is only one sub-category available for this option: -s prof, to gather a ‘customer profile’. Running this command will first ask you to agree to a Standard Disclaimer; agreeing to this will generate the diagnostics in the usual place. This option generates noticeably different data to the others. As well as gathering a lot of the usual stats on the machine, network, environment and server configuration, information on the number of rows, row size and table counts is also collected.

Caveats

The command can take a while to run, especially with multiple repeats (the default is 3x repeats at 5 minute intervals). This needs to be taken into consideration, especially if you are waiting to restart the engine. Considerable disk space may also be required in the $INFORMIXDIR/isa/data directory – and it does not look like this destination can be changed.

While there are options to cover a lot of different scenarios, the generated diagnostics may not be everything Support require for your specific issue – they should be a good starting point at least though.

Conclusion

It can be hard to know exactly which diagnostics to run during an issue with IDS, and you will need to react quickly during a high priority incident. Running ifxcollect can be a relatively quick way to gather enough diagnostics for several different scenarios in order to raise a case with Technical Support.

Disclaimer

Suggestions above are provided “as is” without warranty of any kind, either express or implied, including without limitation any implied warranties of condition, uninterrupted use, merchantability, fitness for a particular purpose, or non-infringement.