KB 99 (JAVA): SAP PI/PO missing messages

Description

Sometimes you may be missing PI message data in Splunk. This can be caused by a number of factors outlined below.

Causes

Max Messages Setting

The PowerConnect Agent uses a SAP PI web service to retrieve PI message metadata, payloads and logs. This web service has a mandatory limit parameter which limits the number of results the web service will return. This is to protect the SAP PI system from large queries which return large numbers of results.

By default the PowerConnect agent will return 100 messages per polled interval per filter e.g. if the agent is configured to poll for PI messages every 1 minute and a single simple filter has been configured, the maximum number of PI messages each minute will be 100. The PowerConnect log shows how many PI messages are being sent per polling interval and can indicate if the max message limit is being hit e.g:

In this instance the max messages setting is 100 and PowerConnect has sent exactly 100 PI messages to Splunk and therefore has hit the limit.

Logged Messages

PI has two types of message persistence with regards to processing steps – staging and logging. The differences between these are documented here: https://blogs.sap.com/2012/11/06/message-staging-and-logging-options-in-advanced-adapter-engine-of-pi-73x/

Logged messages are designed to be ephemeral. They live in a cache for a period of time, then are moved to the database where certain metadata about the message is removed including the Node ID where the message was processed. Once removed from the cache these PI messages are no longer visible to the SAP PI web service as the PowerConnect agent relies on the Node ID as part of its filter so that each PowerConnect agent only retrieves messages for the node it’s running on.

This behaviour can lead to missed messages where the SAP PI cache has been purged before the PowerConnect agent has queried the web service. This is more likely to occur in high volume systems where the PI cache is purged more frequently. 

To confirm whether a PI message is logged, use the PI Message Monitoring console in NWA to inspect the message. If the Quality of Service is BE (Best Effort) and the status of the message has “Log Version” as it’s value then the PI message is logged rather than staged:

Resolution

Max Messages Setting

To raise the limit click on the PI Monitoring tab in the PowerConnect UI, click the Configuration tab then increase the “Max number of messages per minute” setting:

Logged Messages

In this scenario if messages are being missed, it is recommended to convert them to staged messages so that they are persisted with the relevant metadata and are not subject to cache expiry.

Product version

[SAP product version]