Unix system crashing - Need help to identify the issue

Hi,
We are accessing our remote webserver using libcurl.We are using GET method to post the request.The request is a simplified URL.We will receive XML Response for the URL request.
Problem
For Few Scenarios, response for the URL request is received & our component is killed abruptly. Can anyone pls help me what would be the reason for this issue at the earliest.
I have attached the core dump below which will help to track the issue clearly.

core dump: :

fe2d65e0 _malloc_unlocked (800, 0, 6582f8, fe2d63a0, 6ed718, 0) + 230
fe2d6394 malloc (800, 1, e8070, fec574e0, fe3be3c0, fe3c85b8) + 4c
fec574e0 __1c2n6FI_pv_ (800, 1c, e8070, 15540, fec6c9f8, fe3c85b8) + 2c
fe625868 __1cLxercesc_2_5RMemoryManagerImplIallocate6MI_pv_ (65018, 800, 20, b3bf08, 1, b3bf08) + 14
fe6990fc __1cLxercesc_2_5MXMLBufferMgrLbidOnBuffer6M_rn0AJXMLBuffer__ (d63dbc, fe7f0f9a, 37, 8004, 4, 37) + e4
fe6ab880 __1cLxercesc_2_5KXMLScannerLscanXMLDecl6Mkn0BJDeclTypes__v_ (d63ce8, 1, d63dbc, ea610, 0, d63d58) + 18
fe6ab658 __1cLxercesc_2_5KXMLScannerKscanProlog6M_v_ (d63ce8, 6ef458, 1, 7, 1, 1a273f8) + c8
fe611d84 __1cLxercesc_2_5MIGXMLScannerMscanDocument6Mrkn0ALInputSource__v_ (d63ce8, 0, fe611d3c, 0, 18c, fee726fc) + 48
fe594c04 __1cLxercesc_2_5RAbstractDOMParserFparse6Mrkn0ALInputSource__v_ (19e758, fb1fb080, 4f9, 26f308, d63ce8, 65018) + 8c
fee20bb8 __1cKXMLWrapperQXMLWrapperCommon6MpnHLogging_pnMHttpResponse__v_ (19e750, 2fc, fe7c8e98, fee430b8, 0, 26f308) + 12c
fee20a34 __1cKXMLWrapper2t5B6MpnHLogging_pnMHttpResponse_nUt_requestSensitivity__v_ (19e750, e1d38, b46b08, 0, fec6c9f8, 19e750) + 94
ff244da4 __1cJCMPClientKscrRequest6MrnJRWCString_pkcpc_i_ (e36c8, fb1fb404, fb1fb414, fb1fb860, ff257664, ff2470fd) + 1d0
ff2439fc __1cJCMPClientOserviceRequest6MpknMASOF_Message_r3_i_ (e36c8, 44c, fb1fbf3c, 0, fb1fb860, fb1fb40c) + 184
feebb584 __1cRASOF_WorkDirectorLcallService6MpknMASOF_Message_r3_i_ (5a998, 2461e8, fb1fbf3c, ff258030, 2461ec, ff243878) + 1a8
feebbd88 __1cRASOF_WorkerThreadNthreadRoutine6M_v_ (5d338, feed07f0, feed0758, 0, fe027200, 1) + 80
feebbe8c workerThreadRoutine (5d338, fb1fc000, 0, 0, feebbe88, 0) + 4
fe346d4c _lwp_start (0, 0, 0, 0, 0, 0)

The crash dump tells you nothing without the code.

Assuming it's a SEGV, since it's in malloc() your code is almost certainly corrupting your heap.

You probably should use something like this on your code to actually find such bugs:

IBM Software - Rational Purify

Please find the code sinnpet

unsigned int CMPClient::scrGetResponse(RWCString &pck, XMLWrapper *parse, DOMElement * res, char* rspBuf)
{
  //docClient SCR_SUCCESS N/A
  //docClient SCR_RES_SCR_STATUS N/A
  //docClient SCR_RES_ACCOUNT_COUNT count
  //docClient SCR_RES_1_ACCOUNT_NO listOfSearchResults[]!searchResult!billingAccountNumber
  //docClient SCR_RES_1_ACCOUNT_STATUS listOfSearchResults[]!searchResult!billingAccountNumber

  unsigned int result = SCR_SUCCESS;
  try {
    RWCString temp;

    XMLArray *accountDetailArray;
    unsigned int noOfAccountDetail = 0;
    unsigned int noOfAccountDetails = 0;
    unsigned int accountDetailOffset = 0;
    DOMElement *res2;
    const accountDetailBlockSize = SCR_RES_1_ACCOUNT_NO_LEN + SCR_RES_1_ACCOUNT_STATUS_LEN +
    SCR_RES_1_ACCOUNT_NAME_LEN;
    if (res2 = parse->getElement("listOfSearchResults",res, false)) {
      accountDetailArray = parse->getArray("listOfSearchResults!searchResult",res2);
      // ACE received the response without the account number in each search result.
      //Thus, we will be incrementing the account Detail count only if account number present
      noOfAccountDetails = accountDetailArray->numEntries();

      if ( noOfAccountDetails == 0) {
        result = SCR_NO_ACCOUNT_NUMBER_RETURNED;
      } else {
        for(int i=0;i < noOfAccountDetails && i < MAXRESULTCOUNT;i++){
          DOMElement * res1 = accountDetailArray->getEntry(i);
          RWCString AccountNumber="";
          if(parse->getValue("billingAccountNumber", res1, AccountNumber,false)){
            AccountNumber = AccountNumber.strip(RWCString::trailing, MSG_PADDER);
          }
          RWCString AccountStatus="";
          if(parse->getValue("billingAccountStatus", res1, AccountStatus,false)) {
            AccountStatus = AccountStatus.strip(RWCString::trailing, MSG_PADDER);
          }
          RWCString Accountname="";
        if(parse->getValue("name", res1, Accountname,false)) {
          Accountname = Accountname.strip(RWCString::trailing, MSG_PADDER);
        }
        setField(rspBuf, AccountNumber, SCR_RES_1_ACCOUNT_NO_OFFSET + accountDetailOffset,
        SCR_RES_1_ACCOUNT_NO_LEN);
        setField(rspBuf, AccountStatus, SCR_RES_1_ACCOUNT_STATUS_OFFSET + accountDetailOffset,
        SCR_RES_1_ACCOUNT_STATUS_LEN);
        setField(rspBuf, Accountname, SCR_RES_1_ACCOUNT_NAME_OFFSET + accountDetailOffset,
        SCR_RES_1_ACCOUNT_NAME_LEN);
        accountDetailOffset += accountDetailBlockSize;
        noOfAccountDetail++;
      } // End of 1st For loop
      //The below block is used to identify the type of customer.
      unsigned int customerDetailOffset = 0;
      const customerDetailBlockSize = SCR_RES_1_CUSTOMER_TYPE_LEN;
    
      for(int i=0;i < noOfAccountDetails && i < MAXRESULTCOUNT;i++){
        DOMElement * res1 = accountDetailArray->getEntry(i);
        RWCString customerType="";
        RWCString TYPE="";
        if(parse->getValue("type", res1, TYPE,false)) {
          TYPE = TYPE.strip(RWCString::trailing, MSG_PADDER);
          if(TYPE == "COMPANY"){
            customerType = "B";
          }else if (TYPE == "CONSUMER"){
            customerType = "R";
          } else {
            customerType = "O";
          }
          setField(rspBuf, customerType, SCR_RES_1_CUSTOMER_TYPE_OFFSET + customerDetailOffset,
          SCR_RES_1_CUSTOMER_TYPE_LEN);
        }
        customerDetailOffset += customerDetailBlockSize;
      } //End of 2nd For loop
      setField(rspBuf, noOfAccountDetail, SCR_RES_ACCOUNT_COUNT_OFFSET, SCR_RES_ACCOUNT_COUNT_LEN);
    }
  }
}
catch (NotFound &e){
  result = ALL_MISSING_RESPONSE_FIELD; //YYY was MISSING_RESPONSE_FIELD;
  m_logger->logError ("CMPClient::scrGetResponse missing response element: %s, PCK=%s",
  e.what(), (const char *) pck);
}
return result;
}

Please use code tags to preserve indentation and make code readable.

Regards,
Alister

Sorry.. I am not aware of.. Could you please tell me what do you mean by "use code tags"

It means to surround your code snippets with the following markup:

(CODE)
First Line of your code snippet.
...etc...
Last Line of your code snippet.
(/CODE)

But use brackets [] instead of parentheses () around the terms CODE and /CODE.

Regards,
Alister

The first thing you should check is the validity of the XML that you are trying to parse here.

If the XML is part of a bigger stream, I would try to log all XML documents and process return codes to a log file.

XML processes often crash during parse...

Also instead of doing all operations in the try{}catch() {} block you should better check the results of your operations.

i.e.:

accountDetailArray = parse>getArray("listOfSearchResults!searchResult",res2);
noOfAccountDetails = accountDetailArray->numEntries();

Here you should check

if (accountDetailArray == null ) return ((unsigned int)-1); /* just a possible error value */

Also, if this function is true source of your SEGFAULT you should

catch(Error e) { /*any Error*/}

Thanks ikrabbe ..

When we simulate the same secnario with the exact response received during the crash , it worked fine. I have given one of the sample XML response we received

SAMPLE RESPONSE:

[response]
 
<?xml version="1.0" encoding="UTF-8" standalone="yes" ?> 
- <searchCustomerResponse xmlns="capabilities.nat.bt.com/xsd/crm/v3/Customers"
- <listOfSearchResults>
- <searchResult>
<customerAccountId>xxxxx</customerAccountId> 
<name>Mr BROOKE HALLIWELL</name> 
<creditScore /> 
<type>CONSUMER</type> 
<primaryContactId>xxxxx</primaryContactId> 
<billingAccountNumber>xxxxxx</billingAccountNumber> 
<billingAccountStatus>Closed</billingAccountStatus> 
<telNo>xxxxxxx</telNo> 
<restrictedVisibilityIndicator>Y</restrictedVisibilityIndicator> 
- <address>
<id>xxxx</id> 
<shortDescription>16, , , TIBBERSLEY AVENUE, BILLINGHAM, TS23 1JP</shortDescription> 
</address>
<accountManager /> 
</searchResult>
- <searchResult>
<customerAccountId>xxxxxx</customerAccountId> 
<name>BTSS (CORAL RACING)</name> 
<creditScore /> 
<type>ACCOUNT(BUS)</type> 
<billingAccountNumber>xxxxx</billingAccountNumber> 
<billingAccountStatus>Closed</billingAccountStatus> 
<telNo>xxxxx</telNo> 
<restrictedVisibilityIndicator>Y</restrictedVisibilityIndicator> 
- <address>
<id>xxxx</id> 
<shortDescription>1, CHARNWOOD PLACE, , SOUTH ALBION STREET, LEICESTER, LE1 6JL</shortDescription> 
</address>
<accountManager /> 
</searchResult>
</listOfSearchResults>
</searchCustomerResponse>
 
[/response]
 

Note:
For security reasons have masked the private details in response.
I could post with RL , so remove http in xmlns tag.
Regards,
Banu

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?> 
- <searchCustomerResponse xmlns="capabilities.nat.bt.com/xsd/crm/v3/Customers"

1. What is the leading '-' in these lines? Copy&Paste error or real output. This would be one problem

<searchCustomerResponse 

... is never closed by '>'

Please post correct ascii or UTF-8 maybe to pastebin.

What leads you to believe your problem is all in that code?

Since you have both source, binary and core dump; you may just want to fire your debugger and see where it crashed.

Alternatively, compile with debugging symbols and reproduce the problem. Then you should get together with the binary and core dump a pretty good idea what's going on.

PS> the dump you have posted doesn't seem to mention a call to the CMPClient::scrGetResponse() method ?