Skip to main content

Common Call Generators

 

Symptom: BizTalk processing is slowing down.

Resolution

There could be many reasons why BizTalk processing is slowing down. One possible reason is that message parts are piling up in the MessageBox database. You can look into the Parts table to see whether it contains a huge number of records, and whether it keeps growing. You also can look into the job history of the MessageBox_Parts_Cleanup_BizTalkMsgBoxDb job to see whether it takes a long time to complete. A recently discovered performance bug has been fixed in http://support.microsoft.com/kb/2224680. You can get the fix by installing BizTalk Server 2006 R2 SP1 Cumulative Update 2 (CU2) and BizTalk Server 2009 CU2.

 

Symptom: When executing a Business Rule Engine (BRE) policy, you might experience high CPU.

Resolution

This typically occurs because of looping behavior within the policy and/or design of the policy. For additional information about BRE settings that can affect high CPU, see the following TechNet Wiki article:

High CPU when executing a Business Rules Engine (BRE) policy

 

Symptom: You have an orchestration or a schema exposed as a Windows Communication Foundation (WCF) service. You might intermittently receive the following error. The receive location is enabled.

Event Type: Error
Event Source: System.ServiceModel 3.0.0.0
Event Category: WebHost
Event ID: 3
Date: 1/24/2011
Time: 3:14:59 PM
User: MW\svcstlprdbiztalk
Computer: APP-STL-234
Description:
WebHost failed to process a request. Sender Information: System.ServiceModel.ServiceHostingEnvironment+HostingManager/20508338 Exception: System.ServiceModel.ServiceActivationException: The service '/test/test_ws.svc' cannot be activated due to an exception during compilation.  The exception message is: Receive location for address "/test/test_ws.svc" not found. (The BizTalk receive location may be disabled.). ---> Microsoft.BizTalk.Adapter.Wcf.AdapterException: Receive location for address "/test/test_ws.svc" not found. (The BizTalk receive location may be disabled.)
  at Microsoft.BizTalk.Adapter.Wcf.Runtime.ReceiveLocationManager`2.GetEndpointContext(Uri uri)
  at Microsoft.BizTalk.Adapter.Wcf.Runtime.WebServiceHostFactory`3.CreateServiceHost(String constructorString, Uri[] baseAddresses)
  at System.ServiceModel.ServiceHostingEnvironment.HostingManager.CreateService(String normalizedVirtualPath)
  at System.ServiceModel.ServiceHostingEnvironment.HostingManager.ActivateService(String normalizedVirtualPath)
  at System.ServiceModel.ServiceHostingEnvironment.HostingManager.EnsureServiceAvailable(String normalizedVirtualPath)
  --- End of inner exception stack trace ---
  at System.ServiceModel.ServiceHostingEnvironment.HostingManager.EnsureServiceAvailable(String normalizedVirtualPath)
  at System.ServiceModel.ServiceHostingEnvironment.EnsureServiceAvailableFast(String relativeVirtualPath) Process Name: w3wp Process ID: 21236

Resolution

This issue could happen when you have two receive locations that are configured with two different receive handlers (isolated hosts), but they run in the same isolated host instance (w3wp.exe). The issue can be resolved by running the two receive locations in two different isolated instances. You can find more information on this blog:
http://blogs.digitaldeposit.net/saravana/post/2008/08/26/Multiple-Isolated-Host-(Host-Instances)-Multiple-Virtual-directories-different-adapters-what-are-the-rules.aspx.

 

Symptom: After installing .NET Framework 4.0, BizTalk Adapter 2.0 for mySAP Business Suite might throw the following error when trying to receive data from the SAP system.

Event Type: Error
Event Source: BizTalk Adapter v2.0 for mySAP Business Suite
Event Category: None
Event ID: 0
Date: 9/16/2010
Time: 10:48:31 AM
User: N/A
Computer: TestMachine1
Description:
Error in Check Transaction: Failed to load the runtime.
(Exception from HRESULT: 0x80131700)

Resolution

You can use one of two methods to work around the problem:

  1. Manually reregister the sapreceiver.dll with the following command:
    C:\Windows\Microsoft.NET\Framework\v1.1.4322>regasm 
    "C:\Program Files\Microsoft BizTalk Adapter v2.0 for mySAP Business
    Suite\sapreceiver.dll"
  2. To repair Microsoft BizTalk Adapter 2.0 for mySAP Business Suite, open Control Panel and click Add/Remove Programs.

Symptom: Slow performance

Resolution

Although there can be many causes, these are the most common ones:

  • There are several tables that when not maintained, can impact performance:
    dta_DebugTrace (BizTalkDTADb) This table can be manually truncated within SQL after stopping the Tracking host (host that has “Allow Host Tracking” checked). This table is also maintained by the DTA Purge and Archive SQL Agent job. So, if this job is enabled and running, you may need to be more aggressive with the LiveDays and HardDeleteDays values in the job properties.
    dta_MessageInOutEvents (BizTalkDTADb) This table can be manually truncated within SQL after stopping the Tracking host. This table is also maintained by the DTA Purge and Archive SQL Agent job so if this job is enabled and running, you may need to be more aggressive with the LiveDays and HardDeleteDays values in the job properties.
    HostNameQ_Suspended (BizTalkMsgBoxDb) If there are a large number of suspended instances, BizTalk may throttle with a status code of 6, which means the Message Count in Database throttling threshold has been exceeded. If this data is not needed, the BizTalk Terminator tool can be used to delete the instances. If this data is needed, then investigate why they suspended. You can also resume the instances in batches until the backlog is cleared.
    Spool (BizTalkMsgBoxDb) This table contains references from data that is active, suspended and/or dehydrated. If the Spool table gets large, then look into the number of Suspended and/or Dehydrated instances. Also confirm that the BizTalk SQL Agent jobs are executing successfully. Once the Suspended and/or Dehydrated instances are removed, this table will reduce.
    TrackingData_x_x (BizTalkMsgBoxDb) If any or all of these tables exceed the Message Count in Database throttling threshold * 10, BizTalk may throttle with a status code of 6; which means the Message Count in Database throttling threshold has been exceeded. The Tracking host is responsible for moving this data so when there is a large BizTalkDTADb database, the Tracking host may not be able to write the data fast enough to the BizTalkDTADb database. If this tracking data is not needed, the BizTalk Terminator tool can be used to delete the TrackingData_x_x data. If this tracking data is needed, then cleaning up the BizTalkDTADb database will allow the Tracking host to move the data to the BizTalkDTADb database faster.
    A lack of database maintenance; see KB article 952555: How to maintain and troubleshoot BizTalkServer databases provides more detailed information.
  • Throttling with a status code of 6 is the most common cause for slow performance. This occurs when the Message Count in Database throttling threshold has been exceeded, which has a default value of 50,000. There are several tables in the BizTalkMsgBoxDb database that are impacted by this value: HostNameQ_Suspended, TrackingData_x_x and Spool:

    Spool and TrackingData_x_x: If the number of messages in these tables exceed 10 times the Message Count in Database value, then throttling will occur. Since the Message Count in Database default is 50,000, throttling will occur if any of these tables exceed 500,000 (50000*10).
    HostNameQ_Suspended: The row count is included in the 50,000 throttling threshold.

    If you suspect BizTalk is throttling, Performance Monitor can be used to determine the throttling state. Troubleshooting with Performance Monitor provides more information. Once you know the throttling status code, see How BizTalk Server Implements Host Throttling for the reason and a possible resolution.
  • A need to modify TCP settings; see KB article 970406: TCP settings that can impact BizTalk Server
  • BizTalk is clustered as opposed to using the BizTalk group functionality. Instead of a clustering a host, use the built-in load-balancing feature of BizTalk Server by creating the host on multiple computers running BizTalk Server in the group.
  • The Orchestration Profiler can also be used to help determine long-running shapes. This tool uses the Tracking events on the orchestration.
  • Custom components: Microsoft Customer Service and Support (CSS) see many issues where an orchestration calls a custom component. Consider adding tracing to your custom component and to the Expression shape to help determine which piece of code could be causing the delay. If CSS help is needed, add tracing statements to the Expression shape by referencing Microsoft.XLANGS.RuntimeTypes.dll from the project and adding code similar to the following throughout the Expression shape:

    Microsoft.XLANGs.RuntimeTypes.Trace.Tracer.TraceMessage(Microsoft.XLANGs.RuntimeTypes.TraceLevel.Info, "CustomXlang:{0}: hasnext = {1}", System.Convert.ToString(Microsoft.XLANGs.Core.Service.RootService.InstanceId), hasNext.ToString());

    After redeploying the application, capture a BizTalk trace using -all; see KB article 835451: How to usethe BizTalk Adapter Trace Utility inBizTalk Server.

Symptom: Windows could not start the Enterprise Single Sign-On Service service on Local Computer.
Error 0x80131700: 0x80131700

ERROR: Could not contact the SSO server 'localhost'. Check that SSO is configured and that the SSO service is running on that server.
(RPC: 0x800706D9: There are no more endpoints available from the endpoint mapper.)

Could not create SSOSQL. To fix the problem, reinstall SSO or try 'regasm SSOSQL.dll' from a Visual Studio command prompt.
Error Code: 0x80131700

Failed to connect to the SQL database 'SSODB' on SQL Server 'SQLServerName'
0x80131700 (Win32)

Note ENTSSO is a required service for BizTalk and HIS. As a result, these services also fail to start.

Cause: This issue occurs after installing .NET Framework 4.0. The registration of the assembly used by ENTSSO to access SQL Server does not specify the correct version of the .NET Framework. When .NET Framework 4.0 is installed, the assembly will try to use the newer framework and then fail to load. 

Resolution

To resolve this issue, install the 2252691 fix-- Update for Microsoft Enterprise Single Sign-On v4 (KB2252691)--on the ENTSSO server.

The hotfix will update the ENTSSO assembly registration with the correct version of the .NET framework.

More Information: See 2252691 FIX: 0x80131700 error code when starting or configuring the Enterprise Single Sign-On Service.


Symptom: Issues with references in BizTalk Server 2009 projects. You have a BizTalk Server 2009 project with a reference to another BizTalk Server 2009 project. You might notice the following issues:

  1. Referenced items within an orchestration might display a red exclamation point, might be missing, and might result in a build error.


    Resolution: There are two hotfixes available that address reference issues within a BizTalk project:

    977428  FIX: You experience various problems when you develop a BizTalk project that references another BizTalk project in Visual Studio on a computer that is running BizTalk Server 2009

    979153  FIX: BizTalk Server 2009 Orchestration Designer incorrectly indicates that there is an error in a BizTalk Server 2009 project

  2. In a BizTalk Server 2009 project, there is a map using a schema in the referenced BizTalk project. The Schema Type Picker in the map does not show the referenced project and schemas. It only shows local schemas.


    Resolution: In the BizTalk project being referenced, set the Build Action of the schema from None to BtsCompile.


Symptom: With WCF-based line-of-business (LOB) adapters you might notice some unexpected behaviors in BizTalk Server. For example, orchestration instances go into the Dehydrated state; and the receive location does not process new messages. The host instance might not seem that responsive anymore.

Resolution

The problem is caused by threadpool depletion. The threadpool depletion is in turn caused by a bug in the WCF LOB Adapter SDK.  We have released a fix for this issue, and you can get more details from http://support.microsoft.com/kb/2300507. This bug is inside the WCF LOB Adapter SDK, so it affects all adapters (including custom adapters) built on the SDK. The WCF-based SQL adapter is the one most affected just because of the way the SQL adapter works.

Additional Information about LOB Adapter Threading

BizTalk Server uses .NET thread pool to manage threads, and the thread pool size can be tuned per individual host instances through registry settings. In the case of WCF-based LOB adapters, they need quite a few threads to accomplish the job. You might notice that the thread count might be higher than expected. Consider the following:

  • When the adapter is executing receiving, the AdapterChannelListener method executes on its own thread. Here is a sample call stack in a memory dump.
  • 0:000> !clrstack
    OS Thread Id: 0x4b44 (153)
    ESP EIP
    28baeca8 7c82860c [HelperMethodFrame_1OBJ: 28baeca8] System.Threading.WaitHandle.WaitOneNative(Microsoft.Win32.SafeHandles.SafeWaitHandle, UInt32, Boolean, Boolean)
    28baed54 792b68af System.Threading.WaitHandle.WaitOne(Int64, Boolean)
    28baed70 792b6865 System.Threading.WaitHandle.WaitOne(Int32, Boolean)
    28baed84 21c7eb86 Microsoft.ServiceModel.Channels.Common.Channels.AdapterChannelListener`1[[System.__Canon, mscorlib]].OnWaitForChannel(System.TimeSpan)
    28baede8 2103011e System.ServiceModel.Channels.ChannelListenerBase.WaitForChannel(System.TimeSpan)
    28baedfc 21c7e70f Microsoft.ServiceModel.Channels.Common.Channels.AdapterChannelListener`1[[System.__Canon, mscorlib]].OnAcceptChannel(System.TimeSpan)
    28baee64 21035406 System.ServiceModel.Channels.ChannelListenerBase`1[[System.__Canon, mscorlib]].AcceptChannel(System.TimeSpan)
    28baee78 21c7e652 Microsoft.ServiceModel.Channels.Common.Channels.AdapterChannelListener`1[[System.__Canon, mscorlib]].AcceptAsyncCall(System.Object)
    28baeeb4 792c9e4f System.Threading._ThreadPoolWaitCallback.WaitCallback_Context(System.Object)
    28baeebc 792e01ef System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
    28baeed4 792ca3b3 System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(System.Threading._ThreadPoolWaitCallback)
    28baeee8 792ca249 System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(System.Object)
    28baf078 79e71b4c [GCFrame: 28baf078]

  • In another thread, the BizTalk WCF Runtime also dequeues the messages from an internal queue, and sends them to the MessageBox. Here is a sample Dequeue call stack.
  • 0:271> !clrstack
    OS Thread Id: 0x36a4 (271)
    ESP EIP
    1ed0efac 7c82860c [HelperMethodFrame_1OBJ: 1ed0efac] System.Threading.WaitHandle.WaitOneNative(Microsoft.Win32.SafeHandles.SafeWaitHandle, UInt32, Boolean, Boolean)
    1ed0f058 792b68af System.Threading.WaitHandle.WaitOne(Int64, Boolean)
    1ed0f074 792b6865 System.Threading.WaitHandle.WaitOne(Int32, Boolean)
    1ed0f088 792b682d System.Threading.WaitHandle.WaitOne()
    1ed0f090 1b938fa2 Microsoft.BizTalk.Adapter.Wcf.Runtime.BizTalkExecutionQueue.DequeueTask()
    1ed0f0c0 792d6d66 System.Threading.ThreadHelper.ThreadStart_Context(System.Object)
    1ed0f0cc 792e01ef System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
    1ed0f0e4 792d6ce4 System.Threading.ThreadHelper.ThreadStart()
    1ed0f30c 79e71b4c [GCFrame: 1ed0f30c]

  • On another thread, Microsoft.ServiceModel.Channels.dll calls into the adapter to execute. Here is a sample call stack into the adapter.
  • 0:000> !clrstack
    OS Thread Id: 0x3d0c (271)
    ESP EIP
    28aaee5c 7c82860c [HelperMethodFrame: 28aaee5c] System.Threading.Thread.SleepInternal(Int32)
    28aaeeb0 79299275 System.Threading.Thread.Sleep(Int32)
    28aaeeb4 21c7f9f5 MyCompany.MyAdapter.Adapter.WaitForMessage(System.TimeSpan)
    28aaef50 21c7f741 Microsoft.ServiceModel.Channels.Common.Channels.AdapterReplyChannel.WaitForRequest(System.TimeSpan)
    28aaefa0 210ccaa6 System.ServiceModel.Channels.ReplyOneWayChannelListener+ReplyOneWayInputChannel.WaitForMessage(System.TimeSpan)
    28aaefb0 21226a92 System.ServiceModel.Dispatcher.InputChannelBinder.WaitForMessage(System.TimeSpan)
    28aaefc0 21225e34 System.ServiceModel.Dispatcher.ErrorHandlingReceiver.WaitForMessage()
    28aaefec 211771f8 System.ServiceModel.Dispatcher.ChannelHandler.TransactedBatchLoop()
    28aaf028 21177790 System.ServiceModel.Dispatcher.ChannelHandler.SyncTransactionalMessagePump()
    28aaf034 21178610 System.ServiceModel.Dispatcher.ChannelHandler.OnStartSyncMessagePump(System.Object)
    28aaf03c 209639a6 System.ServiceModel.Channels.IOThreadScheduler+CriticalHelper+WorkItem.Invoke2()
    28aaf07c 20963948 System.ServiceModel.Channels.IOThreadScheduler+CriticalHelper+WorkItem.OnSecurityContextCallback(System.Object)
    28aaf084 7928fe65 System.Security.SecurityContext.Run(System.Security.SecurityContext, System.Threading.ContextCallback, System.Object)
    28aaf09c 2096390d System.ServiceModel.Channels.IOThreadScheduler+CriticalHelper+WorkItem.Invoke()
    28aaf0b0 20963890 System.ServiceModel.Channels.IOThreadScheduler+CriticalHelper.ProcessCallbacks()
    28aaf0e4 209636ea System.ServiceModel.Channels.IOThreadScheduler+CriticalHelper.CompletionCallback(System.Object)
    28aaf110 2096365f System.ServiceModel.Channels.IOThreadScheduler+CriticalHelper+ScheduledOverlapped.IOCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)
    28aaf11c 216ee6cd System.ServiceModel.Diagnostics.Utility+IOCompletionThunk.UnhandledExceptionFrame(UInt32, UInt32, System.Threading.NativeOverlapped*)
    28aaf150 7928cdc4 System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)
    28aaf2f0 79e71b4c [GCFrame: 28aaf2f0]

So, for every adapter that receives thread executing, you can expect two additional threads. For example, if there are 20 receiving threads executing the adapter, expect around 20 AdapterChannelListener threads and around 20 DequeueTask threads, for a total of about 60 threads.

If the host that is running the custom adapter is being shared, especially with other transactional-type adapters--for instance, MQ Series Client (MQSC) or MSMQ-- the process might experience thread starvation. In this situation, consider doing the following:

  1. Put the custom adapter in its own host.
  2. Increase MaxWorkerThreads and MaxIOThreads. The default value is 100, and can probably be safely increased to 200-400. Be sure to test these higher values.

Online Information


Symptom: When you use a WCF-based adapter in the BizTalk Adapter Pack, transactions are intermittently stopped due to a Timeout exception. For example, the WCF-based SQL adapter might return the following exception:

Details: “System.Data.SqlClient.SqlException: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.”

This problem usually happens under load, and can occur with the WCF-based Oracle Database, Oracle E-Business Suite, SQL, and SAP adapters.

Resolution

Set the receiveTimeout binding property of the receive location to 24.20:31:23.6470000 (24 days), which is the maximum value.

For more information, see:
Working with BizTalk Adapter for Oracle Database Binding Properties (Adapter Pack 2.0; BizTalk Server 2009)

Working with BizTalk Adapter for Oracle E-Business Suite Binding Properties (Adapter Pack 2.0; BizTalk Server 2009)
Working with BizTalk Adapter for SQL Server Binding Properties (Adapter Pack 2.0; BizTalk Server 2009)

Working with BizTalk Adapter 3.0 for mySAP Business Suite Binding Properties (Adapter Pack 2.0; BizTalk Server 2009)


Symptom:Configuring Certificates for BizTalk EDI/AS2

Resolution

The most common scenario is How-To:

  1. Log onto BizTalk Server as the BizTalk HOST account to install the certificate. This will ensure that the certificate is installed to the host account’s Personal Store in the Certificates snap-in.
  2. Determine the certificate usage, such as encryption or signing. Once you know this, the chart in Configuring Certificates for AS2 can be used to determine which certificate (.cer or .pfx) to install in which certificates store (Personal vs. Other People).


    For more information, see:
    KB article 942253 How to configure certificates that are used to sign and to encrypt AS2 messages in BizTalk Server 2006 R2

  3. The certificate is expired or has been renewed. Treat this as a brand new certificate installation by installing the new certificate into the appropriate store in the Certificates snap-in.

Symptom:The BAM feature has become more popular recently. Besides using the BAM port, some customers are using custom queries to use these BAM activity tables. However, the query performance is very slow.

Resolution

Microsoft Customer Service and Support (CSS) has created a hotfix ( FIX: BizTalk Server 2006 or BizTalk Server 2006 R2 experiences low performance when you execute an SQL query that refers to the "ActivityID" field in the bam_ActivityName_CompletedRelationships table) to improve the query performance by adding a non-clustered index on the activity tables.


Symptom:Unable to configure UDDI

Resolution

When you try to configure UDDI 3.0, which comes with BizTalk Server 2009, you might receive this error: “Unable to connect to the Database Server”. This problem happens when you use a named SQL instance for the UDDI store. A fix is available from http://support.microsoft.com/kb/975684.


Symptom: In BizTalk Administration, you notice that an instance remains in an Active or Ready to Run state. Restarting the BizTalk host instance may resolve the issue.

Resolution

These are the most common causes and associated resolution:

  1. All BizTalk artifacts (Receive, Orchestration, Send and Tracking) are in a single host. Create three new hosts: ReceiveHost, SendHost, and OrchestrationHost. Put all receive locations into the ReceiveHost, all send ports into the SendHost, and all orchestrations into the OrchestrationHost. Dedicate BizTalkServerApplication host to only be the Tracking host. For more information, see Optimizing BizTalk Server Performance.
  2. There are large tables in the BizTalkMsgBoxDb and/or BizTalkDTADb databases. To view the database and table sizes, you can use MsgBoxViewer. To determine which tables are usually the largest and any actions that can be taken, see KB article 952555: How to maintain and troubleshoot BizTalk Server databases. For example, the BizTalkDTADb database is 8 GB but the dta_MessageInOutEvents and dta_DebugTrace tables both have more than 3 million rows. In this scenario, you stop the Tracking host and truncate these tables.
  3. If the instance is in a Ready to Run state, confirm that the BizTalk artifact is started, for instance, the receive location, orchestration, or send port.
  4. A specific adapter and the activity being performed could also be the cause. Examples:

    • The SQL adapter is executing a stored procedure that might be causing a deadlock or a -2 SPID scenario. In this case, investigate the deadlock and, if using transactions, consider rolling back the transaction. This might resolve the -2 SPID and help determine which T-SQL syntax is responsible. If you will use the SQL adapter heavily for transactions, put the SQL Receive Locations in their own host and the SQL Send Ports in their own host.
    • If you use MSMQ heavily, put the MSMQ Receive Locations in their own host and the MSMQ Send Ports in their own host.
      If you have many receive locations using the FILE Adapter, modify the MaxCmds and MaxMpxCt registry values as documented in KB article 810886: "The network BIOS command limit has been reached" error message in Windows Server 2003, in Windows XP, and in Windows 2000 Server.”
    • If there are hundreds of concurrently running orchestration instances, you might need to modify the CLR Hosting settings in the registry. If you use SOAP or the HTTP adapters, you are bound by the ASP.Net settings in the .config file. For more information about modifying these values, see Configuration Parameters that Affect Adapter Performance.

Symptom:In the BizTalk group, there are two BizTalk computers. On BizTalk1, there is an isolated host. On BizTalk2, a client application submits messages to the isolated host on BizTalk1. This returns the following error: “The Messaging Engine failed to register the adapter for "Submit" for the receive location "Test Receive Location". Please verify that the receive location exists, and that the isolated adapter runs under an account that has access to the BizTalk databases.”

This error occurs even when the account is a member of the BizTalk Isolated Host Users group.

Resolution
Create the isolated host instance on the BizTalk computer that hosts the client application.

 

Symptom:There is a BizTalk application that used Pipeline1 and now uses Pipeline2. The BizTalk application is deployed. You modify the application in Visual Studio and redeploy from Visual Studio. When you redeploy, Pipeline1 is used instead of Pipeline2.

Resolution
This occurs when a binding file from a previous deployment is applied. To resolve this, delete the residue binding file and then redeploy. Binding file locations:
C:\Documents and Settings\{user}\Application data\Microsoft\BizTalk Server\Deployment\BindingFiles
C:\Users\{user}\AppData\Local\BizTalk\Deployment\BindingFiles