SharePoint 2013: Event 1000: Faulting application name: Microsoft.Workflow.ServiceHost.exe

Problem

After installing the September 2014 PU to your SharePoint 2013 farm, users began to report that their 2013 workflows are performing sluggishly or not at all.  No users reported any adverse impacts to 2010 workflows.  Checking the Workflow Manager 1.0 services, found that all of the services were running except for the Workflow Manager Backend service, which repeatedly stopped and then restarted. Checking the Windows Server Application log, found these error events recurring frequently: 1000, 1026 and 7031 about every 50 seconds or so.  Checking the Microsoft Workflow Operational log, found a regular cycle of error and critical events: 358, 359, 19, 10 and 358 also occurring every minute or so.  This article captures the information and data examined and the efforts undertaken to resolve this issue.

UPDATE (12/03/14): This error was originally  believed to be due to a patch installation.  These errors occurred again six weeks later, this time without there having been any patches, updates or other software installed.  The errors were eventually traced to a workflow design issue.  Removing the running workflow instances resolved the errors.
 
Troubleshooting

1. Checked Workflow Manager services and related services:
• Service Bus Gateway
• Service Bus Message Broker
• Windows Fabric Host Service
• Workflow Manager Backend - repeatedly stopped and restarted; starting it (when stopped) did not resolve issue.
2. Checked Windows Server Application log:
• Saw events 1000, 1026 and 7031
3. Checked Microsoft Workflow logs
• Saw events 358, 359, 19 and 10
4. Checked Workflow Management Site through IIS:
• Site was started.  Restarting did not resolve issue.
5. Checked Workflow Management Application Pool
• Pool was started.  Restarting it did not resolve issue.
6. Checked local Administrators group:
• Added workflow service account to local Administrators group and then rebooted server, but this did not resolve issue.
7. Executed script per reference 2, but this did not resolve issue.
8. Installed full September 2014 CU, but this did not resolve issue.
9. Performed full uninstallation and then a fresh installation of entire Workflow Manager 1.0 system.
• Issue resolved
10. UPDATE (12/03/14)
11. Issue recurred six weeks later. 
12. More thoroughly reviewed error events occuring in Workflow Operational log
• Noted that the error messages discussed workflow instances and that two workflow instance IDs were repeatedly presented in these error messages.
13. Checked Workflows Settings page for all lists having 2013 workflows attached (4).
• Found one 2013 workflow having two running instances.
14. Removed these instances, and then checked server event logs:
• No more error messages - stopped the moment the instances were removed.
15. Workflow designer reviewed workflow, based upon error message text and input from SharePoint administrator
1. Workflow designer discovered an issue in the workflow.
16. New Solution
1. Revise workflow

Solution

1. Review workflow
2. Revise workflow

References

1. Workflow Manager 1.0 (SharePoint 2013): The process was terminated due to an unhandled exception
2. Update Workflow in SharePoint Server 2013
3. Install & Configure Workflow Manager 1.0
4. Configuring The Service Bus Workflow Farm
5. Workflow Manager 1.0-SharePoint 2013
6. New Workflow Manager 1.0 Getting Started Tutorial
7. Updates for SharePoint 2013 Workflow
8. Installing Workflow Manager 1.0
9. SharePoint 2013: Workflow Manager Installation and Configuration
10. Least Privilege Configuration for Workflow Manager with SharePoint 2013
11. Export SharePoint Designer 2013 Workflow XAML
12. Understanding how to package and deploy workflow in SharePoint 2013
13. SharePoint 2013: Workflow Manager Installation and configuration issues…
14. Installing Workflow Manager Cumulative Update 1
15. Understanding SharePoint 2013 Workflow Backup
16. Configuring Workflow Manager 1.0
17. Workflow Manager Farms for SharePoint 2013 Part Three: Switching an existing farm to use Domain CA issued certificates
18. Monitoring Workflow Manager 1.0
19. Creating a New Farm
20. UPDATE
21. Dispatcher failfast is causing the Workflow Manager backend to terminate
22. SharePoint 2013 workflow fundamentals
23. SharePoint 2013: how to restore Workflow Manager 1.0 workflows in the event of a disaster

Notes

• 2013 Workflow Topology:
• Single App Server: hosts Workflow Manager 1.0, Service Bus 1.0 and Windows Fabric Host
• Two WFEs: host Workflow Manager 1.0 Client
• Note that performing a full un-installation, followed by a new installation, completely wipes out any 2013 workflows that may already have been created.   While performing a restore / recovery operation is theoretically possible, and some admins appear to have done this successfully, you don't have to perform a full Workflow 2013 restore operation in order to recovery the 2013 workflows, which are actually stored in the farm content databases.  Instead, you can recover them by restoring the content databases.
   
  UPDATE (12/3/14): opened Microsoft Support ticket on original issue. Microsoft SharePoint Admin support specialist indicated that the 2013 workflows are actually stored in the 2013 content databases; and that these will become available again on successful re-installation of Workflow Manager 1.0 and re-integration with SharePoint 2013. This was verified.

Events

• Windows Server Application Log
• Error Event 1000:
  Faulting application name: Microsoft.Workflow.ServiceHost.exe, version: 1.0.20922.0, time stamp: 0x505e1b24 Faulting module name: KERNELBASE.dll, version: 6.2.9200.16864, time stamp: 0x531d34d8 Exception code: 0xe0434352 Fault offset: 0x0000000000047b8c Faulting process id: 0x4018 Faulting application start time: 0x01cfe8ac9a5561b0 Faulting application path: C:\Program Files\Workflow Manager\1.0\Workflow\Artifacts\Microsoft.Workflow.ServiceHost.exe Faulting module path: C:\Windows\system32\KERNELBASE.dll
   
• Error Event 1026:
  Application: Microsoft.Workflow.ServiceHost.exe Framework Version: v4.0.30319 Description: The process was terminated due to an unhandled exception. Exception Info: Microsoft.Workflow.Common.FatalException Stack: at Microsoft.Workflow.Common.Fx+<>c__DisplayClass2.<failfast>b__0() at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at System.Threading.ThreadHelper.ThreadStart()
   
• Error Event 7031:
  The Workflow Manager Backend service terminated unexpectedly. It has done this 6121 time(s). The following corrective action will be taken in 30000 milliseconds: Restart the service
   
• Microsoft-Workflow Operational Log
• Error Event 358:
  Dispatcher encountered an unexpected exception: System.Runtime.CallbackException: The call to OnRequestAbort on the WorkflowInstance '35616a0a-26da-4765-b92d-3f6eb5727508' threw an exception. This is considered fatal. See inner exception for details. ---> System.AggregateException: Should be processing if notifying about persistable idle. ---> System.NullReferenceException: Object reference not set to an instance of an object. at System.Activities.Runtime.ActivityExecutor.CancelRootActivity() at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.ScheduleCancel() at Microsoft.Activities.Hosting.HostedWorkflowInstance.Cancel() at Microsoft.Activities.Hosting.WorkflowSession.TryDispatchNextSubroutineWorkItem() at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() --- End of inner exception stack trace --- at Microsoft.Activities.Hosting.HostedWorkflowInstance.NotifyPaused(Exception abortReason) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.AbortInstance(Exception reason, Boolean isWorkflowThread) at System.Activities.Runtime.ActivityExecutor.AbortWorkflowInstance(Exception reason) --- End of inner exception stack trace --- at System.Activities.Runtime.ActivityExecutor.AbortWorkflowInstance(Exception reason) at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.TryPropagateCompletion(HostedWorkflowInstance instance) at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.WorkflowSession.TryStartSubroutine(Guid subroutineId, Guid rootActivityId, Dictionary`2 inArguments, List`1 executionProperties) at Microsoft.Activities.Hosting.WorkflowSession.TryDispatchNextSubroutineWorkItem() at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.Cancel() at Microsoft.Activities.Hosting.WorkflowSession.OnUnhandledException(HostedWorkflowInstance throwingInstance, Exception unhandledException, Boolean isRootFaulted) at System.Activities.Runtime.ActivityExecutor.NotifyUnhandledException(Exception exception, ActivityInstance source) at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.TryPropagateCompletion(HostedWorkflowInstance instance) at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.TryPropagateCompletion(HostedWorkflowInstance instance) at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.WorkflowSession.TryStartSubroutine(Guid subroutineId, Guid rootActivityId, Dictionary`2 inArguments, List`1 executionProperties) at Microsoft.Activities.Hosting.WorkflowSession.TryDispatchNextSubroutineWorkItem() at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.ResumeBookmark(Guid subroutineId, Bookmark bookmark, Object value, Action`1 sessionIdleCallback) at Microsoft.Activities.Hosting.WorkflowSession.ResumeInternal(BookmarkWaiter bookmarkWaiter, Object value, DateTime resumptionTime, Action`1 sessionIdleCallback) at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.ProcessWaiter() at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.Isolate(AsyncCompletion callback, IAsyncResult result) at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.ExceptionHandlingFrame(IAsyncResult result)
   
• Critical Event 359:
  Dispatcher failfast is causing the Workflow Manager backend to terminate. Exception: System.AggregateException: The dispatcher failure rate has reached 11%. InnerException is the last failure; see traces for other failures. ---> System.Runtime.CallbackException: The call to OnRequestAbort on the WorkflowInstance '35616a0a-26da-4765-b92d-3f6eb5727508' threw an exception. This is considered fatal. See inner exception for details. ---> System.AggregateException: Should be processing if notifying about persistable idle. ---> System.NullReferenceException: Object reference not set to an instance of an object. at System.Activities.Runtime.ActivityExecutor.CancelRootActivity() at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.ScheduleCancel() at Microsoft.Activities.Hosting.HostedWorkflowInstance.Cancel() at Microsoft.Activities.Hosting.WorkflowSession.TryDispatchNextSubroutineWorkItem() at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() --- End of inner exception stack trace --- at Microsoft.Activities.Hosting.HostedWorkflowInstance.NotifyPaused(Exception abortReason) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.AbortInstance(Exception reason, Boolean isWorkflowThread) at System.Activities.Runtime.ActivityExecutor.AbortWorkflowInstance(Exception reason) --- End of inner exception stack trace --- at System.Activities.Runtime.ActivityExecutor.AbortWorkflowInstance(Exception reason) at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.TryPropagateCompletion(HostedWorkflowInstance instance) at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.WorkflowSession.TryStartSubroutine(Guid subroutineId, Guid rootActivityId, Dictionary`2 inArguments, List`1 executionProperties) at Microsoft.Activities.Hosting.WorkflowSession.TryDispatchNextSubroutineWorkItem() at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.Cancel() at Microsoft.Activities.Hosting.WorkflowSession.OnUnhandledException(HostedWorkflowInstance throwingInstance, Exception unhandledException, Boolean isRootFaulted) at System.Activities.Runtime.ActivityExecutor.NotifyUnhandledException(Exception exception, ActivityInstance source) at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.TryPropagateCompletion(HostedWorkflowInstance instance) at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.TryPropagateCompletion(HostedWorkflowInstance instance) at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.WorkflowSession.TryStartSubroutine(Guid subroutineId, Guid rootActivityId, Dictionary`2 inArguments, List`1 executionProperties) at Microsoft.Activities.Hosting.WorkflowSession.TryDispatchNextSubroutineWorkItem() at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.ResumeBookmark(Guid subroutineId, Bookmark bookmark, Object value, Action`1 sessionIdleCallback) at Microsoft.Activities.Hosting.WorkflowSession.ResumeInternal(BookmarkWaiter bookmarkWaiter, Object value, DateTime resumptionTime, Action`1 sessionIdleCallback) at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.ProcessWaiter() at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.Isolate(AsyncCompletion callback, IAsyncResult result) at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.ExceptionHandlingFrame(IAsyncResult result) --- End of inner exception stack trace --- ---> (Inner Exception #0) System.Runtime.CallbackException: The call to OnRequestAbort on the WorkflowInstance '35616a0a-26da-4765-b92d-3f6eb5727508' threw an exception. This is considered fatal. See inner exception for details. ---> System.AggregateException: Should be processing if notifying about persistable idle. ---> System.NullReferenceException: Object reference not set to an instance of an object. at System.Activities.Runtime.ActivityExecutor.CancelRootActivity() at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.ScheduleCancel() at Microsoft.Activities.Hosting.HostedWorkflowInstance.Cancel() at Microsoft.Activities.Hosting.WorkflowSession.TryDispatchNextSubroutineWorkItem() at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() --- End of inner exception stack trace --- at Microsoft.Activities.Hosting.HostedWorkflowInstance.NotifyPaused(Exception abortReason) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.AbortInstance(Exception reason, Boolean isWorkflowThread) at System.Activities.Runtime.ActivityExecutor.AbortWorkflowInstance(Exception reason) --- End of inner exception stack trace --- at System.Activities.Runtime.ActivityExecutor.AbortWorkflowInstance(Exception reason) at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.TryPropagateCompletion(HostedWorkflowInstance instance) at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.WorkflowSession.TryStartSubroutine(Guid subroutineId, Guid rootActivityId, Dictionary`2 inArguments, List`1 executionProperties) at Microsoft.Activities.Hosting.WorkflowSession.TryDispatchNextSubroutineWorkItem() at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.Cancel() at Microsoft.Activities.Hosting.WorkflowSession.OnUnhandledException(HostedWorkflowInstance throwingInstance, Exception unhandledException, Boolean isRootFaulted) at System.Activities.Runtime.ActivityExecutor.NotifyUnhandledException(Exception exception, ActivityInstance source) at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.TryPropagateCompletion(HostedWorkflowInstance instance) at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.TryPropagateCompletion(HostedWorkflowInstance instance) at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.WorkflowSession.TryStartSubroutine(Guid subroutineId, Guid rootActivityId, Dictionary`2 inArguments, List`1 executionProperties) at Microsoft.Activities.Hosting.WorkflowSession.TryDispatchNextSubroutineWorkItem() at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.ResumeBookmark(Guid subroutineId, Bookmark bookmark, Object value, Action`1 sessionIdleCallback) at Microsoft.Activities.Hosting.WorkflowSession.ResumeInternal(BookmarkWaiter bookmarkWaiter, Object value, DateTime resumptionTime, Action`1 sessionIdleCallback) at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.ProcessWaiter() at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.Isolate(AsyncCompletion callback, IAsyncResult result) at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.ExceptionHandlingFrame(IAsyncResult result)<--- data-blogger-escaped-li="">
   
• Error Event 19:
  Application terminating due to FailFast. Message : An unrecoverable error occurred. For diagnostic purposes, this English message is associated with the failure: 'The dispatcher failure rate has reached 11%. InnerException is the last failure; see traces for other failures.'., Exception: System.AggregateException: The dispatcher failure rate has reached 11%. InnerException is the last failure; see traces for other failures. ---> System.Runtime.CallbackException: The call to OnRequestAbort on the WorkflowInstance '35616a0a-26da-4765-b92d-3f6eb5727508' threw an exception. This is considered fatal. See inner exception for details. ---> System.AggregateException: Should be processing if notifying about persistable idle. ---> System.NullReferenceException: Object reference not set to an instance of an object. at System.Activities.Runtime.ActivityExecutor.CancelRootActivity() at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.ScheduleCancel() at Microsoft.Activities.Hosting.HostedWorkflowInstance.Cancel() at Microsoft.Activities.Hosting.WorkflowSession.TryDispatchNextSubroutineWorkItem() at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() --- End of inner exception stack trace --- at Microsoft.Activities.Hosting.HostedWorkflowInstance.NotifyPaused(Exception abortReason) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.AbortInstance(Exception reason, Boolean isWorkflowThread) at System.Activities.Runtime.ActivityExecutor.AbortWorkflowInstance(Exception reason) --- End of inner exception stack trace --- at System.Activities.Runtime.ActivityExecutor.AbortWorkflowInstance(Exception reason) at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.TryPropagateCompletion(HostedWorkflowInstance instance) at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.WorkflowSession.TryStartSubroutine(Guid subroutineId, Guid rootActivityId, Dictionary`2 inArguments, List`1 executionProperties) at Microsoft.Activities.Hosting.WorkflowSession.TryDispatchNextSubroutineWorkItem() at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.Cancel() at Microsoft.Activities.Hosting.WorkflowSession.OnUnhandledException(HostedWorkflowInstance throwingInstance, Exception unhandledException, Boolean isRootFaulted) at System.Activities.Runtime.ActivityExecutor.NotifyUnhandledException(Exception exception, ActivityInstance source) at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.TryPropagateCompletion(HostedWorkflowInstance instance) at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.TryPropagateCompletion(HostedWorkflowInstance instance) at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.WorkflowSession.TryStartSubroutine(Guid subroutineId, Guid rootActivityId, Dictionary`2 inArguments, List`1 executionProperties) at Microsoft.Activities.Hosting.WorkflowSession.TryDispatchNextSubroutineWorkItem() at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.ResumeBookmark(Guid subroutineId, Bookmark bookmark, Object value, Action`1 sessionIdleCallback) at Microsoft.Activities.Hosting.WorkflowSession.ResumeInternal(BookmarkWaiter bookmarkWaiter, Object value, DateTime resumptionTime, Action`1 sessionIdleCallback) at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.ProcessWaiter() at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.Isolate(AsyncCompletion callback, IAsyncResult result) at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.ExceptionHandlingFrame(IAsyncResult result) --- End of inner exception stack trace --- ---> (Inner Exception #0) System.Runtime.CallbackException: The call to OnRequestAbort on the WorkflowInstance '35616a0a-26da-4765-b92d-3f6eb5727508' threw an exception. This is considered fatal. See inner exception for details. ---> System.AggregateException: Should be processing if notifying about persistable idle. ---> System.NullReferenceException: Object reference not set to an instance of an object. at System.Activities.Runtime.ActivityExecutor.CancelRootActivity() at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.ScheduleCancel() at Microsoft.Activities.Hosting.HostedWorkflowInstance.Cancel() at Microsoft.Activities.Hosting.WorkflowSession.TryDispatchNextSubroutineWorkItem() at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() --- End of inner exception stack trace --- at Microsoft.Activities.Hosting.HostedWorkflowInstance.NotifyPaused(Exception abortReason) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.AbortInstance(Exception reason, Boolean isWorkflowThread) at System.Activities.Runtime.ActivityExecutor.AbortWorkflowInstance(Exception reason) --- End of inner exception stack trace --- at System.Activities.Runtime.ActivityExecutor.AbortWorkflowInstance(Exception reason) at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.TryPropagateCompletion(HostedWorkflowInstance instance) at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.WorkflowSession.TryStartSubroutine(Guid subroutineId, Guid rootActivityId, Dictionary`2 inArguments, List`1 executionProperties) at Microsoft.Activities.Hosting.WorkflowSession.TryDispatchNextSubroutineWorkItem() at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.Cancel() at Microsoft.Activities.Hosting.WorkflowSession.OnUnhandledException(HostedWorkflowInstance throwingInstance, Exception unhandledException, Boolean isRootFaulted) at System.Activities.Runtime.ActivityExecutor.NotifyUnhandledException(Exception exception, ActivityInstance source) at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.TryPropagateCompletion(HostedWorkflowInstance instance) at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.TryPropagateCompletion(HostedWorkflowInstance instance) at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.WorkflowSession.TryStartSubroutine(Guid subroutineId, Guid rootActivityId, Dictionary`2 inArguments, List`1 executionProperties) at Microsoft.Activities.Hosting.WorkflowSession.TryDispatchNextSubroutineWorkItem() at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.ResumeBookmark(Guid subroutineId, Bookmark bookmark, Object value, Action`1 sessionIdleCallback) at Microsoft.Activities.Hosting.WorkflowSession.ResumeInternal(BookmarkWaiter bookmarkWaiter, Object value, DateTime resumptionTime, Action`1 sessionIdleCallback) at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.ProcessWaiter() at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.Isolate(AsyncCompletion callback, IAsyncResult result) at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.ExceptionHandlingFrame(IAsyncResult result)<--- data-blogger-escaped-.="" data-blogger-escaped-li="">
   
• Critical Event 10:
  Unhandled exception in AppDomain: Microsoft.Workflow.Common.FatalException: An unrecoverable error occurred. For diagnostic purposes, this English message is associated with the failure: 'The dispatcher failure rate has reached 11%. InnerException is the last failure; see traces for other failures.'. ---> System.AggregateException: The dispatcher failure rate has reached 11%. InnerException is the last failure; see traces for other failures. ---> System.Runtime.CallbackException: The call to OnRequestAbort on the WorkflowInstance '35616a0a-26da-4765-b92d-3f6eb5727508' threw an exception. This is considered fatal. See inner exception for details. ---> System.AggregateException: Should be processing if notifying about persistable idle. ---> System.NullReferenceException: Object reference not set to an instance of an object. at System.Activities.Runtime.ActivityExecutor.CancelRootActivity() at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.ScheduleCancel() at Microsoft.Activities.Hosting.HostedWorkflowInstance.Cancel() at Microsoft.Activities.Hosting.WorkflowSession.TryDispatchNextSubroutineWorkItem() at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() --- End of inner exception stack trace --- at Microsoft.Activities.Hosting.HostedWorkflowInstance.NotifyPaused(Exception abortReason) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.AbortInstance(Exception reason, Boolean isWorkflowThread) at System.Activities.Runtime.ActivityExecutor.AbortWorkflowInstance(Exception reason) --- End of inner exception stack trace --- at System.Activities.Runtime.ActivityExecutor.AbortWorkflowInstance(Exception reason) at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.TryPropagateCompletion(HostedWorkflowInstance instance) at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.WorkflowSession.TryStartSubroutine(Guid subroutineId, Guid rootActivityId, Dictionary`2 inArguments, List`1 executionProperties) at Microsoft.Activities.Hosting.WorkflowSession.TryDispatchNextSubroutineWorkItem() at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.Cancel() at Microsoft.Activities.Hosting.WorkflowSession.OnUnhandledException(HostedWorkflowInstance throwingInstance, Exception unhandledException, Boolean isRootFaulted) at System.Activities.Runtime.ActivityExecutor.NotifyUnhandledException(Exception exception, ActivityInstance source) at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.TryPropagateCompletion(HostedWorkflowInstance instance) at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.TryPropagateCompletion(HostedWorkflowInstance instance) at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.WorkflowSession.TryStartSubroutine(Guid subroutineId, Guid rootActivityId, Dictionary`2 inArguments, List`1 executionProperties) at Microsoft.Activities.Hosting.WorkflowSession.TryDispatchNextSubroutineWorkItem() at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.ResumeBookmark(Guid subroutineId, Bookmark bookmark, Object value, Action`1 sessionIdleCallback) at Microsoft.Activities.Hosting.WorkflowSession.ResumeInternal(BookmarkWaiter bookmarkWaiter, Object value, DateTime resumptionTime, Action`1 sessionIdleCallback) at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.ProcessWaiter() at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.Isolate(AsyncCompletion callback, IAsyncResult result) at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.ExceptionHandlingFrame(IAsyncResult result) --- End of inner exception stack trace --- --- End of inner exception stack trace --- at Microsoft.Workflow.Common.Fx.<>c__DisplayClass2.b__0() at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading.ThreadHelper.ThreadStart().

SharePoint 2013: WARNING: The web application is configured with claims authentication mode

Problem

You perform a migration of your farm from 2010 to 2013; or you perform an update on a SharePoint 2013 farm, installing the usual patches and then running either the configuration wizard or psconfig.  The migration or update completes successfully, but warnings are indicated. When you check the Upgrade Status page in Central Administration, you see that there are some warnings associated with the upgrade operation.  When you then open and review the migration or upgrade log, you see warnings similar to:

WARNING The  [WebAppName] web application is configured with claims authentication mode however the content database you are trying to attach is intended to be used against a windows classic authentication mode [ID]

Solution

• Ignore

References

• Avoid Database Attach Error When Upgrading to SharePoint 2013
• December CU gives warning web application is configured with claims authentication mode however the content database you are trying to attach is intended to be used against a windows classic authentication mode.
• Problems with upgrading claims based site from Sharepoint 2010 to Sharepoint 2013 Preview
• One or more web applications are configured to use Windows Classic authentication (SharePoint 2013)

Notes

• We experienced this warning the first time whilst performing test upgrades of the customer farm from 2010 enterprise to 2013 enterprise.  We performed the migration to claims before performing the migration 2013.  We then attempted to resolve the error by performing the migration to claims after migrating to 2013.  We also tried some other variations on the migration, such as updating the 2010 farm through the latest update, but to no avail: the warning persisted.  Regardless, the warning never appeared to be accompanied by actual performance or function problems. So, we chose to ignore the warning and live with it.
• We next sometimes experienced this warning whilst performing routine updates of the target customer farm (ie, installing PUs, CUs, SPs).  We have upgraded the farm 35 times to date, not always in response to actual software update installations, but sometimes to clean up other issues.  Among these 35 upgrades, this warning has occurred 9 times.  The warning has never been accompanied by observable problems with performance or function of the farm.

SharePoint 2013: How to fully uninstall Workflow Manager 1.0

Introduction

This posting walks through the process of completely removing the Workflow Manager 1.0 subsystem (and related components) from a SharePoint 2013 farm.  All references used in developing this procedure are listed below.

NOTE: completing this process will not remove any 2013 workflows that have been created and deployed to the target SharePoint 2013 farm.  2013 workflows are stored in the appropriate content databases, just as 2010 workflows are.  Restoring your 2013 workflows is easily accomplished using Designer 2013.  See SharePoint 2013: how to restore Workflow Manager 1.0 workflows in the event of a disaster  for details.

Discussion

The Workflow Manager 1.0 subsystem, when fully installed to a SharePoint 2013 farm, consists of the following components:

• Databases:
• SbGatewayDatabase
• SbManagement
• SbMessageContainer01
• WFInstanceManagement
• WFManagement
• WFResourceAutomation
• Software:
• Microsoft Web Platform Installer 4.6
• Service Bus 1.0
• Windows Fabric
• Workflow Manager 1.0
• Workflow Manager Client 1.0
• SharePoint Service Applications
• Workflow Service Application Proxy
• Application Folders
• Application Folder: C:\Program Files\Workflow Manager\
• Registry Keys
• HKLM\SOFTWARE\Microsoft\Workflow Manager
• HKLM\Software\Microsoft\Service Bus
• Certificates
• WorkflowOutbound
• SSL
• Encryption (can be the same certificate as used for SSL)
• AppServerGeneratedSBCA

All of these components, except for the Microsoft Web Platform Installer 4.6, registry keys and certificates, must be removed.  The following procedure shows you how.

Procedure

1. Leave the farm
1. Launch the Workflow Manager Configuration Wizard as administrator.  Only this one option will be listed.
2. Copy the current configuration for later reference.
2. Uninstall Software
1. Workflow Manager 1.0
2. Workflow Manager 1.0 Client
3. Windows Fabric
4. Service Bus 1.0
3. Remove Databases
1. SbMessageContainer01
2. SbGateway
3. SbManagement
4. WFInstanceManagement
5. WFManagement
6. WFResourceManagement
4. Remove Service Application
1. Workflow Service Application Proxy
5. Remove application folder
1. C:\Program Files\Workflow Manager\

References

1. Getting started with SharePoint Server 2013 workflow
2. Install and configure workflow for SharePoint Server 2013
3. Uninstalling/Removing SharePoint 2013 Workflow Manager Farm
4. Uninstall Workflow manager client
5. what is opposite of "Register-SPWorkflowService"???
6. Workflow Manager
1. Configuring Workflow Manager 1.0 using Configuration Wizard
1. Leaving a Farm
2. Configuring Workflow Manager 1.0 using PowerShell
1. Managing Farms and Nodes (Workflow Manager 1.0)
2. Managing Certificates (Workflow Manager 1.0)
3. Installing Workflow Manager 1.0
1. Uninstalling Workflow
7. Workflow in SharePoint Server 2013 FAQ
8. How to uninstall Workflow Manager 1.0 when the command ‘Remove-WFHost’ does not work
9. Workflow Manager Farms for SharePoint 2013 Part One: Core Concepts, High Availability, Certificate and SharePoint considerations
10. Error when running workflow form list item - workflow manager SharePoint 2013
11. Export SharePoint Designer 2013 Workflow XAML
12. Workflow development in SharePoint Designer 2013 and Visio 2013
13. Understanding how to package and deploy workflow in SharePoint 2013
14. Exporting Certificates to Client Machines (Workflow Manager 1.0)
15. SharePoint 2013: how to restore Workflow Manager 1.0 workflows in the event of a disaster

Notes

• Uninstallation took approximately one hour, including taking screen shots and writing notes of the process.
• The uninstallation was undertaken to address an issue that presented after installing the September 2014 CU.  The issue resulted in the Workflow Manager Backend service repeatedly stopping and then restarting.  This seemed to have caused all 2013 workflows to perform sluggishly or not at all.  After working through several possible solutions, the remaining one was to uninstall and then re-install the entire Workflow Manager 1.0 subsystem.  This resolved the problem.

SharePoint 2013: users not receiving email notifications from workflows

Problem

Users complain that they are not receiving email notifications from various workflows, such as ticketing workflows.  You test email alerts and find that you receive them successfully.  You ask other users, and you find that they are also receiving email notifications.  A few other users have also raised complaints about not receiving email notifications.  Some of the users that have raised complaints have both standard and elevated privilege, or administrator domain accounts. 

Your SharePoint farm employs Active Directory authentication.  You recently migrated the farm from 2010 to 2013.  When you check the workflow history, you find messages to the affect that the user could not be found.

Discussion

• Description
• The problem involves SharePoint 2013's new username field: as you type a username into this field, it pops up a list of matching user anames.  These names are those associated with the Display Name field of the user's Active Directory account.
• It so happened that some users with both standard and administrator accounts had identical display names.  Thus, when a user was selected for a list item or workflow, as the user's name was typed in, the first item that popped up was not the standard account but the administrator account, which didn't have an email account configured.
• Solution
• For those users with both standard and administrator domain accounts, ensure that the Active Directory Display Name field provides some indication of which type it is.  For example, you might want to append Admin, OU, LA or some other indicator to the display name of the user's administrator account. This way, when that user's name is entered into a username field, both the user's standard and administrator accounts will appear in the popup, but they will be distinguishable from each other.

References

• TIP: Remove domain from username in SharePoint 2010
• User not receiving Sharepoint emails

Notes

• This was eventually found to be a trivial issue, but when it initially occurred, it had us all scratching our heads.  Ultimately, the cause of the problem was identified through careful and exact observation of how usernames were entered.

SharePoint 2013: Organization Browser does not display

Problem

You have a SharePoint 2013 farm.  Windows 7 is deployed among your users.  Their browsers include IE9 and Chrome.  A few users report that the organization browser does not display.  You verify this experience for a user and then perform the following checks:

• Silverlight is installed
• Silverlight is enabled
• Site is in local intranet zone
• Browser being used is IE
• Zoom level is 100%

Solution

1. Uninstall Silverlight
2. Install Silverlight

References

• IE 9 does not display Organization Browser silverlight version
• silverlight won't load in ie9