SharePoint 2013: The Execute method of job definition Microsoft.SharePoint.Diagnostics.SPDiagnosticsMetricsProvider

Problem

You see the following event occuring daily at about 6:12 AM on your SharePoint 2013 farm servers:

Log Name:      Application
Source:        Microsoft-SharePoint Products-SharePoint Foundation
Date:          [date/time]
Event ID:      6398
Task Category: Timer
Level:         Critical
Keywords:      
User:          [DOMAIN/FarmServiceAcct]
Computer:      [SharePoint Server]
Description:
The Execute method of job definition Microsoft.SharePoint.Diagnostics.
SPDiagnosticsMetricsProvider (ID 9cde39fb-4971-4a03-9612-0978098691d7) 
threw an exception. More information is included below.

An update conflict has occurred, and you must re-try this action. The 
object SPWebService was updated by [DOMAIN/FarmServiceAcct], in the 
OWSTIMER (8696) process, on machine [SharePoint Server].  View the 
tracing log for more information about the conflict.
Event Xml:
...

This error is generated if the Config Refresh timer job finds out-of-sync caches among the SharePoint servers.  System topology includes one application server and two web front end servers.

This posting unfortunately does not present a solution, but documents troubleshooting steps and reference articles for future reference.

Troubleshooting

1. Check cache ID
1. APP1: 2248964
2. WFE1: 2248968
3. WFE2: 2248970

   Note: this value is changing constantly.  if you open each cache.ini file on each machine separately, you may get different values - not because the caches are out of sync but due to the cache ID changing from when you open the file on one machine to when you open it on another machine.  To get an accurate snapshot of this value at any moment in time for all machines, have remote sessions open on all machines simulataneously, and then in quick succession make copies of the file on each one.  Then open these copies to determine the actual cache ID.

2. Clear cache
1. First, stopped Timer service on all SharePoint servers.
2. Then on each server, starting with App1:
1. Navigated to C:\ProgramData\Microsoft\SharePoint\Config.
2. Looked for current GUID folder (check dates).
3. Deleted all XML files in this folder.
4. Replaced contents of cache.ini with "1".
3. Started timer service on APP1
1. Waited for it to rebuild fully.

   Note: if you have security software (McAfee, etc) installed, it will significantly consume resources scanning the creation of all of the new cache files, temporarily adversely impacting performance.

2. 1658 XML cache files generated
4. Started timer service on WFE1
1. Waited for it to rebuild fully.
2. 1658 XML cache files generated
3. Refreshed application logs APP1 and WFE1
5. Started timer service on WFE2
1. Waited for it to rebuild fully
2. 1658 XML cache files generated
3. Verify cache IDs
1. After cache rebuild completed, checked contents of each cache.ini:
1. APP1: 2248998
2. WFE1: 2248998
3. WFE2: 2248998

   See note above on getting accurate values for ID.

4. Verify solution
1. Check event logs one day later: same issue recurring
2. Check event logs five days later: same issue recurring.

Solution

1. None found at this time 12/29/14.

References

• Error con el job SPDiagnosticsMetricsProvider
• Sharepoint 2010 – Clearing the Configuration Cache
• Error message when you try to modify or to delete an alternate access mapping in Windows SharePoint Services 3.0: "An update conflict has occurred, and you must re-try this action"
• SharePoint 2013: Event ID 6398 An update conflict has occurred, and you must re-try this action
• SharePoint 2013 Error 6398 The Execute method of job definition Microsoft.SharePoint.Diagnostics.SPDiagnosticsMetricsProvider threw an exception
• Clear SharePoint Config Cache with PowerShell
• Timer Event 6398 - SPDatabaseServerDiagnosticsPerformanceCounter

Notes

• Also checked development farm servers: APP1: 2733002, WFE1: 2733006 and WFE2: 2733006. After performing the above procedure: APP1: 2733071, WFE1: 2733071 and WFE2: 2733071.  Same experience: issue continues to occur after clearing cache.

SharePoint 2013: Critical 6398: The Execute method of job definition Microsoft.Office.Server.UserProfiles.UserProfileImportJob...

Problem

The following event is occuring every minute in the application event log on one of your farm's application servers:

Log Name:      Application
Source:        Microsoft-SharePoint Products-SharePoint Foundation
Date:          [date/time]
Event ID:      6398
Task Category: Timer
Level:         Critical
Keywords:      
User:          [FarmServiceAccount]
Computer:      [ApplicationServer]
Description:
The Execute method of job definition Microsoft.Office.Server.UserProfiles.
UserProfileImportJob (ID deabe75d-3701-43c1-9968-920febb91024) threw an 
exception. More information is included below.

There was no endpoint listening at 
http://[ApplicationServer]:5725/ResourceManagementService/MEX that could 
accept the message. This is often caused by an incorrect address or SOAP 
action. See InnerException, if present, for more details.
Event Xml:
...

 
Checking the Services control panel on the farm application server, I found that the FIM service had stopped.

Solution

• Start or restart the FIM and/or FIM Synchronization services (whichever is stopped).

References

• Error while creating UPSA Synchronization connection in SharePoint 2013
• There was no endpoint listening at... ResourceManagementService/MEX that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details

Notes

• I checked failed jobs in Central Administration Job History listing during the period that this event was occurring (the error event filled the server's Application log), but was unable to find any failed job that occurred at the same times.  The only user profile jobs having one minute scheduling were
  • User Profile Service - System Job to Manage User Profile Synchronization
  • User Profile Service - User Profile to SharePoint Language And Region Synchronization. 
  However, both these jobs showed Succeed for each occurence in the Job History.  If anyone can provide some helpful insight on this puzzling discovery, I would be grateful.  Was it one of these jobs that was generating the error?  Or does this particular task not generate a failed job entry if it fails?

DocAve Manager 6: The username or password is invalid

Problem

You are installing the DocAve agent to a new node that you want to add to your DocAve management group, and you experience the following error while configuring the agent account:

 

This can occur when the DocAve user agent service account has inadequate permissions on the node you are adding it to. 

Solution

• Add the DocAve user agent service account to the local Administrators group.

References

• DocAve 6 Administrator User Guide (p9)
• DocAve 6 Installation Guide (p22)

Notes

• This is the easiest way to resolve this is error.

SharePoint 2013: a review of all methods and their steps for moving document libraries

Introduction

Sometimes, you need to move a document library to another site, such as an archive site.  This can be done in several ways:

• Save as template using the Include Content option
• Bulk copying using the site collection Content and Structure tool
• Bulk copying via Windows Explorer
• Moving using stsadm
• Moving using PowerShell
• Third party tools

This posting reviews each of these methods from a SharePoint 2013 perspective.

Save as Template using the Include Content option

1. Pros
1. Easy to use.
2. Easy access to template after you're done - it's stored in the site collection template gallery.
3. effectively creates a backup copy.
4. Does not require advanced knowledge of PowerShell.
5. Can be done entirely through the web interface.
6. Preserves all public views.
2. Cons
1. Limited to maximum template size set by administrator.
2. Cannot specify security or version settings.
3. Does not preserve version history.
4. Requires at least Design level permission.
5. No log file generated.
3. Procedure
1. Login to the site using an account that has at least Designer level permissions.
2. Navigate to the target site document library.
3. Click the LIBRARY tab to display the Library ribbon.
4. In the Settings group, click the Library Settings button.
5. In the Permissions Management list group of links, click Save document library as template.
6. Enter desired template and file names, check the Include Content checkbox; and then click OK.  The document library is then stored in the site collection's  List Template gallery. 

   If the list is larger than about 50 MB, you will experience an error.

   

   To resolve this error, increase the maximum list template size.  See the references in this section for details.

7. Click the Settings gear icon, and then click Site Settings.
8. In the Web Designer Galleries group, look for List templates. 

   If you don't see it listed here, be sure you are at the site collection level.  You can check this by looking in the Site Collection Administration group: if you can see the link, Go to top level site settings, click it.

9. Click the link, List templates.
10. Verify that your document library template is listed.  Click the Edit button to modify the template title and filename.
11. Now, navigate to the site where you want the document library moved to. 

    This site must exist in the same site collection as the source document library.  If it isn't you can still move it, but you will need to download the template WSP file from the source site collection list template gallery and then upload it to the destination site collection list template gallery.  The list template gallery is just a document library, so, you know how to do this.

12. Click the Settings gear icon, and then click Site contents.
13. Click the add an app icon.
14. Follow the usual steps for adding a new app (ie, document library), only this time choosing the document library template you created.  This creates the document library along with all its content. 

    Note: the document properties Created by and Modified by will be changed for those users who have since left your organization and no longer have Active Directory accounts.  SharePoint will change these to the user account that performed the Save as Template operation.  If you need to preserve such information, you must either copy it to new, custom properties or use a third party tool  

4. References
1. SharePoint 2010: The list is too large to save as a template
2. What is the Size Limit for Saving a Document Library as a Template (SharePoint 2010)
3. How to use Powershell to set Max template document size
4. Increase Maximum size of Site, List Templates in SharePoint
5. Copy or move a library by using a library template
6. Save, download, and upload a SharePoint 2013 site as a template
7. SPWebService.MaxTemplateDocumentSize property
8. Software boundaries and limits for SharePoint 2013
5. Notes
   1. Default maximum document size template: this would seem to be 50MB, per this Microsoft article: Save, download, and upload a SharePoint 2013 site as a template. However, in actual practice, I have found it to be 52.428800 MB. This value was obtained by executing

      $webservice = [Microsoft.SharePoint.Administration.SPWebService]::ContentService
      $webservice.MaxTemplateDocumentSize

      in an elevated SharePoint PowerShell Management shell on a newly deployed SharePoint 2013 farm for which this setting had not yet been modified.

Bulk Copying/Moving using the Site Content and Structure tool (Site Manager)

1.  Pros
1. Does not require knowledge of PowerShell.
2. Can be done entirely through the web interface.
3. Existing security settings are migrated.
2. Cons
1. Cannot specify security or version settings.
2. Version history migrated inconsistently (some are, some are not).
3. Significant complications if document library employs folders: can only be used to copy subfolder by subfolder.
4. Only works to copy documents from one location to another within the same site collection - does not work for copying documents across site collections.
5. File size limited to whatever is set for the farm.
3. Procedure
1. Login to the source site using an account that has at least Designer level permissions.
2. Navigate to the target site document library.
3. Click the LIBRARY tab to display the Library ribbon.
4. In the Settings group, click the Library Settings button.
5. In the Permissions Management list group of links, click Save document library as template.
6. Enter desired template and file names, and then click OK. The document library is then stored in the site collection's List Template gallery. 

   Do not check the Include Content option.

7. Click the Settings gear icon, and then click Site Settings.
8. the Web Designer Galleries group, look for List templates.

   If you don't see it listed here, be sure you are at the site collection level. You can check this by looking in the Site Collection Administration group: if you can see the link, Go to top level site settings, click it.

9. Click the link, List templates.
10. Verify that your document library template is listed. Click the Edit button to modify the template title and filename.
11. Now, navigate to the site where you want the document library moved to.

    Note: This site must exist in the same site collection as the source document library. If it isn't you can still move it, but you will need to download the template WSP file from the source site collection list template gallery and then upload it to the destination site collection list template gallery. The list template gallery is just a document library, so, you know how to do this.

12. Click the Settings gear icon, and then click Site contents.
13. Click the add an app icon.
14. Follow the usual steps for adding a new app (ie, document library), only this time choosing the document library template you created. This creates an empty document library that has the same configuration as the source library.  You need source and destination libraries in order to use the Content and Structure tool, because this tool only copies content from one location to another.
15. Click the Settings gear icon, and then click Site Settings.
16. In the Site Administration group, click Content and structure.
17. Using the tree navigation at left, select the document library that you want to move.  Once you select it, its contents will appear in the Results panel.
18. From the View dropdown, select a view that presents a simple list (no folders) of all of the documents.  By default, 100 items will be shown.

    Note: if your document library employs folders, for some reason, the Site Content and Structure tool will show these even if you select a previously configured view that specifically excludes them.  For example, if your document library employs folders, you can create a simple list view that presents the documents as a single list without folders, and this view works just fine when you are looking at the contents of the library while connected to the document library's page.  On the other hand, the library folders will appear in the left navigation as subitems under the document library.

19. Check the documents that you want to copy - don't use the Move option or you otherwise will not be able to verify that all of them were copied over. 

    Note: for document libraries employing folders, you will only be able to copy items over subfolder by subfolder.  For example, if two documents lie buried three subfolders deep, it will not work to check these, select another subfolder, checking those, and then selecting Copy from the Actions menu: only the currently visible and checked items will be copied.

20. Once you have select afrom the Actions menu, select Copy.  A popup will appear presenting the usual site collection tree structure.
21. Expand the tree to the empty document library that you created back in steps 14, and then click OK.  The popup will close, and you will see progress information appearing temporarily.  Once done, select the new library and you will see listed the documents that you copied over.
22. Repeat this process for all the documents that you want to copy.
23. At the end of this process, use simple column counts to verify that all documents have been copied over. 
24. Once the copied items have been verified, delete the source list or library.
4. References
1. How to Use the Site Content and Structure Manager in SharePoint 2013
2. SharePoint 2013: Moving Lists using the Site Content and Structure Tool
3. Software boundaries and limits for SharePoint 2013
5. Notes
• I found moving lists this way to be advantageous when needing to preserve list item IDs.

Bulk Copying/Moving via Windows Explorer

1. Pros
1. Simple to use.
2. User permissions in Windows Explorer are consistent with user permissions to document library.
3. Does not require elevated permissions.
2. Cons
1. All version history is lost.
2. All security configuration is lost.
3. Metadata (column) properties may be lost or misinterpreted.
4. Must have Contribute (write) permission to destination library.
5. No log file generated.
3. Procedure
1. Navigate to the source document library.
2. Select the LIBRARY tab.
3. In the Connect & Export group, click the Open with Explorer button.  Leave this open. 

   The open with Explorer functionality requires 32-bit ActiveX.  This button will not be enabled if using 64-bit Internet Explorer, Chrome or other browsers not featuring native 32-bit ActiveX support.  Add-ons are available for some of these other browsers, such as Chrome, that will emulate 32-bit ActiveX.

4. Navigate to the destination document library.
5. Select the LIBRARY tab.
6. In the Connect & Export group, click the Open with Explorer button. Leave this open.
7. Copy documents from the source folder to the destination folder.
8. Once the copied items have been verified, delete the source list or library. 

   You must delete the library through the library settings page.  You cannot delete it through the Windows Explorer window.

4. References
1. Copy or move library files by using Open with Explorer
2. How to use the "Open with Explorer" command and how to troubleshoot issues with this option in SharePoint Online
3. Plan browser support in SharePoint 2013
4. IE Tab Extension for Chrome
5. Notes
• To determine whether Internet Explorer is 32 or 64 bit, select About Internet Explorer from the Help menu.  If the browser is 64-bit, the Version will indicate this.

Moving using stsadm

1. Pros
1. Simpler permissions to use.
2. Finer control over export.
3. Preserves security and most version history.
4. Generates log file for export and import operations.
2. Cons
1. Must be member of local administrator's group.
2. Deprecated.
3. Procedure
1. Open a command shell on a SharePoint farm server.
2. Execute the following command:
    
   stsadm -o export -url "<source library url>" -filename " "<save file path and name>" -includeusersecurity -versions 4.
    
1. Source Library URL is URL seen in the address bar when viewing the library.  So, if the library is in site Marketing and is called Documents, the URL you would enter would be: http:/Contoso/Marketing/Documents.   Note that you don't include the extra part that points to the particular view, such as "Forms/Allitems.aspx."
2. Save File Path and Name is local or network location and file name.
3. In the same browser, execute the following command:
   
   stsadm -o import -url "<destination library url>" -filename "<file path and name>"
    
4. Delete the source library.
4. References
1. Index for Stsadm operations and properties (Office SharePoint Server)
2. Some key differences between stsadm export and backup operations
3. Stsadm to Windows PowerShell mapping in SharePoint 2013
4. Complete reference of all STSADM operations (with parameters) in MOSS 2007 SP1
5. Exporting and Importing site, library or list using STSADM SharePoint Administration tool
6. Export: Stsadm operation (Office SharePoint Server)
5. Notes

Moving using PowerShell

1. Pros
1. Can be scheduled as a task
2. Finer control over export process via parameters
3. Generates log file
2. Cons
1. Requires advanced scripting knowledge.
2. Must be member of SharePoint_Shell_Access role.
3. Procedure
1. Login to any SharePoint server using an Administrator account.
2. Launch an elevated instance of the SharePoint Management Shell.
3. Execute the following:
     Export-SPWeb -Identity "<sourceurl>" -ItemURL "<itemUrl>" -Path "<path>" -IncludeUserSecurity -IncludeVersions "All" -CompressionSize 1000 -Force
1. <sourceurl>: this is the url to the site containing the document library.  If the site containing the library was Marketing, you might enter something like this: http:/SharePointFarm.Com/Marketing/.  That's it.  Nothing else.  Note the concluding forward slash.  I left out one of the begining forward slashes so that it wouldn't be rendered in this page as a hyperlink.
2. <itemUrl>: this is the name of document library and the extra portion containing the view.  For example, if the document library was called Documents, you might enter something like this: Documents/Forms/AllItems.aspx.  Note that it does not start with a forward slash.
3. -CompressionSize 1000 -Force: these last parameters ensure that the export generates a single file.  Note that compression size is in MB.
4. After finishing the export, import the file to its new location by executing the following:
   Import-SPWeb -Identity "<destinationurl>" -Path "<path>", where
• <destinationurl> is the URL to the site to which you want to import the library.  For example, if your site was marketing, it would be htp://contoso.com/marketing/.
• <path> is the path and file name of the list to be imported.  For example, if the file was stored in the Imports folder off the root, it would be: C:\Imports\MyLibrary.cmp.
4. References
1. Import-SPWeb
2. Export-SPWeb
3. Export sites, lists, or document libraries in SharePoint 2013
4. Import a list or document library in SharePoint 2013
5. Cannot find an SPWeb object with Id or Url
6. Get-SPWeb : Cannot find an SPWeb object with Id or Url
7. Software boundaries and limits for SharePoint 2013
8. SharePoint 2013: To export a site, list, or document library
9. Export/Import-SPSIte :- What is the max size limit that can be moved using this command??
10. Use Windows PowerShell to administer SharePoint 2013
5. Notes

Using Central Administration

1. Pros
1. Finer control over export process via parameters
2. Can be performed online using simple web interface
3. Doesn't require advanced knowledge
4. Generates a log file.
2. Cons
1. Requires user to be a member of the farm administrator's group
2. Can only be used to export list - cannot be used to import a list.
3. No control over other parameters, such as compression or multiple files
3. Procedure
1. Launch Central Administration
2. Go: Backup and Restore > Granular Backup > Export a site or list
3. Configure as appropriate
4. Click the Start Export button. 

   If a local URL is entered (eg, D:\test.cmp), the file will be saved to the farm application server that is hosting Central Administration.

5. To then import this, you must use PowerShell or stsadm. 
4. References
1. Export sites, lists, or document libraries in SharePoint 2013
2. Software boundaries and limits for SharePoint 2013
3. SharePoint 2013: To export a site, list, or document library
5. Notes

Third Party Tools

1. Pros
1. Can preserve all version history and security information
2. Don't require advanced technical knowledge
2. Cons
1. Legal and regulatory Import/Export considerations (if vendor is foreign to your country)
2. Costs
3. Listing
1. SharePoint List Transfer
2. CopyMove for SharePoint 2013
3. Content Matrix
4. ShareGate
5. Kroll Ontrack PowerControls
4. Notes

SharePoint 2013: Cannot find an SPWeb object with Id or URL

Problem

You've logged into a farm server with the SharePoint Setup User Administrator account.  You've opened a SharePoint management shell with elevated privileges.  You are attempting to import a document library using the SPWeb object but experience this error:

Import-SPWeb : Cannot find an SPWeb object with Id or URL : <URL>...

This can occur when the account you are logged in as has not been added to the SharePoint_Shell_Access role of the content and/or farm configuration databases.  Generally, you want your administrator account to be added to this role for all farm content databases and the farm configuration databases so that you can manage them from a PowerShell.

In the following procedure, you will first get the GUIDs associated with the content databases.  Then, you will check each content database and the farm configuration database for the accounts that have had the SharePoint_Shell_Access role added to them.  Lastly, you will add the role to the desired account for each database.

Solution

1. Note down the following information:
1. The account you used trying to perform the import that resulted in the error (call it User.Account)
2. The name of the content database containing the site into which you wanted to perform the import (call Content_A).
2. Login to a SharePoint farm server using the SharePoint Setup User Administrator account.
3. Launch an elevated instance of the SharePoint 2013 Management Shell.
4. Execute Get-SPContentDatabase.  This returns a list of all farm content databases, along with summary information and including the database GUID.
5. Note down the content databases and their GUIDs.
6. Execute Get-SPShellAdmin <GUID> for each GUID noted in step 5).  This will return a list of all user accounts added to the SharePoint_Shell_Access role for that database.
7. Note down where the account in step 1) above is missing from the role.
8. Execute: Get-SPShellAdmin.  This will return a list of all user accounts added to the SharePoint_Shell_Access role for the farm configuration database.
9. Execute: Add-SPShellAdmin -UserName DOMAIN\User.Account <GUID> for each content database identified in step 5). Executing this command adds this role for both the content database and for the farm configuration database. 

   If you experience an error trying to add this role to that account, perform the steps in this reference.

10. Repeat step 6) - 8) to verify that the SharePoint_Shell_Access role has now been added to the desired account.

References

• SharePoint 2013: Cannot add to the SharePoint_Shell_Access role
• Add-SPShellAdmin
• Get-SPShellAdmin
• Set Privileges for Using PowerShell to Manage SharePoint
• Plan for administrative and service accounts in SharePoint 2013
• Initial deployment administrative and service accounts in SharePoint 2013

Notes

• If you experience an error trying to add this account to that role, perform the steps in this reference.

SharePoint 2013: How to create a cross-site lookup column using custom site columns

Problem

I needed to create a list-based, work tracking system for my IT team.  This list would need to link to the IT helpdesk ticketing list used by the general user base.  The ticketing list was in a site visible to the general user, while the work tracking list would be in a subsite visible only to the IT team.  Column lookups only work for lists within the same site.  The challenge then was to expose a list as a lookup to a subsite.

This challenge can be resolved by using custom site columns.  Custom site columns created at the site collection level are available to all sites within the site collection. Custom site columns created at the site level are available within that site and within its subsites.

In this walkthrough, I'll create a custom column at the site level, since this column won't be needed outside of the parent site.  This list will then be exposed as a lookup to another list contained within a subsite of the original list.

Procedure

1. Logged in as a site collection administrator, navigate to the parent site containing the ticket list.
2. Go: Settings > Site Settings > Web Designer Galleries > Site columns.
3. Click Create.
4. Enter a name for the new site column.  For this walkthrough, I'll call it ServiceDeskTickets.
5. Select Lookup (information already on this site) as the type.
6. Select a group to categorize this site column or enter a new group name.  For this walkthrough, the group will be Custom Columns.
7. Configure the next two settings as desired.  Next comes the critical part.
8. From the Get information from dropdown, select the list you want to expose, in this case the service desk ticket list. 

   If you don't see your desired list here, this is probably because you did not go to the right site before clicking Site columns - or you may be at the root site.  Just cancel out of everything, and then start over. 

   After you make your selection, the page refreshes, updating the various options and data listed in the Additional Column Settings group.
9. Then from the In this column: dropdown, select the primary field from this list that you want to expose. For my needs, I chose the ticket list ID, since this would lead to a nicely formatted dropdown to select from in the subsite list.
10. Select any additional list columns that you want to also be exposed to the subsite. 

    You may not see all of the source columns listed here. This is because certain column types cannot be used to create lookup column types. See this reference for additional details on what column types can and can't be exposed in this way.  There are workarounds for still exposing some of these missing columns if you encounter this.

    .
11. Click OK. 
12. Navigate to the subsite, and then to the target list in which you want to expose the cross-site lookup.
13. On the list's LIST ribbon, click List Settings.
14. Scroll down and then click Add from existing site columns.
15. From the Select site columns from dropdown, select Custom Columns.
16. In the Available site columns multi-select box, select ServiceDeskTickets, and then click OK.  The List Information page is displayed again, and you can now see all of the lookup columns that were added to the target list.

    To change the name of any lookup column to something more user-friendly, just click on it.  To change what additional lookup columns will be added, just click on the primary lookup column: this will take you to a page where you can customize what additional lookup columns will be added along with the primary lookup column.

17. You're done.

References

• Creating a cross site lookup field without code
• Using Site Columns in Sharepoint 2013
• Plan content types and workflows in SharePoint 2013
• Projected Lookup Field - Lookup Column Additional Fields Missing
• Content Type or Lists Lookup Column can't show additional fields which are Choice or lookup columns
• Content Type enforcing foreign key relationship
• Create list relationships by using unique and lookup columns

Notes

• Many thanks to Bram Nuyts for presenting the original critical steps necessary for implementing cross-site lookup columns.
• To expose a site list to a sister site or site higher in the site hierarchy, you must expose the source list as a site column configured at the site collection level.
• Working with Custom Content Types
• Custom content types added to the target list may not include additional lookup columns that you create later. 
• Using Additional Columns: 
• when you define a site or site collection column based upon a column in an existing list, you have the option of also making other columns available in that list as additional columns.  However, not all list columns will be usable in this way, from my experience.  For example, if you have custom columns, lookup columns, etc in the list, these will not be available (or even visible) when you select additional columns during the site column definition process.  Neither is the Choice column.
• There appears to be some disconnect within Microsoft articles on this very issue.  If you read the Microsoft article on this topic for 2013, you get one list of columns that can be used as additional columns.  However, if you read the earlier Microsoft article on this topic for 2010, you get a significantly reduced list of columns.  From my experience, Microsoft's 2010 version of this article is the more accurate one.

SharePoint 2013: Error loading navigation: TermStore not found - Common solutions

Introduction

In this posting, I present two causes of this problem, that I have experienced, and the solutions I implemented that resolved them.  The two causes involve minor interruptions to network connectivity and server restart.

Network Connectivity Interruptions

Problem

Your network experienced some very minor, very temporary interruptions in service.  Afterwards, some users were unable to connect to the SharePoint 2013 farm and others experienced an error in the global navigation bar:

Error loading navigation: TermStore not found (Correlation ID:...)

Note that a correlation ID appears with this error.  Checking farm health reports, a rule failure of the Security Token Service on WFE2 was found.  However, on investigation, all services were found to be running on this server, and all IIS application pools were found to be started.  Review of the WFE2 error logs found application errors involving claims authentication.  The server was up and IIS and all other services were found to be running.

Solution

• Restart the SecurityTokenClaimsApplicationPool.

Notes

• NLB was configured with WFE2 as primary and WFE1 as secondary.  Those users routed to WFE2 experienced failure connecting to the farm, while those routed to WFE1 just experienced the navigation error.  WFE2 eventually recovered partially on its own, but the navigation issue persisted.
• My sequence of troubleshooting steps for this issue:
1. Review farm health reports
2. Review server event logs
3. Check services on all farm servers
4. Check application pools on all farm servers

Server Restart

Problem

The farm servers are restarted.  Afterwards, users connecting to their site see the following error message displayed in the global navigation bar:

Error loading navigation: TermStore not found

Note that no correlation ID appears in this case for the error, unlike the earlier example.  Checking farm health reports, you find NO failure of the Security Token service on any any web front end. Checking IIS, you find that all application pools and sites have started.  Checking server services, you find that both FIM service, and the Search (OSearch) and Timer services, are in the stopped state.

Solution

• Start the Timer service.

Notes

• None

DocAve 6 Administrator: An error occurred while loading the node. You may not have enough permissions.

Problem

You administer a SharePoint 2013 farm.  You are working in DocAve 6 SP4 Administrator for SharePoint 6.  You want to view the SharePoint topology and are preparing to select the scope in the Scope panel at left.  You click on the top farm heading to expand it, and this reveals all of the web applications in your farm.  Next, you click on one of the web applications to expand it.  When you do this, an error message is displayed:

You also notice that the node does expand, but that not all of the site collections under this node are listed.

This can occur when the DocAve agent service account is not properly mapped to one or more web application content databases.

Solution

1. Launch SQL Server Management Studio as administrator.
2. Expand Security\logins.
3. Double-click the DocAve agent service account.
4. Select the User Mapping page.
5. Scroll down the list of databases until you find the web application content databases.
6. Verify that the DocAve agent service account is mapping as db_owner to each of these.  If it isn't, make it so.
7. Click OK.

References

• DocAve 6 Administrator User Guide (p10)
• DocAve 6 Installation Guide (p9)

Notes

• None

Windows Server 2012: The parameter is incorrect

Problem

You attempt to change the properties of a Windows event log, but every time, after clicking OK, a prompt appears, stating, "The parameter is incorrect."  Though these settings are enabled and editable, they are controlled by GPO.

Solution

1. On the problematic machine, launch an elevated command prompt.
2. Execute the command Wevtutil using the appropriate parameters. For example, to set the System log to Overwrite events as needed, you would execute the following:

   wevtutil sl System /rt:false /ab:false

References

• wevtutil

Notes

• The change is implemented immediately.

SharePoint 2013: how to change the document library URL

Introduction

Sometimes, you want or need to change a document library or list URL.  This can't be done using any feature available through the document library or list Settings page.  It can be done using SharePoint Designer.  This posting shows you how.  This posting is valid whether you are using SharePoint 2007, 2010 or 2013.  The same steps apply.

Procedure

1. Launch SharePoint Designer.
2. Connect to the target site.
3. In the Navigation panel, click All Files.
4. Scroll down to the list or library.
5. Right-click the list or library, and then select Rename.
6. Enter the URL name for the list or library, and then press the ENTER key. 

   Once you press Enter, the URL name will be changed to what you entered, AND the list or library title will also be changed to the same thing.  That's OK: you can change the title later - but don't do it here.

7. Click on the new  name that you entered.
8. In the List Information group, look for the Name field, and then click it.
9. Edit as desired, and then press the ENTER key. 

   Though it states Name, it actually only changes the Title of the list or library.

References

• EndUserSharePoint.com: How can I change the name of an existing Document Library

Notes

• After changing the document library or list URL, it may take a few minutes for this changed to be propogated to your site navigation, whether managed or structural.  Don't worry: you may get Page not Found errors displayed in the browser if you click on the quicklink to the library.  Just wait a minute or two for navigation to be updated throughout the site.

SharePoint 2013: Cannot add to the SharePoint_Shell_Access role

Problem

You attempt to add a new SharePoint Management Shell administrator using the Add-SPShellAdmin commandlet, while logged in using an account that is a member of the local Administrator group.  Every time you attempt to run the commandlet, you experience the following error message presented in the shell:

add-spshelladmin
"Cannot add <username> to the SharePoint_Shell_Access role of the database <farm configuration database>.  A possible cause of this error is that the account name was already added to the database as a login using a different user name than the account name."

In SQL Server Management Studio (SSMS), you check the login properties of the account and verify that it is currently mapped as DBO to the database.  Through SSMS, you try to uncheck the role membership, but experience an error.  You use the usual sp_changedbowner procedure, but without success. 

Solution

1. Run SSMS as administrator.
2. Right-click on the farm configuration database, and then choose Properties.
3. Select the Files page.
4. In the Owner field, change this to sa, and then click OK.
5. Exit SSMS.
6. Logon to any SharePoint server using the SharePoint Setup User Administrator account.
7. Launch a SharePoint Management Shell as administator.
8. Re-execute the Add-SPShellAdmin commandlet.

References

• Add-SPShellAdmin
• Add-SPShellAdmin–user cannot exist in database
• sp_changedbowner (Transact-SQL)
• Cannot unmap login from db_owner role
• Cannot find an SPWeb object with Id or Url

DocAve 6 Manager for SharePoint 2013: Installation Guide

Introduction

This posting walks through the process of preparing for and installing DocAve 6 Manager for SharePoint 2013.  Preparing for DocAve Manager involves configuring account permissions in three areas: local, SQL and SharePoint; and adding the Web Server Role to the prospective server to be used for hosting DocAve Manager.  This guide presents steps for installing to Windows Server 2012, on Hyper-V 2012, where servers are configured with both external and private networking.

Step 1: Identify Resources

1. Identify the server that you want to use as the DocAve Manager management server.
2. Verify that this server meets the following technical requirements:
1. 54-bit, 4 cores
2. 4 GB RAM
3. 60 GB HD

Step 2: Create new domain account for DocAve Manager agent service

1. Create new domain account for DocAve SharePoint Agent, for example, spDocAve

Step 3: Provision DocAve agent service account

1. Provision local permissions for account
1. Add the DocAve agent service account to local Administrators group on each farm server.
2. Provision SQL Server permissions
1. Grant dbcreator and securityadmin roles on farm SQL Server
2. Grant db_owner role on all farm databases
3. Provision SharePoint permissions
1. Add to Farm Administrators group
1. Central Administration > Security > Manage the farm administrators group
2. Grant Full Control and Operates as System account all web applications
1. Central Administration > Application Management > Manage Web Applications
3. Grant Create Personal Site, Follow People and Edit Profile and Use Tags and Notes permissions to the User Profile Service Application.
1. Central Administration > Application Management > Mange Service Applications > User Profile Service (click on it) > People > Manage User Permissions
4. Grant Term Store Administrator to Managed Metadata Service
1. Central Administration > Application Management > Mange Service Applications (select, don't click) > Administrators
5. Grant Full Control to Search Service application
1. Central Administration > Manage Service Applications > Search Service (select only) > Administrators

Step 4: Configure Firewall

1. If your farm has both external and private networks (with private unrestricted), configure exception for port 14000 on external NIC to enable management interface access.
2. If the farm has only one network, need to configure port exceptions as indicated in DocAve documentation.

Step 5: Configure Server

1. Roles
1. Add Web Server (IIS)
1. Application Development
1. ASP.NET 3.5
2. .NET Extensibility 3.5
3. ISAPI Extensions and ISAPI Filters
2. Management Tools
1. IIS Management Console
2. IIS6 Management Compatability
3. IIS6 Metabase Compatability
2. Features
1. .NET Framework 3.5.1
2. HTTP Activation
3. Non-HTTP Activation
4. WCF Services
5. TCP Port Sharing (and started)
6. Common HTTP Features
1. Static Content
2. Default Content
7. PowerShell 2.0 or greater
3. Services
1. Windows process Activation Service
1. Started
2. Process Model, .NET Environment and Configuration APIs installed
2. WWW Publishing Service
1. Started
3. IIS Admin Service
1. Started

Step 6: Install DocAve Manager

1. login to the prospective DocAve management server using the SharePoint Setup User Administrator account (eg, spAdmin).
2. Download the DocAve Management software (zip), and save locally.
3. Extract locally.
4. Right-click Setup.exe and run as administrator.

Step 7: Intall DocAve Agent

1. Login to a SharePoint farm server using the SharePoint Setup User Administrator account (eg, spAdmin).
2. Verify that your designated DocAve agent service account has been added to the local Administrators group.  If it's not, add it.
3. Download the DocAve Management software (zip), and save locally.
4. Right-click Setup.exe and run as administrator.

Step 8: Setup Administrator Access to DocAve (for NTLM)

1. Launch the DocAve management interface.
2. Login (default: admin/admin).

   OOTB, there is a single local administrative account, admin, that enables you to access the DocAve Control Panel. To add domain accounts, you must first configure DocAve to recognize NTLM authentication.

3. Click a Control Panel link.
4. In the Authentication Manager group, click Authentication Manager.
5. On the Ribbon, in the Integration group, click Windows Authentication.
6. From the Authentication Type dropdown, select NTLM.
7. Click OK.
8. On Control Panel, in the Account Manager group, click Account Manager.
9. On the Ribbon, in the View group, click Users.
10. On the Ribbon, in the Actions group, click Add User.
11. From the User Type dropdown, select Windows User/Group Name.
12. In the Windows User/Group Name box, enter a user account as DOMAIN\UserName.
13. In the Group area, select the Add user to administrators group radio button.
14. Click OK.

References

• TCP Port Sharing and Web Server(IIS) Under Application Server in Windows Server 2012
• IIS Admin Service ‘missing’ in Lync

Notes

• The Installation UI is buggy.  You'll find that if you use the TAB key, the cursor focus can disappear, even when tabbing from one adjacent field to the next.  It's also possible to put the installation wizard into a permanently ambiguous state, from which the only exit is to terminate it and start over.
• Invalid username/password.  I found that during the intsallation, at the Control Service Configuration panel, after entering the username and password I wanted to use for a new application pool, I kept experiencing invalid username/password errors.  On a hunch, I added the DocAve agent service account to the local Administrators group and then tried again: success. This is mentioned in the DocAve Installation guide at page 23, first sentence.
• Installation Rule Scanning: Web Server (IIS) Role: Failed.  If you see this rule violation,
  
  it is likely due to not installing the IIS 6 Management Compatibility and the ASP.NET 3.5 features, which are not default role features installed automatically.  This rule violation will be accompanied by the IIS Admin Service rule violation.  This is because the IIS Admin Service is only installed when IIS 6 compatibility features are installed.  It's easy to check.  Just launch the Add roles and features tool.  Then expand Web Server (IIS) > Web Server > Application Development.  ASP.NET 3.5, ISAPI Extensions and ISAPI Filters won't be installed. Checking the box for ASP.NET 3.5 will automatically also select the other two.  Next, expand Web Server (IIS) > Web Server > Management Tools > IIS 6 Management Compatibility:
  
  Check the box for IIS 6 Management Compatibility; then complete installation.
• Page Not Found!!  If you see this when you fire up the DocAve UI for the first time, don't panic. Check the IE Enhanced Security Configuration setting.  If it's enabled, this will block the UI scripts from executing.  Disabled this setting and try again.

SQL Server 2012: Detection of product..., feature SQL_Tools_ANS failed during request for component

Problem

You manage a SharePoint 2013 farm having a single SQL Server 2012 instance.   You see the following warnings appear in the SQL Server application log:  
 

Log Name:      Application
Source:        MsiInstaller
Date:          [Date/Time]
Event ID:      1001
Task Category: None
Level:         Warning
Keywords:      Classic
User:          SYSTEM
Computer:      [SQL Server 2012]
Description:
Detection of product '{A7037EB2-F953-4B12-B843-195F4D988DA1}', feature 
'SQL_Tools_ANS' failed during request for component 
'{6E985C15-8B6D-413D-B456-4F624D9C11C2}'

Log Name:      Application
Source:        MsiInstaller
Date:          [Date/Time]
Event ID:      1004
Task Category: None
Level:         Warning
Keywords:      Classic
User:          SYSTEM
Computer:      [SQL Server 2012]
Description:
Detection of product '{A7037EB2-F953-4B12-B843-195F4D988DA1}', feature 
'SQL_Tools_ANS', component '{0CECE655-2A0F-4593-AF4B-EFC31D622982}' 
failed.  The resource '' does not exist.

This warning does not appear to be associated with any observable adverse impact to the farm or its capabilities.

Solution

1. As of the date of this posting, install SQL Server 2012 SP2. 

   This service pack is a servicing-only release.  It is also a rollup of all CUs that have been published in SP1 through SP1 CU9 and thus includes the hotfix previously published that fixes this specific problem.

References

• Error: Detection of product ‘{}’, feature ‘SQL_Tools_Ans’ failed during request for component
• KB2793634: Windows Installer starts repeatedly after you install SQL Server 2012 SP1
• After applying Service Pack 1 for SQL Server 2012 you may encounter a known issue! Details inside…..
• Microsoft® SQL Server® 2012 Service Pack 2 (SP2)

 Notes

• UPDATE 2/3/15: issue has not appeared to date.  Confirming resolution through installation of SP2.

Performance Monitor: How to resolve the task is disabled error when starting a data collector

Problem

You are attempting to configure Windows Server 2012 Performance Monitor to collect data in order to analyze CPU and memory load performance of a SharePoint 2013 web front end.  In Performance Monitor, you select the default Server Manager Performance data collector set, and then click the Start button.  You then experience the error message:

Solution

1. Launch Task Scheduler using an account having local Administrator permissions.
2. In the left task pane, expand the tree like so: Task Scheduler Library > Microsoft > Windows > PLA.
3. Select the PLA group.  Note that the status of the Server Manager Performance Monitor is Disabled.
4. In the right Results pane, right-click Server Manager Performance Monitor.  A menu appears
   
   
5. Click Enable.  Note that now the status of the Server Manager Performance Monitor item is Ready:
   

References

• Error enabling Data Collector Set on Windows 2012

Notes

• Thanks to Happy SysAdmin for first helping me understanding this issue.

SharePoint 2013: How to create a global managed navigation structure using the Managed Metadata service application

Introduction

This posting walks you through the process of creating a managed global top navigation structure for your farm.  You will be using the Managed Metadata service application to construct a term set that will be used for site navigation, in which each term is associated with a specific URL.  The term set will be constructed at the farm level, in Central Administration, and will thus be available to any site collection in the farm through linked cloning.

Note that this procedure presented here is effective for farms containing several site collections, but would be impractical for farms containing many site collections.

Procedure

1. Create new term store group
1. Launch Central Administration as a farm administrator
2. Go: CA > Application Management > Manage service applications > Managed Metadata Service
3. In the left Term Store panel, hover your cursor over Managed Metadata Service, and then click the drop-down.  One menu option will be listed: New Group.
4. Click this option.  A new orange bar will appear just below the System group.
5. Enter a name for this new term store group.  For example, Global Top Navigation.
6. Once you have entered the name, click anywhere on the white space in the General panel.  The General panel updates to display fields for customizing this group.
7. Enter a description for the new term store group.  For example, Farm top navigation.
8. Enter the AD accounts of those persons who you want to manage this group and those who will be able to contribute to it.
9. Click Save.  This completes the steps for creating the new term store group.
10. If you want to start over, just right click on this group name, and then choose, Delete Group.
2. Create new term set
1. Hover your cursor over the new term store group, and then click the dropdown arrow.  Three options will be displayed:
1. New Term Set,
2. Import Term Set and
3. Delete Group
2. Click New Term Set.  An orange bar will appear just below the new term store group.
3. Enter a name for the new term set.  For example, Top Navigation Farm.
4. Click on the white space in the General tab.  The General tab will update to display fields for customizing the new term set.  You'll also see three additional tabs:
1. Intended User
2. Custom Sort and
3. Custom Properties
5. On the General tab, enter the information as appropriate.
6. Click Save.
7. Select the Intended tab.
8. Uncheck Available for Tagging.
9. Check Use this Term Set for Site Navigation.
10. Click Save.  This completes the steps for creating a term set.

    A Term Set, when used for navigation, corresponds to an entire menu set - in other words, all of the various menus that would appear in the page's global navigation bar.  Reserve this first term set as the baseline and source.  Create additional term sets that link to this source and that are individually used for each site collection.  One term set can be used for one site collection.  But one term set can be used as the source for however many other term sets.

3. Create new top menu terms
1. Hover your cursor over the new term set, and then click the dropdown arrow.  Several options will be displayed:
1. Create Term
2. Copy Term Set
3. Reuse Terms
4. Pin Term with Children
5. Move Term Set
6. Delete Term Set
2. Click Create Term.  An orange bar will appear just below the new term set.
3. Enter a name for the new term.  For example, Home.
4. Click on the white space in the General tab. The General tab will update to display fields for customizing the new term.  Three other tabs will also appear:
1. Navigation
2. Term-Driven Pages
3. Custom Properties
5. Select the Navigation tab.
6. In the Visibility in Menus section, check both options.
7. In the Navigation Node Type section, select Simple Link or Header, and then enter the URL for the menu option.
8. Click Save.
9. Repeat this process for each of the top menus that you want to appear in the global navigation bar. At the end of this, you will have a list of top menu terms appearing below Top Navigation Farm.  For example, this is what you might see, when you look in the Taxonomy Term Store panel:
1. Global Top Navigation
1. Top Navigation Farm
1. Home
2. Products
3. Resources
4. Support
5. About
10. After you have created all the top menu terms, select the new term set.  The right panel updates to display the usual tabs.
11. Select the Custom Sort tab.
12. In the Custom Sort Order section, select the Use custom sort order option.  A list will appear corresponding to the top menu terms that you created.
13. Using the appropriate sort order dropdowns, set the order of these as desired.  This is the way they will appear in the global navigation bar.
14. Click Save.
4. Create menu option terms
1. Hover your cursor over a new term, and then click the dropdown arrow. Several options will be displayed:
1. Create Term
2. Copy Term
3. Copy Term with Children
4. Reuse Terms
5. Pin Term with Children
6. Merge Terms
7. Deprecate Term
8. Move Term
9. Delete Term
2. Click Create Term.
3. An orange bar will appear just below the new term.  Enter a name for the new term. For example, if you created it under Products, it might be ProductA.
4. Click on the white space in the General tab. The General tab will update to display fields for customizing the new term. Three other tabs will also appear, just like previously.  Indeed, from here on out, as you create new terms and subterms, etc, the tabs that you see will remain the same.
5. Select the Navigation tab.
6. In the Visibility in Menus section, check both options.
7. In the Navigation Node Type section, select Simple Link or Header, and then enter the URL for the menu option.
8. Click Save.
9. Repeat this process for each of the terms that you want to see appear when the user clicks on the top menu term.  For example, if you created several menu option terms under the Products top menu term, you might see this:
1. Global Top Navigation
1. Top Navigation Farm
1. Home
2. Products
1. ProductA
2. ProductB
3. ProductC
3. Resources
4. Support
5. About
10. After you have created all the menu option terms, select the parent top menu term. The right panel updates to display the usual tabs.
11. Select the Custom Sort tab.
12. In the Custom Sort Order section, select the Use custom sort order option. A list will appear corresponding to the menu option terms that you created. 

    NOTE: by default, these items will be sorted alphabetically ascending.

13. Using the appropriate sort order dropdowns, set the order of these as desired. This is the way they will appear in the global navigation bar, after the user clicks on the top menu item.
14. Click Save.
15. Repeat this process for each of the other top menu terms.  At the end of all this, you might have something like the following:
1. Global Top Navigation
1. Top Navigation Farm
1. Home
2. Products
1. ProductA
2. ProductB
3. ProductC
3. Resources
1. ResourcesA
2. ResourcesB
4. Support
5. About
1. Help
2. Contact
3. Comment
16. This completes creating the Top Navigation Farm term set.  This term set will not actually be linked to the global navigation of any one site collection.  Rather, it will be the template that other term sets will link to.  In the next step, you will create a new term set, and this term set will be one that you can use for the managed navigation of a single site collection.  This term set implements the term set template you created previously using something called "pinning", which is binding a term to another term so that the first takes the value of the second.  You can't pin term groups or term sets.  But you can pin terms.  Thus, to create a clone term set from the template, you effectively have to create a new term set and terms, but then pin each of these terms to their corresponding term in the template.
5. Create first site collection Top Navigation term set
1. Hover your cursor over the template term store group, and then click the dropdown arrow.
2. Click New Term Set. An orange bar will appear just below the new term store group.
3. Enter a name for the new term set.  Let's call it, Top Navigation Site Collection A.
4. Click on the white space in the General tab. The General tab will update to display fields for customizing the new term set.
5. On the General tab, enter the information as appropriate.
6. Click Save.
7. Select the Intended tab.
8. Uncheck Available for Tagging.
9. Check Use this Term Set for Site Navigation.
10. Click Save.  Now lets add terms to it. This will proceed a bit differently, as now we want to only "pin" existing terms to it, not create new ones.
11. Hover your cursor over Top Navigation Site Collection A, and then click the dropdown arrow.
12. Select Pin Term with Children.  A popup dialog appears listing the current contents of the Taxonomy Term Store.
13. Expand the Global Top Navigation group.  You will see Top Navigation Farm and Top Navigation Site Collection A.
14. Expand Top Navigation Farm.  You will see the terms you created previously.
15. Select the Home term, and then click OK.  The popup closes, and now, in the Term Store, you will see it updated to display something like this:
1. Global Top Navigation
1. Top Navigation Farm
2. Top Navigation Site Collection A
1. Home
16. Repeat this process for each of the other terms.  At the end of this, you will see something like the following:
1. Global Top Navigation
1. Top Navigation Farm
2. Top Navigation Site Collection A
1. Home
2. Products
3. Resources
4. Support
5. About
17. Note that each of these pinned terms also has presents the children you configured for that term previously.  In fact, any data change that you now make to the template term set will be reflected in this new term set.  For example, if you changed the URL for the Home term in Top Navigation Farm, that change will instantly be reflected in the Home term for Top Navigation Site Collection A.  And so on.  Top Navigation Site Collection A is effectively a linked clone of Top Navigation Farm. Next, let's configure the term sort order.
18. Select Top Navigation Site Collection A.
19. Select the Custom Sort tab.
20. Configure the term order as desired, and then click Save.
21. Repeat this process, creating a new site collection term set for each of the site collections in your farm.  At the end of all this, you might have something similar to this:
1. Global Top Navigation
1. Top Navigation Farm
2. Top Navigation Site Collection A
3. Top Navigation Site Collection B
4. Top Navigation Site Collection C
22. Each of these site collection terms sets is now ready to be associated with a single site collection.
23. Exit the Managed Metadata Service Application.
6. Configure a site collection to use a Top Navigation term set
1. Connect to a farm site collection, let's call it, Site Collection A, using an account provisioned as a site collection administrator.
2. Click the gear icon, and then click Site settings.
3. On the site collection settings page, in the Look and Feel section, click Navigation.
4. On the Navigation settings page, in the Global Navigation section, select Managed Navigation.
5. In the Managed Navigation: Term Set section, select the Site Collection A term set.
6. Scroll to the top or bottom of the page, and then click OK.  The change to global navigation will be instantaneous.
7. Repeat this process for each of the other site collections, assigning it its appropriate term set having a similar name.  At the end of all this, you will have configured consistent global navigation for all your site collections using term sets cloned and now managed from a single template term set.

References

• TechNet: Managed metadata and navigation in SharePoint 2013
• TechNet: Managed navigation in SharePoint 2013
• Navigation - Using Managed Metadata Feature of SharePoint 2013
• Configuring the SharePoint 2013 Managed Metadata service for Navigation
• Managed Navigation in SharePoint 2013

Notes

• Error loading navigation: The Managed Navigation term set is improperly attached to the site: Ocassionally, while editing the Managed Navigation structure, you may see this error appear where the global navigation bar should be when refreshing the user pages after a term set change.  To resolve this, try switching the site navigation back and forth between managed and structured navigation, a couple of times.  My experience has been that about 2-3 tries usually resolves this problem. 
• How to tell in Term Store, when a navigation term set is in use: its font will be slightly larger and the font color will be gray.
• Many thanks to Mythili Ranganathan for her excellent posting on this topic that provided me with the critical initial introduction to implementing managed navigation in SharePoint 2013.  My posting builds upon her foundational writing.

SharePoint 2013: Event 1000: Faulting application name: Microsoft.Workflow.ServiceHost.exe

Problem

After installing the September 2014 PU to your SharePoint 2013 farm, users began to report that their 2013 workflows are performing sluggishly or not at all.  No users reported any adverse impacts to 2010 workflows.  Checking the Workflow Manager 1.0 services, found that all of the services were running except for the Workflow Manager Backend service, which repeatedly stopped and then restarted. Checking the Windows Server Application log, found these error events recurring frequently: 1000, 1026 and 7031 about every 50 seconds or so.  Checking the Microsoft Workflow Operational log, found a regular cycle of error and critical events: 358, 359, 19, 10 and 358 also occurring every minute or so.  This article captures the information and data examined and the efforts undertaken to resolve this issue.

UPDATE (12/03/14): This error was originally  believed to be due to a patch installation.  These errors occurred again six weeks later, this time without there having been any patches, updates or other software installed.  The errors were eventually traced to a workflow design issue.  Removing the running workflow instances resolved the errors.
 
Troubleshooting

1. Checked Workflow Manager services and related services:
• Service Bus Gateway
• Service Bus Message Broker
• Windows Fabric Host Service
• Workflow Manager Backend - repeatedly stopped and restarted; starting it (when stopped) did not resolve issue.
2. Checked Windows Server Application log:
• Saw events 1000, 1026 and 7031
3. Checked Microsoft Workflow logs
• Saw events 358, 359, 19 and 10
4. Checked Workflow Management Site through IIS:
• Site was started.  Restarting did not resolve issue.
5. Checked Workflow Management Application Pool
• Pool was started.  Restarting it did not resolve issue.
6. Checked local Administrators group:
• Added workflow service account to local Administrators group and then rebooted server, but this did not resolve issue.
7. Executed script per reference 2, but this did not resolve issue.
8. Installed full September 2014 CU, but this did not resolve issue.
9. Performed full uninstallation and then a fresh installation of entire Workflow Manager 1.0 system.
• Issue resolved
10. UPDATE (12/03/14)
11. Issue recurred six weeks later. 
12. More thoroughly reviewed error events occuring in Workflow Operational log
• Noted that the error messages discussed workflow instances and that two workflow instance IDs were repeatedly presented in these error messages.
13. Checked Workflows Settings page for all lists having 2013 workflows attached (4).
• Found one 2013 workflow having two running instances.
14. Removed these instances, and then checked server event logs:
• No more error messages - stopped the moment the instances were removed.
15. Workflow designer reviewed workflow, based upon error message text and input from SharePoint administrator
1. Workflow designer discovered an issue in the workflow.
16. New Solution
1. Revise workflow

Solution

1. Review workflow
2. Revise workflow

References

1. Workflow Manager 1.0 (SharePoint 2013): The process was terminated due to an unhandled exception
2. Update Workflow in SharePoint Server 2013
3. Install & Configure Workflow Manager 1.0
4. Configuring The Service Bus Workflow Farm
5. Workflow Manager 1.0-SharePoint 2013
6. New Workflow Manager 1.0 Getting Started Tutorial
7. Updates for SharePoint 2013 Workflow
8. Installing Workflow Manager 1.0
9. SharePoint 2013: Workflow Manager Installation and Configuration
10. Least Privilege Configuration for Workflow Manager with SharePoint 2013
11. Export SharePoint Designer 2013 Workflow XAML
12. Understanding how to package and deploy workflow in SharePoint 2013
13. SharePoint 2013: Workflow Manager Installation and configuration issues…
14. Installing Workflow Manager Cumulative Update 1
15. Understanding SharePoint 2013 Workflow Backup
16. Configuring Workflow Manager 1.0
17. Workflow Manager Farms for SharePoint 2013 Part Three: Switching an existing farm to use Domain CA issued certificates
18. Monitoring Workflow Manager 1.0
19. Creating a New Farm
20. UPDATE
21. Dispatcher failfast is causing the Workflow Manager backend to terminate
22. SharePoint 2013 workflow fundamentals
23. SharePoint 2013: how to restore Workflow Manager 1.0 workflows in the event of a disaster

Notes

• 2013 Workflow Topology:
• Single App Server: hosts Workflow Manager 1.0, Service Bus 1.0 and Windows Fabric Host
• Two WFEs: host Workflow Manager 1.0 Client
• Note that performing a full un-installation, followed by a new installation, completely wipes out any 2013 workflows that may already have been created.   While performing a restore / recovery operation is theoretically possible, and some admins appear to have done this successfully, you don't have to perform a full Workflow 2013 restore operation in order to recovery the 2013 workflows, which are actually stored in the farm content databases.  Instead, you can recover them by restoring the content databases.
   
  UPDATE (12/3/14): opened Microsoft Support ticket on original issue. Microsoft SharePoint Admin support specialist indicated that the 2013 workflows are actually stored in the 2013 content databases; and that these will become available again on successful re-installation of Workflow Manager 1.0 and re-integration with SharePoint 2013. This was verified.

Events

• Windows Server Application Log
• Error Event 1000:
  Faulting application name: Microsoft.Workflow.ServiceHost.exe, version: 1.0.20922.0, time stamp: 0x505e1b24 Faulting module name: KERNELBASE.dll, version: 6.2.9200.16864, time stamp: 0x531d34d8 Exception code: 0xe0434352 Fault offset: 0x0000000000047b8c Faulting process id: 0x4018 Faulting application start time: 0x01cfe8ac9a5561b0 Faulting application path: C:\Program Files\Workflow Manager\1.0\Workflow\Artifacts\Microsoft.Workflow.ServiceHost.exe Faulting module path: C:\Windows\system32\KERNELBASE.dll
   
• Error Event 1026:
  Application: Microsoft.Workflow.ServiceHost.exe Framework Version: v4.0.30319 Description: The process was terminated due to an unhandled exception. Exception Info: Microsoft.Workflow.Common.FatalException Stack: at Microsoft.Workflow.Common.Fx+<>c__DisplayClass2.<failfast>b__0() at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at System.Threading.ThreadHelper.ThreadStart()
   
• Error Event 7031:
  The Workflow Manager Backend service terminated unexpectedly. It has done this 6121 time(s). The following corrective action will be taken in 30000 milliseconds: Restart the service
   
• Microsoft-Workflow Operational Log
• Error Event 358:
  Dispatcher encountered an unexpected exception: System.Runtime.CallbackException: The call to OnRequestAbort on the WorkflowInstance '35616a0a-26da-4765-b92d-3f6eb5727508' threw an exception. This is considered fatal. See inner exception for details. ---> System.AggregateException: Should be processing if notifying about persistable idle. ---> System.NullReferenceException: Object reference not set to an instance of an object. at System.Activities.Runtime.ActivityExecutor.CancelRootActivity() at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.ScheduleCancel() at Microsoft.Activities.Hosting.HostedWorkflowInstance.Cancel() at Microsoft.Activities.Hosting.WorkflowSession.TryDispatchNextSubroutineWorkItem() at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() --- End of inner exception stack trace --- at Microsoft.Activities.Hosting.HostedWorkflowInstance.NotifyPaused(Exception abortReason) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.AbortInstance(Exception reason, Boolean isWorkflowThread) at System.Activities.Runtime.ActivityExecutor.AbortWorkflowInstance(Exception reason) --- End of inner exception stack trace --- at System.Activities.Runtime.ActivityExecutor.AbortWorkflowInstance(Exception reason) at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.TryPropagateCompletion(HostedWorkflowInstance instance) at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.WorkflowSession.TryStartSubroutine(Guid subroutineId, Guid rootActivityId, Dictionary`2 inArguments, List`1 executionProperties) at Microsoft.Activities.Hosting.WorkflowSession.TryDispatchNextSubroutineWorkItem() at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.Cancel() at Microsoft.Activities.Hosting.WorkflowSession.OnUnhandledException(HostedWorkflowInstance throwingInstance, Exception unhandledException, Boolean isRootFaulted) at System.Activities.Runtime.ActivityExecutor.NotifyUnhandledException(Exception exception, ActivityInstance source) at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.TryPropagateCompletion(HostedWorkflowInstance instance) at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.TryPropagateCompletion(HostedWorkflowInstance instance) at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.WorkflowSession.TryStartSubroutine(Guid subroutineId, Guid rootActivityId, Dictionary`2 inArguments, List`1 executionProperties) at Microsoft.Activities.Hosting.WorkflowSession.TryDispatchNextSubroutineWorkItem() at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.ResumeBookmark(Guid subroutineId, Bookmark bookmark, Object value, Action`1 sessionIdleCallback) at Microsoft.Activities.Hosting.WorkflowSession.ResumeInternal(BookmarkWaiter bookmarkWaiter, Object value, DateTime resumptionTime, Action`1 sessionIdleCallback) at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.ProcessWaiter() at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.Isolate(AsyncCompletion callback, IAsyncResult result) at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.ExceptionHandlingFrame(IAsyncResult result)
   
• Critical Event 359:
  Dispatcher failfast is causing the Workflow Manager backend to terminate. Exception: System.AggregateException: The dispatcher failure rate has reached 11%. InnerException is the last failure; see traces for other failures. ---> System.Runtime.CallbackException: The call to OnRequestAbort on the WorkflowInstance '35616a0a-26da-4765-b92d-3f6eb5727508' threw an exception. This is considered fatal. See inner exception for details. ---> System.AggregateException: Should be processing if notifying about persistable idle. ---> System.NullReferenceException: Object reference not set to an instance of an object. at System.Activities.Runtime.ActivityExecutor.CancelRootActivity() at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.ScheduleCancel() at Microsoft.Activities.Hosting.HostedWorkflowInstance.Cancel() at Microsoft.Activities.Hosting.WorkflowSession.TryDispatchNextSubroutineWorkItem() at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() --- End of inner exception stack trace --- at Microsoft.Activities.Hosting.HostedWorkflowInstance.NotifyPaused(Exception abortReason) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.AbortInstance(Exception reason, Boolean isWorkflowThread) at System.Activities.Runtime.ActivityExecutor.AbortWorkflowInstance(Exception reason) --- End of inner exception stack trace --- at System.Activities.Runtime.ActivityExecutor.AbortWorkflowInstance(Exception reason) at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.TryPropagateCompletion(HostedWorkflowInstance instance) at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.WorkflowSession.TryStartSubroutine(Guid subroutineId, Guid rootActivityId, Dictionary`2 inArguments, List`1 executionProperties) at Microsoft.Activities.Hosting.WorkflowSession.TryDispatchNextSubroutineWorkItem() at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.Cancel() at Microsoft.Activities.Hosting.WorkflowSession.OnUnhandledException(HostedWorkflowInstance throwingInstance, Exception unhandledException, Boolean isRootFaulted) at System.Activities.Runtime.ActivityExecutor.NotifyUnhandledException(Exception exception, ActivityInstance source) at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.TryPropagateCompletion(HostedWorkflowInstance instance) at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.TryPropagateCompletion(HostedWorkflowInstance instance) at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.WorkflowSession.TryStartSubroutine(Guid subroutineId, Guid rootActivityId, Dictionary`2 inArguments, List`1 executionProperties) at Microsoft.Activities.Hosting.WorkflowSession.TryDispatchNextSubroutineWorkItem() at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.ResumeBookmark(Guid subroutineId, Bookmark bookmark, Object value, Action`1 sessionIdleCallback) at Microsoft.Activities.Hosting.WorkflowSession.ResumeInternal(BookmarkWaiter bookmarkWaiter, Object value, DateTime resumptionTime, Action`1 sessionIdleCallback) at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.ProcessWaiter() at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.Isolate(AsyncCompletion callback, IAsyncResult result) at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.ExceptionHandlingFrame(IAsyncResult result) --- End of inner exception stack trace --- ---> (Inner Exception #0) System.Runtime.CallbackException: The call to OnRequestAbort on the WorkflowInstance '35616a0a-26da-4765-b92d-3f6eb5727508' threw an exception. This is considered fatal. See inner exception for details. ---> System.AggregateException: Should be processing if notifying about persistable idle. ---> System.NullReferenceException: Object reference not set to an instance of an object. at System.Activities.Runtime.ActivityExecutor.CancelRootActivity() at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.ScheduleCancel() at Microsoft.Activities.Hosting.HostedWorkflowInstance.Cancel() at Microsoft.Activities.Hosting.WorkflowSession.TryDispatchNextSubroutineWorkItem() at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() --- End of inner exception stack trace --- at Microsoft.Activities.Hosting.HostedWorkflowInstance.NotifyPaused(Exception abortReason) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.AbortInstance(Exception reason, Boolean isWorkflowThread) at System.Activities.Runtime.ActivityExecutor.AbortWorkflowInstance(Exception reason) --- End of inner exception stack trace --- at System.Activities.Runtime.ActivityExecutor.AbortWorkflowInstance(Exception reason) at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.TryPropagateCompletion(HostedWorkflowInstance instance) at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.WorkflowSession.TryStartSubroutine(Guid subroutineId, Guid rootActivityId, Dictionary`2 inArguments, List`1 executionProperties) at Microsoft.Activities.Hosting.WorkflowSession.TryDispatchNextSubroutineWorkItem() at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.Cancel() at Microsoft.Activities.Hosting.WorkflowSession.OnUnhandledException(HostedWorkflowInstance throwingInstance, Exception unhandledException, Boolean isRootFaulted) at System.Activities.Runtime.ActivityExecutor.NotifyUnhandledException(Exception exception, ActivityInstance source) at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.TryPropagateCompletion(HostedWorkflowInstance instance) at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.TryPropagateCompletion(HostedWorkflowInstance instance) at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.WorkflowSession.TryStartSubroutine(Guid subroutineId, Guid rootActivityId, Dictionary`2 inArguments, List`1 executionProperties) at Microsoft.Activities.Hosting.WorkflowSession.TryDispatchNextSubroutineWorkItem() at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.ResumeBookmark(Guid subroutineId, Bookmark bookmark, Object value, Action`1 sessionIdleCallback) at Microsoft.Activities.Hosting.WorkflowSession.ResumeInternal(BookmarkWaiter bookmarkWaiter, Object value, DateTime resumptionTime, Action`1 sessionIdleCallback) at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.ProcessWaiter() at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.Isolate(AsyncCompletion callback, IAsyncResult result) at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.ExceptionHandlingFrame(IAsyncResult result)<--- data-blogger-escaped-li="">
   
• Error Event 19:
  Application terminating due to FailFast. Message : An unrecoverable error occurred. For diagnostic purposes, this English message is associated with the failure: 'The dispatcher failure rate has reached 11%. InnerException is the last failure; see traces for other failures.'., Exception: System.AggregateException: The dispatcher failure rate has reached 11%. InnerException is the last failure; see traces for other failures. ---> System.Runtime.CallbackException: The call to OnRequestAbort on the WorkflowInstance '35616a0a-26da-4765-b92d-3f6eb5727508' threw an exception. This is considered fatal. See inner exception for details. ---> System.AggregateException: Should be processing if notifying about persistable idle. ---> System.NullReferenceException: Object reference not set to an instance of an object. at System.Activities.Runtime.ActivityExecutor.CancelRootActivity() at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.ScheduleCancel() at Microsoft.Activities.Hosting.HostedWorkflowInstance.Cancel() at Microsoft.Activities.Hosting.WorkflowSession.TryDispatchNextSubroutineWorkItem() at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() --- End of inner exception stack trace --- at Microsoft.Activities.Hosting.HostedWorkflowInstance.NotifyPaused(Exception abortReason) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.AbortInstance(Exception reason, Boolean isWorkflowThread) at System.Activities.Runtime.ActivityExecutor.AbortWorkflowInstance(Exception reason) --- End of inner exception stack trace --- at System.Activities.Runtime.ActivityExecutor.AbortWorkflowInstance(Exception reason) at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.TryPropagateCompletion(HostedWorkflowInstance instance) at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.WorkflowSession.TryStartSubroutine(Guid subroutineId, Guid rootActivityId, Dictionary`2 inArguments, List`1 executionProperties) at Microsoft.Activities.Hosting.WorkflowSession.TryDispatchNextSubroutineWorkItem() at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.Cancel() at Microsoft.Activities.Hosting.WorkflowSession.OnUnhandledException(HostedWorkflowInstance throwingInstance, Exception unhandledException, Boolean isRootFaulted) at System.Activities.Runtime.ActivityExecutor.NotifyUnhandledException(Exception exception, ActivityInstance source) at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.TryPropagateCompletion(HostedWorkflowInstance instance) at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.TryPropagateCompletion(HostedWorkflowInstance instance) at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.WorkflowSession.TryStartSubroutine(Guid subroutineId, Guid rootActivityId, Dictionary`2 inArguments, List`1 executionProperties) at Microsoft.Activities.Hosting.WorkflowSession.TryDispatchNextSubroutineWorkItem() at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.ResumeBookmark(Guid subroutineId, Bookmark bookmark, Object value, Action`1 sessionIdleCallback) at Microsoft.Activities.Hosting.WorkflowSession.ResumeInternal(BookmarkWaiter bookmarkWaiter, Object value, DateTime resumptionTime, Action`1 sessionIdleCallback) at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.ProcessWaiter() at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.Isolate(AsyncCompletion callback, IAsyncResult result) at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.ExceptionHandlingFrame(IAsyncResult result) --- End of inner exception stack trace --- ---> (Inner Exception #0) System.Runtime.CallbackException: The call to OnRequestAbort on the WorkflowInstance '35616a0a-26da-4765-b92d-3f6eb5727508' threw an exception. This is considered fatal. See inner exception for details. ---> System.AggregateException: Should be processing if notifying about persistable idle. ---> System.NullReferenceException: Object reference not set to an instance of an object. at System.Activities.Runtime.ActivityExecutor.CancelRootActivity() at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.ScheduleCancel() at Microsoft.Activities.Hosting.HostedWorkflowInstance.Cancel() at Microsoft.Activities.Hosting.WorkflowSession.TryDispatchNextSubroutineWorkItem() at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() --- End of inner exception stack trace --- at Microsoft.Activities.Hosting.HostedWorkflowInstance.NotifyPaused(Exception abortReason) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.AbortInstance(Exception reason, Boolean isWorkflowThread) at System.Activities.Runtime.ActivityExecutor.AbortWorkflowInstance(Exception reason) --- End of inner exception stack trace --- at System.Activities.Runtime.ActivityExecutor.AbortWorkflowInstance(Exception reason) at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.TryPropagateCompletion(HostedWorkflowInstance instance) at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.WorkflowSession.TryStartSubroutine(Guid subroutineId, Guid rootActivityId, Dictionary`2 inArguments, List`1 executionProperties) at Microsoft.Activities.Hosting.WorkflowSession.TryDispatchNextSubroutineWorkItem() at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.Cancel() at Microsoft.Activities.Hosting.WorkflowSession.OnUnhandledException(HostedWorkflowInstance throwingInstance, Exception unhandledException, Boolean isRootFaulted) at System.Activities.Runtime.ActivityExecutor.NotifyUnhandledException(Exception exception, ActivityInstance source) at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.TryPropagateCompletion(HostedWorkflowInstance instance) at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.TryPropagateCompletion(HostedWorkflowInstance instance) at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.WorkflowSession.TryStartSubroutine(Guid subroutineId, Guid rootActivityId, Dictionary`2 inArguments, List`1 executionProperties) at Microsoft.Activities.Hosting.WorkflowSession.TryDispatchNextSubroutineWorkItem() at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.ResumeBookmark(Guid subroutineId, Bookmark bookmark, Object value, Action`1 sessionIdleCallback) at Microsoft.Activities.Hosting.WorkflowSession.ResumeInternal(BookmarkWaiter bookmarkWaiter, Object value, DateTime resumptionTime, Action`1 sessionIdleCallback) at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.ProcessWaiter() at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.Isolate(AsyncCompletion callback, IAsyncResult result) at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.ExceptionHandlingFrame(IAsyncResult result)<--- data-blogger-escaped-.="" data-blogger-escaped-li="">
   
• Critical Event 10:
  Unhandled exception in AppDomain: Microsoft.Workflow.Common.FatalException: An unrecoverable error occurred. For diagnostic purposes, this English message is associated with the failure: 'The dispatcher failure rate has reached 11%. InnerException is the last failure; see traces for other failures.'. ---> System.AggregateException: The dispatcher failure rate has reached 11%. InnerException is the last failure; see traces for other failures. ---> System.Runtime.CallbackException: The call to OnRequestAbort on the WorkflowInstance '35616a0a-26da-4765-b92d-3f6eb5727508' threw an exception. This is considered fatal. See inner exception for details. ---> System.AggregateException: Should be processing if notifying about persistable idle. ---> System.NullReferenceException: Object reference not set to an instance of an object. at System.Activities.Runtime.ActivityExecutor.CancelRootActivity() at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.ScheduleCancel() at Microsoft.Activities.Hosting.HostedWorkflowInstance.Cancel() at Microsoft.Activities.Hosting.WorkflowSession.TryDispatchNextSubroutineWorkItem() at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() --- End of inner exception stack trace --- at Microsoft.Activities.Hosting.HostedWorkflowInstance.NotifyPaused(Exception abortReason) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.AbortInstance(Exception reason, Boolean isWorkflowThread) at System.Activities.Runtime.ActivityExecutor.AbortWorkflowInstance(Exception reason) --- End of inner exception stack trace --- at System.Activities.Runtime.ActivityExecutor.AbortWorkflowInstance(Exception reason) at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.TryPropagateCompletion(HostedWorkflowInstance instance) at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.WorkflowSession.TryStartSubroutine(Guid subroutineId, Guid rootActivityId, Dictionary`2 inArguments, List`1 executionProperties) at Microsoft.Activities.Hosting.WorkflowSession.TryDispatchNextSubroutineWorkItem() at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.Cancel() at Microsoft.Activities.Hosting.WorkflowSession.OnUnhandledException(HostedWorkflowInstance throwingInstance, Exception unhandledException, Boolean isRootFaulted) at System.Activities.Runtime.ActivityExecutor.NotifyUnhandledException(Exception exception, ActivityInstance source) at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.TryPropagateCompletion(HostedWorkflowInstance instance) at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.TryPropagateCompletion(HostedWorkflowInstance instance) at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.WorkflowSession.TryStartSubroutine(Guid subroutineId, Guid rootActivityId, Dictionary`2 inArguments, List`1 executionProperties) at Microsoft.Activities.Hosting.WorkflowSession.TryDispatchNextSubroutineWorkItem() at Microsoft.Activities.Hosting.WorkflowSession.OnPaused(HostedWorkflowInstance pausedInstance) at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.OnNotifyPaused() at System.Activities.Runtime.ActivityExecutor.OnSchedulerIdle() at System.Activities.Runtime.Scheduler.OnScheduledWork(Object state) at System.Runtime.Fx.SendOrPostThunk.UnhandledExceptionFrame(Object state) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Run() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ReleaseLock() at Microsoft.Activities.Hosting.HostedWorkflowInstance.ResumeBookmark(Bookmark bookmark, Object value) at Microsoft.Activities.Hosting.WorkflowSession.ResumeBookmark(Guid subroutineId, Bookmark bookmark, Object value, Action`1 sessionIdleCallback) at Microsoft.Activities.Hosting.WorkflowSession.ResumeInternal(BookmarkWaiter bookmarkWaiter, Object value, DateTime resumptionTime, Action`1 sessionIdleCallback) at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.ProcessWaiter() at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.Isolate(AsyncCompletion callback, IAsyncResult result) at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.ExceptionHandlingFrame(IAsyncResult result) --- End of inner exception stack trace --- --- End of inner exception stack trace --- at Microsoft.Workflow.Common.Fx.<>c__DisplayClass2.b__0() at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading.ThreadHelper.ThreadStart().