SharePoint 2013: AppFabric service repeatedly stops

Problem

You discover that the AppFabric Caching Service, seen in the Services control panel, repeatedly stops.  You can start it, and it runs awhile, but then it stops.

History

• Errors appear to have begun after changing the AppFabric identity from its default (on new farm installation, this is the farm service account) to a service account (eg, spService).

Troubleshooting

1. Check Service Status: Executing Use-CacheCluster; Get-CacheHost, while the service is running, you find system status to be UNKNOWN.  If you execute this while the service is stopped, you see a status of DOWN.
2. Check Application Log: reviewing the server's logs, you see these events occurring regularly:
   
   

   Error Event 111: Microsoft-Windows Server AppFabric Caching
   AppFabric Caching service crashed with exception 
   {Microsoft.ApplicationServer.Caching.DataCacheException:...
   
   Error Event 1000: Application Error
   Faulting application name: DistributedCacheService.exe, version: 1.0.4632.0,
    time stamp: 0x4eafeccf...
   
   Error Event 2016: .NET Runtime
   Application: DistributedCacheService.exe
   Framework Version: v4.0.30319
   Description: The process was terminated due to an unhandled exception...
   
   Error Event 7031: Service Control Manager
   The AppFabric Caching Service service terminated unexpectedly.  It has done 
   this 1 time(s).  The following corrective action will be taken in 60000 
   milliseconds: Restart the service...
    
   Error Event 7034: Service Control Manager
   The AppFabric Caching Service service terminated unexpectedly.  It has done 
   this 26 time(s)...
    
   Warning Event 6: Microsoft Windows Fabric
   {7bf7299f000000000000000000000000} failed to refresh lookup table, 
   exception: {Microsoft.Fabric.Common.OperationCompletedException: 
   Operation completed with an exception ---> System.TimeoutException: 
   The operation has timed out.... [service account]
    

3. Check Permissions: added AppFabric identity to local Administrators group, but this failed to resolve problem. Changed AppFabric identity back to the farm service account (eg, spFarm). No new errors appeared.  Thus, the problem scope was narrowed to the service account.  Next: needed to determine what about the farm service account enabled it to successfully run the AppFabric service when the application service account could not.
4. Check Database Role Memberships: reviewed and compared farm configuration database (eg, Config) roles of farm and application service accounts: farm service account mapped as: public, db_owner, SPDataAccess, SharePoint_Shell_Access, WSS_Content_Application_Pools, db_accessadmin; application service account mapped as: public and WSS_Content_Application_Pools only.  Thus, application service account did not have sufficient access to farm configuration database.  Mapped SPDataAccess database role to application service account; changed AppFabric identity back to application service account, then monitored: no new errors.

Solution

• Ensure that the AppFabric service identity has the appropriate access to the farm configuration database.  Mapping it to the SPDataAccess role is sufficient.

References

• AppFabric Distributed Cache keeps crashing in SharePoint 2013 – Fixed!
• Both Distributed Cache SharePoint Service and AppFabric Caching Service stop
• AppFabricCachingService – Service Status: Unknown
• AppFabric Caching and SharePoint: Concepts and Examples (Part 1)
• Account permissions and security settings in SharePoint 2013
• My Sites and The SPDataAccess SQL Role

Notes

• Small, two server, two-tier farm
• SQL Server 2012 Standard
• SharePoint 2013 Enterprise
• New deployment.