Tuesday, May 20, 2014

SharePoint 2013: FatalError: Object Search Service failed in event OnBackup

Problem

You recently deployed a new SharePoint Server 2013 farm. You attempted to perform a full backup through Central Administration for the first time.  After the backup completes, you review the Backup job status and find a number of errors, all involving backup of the Search service databases. You see this error in the status page:
Object Search Service failed in event OnBackup. For more information, see the spbackup.log or sprestore.log file located in the backup directory. FaultException: Management called failed with System.InvalidOperationException: 'Job failed: Have tried to perform backup/restore operation twice on all in-sync members in cluster SP0e4f5d0f2fce.0, but none succeeded. Last failure message: Microsoft.Ceres.SearchCore.Seeding.SnapshotTransferException: Could not send chunk ms\%default\gen.000000000000007e.state: Localpath: [0-349> to target BackupDirectoryTarget[directory=[PathToBackupFolder]\spbr0000\I.0.0,validateTransfers=False] at Microsoft.Ceres.SearchCore.Seeding.SnapshotSender.SendChunks(ISnapshot snapshot, ISeedSource source, ISeedTarget target, SeedStatus status, Func`1 checkAborted, Int32 targetFragIndex) at Microsoft.Ceres.SearchCore.Seeding.SnapshotSender.FirstPhaseTransfer(ISeedSource source, ISeedTarget target, Action`1 updateProgress, Func`1 shouldAbort) at Microsoft.Ceres.SearchCore.Seeding.BackupWorker.BackupWork.DoFirstPhaseWork()' at at Microsoft.Ceres.SearchCore.IndexController.BackupService.ThrowOnFailure(JobStatus status) at Microsoft.Ceres.SearchCore.IndexController.BackupService.ProgressFirstPhase(String handle) at Microsoft.Ceres.SearchCore.IndexController.IndexControllerManagementAgent.WrapCall[T](Func`2 original)
Opening the backup log file, spbackup, you see that the backup process was able to communicate with the backend just fine, as you'll see plenty of messages like these:
...
[Date/Time] Progress: [Search_AdminDB] 90 percent complete.
[Date/Time] Progress: [Search_AdminDB_CrawlStore] 100 percent complete.
[Date/Time] Progress: [Search_AdminDB_LinksStore] 91 percent complete.
[Date/Time] Progress: [Search_AdminDB_AnalyticsReportingStore] 91 percent 
complete.
[Date/Time] Progress: [Search_AdminDB] 97 percent complete.
[Date/Time] Verbose: [Search_AdminDB_CrawlStore] SQL Server Message: 
Processed 3 pages for database 'Search_AdminDB_CrawlStore', file 
'Search_AdminDB_CrawlStore_log' on file 1.
[Date/Time] Progress: [Search_AdminDB_LinksStore] 95 percent complete.
...
which indicates that the backup process is communicating with the backend just fine and thus this SQL Server interaction is not associated with the failure.  Looking farther down, towards the completion section of the log, you see errors occuring again:
...
[Date/Time] FatalError: Object Search_AdminDB failed in event
OnBackupComplete. For more information, see the spbackup.log or 
sprestore.log file located in the backup directory.
 Aborted due to error in another component.
[Date/Time] Verbose: Starting object: Search_AdminDB_CrawlStore.
[Date/Time] FatalError: Object Admin (C: on [Search Host Server]) 
failed in event OnBackupComplete. For more information, see the spbackup.log 
or sprestore.log file located in the backup directory.
 Aborted due to error in another component.
...
Lastly, looking at the application event log on the Search Service host server, you see the following timer event error:
Log Name:      Application
Source:        Microsoft-SharePoint Products-SharePoint Foundation
Date:          [Date/Time]
Event ID:      6398
Task Category: Timer
Level:         Critical
Keywords:      
User:          [Farm Account]
Computer:      [Search host server]
Description:
The Execute method of job definition Microsoft.SharePoint.Administration.Backup.SPBackupRestoreJobDefinition (ID 26eac21f-d391-4024-a82f-8ee0b5738ba3) threw an exception. More information is included below.
The backup job failed. For more information, see the error log that is located in the backup directory.
Event Xml:
...

Solution
  1. Identify the Search Service Account.
  2. Navigate to the intranet folder to which you intend to write full farm backups
  3. Grant the Search Service Account the following permissions to this folder:
    1. Read
    2. Write
References
  1. SharePoint 2013 Farm Backup Error : Object Search Service Application failed in event OnBackup. Could not send chunk to target BackupDirectoryTarget
  2. Plan for administrative and service accounts in SharePoint 2013
  3. Account permissions and security settings in SharePoint 2013
  4. Create and configure a Search service application in SharePoint Server 2013
  5. Configure backup and restore permissions in SharePoint 2013
  6. Back up Search service applications in SharePoint 2013
Notes
  • General: the issue involves service accounts and permissions to the destination backup folder. 
  • Backup log file: this will be found in the same folder containing the farm backup files.  For example, if your backup folder is SharePointBackup, inside this folder you will find a number of subfolders, each corresponding to a previous farm backup, numbered: spbr0000, spbr0001, spbr0002 and so on.  Open the most recent one to view the log of the most recent backup.
  • Search Service Account Access to Backup: this is new from 2010.  For 2010, you had to grant access to the SQL Server Service Account.  I was not able to find any discussion on this among all of the expected Microsoft documentation resources (see references 2 - 6).  Thanks to Amol Meshe for resolving this one.

1 comment:

Anonymous said...

thanks a lot.