SharePoint 2013: How to deploy a search service application using PowerShell

Introduction

SharePoint 2013 Deployment Series

How to setup a database server alias

How to configure cache

How to deploy a search service application using PowerShell

You have deployed a new SharePoint Server 2013 farm.  You now wish to provision the farm's Search Service Application and to do so via PowerShell so as to be able to define simple custom search database names.  This posting shows you how.  This posting assumes a five-server farm, including: one database server, one application server, two web front end servers and one Office Web Apps server. All servers are VMs hosted on Microsoft Hyper-V.  And it assumes two service accounts will be used: a content access account and an account used to run the Admin and query services.  The search topology that will be deployed will be the following:

SSL is not used to connect to the primary web application content source.

Procedure

1. Remove Existing Search
1. Remove the existing Search Service Application (if it exists):
1. In Central Administration, navigate to Manage Service Applications.
2. Select the row corresponding to the search service application.
3. Select the SERVICE APPLICATIONS tab, and then click the Delete button.
4. At the prompt, enable the option to remove all data as well.
5. Check the Manage Service Applications page to verify that all three of these have been removed:
1. Search Administration Web Service for Search Service
2. Search Service
3. Search Service Application Proxy
6. Check SQL Server to verify that all the search content and associated transaction log databases have been removed.
2. Remove Search Application Pools
1. In the SharePoint Management Shell, execute Get-SPServiceApplicationPool.
2. Verify that no search application pools are listed.
3. If any are, remove them using Remove-SPServiceApplicationPool.
2. Preparation
1. Verify setup user administrator account (eg, spAdmin) is member of local Administrator group on all farm servers.
2. Identify service accounts and passwords:
1. Content Access service account
2. Administration and query service account
3. Map the Admin service account to diagnostic logging database as dbo.
4. Identify the web front end server names
1. WFE1
2. WFE2
5. Identify the drive on the servers where you want to store indexes.  This drive must be the same on all farm servers hosting SharePoint.
1. (eg) D:\SPIndex
6. Verify that any application pools created for previous installations (if any) of the Search Service Application have been removed.
1. (eg) Search Service AppPool
3. Execution
1. Instantiate Search Services
1. Remote into the application server as the setup user administrator account.
2. Open the SharePoint 2013 Management Shell as administrator.
3. Download this script: Enterprise Search Deployment Script.
4. Edit variable as appropriate to your farm.
5. Run the script.
6. After completion of the script, navigate to the Search Service Administration page and verify that the search service topology is similar to what is presented above.
7. Verify that the default content access service account is set as the crawl account.
2. Configure Content Sources
1. In CA, navigate to Content Sources, and then click on Local SharePoint sites (it will be the only item in the list at present).
2. Note the sources (URLs) present.
3. Create individual content sources for each of these URLs so that they can be crawl scheduled individually.  So, for example, configure the following entries:
1. Local SharePoint sites: point it to your primary web application.  If your farm has more than one web application supporting user websites, creating individual entries for each.
2. My Sites Content: point this to your farm's My Site web application.
3. User Profile Content: point this to the URL for crawling user profiles.  It will have the form like: sps3://MySites.com (non-SSL).
4. Configure crawl schedules for each content source as appropriate.  Create full and incremental schedules.
3. Configure Global Search Center
1. Create a Search Center.  Deploy this within the primary web application or in a new web application as appropriate to your needs.  I recommend adding it to your existing primary web application.
2. In CA, navigate to the Search Service Administration page, and then set the Global Search Center URL to the Search Center you have created for your web application.
4. Configure SSL Content sources
1. If your search service needs to use SSL to crawl certain content sources, you will need to configure the service to ignore SSL warnings.
2. In CA, go to: General Application Settings > Search > Farm Search Administration.
3. In the Farm-Level Search Settings group, look for Ignore SSL warnings.
4. Click No.
5. Enable Ignore SSL certificate name warnings.
6. Click OK.
4. Miscellaneous
1. If you plan to perform farm backups through Central Administration or PowerShell, be sure to grant the Search service account read and write permissions to the backup folder.  Otherwise, farm backups will fail on backup of the Search service configuration and data.  This is new from 2010.

References

• Manage search components in SharePoint Server 2013
• SharePoint 2013 Unable to change topology when Generation controller is not active
• Manage the index component in SharePoint Server 2013
• Get-SPEnterpriseSearchStatus
• Manage the search topology in SharePoint Server 2013
• Remove-SPServiceApplicationPool
• Get-SPServiceApplicationPool
• Configure a service application by using a Windows PowerShell script (SharePoint Server 2010)
• Service application cmdlets in SharePoint 2013
• Deploy people search in SharePoint Server 2013
• SharePoint 2013: FatalError: Object Search Service failed in event OnBackup
• Enterprise Search Deployment Script
• Remove-SPServiceApplication
• Delete a service application in SharePoint 2013

Notes

• Farm topology: This procedure was executed on a five-server farm (including OWA and SQL), Windows Server 2012 VMs.
• Deleting the Search Service Application through Central Administration does not remove the application pools that were created for the Search Service Application.  Thus, to completely remove all artifacts associated with the Search Service Application, and thus be able to start over from scratch, you must also manually remove those application pools that were created for it.
• Default Content Sources: After deploying the Search Service Application, you will need to configure content sources. Your farm content sources (aside from CA) will automatically have their URLs added to the Local SharePoint sites group - including My Sites.  I recommend separating content sources, so as to be able to implement separate crawl schedules for them.  Not all content sources require as frequent crawling as your primary web applications.
• User profile content: if you want to enable searches of user profiles, you will need to add the URL for this manually.
• Search Service Account access to backup folder: I don't know how or why, but in SharePoint 2013, the Search Service Account plays a role in the farm backup process when it comes to backup of Search. This was not the case in SharePoint 2010, where I only had to grant the farm and sql server service accounts this access.  I discovered this new wrinkle while actually trying to perform a full farm backup for the first time and then troubleshooting and resolving its initial failure.
• Removing Search via PowerShell.  First, get the Identity of the application.  Then run the command to remove:

  Get-SPServiceApplication | ?{$_.name -like "*Search*"}
  $spapp = Get-SPServiceApplication -identity [
  Remove-SPServiceApplication $spapp -RemoveData

  If you don't use the "-RemoveData" switch parameter, the search databases and their associated transaction logs will remain; and then you'll see "A databases exception occurred..." error appear in the shell window when you try to execute New-SPEnterpriseSearchServiceApplication.  
• 
  

SharePoint 2013: FatalError: Object Search Service failed in event OnBackup

Problem

You recently deployed a new SharePoint Server 2013 farm. You attempted to perform a full backup through Central Administration for the first time.  After the backup completes, you review the Backup job status and find a number of errors, all involving backup of the Search service databases. You see this error in the status page:

Object Search Service failed in event OnBackup. For more information, see the spbackup.log or sprestore.log file located in the backup directory. 
 FaultException: Management called failed with System.InvalidOperationException: 'Job failed: Have tried to perform backup/restore operation twice on all in-sync members in cluster SP0e4f5d0f2fce.0, but none succeeded. Last failure message: Microsoft.Ceres.SearchCore.Seeding.SnapshotTransferException: Could not send chunk ms\%default\gen.000000000000007e.state: Localpath: [0-349> to target BackupDirectoryTarget[directory=[PathToBackupFolder]\spbr0000\I.0.0,validateTransfers=False] 
   at Microsoft.Ceres.SearchCore.Seeding.SnapshotSender.SendChunks(ISnapshot snapshot, ISeedSource source, ISeedTarget target, SeedStatus status, Func`1 checkAborted, Int32 targetFragIndex) 
   at Microsoft.Ceres.SearchCore.Seeding.SnapshotSender.FirstPhaseTransfer(ISeedSource source, ISeedTarget target, Action`1 updateProgress, Func`1 shouldAbort) 
   at Microsoft.Ceres.SearchCore.Seeding.BackupWorker.BackupWork.DoFirstPhaseWork()' at    at Microsoft.Ceres.SearchCore.IndexController.BackupService.ThrowOnFailure(JobStatus status) 
   at Microsoft.Ceres.SearchCore.IndexController.BackupService.ProgressFirstPhase(String handle) 
   at Microsoft.Ceres.SearchCore.IndexController.IndexControllerManagementAgent.WrapCall[T](Func`2 original)

Opening the backup log file, spbackup, you see that the backup process was able to communicate with the backend just fine, as you'll see plenty of messages like these:

...
[Date/Time] Progress: [Search_AdminDB] 90 percent complete.
[Date/Time] Progress: [Search_AdminDB_CrawlStore] 100 percent complete.
[Date/Time] Progress: [Search_AdminDB_LinksStore] 91 percent complete.
[Date/Time] Progress: [Search_AdminDB_AnalyticsReportingStore] 91 percent 
complete.
[Date/Time] Progress: [Search_AdminDB] 97 percent complete.
[Date/Time] Verbose: [Search_AdminDB_CrawlStore] SQL Server Message: 
Processed 3 pages for database 'Search_AdminDB_CrawlStore', file 
'Search_AdminDB_CrawlStore_log' on file 1.
[Date/Time] Progress: [Search_AdminDB_LinksStore] 95 percent complete.
...

which indicates that the backup process is communicating with the backend just fine and thus this SQL Server interaction is not associated with the failure.  Looking farther down, towards the completion section of the log, you see errors occuring again:

...
[Date/Time] FatalError: Object Search_AdminDB failed in event
OnBackupComplete. For more information, see the spbackup.log or 
sprestore.log file located in the backup directory.
 Aborted due to error in another component.
[Date/Time] Verbose: Starting object: Search_AdminDB_CrawlStore.
[Date/Time] FatalError: Object Admin (C: on [Search Host Server]) 
failed in event OnBackupComplete. For more information, see the spbackup.log 
or sprestore.log file located in the backup directory.
 Aborted due to error in another component.
...

Lastly, looking at the application event log on the Search Service host server, you see the following timer event error:

Log Name:      Application
Source:        Microsoft-SharePoint Products-SharePoint Foundation
Date:          [Date/Time]
Event ID:      6398
Task Category: Timer
Level:         Critical
Keywords:      
User:          [Farm Account]
Computer:      [Search host server]
Description:
The Execute method of job definition Microsoft.SharePoint.Administration.Backup.SPBackupRestoreJobDefinition (ID 26eac21f-d391-4024-a82f-8ee0b5738ba3) threw an exception. More information is included below.
The backup job failed. For more information, see the error log that is located in the backup directory.
Event Xml:
...

Solution

1. Identify the Search Service Account.
2. Navigate to the intranet folder to which you intend to write full farm backups
3. Grant the Search Service Account the following permissions to this folder:
1. Read
2. Write

References

1. SharePoint 2013 Farm Backup Error : Object Search Service Application failed in event OnBackup. Could not send chunk to target BackupDirectoryTarget
2. Plan for administrative and service accounts in SharePoint 2013
3. Account permissions and security settings in SharePoint 2013
4. Create and configure a Search service application in SharePoint Server 2013
5. Configure backup and restore permissions in SharePoint 2013
6. Back up Search service applications in SharePoint 2013

Notes

• General: the issue involves service accounts and permissions to the destination backup folder. 
• Backup log file: this will be found in the same folder containing the farm backup files.  For example, if your backup folder is SharePointBackup, inside this folder you will find a number of subfolders, each corresponding to a previous farm backup, numbered: spbr0000, spbr0001, spbr0002 and so on.  Open the most recent one to view the log of the most recent backup.
• Search Service Account Access to Backup: this is new from 2010.  For 2010, you had to grant access to the SQL Server Service Account.  I was not able to find any discussion on this among all of the expected Microsoft documentation resources (see references 2 - 6).  Thanks to Amol Meshe for resolving this one.

SharePoint 2013: Product Configuration Wizard stuck on task 9 of 10

Problem

You have installed SP1 or some other cumulative update on all your farm servers without issue.  You then run the SharePoint Products Configuration Wizard on each machine.  It completes without issue on one or two machines, but then, on the next machine, it nearly completes but then appears to remain stuck on configuration task 9 of 10.  You wait an hour, but it still remains stuck on task 9 of 10.  You then check the Upgrade Status page in Central Administration

Solution

1. Delete the SharePoint Products Configuration Wizard dialog.  Terminate the process of necessary to remove it.
2. Open the Services.msc panel, and then look for the SharePoint Timer Service.
3. Stop this service.
4. Open Windows Explorer, and then navigate to the cache folder at: C:\ProgramData\Microsoft\SharePoint\Config.
5. Look for the most recent cache folder, and then open it.
6. Take note of how many configuration files are in this folder.  For example, looking in this folder for one of my WFEs, I see 1736, including the cache.ini.
7. Delete all files in this folder EXCEPT cache.ini.
8. Open the cache.ini file in a text editor, and then randomly modify the number, but keep it at the same number of digits.
9. Save the cache.ini file.
10. Start the SharePoint Timer Service.
11. Now watch the configuration folder.  It will start filling up with new configuration cache files quickly.  When it reaches the number you noted down in step 6, and you no longer see any new files being created, proceed to the next step.
12. Open a command prompt as Administrator.
13. Run the following command: Psconfig.exe -cmd upgrade -inplace b2b -wait -force
14. Check the Upgrade Status.

References

• Command-line reference for the SharePoint Products and Technologies Configuration Wizard (Office SharePoint Server)
• SharePoint 2013 Product configuration wizard failing at step 9/10
• sharepoint 2013 - After SP1 installation - Sharepoint products Configuration Wizard stuck

Notes

• Upgrade Status page: CA > Upgrade and Migration > Upgrade and Patch Management > Check upgrade status.
• Cache folder: You may get a Folder not found error trying to navigate to this folder.  If so, try navigating first to one of the higher tier folders, then double-clicking on each subfolder in turn.
• The path to psconfig is here: c:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\15\BIN\.
• Seems stuck on 10%: I've experienced this several times over the years.  Before determining that the upgrade process is actually experiencing problems, just do a simple check of the Upgrade log.  Check to see what the most recent time stamp is: if its quite recent then the upgrade process is likely moving forward successfully.  Check back again a moment later: if you see a new time stamp then again it is likely that the upgrade process is successfully moving forward and is just not updating the percentage complete in a meaningful way.  
• Actually stuck on 10%: then there is the case where you check back after awhile - maybe after a long while - and no new time stamp entry has been added to the Upgrade log - even after an hour or two.  In this case, there might be a problem.  Be sure to review the Upgrade error log to see if there are any unusual errors.  In this case, which I have experienced just once, fortunately.  This occurred during an upgrade launched by this command: PSConfig.exe -cmd upgrade -inplace b2b -force -cmd applicationcontent -install -cmd installfeatures.  Nothing unusual seemed to occur until it reached Step 5 of 6.  Checking the upgrade log found no issues; there was an upgrade error log generated, but the errors logged were the pesky "web application is configured with claims authentication mode however the content database you are trying to..." warnings that I ignore.  I waited an hour and then followed the procedure above, and I was able to successfully complete the upgrade. 
• Force upgrade to end with error: when it seems like the upgrade is stuck, you can force it to end, albeit with error, by stopping the timer and then restarting it.

SharePoint 2013: The process was terminated due to an unhandled exception

Problem

You have a SharePoint Server 2013 farm, with one application and two web front end (WFE) servers in a traditional topology.  The farm servers are VMs hosted on Hyper-V.  Farm servers have both private and external NICs configured.  The private network is limited to farm servers.  A HOST file is used to enable private network routing among farm servers.  The Distributed cache service is running on the application server and the two WFEs.  You review the Application Event log for one of your web front end (WFE) servers and find the following set of events appearing in the log on an hourly basis:

Log Name:      Application
Source:        Application Error
Date:          [date/Time]
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      [WFE]
Description:
Faulting application name: WerFault.exe, version: 6.2.9200.16659, time stamp: 0x51db3bf4
Faulting module name: wer.dll, version: 6.2.9200.16384, time stamp: 0x501081cc
Exception code: 0xc0000005
Fault offset: 0x0000000000021fe5
Faulting process id: 0x318c
Faulting application start time: 0x01cf62c15e954491
Faulting application path: C:\Windows\system32\WerFault.exe
Faulting module path: C:\Windows\system32\wer.dll
Report Id: 9d3c283d-ceb4-11e3-9405-00155d38891a
Faulting package full name: 
Faulting package-relative application ID: 
Event Xml:
...

and

Log Name:      Application
Source:        Application Error
Date:          [date/time]
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      [WFE]
Description:
Faulting application name: DistributedCacheService.exe, version: 1.0.4632.0, time stamp: 0x4eafeccf
Faulting module name: KERNELBASE.dll, version: 6.2.9200.16815, time stamp: 0x52f2ca60
Exception code: 0xe0434352
Fault offset: 0x00000000000264a8
Faulting process id: 0x3588
Faulting application start time: 0x01cf62c03cd277d1
Faulting application path: C:\Program Files\AppFabric 1.1 for Windows Server\DistributedCacheService.exe
Faulting module path: C:\Windows\system32\KERNELBASE.dll
Report Id: 9c501e53-ceb4-11e3-9405-00155d38891a
Faulting package full name: 
Faulting package-relative application ID: 
Event Xml:
...

and

Log Name:      Application
Source:        .NET Runtime
Date:          [date/time]
Event ID:      1026
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      [WFE]
Description:
Application: DistributedCacheService.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: Microsoft.ApplicationServer.Caching.DataCacheException
Stack:
   at Microsoft.ApplicationServer.Caching.VelocityWindowsService.StartServiceCallback(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

Event Xml:
...

You check the Distributed Cache service on the WFEs through Manage Services on Server in Central Administration and see that the services are running on all servers. You then open a SharePoint Management Shell as administrator on one of the WFEs, and you run Use-CacheCluster and Get-CacheHost to check the status of the cache hosts.  The status shows the application server status as UNKNOWN, the local server as UP and the other WFE as UNKNOWN.  You then remote into the application server and repeat the powershell commands, and this time the status is: application server UP and the two WFEs UNKNOWN.  A similar result is experienced for the second WFE.

Solution

1. Check the HOST file on each of the farm servers and verify that the host names in the file are fully qualified.

References

1. Cache Administration with Windows PowerShell (AppFabric 1.1)
2. AppFabric 1.1 caching service crashes with System.UriFormatException: Invalid URI: The hostname could not be parsed
3. AppFabric Event ID 1000 and Event ID 1026 with SharePoint 2013
4. Raw error past data on PASTEBIN by Anonymous
5. AppFabric Caching and SharePoint: Concepts and Examples (Part 1)

Notes

• The primary posting that helped solve this was [3]. Hat tip to the experts at Sterling International Consulting Group for identifying this one as it relates to static private networking.

SharePoint 2013: Critical Event 6398: The Execute method of job definition Microsoft.SharePoint.Administration.SPAppInstallationJobDefinition... threw an exception

Problem

You review the Application event log on one of your SharePoint Server 2013 farm web front end (WFE) servers and find the following critical event occuring every five minutes:

Log Name:      Application
Source:        Microsoft-SharePoint Products-SharePoint Foundation
Date:          [Date/Time]
Event ID:      6398
Task Category: Timer
Level:         Critical
Keywords:      
User:          [FarmAccount]
Computer:      [WFE1]
Description:
The Execute method of job definition 
Microsoft.SharePoint.Administration.SPAppInstallationJobDefinition 
(ID 16bfab57-4403-44da-9092-37f0f81fb32e) threw an exception. More 
information is included below.

Access to the path 'C:\ProgramData\Microsoft\SharePoint\AppInstallation'
is denied.
Event Xml:
...

Solution

1. Open Windows Explorer, and navigate to C:\ProgramData\Microsoft.
2. Right-click the SharePoint subfolder, and then choose Properties.
3. Select the Security tab.
4. Take  note of any entries here that display as SIDS only.
5. Perform SID lookup, both against AD and against the local machine.
6. Verify that the following two user groups are added and that they have been configured with the following permissions:
1. WSS_ADMIN_WPG
1. Full Control
2. WSS_WPG
1. Read & Execute
2. List folder contents
3. Read
7. Remove any corrupted accounts/groups that appear here (you will see their SIDS only).
8. Reboot the server.

References

• Sharepoint Execute method, Access denied to AppInstallation not clear service accounts
• Well-known security identifiers in Windows operating systems
• Windows PowerShell Tip of the Week: Determining the SID for a Local User Account
• Account permissions and security settings in SharePoint 2013

Notes

• I have found that performing a full re-installation of SharePoint Server 2013 (including configuration) seems to sometimes cause corruptions among the local user groups that SharePoint configures and that existing user groups are not properly cleaned up or overwritten when performing a re-installation.
• I have found that this critical event can be temporarily resolved by adding the (AppFabric) Distributed Cache service account to the local Administrators group on each server hosting the Distribute Cache service.  I had to reboot the server after doing this, or the new account privileges was not realized.
• Through investigation, the SIDS that I found here did not match up with the current SIDS for the WSS_ADMIN_WPG and WSS_WPG user groups already configured on the WFEs (the SIDS will be different from WFE to WFE - these are local user groups).  I think that these orphaned SIDS may reflect earlier SIDS for these same user groups from a previous installation.
• The farm service account is a member of the WSS_WPG and WSS_ADMIN_WPG local user groups.

SharePoint 2013: Unable to change topology when Generation controller is not active

Problem

You are attempting to modify the existing farm search topology by adding a second WFE and adding index and query components to this second WFE.  Within your SharePoint Management PowerShell window, you clone the current enterprise search topology, and make the appropriate modifications, but when you then try to activate it, you are presented with the following error: 

Exception calling "Activate" with "0" argument(s): "Topology activation failed. Management called failed with System.InvalidOperationException: 'Unable to change topology when Generation controller is not active' at at Microsoft.Ceres.SearchCore.IndexController.IndexController.IsEmptyIndex(String indexSystemName) at Microsoft.Ceres.SearchCore.IndexController.IndexControllerManagementAgent.WrapCall[T](Func`2 original)" At line:1 char:1 + $Clone.Activate() + ~~~~~~~~~~~~~~~~~ + CategoryInfo : NotSpecified: (:) [], MethodInvocationException + FullyQualifiedErrorId : SearchTopologyActivationException SharePoint 2013 Unable to change topology when Generation controller is not active

You then check the Search Service Topology and note that the index component on the WFE currently hosting the index and query components is in a degraded state.  This is the only index component for the farm.  You then engage in troubleshooting.

Troubleshooting

1. Action: attempted to reset index.
• Result: would not complete.
2. Action: attempted to delete Search Service Application
• Result: would not complete.
3. Action: perform steps as in reference [2], then removed and re-installed the Search Service application.
• Result: was able to perform a delete of the Search Service Application and then build a new search service application.
4. Action: viewed search topology on Search Service Administration page.
• Result: index partition no longer displays in degraded state.

Solution

1. Get out of any still running index reset or search application deletion prompts that display that they are still busy.
2. If you are not there already, remote into the farm server hosting the search service.
3. Stop the SharePoint Timer Service on this server.
4. Open up Windows Explorer, and then navigate to C:\ProgramData\Microsoft\SharePoint\Config\.  You'll see one or more folders at this location named by a GUID.  Look for the most recent one. 

   Note: this posting assumes that the SharePoint 2013 binaries were installed default (to the C: drive).

5. Open this folder.  You will see a number of XML files. 
6. Delete all XML files in this folder only.
7. Now look for the file cache.ini.  It should be the only one remaining.
8. Open this file in a text editor.  It will contain a single six-digit number.
9. Edit this number randomly, but keeping it at six-digits.
10. Save the file.
11. Restart the SharePoint Timer Service.
12. Now delete the Search Service Application.
13. Rebuild the Search Service Application.

References

1. Create new Index Component and add it to a clone topology?
2. SharePoint 2013 Unable to change topology when Generation controller is not active
3. An update conflict has occurred, and you must re-try this action. The object SearchServiceApplication Name={FAST SSA} was updated by {account}, in OWSTIMER (5836) process, on machine {server name}.
4. One or more servers is not responding (SharePoint Foundation 2010)

Notes

• If your site displays an HTTP 500 Internal Server error page, after performing these stops (specifically, stopping and starting the SharePoint Timer Service), check IIS to verify that all application pools are started on the WFE you were working on.

SharePoint 2013: My Site content is not being crawled

Problem

You have deployed a new Search Service Application and launched a full crawl of all content among all sites, including your farm My Site instance.  On reviewing the crawl log, you see the following warning associated with My Site crawl:

This item and all items under it will not be crawled because the owner has set the NoCrawl flag to prevent it from being searchable.

You're not sure where this setting is.

Solution

1. In any browser, navigate to the root My Site URL.
2. From the Settings gear icon (top right corner), click Site Settings.  The Site Settings page is displayed.
3. In the Search group, look for Search and offline availability.
4. Click this link.  The Search and Offline Availability page is displayed.
   
5. For the Indexing Site Content setting, select Yes.
6. Click OK.
7. Start a new crawl.

References

• This item and all items under it will not be crawled because the owner has set the NoCrawl flag to prevent it from being searchable
• Enable content to be searchable

Notes

• This setting used to be located in the Site Administration group in SharePoint Server 2010.

SharePoint 2013: Built-in accounts are used as application pool or service identities

Problem

You find the following entry in the SharePoint 2013 Central Administration, Review problems and solutions, All Reports listing:

Title	Built-in accounts are used as application pool or service identities.
Severity	2 - Warning
Category	Configuration
Explanation	Using built-in accounts like Network Service or Local System as application pool or as service identities is not supported in a farm configuration.  The following services are currently running as built-in identities on one or more servers: c2wts(Windows Service)
Remedy	Browse to  .../_admin/FarmCredentialManagement.aspx and change the account used for the services listed in the explanation. For more information about this rule, see "http://go.microsoft.com/fwlink/?LinkID=142699".
Failing Servers
Failing Services	SPTimerService (SPTimerV4)
Rule Settings	View

  
Solution

1. Determine the SharePoint service account that you want to use as the identity for the Claims to Windows Token Service.
1. This service account should be a domain account with standard user privileges. 

   For my environments, I generally use the services account (eg, spService) as the identity for C2WTS.

2. If your environment is locked down, ensure that the network GPO is modified to grant the service account the Impersonate a client after authentication and Log on as a service local user rights assignments.
3. Using the SharePoint Setup/Administrator account (eg, spadmin), launch Central Administration, and then navigate to Security > Configure service accounts.
4. From the upper dropdown, select Windows Service - Claims to Windows Token Service.
5. From the lower dropdown, select the desired service account.
6. Click OK.
7. Check the Events log on the server on which the service is started to ensure that this account re-assignment did not generate any errors.

References

1. Claims to Windows Token Service (c2WTS)
2. c2wts - Windows Service account SharePoint Server 2010
3. Built-in accounts are used as application pool or service identities (SharePoint Foundation 2010)
4. How to configure the Claims to Windows Token Service (C2WTS) for SharePoint 2013
5. Local Policy Settings
6. SharePoint 2013: Service Account Configurations and Permissions

Notes

• Remarkably, the SharePoint 2013 Health Analyzer rules reference does not include an entry for this issue, though the 2010 rules reference does.  A link to the appropriate rule reference for this issue is provided in the References section.
• Reference 4 indicates that the C2WTS service account should have the Act as part of the operating system user right assignment in local policy.  However, in its Local Policy Settings reference, Microsoft states that,

  This user right is extremely powerful; anyone with this right can take complete control of the computer and erase virtually any evidence of their activities.
  
  Limit the Act as part of the operating system user right to as few accounts as possible—it should not even be assigned to the Administrators group under normal circumstances. When a service requires this user right, configure the service to log on by using the local System account, which has this user right inherently. Do not create a separate account and assign this user right to it.
  
  This user right is rarely needed by any accounts other than the local System account.

  Thus, it's not clear to me that the C2WTS service account actually requires this user right assignment.  The other two rights assignments, Impersonate a client after authentication and Log on as a service, are common assignments made to service accounts by the SharePoint Products Configuration Wizard, and thus I have no difficulty with these.  More conclusively, I have set a domain account to run C2WTS that does not have the Act as part of the operating system user right assignment, and the service ran just fine without generating any events on the server.

SharePoint 2013: verifying workflow setup returns HTTP 403 Forbidden error

Problem

You have completed installation and configuration of Workflow Manager 1.0 to the farm server hosting this application and workflow manager clients to the web front end (WFE) servers.  You now attempt to verify correct setup.  You launch a browser, and then try to connect to the fully qualified domain name of the server + :12290/, as indicated.  You then experience the error:

Solution

1. Launch IE as administrator, and then connect to the URL.
2. In the alternative, first launch Central Administration, and then connect to the URL using the same or different tab of this browser instance.

References

• Troubleshooting Workflow Manager 1.0 Management and Execution

Notes

• For this posting, Workflow Manager 1.0 was installed to a farm application server.
• The farm is installed to Windows Server 2012 VMs.

SharePoint 2013: 500 Internal Server Error and Security Token Service Failure

Problem

You connect to the primary website for your customers and experience the following response:

You check farm server logs, and, on one of the web front ends (WFE), you see application errors like the following:

Log Name:      Application
Source:        Microsoft-SharePoint Products-SharePoint Foundation
Date:          [date/time]
Event ID:      8306
Task Category: Claims Authentication
Level:         Error
Keywords:      
User:          [search or other service account]
Computer:      [name of problematic WFE]
Description:
An exception occurred when trying to issue security token: The server was 
unable to process the request due to an internal error.  For more 
information about the error, either turn on IncludeExceptionDetailInFaults 
(either from ServiceBehaviorAttribute or from the servicedebug configuration 
behavior) on the server in order to send the exception information back to 
the client, or turn on tracing as per the Microsoft .NET Framework SDK 
documentation and inspect the server trace logs..
Event Xml:
...

This error may feature various service accounts due to different farm services being affected.  You then connect to Central Administration, review the health report, and see the following rule violation:

The Security Token Service is not available

and the server triggering this rule violation is one of the farm's WFEs.

Solution

1. Remote into the problematic WFE.
2. Open IIS Manager, and then navigate to Application Pools.
3. View the state of the SecurityTokenServiceApplicationPool.
• If it is started, stop and restart it.
• If it is stopped, start it.
4. Open a command prompt as administrator.
5. Perform an IISReset /NoForce
6. Connect to the primary customer site again.

References

• The Security Token Service is not available (SharePoint 2013)
• How to resolve HTTP 500 Internal Server Error in SharePoint 2013 site
• SharePoint Server Check V1.1

Notes

• About this posting: this posting consolidates my notes regarding troubleshooting and resolving one cause of an HTTP 500 error.  In this scenario, the HTTP 500 error is associated with an issue involving the Security Token Service.
• Central Admin may not be affected: If you don't host CA on one of your WFEs, it won't be affected by this issue.  This is the case here, where the farm's Central Administration site is not hosted on one of the WFEs (it employs the traditional rather than streamlined topology).  This scenario helps demonstrate why you may want to choose the traditional topology over a streamlined one: had CA also been hosted on one of the WFEs, it too would have been affected by the SecurityTokenService issue and would not have been available to provide useful troubleshooting information.
• What to do if it's not Security Token Service: If you're looking at this posting, and your HTTP 500 Internal Server experience is not associated with a Security Token Service Failure, then glance over this checklist:
  • In Services, verify SharePoint Timer Service (SPTimerV4) is running
  • In IIS, verify website is started
  • In IIS, verify SharePoint Web Services Root application pool is started.
  I have experienced that if the SharePoint Web Services Root application pool is stopped on one of the WFEs, even though they're in NLB configuration, this will bring down the entire website.  I don't understand why, and I haven't had the time to explore it further.
• Some users see HTTP 500 but other are still able to access: farm topologies employing some form of network load balancing (NLB) will present seemingly inconsistent experiences here to the administrator trying to troubleshoot.  This is due to NLB routing some requests to the failing WFE and other requests to the remaining healthy WFEs.
• PersistedNavigationTermSetSyncJobDefinition failure: if you are using managed navigation, and the security token service fails on one of the farm WFEs, you may see this error appearing in the server log at fairly regular intervals (eg, every 15 minutes or so):

  Log Name:      Application
  Source:        Microsoft-SharePoint Products-SharePoint Foundation
  Date:          [date/time]
  Event ID:      6398
  Task Category: Timer
  Level:         Critical
  Keywords:      
  User:          [farm service account]
  Computer:      [WFE on which security token service is failing]
  Description:
  The Execute method of job definition Microsoft.SharePoint.Publishing.Internal.
  PersistedNavigationTermSetSyncJobDefinition 
  (ID 5cfde201-6fd8-4ee3-8920-95a7d017a22f) 
  threw an exception. More information is included below.
  
  The server was unable to process the request due to an internal error.  
  For more information about the error, either turn on 
  IncludeExceptionDetailInFaults (either from ServiceBehaviorAttribute or 
  from the servicedebug configuration behavior) on the server in order 
  to send the exception information back to the client, or turn on tracing 
  as per the Microsoft .NET Framework SDK documentation and inspect the 
  server trace logs.
  Event Xml:
  

  and this one also this one, though it may occur at varying times:

  Log Name:      Application
  Source:        Microsoft-SharePoint Products-SharePoint Server
  Date:          [date/time]
  Event ID:      8088
  Task Category: Taxonomy
  Level:         Warning
  Keywords:      
  User:          NT AUTHORITY\IUSR
  Computer:      [problematic WFE]
  Description:
  The Managed Metadata Service 'Managed Metadata Service' is inaccessible.
  Event Xml:
  ...
  

• SharePoint Check: I developed a simple PowerShell script that interrogates farm services, IIS features and other services and presents the results listed in the shell.  One thing it returns is the state of the IIS application pools.  

SharePoint 2013: Service Account Configurations and Permissions

Introduction

Below are my notes in tabular form for SharePoint Server 2013 service account configurations and permissions.  The data presented here is drawn from: 1) cited references, 2) Microsoft documentation, 3) blog postings, 4) local security policy settings data exports, 5) SharePoint ULS log messages and 6) trial and error.  Such settings as these are normally configured by the SharePoint 2013 Products Configuration Wizard on the local machine.  However, in locked down environments, these settings will be overwritten by GPO.  Therefore, in locked down environments, you will need to have these settings added to the GPO.  Bolding settings are absolutely essential and deployment will fail without them.  The non-bolded settings will not stop deployment or operations, but will affect performance and have other affects.

Service Account Configurations and Settings

Service Account	Suggested Name	Domain Permissions	Local Permissions [see notes 1,2,3]
User administrator	spAdmin	User	Local Admin
Farm service	spFarm	User AD Read for all service accounts [see note 5]	Local Admin [see note 4] Allow log on locally Adjust memory quotas for a process Impersonate a client after authentication Log on as a batch job Log on as a service Replace a process level token
Web Application Pool	spApp	User	Impersonate a client after authentication Log on as a batch job Log on as a service
Services Pool	spService	User	Adjust memory quotas for a process Log on as a batch job Log on as a service Replace a process level token [see note 9]
Search	spSearch	User [see note 8]	Adjust memory quotas for a process Impersonate a client after authentication Replace a process level token
SQL Server	spSQL	User AD Read for workflow service account [see note 6] and SQL backup service account [see note 7]	Adjust memory quotas for a process Bypass traverse checking Log on as a service Replace a process-level token
Workflow	spWorkfl	User [see note 10]	Log on as a service Impersonate a client after authentication Allow logon locally Logon as a batch job
Super User	spSuperU	User	None
Super Reader	spSuperR	User	None
AD profile access	spProfile	User Replicate Directory Changes	None

References

• Account permissions and security settings in SharePoint 2013
• SharePoint Portal Server 2003: Changing Access Accounts and Passwords
• SQL Server 2000: Changing Access Accounts and Passwords
• SQL Server 2012: Configure Windows Service Accounts and Permissions
• Privilege Constants
• User Rights Assignment
• Services permissions
• Grant User Rights to a Service Account
• Plan for administrative and service accounts in SharePoint 2013
• SharePoint 2013: Processing this item failed because of an unknown error when trying to parse its contents
• SharePoint 2013: - Failed to launch sandbox process, SE_ASSIGNPRIMARYTOKEN_NAME and SE_INCREASE_QUOTA_NAME privileges are require
• SharePoint 2013: An exception occurred when trying to establish endpoint for context: Could not load file or assembly...
• How to setup SharePoint 2010 user profile synchronization and MySites; search for "rights"
• TechNet: User Rights: Windows Server 2008
• TechNet: User Rights Assignment: Windows Server 2012
• SharePoint 2013 Service Accounts Best Practices Explained: interesting somewhat different perspective than mine - good to get different views on this so as to more informatively craft the best policy for your own situation.

Notes

1. If your AD environment doesn't control user rights assignments via GPO, you don't need to configure these yourself as they are done automatically for you by the SharePoint Products Configuration Wizard.
2. If you're not sure what user rights assignments are being configured, you can determine them by: 1) performing a configuration change and then 2) opening the local Security Policy on the local machine (most likely the Application or Batch server), and then viewing User Rights Assignments - do an export to save these for reference. Repeat this after every new application and service instantiation and you will build up a pretty good understanding of what is being configured here. NOTE: if you have GPO pushed out regularly, you will need to view and export these assignments quickly - before the GPO is pushed out again, as this will overwrite these assignments.
3. SharePoint Service Account User Rights Assignments: I have not been able to find any definitive, office Microsoft presentation of Local Security Policy user rights assignments for SharePoint 2007, 2010 or 2013. I did find them for SharePoint Portal Server 2003. Microsoft's TechNet pages for account permissions and security setting for SharePoint 2007, 2010 and 2013 do not discuss user rights assignments at all. On the other hand, Microsoft does provide these critical settings for another flagship application, SQL Server, doing so from SQL Server 2000 through 2012. Thus, obivously, Microsoft recognizes the importance of these settings to its applications. It's thus very unclear why Microsoft has failed to disclose these critical settings for SharePoint, as these settings play just as critical a role in the successful deployment and performance of SharePoint as they do for SQL Server.
4. The farm service account (aka, database access account) only needs to be a member of the local admin group during installation and configuration tasks.  During normal operation, the farm service account does not need to be a local administrator.
5. AD Read.  This Service Permission grants the ability to the farm service account, or any other account, to be able to read critical account security information, such as account restrictions.  Note that this permission is actually composed of a number of sub-permissions, including the Read account restrictions permission, which is critical.  I'm not presently sure which other specific read sub-permissions are necessary, so, simply selecting Read covers all the bases.  I will update this posting when I know more.
6. This permission is needed for the successful deployment of SharePoint Server 2013 Workflow Manager.  This permission should be granted to whatever account you have running the SQL Server Service, be it Local Service or what not.  If you do not grant this permission, your Workflow Manager configuration will fail on the first step with error message:

   Could not obtain information about Windows NT group/user '{your workflow service account}', error code 0x5.

   When you then look at the log file for the workflow deployment, you see this conclusive message:

   ...System.Data.SqlClient.SqlException: Could not obtain information about Windows NT group/user '{your workflow service account}', error code 0x5...

7. This is the account that you used to login to SQL Server and create the scheduled farm database maintenance plans.  For example, if you logged into SQL Server Management Studio as the farm service account, and then created the farm database scheduled maintenance plans while logged in as this account, you would grant the SQL Server service account (eg, spSQL) AD Read permission to the farm service account.  If you do not grant the AD Read permission, you will experience the following SQL Server error when attempting to execute a maintenance plan:

   [date] [time],,Error,[298] SQLServer Error: 15404<c/> Could not obtain 
   information about Windows NT group/user [Account you are logged in as
   ]<c/> error code 0x5. [SQLSTATE 42000] (ConnIsLoginSysAdmin)

8. There are a number of postings concerning the user rights assignments necessary in order for the search service to run properly.  Read this posting to find more references on this topic.
9. Without these user rights, the application pool run by this account will not remain started.
10. Verify that the SharePoint Setup User Administrator, SQL Server and Farm Service accounts have AD Read permission to the service account that will be used to run Workflow Manager.

SharePoint 2013: drag files here disabled

Problem

You are remoted into a farm server.  You connect to a SharePoint 2013 website while in this remote session, and then navigate to a document folder.  You then attempt to drag a file onto the folder, but the drag files here option seems disabled.

Solution

1. Verify that the Desktop Experience feature has been setup on the server [1,2].
2. If it is already, then check the browser type and version you are using on the server [3].

References

1. Enable Desktop Experience on Servers to Access SharePoint Web Folders
2. Desktop Experience Overview
3. SharePoint Server 2013: Drag and drop contents to Library

SharePoint 2013 Visio Service: the server failed to process the request

Problem

A user reports the following error when attempting to view a Visio web drawing file in his My Site document folder:

Solution

• Ensure that the Visio Graphics Service account is mapped as DBO to the My Site content database.

References

• Visio - bug - The server failed to process the request
• Visio Graphics Service administration in SharePoint Server 2013
• Get-SPServiceApplicationPool
• Get-SPServiceApplication

Notes

• If you added My Site after creating a new Visio Graphics Service, the Visio Graphics Service service account is not automatically mapped to the My Site web application content database.
• To find the Visio Graphics Service identity, you can use IIS or Powershell:
• IIS: launch IIS and view the application pools: select an application pool, and then click View Applications.  eventually you'll find the service account running the service.
• PowerShell: First, look in the Service Applications list for the name of your Visio Service. Then execute this script to generate a list of service applications and their associated application pools:
  

  Get-SPServiceApplication -Name "{Your Visio Service name}"

  Note down the application pool associated with the Visio Graphics Service.  Then execute this script to see a list of application pools and their service accounts:

  Get-SPServiceApplicationPool -Identity "{Your Application Pool  Name}"

  Now you know what service account to map as dbo to the My Site content database.

SharePoint 2013 MySite: 404 - The resource cannot be found

Problem

You have just finished deploying a new MySite service to your SharePoint 2013 farm situated in the corporate intranet.  Setting up the web application and creating a new site collection were completed without issue.  Your user profile service works fine, and you completed User Profile configuration for MySite.  However, when you then click on Sites or SkyDrive, you are presented with the 404 - The resource cannot be found... error page.

Solution

1. Verify that the My Site Settings, My Site Host location (URL) matches the URL of the My Site web application.
2. Verify that the My Site Settings, Personal Site Location matches one of the Managed Paths, Wildcard inclusion paths.  For example, if your Personal Site Location is personal (the default), be sure that you have also configured a managed path called personal. 

   The default managed wildcard inclusion path is site.

3. Verify that you have a DNS entry name that matches the name assigned to your My Site web application and that it points to the appropriate server (single-WFE) or cluster (multiple WFE). 

   If you are using a HOST file in preparation for a migration effort, be sure that you have added an entry for the My Site web application.

References

• DNS: The Hosts file <file name> on the DNS server should be empty
• Configure My Sites in SharePoint Server 2013
• How to Configure SharePoint 2013 My Sites

Notes

• Setting the My Site Settings, Personal Site Location to a path that hasn't been configured in Managed Paths, Wildcard inclusion can cause other odd affects, such as:
• Seemingly being able to create your own My Site, but then, when you subsequently click About Me, Sites or Skydrive, it's as if you are creating your My Site fresh for the first time all over again.
• Portions of your My Site page display the usual helpful "Sorry, there's been a problem" error message.

SharePoint 2013: OWA Previews not displaying in FAST Search Results

Problem

You have deployed Office Web Apps (OWA) to your internal Sharepoint 2013 farm situated on the corporate intranet.  The OWA farm does not employ encrypted connections (HTTPS).  Thus, the WOPI zone is internal-http. You have successfully tested the new OWA farm by navigating to its WOPI discovery page.  However, you are not able to view a preview of a document, either in a document library or in a search result.

Solution

1. To get previews to display in document libraries:
1. Ensure that AllowOAuthOverHttp is True.
1. Execute:

   (Get-SPSecurityTokenServiceConfig).AllowOAuthOverHttp

2. If this is False, execute the following:

   $config = (Get-SPSecurityTokenServiceConfig)
   $config.AllowOAuthOverHttp = $true
   $config.Update()

2.  Ensure that document type zones match the OWA farm's WOPI zone
1. First check document type zones by executing:

   Get-SPWOPIBinding
   

2. Next, check the OWA farm zone by executing:

   Get-SPWOPIZone
   

3. If these are different, set the WOPI zone of the OWA farm to match the WOPI zones of the document types.  The default zone for the OWA farm is internal-https.  You'll need to change this to internal-http.  Do this by executing:

   Set-SPWOPIZone -zone "internal-http"
   

2. To get previews to display in search results:
1. Delete the index and rebuild it.  An incremental won't do.
1. Logon as farm administrator
2. In Central Administration, navigate to Search Service Administration.
3. Click Index Reset.
4. After the index reset is completed, start a new full crawl.
5. After the full crawl is completed, perform a new search and then test the preview capability again.

References

• SharePoint 2013 – Office Web Apps Preview Not Showing
• Office Online and FAST Search Document Thumbnail and Preview Scenarios
• SharePoint Server 2013 search results do not display document previews or thumbnails from SharePoint Server 2010 sites
• Deploy Office Web Apps Server
• Configure Office Web Apps for SharePoint 2013
• Get-SPSecurityTokenServiceConfig
• Get-SPWOPIBinding
• Get-SPWOPIZone