Friday, February 28, 2014

SharePoint 2010: How to restore multiple web application content databases

Introduction

This posting walks through the process of performing a restore of multiple content databases associated with a single SharePoint 2010 web application.

Procedure
  1. Copy the content database backups to a local folder on the database server.
  2. Perform a SharePoint detach of the content databases from web application:
    1. Go: CA > Application Management > Manage Content Databases.
    2. Remove each database.
  3. Detach old/corrupted content database(s) (SQL Server Management Studio)
  4. Restore backup content database(s) (SQL Server Management Studio)
  5. Add farm service and farm administration accounts to database logins and map appropriately (SQL Server Management Studio)
  6. Perform a SharePoint attach of the content database(s) (through SharePoint Central Administration or via PowerShell script)
  7. Refresh browser.
References
Notes
  • There really is nothing more to do, when restoring multiple content databases associated with a web application - no special procedure, script to run, special order, etc. You don't even need to perform an IISReset. Just attach them and go.  SharePoint will detect that multiple databases are associated with the web application and relink everything appropriately.
  • You may find, after attaching the content databases (either via CA or via PowerShell script), that the site collection(s) in one of the content databases are not visible, and navigating to the URL for one of these site collections returns the "404" error.  The basis for this experience is unclear but can be resolved by performing a SharePoint detach/attach of these databases in CA.  You may need to repeat this several times.
    • TIP: the following procedure has worked for me when I encountered this problem: first perform the SharePoint attach of the content database containing the main site collection - the one hosting the default site collection.  Next, before performing SharePoint attachments of the other content databases, open a browser and navigate to the default site collection.  Then, perform a SharePoint attach of the next content database, open a browser and navigate to a site collection in this next content database; and so on.
  • Performing SharePoint attachments of content databases by using Central Administration may require a database upgrade afterwards.  On the other hand, using the Mount-SPContentDatabase PowerShell script automatically performs the database upgrade as a part of the attach process.
  • I have performed a restore in this fashion, for a farm hosting three site collections and about 150 sites, in about 1 hour, and that's with lots of double-checking.
  • I've bolded SharePoint when referring to performing an attachment through SharePoint Central Administration or via PowerShell script, as opposed to performing such operations directly with SQL Server.  The terminology here is a bit ambiguous, as these words are also used when performing attach/detach operations when working with SQL Server.  However, the two are effectively different operations.

Wednesday, February 26, 2014

SharePoint 2007: The Search Application is not provisioned

Problem

On navigating to SSP, and going to Search Administration, I saw this message for Crawl status:
The search application for [SearchAppName] on server [ServerName] is not provisioned.  Confirm that the Windows SharePoint Services Timer service and Windows SharePoint Services Administration service are running on the server.
This was a legacy SharePoint farm that I had been assigned management for.  Farm was operational and accessible by users.  However, my review found Search nonoperational.  Performed standard checks.  Cited references were helpful in focusing resolution efforts productively.

Troubleshooting
  1. Check Windows SharePoint Services Timer Service
    • Result: started; logon: admin account.
  2. Check Windows SharePoint Services Administration service
    • Result: started (obviously); logon: local system
  3. Check Office SharePoint Server Search service (through CA)
    • Result: started; logon: admin account
  4. Stop/Start Office SharePoint Server Search service
    • Result: success
  5. Check Shared Services Provider configuration
    1. Result: Index Server not configured.
    2. Action: set Index Server; re-entered SSP Service Credentials password; clicked OK.
  6. Check Search Administration
    1. Result: experienced small delay in loading page (good sign); on load, crawl status indicating crawling.  Successful resolution of issue. 
References

Tuesday, February 18, 2014

SharePoint 2010: Health Analyzer: The Security Token Service is not available

Problem

You find the following entry in the SharePoint 2010 Central Administration Review problems and solutions All Reports listing:

TitleThe Security Token Service is not available
Severity2 - Warning
CategoryAvailability
ExplanationThe Security Token Service is not issuing tokens. The service could be malfunctioning or in a bad state.
RemedyAdministrator should try to restart the Security Token Service on the boxes where it is not issuing tokens. If problem persists, further troubleshooting may be available in the KB article. For more information about this rule, see "http://go.microsoft.com/fwlink/?LinkID=160531".
Failing Servers[NameOfFailingServer]
Failing ServicesSPSecurityTokenService (SecurityTokenService)
Rule SettingsView
 
And looking in the Windows Server Application log, you see the following error event occuring once every 15 minutes:
Log Name:      Application
Source:        Microsoft-SharePoint Products-SharePoint Foundation
Date:          [date/time]
Event ID:      8306
Task Category: Claims Authentication
Level:         Error
Keywords:     
User:          [domain\FarmServiceAccount]
Computer:      [NameOfFailingServer]
Description:
An exception occurred when trying to issue security token: Could not 
connect to 
http://localhost:[port]/SecurityTokenServiceApplication/securitytoken.svc/actas. 
TCP error code 10061: No connection could be made because the target 
machine actively refused it 127.0.0.1:[port]. .
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-SharePoint Products-SharePoint Foundation" 
Guid="{6FB7E0CD-52E7-47DD-997A-241563931FC2}" />
    <EventID>8306</EventID>
    <Version>14</Version>
    <Level>2</Level>
    <Task>47</Task>
    <Opcode>0</Opcode>
    <Keywords>0x4000000000000000</Keywords>
    <TimeCreated SystemTime="[date/time]" />
    <EventRecordID>29898</EventRecordID>
    <Correlation ActivityID="{4F0A4944-93C2-44D8-9C81-68162604774B}" />
    <Execution ProcessID="7128" ThreadID="3156" />
    <Channel>Application</Channel>
    <Computer>[NameOfFailingServer]</Computer>
    <Security UserID="S-1-5-21-3026233045-20759957-1393672501-82644" />
  </System>
  <EventData>
    <Data Name="string0">Could not connect to 
http://localhost:[port]/SecurityTokenServiceApplication/securitytoken.svc/actas. 
TCP error code 10061: No connection could be made because the target machine 
actively refused it 127.0.0.1:[port]. </Data>
  </EventData>
</Event>
Solution
  1. Solution A: Restart the Security Token Service application pool:
    1. As Administrator, launch IIS Manager.
    2. In the Connections pane (at left), expand the tree to view Application Pools.
    3. Select Application Pools.  This updates the results pane (center pane).
    4. Right-click on SecurityTokenServiceApplicationPool, and then click Stop.
    5. Right-click on SecurityTokenServiceApplicationPool, and then click Start.
    6. Go back to Central Administration, Review problems and solutions :All Reports view.
    7. Click on the issue, The Security Token Service is not available.
    8. Click Re-analyze Now.
    9. Click Close.
    10. Refresh after several minutes.
  2. Solution B: Re-provision the service
    1. Launch the SharePoint Management Shell as Administrator.
    2. Run the following script:
      $sts = Get-SPServiceApplication | ?{$_ -match "Security"}
      $sts.Status
      $sts.Provision()​
      
    3. Go back to Central Administration, Review problems and solutions :All Reports view.
    4. Click on the issue, The Security Token Service is not available.
    5. Click Re-analyze Now.
    6. Click Close.
    7. Refresh after several minutes.
References

Saturday, February 15, 2014

SharePoint 2010 Health Analyzer: One or more servers is not responding: SharePoint 2010 Timer Service Stopped

Problem

You find the following entry in the SharePoint 2010 Central Administration Review problems and solutions All Reports listing:

TitleOne or more servers is not responding.
Severity1 - Error
CategoryAvailability
ExplanationThe following servers have not executed any timer jobs in the last 2 hours: [SharePointServer1].  This can happen if the server was shut down or lost network connectivity, or if the timer service on that server has crashed, hung, or been stopped.
RemedyEnsure that the server(s) listed above are running and connected to the network.  If the timer service is not started, restart the service by typing the following command at the command prompt on each server: "net start SPTimerV4."  If the server was intentionally removed from the farm, remove the record of the server from the SharePoint topology in the Central Administration site at http://[pathtoCA]. For more information about this rule, see"http://go.microsoft.com/fwlink/?LinkID=142656".
Failing Servers 
Failing ServicesSPTimerService (SPTimerV4)
Rule SettingsView
 
Troubleshooting
  1. Reviewed job history for SharePointServer1.
    • Result: no job currently scheduled, nor any job completed since midnight.
  2. Remoted into SharePointServer1, opened Services, scrolled to SharePoint 2010 Timer, and then viewed status.
    • Result: service is set to Auto and is stopped.
  3. Started this service and monitored for a short while.
    • Result: service remained started.
  4. Returned to CA, refreshed Job History page for SharePointServer1:
    • Result: new jobs began appearing in history.  All jobs completed successfully except for two, which were repeatedly aborted: Service Application Instance Provisioning Job, Application Server Administration Service Timer Job.
  5. Clicked Aborted status for Service Application Instance Provisioning Job.
    • Result: message: "The administration service job definition 'SecurityTokenServiceApplication' (id 5e0bad8b-948b-460a-8bb2-19da7e10fd9c) was not executed because the administration service on this server is not started. This job definition can be run manually using 'stsadm -o execadmsvcjobs'. "
  6. Clicked Aborted status for Application Server Administration Service Timer Job.
    • Result: message: The administration service job definition 'job-application-server-admin-service' (id 43b74f53-53ba-4eb2-981d-28c13bf012e2) was not executed because the administration service on this server is not started. This job definition can be run manually using 'stsadm -o execadmsvcjobs'.
  7. Remoted into SharePointServer1, opened Services, scrolled to SharePoint 2010 Administration, and then viewed status.
    • Result: service is set to Auto and is stopped.  Also noted that its logon is set to Local System.
  8. Remoted into SharePointServer2 to compare.  Opened Services, scrolled to SharePoint 2010 Administration, and then viewed status.
    • Result: service is set to Auto and is started.  Also noted that its logon is set to Local System.
  9. Remoted into SharePointServer1, opened Services, scrolled to SharePoint 2010 Administration, and then started service and observed.
    • Result: service remained started.
  10. Returned to CA, refreshed Job History page for SharePointServer1, and looked for SharePoint 2010 Timer Service Job status.
    • Result: job repeatedly being completed successfully.
  11. Looked for Service Application Instance Provisioning Job status.
    • Result: this job did not reappear.
  12. Returned  SharePoint 2010 Central Administration Review problems and solutions All Reports listing, and clicked on error.  Clicked Re-analyze Now button.
    • Result: Severity changed to "4 - Success"
Summary

This was a straightforward SharePoint Health Analyzer rule issue to resolve.  It helped to correlate the issue against Windows Server logs.

References
Notes
  • Starting the SharePoint 2010 Administration will generate an error event in the server Application log: EventID 2137: The SharePoint Health Analyzer detected an error.  One or more services have started or stopped unexpectedly.

SQL Server 2008: An exception occurred while enqueueing a message in the target queue

Problem

During routine check of the farm database server event logs, you see the following Application event occuring at the rate of about 77 repetitions per minute:
Log Name:      Application
Source:        MSSQLSERVER
Date:          [date/time]
Event ID:      28005
Task Category: Server
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      [databasename]
Description:
An exception occurred while enqueueing a message in the target queue. 
Error: 15404, State: 19. Could not obtain information about Windows NT 
group/user '[domain\accountname]', error code 0x5.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="MSSQLSERVER" />
    <EventID Qualifiers="49152">28005</EventID>
    <Level>2</Level>
    <Task>2</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="[date/time]" />
    <EventRecordID>1127225</EventRecordID>
    <Channel>Application</Channel>
    <Computer>[databasename]</Computer>
    <Security />
  </System>
  <EventData>
    <Data>15404</Data>
    <Data>19</Data>
    <Data>Could not obtain information about 
Windows NT group/user '[domain\accountname]', error code 0x5.</Data>
    <Binary>656D000010000000100000004F00430053002D00
560053002D004E0041005600530051004C00440031000000070000006D00610073007400650072
000000</Binary>
  </EventData>
</Event>
Troubleshooting
  1. Reviewed server Application log to find start of issue:
    • Results: 33,000 repetitions of event completely flooded log through beginning - log was overwriting itself.  Estimated duration of event thus far: (33K total occurrences in log) / (77 occurrences per cycle) / (~ 63 seconds per cycle) > = 7 hours.
  2. Checked farm user content sites:
    • Result: no issues; users able to access content
  3. Checked Central Administration Health Analyzer Reports:
    • Result: no related issues.
  4. Launched SQL Server Configuration Manager and reviewed log on for SQL Server service: local system account.  Changed this to standard domain account per discussion in this reference.
    • Result: no resolution.
  5. Stopped and restarted SQL Server service
    • Result: no resolution.
  6. Restarted the server:
    • Result: no resolution.
  7. Reviewed the DBOs for all farm databases by using this statement:
    SELECT name, suser_sname(owner_sid) FROM sys.databases
    as discussed in this reference.
    • Result: did not find any NULL values as discusses in the reference, thus not helpful
  8. Checked locked status by executing this statement,
    SELECT LOGINPROPERTY('sa','IsLocked')
    according to this reference:
    • Result: not locked, thus not helpful
  9. Tested ability of problematic account to impersonate using this statement,
  10. EXECUTE AS LOGIN = ‘[yourdomain]\me’
    Go
    as discussed in this reference:
    • Results: failed. Tried again on problematic account: succeeded.  Tried on other service accounts: failed.
  11. Checked databases, service accounts and service account roles on database server
    • Results: found three databases created under (and thus owned by) the farm account, which was unusual, as these are usually created by the admin account. Databases involved the securestore and web analytics services. Neither the admin nor the farm account were mapped to these databases (this accounted for an unrelated error involving failed database upgrade - psconfig).
  12. Checked with systems admins regarding patching
    • Results: servers had been patched with latest Windows server CUs the previous day.  Found some older discussion associating eventID 28005 with patching.
  13. Again checked the farm databases against the error details
    • Result: again noted that the account identified in the error, the farm service account, was identified as the DBO for three particular databases; and that these databases were all created recently.  Also noted the owner for a new content database, created recently, also used the farm account.
  14. Changed the DBO for these three databases to my own administrator account using this TSQL statement:
    ALTER AUTHORIZATION ON DATABASE::NewContentDb TO "MyAdministratorAccount";
    • Result: no resolution, but improved my understanding of issue.  28005 error now presented my administrator account in its error message, rather than the farm service account.  This help bound the scope of the issue by demonstrating that I could affect the error outcome, even in a minor way.
  15. Changed the DBO for the new content database to the SA using the same TSQL statement:
    • Result: no resolution.
  16. Changed the DBO for these three databases to the SA.
    • Result: success.  EventID 28005 stopped appearing in the server Application log.
Solution
  1. Changed the DBO for the three particular databases to the SA (using the same TSQL statement shown above).
    • Result: success. 24 hours later, the error message still has not appeared.
Summary

The references pointed the way to the a solution, this being changing the DBO for databases to the SA.  This is not the optimal solution, but it at least resolves the immediate problem.  Note that not all databases had to have their DBO changed.  Only those associated with the domain account presented in the error event.  There remain databases in the farm, after implementing the solution above, for which the owner remains the farm administrator account.  And no error is being generated for them. So, it's unclear what the root cause here is.

One remaining task to be performed is to obtain a domain account for which the 'Allowed to authenticate' security setting has been enabled, as suggested by this reference (scroll to the bottom of the page).  This will require sysadmin assistance and will thus be implemented at a later date.

References
Notes
  • Farm topology: Windows Server 2008 R2, SQL Server 2008 R2 Standard, SharePoint 2010 Server Enterprise.
  • Methodology: the troubleshooting steps were wide ranging in order to more definitively bound the scope of the issue and improve understanding of it.

Thursday, February 13, 2014

SharePoint 2010: Read error on file

Problem

You are attempting to save a list as a template, and including its data, but are presented with this error:
Read error on file "_catalogs/.../MyListTemplate.stp"
where MyListTemplate is the name you provided when initiating the save to template process.

Troubleshooting

Taking note of the correlation ID for the error, you launch ULSViewer and review the trace log.  You find a number of events associated with this correlation ID.  One of these events in particular triggers your interest:
Throttled:Query exceeds lookup column threshold...
You then launch Central Administration, navigate to CA > Application Management > Web Applications > Manage web applications.  You select the target web application, and then go: General Settings > Resource Throttling.  You adjust the List View Lookup Threshold to a larger number and then re-attempt saving the list as a template.  The error presents itself again.  Increasing the List View Lookup Threshold to a still larger value does not resolve this.

Solution
  1. Open the SharePoint 2010 Management Shell
  2. Execute the following:
    $Web = Get-SPWeb "http://PathToTargetSite"
    $List = $Web.Lists["YourListName"]
    $List.EnableThrottling = $false
    $List.Update()
    
  3. Re-attempt the save list as template with data process.
  4. Once completed, re-execute the above script, this time setting EnableThrottling to $True.
References

SharePoint 2010: The list is too large to save as a template

Problem

You attempt to save a list as a template, including its data, and you are presented with this error:
 

Solution
  1. Increase the MaxTemplateDocumentSize property using either PowerShell or stsadm.  Default is 50 MB.  Maximum is 500 MB.
  2. Using PowerShell:
    1. Execute the following commands in a PowerShell command window to increase the maximum to 500 MB:
      [void][System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint")
      [void][System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint.Administration")
      $docSize = 500000000
      $webservice = [Microsoft.SharePoint.Administration.SPWebService]::ContentService
      $webservice.MaxTemplateDocumentSize = $docSize
      $webservice.Update()
  3. Using stsadm:
    1. Run the following command at the command prompt:
      stsadm -o setproperty -propertyname max-template-document-size -propertyvalue 500000000
Notes
  • The PowerShell commands can be executed on any server in the SharePoint farm.  Note that if you use
  • stsadm can be executed in the command window on any SharePoint server in the SharePoint farm.  If you experience an error trying to execute stsadm, check the DOS Path environment variable to be sure that it includes the path to the stsadm binary, which, by default for 2010, is located here:
    C:\Program Files\Common Files\Microsoft Shared\web server extensions\14\BIN 
References

Wednesday, February 5, 2014

SharePoint 2010 List Forms: Recover the default list or document library edit form

Problem

The default OOTB edit form for a particular document library (or for any list for that matter) has been lost, revised, edited, or whatever, such that the form no longer features the standard default List Form Web Part.  This is the list form web part (LFWP) that only displays the three fields, like so:
Because of these changes, certain fields are no longer displayed or displayed in a way such that they are not as usable as before.  For example, the Name field may display the filename now only as a read-only text, and not in an editable text box.  Other fields may not even be displayed.  There may also be a break between the ability to change column ordering using the features on the document library's or list's Settings page.  Whatever the case may, you want to get the original OOTB edit form back.  Here's how.  You'll need SharePoint Designer 2010 to do it.

Solution
  1. Launch SharePoint Designer 2010.
  2. Connect to the target site.
  3. On the Navigation pane, at left, click Lists and Libraries.
  4. On the Results pane, click on the target list or library.  The Results pane will change to show the properties and objects associated with the list or library you clicked.
  5. Look for the Forms group.  It will list all of the New, Edit and View forms created for this list or library.  If this is for a library, you will also see an Upload form item listed there too.
  6. Click the New... button for this group.
  7. Enter a name for this new default edit form, and the select the Edit item form option.
  8. Click OK.
  9. This dialog will close and a new edit form will appear listed in the Forms group.
  10. Click this item to open it in edit mode.  What you will see is all of the fields layed out nicely.  This is the DataForm Web Part.  You don't want this.
  11. Click anywhere in this form.  You'll see the DataFormWebPart title appear just above this form.
  12. Select this title.  This then selects the entire web part.
  13. Click the Delete key.
  14. Click anywhere inside the now empty web part zone.
  15. Up above, select the Insert ribbon.
  16. In the Controls group, click the SharePoint item.  A dropdown will open.
  17. Click List Form..., down at the bottom.  The List or Document Library Form dialog appears.
  18. Select the target list or document library, and then select the Edit item form option.  Leave the Show standard toolbar option checked.
  19. Click OK.  This inserts the LFWP onto the page.
  20. Save the page.
  21. On the Forms group, select the new edit form listed there (don't click on the filename itself).  This updates the ribbon.
  22. On the ribbon, select the Forms tab.
  23. Click the Set as Default button.
  24. Test.
References

SharePoint 2010: an unexpected error has occurred

Problem

Helpdesk notified that the entire SharePoint Server 2010 farm was inaccessible: connecting to the farm resulted in the following error being displayed in the browser:

Troubleshooting
  1. Checked farm servers:
    • all up and running.
  2. Remoted into the application server and launched Central Administration:
    • connecting to this site resulted in same error.
  3. Remoted into the database server and launched SQL Server Management Studio:
    • SQL Server service running.
  4. On a web front end server, launched Server Manager and checked the Application log:
    • Group of 8306 errors occuring at 2:10 AM: An exception occurred when trying to issue security token: This implementation is not part of the Windows Platform FIPS validated cryptographic algorithms..
    • A large volume of 5586, 6398, 8088, 5240, 6772 and other errors began appearing at approximately 11:20 PM.
  5. Checked development farms:
    • same error appearing on connection; same errors appearing in Application logs.
  6. Checked FIPS-related security settings:
    1. Computer Configuration > Windows Settings > Security Settings > Local Policies > Security Options > System cryptography: Use FIPS compliant algorithms for encryption... this was set to Enabled.
Solution
  1. The setting change was pushed out at the corporate NOC level (above us), thus, we could not change it directly.
  2. Instead, department sysadmin added a new instance of the FIPS policy setting to the AD container but configured it at a higher priority.  Thus, after the corporate policy set the FIPS setting to Enabled, the department policy set the FIPS setting back to DISABLED.
  3. Forced the update to the servers in this container.
  4. Reset IIS on farm servers.
  5. Check security settings: security setting now DISABLED.
References

Monday, February 3, 2014

SharePoint 2010: Error: Additions to this Web site have been blocked

Problem

A subcontractor has been provided administrator access to a SharePoint 2010 farm site and has been restoring content databases and configuring a new web application.  Performing routine maintenance and oversight checks, you attempt to make a configuration change to the main site collection and experience the following error:
Solution
  1. Login to Central Administration under a farm administrator account.
  2. Go: CA > Application Management > Site Collections > Configure quotas and locks.
  3. From the Site Collection dropdown, select the appropriate site collection.
  4. In the Site Lock Information section, set the Lock status for this site to: Not locked.
  5. Click OK.
References

Sunday, February 2, 2014

Windows Media Player: Rip CD option greyed out

Problem

You launch Windows Media Player to rip a new CD or refresh the tracks of an old CD you've already ripped before.  After Media Player recognizes the CD in the disk player, it displays the tracks on the CD, but the Rip CD option is greyed out.

Solution

This option is greyed out because no tracks have been selected.  To select a track, enter a check into the checkbox before each track title.  The moment you check a box, the Rip CD option will be enabled.

References
Notes
  • Windows 7
  • Windows Media Player 12