SharePoint 2013 OWA: Server Error: We're sorry. An Error has occurred. We've logged the error...

Problem

Users attempt to view online a Microsoft Office document in a site library, but experience the following error:

This error is seen when attempting to view any type of Office document.  You then login to the Office Web Apps 2013 server, view its Application log, and find it rapidly filling up with multiple versions of Event IDs 1000 and 1026, such as:

Log Name:      Application
Source:        Application Error
Date:          [date/time]
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      [OWA servername]
Description:
Faulting application name: BroadcastWatchdog_App.exe, version: 
15.0.4502.1000, time stamp: 0x512d264c
Faulting module name: KERNELBASE.dll, version: 6.2.9200.16864, 
time stamp: 0x531d34d8
Exception code: 0xe0434352
Fault offset: 0x0000000000047b8c
Faulting process id: 0x1b54
Faulting application start time: 0x01cfa51426c90393
Faulting application path: C:\Program Files\Microsoft Office Web 
Apps\BroadcastServicesWatchdog_App\BroadcastWatchdog_App.exe
Faulting module path: C:\Windows\system32\KERNELBASE.dll
Report Id: 6525fef4-1107-11e4-93ff-00155d38891e
Faulting package full name: 
Faulting package-relative application ID: 
Event Xml:

and

Log Name:      Application
Source:        Application Error
Date:          [date/time]
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      [OWA servername]
Description:
Faulting application name: Microsoft.Office.Excel.Server.EcsWatchdog.exe, 
version: 15.0.4511.1000, time stamp: 0x5164ab86
Faulting module name: KERNELBASE.dll, version: 6.2.9200.16864, 
time stamp: 0x531d34d8
Exception code: 0xe0434352
Fault offset: 0x0000000000047b8c
Faulting process id: 0x49c
Faulting application start time: 0x01cfa51428fbb3ad
Faulting application path: C:\Program Files\Microsoft Office Web
Apps\ExcelServicesEcsWatchdog\Microsoft.Office.Excel.Server.EcsWatchdog.exe
Faulting module path: C:\Windows\system32\KERNELBASE.dll
Report Id: 66efc6c6-1107-11e4-93ff-00155d38891e
Faulting package full name: 
Faulting package-relative application ID: 
Event Xml:

and

Log Name:      Application
Source:        .NET Runtime
Date:          [date/time]
Event ID:      1026
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      [OWAservername]
Description:
Application: WordViewerAppManagerWatchdog.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.TypeInitializationException
Stack:
   at Microsoft.Office.Web.Common.ServiceInstanceFinder.
      GetLocalAgentInstance(Microsoft.Office.Web.Common.OfficeServiceType)
   at Microsoft.Office.Web.Common.WatchdogHelper.PrepareRegistrations
      (Microsoft.Office.Web.Common.OfficeServiceType)
   at Microsoft.Office.Web.Common.WatchdogHelper.WatchMachines
      (Microsoft.Office.Web.Common.OfficeServiceType, CheckServiceInstance, 
         Microsoft.Office.Web.Common.OfficeServiceType, System.String)
   at Microsoft.Office.Web.WordViewerWatchdog.Program.Main()
Event Xml:

and many others of a similar nature.  You recall that you recently installed updates to the OWA server, via Systems Center Configuration Manager (SCCM), and that some of these involved OWA.

Discussion

Applying updates to Office Web Apps server via SCCM or Windows Updates is not supported and may cause service application failure and loss of OWA functionality to all your users.  Recovery from this situation involves rebuilding the entire OWA service application and its bindings to the SharePoint farm; as well as modifications to your organization's Windows patching SOP. 

Total Solution Time Estimate: 3 hours.

Solution

1. Login to any server in the farm that hosts SharePoint 2013, under the Setup User Administrator account (eg, spAdmin). 

   NOTE: always use the SharePoint Setup User Administrator account for any SharePoint installation and configuration activities.

2. Open a SharePoint Management Shell with elevated privileges (as administrator).
3. Execute the following:

   Remove-SPWOPIBinding

   This removes the Office application bindings that SharePoint has to the OWA server.
4. Logout of the SharePoint server.
5. Login under any administrator account to the OWA server.
6. Open a PowerShell window with elevated privileges (as administrator).
7. Execute the following:

   Import-Module -Name OfficeWebApps
   Remove-OfficeWebAppsMachine

   This doesn't uninstall the OWA software; it only removes the local OWA server from the OWA farm. 

   NOTE: if you have more than one OWA server in the OWA farm, you must first execute this command on each of the child OWA servers first.  Then you can execute this command on the final, master OWA server.

8. Uninstall the software using standard Windows uninstallation method.

   NOTE: you must uninstall the OWA software and rebuild fresh due to how the updates were previously installed.  Uninstalling the OWA software will also get rid of the updates.

9. Restart the OWA server.
10. Re-install OWA.
11. Re-install updates.
12. Restart the OWA server.
13. Login to the OWA server under an administrator account.
14. Open a PowerShell window with elevated privileges (as administrator).
15. Execute the following:

    Import-Module -Name OfficeWebApps
    New-OfficeWebAppsFarm -InternalURL "http://servername" -AllowHttp -EditingEnabled

    where servername is the actual name of the OWA server host machine.  Do not use the fully qualified name.  Executing this command creates a new OWA farm.
16. Next, execute the following:

    Get-OfficeWebAppsMachine

    This returns the Roles and health status of the local OWA server.  For my OWA farm, I wanted the roles to be All and health status Healthy.
17. Logout of the OWA server.
18. Open any browser, and then connect to the following:

    http:/servername/hosting/discovery

    If everything's working correctly, you will see the service discovery file in XML format.
19. Login to any SharePoint farm server (hosting SharePoint), using your farm's SharePoint Setup User Administrator account.
20. Open a SharePoint Management Shell with elevated privileges (as administrator).
21. Execute the following:

    Get-SPWOPIBinding

    This should return nothing, since all WOPI bindings should have been removed back in step 3.  If it doesn't, execute Remove-SPWOPIBinding.
22. In this same SharePoint Management shell, execute the following:

    New-SPWOPIBinding -ServerName <WacServerName> -AllowHTTP

    where ServerName is the actual name of the OWA server host machine. Do not use the fully qualified name.  This command builds the bindings in SharePoint to the OWA service application.
23. In this same SharePoint Management shell, execute the following:

    Get-SPWOPIBinding

    This should now return a list of Office applications and actions and their bindings.
24. In the same SharePoint Management shell, execute the following:

    Get-SPWOPIZone

    This should return internal-http, given the WOPI authentication type that was configured in step 22, "-AllowHTTP."
25. In the same SharePoint Management shell, execute the following:

    (Get-SPSecurityTokenServiceConfig).AllowOAuthOverHttp

    This should return True, if you are not using HTTPS for connections to your farm.
26. OWA services should be available immediately, and you can test right away.

References

• Office Web Apps Server
• Deploy Office Web Apps Server
• Configure Office Web Apps for SharePoint 2013
• Apply software updates to Office Web Apps Server
• Plan for administrative and service accounts in SharePoint 2013
• Plan for least-privileged administration in SharePoint 2013
• Windows PowerShell for Office Web Apps Server
• Remove-SPWOPIBinding
• Remove-OfficeWebAppsMachine
• New-OfficeWebAppsFarm
• Get-OfficeWebAppsMachine
• Get-SPWOPIBinding
• New-SPWOPIBinding
• Get-SPWOPIZone
• Get-SPSecurityTokenServiceConfig
• Office Updates
• Fun configuring Office web apps 2013 (OWA)
• A quick-guide to setting up OWA with SharePoint 2013 – start to finish: lots of good and accurate content to help you get started.
• Install, Configure & Monitor Office Web Apps 2013 for SP 2013: good section on troubleshooting and monitoring.
• How to remove Office Web Apps 2013 with Service Pack 1 and apply Security Update 2878219: this was the primary reference I used to guide my recovery efforts.

Notes

• This solution assumes a single-server internal OWA farm that exposes its service over HTTP (not HTTPS) for an internal corporate environment and that has Microsoft Office volume licensing.
• It also assumes an internal SharePoint 2013 farm that uses authentication over HTTP and does not employ encrypted connections.