SharePoint 2013: Configuration Failed, SPUpdatedConcurrencyException was thrown

Problem

You installed the SharePoint Server 2013 March 2015 CU. After rebooting, you launch the SharePoint Products Configuration Wizard. The wizard completes steps 9 through 10, but then presents a failure message on the last step:

Reviewing the upgrade log file, you see this error:

An exception of type Microsoft.SharePoint.PostSetupConfiguration.PostSetupConfigurationTaskException was thrown.
Additional exception information: Failed to upgrade SharePoint Products.

You then open an elevated command shell using the SharePoint Setup User Administrator account and execute: PSConfig.exe -cmd upgrade -inplace b2b -force -wait. You see the usual messages display and eventually the message displays 100%. However, it appears to remain stuck on displaying 100% event after half an hour. Looking in the upgrade log again, you find this message:

An exception of type Microsoft.SharePoint.Administration.SPUpdatedConcurrencyException was thrown. Additional exception information: An update conflict has occurred, and you must re-try this action. The object SPUpgradeSession Name=Upgrade-20150312-141528-847 was updated by [farm service account], in the PSCONFIG (7904) process, on machine [farm server]. View the tracing log for more information about the conflict.

The issue involves a concurrency issue, just like the PowerShell error message indicates. What this means is that multiple upgrade processes were attempting to engage the same configuration table concurrently and a race condition occurred. There are several different solutions here that can be implemented. These are presented in the next section.

Solutions

These solution paths are listed in order of increasing complexity and in the suggested order in which to engage them.  Note: if you experience an apparent "Stuck on 10%, 20%... or even 100%" issue, see the Troubleshooting section below.  That's a different issue then the solutions listed below will resolve.

1. Re-execute PSCONFIG, but ensuring that you do it on only one server at a time, each time waiting for it to complete before executing it on the next one.
2. Re-executing the PSCONFIG command-line tool:

   Psconfig.exe -cmd upgrade -inplace b2b -wait -force

   Try it two or more times. If the same error occurs each time, move on to the next solution option.
3. If executing the command-line tool in the SharePoint Management Shell resulted in the error, try executing it in an elevated DOS command shell.
4. Re-execute the PSCONFIG command-line tool without the -force parameter, like so;

   Psconfig.exe -cmd upgrade -inplace b2b -wait

5. Re-execute the PSCONFIG command-line tool and include the ApplicationContent and InstallFeatures commands, like so:

   PSConfig.exe -cmd upgrade -inplace b2b -force -cmd applicationcontent -install -cmd installfeatures

6. Clear the SharePoint Configuration Cache, and then re-execute the command-line tool (either in and elevated SharePoint Management Shell or DOS command shell).
7. Re-execute PSCONFIG but specifying the affinity (see reference 15 for details).

How to Clear Upgrade Progress Flag and Configuration Cache

1. Clear the upgrade in progress flag
1. Open an elevated command shell.
2. Navigate to: c:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\15\BIN\.
3. Execute the following: stsadm -o getproperty -propertyname command-line-upgrade-running. If this returns <Property Exist="Yes" Value="no" />, go directly to step 2). Otherwise, continue to the next step.
4. Execute the following: stsadm –o setproperty –propertyname command-line-upgrade-running –propertyvalue no.
2. Clear the configuration cache
1. Open the Services.msc control panel, and then look for the SharePoint Timer Service.
2. Stop this service.
3. Open Windows Explorer, and then navigate to the cache folder at: C:\ProgramData\Microsoft\SharePoint\Config.
4. Look for the most recent cache folder, and then open it.
5. Delete all files in this folder EXCEPT cache.ini.
6. Open the cache.ini file in a text editor, and then randomly modify the number, say, to "1".
7. Save the cache.ini file.
8. Start the SharePoint Timer Service.
9. Wait for the configuration cache to fill up again. About 1700 items. Once it stabilizes, proceed to the next step
3. Perform a new upgrade
1. Open an elevated command prompt as Administrator.
2. Run the following command: Psconfig.exe -cmd upgrade -inplace b2b -wait -force. NOTE: The following services will be stopped after executing this command:
   1. SharePoint Timer Service
   2. SharePoint Server Search 15
   3. SharePoint Search Host Controller
   4. SharePoint Administration
   5. World Wide Web Publishing Service
   They will be automatically restarted again at the end of the upgrade process.

Troubleshooting

1. Stuck on 100%. You may experience an apparent "stuck on 100%" issue, even after performing the steps above. Wait before repeating the process or engaging in more troubleshooting. I have experienced delays of up to 90 minutes, between seeing "100.00%" and seeing the message that the upgrade was complete. To check if this is the case, open the Upgrade log file, and then search for the phrase "End processing children of [SPFarm Name=[farm configuration database name] for upgrade" followed by "Exiting upgrade for [SPFarm Name=[farm configuration database name]]. If these are the last two statements currently in the Upgrade file, it is likely that the upgrade is progressing fine and simply requires more time to complete. This has been my experience to date. For example, once, while upgrading two WFEs and one APP, I observed that in each case, the upgrade seemed to get stuck just after presenting 100%. But when I looked into the Upgrade file in each server, I found the above statements at the end of the file; and in every case, the process eventually completed successfully.
2. Stuck on 10%.  You may also experience an apparent "stuck on 10%" issue.  I have experienced this message lasting nearly two hours, and at the conclusion the message still states "10%" but no followed by "Successfully upgraded SharePoint Products.  The screen shot below shows the proof.  
   
   From experience, my suggestion is that if you see it apparently stuck on 10%, 20% or even 100%, don't panic: just let it run its course. And yes this may take two hours or more.  Just let it run.
3. Feature (Id = [...]) is referenced in database [...], but isn't installed on the current farm. If you experience this error after executing PSCONFIG in an elevated SharePoint Management Shell, open an elevated command shell and execute this command:

   PSConfig.exe -cmd upgrade -inplace b2b -force -cmd applicationcontent -install -cmd installfeatures

References

• Primary
• SharePoint 2013: Product Configuration Wizard stuck on task 9 of 10
• Concurrency Error with PSConfig
• What is the SharePoint Configuration Cache?
• Psconfig command-line reference (SharePoint Server 2010)
• Miscellaneous background
1. SharePoint Configuration wizard stuck and never ending when applying service pack. "SyncUpgradeTimerJob: sleeping for 10 seconds" in Upgrade.log
2. PSConfig fails - Failed to upgrade SharePoint Products
3. Problem running config wizard after install of SharePoint 2010 Oct CU
4. PostSetupConfigurationTaskException - Failed to upgrade SharePoint Products
5. SharePoint Configuration Failure
6. Upgrade Session Failed @ 100% Complete?
7. An exception of type Microsoft.SharePoint.PostSetupConfiguration.PostSetupConfigurationTaskException was thrown. Additi onal exception information
8. User, group, or role 'WSS_Content_Application_Pools' already exists in axapta
9. PS2013 ULS Error: "patchca not found. Falling back"
10. An exception of type Microsoft.SharePoint.PostSetupConfiguration.PostSetupConfigurationTaskException was thrown
11. After you apply Windows SharePoint Services 3.0 SP1, an event is logged in the Application log on the server: "User, group, or role 'WSS_Content_Application_Pools' already exists in the current database"
12. Error message when you run the "SharePoint Products and Technologies Configuration Wizard" when you upgrade to SharePoint Server 2007 or to Windows SharePoint Services 3.0: "Failed to initialize SharePoint Product and Technologies upgrade"
13. PSCONFIG stuck at 100% on SP2013 farm
14. A Deeper Look: Patching SharePoint 2013 Farms
15. SharePoint 2010 /2013: “An exception of type Microsoft.SharePoint.Administration.SPUpdatedConcurrencyException was thrown” while installing an update

Notes

• One variation of the above is where the upgrade remained stuck on 10% or some other percentage. For example, I experienced the issue noted above after running the SharePoint Productions Configuration wizard and seeing it complete successfully, but then seeing the message Product / patch installation or server upgrade required appear in the Central Administration Problems report. I then executed the psconfig command, but this remained stuck at 10%. To resolve this, I performed the solution steps identified above.
• UPDATE (5/13/15): Another variation: after restarting a server (after installing Windows updates), opened elevated shell and ran psconfig. During step 3 of 4, the SPUpdatedConcurrencyException error was presented; and this occurred without even reaching the percent "%" phase of this stage. While beginning to implement the steps above, I noticed that a number of services were stopped, including SharePoint Administration and Timer. I started all stopped SharePoint services, then opened an elevated command shell, and ran psconfig. This time, the process ran through 100%.
• UPDATE (09/13/16): I needed to execute the command-line tool in order to resolve a database upgrade warning (I mounted content databases from one farm to another, and the one farm was at a patch level one month behind).  I ran into the SPUpdatedConcurrencyException error again while executing the PSCONFIG command-line tool.  Re-executing the command-line tool didn't solve it, and neither did switching shells (PS to DOS). Adding the ApplicationContent and InstallFeatures commands did the trick.  Even still, it seemed to get stuck on 10%.  After waiting half an hour, I locked the server and went home.  The next day, it showed completed successfully, but... still displaying 10%.  Note that in this case, all that was being updated were several content databases - the SP binaries of the target farm had already been updated and upgraded.

  NOTE: be sure to check SharePoint server services after running PSCONFIG.  I've experienced PSCONFIG completing successfully, but leaving critical services stopped, such as the World Wide Web Publishing Service, SharePoint Search Host Controller, SharePoint Server Search 15 and SharePoint Timer Services.  Also, if you executed it first on the app server, it automatically tries to run on the other SharePoint servers to, and when it fails, it fails on all of them.  Thus, if you finally get it to run successfully on the app server, check the other SP servers to see whether it was run again on these.  if it wasn't, you'll need to run PSCONFIG on these as well.