We recently had the SCCM Backup task start failing on one of our Primary Sites. Worked fine for years, then all of a sudden it just stopped working. After doing a little digging, I could see the backup was failing on the VSS Initialization during the SQL portion of the backup.
In the SMSBKUP.LOG on the Primary I found errors like:
ERROR: SQL Backup task failed. Error message - Error: VSS Initialization failed. STATMSG: ID=5052 SEV=E LEV=M SOURCE="SMS Server" COMP="SMS_SITE_BACKUP" SYS=<ServerName>.<DomainName> SITE=<SiteCode> PID=8292 TID=19584 GMTDATE=Tue May 04 06:38:15.396 2021 ISTR0="\<ServerName>" ISTR1="CM_<SiteCode>;" ISTR2="Error: VSS Initialization failed." ISTR3="" ISTR4="" ISTR5="" ISTR6="" ISTR7="" ISTR8="" ISTR9="" NUMATTRS=0 Error: Sql Server could not prepare for the Backup.
Looking on our SQL Cluster there is a folder called <Drive>:\SMS_<PrimarySiteServerName>\logs and in here I looked at smssqlbkup.log that showed more VSS related errors:
Starting VSS initialization… m_pBackupComponents->InitializeForBackup() failed. Error code = 0x80,042,302.Error description = . Error: VSS Initialization failed… Sql backup failed. Please see previous errors.
Also notable was that the last time the backup worked lined up with the last time the SQL Cluster had rebooted as well (check out https://sccmpowershell.com/get-the-last-boot-time-using-powershell for getting the last boot time).
I first tried restarting the SQL Server VSS Writer service on the cluster and also the SMS services on the Primary Site, including SMS_SITE_VSS_WRITER and the Microsoft Software Shadow Copy Provider services and running the backup, no luck.
Time to Google! After some searching there was a useful comment in this Microsoft forum thread.
Next I ran the following on both the Primary and on the active SQL cluster node:
vssadmin list writers
On the Primary Site Server it returned a list of all of the VSS Writers available but on the active SQL Cluster node, NOTHING was returned and it just errored with a failure.
Error: A Volume Shadow Copy Service component encountered an unexpected error. Check the Application event log for more information.
Next I ran the command again on the secondary (passive) node and got the list of VSS Writers back without any issues. All items should show a State of “Stable”. So I rebooted the active SQL node and failed over to the secondary node. After the server rebooted I ran the vssadmin list writers command again and the list of VSS Writers returned without any issues.
I then ran the SCCM backup by manually starting the SMS_SITE_BACKUP service on the Primary Site Server. Voila! Backup was successful.
Hope this helps!