SCCM Client Not Working on Some Systems After 2103 Upgrade

We recently upgraded to Microsoft Endpoint Configuration Manager (MECM) 2103 (aka SCCM 2103) and ran into an issue with the Client Upgrade on about 350 of our 3000+ servers.

When we do our upgrades we normally enable automatic client upgrades for Workstations, but for Servers we use a scheduled Software Deployment to do the upgrade. Most of our servers are Pull Distribution Points so those upgrade the client automatically when the pull DP upgrades, but we still had about 550 servers that needed the deployment.

After setting up the deployment and scheduling it for Friday night, by Sunday, 350 servers were still not showing the new client version which is not normal for us as we usually have a much higher success rate for this deployment.

Looking at the client logs, I could see that the clients ran the upgrade deployment and ccmsetup.log showed that it completed successfully, but something was clearly amiss. In the Configuration Manager applet, not all of the tabs were there and Actions only had 2 actions so the client wasn’t processing policies.

In the ccmexec.log I found the following errors:

System Task 'UpdatesDeploymentStartupTask' is disabled due to absent or unsupported parent application 'ConfigMgr_Client'. 
System Task 'CoMgmt_Startup' is disabled due to absent or unsupported parent application 'ConfigMgr_Client'.    
System Task 'CIStore_Startup' is disabled due to absent or unsupported parent application 'ConfigMgr_Client'.    
System Task 'SettingsAgentStartup' is disabled due to absent or unsupported parent application 
System Task 'CTMStartup' is disabled due to absent or unsupported parent application 'ConfigMgr_Client'.    
System Task 'SysPerfStartup' is disabled due to absent or unsupported parent application 'ConfigMgr_Client'.    
System Task 'CcmEvalTaskStartup' is disabled due to absent or unsupported parent application
System Task 'DeltaDownloadServiceStartup' is disabled due to absent or unsupported parent application
System Task 'ClientIDManagerStartup' is disabled due to absent or unsupported parent application 'CCM_Framework'.
System Task 'MtrMgrStartup' is disabled due to absent or unsupported parent application 'ConfigMgr_Client'.
System Task 'CIStateStore_Startup' is disabled due to absent or unsupported parent application
System Task 'LSMaintenance' is disabled due to absent or unsupported parent application 'ConfigMgr_Client'.
System Task 'EndpointProtectionAgentStartup' is disabled due to absent or unsupported parent application 'ConfigMgr_Client'.
System Task 'StatusAgentStartup' is disabled due to absent or unsupported parent application 'CCM_Framework'.
System Task 'CcmCloudStartup' is disabled due to absent or unsupported parent application 'ConfigMgr_Client'. 
System Task 'CIAgent_Startup' is disabled due to absent or unsupported parent application 'ConfigMgr_Client'. 
System Task 'M365A_Startup' is disabled due to absent or unsupported parent application 'ConfigMgr_Client'.
System Task 'CertEnrollAgentStartupTask' is disabled due to absent or unsupported parent application 'ConfigMgr_Client'.
System Task 'SensorStartup' is disabled due to absent or unsupported parent application 'ConfigMgr_Client'. 
System Task 'UserAffinityTaskStartup' is disabled due to absent or unsupported parent application 'ConfigMgr_Client'.
System Task 'CertificateMaintenance' is disabled due to absent or unsupported parent application 'CCM_Framework'.
System Task 'CIDownloader_Startup' is disabled due to absent or unsupported parent application 'ConfigMgr_Client'.
System Task 'FileBITSStartup' is disabled due to absent or unsupported parent application 'CCM_Framework'.
System Task 'ScanAgentStartupTask' is disabled due to absent or unsupported parent application 'ConfigMgr_Client'.
System Task 'CMBITSManagerStartup' is disabled due to absent or unsupported parent application 'ConfigMgr_Client'.
System Task 'RebootCoordStartup' is disabled due to absent or unsupported parent application 'ConfigMgr_Client'.
System Task 'SoftwareCatalogUpdateStartupTask' is disabled due to absent or unsupported parent application 'ConfigMgr_Client'.
System Task 'ComplRelayAgentStartupTask' is disabled due to absent or unsupported parent application 'ConfigMgr_Client'.
System Task 'WinPEBranchCacheStartup' is disabled due to absent or unsupported parent application 'ConfigMgr_Client'.
System Task 'SrvWinMgrStartup' is disabled due to absent or unsupported parent application 'ConfigMgr_Client'.
System Task 'BLM_Startup' is disabled due to absent or unsupported parent application 'ConfigMgr_Client'.
System Task 'DCMAgent_Startup' is disabled due to absent or unsupported parent application
System Task 'SCStartUp' is disabled due to absent or unsupported parent application 'ConfigMgr_Client'. 
System Task 'BgbAgentStartupTask' is disabled due to absent or unsupported parent application
System Task 'EEAStartup' is disabled due to absent or unsupported parent application 'ConfigMgr_Client'.
System Task 'DTSStartup' is disabled due to absent or unsupported parent application 'CCM_Framework'.
System Task 'SrcUpdateStartup' is disabled due to absent or unsupported parent application
System Task 'ExecmgrStartup' is disabled due to absent or unsupported parent application 'ConfigMgr_Client'. 
System Task 'PwrAgentStartupTask' is disabled due to absent or unsupported parent application 'ConfigMgr_Client'.

In my many years working with SCCM, I had never seen these errors before. Googling found NOTHING! Always great to be the first to run into something. 🙂

I tried removing the client and re-installing, but same issue. Completely removed the client, deleted ccmsetup folder, re-installed again, that worked on one system but not on another. Inconsistent results are never good when you have 350 systems to remediate! I tried re-pushing the client from the console both with and without the option to remove the client first. It would install but then ccmexec.log showed the same errors again.

I decided to check 2 things. First, of the 350, which ones are OFFLINE. See my post for Simple Ping Test to List of Computers for doing this. Next, I decided to cross reference my list with our list of Distribution Points to see if these were DPs or not.

SELECT ServerName, SMSSiteCode FROM v_DistributionPoints WHERE ServerName IN ('SERVER1', 'SERVER2', 'etc') ORDER BY ServerName

I had copied the list of the 350 systems from my Collection into Excel and then put the above SQL query results into the sheet too and did a vlookup to compare the lists. If you’re not good with vlookups, this is my favourite online tool for comparing two lists courtesy of MIT (Compare Two Lists).

Out of the 350, over 300 of them were NOT Distribution Points. But they used to be! We have a server refresh project going on and are replacing servers with new ones at many of our locations. As part of this process, the old DP gets deleted as a Site System from SCCM and of course, the DP role along with it. Turns out when you do this via the console, or the PS cmdlets, SCCM apparently doesn’t actually uninstall the Pull DP from the system even though it’s gone from the SCCM console and database. Do better Microsoft.

On one of the affected servers I ran an uninstall for <Drive>:\SMS_DP%\sms\bin\PullDP.msi. Once this completed, voila! The client on the server started working almost immediately, the errors went away, all of the tabs showed up in the Control Panel applet and the Actions tab was fully populated. The client ran its Discovery cycle and the object updated in my Collection to show the new client version.

Solution

Now, how do I repeat this on the remaining 299 systems? I did some searching and found this thread on StackOverflow and found the solution in 3 simple lines of code. I wrapped a ForEach loop around it and ended up with a script I could pass the list of target servers to.

$Servers = Get-Content -Path "$PSScriptRoot\Servers.txt"

ForEach ($Server in $Servers) {
    Write-Host "Processing server $Server. Removing PullDP…"
    $command = "msiexec.exe /X <Drive>:\SMS_DP$\sms\bin\PullDP.msi /qn /lv <Drive>:\PullDP_Uninstall.log"
    $ScriptBlock = [Scriptblock]::Create($command)
    Invoke-Command -ComputerName $Server -ScriptBlock $ScriptBlock
}

This worked like a charm to successfully remediate the clients.

You might be wondering about the other 50 not in the list. Most of those are new servers offline for shipping and a handful are having various other issues other than this issue.

Additional Update

We later found some servers that were still Distribution Points where the client and Pull DP were not working after the 2103 upgrade. I opened a case with MS and sent them DistMgr logs from the Primary, but we could see that CCMExec.log for the client was showing the same errors as above and DistMgr was basically not attempting to initiate a re-install/upgrade of the DP. I flipped it to a Standard DP but still no joy.

I sent MS historical DistMgr log files for analysis but based on my previous experience I knew that if I ran the uninstall for PullDP.msi the client would start working (which it did). In the DistMgr.log on the Primary Site Server there wasn’t much to go on, but I could see that it was showing the log entry:

File \\<SERVER_FQDN>\SMS_DP$\sms\bin\\smsdp.dll version is 5.0.9049.1000

What was strange is that this DLL was the only one being referenced for that DP. There is a list of about 15-20 that should show in the log file. DistMgr wasn’t recognizing it as a fully installed DP, but it wasn’t trying to repair/install it either. We needed to get the DP working again as it was at a live location so while MS was analyzing logs, I decided to rename this DLL to smsdp.old.

After the smsdp.dll was renamed on the target server I waited for DistMgr on the primary to check the server again (it does this about every 20-22mins).

Boom! DistrMgr woke up and recognized the DP as being uninstalled and re-installed it.

PullDP ["Display=\\<SERVER_FQDN>\"]MSWNET:["SMS_SITE=<SiteCode>"]\\<SERVER_FQDN>\ is marked Uninstalled
…
Found .ins file for remote server ["Display=\<SERVER_FQDN>\"]MSWNET:["SMS_SITE=<SiteCode>"]\<SERVER_FQDN>\, will try to verify the selected drive and install DP files on this server
…
DP configuration thread done for distribution point <SERVER_FQDN>

Once this was finished, I flipped the server back to a PullDP in the console and waited for that to update which took a while, but it eventually did. I then redistributed content to the DP and packages successfully started staging to it again.

Happy computing!

4 thoughts on “SCCM Client Not Working on Some Systems After 2103 Upgrade”

  1. We’re doing our 2103 upgrade today and I ran into this same problem. The problem servers are all Pull DPs, but not all our Pull DPs are experiencing the issue.

    In our case, reconfiguring them to be normal DPs got the pulldp.msi uninstalled and the cm client immediately started working. Once they were fixed up I reconfigured them to be Pull DPs again.

    Thanks for the tip!

    Reply
    • Glad it helped! I just added an Additional Update section as well. Interesting that for yours just flipping it from Pull to Standard and back again was enough. We weren’t so lucky on a few of ours and had to rename the DP DLLs before the DP would fully upgrade and take packages again.

      Reply
  2. I’ve got a similar client upgrade issue with 2107. Performed an upgrade to the base 2107 version on a small site and all DPs auto upgraded the client to that version. But there is also a KB for 2107 so I installed that and I don’t think that triggers a further client upgrade on any systems for some reason. Not sure why but might not be an issue. So I updated our client upgrade package for servers and deployed it to the SCCM servers collection.

    It worked on all but two DPs and I just cannot get them upgraded. The client appears to install fine like yours. It’s a PKI environment and I can see the right client authentication cert being selected but then it immediately complains that no certificate exists. The error is in ClientIDManagerStartup.log: Error RegTask: Failed to get certificate. Error: 0x80004005.

    There are so many posts with references to that error, saying to stop the service, delete a file from RSA\MachineKeys and restart the service. Everyone says yay it fixed my issue but it doesn’t work here. Even though it’s a different error to yours, I’m convinced it’s because these are DPs. It’s even more annoying given that DPs don’t even need a client to function but apparently somehow they can prevent the client from working properly.

    I know it’s only 2 DPs and not 350, but I’m hoping they won’t need to be uninstalled, upgraded, reinstalled.

    Reply

Leave a Reply to Jared Cancel reply