3. Best Practices in Architecting& Implementing WSUS Greg Shields Partner & Principal Technologist Concentrated Technologywww.ConcentratedTech.com Session Code: WSV208
4. Presentation Outline (hidden slide): Speakers:complete this slide using the session information found at the speaker portal. Title: Best Practices in Architecting and Implementing Windows Server Update Services Technical Level: 200 Intended Audience: Technologists, Systems Administrators, WSUS Implementers Objectives (what do you want the audience to take away from this session): 1. Bad WSUS architectures will equal poor update installation. 2. Using WSUS for servers is different than for workstations. 3. What are the real-world best practices for applying updates. Presentation Outline (including demos):
5. Agenda Topics Part I: Architecting & Implementing WSUS Part II: Troubleshooting WSUS Part III: Tips & Tricks for Using WSUS 5
7. WSUS Product Vision Simple, zero-cost solution for distributing Microsoft Updates content in a corporation. A “free” RTW add-on for Windows Server Solution only distributes Microsoft Updates Distributing 3rd party patches require purchasing advanced management tools such as SCE or Configuration Manager 2007 Provides a foundation for Update Management across Microsoft products: SCE, Configuration Manager 2007, MBSA, WU, SBS, Forefront, … Consistent scan results Unified client scan mechanism (WUA) irrespective of which server actually manages the updates.
8. WSUS Momentum Over 500,000 distinct WSUS servers synched with Microsoft Update last month Used by over 60% medium/large orgs and built into SBS WSUS 3 released April 30 2007 Huge improvements in performance, deployment options, reporting and UI Easy in-place upgrade from WSUS2 WSUS 3.0 SP1 released Feb 7, 2008 WSUS 3.0 SP2 released Jan 26, 2009
10. WSUS 3.0 SP1/SP2 Adds Features WSUS 3 SP1 added the following features: Installs on Windows Server 2008, integrated with Server Manager (after installing Server Manager update KB940518) API enhancements for advanced management tools Bug fixes WSUS 3 SP2 adds: Installs on Windows Server 2008 R2 Supports managing Win7 clients Support for BranchCache Auto-approval rules with deadlines Bug fixes (DSS gets languages from USS, target groups sorted alphabetically, more robust setup upgrade) Compliance against approved updates
11. New Features in WSUS SP2 Greg Shields Partner & Principal Technologist Concentrated Technology www.ConcentratedTech.com demo
12. Elements of ArchitectureWhy Architecture? Problems are usually results of improper architecture A correct architecture will drive a better design Especially in situations of administrator distrust or insufficient bandwidth Design your WSUS solution with the same goals as your AD solution Roaming users should be dealt with separately 12
13. “Simple” Architecture Single, well-connected site WSUS Updates from MU Clients update from WSUS Single server can handle 25,000 clients 50K clients with 2x front-end servers and big SQL back-end Remote SQL configuration reduces server load Front-end handles update sync load Back-end handles reporting load 13
14. “Simple, with Groups” Architecture 14 Largest use case in production today Driving forces to move to Machine Groups: Differing patching requirements or schedules Test groups Servers vs. Workstations Politics Not necessarily used for load distribution
15. WSUS Chaining Chaining involves downstream servers getting updates (and sometimes Group data) from upstream servers Options for chaining Distributed vs. Centralized model “Autonomous Mode” vs. “Replica Mode” Chaining solves the problem of “mesh” or “fully independent” architectures Wastes resources and bandwidth Not that some situations don’t mandate “mesh” or “fully independent” architectures! 15
16. “Centralized” Architecture Downstream servers are replicas of primary server Little downstream control over servers Downstream admins drop machines into predefined groups All update approvals and schedule done at primary server 16
17. “Distributed” Architecture Downstream servers obtain updates from primary server, except: Update approvals do not flow down. Assigned at each site individually. Downstream admins have greater control. Can create groups and assign approvals. Used for distribution rather than control of updates 17 Combinations of centralized anddistributed possible. Depends onintra-IT trust model.
18. “Disconnected” Architecture Many environments don’t have Internet connectivity. Test/dev, government, classified, air gap environments Data must be imported from “the outside” Any the previous architectures will work Manual import process required Gives CM/QA/Security the option to review updates prior to bringing “inside”. 18 Sneakernet
19. “Disconnected” Architecture Match advanced options between source and target. Express installation files & languages must match. Backup & restore updates from source to target. Back up C:SUSSUSContent Restore to the same location on the target server. Transfer update metadata from source to target. Navigate to C:rogram Filespdate Servicesools Export metadata using wsusutil.exe export {packageName} {logFile} Import with wsusutil.exe import {packageName} {logFile} packageName & logFileare unique names you choose 19 Database validation can take multiple hours to complete!
20. “Roaming” Architecture Manages updates for external resources WSUS servers distribute approval metadata Clients download updates from Windows Update directly. Extra security for internet-facing WSUS server Useful separate architecture for mostly off-net clients 20 Laptop WSUS Laptops
21. “Roaming” Architecture Four Steps to Internet-facing WSUS Build server in DMZ and position behind ISA proxy Locate database on server not reachable from Internet Enable SSL for communications Host content on Microsoft Update 21 Laptop WSUS Laptops
22. “High Availability” Architecture WSUS 3.0 includes native support for high availability NLB Clusters connect multiple WSUS web servers via a single cluster IP SQL Cluster manages the database No single point of failure Critical: This design isuseful for availability,but does little forperformance. 22
23. Managing Branch Offices Branch offices are typically managed through replica WSUS servers Replica servers take all orders from the central server. Settings at the top flow downward, but take time. Alternatively, unify architecture through a single “central server” Single server manages all clients across all offices Deploy ISA proxy in the branch Enable BITS peer-caching Use delta files to reduce network traffic. 10x more server disk space 4x less client download 23
24. Upgrade deployment WSUS 3 SP1 setup supports in-place upgrade One-way upgrade (no rollback) Can’t be done from WSUS 2 on Windows Server 2000 or using SQL 2000 Alternative is migration upgrade: Install second server If original server is WSUS2 SP1: Perform disconnected replica steps (wsusutil, ntbackup, wsusmigrate) Switch over client via policy If original server is also WSUS3 Configure new server to be a replica of the first and sync After sync, configure new server to be autonomous Upgrade hierarchy from top down
26. Errors and Error Codes Numerous WSUS error codes exist. A complete list of all WSUS error codes is available on-line at http://inetexplorer.mvps.org/archive/ windows_update_codes.htm For example, 0x8DDD0018 occurs when one of these services is Disabled Automatic Updates BITS Event Log 26
27. Errors and Error Codes II 0x80072EE2, 0x80072EFD This issue occurs because the Windows Update client did not receive a timely response from the Windows Update Web site server. Likely a proxy configuration, personal firewall, or trusted hosts problem 27
28. Errors and Error Codes III 0x80246008, 0x8024402C Caused by BITS malfunctioning or corrupted. Download and extract the BITSAdmin tool from the Windows Support Tools CD. Bitsadmin /util /repairservice /force If that doesn’t work, try a BITS re-install Though if you do a BITS re-install, clear out the %SystemRoot%oftwareDistribution folder and reboot when done. Its worth mentioning here that thereis no “backup” download process for WUA. …like HTTP or FTP…If BITS is non-functional, so is patching! 28
29. Errors and Error Codes IV 0x80244019 This error is often caused when the Proxy server is not properly configured. Ensure that your Proxy server allows Anonymous access to these external addresses: http://windowsupdate.microsoft.com http://*.windowsupdate.microsoft.com https://*.windowsupdate.microsoft.com http://*.update.microsoft.com https://*.update.microsoft.com http://*.windowsupdate.com http://download.windowsupdate.com http://download.microsoft.com http://*.download.windowsupdate.com http://wustat.windows.com http://ntservicepack.microsoft.com Microsoft doesnot publish the IP’sassociated with theseFQDN’s.So, if you do perimeternetwork security by IPyou’ve gotta’ stayon the ball with these! 29
30. WUA Client Issues To enable auto-updates, ensure: Anonymous access granted to Self Update virtual directory on WSUS server Auto-updates requires TCP/80 to function on WSUS server Be aware of GP replication times 90 to 120 minute GP refresh timing will impact speed of clients becoming visible in WSUS admin tool Be aware of AU detection frequency times WUA client set to check with server every 22 hours (minus offset). When WUA checks in is when it checks WUA version. Need to do wuauclt /detectnow to force this to occur on-demand. 30
31. WUA Client Issues II Known issue with imaged workstations: If you image your workstations (and who doesn’t these days!), you must change SID Sysinternals NewSID, Microsoft SysPrep Not doing this will prevent WUA from contacting WSUS To fix this problem: Run one of the above tools to change the SID HKLMoftwareicrosoftindowsCurrentVersionindowsUpdate Delete PingID, SUSClientID, and AccountDomainSID values Restart wususerv service Run wuauclt /resetauthorization /detectnow 31
32. WUA Client Issues III Disabling the Automatic Updates Service or the BITS Service at any point in the past prevents it from starting properly when you need it! Reset permissions on these services to re-enable functionality. Use the Service Control Resource Kit tool (sc.exe) to do this: sc sdset bits "D:(A;;CCLCSWRPWPDTLOCRRC;;;SY)(A;;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;BA)(A;;CCLCSWLOCRRC;;;AU)(A;;CCLCSWRPWPDTLOCRRC;;;PU)" sc sdsetwuauserv "D:(A;;CCLCSWRPWPDTLOCRRC;;;SY)(A;;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;BA)(A;;CCLCSWLOCRRC;;;AU)(A;;CCLCSWRPWPDTLOCRRC;;;PU)“ Every disabled client needs this! 32
34. Optimize Patch Distribution In large, multi-site environments low bandwidth may cause problems for remote offices. Distributing updates to downstream servers is big problem Potential solutions: Ensure downloading only the languages you need Configure patch distribution to occur in the evenings. Stagger patch distributions between tiered sites Express installation files can exacerbate this. The bandwidth savings in express installation files occurs from WSUS server to client, not between WSUS servers. Throttle BITS 34
35. Throttling BITS BITS can be throttled either on the WSUS server or additionally on all the clients. Alleviates network saturation during update distribution and during client installation Be aware that this does slow down update distributions! Throttle BITS in Group Policy: Computer Configuration | Administrative Templates | Network | Background Intelligent Transfer Service Two settings: Maximum network bandwidth that BITS uses Limit by Kbps based on time of day or at all times Be aware that Kbps is kiloBITS not kiloBYTES (divide by 8) Timeout (in days) for inactive jobs 35
36. DNS Netmask Ordering Non-centralized architectures can better route clients through DNS Netmask ordering. Microsoft DNS Round Robin will first provide an IP address in the same subnet as the requestor. If no IP exists in the same subnet, a random IP will be selected. All WSUS hosts must respond to the same FQDN. DNS FQDN record is populated with IP addresses of all WSUS servers in the network. 36
37. Server Tuning Run cleanup and DB defrag every few months Cleanup wizard is a feature in WSUS 3 Removes stale computers and updates DB index defrag script available on ScriptCenter keeps the server running fast Look out: Take care to not remove computers that are still active (but having trouble contacting the server) Populate from AD sample tool can help In a hierarchy, need to run cleanup on each WSUS server. Clean computers from bottom-up Clean updates from top-down (or between sync intervals) Can be automated through the API
38. 38 Considerations for Updating Servers Servers require more care than workstations… A rebuild is usually not an acceptable solution for a failed patch installation. Outage windows are shorter. But in some ways servers are easier… Data and system drives usually separated. Hardware configuration is usually more stable or well-understood. Service isolation and redundancy – in larger environments – limits exposure/risk. People typically aren’t “surfing” on servers. The RAID 1 Undo Trick…
39. What About Reboots? I’ve said this before, and I’ll say it again: “If you have a patch management plan without a reboot strategy, you don’t have a patch management plan.” Three methods: Client-initiated WSUS-initiated Script-initiated Two methodologies: Scheduled reboots vs. rebooting for patch installation I will argue in favor of scheduled, forced rebootsover mid-day reboots. 39
40. Handling Reboots RebootFile = "computers.txt“ LogFile = "results.txt" Set fso = CreateObject("Scripting.FileSystemObject") Set f = fso.OpenTextFile(RebootFile, 1, True) Set objTextFile = fso.OpenTextFile(LogFile, 2, True) On Error resume next Do While f.AtEndOfLine <> True strComputer = f.ReadLine Set objWMIService = GetObject("winmgmts:{impersonationLevel=impersonate}!" & strComputer & "ootimv2") If Err.Number <> 0 Then objTextFile.WriteLine(strComputer & " is not responding.") Err.Clear Else Set colOperatingSystems = objWMIService.ExecQuery("Select * from Win32_OperatingSystem") objTextFile.WriteLine(strComputer & " is rebooting.") For Each objOperatingSystem in colOperatingSystems ObjOperatingSystem.Reboot() Next End If Loop
41. Custom Reports UI supports basic customization (filters) Advanced customization can be built on WSUS (.Net) API Can use of PowerShell scripts to generate reports Public read-only SQL views Can use SSRS to generate reports (if full SQL) Samples available from MSDN E.g., compliance against approved updates
42. Match KBs to MSRCs Ever wish you had a nice mapping of knowledgebase numbers to MSRC numbers? “The Q-numbers to the MS-numbers” This script outputs a .CSV file that provides just that mapping Add the name of your WSUS server into the top line of the script: strWSUSServer = “<Enter WSUS Server here>" 42
43. Match KBs to MSRCs strWSUSServer = “<Enter WSUS Server here>" Set fso = CreateObject("Scripting.FileSystemObject") Set objTextFile = fso.OpenTextFile("OUTPUT.csv", 2, True) objTextFile.WriteLine("MS Number,Q Number") Set conn = CreateObject("ADODB.Connection") Set rs = CreateObject("ADODB.Recordset") dbconn = "Driver={SQL Server};Server=" & strWSUSServer & ";Database=SUSDB" conn.opendbconn strSQLQuery = "SELECT dbo.tbSecurityBulletinForRevision.SecurityBulletinID, dbo.tbLocalizedProperty.Title FROM dbo.tbLocalizedPropertyForRevision INNER JOIN dbo.tbLocalizedProperty ON dbo.tbLocalizedPropertyForRevision.LocalizedPropertyID = dbo.tbLocalizedProperty.LocalizedPropertyID INNER JOIN dbo.tbSecurityBulletinForRevision ON dbo.tbLocalizedPropertyForRevision.RevisionID = dbo.tbSecurityBulletinForRevision.RevisionID WHERE (dbo.tbLocalizedPropertyForRevision.LanguageID = 1033) ORDER BY dbo.tbSecurityBulletinForRevision.SecurityBulletinID" rs.OpenstrSQLQuery, conn, 3, 3 While Not rs.EOF objTextFile.WriteLine(rs.Fields(0).Value & "," & Replace(rs.Fields(1).Value, ",", "")) rs.MoveNext Wend WScript.Echo "Done!"
44. Agent Control Use WUA API to control the agent Custom install schedules Updating servers in web farms Implementing “install now” functionality
45. On-Demand Patching(You Patch Now!) Ever wish you had a WSUS “Big Red Button”? Such a button might automatically download and install all approved patches and reboot if necessary… How about this VBScript? Run this script from any server console Immediately downloads and installs all approved patches. If a reboot is required, it will then reboot the server. 45
46. The WSUS Big Red Button Set fso = CreateObject("Scripting.FileSystemObject") Set objAutomaticUpdates = CreateObject("Microsoft.Update.AutoUpdate") objAutomaticUpdates.EnableService objAutomaticUpdates.DetectNow Set objSession = CreateObject("Microsoft.Update.Session") Set objSearcher = objSession.CreateUpdateSearcher() Set objResults = objSearcher.Search("IsInstalled=0 and Type='Software'") Set colUpdates = objResults.Updates Set objUpdatesToDownload = CreateObject("Microsoft.Update.UpdateColl") intUpdateCount = 0 For i = 0 to colUpdates.Count - 1 intUpdateCount = intUpdateCount + 1 Set objUpdate = colUpdates.Item(i) objUpdatesToDownload.Add(objUpdate) Next ‘<<This is only the first half of the script. Add the code from the next page to ‘create the full script>>
47. The WSUS Big Red Button ‘<<Add this half to the code on the previous page!>> If intUpdateCount = 0 Then WScript.Quit Else Set objDownloader = objSession.CreateUpdateDownloader() objDownloader.Updates = objUpdatesToDownload objDownloader.Download() Set objInstaller = objSession.CreateUpdateInstaller() objInstaller.Updates = objUpdatesToDownload Set installationResult = objInstaller.Install() Set objSysInfo = CreateObject("Microsoft.Update.SystemInfo") If objSysInfo.RebootRequired Then Set objWMIService = GetObject("winmgmts:{impersonationLevel=impersonate,(Shutdown)}!localhostootimv2") Set colOperatingSystems = objWMIService.ExecQuery("Select * from Win32_OperatingSystem") For Each objOperatingSystem in colOperatingSystems objOperatingSystem.Reboot() Next End If End If
48. Other API Uses ISVs use APIs for many other features as well Distribute 3rd party updates (quite complex) Gather software and hardware inventory Distribute updates to non-Windows devices Your starting point is http://technet.microsoft.com/en-us/wsus/bb466192.aspx API Samples Diagnostic Tools Header Files
49. Summary WSUS is simple to use, but scales to enterprise Flexible server deployment options Single server, scale up, branch office, scale out, disconnected, roaming laptops Flexible update deployment options Peer caching, delta patching, auto approval rules, auto-reapprove revisions Periodically tune the server (defrag + cleanup) Public API and DB views can be used to extend the base functionality for many advanced scenarios Starting point for all WSUS information http://www.microsoft.com/updateservices
51. Required Slide Speakers, TechEd 2009 is not producing a DVD. Please announce that attendees can access session recordings at TechEd Online. www.microsoft.com/teched Sessions On-Demand & Community www.microsoft.com/learning Microsoft Certification & Training Resources http://microsoft.com/technet Resources for IT Professionals http://microsoft.com/msdn Resources for Developers www.microsoft.com/learning Microsoft Certification and Training Resources Resources
