Hear from the engineers about the agent architecture, best practices and troubleshooting tips to keep things running smoothly: http://dell.to/1GDYpr8

1. Dell World 2014
KACE Agent Architecture and Troubleshooting
Overview
Allen Tsai: Principal Engineer
Rob Napier: Principal Engineer November, 06, 2014
Dell World
User Forum

2. Dell World 2014
• Changes in 6.0/6.3
• Key field issues addressed
• What a healthy agent looks like
• Data files
• Agent plugins
• Log file
• Windows installer
• AMP watchdog
• Agent security
• Debugging tips
Overview

3. Dell World 2014
• Application Blacklisting for Windows and Mac
• Updated User Alert (Windows and Mac parity)
• Expanded inventory collection
• Improved reliability
• Improved security
• AMP watchdog
Changes in 6.0 & 6.3

4. Dell World 2014
• KA-334 : Cannot replicate patches to UNC path
• KA-1231: Replicated large files keep on growing in size
• KA-1328: AMPAgent incorrectly terminates csrss.exe as its child process
upon exit
• KA-231: AMPAgent can utilize too much CPU on Mac and Windows
Key Field Issues Addressed in 6.0/6.3

5. Dell World 2014
• AMPAgent service/daemon running
• Valid amp.conf with the proper host specified
• Valid amp_auto.conf (6.3) with AMP port 52230
• Network characteristics of a healthy agent
• Heartbeat every 20 seconds
• Regular inventory (interval specified by K1)
• Cycle thru read/write operations (visible in debug)
What a healthy agent looks like

6. Dell World 2014
• amp.conf
• amp_auto.conf (6.3)
• inventory.xml
• kinventory.db
• InventoryData.Software
• ksw_process.db
• ksw_timestamps.db
• Metering_data.txt
• kbots_cache/kbots.xml
Data Files

7. Dell World 2014
• host=<hostname> : all you need in amp.conf is a host field, and the agent
can fill in the rest once it’s connected to the server.
• debug=true|all : use debug field to increase logging, but watch out for
rollover.
• The log will roll over at 4 MB and 5 old logs are kept.
• Starting with 6.3, all the below calculated properties are stored in
amp_auto.conf
• wto, rto, cto, crto : write, read, connect and connect-retry timeouts
• servercompress : control whether the agent automatically compress uploads
• maxDownloadSpeed : set the max speed in KB/s that the agent can download payloads at
• processtimeout=xxx : override default process timeout in milliseconds
• ampurl, ampport, weburl, webport, companyname, splashtext, etc… : some variables are
controlled by agent and should not be changed by the user
amp.conf & amp_auto.conf

8. Dell World 2014
• kinventory.db
• SQLite database used for inventory capture and for generating inventory.xml file.
• Useful for debugging when inventory.xml is missing or incomplete.
• Can be deleted to have kinventory repopulate in case of suspected corruption.
• inventory.xml
• The XML that describes the machine generated from the information in
kinventory.db
• Useful to check if agent is collecting the information correctly when troubleshooting
incorrect inventory data
kinventory.db & inventory.xml

9. Dell World 2014
• Software inventory collected to match against Dell Software Catalog
• Contains the list of all binaries and their attributes from the entire file system
• Contains some additional information such as Windows add/remove registry
keys
• Used by K1 to determine all the software titles installed on the system
• Not to be confused with regular inventory, which is more hardware oriented
InventoryData.software

10. Dell World 2014
• SQLite database introduced in 5.5 to capture the software metering data.
• This database stores all the real time metering data as processes
launch/terminate, and will be flushed when server asks for a report.
• Maximum rows is configurable. The default maximum is 5000 rows and can
be adjusted depending on the software meter flush interval.
• On the Mac, there are multiple copies of ksw_process database, one per
logged in user. The databases will have _username appended to the base
filename.
• E.g.: ksw_process_atsai.db
ksw_process.db

11. Dell World 2014
• SQLite database introduced in 5.5 to store information such as last alive date
• Used to recover in case software metering process itself is terminated and
unable to determine the termination time for process it is monitoring.
Provides a best guess answer as to when processed we were monitoring
terminated once software meter process starts back up.
• Like ksw_process.db, there are multiple DB on the Mac, one per user with
_username appended to the base filename.
• E.g.: ksw_timestamp_atsai.db
ksw_timestamp.db

12. Dell World 2014
• Software meter results flushed from ksw_process.db
• Contains a list of all processes that ran on the system, their attributes, start
time and end time as well as user that launched the process.
• This is generated when K1 asks the agent to flush its result and upload to
server.
metering_data.txt

13. Dell World 2014
• Lists all online and offline scripts.
• Provides ID and VERSION of active scripts, located in the same folder.
• Used by the boot, login and offline script engines to loop through and look
for applicable scripts.
• Updated by kbot number 3.
• C:Program Files (x86)DellKACErunkbot.exe 3 0
kbots_cache/kbots.xml

14. Dell World 2014
• The primary functions of the agent arehandled by the AMPAgent
service/daemon and 4 supporting plugins
• pluginWeb: handles script downloads, replication, log uploads.
• pluginRunProcess: handles scripts and runs processes thru runkbot.
• pluginPatching: handles detecting/deploying of patches.
• pluginDesktopAlerts: handles displaying broadcast user alerts or pre-install script
alerts.
Agent Plugins

15. Dell World 2014
• With 6.0, we consolidated all agent logs into a single log file KAgent.log.
• This allows for a better trace of exact events that happened on the agent
without having to cross reference all the logs and match up by time.
• The log contain the date, module and the function name that generated the
log.
[2014-10-14.19:58:32][KInventory:CInventoryData::Initi] KInventory InventoryData opened DB successfully
[2014-10-14.19:58:32][KInventory:CInventoryData::Initi] KInventory InventoryData populated the DB
[2014-10-14.19:59:15][KInventory:runInventory ] KInventory Inventory Capture completed and stored in
C:ProgramDataDellKACEinventory.xml
[2014-10-14.19:59:15][KCopy:UploadUsingCurl ] UploadFile: Server gzip compression is active
[2014-10-14.19:59:15][KCopy:UploadUsingCurl ] UploadFile: uploading file C:ProgramDataDellKACEinventory.xml.gz to
https://engk1agent3/service/inventory.php?KUID=F2C603AD-08C8-48D3-A556-25F2702F6D89&VERSION=6.0.32
Log File starting 6.0

16. Dell World 2014
• By default, Windows agent provisioning, removal and updates produce two debug logs
files:
• ampmsi.log: msiexec /L*v log file, found in the %TEMP% folder
• ampinstaller.log: custom action log file, found in the %TEMP% folder as well as the KACE data
folder
• The log files indicate when agent installed and/or uninstalled, and flags and properties
passed in.
• When run non-silently, the agent will honor the current locale language if is one of the
10 supported languages. Otherwise, default is English.
• Use the new GPO Provisioning tool to help create the GPO. Will not need to use the
setlang VB script to override default locale anymore if use the GPO Provisioning tool.
• https://www.kace.com/support/resources/kb/solutiondetail?sol=133776
Windows Installer

17. Dell World 2014
• /i <msi file> : install msi file (example: msiexec /i amp.msi)
• /x <msi file> : The preferred way to uninstall agent is to run “AMPTools uninstall”
• /qn : silent install (example: msiexec /qn /i amp.msi)
• /L*v <log_file> : create log file (example: msiexec /L*v amp.log /i amp.msi)
• HOST=<host_name> : set amp.conf host value (example: msiexec /i amp.msi
HOST=kbox7.acme.com)
• Alternatively: you can append to msi filename (example: msiexec /i amp_kbox7.acme.com)
• DEBUG=true : set amp.conf debug value (example: msiexec /i amp.msi HOST=kbox7 DEBUG=true)
• NOHOOKS=1 : don’t install boot & logon hook dlls (msiexec /i amp.msi HOST=kbox7
NOHOOKS=1)
• CLONEPREP=1 : do not start AMP service until next reboot (msiexec /i amp.msi CLONEPREP=1)
Windows Installer Common Properties

18. Dell World 2014
• Introduced in 6.3, AMPWatchDog monitors the health of agent and perform
simple recovery to address majority of the common issues
• Conditions which AMPWatchDog monitors currently:
• AMPAgent executable exists in expected location
• AMP configuration file exist with server host
• AMP Service/Daemon not running (Restart agent)
• Inventory.xml exists and is less than 3.5 times the configured frequency (Restart
agent)
• ampport=port exists in amp_auto.conf (Restart agent)
• Logs netstat output relevant to ampagent connection
AMPWatchDog

19. Dell World 2014
• Agent by default will accept http connection if https can not be established with K1
• Uses OpenSSL 1.0.1h with Heartbleed addressed
• Agent is not affected by the POODLE vulnerability
• Only publicly signed certificates honored
• The signing root authority must be included in the curl certificate bundle (cacert.pem
in agent data directory, exported from Mozilla)
• Different SSL options to set in amp.conf
• sslrequired: Agent will not fall back to http if https can not be established
• verifyssl: Agent will verify the server certificate, implied sslrequired
• TLS will be on by default in the future
Agent Security

20. Dell World 2014
• What are the basic stuff to look for when suspecting agent issues?
• Turn on Debug=true
• Is agent running?
• Is the KAgent.log being updated? (under users directory)
• Is the agent connected? DNS issues? Use FQDN when possible. Can you resolve the
host and telnet to port 52230?
• Agent is 32 bits, remember that for any registry/file system OS redirections
• Test agent functionalities by running runkbot 2 0
• Windows: c:Program Files (x86)DellKACErunkbot 2 0
• Mac: /Library/Application Support/Dell/KACE/bin/runkbot 2 0
• Linux: /opt/dell/kace/bin/runkbot 2 0
• Are there crash dumps?
• Windows: c:ProgramDataDellKACE*.dmp
• Mac: Console
Trouble Shooting

21. Dell World 2014
• Eliminate possible system conflicts (disable firewall, turn off AV program,
etc.)
• Look at the log file in the KACE data folder
• Win XP: “C:Documents and SettingsAll UsersDellKACEuser
• Vista and Win 7: C:ProgramDataDellKACEuser
• Mac: /Library/Application Support/Dell/KACE/data/user
• Linux: /var/dell/kace/user
• Verify valid amp.conf file, and regenerate it if needed using AMPTools
• AMPTools resetconf host=kbox7.acme.com
• Verify existence of valid kbots_cache/kbots.xml files and supporting script
xml files
Debugging Tips

22. Dell World 2014
• Verify K1000 host name resolves using browser or command line
• ping kbox7.acme.com
• telnet kbox7.acme.com 52230
• Enable debugging by running “AMPTools debug=true” which will set debug
value in amp.conf and restart the agent.
• Alternatively, you can temporarily enable debugging on Windows without
restarting the agent using “sc control ampagent 199” (This only enables
debug for AMPAgent, not all binaries)
Debugging Tips (continued)

23. Dell World 2014
• HexDump
• Setting HEX_DMP environment variable before starting AMPAgent will cause
AMPAgent to log the exact informtion it tries to send and receive over the wire.
• HeartBeat
o [Smurf_write_SYNC ] -------------------------------------------------------------
------------------- [Smurf_write_SYNC ] 00000000 00 00 00 01 05
..... [Smurf_write_SYNC ] 00000005 ----------------------------------------------
-----------------------
• Agent connection string
o [Smurf_write_SYNC ] -------------------------------------------------------------
-------------------
o [Smurf_write_SYNC ] 00000000 00 00 00 01 02 00 06 00 00 00 21 36 34 34 35 46
..........!6445F
o [Smurf_write_SYNC ] 00000010 45 42 34 34 34 33 37 34 36 33 31 41 30 33 39 35
EB444374631A0395
o [Smurf_write_SYNC ] 00000020 30 42 31 31 32 34 39 37 34 31 41 00 00 00 00 0a
0B11249741A.....
o [Smurf_write_SYNC ] 00000030 30 2c 32 2c 34 2c 36 2c 35 00 00 00 00 6b 76 65
0,2,4,6,5....kve
Debugging Tips (continued)

24. Dell World 2014
• Basic information to collect when observed crashes
• Collect crash dumps or crash call stack
• Windows 2008/Vista and newer:
HKEY_LOCAL_MACHINESOFTWAREMicrosoftWindowsWindows Error
ReportingLocalDumps
• Mac: Agent crashes will show under System Diagnostic Report in Console. Collect
the crash call stack.
• Collect agent version
• Collect agent log in debug mode
• All the information are important in order to debug and simulate the failure
successfully. Need the agent version in order to match the crash dump up
with the debug symbols.
Crashes Diagnostics

25. Dell World User Forum
Q & A

26. Dell World User Forum
Thank you.