VMS Troubleshooting Guide

Digital Video Surveillance Basic Troubleshooting Guide
Compliments of Seneca < http://www.senecadata.com >
Contents
VMS Troubleshooting Questions ..............................................................................................................2
Typical DSS system view .......................................................................................................................2
A) Basic ..............................................................................................................................................2
B) System Topology Information.......................................................................................................3
C) Questions if intermittent camera connectivity problems exist....................................................3
D) Things to look out for that might cause issues. ............................................................................3
E) Local IT/Engineering gets involved for the following troubleshooting.........................................3
TOOLS........................................................................................................................................................4
Basic Windows Performance Tools Detailed Usage .............................................................................4
Task Manager/Resource Monitor.........................................................................................................5
Resource Monitor .................................................................................................................................6
PERFMON: Dynamic and long term resource viewing and collecting................................................7
RealTemp …. Looking for hot systems ................................................................................................9
GPUZ a look at your video subsystem...................................................................................................9

VMS Troubleshooting Questions
Typical DSS system view:
A) Basic
1. What is the problem environment...i.e....what is the server/client being asked to do and what is
it not doing?
 "Are you using the appliance for recording, viewing or both?"
o Note that large scale viewing on a recorder is not recommended.
 "What are you having trouble with…. recording, viewing or both?"
 Physical Environment (looking for Temperature, Humidity, Vibration, Dust) (RealTemp
log)
 Windows Software/Hardware environment (Task Manager and Resource View…. CPU,
Memory, HDD, Network…anything either pegged or zero?)
 If a RAID array is being used…is it in Writeback Always mode?
2. If you are having problems viewing video on a separate client / workstation
 How many simultaneous streams on the system?
 Live or Playback (Playback stresses server HDD whereas Live does not)
 Resolution & FPS total being attempted

 One Client machine or All Client machines (All suggests network issue or underspec’d
system)
3. If application is crashing,
 Look at GPU utilization and memory utilization (GPUZ is a great tool for monitoring this)
*note some Client application use hardware resources differently
 Where is the application memory at the time of the crash (use Task Manager-Resource
view)
B) System Topology Information
1. How many NICs are being utilized
2. NIC bandwidth (BW) =? (Resource Monitor)
3. Network Topology =? (Subnet usage important) i.e. the recorder is on separate subnet from the
client viewing application, are they sharing BW with Corporate data?
4. Storage BW =? (Only for a Server/Storage appliance) (Resource Monitor)
5. What VMS are you running? (VMSs do use resources differently)
6. What version are you on?
7. System hardware details = (BOM)
C) Questions if intermittent camera connectivity problems exist
1. Is one camera affected, some or all?
2. If all cameras, are they the same brand, make, model, and firmware version?
3. If ‘some’ then are they connected to a common device?
4. Using TCP or UDP for your RTSP stream? UDP will drop packets and frames, causing loss.
D) Things to look out for that might cause issues.
1) Is there other network traffic or is it closed for security only?
2) Are there windows updates and other utilities such as defrag, virus scan etc running on the
storage discs?
3) Is windows firewall on / off if their network policy allows for it to be off please turn it off
E) Local IT/Engineering gets involved for the following troubleshooting.
1. If some cameras, what's common, the switch? The network? The subnet?
 Good managed switches will give you statistics on dropped packets and other issues. I
would do that first before setting up NAGIOS or MRTG from scratch to troubleshoot.
Besides, those packages require switches to have that basic functionality anyway (sFlow
or SNMP) so look at the switch first!
2. Network cabling good? A bad cable causes issues.
3. If you rule out the network, then look at your storage and your hosts.

TOOLS
Here are a few tools that are easily used on a Windows system to aid in debugging.
A) Windows has Performance Monitor.
a. "Setup metrics on CPU, RAM, page faults, network statistics and I/O, disk stats and I/O,
disk queue depth, etc.
b. Monitor all statistics against known good numbers. " (see below)
B) Windows Event Logs.
a. Use the filtering option on the right to look for warnings/errors and critical events
C) Windows Task Manager which is a high level view of everything.
D) Storage logs.
a. Controller and disk failures can cause I/O to go crazy on those path's as rewrite
operations are sent out. This may bring down entire production critical systems on top-
tier storage arrays.
b. LSI RAID controllers have a ‘lsigetwin’ tool available for automatic dump of info.
E) File system integrity.
 How does SCANDISK come back? Bad records and bad allocation tables can do bad
things.
Basic Windows Performance Tools Detailed Usage
Here are details basic Windows tools that will help you determine how well your Windows PC is
performing.

Task Manager/Resource Monitor
Figure 1 What to open within Task Manager
How to get this:
Right Mouse Button on the Task Bar as shown below then pick Task Manager. An alternate method is
to use the Alt-Ctrl-Del combo and select it.
Figure 2 Using RMB to open Task Manager

Once Task Manager (TM) is opened, select the ‘Performance’ tab and also select ‘Resource Monitor’
which will open a new window of information as shown in Figure 1.
Take away from this window: A quick view of the CPU and Memory being used up shown by the yellow
square.
Any value greater than 70% should raise a flag.
Resource Monitor
The Resource Monitor window looks like this:
‘Click’ on the yellow highlighted items to expand each section.
Take away from this window:
This adds in information about the Storage (Hard Disks) and Network resources being used up.
Once again, you do not want to see a number higher than 70% for the Network.
For the Storage, under the ‘Disk’ tab, you are verifying that the video is being written to the correct
array [‘File’ column] and that the ‘Queue Length’ [red square] is not very large (single digits…preferably
less than one) as shown in Figure 3.

Figure 3 Storage Resource tab
If the resource usage of all these elements is less than 70%, then it is possible to give the system more
work to do.
PERFMON: Dynamic and long term resource viewing and collecting.
Create a User Defined data collector set as follows:
1) Start typing ‘performance’ in the Win7 command line and select Performance Monitor (perfmon on
other versions)

2) To collect the information in real time, add the counters under Performance->Monitoring Tools-
>Performance Monitor. Save this resulting setup file in the documents dir for easy recall later
(RMB…. Save Settings).
View of ‘add counters’ and the order of use pic:
(a) Step 3 is scroll to desired counter
(b) Step 4 is select counter
(c) Step 5 is expand counter list
(d) Step 6 is select desired counters
(e) Step 7 adds them to the list
Add these counters :
b) From each archiving LogicalDisk
i) Disk Read Bytes/sec for the drive where the live/archived video are stored
ii) Disk Write Bytes/sec for the drive where the live/archived video are stored
iii) Avg. Disk Queue Length for the drive where the live/archived video are stored (IMPORTANT)
iv) Current Disk Queue length for the drive where the live/archived video are stored
(IMPORTANT)
v) Disk Writes/sec for the drive where the live/archived video are stored
c) From Network Interface <select the NICs you are using>
i) Bytes Received/sec
ii) Bytes Sent/sec
iii) Bytes Total/sec
iv) Packets Received Errors
v) Packets Received Discarded
d) From Processor
i) %Processor Time for the _Total and all logical processor instances
e) From Memory
i) Pages/sec
ii) Available Mbytes (lets you discover memory leaks)
3) On Performance Monitor (1 in pic above) do RMB then create Data Collector <<<< do this to record
a log file to review later or run over a long time >>>>

4) Type a name such as XYZ Server Storage Certification
5) Browse for the path where you want to save the logs, then Finish
RealTemp …. Looking for hot systems
Activate the log file as shown….
The log file is RealTempLog.txt and stored where RealTemp is installed
GPUZ a look at your video subsystem
This tool tells us about the Video Card and will log the usage. Activate the logging.
Note that on systems with multiple Video sources (ie mobo and card), you need to select the one you
are using. This can be done dynamically.

VMS Troubleshooting Guide

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a VMS Troubleshooting Guide

Semelhante a VMS Troubleshooting Guide (20)

VMS Troubleshooting Guide