SysMon Documentation

SysMon is a generic system monitoring tool. It captures a range of events in real time, processes them, and enables their viewing. SysMon is built as extendable and a very flexible platform. SysMon is a client server system. The event collection is the server side and is performed by a resident service with a very small foot print. The client is a windows application responsible of the graphic user interface. The client connects to the server through TCP/IP.

SysMon Plug-ins

The capturing of events is performed by pluggable software modules called SysMon plug-ins. Each plug-in is responsible of capturing a certain type of events. The plug-ins currently available are:

Debug Output: captures debug output from software
Events: captures system events
IIS: captures hits on the Internet information services web server
Log4j: captures debug messages from Java applications through log4j
Cron: Generates events every configurable amount of time thus enables activating logic at any desired interval.
Performance counters: Collects information from any component capable of emitting data through windows performance counters interface.

SysMon Processors

SysMon processors are pluggable software modules responsible of processing the captured events. The currently available are:

Scripting Engine: Capable of executing custom logic in javascript.

File-System Log: writes the events to the file system.

The processors are arranged in a tree structure. Each level in the tree will process each event and then route it to its child processors. This tree structure is called a flow.

SysMon Client Overview

SysMon client is responsible of the graphical user interface of the system. Launching the client can be done in several methods:

By double clicking on the SysMon taskbar icon
By right clicking on the SysMon taskbar icon and selecting open from the popup menu.
By the Windows Start Menu->Programs->SysMon->SysMon Viewer
By double clicking on SysMon.exe in windows explorer

The SysMon client connects to the SysMon server through TCP/IP protocol.

Here is the client main screen:

Each line represents an event. By double clicking on an event, the event details window is opened:

File->Connect

Use this operation in order to connect to a SysMon server on the network.

IP/Name: type the IP address or the host name of the computer on which the SysMon server you wish to connect to is running. The drop-down list contains recent computers that where connected to from this client.

Port: The TCP/IP port which the SysMon server is configured to listen on. The default is 8500.

Password: The password that is configured on the SysMon server. When connecting to the SysMon server on the local host, the password is not needed and is ignored.

Remember Password: Check if you with the password to be saved on this client machine.

In order for a connect operation to succeed, the following conditions must be met:

SysMon Server must be running on the computer that you are trying to connect to
The network must enable TCP/IP connection between the client and the server
The SysMon Server must be configured to enable remote connections from the client
The password used must match the configured password on the SysMon server

If the client is already connected to a SysMon server, it will first disconnect from it.

File->Connect Local

Use this operation in order to connect to a SysMon server on the local host. If the SysMon server is not running, the client will attempt to launch it.

If the client is already connected to a SysMon server, it will first disconnect from it.

File->Disconnect

Use this operation in order to disconnect the client from the server. If the client is currently viewing a file, the file will be closed.

File->Save

Use this operation in order to save any changes made to the file being viewed.

File->Save As

Use this operation in order to save any changes made to the file being viewed into another file name.

The selection only option will save only the selected lines.

File->Open File

Use this operation in order to open a previously saved SysMon log file.

File->Recent File

Use this operation in order to open a previously saved SysMon log file that was recently opened in the client.

File->Print

Use this operation in order to print the log.

File->Exit

Use this operation in order to exit the SysMon client. Closing the client will not close the server. The server will continue monitoring the system while the client is closed.

Edit->Undo

Use this operation in order to undo the last deletion performed.

Edit->Cut

Use this operation in order to delete the selected lines from the log and place them in the clip board. This enables a paste operation.

Edit->Copy

Use this operation in order to copy the selected lines from the log to the clip board. This enables a paste operation.

Edit->Delete

Use this operation in order to delete the selected lines from the log.

Edit->Delete All

Use this operation in order to delete all the data from the log.

Edit->Select All

Use this operation in order to select all the lines in the log.

Edit->Find

Use this operation in order to find information in the log.

Text to find: type the text you wish to find.

Match whole word only: will only find the typed text if it is a complete word.

Match case: find the text only if there is a precise match in letters case.

Regular expression: accept regular expression in the typed text in order to extend the search capabilities. Click on the black arrow to the right of the text input line in order to open a menu with all the different regular expressions you can use:

Name	Regular Expression	Description
Any Character	.	Accept any single character
Character in/Not in Range	[ ]	Any one of the characters contained in the brackets, or any of an ASCII range of characters separated by a hyphen (-). For example, x[abc]y matches xay, xby and xcy, and r[eo]+d matches red, rod, reed, and rood, but not reod or roed. x[0-9] matches x0, x1, x2, and so on. If the first character in the brackets is a caret (^), then the regular expression matches any characters except those in the brackets
Beginning of line	^	Match text only if it starts at the beginning of the line. For example, ^abc
End of line	$	Match text only if it ends at the end of the line. For example, xyz$
Or	\\|	Or. For example abc\\|xyz will match either abc or xyz.
0 or More Matches	*	Match any number of occurrences of the expression preceding the . For example, HTP will match HTTP and HP.
1 Or More Matches	+	Match at least one occurrence of the expression preceding the +. For example, HT+P will match HTTP but not HP.
Group	\{\}	Any sequence of characters between the escaped braces. For example, \{ju\}+fruit finds jufruit, jujufruit, jujujufruit, and so on. Note that it will not find jfruit, ufruit, or ujfruit, because the sequence ju is not in any of those strings
Word Bound	\b	Word bound. For example, \bed will find education but not medical
Not Word Bound	\B	Not word bound. For example, \Bed will find medical but not education

The regular expression menu includes some pre-set combinations for convenience:

Name	Regular Expression
Whitespace	[ \t]
Alphanumeric character	[0-9a-zA-Z]
Alphabetic character	[a-zA-Z]
Decimal digit	[0-9]
Hexadecimal digit	[0-9a-fA-F]

Direction: Up or Down. Search for the next match in older events relative to the currently selected event (Up) or in newer events (Down).

Source: type text to be matched in the event source column. If left blank, all sources are matched.

Edit->Find Next

Use this operation in order to find the next match of the current find parameters. If the SHIFT key is held together with F3, the current search direction is reversed.

Edit->Find Prev

Use this operation in order to find the previous match of the current find parameters. If the SHIFT key is held together with F3, the current search direction is reversed.

Edit->Toggle Bookmark

Use this operation in order to toggle the bookmark state at the current log line. This means that if in the current log line there is no bookmark, a bookmark will be set and visa versa. Setting a bookmark on a log line allows quick finding of that line by using the Next Bookmark and Prev Bookmark commands.

Edit->Next Bookmark

Use this operation in order to jump to the next bookmark.

Edit->Prev Bookmark

Use this operation in order to jump to the previous bookmark.

Edit->Remove All Bookmarks

Use this operation in order to delete all the bookmarks.

View->Toolbar

Use this operation in order to toggle the visibility state of the toolbar.

View->Status Bar

Use this operation in order to toggle the visibility state of the Status Bar.

View->Auto Scroll

Use this operation in order to toggle the log auto scroll feature. When auto scroll is enabled, the log will automatically scroll in order to make every new line added to it visible.

View->Always on top

Use this operation in order to toggle the always on top feature. When always on top is enabled, SysMon window will become always on top of other windows. It may still be covered by other windows that are also marked as always on top.

View->Filter/Highlight

This option opens the View Filters page in the settings dialog.

This configuration page enables to either highlight/decorate log lines or to hide them based on find criteria. See find for details.

View->Font

This option opens windows standard font selection dialog:

Use this dialog to select the desired font to user in order to show the log data. This font will also be used for printing.

Tools->Settings

This option opens the settings dialog. See Configuration for details.

Help->Contents

This option will open the help contents:

Help->Index

This option will open the help index:

Help->SysMon Web Homepage

This option will open the default web browser and will navigate to http://www.sysmon.com

Help->Check for Updates

This option will connect to http://www.sysmon.com and will check for new updates.

Help->About SysMon

This option will open the about dialog:

SysMon Server Overview

SysMon server is implemented as a system service. It can automatically start when the operating system is started. The server listens on a TCP/IP port (the default port is 8500) for incoming client connections. Multiple clients may connect to the server concurrently. The server performs event collection through special modules called SysMon plug-ins. Once an event is captured, it is processed by SysMon processors. A processor is a software module that receives a notification for each event captured. It can then process this event. After processing the event, the event is routed to the next processors that forming a processing flow. This mechanism forms a very flexible event capturing and processing system.

Management

SysMon is resident in a system service.

Stopping SysMon service:

Open a command console by Start->Run, type ‘ cmd’<enter>
Type ‘net stop SysMonSvc’<enter>

Starting SysMon service:

Open a command console by Start->Run, type ‘cmd’<enter>
Type ‘net start SysMonSvc’<enter>

The SysMon service is configured by default to start automatically on system startup.

Configuration Overview

Configuration of SysMon is performed by a SysMon client application. In order to configure a server, the client has to connect to it first. Connecting to a server running on the same machine as the client does not require any password. Connecting to a remote server requires both a password and proper security settings on the server. The server must also allow remote configuration.

You configure SysMon through the settings dialog:

Use this dialog to configure both SysMon client and server. The client configuration includes:

· The columns to show

· The View Filters

The server configuration includes:

· General settings

· Security

· Plugins

· Flow

· Processors settings:

o Scripting

o File-System Log

General Configuration

The General configuration page contains two groups:

General

o Automatically start service at system startup: check to have SysMon service launch automatically when Windows start
o Show icon in the taskbar’s status area: check to have SysMon add an icon to the status area

Columns

o Check/uncheck to show/hide specific columns in the SysMon viewer.

View Filters Configuration

Use this configuration page to control log lines highlighting/hiding. Log lines selection is based on find criteria. See find for details.

Security Configuration

Use this configuration page to control connections from remote SysMon viewers to SysMon server.

Allow remote connections: check to enable SysMon viewers from other machines to connect to this SysMon server

Allow remote configuration: check to enable SysMon viewers from other machines to configure this SysMon server

Limit number of remote connections: The SysMon server will prevent more that the specified number of remote viewers to connect to it

Remote connection password: remote connection viewers will be prompted to enter this password

Allow remote connections only from: Enter a list of IP addresses that SysMon server will allow to connect to it

Don’t allow remote connections from: Enter a list of IP addresses that SysMon server will prevent from connecting to it.

Add IP addresses to the list by clicking on the Add button

Remove IP addresses from the list by clicking on the Remove button

Edit an IP address in the list either by clicking on the address once and then on the Edit button or by double clicking on the address

Plug-ins Configuration

Use this configuration page to select which SysMon plug-ins to activate.

Flow Configuration

Use this configuration screen to define SysMon’s event processing flow. Each event captured by the plug-ins is passed through the processors flow. Each event, at each flow node is routed to each of the node’s children starting at the flow’s root level. Manipulate the flow structure by right clicking on flow nodes and selecting operations from the pop-up menu. You can also use drag and drop of nodes in the flow in order to move entire branches.

Scripting Engine Processor Configuration

Use this configuration page to configure the scripting engine event processor. The script drop down list contains all the scripts under the scripts folder on the SysMon server. A script should be written in the javascript language. Scripts can be included by other scripts by using the @include directive (For example: @include=mylib.js).

A script can:

manipulate events by changing their properties
create new events
drop events
perform http hits
Send emails
Generate a SysMon alert event
Execute a program
Ping a host

For each event, SysMon scripting engine will invoke a function in the script called onEvent and pass the event as a parameter. In Jscript for example:

function onEvent( ev )

{

// function body

return true;

}

The return value from the function determines whether the event is dropped or not. true means the event is routed forward and false means it will be dropped.

The event properties are:

data: a dictionary of values. Each plug-in has a list of data values:

dbg:

§ ProcessId (integer): The process identifier that generated this debug output

§ GenerationTime (time): The time when this event was captured

§ Strings (array of stings): All the string values that this event contains

§ CategoryId (integer): An identifier of the event category

§ EventId (integer): An identifier of this event as provided by the event generator

§ CategoryName (string): A classification as provided by the event generator

§ Type (integer): A classification of the event severity:

· 0 = Success

· 1 = Error

· 2 = Warning

· 4 = Information

· 8 = Audit success

· 16 = Audit failure

§ Log (string): The name of the log to which the event was destined: Application/Security/System

§ User (string): The user context of this event

§ GeneraionTime (time): The time when this event was generated

cron:

§ GeneraionTime (time): The time when this event was generated

§ name: the name of this timer

§ interval: the number of milliseconds this timer is configured to delay from the previous one.

IISMon:

§ GeneraionTime (time): The time when this event was generated

§ HttpStatus: the http status of the monitored hit

§ Duration: the number of milli seconds it took to process the http hit

§ Secured: True if the hit was protected by SSL.

§ Operation: the http request verb (GET, POST…)

§ ContentLength: the number of bytes in the response

§ User: The user as passed in the request

Log4j:

§ GeneraionTime (time): The time when this event was generated

§ threadName: The name of the java thread generating the message

PerfCounters:

§ GeneraionTime (time): The time when this event was generated

§ <The configured counter name>

plugin: The capturing plug-in short name
text: The event body
source: A string identifying the event generator
time: The time in which this event was captured, The time may differ from the event generation time

Example of manipulating event properties:

function onEvent( ev )

{

ev.text = “this is the modified text”;

return true;

}

The available functions

addEvent

Adds an event to be processed

Object addEvent(string text, string source, string data, string typedef, string plugin, int type)

text: the event title

source: identifies the event origin

data: the event data as a javascript object.

Typedef: defines the type of each of the data elements. Each data type is encoded by a single character. Here is the type definition encoding:

Character	Type
s	String
n or d	Integer
A	Array of strings
t	Time as a double value that represents vb variant date time format
b	Boolean (0 or 1)

plugin: identifies the plugin generating this event

type: 0=info, 1=warning, 2=error, 3=alert

Return value: an event object

Sample:

var data = new Object();

data.aaa=”111”;

data.bbb=222;

addEvent(“text”, “my script”, data, “s;d”, “scr”, 0);

urlHit

Performs an http hit

string urlHit(string url, string params, int timeout)

url: the url to hit. May include GET parameters

params: POST parameters

timeout: the time in milli-seconds to wait for a response

Return value: the response body as a string

Sample:

var response = urlHit(“http://www.sysmon.com/update?ver=1.0.0.9”, “param1=val1&param2=val2”, 5000);

sendEmail

Sends an email message.

string sendEmail(string from, string/array to, string subject, string body, string host, int port, string user, string password)

from: identifies the sender

to: email address of recipient(s)

subject: message title

body: message body

host: server name to use for sending (localhost is the default)

port: the tcp/ip port to connect to in the host (25 is the default)

user: the user name used when connecting to host (optional)

password: the password to use when connecting to host (optional)

Return value: error message if failed

Sample:

var err = sendEmail(“me”, “you@yourdomain.com”, “sysmon”, “Please download the latest version”);

alert

Adds an alert event. It is a short form for addEvent for showing alert events

void alert(string message)

message: the alert message title

Sample:

alert(“Disk space is low”);

As a result, sysmon will add an alert event to the log and show a balloon out of the taskbar icon:

exec

Executes a command on the host operating system

bool exec(string command)

command: the command to execute

Return value: true on success else false

Sample:

var success = exec(“notepad.exe”);

ping

Sends a series of icmp messages to a host

int ping(string host, int numPackets, int timeout)

host: the target host name

numPackets: the number of icmp messages to send

timeout: the time limit in milli-seconds

Return value: a handle to the ping request

Notes:

When the target host replies, a callback function named onPingReply is called.

void onPingReply(int handle, int time, int addr, int index)

handle: the ping handle that was returned from the ping call

time: the time in milli-seconds it took for the host to reply

addr: the tcp/ip address of the target host

index: the 0 based index of the icmp message

When there is an error, a callback function named onPingError is called.

void onPingError(int handle, string error, int index)

handle: the ping handle that was returned from the ping call

error: the error message

index: the 0 based index of the icmp message

Sample:

var total, min, max, numReplies, host_addr;

function onEvent( ev )

{

if(ev.source == "cron" && ev.data["name"] == "ping") {

total = 0;

min = 10000000;

max = 0;

numReplies = 0;

ping("www.google.com");

return false;

}

return true;

}

function onPingError(handle, err, index)

{

alert(err);

if(index == 4)

report_ping();

}

function onPingReply(handle, time, addr, index)

{

numReplies++;

total += time;

if(time < min)

min = time;

if(time > max)

max = time;

host_addr = addr;

if(index == 4)

report_ping();

}

function report_ping()

{

if(numReplies == 0)

return;

var msg = "Reply from " + host_addr + ". Replies=" + numReplies + "/5 Avg=" + (total/numReplies) + " min=" + min + " max=" + max + " total=" + total;

addEvent(msg, "ping");

}

abortPing

Aborts a ping operation.

void abortPing(int handle)

handle: the ping handle that was returned from the ping call

File-System Log Processor Configuration

Use this configuration page to configure the file system log.

Path: select the path in which to save the log data

Size Limit: Specify the number of mega-bytes the log files may not exceed.

Time Limit: Specify the amount of time to keep data on the server.

The log will delete the oldest data every time the size or time exceeds the limit.