![[symon]](img/symon.png)
symon(8) manual:
SYMON(8) System Manager's Manual SYMON(8)
NAME
symon – system monitor
SYNOPSIS
symon [-dtuv] [-f filename]
DESCRIPTION
symon is a lightweight system monitor that measures cpu, load,
filesystem, interface, disk, memory, pf, pf queues, mbuf, proc and sensor
statistics every 5 seconds. This information is then spooled to symux(8)
for further processing.
symon has been designed to inflict minimal performance and security
impact on the system it monitors. symux(8) has performance impact
proportional to the amount of streams it needs to manage. Ideally symux
should live on a different system and collect data from several symon
instances in a LAN.
By default, symon will drop privileges and chroot(2) to home of the
_symon user. This behaviour is not strictly needed for the cpu, mem,
mbuf, disk debug and interface probes as these will work even when symon
is started as nobody.
The options:
-d Stop symon from becoming a daemon and show debug information on
stdout.
-f filename
Read configuration from filename instead of /etc/symon.conf.
-t Test configuration file and exit.
-u By default symon will chroot(2) into _symon user home directory.
The -u disables this behaviour.
-v Show version information.
CONFIGURATION
symon obtains configuration data from /etc/symon.conf. The configuration
file contains monitor stanzas that define what resources should be
monitored and to which symux(8) the information should be streamed to.
Multiple monitor statements are allowed. Whitespace, newlines and text
behind '#' are ignored. The format in BNF:
monitor-rule = "monitor" "{" resources "}" [every]
"stream" ["from" host] ["to"] host [ port ]
resources = resource [ version ] ["(" argument ")"]
[ ","|" " resources ]
resource = "cpu" | "cpuiow" | "debug" | "df" | "flukso" |
"if" | "io" | "load" | "mbuf" | "mem" | "pf" |
"pfq" | "proc" | "sensor" | "smart"
version = number
argument = number | name
every = "every" time
time = "second" | number "seconds"
host = ip4addr | ip6addr | hostname
port = [ "port" | "," ] portnumber
Note that symux(8) data files default to receiving data every 5 seconds.
Adjusting the monitoring interval will also require adjusting the
associated symux(8) datafile(s).
The pf probe will return data that is collected for the loginterface set
in /etc/pf.conf(5).
The Linux io, df, and smart probes support device names via id, label,
path and uuid.
The FreeBSD io, df, and smart probes support gpt names, ufs names, ufs
ids and paths.
The OpenBSD io probe supports device uuids.
EXAMPLE
Here is an example OpenBSD symon.conf that monitors cpu, memory, pf,
interfaces xl0/de0/lo0/wi0, disks wd[0-3]/cd[0-1], debug variables debug0
to debug19 and streams that information to localhost on port 2100.
monitor { cpu(0), mem, pf, if(xl0), if(de0),
if(lo0), if(wi0), io(wd0), io(wd1),
io(wd2), io(wd3), io(cd0), io(cd1),
io(ccd0), df(sd0a), df(sd0d), df(sd0e),
debug, proc(httpd) } stream to 127.0.0.1 2100
EXAMPLE
Here is an example Linux symon.conf that monitors cpu including iowait,
memory, load, interface eth0, io and df for a set of disks every 5
seconds. Smart data is to be collected every 60 seconds. Disks in the
smart and io statements are identified using ids, filesystem volumes in
df using labels.
monitor { smart(ata-Hitachi_HDS722020ALA330_JK1130ABABABAB),
smart(ata-Hitachi_HDS722020ALA330_JK1130ACACACAC),
} every 60 seconds stream to 192.168.0.2 port 2100
monitor { cpuiow(0), cpuiow(1), mem, if(eth0),
io(ata-Hitachi_HDS722020ALA330_JK1130ABABABAB),
io(ata-Hitachi_HDS722020ALA330_JK1130ACACACAC),
df(data_1),
df(data_2),
df(data_3),
df(home),
df(var),
load
} stream to 192.168.0.2 port 2100
SIGNALS
SIGHUP Causes symon to read /etc/symon.conf. symon will keep the old
configuration if errors occured during parsing of the
configuration file. Note that the chroot(2) may cause resources
to become unattainable, most notably the configuration file
itself.
FILES
/var/run/symon.pid
Contains the program id of the symon daemon.
/etc/symon.conf
symon system wide configuration file.
BUGS
Every monitored resource mentioned /etc/symon.conf gets queried.
Mentioning, for example, cpu(0) twice for different muxes will result in
two distinct cpu(0) measurement actions.
The proc module is too simple: memory shared between two instances of the
same process is simply counted twice.
symon does not check whether all resources mentioned in /etc/symon.conf
exist.
AUTHOR
Willem Dijkstra <wpd@xs4all.nl>. Daniel Hartmeier helped to port to big-
endian architectures. Matthew Gream helped to port symon to other BSD
platforms.
Port contributors: Marc Balmer, Tito Dal Canton, Matthew Gream, Daniel
Hartmeier, Lars Kotthoff, Constantine A. Murenin, J. Martin Petersen,
Fredrik Soderblom, Harm Schotanus and Martin van der Werff.
Valeriy Leshchinskiy maintains a windows symon client at
https://github.com/ValHazelwood/SymonClient .
SEE ALSO
symux(8)
Linux 6.13.5-arch1-1 April 4, 2012 Linux 6.13.5-arch1-1
symux(8) manual:
SYMUX(8) System Manager's Manual SYMUX(8)
NAME
symux – symon stream multiplexer
SYNOPSIS
symux [-dltv] [-f filename]
DESCRIPTION
symon(8) is a lightweight system monitor that measures cpu, load,
filesystem, interface, disk, memory, pf, pf queues, mbuf, proc and sensor
statistics every 5 seconds. This information is then spooled to symux for
further processing.
symon(8) has been designed to inflict minimal performance and security
impact on the system it monitors. symux has performance impact
proportional to the amount of streams it needs to manage. Ideally symux
should live on a different system and collect data from several symon(8)
instances in a LAN. symux stores the incoming streams in .rrd files and
distributes the information to connected listeners. Listeners can connect
to symux on a fifo and receive incoming symon(8) transmissions decoded
into ascii.
symux needs no specific privileges besides being able to open it's ports
and the rrd files. It should be run as nobody.
The options:
-d Stop symux from becoming a daemon and show debug information on
stdout. Use this setting to find hosts or specific statistics
that do get sent, but are ignored due to configuration.
-f filename
Read configuration from filename instead of /etc/symux.conf.
-l List rrd files found in active configuration.
-t Test configuration file and exit.
-v Show version.
CONFIGURATION
symux obtains configuration data from /etc/symux.conf. The configuration
file contains one mux stanza that defines on what host address and port
symux should listen to for incoming monitored data. There is a source
section for every host that is to be monitored. The source section
defines what data to accept and where to write that data to. In the case
that a source is of another address family than the mux stanza, i.e.
source = ipv6 with mux = ipv4, a listen port of the sources' family is
opened using the unspecified address. Whitespace, newlines and text
behind '#' are ignored. The format in BNF:
stmt = mux-stmt | source-stmt
mux-stmt = "mux" host [ port ]
host = ip4addr | ip6addr | hostname
port = [ "port" | "," ] portnumber
source-stmt = "source" host "{"
accept-stmts
[ write-stmts ]
[ datadir-stmt ] "}"
accept-stmts = accept-stmt [accept-stmts]
accept-stmt = "accept" "{" resources "}"
resources = resource [ version ] ["(" argument ")"]
[ ","|" " resources ]
resource = "cpu" | "cpuiow" | "debug" | "df" | "flukso" |
"if" | "io" | "load" | "mbuf" | "mem" | "pf" |
"pfq" | "proc" | "sensor" | "smart"
version = number
argument = number | interfacename | diskname
datadir-stmt = "datadir" dirname
write-stmts = write-stmt [write-stmts]
write-stmt = "write" resource "in" filename
Note that
port in the mux-stmt specifies the port-number for the udp port
(incoming symon(8) traffic).
version
is needed to distinguish between the same type of information
(i.e. io ) coming from different versions of OpenBSD. If no
version number is supplied, the latest will be assumed.
datadir
will guess filenames for all accepted streams. write statements
always take precendence over a datadir statement.
EXAMPLE
Here is an example symux.conf that listens to udp port 2100 on lo0, and
accepts cpu, memory, pf, interfaces xl0/de0/lo0/wi0, disks
wd[0-3]/cd[0-1], disk free blocks of three partition streams from a
symon(8) on localhost.
mux 127.0.0.1 2100
source 127.0.0.1 {
accept { cpu(0), mem, pf,
if(xl0), if(de0),
if(lo0), if(wi0),
io(wd0), io(wd1), io(wd2),
io(wd3), io(cd0), io(cd1),
df(sd0a), df(sd0d), df(sd0e) }
datadir "/var/www/symon/rrds/localhost"
}
LISTENERS
symux offers received symon(8) data to other programs via a fifo. An
example of a listener session:
nexus:~/project/symon$ cat /var/run/symux.fifo
10.0.0.1;mem::1077662160:7630848:53850112:469417984:0:25600;cpu:0:
1077662160:0.00:0.00:0.30:0.20:99.50;io:wd0:1077662160:2074:12759:
0:30736384:131780608;
10.0.0.2;mbuf::1077658247:138:74:0:0:41:0:23:0:90:360:868352:25:0:
0:0;pf::1077658247:700930123:535398451:0:352:1107229:706391:119833
9:4:0:0:2:3:29:4109383:83291:83262:980325:0:1:6:0:0;mem::107765824
7:79155200:131956736:391430144:0:536739840;cpu:0:1077658247:0.50:0
552:0;if:lo0:1077658247:147104:147104:45868177:45868177:0:0:0:0:0:
0;if:xl0:1077658247:284267:452077:150620236:273265863:372:89478:0:
0:0:0;if:de0:1077658247:1813721:1197722:729054136:568900227:101:2:
0:0:198:0;
^C
The format is symon-version : symon-host-ip : stream-name :
stream-argument : timestamp : data
Data formats:
cpu Time spent in ( user, nice, system, interrupt, idle ). Total time
is 100, data is offered with precision 2.
cpuiow Time spent in ( user, nice, system, interrupt, idle, iowait ).
Total time is 100, data is offered with precision 2.
debug Kernel variables debug0 to debug19. ( debug0 : ... : debug19 ).
Values are 32 bit unsigned integers.
df Disk free statistics ( blocks : bfree : bavail : files : ffree :
syncwrites : asyncwrites ). Values are 64 bit unsigned integers.
load Load averages for the last 1, 5, and 15 minutes ( load1, load5,
load15 ). Data is offered with prec ision 2 and a maximum of 655.
if Alias for if2. See below.
if1 Pre OpenBSD 4.3 interface counters ( packets_in, packets_out,
bytes_in, bytes_out, multicasts_in, multicasts_out, errors_in,
errors_out, collisions, drops ). Values are 32 bit unsigned
integers.
if2 Interface counters ( ipackets, opackets, ibytes, obytes, imcasts,
omcasts, ierrors, oerrors, collisions, drops ). Values are 64 bit
unsigned integers.
io Alias for io2. See below.
io1 Pre OpenBSD 3.5 io/disk counters ( total_transfers, total_seeks,
total_bytes ). Values are 64 bit unsigned integers.
io2 Io/disk counters ( rxfer, wxfer, seeks, rbytes, wbytes). Values
are 64 bit unsigned integers.
mbuf Mbuf statistics ( totmbufs : mt_data : mt_oobdata : mt_control :
mt_header : mt_ftable : mt_soname : mt_soopts : pgused : pgtotal
: totmem : totpct : m_drops : m_wait : m_drain ).
mem Alias for mem2. See below.
mem1 Pre symon 2.78 memory counters ( real_active, real_total, free,
swap_used, swap_total ). All values are in bytes rounded to page
boundaries. Values are 32 bit unsigned integers.
mem2 Memory in ( real_active, real_total, free, swap_used, swap_total
). All values are in bytes rounded to page boundaries. Values are
64 bit unsigned integers.
pf Packet filter statistics ( bytes_v4_in : bytes_v4_out :
bytes_v6_in : bytes_v6_out : packets_v4_in_pass :
packets_v4_in_drop : packets_v4_out_pass : packets_v4_out_drop :
packets_v6_in_pass : packets_v6_in_drop : packets_v6_out_pass :
packets_v6_out_drop : states_entries : states_searches :
states_inserts : states_removals : counters_match :
counters_badoffset : counters_fragment : counters_short :
counters_normalize : counters_memory ). Values are 64 bit
unsigned integers.
pfq pf/altq queue statistics ( sent_bytes : sent_packets : drop_bytes
: drop_packets ). Values are 64 bit unsigned integers.
proc Process statistics ( number : uticks : sticks : iticks : cpusec :
cpupct : procsz : rsssz ).
sensor Single sensor measurement offered with 7.6 precision. Value
depends on sensor type.
smart SMART attributes ( read_error_rate: reallocated_sectors:
spin_retries: air_flow_temp: temperature: reallocations:
current_pending: uncorrectables: soft_read_error_rate:
g_sense_error_rate: temperature2: free_fall_protection ). Values
depend on drive model and may change between models.
flukso Average pwr sensor value offered with 7.6 precision. Value is a
moving average and will depend on the number of measurements seen
in a particular symon interval.
SIGNALS
SIGHUP Causes symux to read /etc/symux.conf or the file specified by the
-f flag. symux will keep the old configuration if errors occured
during parsing of the configuration file.
FILES
/var/run/symux.pid
Contains the program id of the symux daemon.
/var/run/symux.fifo
The named pipe / fifo that transmits an ascii version of the
incoming measurements.
/etc/symux.conf
symux system wide configuration file.
LEGACY FORMATS
symux supports symon(8) clients that send
pre OpenBSD 3.5 disk statistics.
These streams should be identified as io1(<disk>) instead of
io(<disk>) in /etc/symux.conf. Also note that symon(8) measures
io1 or io2 depending on whether it was compiled on a host that
supports version 1 or 2. The rrd datastructures of these streams
differ and there is no easy way to change an io1 rrd into an io2
rrd.
pre symon 2.78 mem/if statistics.
These streams should be identified as if1(<interface>) and mem1()
in /etc/symux.conf. symon versions 2.78 and up will always report
if2 and mem2 statistics. The rrd files for the old and new probes
are identical and need not be changed.
symux will output what version of information it is offered by symon(8)s
on the network when started with the -d flag.
BUGS
symux writes incoming data to rrd files "in process". An rrdupdate on a
somewhat stale rrdfile -- with the last data from quite some time in the
past -- is a very expensive operation. This can cause symux to lockup
while rrdupdate is updating the rrd file. symux will be unresponsive
during this process.
AUTHOR
Willem Dijkstra <wpd@xs4all.nl>. Daniel Hartmeier helped to port to big-
endian architectures. Matthew Gream helped to port symon to other BSD
platforms.
Port contributors: Marc Balmer, Tito Dal Canton, Matthew Gream, Daniel
Hartmeier, Lars Kotthoff, Constantine A. Murenin, J. Martin Petersen,
Fredrik Soderblom, Harm Schotanus and Martin van der Werff.
SEE ALSO
symon(8)
Linux 6.13.5-arch1-1 April 4, 2012 Linux 6.13.5-arch1-1
Installation instructions for symon, also accessable as the file README.md inside the symon-version.tar.gz:
() ()
symon
symon is a lightweight system monitor that measures cpu, load,
filesystem, interface, disk, memory, pf, pf queues, mbuf, proc and
sensor statistics every 5 seconds. This information is then spooled to
symux(8) for further processing.
symon has been designed to inflict minimal performance and security
impact on the system it monitors. symux(8) has performance impact
proportional to the amount of streams it needs to manage. Ideally
symux should live on a different system and collect data from several
symon instances in a LAN.
By default, symon will drop privileges and chroot(2) to home of the
symon user. This behaviour is not strictly needed for the cpu, mem,
mbuf, disk debug and interface probes as these will work even when
symon is started as nobody.
symon has been ported to FreeBSD, NetBSD, OpenBSD and Linux.
Rationale
There are many system monitoring tools; what is unique to symon: -
lightweight, aim is to get the data at minimum cpu cost - measure every
5 seconds, this high measurement speed makes e.g. network burst
throttling visible - can be used to feed real-time dashboards that show
health of all hosts in a deployment
Suite
Currently the suite consists of these parts: - symon - lightweight
system monitor. Can be run with privleges equivalent to nobody on the
monitored host. Offers no functionality but monitoring and forwarding
of measured data. - symux - persists data. Incoming symon streams are
stored on disk in rrd files. symux offers systems statistics as they
come in to 3rd party clients. - SymuxClient.pm - generic perl symux
client. Could, for instance, be used to get the hourly amount of data
that was transmitted on a particular interface. - syweb - draws
rrdtool pictures of the stored data. syweb is a php script that can
deal with chrooted apaches. It can show all systems that are monitored
in one go, or be configured to only show a set of graphs. (separate
package) - sylcd - symux client that drives CrystalFontz and HD44780
lcds. sylcd shows current network load on a specific host. (separate
package)
Building & Installing
Privileges
symux needs read and write access to its rrdfiles.
symon needs to interface with your kernel. Depending on your host
system this leads to different privilege requirements:
OpenBSD: - no privs: cpu, debug, df, if, io, mbuf, mem, proc, sensor -
rw on /dev/pf for pf
NetBSD: - no privs: cpu, debug, df, if, io, mbuf, proc - r on
/dev/sysmon for sensor
FreeBSD: - no privs: all - non-chroot on FreeBSD 5.x for CPU ticks in
proc - rw on /dev/pf for pf and pfq
Linux: - r on /proc/net/dev: if - r on /proc/stat: cpu, cpuiow - r on
/proc/meminfo: mem
all: - r on chroot/etc/localtime for proper timezone logging
Real quick on OpenBSD
Install binary packages using:
pkg_add symon # on all monitored hosts
pkg_add symux # on the loghost
pkg_add syweb # also on the loghost
Or build from source:
/usr/ports/net/rrdtool && make install) &&
make &&
make install &&
vi /etc/symux.conf /etc/symon.conf &&
~symon/symux/c_smrrds.sh all &&
/usr/local/libexec/symux &&
useradd -d /var/empty -L daemon -c 'symon Account' -s /sbin/nologin _symon
/usr/local/libexec/symon
Note that the syweb package to show the data stored in the rrd files is
a separate package.
Less quick, but all OSes
• Install rrdtool on the host that will also run your symux gatherer.
BSDs: cd /usr/ports/net/rrdtool && make install
• Check Makefile.inc for settings. Things to watch out for are:
• PREFIX = Where does the installation tree start. Defaults to
/usr/local.
• BINDIR = Where should the daemons be put, relative to $PREFIX.
Defaults to libexec.
• MANDIR = Where should the manuals be installed, relative to
$PREFIX. Defaults to man.
• SHRDIR = Where are the example configurations to be installed.
Defaults to share/symon.
• RRDDIR = $RRDDIR/include should yield rrd.h. Define SYMON_ONLY in
the environment or on the make command line to render this mute.
• INSTALLUSER / GROUPDIR / GROUPFILE = user and groups that install
should use.
• you can define SYMON_ONLY if you do not want to compile symux / do
not have rrdtool installed.
• symon/platform/os/Makefile.inc is read before Makefile.inc; define
your vars in the environment, or in Makefile.inc with != to force
overwriting the defaults.
BSDs: Run make && make install
Linux: Run pmake && pmake install || bmake && bmake install
• Create an /etc/symon.conf for each monitored host and one symux.conf
for the gatherer host. See the manual pages on how to specify
alternative configuration file locations, if necessary. Note that
there are example configurations for both in $PREFIX/$SHRDIR.
• Create the rrd files where the incoming symon data is to be stored.
$PREFIX/$SHRDIR/c_smrrds.sh and symux -l are your friends. Note that
syweb expects an .../machine/*.rrd style directory structure
somewhere under /var/www.
• Ensure that /etc/localtime is accessible by symon/symux. Failing
this you will get log messages in GMT. Note that etc is chroot/etc
when symon chroots.
• Both symon and symux will daemonize if started normally. Start them
with debugging on initially to iron out any configuration problems:
$PREFIX/$BINDIR/symux -d &
$PREFIX/$BINDIR/symon -d
• Remove -d flags and check system logs for any failures.
• Only if you need the web interface: download and install syweb.
Getting measurements without the web
The client directory contains a perl module SymuxClient.pm that can be
used to read measurements as they come in at the symux host. A sample
Perl program called getsymonitem.pl shows how to use the module.
Example:
nexus$ cat /var/run/symux.fifo | getsymonitem.pl 127.0.0.1 2100 127.0.0.1 "cpu(0)" "idle"
93.40
Historical data can be gathered using rrdfetch(1) from symux’s rrd
files.
Portability
This package was originally built as an OpenBSD application. It now
has support for FreeBSD, NetBSD and Linux.
My mental rules for extensions
• Cpu cost matters more than storage.
• Structures are not overloaded in functionality even if that would be
possible memberwise. Overloading = code can be understood in more
than one way and that leads to more maintenance and bugs.
• Defined values are not overloaded in functionality if possible. Same
reason as above.
• Error messages carry __FILE__, __LINE__ if they are failed assertions
and plain English if they can be attributed to the end-user.
• Marshalling is hard. Alignment is carried out by bcopy.
• Support improvements of kernel structures by adding a new stream
type. Goal is to support both the old and the new, so that a
deployment can slowly move towards newer installations without losing
metrics.
• add a new stream id with a postfix (i.e. IO2)
• make old stream accessable via lex with previous postfix (i.e. io1)
()
Installation instructions for syweb, also accessable as the file INSTALL inside the syweb-version.tar.gz:
Installation ------------ Requirements to show a graph: - PHP: Language version needed is 7.4+. - Program files: Download syweb-version.tar.gz from the website and copy htdocs/ and symon/ from the archive into ~www. - Data files: Make your symon rrds accessable to php as run by your http server. Move them into ~www/symon/rrds and update symux. - Configuration: Customise ~www/htdocs/syweb/setup.inc. The supplied setup.inc is intended for chrooted httpd using php-fpm on OpenBSD. - RRDtool: Must be available in whatever web container you are using. E.g. for OpenBSD chrooted httpd, rrdtool must be available inside the chroot. - Date & time: chrooted httpd may have a different idea of timezone. Add etc/localtime to chroot if needed. Point your browser to http://127.0.0.1/syweb/configtest.php to test the configuration. This page will show if the application cannot deal with a setting in setup.inc. Point your browser to http://127.0.0.1/syweb/index.php for the main page. You can move index_noui.php to index.php if you do not need the ui. RRDTool and chrooted httpd -------------------------- RRDTool has a large number of requirements that it must be able to access in order to run. This can require copying a fair number of files inside your chrooted httpd. - Dynamically linked libraries: find out using ldd rrdtool - Dynamically linking kit: ld.so must be available inside chroot to do the dynamic linking, and must be fed a chroot adjusted ~www/var/run/ld.so.hints - Fonts data: a subset of fonts from /usr/X11R6/lib/X11/fonts/TTF/DevaVuSansMono* must be available in a ~www/fontdirectory. - Font configuration: "fontdirectory" must be specified in chroot ~www/etc/fonts/fonts.conf. And finally: - sh binary: this is needed by PHP to be able to execute rrdtool. All these steps are detailed for OpenBSD in install_rrdtool.sh.