pages tagged rsyslogFeeding the Cloudhttps://feeding.cloud.geek.nz/tags/rsyslog/Feeding the Cloudikiwiki2023-10-28T23:54:26ZRemote logging of Turris Omnia log messages using syslog-ng and rsysloghttps://feeding.cloud.geek.nz/posts/remote-logging-turris-omnia-router/
<a href="https://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution-ShareAlike 4.0 International License</a>
2023-10-28T23:54:26Z2022-07-02T03:45:00Z
<p>As part of debugging an upstream connection problem I've been seeing
recently, I wanted to be able to monitor the logs from my <a href="https://www.turris.com/en/omnia/overview/">Turris
Omnia</a> router. Here's how I
configured it to send its logs to a server I already had on the local
network.</p>
<h2 id="Server_setup">Server setup</h2>
<p>The first thing I did was to open up my server's
<a href="https://www.rsyslog.com/">rsyslog</a> (Debian's default syslog server) to
remote connections since it's going to be the destination host for the
router's log messages.</p>
<p>I added the following to <code>/etc/rsyslog.d/router.conf</code>:</p>
<pre><code>module(load="imtcp")
input(type="imtcp" port="514")
if $fromhost-ip == '192.168.1.1' then {
if $syslogseverity <= 5 then {
action(type="omfile" file="/var/log/router.log")
}
stop
}
</code></pre>
<p>This is using the latest rsyslog configuration method: a handy scripting
language called
<a href="https://www.rsyslog.com/doc/v8-stable/rainerscript/index.html">RainerScript</a>.
<a href="https://wiki.archlinux.org/title/Rsyslog#Severity_levels">Severity level</a> 5
maps to "notice" which consists of <em>unusual non-error conditions</em>, and
<code>192.168.1.1</code> is of course the IP address of the router on the LAN side.
With this, I'm directing all router log messages to a separate file,
filtering out anything less important than severity 5.</p>
<p>In order for rsyslog to pick up this new configuration file, I restarted it:</p>
<pre><code>systemctl restart rsyslog.service
</code></pre>
<p>and checked that it was running correctly (e.g. no syntax errors in the new
config file) using:</p>
<pre><code>systemctl status rsyslog.service
</code></pre>
<p>Since I added a new log file, I also setup log rotation for it by putting
the following in <code>/etc/logrotate.d/router</code>:</p>
<pre><code>/var/log/router.log
{
rotate 4
weekly
missingok
notifempty
compress
delaycompress
sharedscripts
postrotate
/usr/lib/rsyslog/rsyslog-rotate
endscript
}
</code></pre>
<p>In addition, since I use
<a href="https://packages.debian.org/stable/logcheck">logcheck</a> to monitor my server
logs and email me errors, I had to add <code>/var/log/router.log</code> to
<code>/etc/logcheck/logcheck.logfiles</code>.</p>
<p>Finally I opened the rsyslog port to the router in my server's firewall by
adding the following to <code>/etc/network/iptables.up.rules</code>:</p>
<pre><code># Allow logs from the router
-A INPUT -s 192.168.1.1 -p tcp --dport 514 -j ACCEPT
</code></pre>
<p>and ran <code>iptables-apply</code>.</p>
<p>With all of this in place, it was time to get the router to send messages.</p>
<h2 id="Router_setup">Router setup</h2>
<p>As <a href="https://forum.turris.cz/t/remote-log-how-to-configure/992/3">suggested on the Turris
forum</a>, I
ssh'ed into my router and added this in <code>/etc/syslog-ng.d/remote.conf</code>:</p>
<pre><code>destination d_loghost {
network("192.168.1.200" time-zone("America/Vancouver"));
};
source dns {
file("/var/log/resolver");
};
log {
source(src);
source(net);
source(kernel);
source(dns);
destination(d_loghost);
};
</code></pre>
<p>Setting the <a href="https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.33/administration-guide/time-zone">timezone</a> to the same as my server was needed because the router
messages were otherwise sent with UTC timestamps.</p>
<p>To ensure that the destination host always gets the same IP address
(<code>192.168.1.200</code>), I went to the <a href="https://192.168.1.1/cgi-bin/luci/admin/network/dhcp">advanced DHCP configuration
page</a> and added a
<em>static lease</em> for the server's MAC address so that it always gets assigned
<code>192.168.1.200</code>. If that wasn't already the server's IP address, you'll have
to restart it for this to take effect.</p>
<p>Finally, I restarted the syslog-ng daemon on the router to pick up the new
config file:</p>
<pre><code>/etc/init.d/syslog-ng restart
</code></pre>
<h2 id="Testing">Testing</h2>
<p>In order to test this configuration, I opened three terminal windows:</p>
<ol>
<li><code>tail -f /var/log/syslog</code> on the server</li>
<li><code>tail -f /var/log/router.log</code> on the server</li>
<li><code>tail -f /var/log/messages</code> on the router</li>
</ol>
<p>I immediately started to see messages from the router in the third window
and some of these, not all because of my severity-5 filter, were flowing to
the second window as well. Also important is that none of the messages make
it to the first window, otherwise log messages from the router would be mixed
in with the server's own logs. That's the purpose of the <code>stop</code> command in
<code>/etc/rsyslog.d/router.conf</code>.</p>
<p>To force a log messages to be emitted by the router, simply ssh into it and
issue the following command:</p>
<pre><code>logger Test
</code></pre>
<p>It should show up in the second and third windows immediately if you've got
everything setup correctly</p>
<h2 id="Timezone_problems">Timezone problems</h2>
<p>If I do the following on my router:</p>
<p> /etc/init.d/syslog-ng restart
logger TestA</p>
<p>I see the following in <code>/var/log/messages</code>:</p>
<p> Aug 14 20:39:35 hostname syslog-ng[9860]: syslog-ng shutting down; version='3.37.1'
Aug 14 20:39:36 hostname syslog-ng[10024]: syslog-ng starting up; version='3.37.1'
Aug 15 03:39:49 hostname root: TestA</p>
<p>The correct timezone is the one in the first two lines. Other daemon
messages are displayed using an incorrect timezone like <code>logger</code>.</p>
<p>Thanks to a <a href="https://lists.balabit.hu/pipermail/syslog-ng/2022-August/thread.html#26509">very helpful <code>syslog-ng</code> mailing list thread</a>, I found that this is actually an <a href="https://gitlab.nic.cz/turris/os/packages/-/issues/471">upstream OpenWRT bug</a>.</p>
<p>My favourite work-around is to tell syslog-ng to simply ignore the timestamp
provided by the application and to use the time of reception (of the log
message) instead. To do this, simply change the following in
<code>/etc/syslog-ng.conf</code>:</p>
<pre><code>source src {
internal();
unix-dgram("/dev/log");
};
</code></pre>
<p>to:</p>
<pre><code>source src {
internal();
unix-dgram("/dev/log", keep-timestamp(no));
};
</code></pre>
<p>Unfortunately, I wasn't able to fix it in a way that would survive a
<code>syslog-ng</code> package update, but since this is supposedly fixed in Turris 6.0,
it shouldn't be a problem for much longer.</p>
Debugging OpenWRT routers by shipping logs to a remote syslog serverhttps://feeding.cloud.geek.nz/posts/debugging-openwrt-routers-by-shipping/
<a href="https://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution-ShareAlike 4.0 International License</a>
2022-07-02T02:45:10Z2012-01-14T08:45:00Z
<p>Trying to debug problems with consumer-grade routers is notoriously difficult due to a lack of decent debugging information. It's quite hard to know what's going on without at least a few good error messages.</p>
<p>Here is how I made my <a href="https://openwrt.org/">OpenWRT</a>-based <a href="http://gargoyle-router.org/">Gargoyle</a> router send its log messages to a network server running <a href="http://rsyslog.com/">rsyslog</a>.</p>
<h3 id="Server_Configuration">Server Configuration</h3>
<p>Given that the router (<code>192.168.1.1</code>) will be sending its log messages on UDP port 514, I started by opening that port in my firewall:</p>
<pre><code>iptables -A INPUT -s 192.168.1.1 -p udp --dport 514 -j ACCEPT
</code></pre>
<p>Then I enabled the UDP module for rsyslog and redirected messages to a separate log file (so that it doesn't fill up <code>/var/log/syslog</code>) by putting the following (a modified version of <a href="http://rsyslog.com/storing-messages-from-a-remote-system-into-a-specific-file/">these instructions</a>) in <code>/etc/rsyslog.d/10-gargoyle-router.conf</code>:</p>
<pre><code>$ModLoad imudp
$UDPServerRun 514
:fromhost-ip, isequal, "192.168.1.1" /var/log/gargoyle-router.log
& ~
</code></pre>
<p>The name of the file is important because this configuration snipet needs to be loaded before the directive which writes to <code>/var/log/syslog</code> for the discard statement (the "& ~" line) to <a href="http://lists.adiscon.net/pipermail/rsyslog/2012-January/014201.html">work correctly</a>.</p>
<h3 id="Router_Configuration">Router Configuration</h3>
<p>Finally, I followed the <a href="http://www.gargoyle-router.com/wiki/doku.php?id=remote_syslog">instructions</a> on the Gargoyle wiki to get the router to forward its log messages to my server (<code>192.168.1.2</code>).</p>
<p>After logging into the router via ssh, I ran the following commands:</p>
<pre><code>uci set system.@system[0].log_ip=192.168.1.2
uci set system.@system[0].cronloglevel=7
uci commit
</code></pre>
<p>before rebooting the router.</p>
<p>Now whenever I have to troubleshoot network problems, I can keep a terminal open on my server and get some visibility on what the router is doing:</p>
<pre><code>tail -f /var/log/gargoyle-router.log
</code></pre>