From 5c300d2399398a7384728ef11b00642b8cfe5128 Mon Sep 17 00:00:00 2001 From: varac Date: Tue, 2 Dec 2014 20:21:26 +0100 Subject: Ignore bigcouch conflict errors, mainly coming from tapicero creating new users (Feature #6481) There are potentially many tapicero daemons running, and they all try to do the same thing at the same time. It is basically designed to create race conditions. All tapicero daemons try to create the user db at the same time. Only one of them wins the race and actually creates it. We need to fix this later (see https://leap.se/code/issues/6480) but for now, we ignore them because conflict errors should be handled by the applictation anyway. Change-Id: I91095b1901d238e3d199954ba3716023d3fd49c1 --- puppet/modules/site_check_mk/files/agent/logwatch/bigcouch.cfg | 2 ++ 1 file changed, 2 insertions(+) (limited to 'puppet/modules/site_check_mk/files/agent') diff --git a/puppet/modules/site_check_mk/files/agent/logwatch/bigcouch.cfg b/puppet/modules/site_check_mk/files/agent/logwatch/bigcouch.cfg index 28f333b0..d274a676 100644 --- a/puppet/modules/site_check_mk/files/agent/logwatch/bigcouch.cfg +++ b/puppet/modules/site_check_mk/files/agent/logwatch/bigcouch.cfg @@ -6,6 +6,8 @@ I 127.0.0.1 localhost:5984 .* ok # https://leap.se/code/issues/5246 I Shutting down group server + # ignore bigcouch conflict errors, mainly coming from tapicero creating new users + I Error in process.*{{nocatch,conflict} # ignore "Uncaught error in HTTP request: {exit, normal}" error # it's suppressed in later versions of bigcouch anhow # see https://leap.se/code/issues/5226 -- cgit v1.2.3 From dd888ede2897d56e1b7273160b99f990cd4bcf31 Mon Sep 17 00:00:00 2001 From: varac Date: Tue, 9 Dec 2014 10:07:12 +0100 Subject: logwatch: ignore postfix errors on lost connection (Bug #6476) Change-Id: I0b1eec11a3b3da39d65572b6bee8b3ce892e08ac --- puppet/modules/site_check_mk/files/agent/logwatch/syslog_tail.cfg | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) (limited to 'puppet/modules/site_check_mk/files/agent') diff --git a/puppet/modules/site_check_mk/files/agent/logwatch/syslog_tail.cfg b/puppet/modules/site_check_mk/files/agent/logwatch/syslog_tail.cfg index 450b9e90..60881e22 100644 --- a/puppet/modules/site_check_mk/files/agent/logwatch/syslog_tail.cfg +++ b/puppet/modules/site_check_mk/files/agent/logwatch/syslog_tail.cfg @@ -1,7 +1,9 @@ # some general patterns + I Error: Driver 'pcspkr' is already registered, aborting... +# ignore postfix errors on lost connection (Bug #6476) + I postfix/smtpd.*SSL_accept error from.*lost connection C panic C Oops - I Error: Driver 'pcspkr' is already registered, aborting... C Error C error W generic protection rip -- cgit v1.2.3 From a8334795fd393fac6e5055085fd7ab94b3b3295a Mon Sep 17 00:00:00 2001 From: Micah Anderson Date: Tue, 9 Dec 2014 21:03:02 -0500 Subject: Ignore rexi_EXIT bigcouch error (Bug #6512) Change-Id: I03842b65329aabb012cc2c7514007e174cbd8fc0 --- puppet/modules/site_check_mk/files/agent/logwatch/bigcouch.cfg | 2 ++ 1 file changed, 2 insertions(+) (limited to 'puppet/modules/site_check_mk/files/agent') diff --git a/puppet/modules/site_check_mk/files/agent/logwatch/bigcouch.cfg b/puppet/modules/site_check_mk/files/agent/logwatch/bigcouch.cfg index d274a676..5cd2a47b 100644 --- a/puppet/modules/site_check_mk/files/agent/logwatch/bigcouch.cfg +++ b/puppet/modules/site_check_mk/files/agent/logwatch/bigcouch.cfg @@ -13,6 +13,8 @@ # see https://leap.se/code/issues/5226 I Uncaught error in HTTP request: {exit,normal} I Uncaught error in HTTP request: {exit, + # Ignore rexi_EXIT bigcouch error (Bug #6512) + I Error in process <[0-9.]+> on node .* with exit value: {{rexi_EXIT,{killed,\[{couch_db,collect_results C Uncaught error in HTTP request: {error, C Response abnormally terminated: {nodedown, C rexi_DOWN,noproc -- cgit v1.2.3 From a70488de29e8721191de1f6a504dcc35e5d74770 Mon Sep 17 00:00:00 2001 From: varac Date: Wed, 10 Dec 2014 11:55:24 +0100 Subject: ignore transient Tapicero errors when creating a db (Bug #6511) Change-Id: I0939070482fad4f99f03e41094a3df42ff5063e4 --- puppet/modules/site_check_mk/files/agent/logwatch/syslog/tapicero.cfg | 2 ++ 1 file changed, 2 insertions(+) (limited to 'puppet/modules/site_check_mk/files/agent') diff --git a/puppet/modules/site_check_mk/files/agent/logwatch/syslog/tapicero.cfg b/puppet/modules/site_check_mk/files/agent/logwatch/syslog/tapicero.cfg index 93ce0311..3aec4ad0 100644 --- a/puppet/modules/site_check_mk/files/agent/logwatch/syslog/tapicero.cfg +++ b/puppet/modules/site_check_mk/files/agent/logwatch/syslog/tapicero.cfg @@ -3,6 +3,8 @@ # instances, so we ignore it # see https://leap.se/code/issues/5168 I tapicero.*RestClient::PreconditionFailed: +# Ignore transient Tapicero errors when creating a db (#6511) + I tapicero.*Creating database user-[[:alnum:]]* failed \(trying again soon\): RestClient::InternalServerError: 500 Internal Server Error C tapicero.*Creating database.*failed due to: C tapicero.*failed W tapicero.*Couch stream ended unexpectedly. -- cgit v1.2.3 From 9076e8ccf62bd4cf25cd22af98d3536aff361b51 Mon Sep 17 00:00:00 2001 From: Micah Anderson Date: Wed, 10 Dec 2014 10:48:10 -0500 Subject: update ffa53ef321bbfd771afff1ccb230d1b5e4f9ab00 to fix ordering requirement in logwatch, remove extended regexp character class and also ignore "Writing security" lines Change-Id: I7d33725db06a40361a3b04f9591adeb6a025bf77 --- puppet/modules/site_check_mk/files/agent/logwatch/syslog/tapicero.cfg | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) (limited to 'puppet/modules/site_check_mk/files/agent') diff --git a/puppet/modules/site_check_mk/files/agent/logwatch/syslog/tapicero.cfg b/puppet/modules/site_check_mk/files/agent/logwatch/syslog/tapicero.cfg index 3aec4ad0..c998322c 100644 --- a/puppet/modules/site_check_mk/files/agent/logwatch/syslog/tapicero.cfg +++ b/puppet/modules/site_check_mk/files/agent/logwatch/syslog/tapicero.cfg @@ -1,10 +1,10 @@ +# Ignore transient Tapicero errors when creating a db (#6511) + I tapicero.*(Creating database|Writing security to|Uploading design doc to) user-.* failed (\(trying again soon\)|twice due to): RestClient::InternalServerError: 500 Internal Server Error C tapicero.*RestClient::InternalServerError: # possible race condition between multiple tapicero # instances, so we ignore it # see https://leap.se/code/issues/5168 I tapicero.*RestClient::PreconditionFailed: -# Ignore transient Tapicero errors when creating a db (#6511) - I tapicero.*Creating database user-[[:alnum:]]* failed \(trying again soon\): RestClient::InternalServerError: 500 Internal Server Error C tapicero.*Creating database.*failed due to: C tapicero.*failed W tapicero.*Couch stream ended unexpectedly. -- cgit v1.2.3 From 8c0fa3991f03c31fa2b4d61f20c824b86e1efa1c Mon Sep 17 00:00:00 2001 From: Micah Anderson Date: Thu, 11 Dec 2014 15:55:00 -0500 Subject: logwatch: ignore ipv6 icmp errors (Bug #6540) Change-Id: I198c5245c7e73d6dd7a7d9725fac1eb9a8f425a5 --- puppet/modules/site_check_mk/files/agent/logwatch/syslog_tail.cfg | 2 ++ 1 file changed, 2 insertions(+) (limited to 'puppet/modules/site_check_mk/files/agent') diff --git a/puppet/modules/site_check_mk/files/agent/logwatch/syslog_tail.cfg b/puppet/modules/site_check_mk/files/agent/logwatch/syslog_tail.cfg index 60881e22..b19ac241 100644 --- a/puppet/modules/site_check_mk/files/agent/logwatch/syslog_tail.cfg +++ b/puppet/modules/site_check_mk/files/agent/logwatch/syslog_tail.cfg @@ -11,3 +11,5 @@ # 401 Unauthorized error logged by webapp and possible other # applications C Unauthorized +# ignore ipv6 icmp errors for now (Bug #6540) + I kernel: .*icmpv6_send: no reply to icmp error -- cgit v1.2.3 From a64b2081b872823f2d539b48151bc0d323b42453 Mon Sep 17 00:00:00 2001 From: Micah Anderson Date: Thu, 11 Dec 2014 21:22:49 -0500 Subject: Ignore additional tapicero message (#6542): tapicero[921]: Checking security of user-1b3b1fb78db851190fa72dac01207b8d failed (trying again soon): RestClient::ResourceNotFound: 404 Resource Not Found: {"error":"not_found","reason":"Database does not exist."}") tapicero recovers from this error Change-Id: Ic105823ddc282512000e6d7445539428581eb997 --- puppet/modules/site_check_mk/files/agent/logwatch/syslog/tapicero.cfg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'puppet/modules/site_check_mk/files/agent') diff --git a/puppet/modules/site_check_mk/files/agent/logwatch/syslog/tapicero.cfg b/puppet/modules/site_check_mk/files/agent/logwatch/syslog/tapicero.cfg index c998322c..d9f0eafc 100644 --- a/puppet/modules/site_check_mk/files/agent/logwatch/syslog/tapicero.cfg +++ b/puppet/modules/site_check_mk/files/agent/logwatch/syslog/tapicero.cfg @@ -1,5 +1,5 @@ # Ignore transient Tapicero errors when creating a db (#6511) - I tapicero.*(Creating database|Writing security to|Uploading design doc to) user-.* failed (\(trying again soon\)|twice due to): RestClient::InternalServerError: 500 Internal Server Error + I tapicero.*(Creating database|Checking security of|Writing security to|Uploading design doc to) user-.* failed (\(trying again soon\)|twice due to): (RestClient::Resource Not Found|RestClient::InternalServerError): (404 Resource Not Found|500 Internal Server Error) C tapicero.*RestClient::InternalServerError: # possible race condition between multiple tapicero # instances, so we ignore it -- cgit v1.2.3 From 6df0eb2921db5ea32662036a9fe297cdfb5042ee Mon Sep 17 00:00:00 2001 From: Micah Anderson Date: Thu, 11 Dec 2014 19:01:16 -0500 Subject: Move kernel ipv6 log message up before the 'C error' line to it is caught (#6540) Change-Id: I1fe8d4cf60532dfe01cfb3a014c4cbeb4acdc479 --- puppet/modules/site_check_mk/files/agent/logwatch/syslog_tail.cfg | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) (limited to 'puppet/modules/site_check_mk/files/agent') diff --git a/puppet/modules/site_check_mk/files/agent/logwatch/syslog_tail.cfg b/puppet/modules/site_check_mk/files/agent/logwatch/syslog_tail.cfg index b19ac241..b8f47434 100644 --- a/puppet/modules/site_check_mk/files/agent/logwatch/syslog_tail.cfg +++ b/puppet/modules/site_check_mk/files/agent/logwatch/syslog_tail.cfg @@ -5,11 +5,11 @@ C panic C Oops C Error +# ignore ipv6 icmp errors for now (Bug #6540) + I kernel: .*icmpv6_send: no reply to icmp error C error W generic protection rip W .*Unrecovered read error - auto reallocate failed # 401 Unauthorized error logged by webapp and possible other # applications C Unauthorized -# ignore ipv6 icmp errors for now (Bug #6540) - I kernel: .*icmpv6_send: no reply to icmp error -- cgit v1.2.3 From 970af8cf4a7670c01cd06b45a09f010d91949a2f Mon Sep 17 00:00:00 2001 From: Micah Anderson Date: Tue, 16 Dec 2014 10:56:41 -0500 Subject: ignore additional bigcouch error messages (#6512) Change-Id: Ie51fb485bcae9a9467c465bdd1b4a5785023db04 --- puppet/modules/site_check_mk/files/agent/logwatch/bigcouch.cfg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'puppet/modules/site_check_mk/files/agent') diff --git a/puppet/modules/site_check_mk/files/agent/logwatch/bigcouch.cfg b/puppet/modules/site_check_mk/files/agent/logwatch/bigcouch.cfg index 5cd2a47b..3d5ada42 100644 --- a/puppet/modules/site_check_mk/files/agent/logwatch/bigcouch.cfg +++ b/puppet/modules/site_check_mk/files/agent/logwatch/bigcouch.cfg @@ -14,7 +14,7 @@ I Uncaught error in HTTP request: {exit,normal} I Uncaught error in HTTP request: {exit, # Ignore rexi_EXIT bigcouch error (Bug #6512) - I Error in process <[0-9.]+> on node .* with exit value: {{rexi_EXIT,{killed,\[{couch_db,collect_results + I Error in process <[0-9.]+> on node .* with exit value: {{rexi_EXIT,{(killed|noproc|shutdown),\[{couch_db,collect_results C Uncaught error in HTTP request: {error, C Response abnormally terminated: {nodedown, C rexi_DOWN,noproc -- cgit v1.2.3 From 8c89257f8b600d30e47f9321ef957af719b4ea21 Mon Sep 17 00:00:00 2001 From: Micah Anderson Date: Tue, 16 Dec 2014 11:06:28 -0500 Subject: Ignore postfix "too many errors after DATA" logwatch msg (Bug #6545) Change-Id: I0abeb88f7b6548e5742bd3d99b2f4e5d9c6cf421 --- puppet/modules/site_check_mk/files/agent/logwatch/syslog_tail.cfg | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) (limited to 'puppet/modules/site_check_mk/files/agent') diff --git a/puppet/modules/site_check_mk/files/agent/logwatch/syslog_tail.cfg b/puppet/modules/site_check_mk/files/agent/logwatch/syslog_tail.cfg index b19ac241..71395c50 100644 --- a/puppet/modules/site_check_mk/files/agent/logwatch/syslog_tail.cfg +++ b/puppet/modules/site_check_mk/files/agent/logwatch/syslog_tail.cfg @@ -2,14 +2,16 @@ I Error: Driver 'pcspkr' is already registered, aborting... # ignore postfix errors on lost connection (Bug #6476) I postfix/smtpd.*SSL_accept error from.*lost connection +# ignore postfix too many errors after DATA (#6545) + I postfix/smtpd.*too many errors after DATA from C panic C Oops C Error +# ignore ipv6 icmp errors for now (Bug #6540) + I kernel: .*icmpv6_send: no reply to icmp error C error W generic protection rip W .*Unrecovered read error - auto reallocate failed # 401 Unauthorized error logged by webapp and possible other # applications C Unauthorized -# ignore ipv6 icmp errors for now (Bug #6540) - I kernel: .*icmpv6_send: no reply to icmp error -- cgit v1.2.3 From 8d6c05adf08eb2ddc29c4481ea06fcf85dda7b26 Mon Sep 17 00:00:00 2001 From: Micah Anderson Date: Tue, 16 Dec 2014 20:00:01 -0500 Subject: Ignore "Generic server terminating" bigcouch message (Feature #6544) Change-Id: I73defd7964501e4eabe7dd05c02887e7aeb2f063 --- puppet/modules/site_check_mk/files/agent/logwatch/bigcouch.cfg | 4 ++++ 1 file changed, 4 insertions(+) (limited to 'puppet/modules/site_check_mk/files/agent') diff --git a/puppet/modules/site_check_mk/files/agent/logwatch/bigcouch.cfg b/puppet/modules/site_check_mk/files/agent/logwatch/bigcouch.cfg index 5cd2a47b..434d5503 100644 --- a/puppet/modules/site_check_mk/files/agent/logwatch/bigcouch.cfg +++ b/puppet/modules/site_check_mk/files/agent/logwatch/bigcouch.cfg @@ -15,6 +15,10 @@ I Uncaught error in HTTP request: {exit, # Ignore rexi_EXIT bigcouch error (Bug #6512) I Error in process <[0-9.]+> on node .* with exit value: {{rexi_EXIT,{killed,\[{couch_db,collect_results + # Ignore "Generic server terminating" bigcouch message (Feature #6544) + I Generic server <.*> terminating + I {error_report,<.*>, + I {error_info, C Uncaught error in HTTP request: {error, C Response abnormally terminated: {nodedown, C rexi_DOWN,noproc -- cgit v1.2.3 From 547354d07972130fc41cc79a80085731b696f887 Mon Sep 17 00:00:00 2001 From: Micah Anderson Date: Wed, 17 Dec 2014 15:21:09 -0500 Subject: Ignore Soledad "Timing out client" warning (Bug #6566) Change-Id: I6d3fa5028ba6eaca7b21a7e850136ef980f6e782 --- puppet/modules/site_check_mk/files/agent/logwatch/soledad.cfg | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) (limited to 'puppet/modules/site_check_mk/files/agent') diff --git a/puppet/modules/site_check_mk/files/agent/logwatch/soledad.cfg b/puppet/modules/site_check_mk/files/agent/logwatch/soledad.cfg index 623d1e46..3af5045b 100644 --- a/puppet/modules/site_check_mk/files/agent/logwatch/soledad.cfg +++ b/puppet/modules/site_check_mk/files/agent/logwatch/soledad.cfg @@ -2,4 +2,5 @@ C WSGI application error C Error C error - W Timing out client: +# Removed this line because we determined it was better to ignore it (#6566) +# W Timing out client: -- cgit v1.2.3 From 429d49824988873870911bf3425d4ca1118e8605 Mon Sep 17 00:00:00 2001 From: Micah Anderson Date: Wed, 17 Dec 2014 16:04:37 -0500 Subject: Update to logwatch ignore for tapicero Change-Id: I1d8cedfeb1153312c13f7f182c7ac3b031647dd4 --- puppet/modules/site_check_mk/files/agent/logwatch/syslog/tapicero.cfg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'puppet/modules/site_check_mk/files/agent') diff --git a/puppet/modules/site_check_mk/files/agent/logwatch/syslog/tapicero.cfg b/puppet/modules/site_check_mk/files/agent/logwatch/syslog/tapicero.cfg index d9f0eafc..d00b8a75 100644 --- a/puppet/modules/site_check_mk/files/agent/logwatch/syslog/tapicero.cfg +++ b/puppet/modules/site_check_mk/files/agent/logwatch/syslog/tapicero.cfg @@ -1,5 +1,5 @@ # Ignore transient Tapicero errors when creating a db (#6511) - I tapicero.*(Creating database|Checking security of|Writing security to|Uploading design doc to) user-.* failed (\(trying again soon\)|twice due to): (RestClient::Resource Not Found|RestClient::InternalServerError): (404 Resource Not Found|500 Internal Server Error) + I tapicero.*(Creating database|Checking security of|Writing security to|Uploading design doc to) user-.* failed (\(trying again soon\)|(twice )? due to): (RestClient::Resource Not Found|RestClient::InternalServerError): (404 Resource Not Found|500 Internal Server Error) C tapicero.*RestClient::InternalServerError: # possible race condition between multiple tapicero # instances, so we ignore it -- cgit v1.2.3 From 82abc7fa7b9dd7275365d6a185b51803a34368a3 Mon Sep 17 00:00:00 2001 From: varac Date: Wed, 17 Dec 2014 22:17:08 +0100 Subject: Check_mk logwatch: ignore openvpn warnings (Feature #6568) Change-Id: I0d30afbcc6dcb90c6716f7c6bb0bca3e6ae0964a --- .../modules/site_check_mk/files/agent/logwatch/syslog/openvpn.cfg | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) (limited to 'puppet/modules/site_check_mk/files/agent') diff --git a/puppet/modules/site_check_mk/files/agent/logwatch/syslog/openvpn.cfg b/puppet/modules/site_check_mk/files/agent/logwatch/syslog/openvpn.cfg index d58e876d..ac17c0ca 100644 --- a/puppet/modules/site_check_mk/files/agent/logwatch/syslog/openvpn.cfg +++ b/puppet/modules/site_check_mk/files/agent/logwatch/syslog/openvpn.cfg @@ -2,6 +2,12 @@ # suddenly hangup before properly establishing # a tls connection I ovpn-.*TLS Error: Unroutable control packet received from - I ovpn-.*TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity) + I ovpn-.*TLS Error: TLS key negotiation failed to occur within 60 seconds \(check your network connectivity\) I ovpn-.*TLS Error: TLS handshake failed + I ovpn-.*TLS Error: TLS object -> incoming plaintext read error + I ovpn-.*Fatal TLS error \(check_tls_errors_co\), restarting + I ovpn-.*TLS_ERROR: BIO read tls_read_plaintext error: error:140890B2:SSL routines:SSL3_GET_CLIENT_CERTIFICATE:no certificate + + I ovpn-.*SIGUSR1\[soft,tls-error\] received, client-instance restarting + I ovpn-.*VERIFY ERROR: depth=0, error=certificate has expired -- cgit v1.2.3 From d024a172ee4b2dc7fca4fe4251c930ff437f1f1b Mon Sep 17 00:00:00 2001 From: Micah Anderson Date: Thu, 18 Dec 2014 14:07:04 -0500 Subject: update tapicero logwatch messages to remove extra space Change-Id: I0149ac2e767531d9724b57b9e3bdae7943f954ff --- puppet/modules/site_check_mk/files/agent/logwatch/syslog/tapicero.cfg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'puppet/modules/site_check_mk/files/agent') diff --git a/puppet/modules/site_check_mk/files/agent/logwatch/syslog/tapicero.cfg b/puppet/modules/site_check_mk/files/agent/logwatch/syslog/tapicero.cfg index d00b8a75..e5721eea 100644 --- a/puppet/modules/site_check_mk/files/agent/logwatch/syslog/tapicero.cfg +++ b/puppet/modules/site_check_mk/files/agent/logwatch/syslog/tapicero.cfg @@ -1,5 +1,5 @@ # Ignore transient Tapicero errors when creating a db (#6511) - I tapicero.*(Creating database|Checking security of|Writing security to|Uploading design doc to) user-.* failed (\(trying again soon\)|(twice )? due to): (RestClient::Resource Not Found|RestClient::InternalServerError): (404 Resource Not Found|500 Internal Server Error) + I tapicero.*(Creating database|Checking security of|Writing security to|Uploading design doc to) user-.* failed (\(trying again soon\)|(twice )?due to): (RestClient::Resource Not Found|RestClient::InternalServerError): (404 Resource Not Found|500 Internal Server Error) C tapicero.*RestClient::InternalServerError: # possible race condition between multiple tapicero # instances, so we ignore it -- cgit v1.2.3