' [15:42:02] PROBLEM - phabdigest-bots-weekly service on tools2002 is CRITICAL: CRITICAL: Status of the systemd unit phabdigest-bots-weekly [15:42:26] RECOVERY - Phabricator on tools2002 is OK: HTTP OK: HTTP/1.1 200 OK - 21794 bytes in 0.636 second response time [15:42:35] yes i know i broke stuff [15:42:36] RECOVERY - puppet service on tools2002 is OK: OK: Status of the systemd unit run-puppet [15:42:56] https://www.irccloud.com/pastebin/JHJFdRkC/ [15:43:06] MacFan4000: does ^ make sense [15:47:16] PROBLEM - puppet service on bots2002 is UNKNOWN: Remote Icinga instance 'bots2002.fossbots.org' is not connected to 'tools2002.fossbots.org' [15:47:36] PROBLEM - FOSSBots SSL on bots2002 is UNKNOWN: Remote Icinga instance 'bots2002.fossbots.org' is not connected to 'tools2002.fossbots.org' [15:47:46] PROBLEM - streambot service on bots2002 is UNKNOWN: Remote Icinga instance 'bots2002.fossbots.org' is not connected to 'tools2002.fossbots.org' [15:47:48] PROBLEM - disk / on bots2002 is UNKNOWN: Remote Icinga instance 'bots2002.fossbots.org' is not connected to 'tools2002.fossbots.org' [15:47:54] PROBLEM - ssh #page on db2002 is UNKNOWN: Remote Icinga instance 'db2002.fossbots.org' is not connected to 'tools2002.fossbots.org' [15:47:56] fossbots.org' [15:48:00] PROBLEM - ping6 #page on bots2002 is UNKNOWN: Remote Icinga instance 'bots2002.fossbots.org' is not connected to 'tools2002.fossbots.org' [15:48:04] PROBLEM - disk on db2002 is UNKNOWN: Remote Icinga instance 'db2002.fossbots.org' is not connected to 'tools2002.fossbots.org' [15:48:08] PROBLEM - load on db2002 is UNKNOWN: Remote Icinga instance 'db2002.fossbots.org' is not connected to 'tools2002.fossbots.org' [15:48:14] PROBLEM - ufw service on bots2002 is UNKNOWN: Remote Icinga instance 'bots2002.fossbots.org' is not connected to 'tools2002.fossbots.org' [15:48:17] PROBLEM - load on bots2002 is UNKNOWN: Remote Icinga instance 'bots2002.fossbots.org' is not connected to 'tools2002.fossbots.org' [15:48:21] PROBLEM - disk / on db2002 is UNKNOWN: Remote Icinga instance 'db2002.fossbots.org' is not connected to 'tools2002.fossbots.org' [15:48:22] PROBLEM - puppet service on db2002 is UNKNOWN: Remote Icinga instance 'db2002.fossbots.org' is not connected to 'tools2002.fossbots.org' [15:48:27] PROBLEM - Apache on bots2002 is UNKNOWN: Remote Icinga instance 'bots2002.fossbots.org' is not connected to 'tools2002.fossbots.org' [15:48:29] PROBLEM - http on bots2002 is UNKNOWN: Remote Icinga instance 'bots2002.fossbots.org' is not connected to 'tools2002.fossbots.org' [15:48:31] PROBLEM - disk on bots2002 is UNKNOWN: Remote Icinga instance 'bots2002.fossbots.org' is not connected to 'tools2002.fossbots.org' [15:48:32] PROBLEM - users on bots2002 is UNKNOWN: Remote Icinga instance 'bots2002.fossbots.org' is not connected to 'tools2002.fossbots.org' [15:48:35] PROBLEM - ping6 #page on db2002 is UNKNOWN: Remote Icinga instance 'db2002.fossbots.org' is not connected to 'tools2002.fossbots.org' [15:48:35] .in 8h mass delete icinga spam emails [15:48:35] Sario: Okay, will remind at 09/May/2022 6:48PM [15:48:39] PROBLEM - StreamBot on bots2002 is UNKNOWN: Remote Icinga instance 'bots2002.fossbots.org' is not connected to 'tools2002.fossbots.org' [15:48:41] PROBLEM - users on db2002 is UNKNOWN: Remote Icinga instance 'db2002.fossbots.org' is not connected to 'tools2002.fossbots.org' [15:48:43] PROBLEM - procs on bots2002 is UNKNOWN: Remote Icinga instance 'bots2002.fossbots.org' is not connected to 'tools2002.fossbots.org' [15:50:15] MirahezeBot: i assume the page is yyou [15:50:20] MacFan4000: * [15:50:26] i'm looking/fixing now [15:51:34] [02puppet] 07RhinosF1 closed pull request 03#301: dbbackups: add - 13https://github.com/FOSSBots/puppet/pull/301 [15:51:35] [url] Page not found · GitHub · GitHub | github.com [15:51:36] [02puppet] 07RhinosF1 pushed 031 commit to 03master [+2/-0/±0] 13https://github.com/FOSSBots/puppet/compare/8a3ac2eea5a2...77bbbce07033 [15:51:36] [url] Comparing 8a3ac2eea5a2...77bbbce07033 · FOSSBots/puppet · GitHub | github.com [15:51:37] [02puppet] 07RhinosF1 0377bbbce - dbbackups: add (#301) * dbbackups: add * Create backup-db.sh [15:51:39] [02puppet] 07RhinosF1 deleted branch 03RhinosF1-patch-1 - 13https://github.com/FOSSBots/puppet [15:51:39] [url] Page not found · GitHub · GitHub | github.com [15:51:40] [02puppet] 07RhinosF1 deleted branch 03RhinosF1-patch-1 [15:54:16] [02puppet] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/FOSSBots/puppet/compare/77bbbce07033...e5ef42d53f81 [15:54:17] [url] Comparing 77bbbce07033...e5ef42d53f81 · FOSSBots/puppet · GitHub | github.com [15:54:18] [02puppet] 07MacFan4000 03e5ef42d - Update bots2002.yaml [15:54:42] [02puppet] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/FOSSBots/puppet/compare/e5ef42d53f81...fefb15366bf5 [15:54:43] [url] Comparing e5ef42d53f81...fefb15366bf5 · FOSSBots/puppet · GitHub | github.com [15:54:43] [02puppet] 07MacFan4000 03fefb153 - Update tools2002.yaml [15:55:04] [02puppet] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/FOSSBots/puppet/compare/fefb15366bf5...3e2f9d6d1b60 [15:55:04] [url] Comparing fefb15366bf5...3e2f9d6d1b60 · FOSSBots/puppet · GitHub | github.com [15:55:05] [02puppet] 07MacFan4000 033e2f9d6 - Update db2002.yaml [15:55:31] RECOVERY - disk on bots2002 is OK: DISK OK - free space: / 21096 MB (67% inode=93%); /boot/efi 126 MB (99% inode=-); [15:55:33] RECOVERY - StreamBot on bots2002 is OK: PROCS OK: 1 process with command name 'python3' [15:55:35] RECOVERY - disk / on bots2002 is OK: DISK OK - free space: / 21096 MB (67% inode=93%); /boot/efi 126 MB (99% inode=-); [15:55:55] RECOVERY - sopel prod service on bots2002 is OK: OK: Status of the systemd unit mirahezebotprodlibera [15:56:01] RECOVERY - Apache on bots2002 is OK: PROCS OK: 3 processes with command name 'apache2' [15:56:03] RECOVERY - load on bots2002 is OK: OK - load average: 0.29, 0.31, 0.28 [15:56:05] RECOVERY - apt on bots2002 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [15:56:07] RECOVERY - ufw service on bots2002 is OK: OK: Status of the systemd unit ufw [15:56:09] RECOVERY - FOSSBots SSL on bots2002 is OK: OK - Certificate 'CloudFlare Origin Certificate' will expire on Sun Mar 15 23:05:00 2037 +0000. [15:56:13] RECOVERY - Discord-relay on bots2002 is OK: PROCS OK: 1 process with command name 'matterbridge' [15:56:19] RECOVERY - streambot service on bots2002 is OK: OK: Status of the systemd unit streambot [15:56:21] RECOVERY - http on bots2002 is OK: HTTP OK: HTTP/1.1 200 OK - 10975 bytes in 0.004 second response time [15:56:23] RECOVERY - procs on bots2002 is OK: PROCS OK: 122 processes [15:59:07] [02puppet] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/FOSSBots/puppet/compare/3e2f9d6d1b60...938a6185b80c [15:59:07] [url] Comparing 3e2f9d6d1b60...938a6185b80c · FOSSBots/puppet · GitHub | github.com [15:59:08] [02puppet] 07RhinosF1 03938a618 - Update site.pp [16:00:10] [02puppet] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/FOSSBots/puppet/compare/938a6185b80c...156a0838c6b0 [16:00:11] [url] Comparing 938a6185b80c...156a0838c6b0 · FOSSBots/puppet · GitHub | github.com [16:00:12] [02puppet] 07RhinosF1 03156a083 - Update dbbackups.pp [16:01:23] [02puppet] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/FOSSBots/puppet/compare/156a0838c6b0...ff775802d1a8 [16:01:24] [url] Comparing 156a0838c6b0...ff775802d1a8 · FOSSBots/puppet · GitHub | github.com [16:01:25] [02puppet] 07RhinosF1 03ff77580 - Update dbbackups.pp [16:06:06] [02puppet] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/FOSSBots/puppet/compare/ff775802d1a8...db7c86043c0a [16:06:06] [url] Comparing ff775802d1a8...db7c86043c0a · FOSSBots/puppet · GitHub | github.com [16:06:07] [02puppet] 07RhinosF1 03db7c860 - Update dbbackups.pp [16:06:34] RECOVERY - ssh #page on db2002 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [16:06:36] inode=96%); /boot/efi 126 MB (99% inode=-); [16:06:40] RECOVERY - ping6 #page on db2002 is OK: PING OK - Packet loss = 0%, RTA = 0.28 ms [16:06:42] RECOVERY - ufw service on db2002 is OK: OK: Status of the systemd unit ufw [16:06:56] RECOVERY - Puppet on db2002 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [16:07:04] RECOVERY - mariadb service on db2002 is OK: OK: Status of the systemd unit mariadb [16:07:08] RECOVERY - apt on db2002 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [16:07:14] RECOVERY - disk / on db2002 is OK: DISK OK - free space: / 26386 MB (84% inode=96%); /boot/efi 126 MB (99% inode=-); [16:10:13] RECOVERY - puppet service on db2002 is OK: OK: Status of the systemd unit run-puppet [16:14:31] RECOVERY - phabdigest-bots-weekly service on tools2002 is OK: OK: Status of the systemd unit phabdigest-bots-weekly [16:16:03] [02puppet] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://github.com/FOSSBots/puppet/commit/937e15a15c62 [16:16:05] [02puppet] 07RhinosF1 03937e15a - monitoring: monitor DB backup script [16:16:06] [02puppet] 07RhinosF1 created branch 03RhinosF1-patch-1 - 13https://github.com/FOSSBots/puppet [16:16:07] [url] Page not found · GitHub · GitHub | github.com [16:16:08] [02puppet] 07RhinosF1 opened pull request 03#304: monitoring: monitor DB backup script - 13https://github.com/FOSSBots/puppet/pull/304 [16:16:08] [url] Page not found · GitHub · GitHub | github.com [16:17:46] [02puppet] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/FOSSBots/puppet/compare/db7c86043c0a...c08ef384c6d7 [16:17:46] [url] Comparing db7c86043c0a...c08ef384c6d7 · FOSSBots/puppet · GitHub | github.com [16:17:47] [02puppet] 07MacFan4000 03c08ef38 - move web repo [16:18:12] RhinosF1: i need an invite to the private [16:19:53] Zppix: try join now [16:52:31] [02puppet] 07sario528 closed pull request 03#304: monitoring: monitor DB backup script - 13https://github.com/FOSSBots/puppet/pull/304 [16:52:31] [url] Page not found · GitHub · GitHub | github.com [16:52:32] [02puppet] 07sario528 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/FOSSBots/puppet/compare/c08ef384c6d7...67fdb4f59a4e [16:52:33] [url] Comparing c08ef384c6d7...67fdb4f59a4e · FOSSBots/puppet · GitHub | github.com [16:52:34] [02puppet] 07RhinosF1 0367fdb4f - monitoring: monitor DB backup script (#304) [16:54:03] [02puppet] 07sario528 deleted branch 03RhinosF1-patch-1 - 13https://github.com/FOSSBots/puppet [16:54:03] [url] Page not found · GitHub · GitHub | github.com [16:54:04] [02puppet] 07sario528 deleted branch 03RhinosF1-patch-1 [16:54:50] [02puppet] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/FOSSBots/puppet/compare/67fdb4f59a4e...74913c34477a [16:54:51] [url] Comparing 67fdb4f59a4e...74913c34477a · FOSSBots/puppet · GitHub | github.com [16:54:52] [02puppet] 07MacFan4000 0374913c3 - Update init.pp [16:55:54] [02puppet] 07sario528 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/FOSSBots/puppet/compare/74913c34477a...9a0f9863b914 [16:55:54] [url] Comparing 74913c34477a...9a0f9863b914 · FOSSBots/puppet · GitHub | github.com [16:55:55] [02puppet] 07sario528 039a0f986 - Update README.md [17:01:54] [02puppet] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/FOSSBots/puppet/compare/9a0f9863b914...675b7c9bb27a [17:01:55] [url] Comparing 9a0f9863b914...675b7c9bb27a · FOSSBots/puppet · GitHub | github.com [17:01:56] [02puppet] 07MacFan4000 03675b7c9 - Update motd.epp [17:05:11] [02puppet] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/FOSSBots/puppet/compare/675b7c9bb27a...241a0129911f [17:05:12] [url] Comparing 675b7c9bb27a...241a0129911f · FOSSBots/puppet · GitHub | github.com [17:05:13] [02puppet] 07MacFan4000 03241a012 - update phab mailer address [21:11:47] [02sopel] 07dependabot[bot] opened pull request 03#112: build(deps): bump xmltodict from 0.12 to 0.13.0 - 13https://github.com/MirahezeBots/sopel/pull/112 [21:11:48] [url] Page not found · GitHub · GitHub | github.com [21:11:49] [02sopel] 07dependabot[bot] pushed 031 commit to 03dependabot/pip/xmltodict-0.13.0 [+0/-0/±1] 13https://github.com/MirahezeBots/sopel/commit/0630f0e76b8f [21:11:50] [02sopel] 07dependabot[bot] 030630f0e - build(deps): bump xmltodict from 0.12 to 0.13.0 Bumps [xmltodict](https://github.com/martinblech/xmltodict) from 0.12 to 0.13.0. - [Release notes](https://github.com/martinblech/xmltodict/releases) - [Changelog](https://github.com/martinblech/xmltodict/blob/master/CHANGELOG.md) - [Commits](https://github.com/martinblech/xmltodict/compare/v0.12.0...v0.13.0) --- updated- [21:11:50] dependencies: - dependency-name: xmltodict dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] [21:11:51] [url] GitHub - martinblech/xmltodict: Python module that makes working with XML feel like you are working with JSON | github.com [21:11:51] [url] Releases · martinblech/xmltodict · GitHub | github.com [21:11:52] [url] xmltodict/CHANGELOG.md at master · martinblech/xmltodict · GitHub | github.com [21:11:52] [02sopel] 07dependabot[bot] labeled pull request 03#112: build(deps): bump xmltodict from 0.12 to 0.13.0 - 13https://github.com/MirahezeBots/sopel/pull/112 [21:11:52] [url] Page not found · GitHub · GitHub | github.com [21:11:53] [02sopel] 07dependabot[bot] created branch 03dependabot/pip/xmltodict-0.13.0 - 13https://github.com/MirahezeBots/sopel [21:11:53] [url] Comparing v0.12.0...v0.13.0 · martinblech/xmltodict · GitHub | github.com [21:11:54] [url] Page not found · GitHub · GitHub | github.com [21:56:19] [02puppet] 07MacFan4000 commented on pull request 03#298: Bump wheel from 0.36.2 to 0.37.1 in /modules/sopel/files - 13https://github.com/FOSSBots/puppet/pull/298#issuecomment-1121624073 [21:56:19] [url] Bump wheel from 0.36.2 to 0.37.1 in /modules/sopel/files by dependabot[bot] · Pull Request #298 · FOSSBots/puppet · GitHub | github.com [21:56:20] [02puppet] 07dependabot[bot] closed pull request 03#298: Bump wheel from 0.36.2 to 0.37.1 in /modules/sopel/files - 13https://github.com/FOSSBots/puppet/pull/298 [21:56:21] [url] Page not found · GitHub · GitHub | github.com [21:56:22] [02puppet] 07dependabot[bot] deleted branch 03dependabot/pip/modules/sopel/files/wheel-0.37.1 - 13https://github.com/FOSSBots/puppet [21:56:22] [url] Page not found · GitHub · GitHub | github.com [21:56:23] [02puppet] 07dependabot[bot] deleted branch 03dependabot/pip/modules/sopel/files/wheel-0.37.1 [21:56:25] [02puppet] 07dependabot[bot] commented on pull request 03#298: Bump wheel from 0.36.2 to 0.37.1 in /modules/sopel/files - 13https://github.com/FOSSBots/puppet/pull/298#issuecomment-1121624110 [21:56:25] [url] Bump wheel from 0.36.2 to 0.37.1 in /modules/sopel/files by dependabot[bot] · Pull Request #298 · FOSSBots/puppet · GitHub | github.com [21:59:05] [02puppet] 07MacFan4000 commented on pull request 03#300: Bump praw from 5.4.0 to 7.5.0 in /modules/sopel/files - 13https://github.com/FOSSBots/puppet/pull/300#issuecomment-1121626039 [21:59:06] [url] Bump praw from 5.4.0 to 7.5.0 in /modules/sopel/files by dependabot[bot] · Pull Request #300 · FOSSBots/puppet · GitHub | github.com [21:59:07] [02puppet] 07dependabot[bot] closed pull request 03#300: Bump praw from 5.4.0 to 7.5.0 in /modules/sopel/files - 13https://github.com/FOSSBots/puppet/pull/300 [21:59:07] [url] Page not found · GitHub · GitHub | github.com [21:59:08] [02puppet] 07dependabot[bot] deleted branch 03dependabot/pip/modules/sopel/files/praw-7.5.0 [21:59:10] [02puppet] 07dependabot[bot] deleted branch 03dependabot/pip/modules/sopel/files/praw-7.5.0 - 13https://github.com/FOSSBots/puppet [21:59:10] [url] Page not found · GitHub · GitHub | github.com [21:59:11] [02puppet] 07dependabot[bot] commented on pull request 03#300: Bump praw from 5.4.0 to 7.5.0 in /modules/sopel/files - 13https://github.com/FOSSBots/puppet/pull/300#issuecomment-1121626078 [21:59:12] [url] Bump praw from 5.4.0 to 7.5.0 in /modules/sopel/files by dependabot[bot] · Pull Request #300 · FOSSBots/puppet · GitHub | github.com [21:59:17] [02puppet] 07MacFan4000 commented on pull request 03#299: Bump prawcore from 0.14.0 to 2.3.0 in /modules/sopel/files - 13https://github.com/FOSSBots/puppet/pull/299#issuecomment-1121626151 [21:59:17] [url] Bump prawcore from 0.14.0 to 2.3.0 in /modules/sopel/files by dependabot[bot] · Pull Request #299 · FOSSBots/puppet · GitHub | github.com [21:59:18] [02puppet] 07dependabot[bot] closed pull request 03#299: Bump prawcore from 0.14.0 to 2.3.0 in /modules/sopel/files - 13https://github.com/FOSSBots/puppet/pull/299 [21:59:19] [url] Page not found · GitHub · GitHub | github.com [21:59:20] [02puppet] 07dependabot[bot] deleted branch 03dependabot/pip/modules/sopel/files/prawcore-2.3.0 - 13https://github.com/FOSSBots/puppet [21:59:20] [url] Page not found · GitHub · GitHub | github.com [21:59:21] [02puppet] 07dependabot[bot] commented on pull request 03#299: Bump prawcore from 0.14.0 to 2.3.0 in /modules/sopel/files - 13https://github.com/FOSSBots/puppet/pull/299#issuecomment-1121626168 [21:59:22] [url] Bump prawcore from 0.14.0 to 2.3.0 in /modules/sopel/files by dependabot[bot] · Pull Request #299 · FOSSBots/puppet · GitHub | github.com [21:59:23] [02puppet] 07dependabot[bot] deleted branch 03dependabot/pip/modules/sopel/files/prawcore-2.3.0 [22:01:29] [02puppet] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/FOSSBots/puppet/compare/241a0129911f...144070f17502 [22:01:30] [url] Comparing 241a0129911f...144070f17502 · FOSSBots/puppet · GitHub | github.com [22:01:31] [02puppet] 07MacFan4000 03144070f - phab: update mail server [22:12:16] [02puppet] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/FOSSBots/puppet/compare/144070f17502...544a197ee14a [22:12:17] [url] Comparing 144070f17502...544a197ee14a · FOSSBots/puppet · GitHub | github.com [22:12:18] [02puppet] 07MacFan4000 03544a197 - update postfix relay [22:16:21] CUSTOM - DB Backups on db2002 is OK: OK: Status of the systemd unit backup-db macfan testing emails -macfan [22:33:25] [02puppet] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/FOSSBots/puppet/compare/544a197ee14a...c314ac01864f [22:33:26] [url] Comparing 544a197ee14a...c314ac01864f · FOSSBots/puppet · GitHub | github.com [22:33:27] [02puppet] 07MacFan4000 03c314ac0 - Update commands.conf [22:35:49] CUSTOM - DB Backups on db2002 is OK: OK: Status of the systemd unit backup-db macfan test email [22:37:02] ok, mail through zoho is fully working [22:37:15] (for phab,phabdigests,icinga) [23:48:35] Sario: mass delete icinga spam emails