[13:26:57] <Not-9244>	 [02puppet] 07MacFan4000 commented on pull request 03#253: relays: add a new one - 13https://github.com/MirahezeBots/puppet/pull/253#issuecomment-1046237932
[13:26:58] <Not-9244>	 [02puppet] 07MacFan4000 closed pull request 03#253: relays: add a new one - 13https://github.com/MirahezeBots/puppet/pull/253
[13:31:21] <MacFan4000>	 $log copied phabdigests config files from tools1 to tools2002
[13:31:30] <Void-bot>	 Saved item "copied phabdigests config files from tools1 to tools2002"
[13:32:49] <MacFan4000>	 $log switched phabdigests dns to point to tools2002
[13:32:58] <Void-bot>	 Saved item "switched phabdigests dns to point to tools2002"
[15:31:46] <Not-9244>	 [02puppet] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+0/-0/±1] 13https://github.com/MirahezeBots/puppet/commit/187298100372
[15:31:47] <Not-9244>	 [02puppet] 07MacFan4000 031872981 - phab: read-only
[15:31:49] <Not-9244>	 [02puppet] 07MacFan4000 created branch 03MacFan4000-patch-1 - 13https://github.com/MirahezeBots/puppet
[15:31:50] <Not-9244>	 [02puppet] 07MacFan4000 opened pull request 03#271: phab: read-only - 13https://github.com/MirahezeBots/puppet/pull/271
[15:38:41] <Not-9244>	 [02puppet] 07MacFan4000 closed pull request 03#271: phab: read-only - 13https://github.com/MirahezeBots/puppet/pull/271
[15:38:42] <Not-9244>	 [02puppet] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/MirahezeBots/puppet/compare/0d672a5c33a6...4101235ddfbb
[15:38:44] <Not-9244>	 [02puppet] 07MacFan4000 034101235 - phab: read-only (#271)
[15:44:26] <MacFan4000>	 $log phab readonly and SET GLOBAL read_only = 1;
[15:44:38] <Void-bot>	 Saved item "phab readonly and SET GLOBAL read_only = 1;"
[16:07:40] <icinga-mhbots>	 PROBLEM - procs on db2002 is WARNING: PROCS WARNING: 271 processes   
[16:10:40] <icinga-mhbots>	 RECOVERY - procs on db2002 is OK: PROCS OK: 231 processes   
[16:23:15] <Not-9244>	 [02puppet] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/MirahezeBots/puppet/compare/4101235ddfbb...1c65de8d0de4
[16:23:17] <Not-9244>	 [02puppet] 07MacFan4000 031c65de8 - roundcube: switch to db2002
[16:25:58] <MacFan4000>	 $log switch dns for webmail to tools2002
[16:26:10] <Void-bot>	 Saved item "switch dns for webmail to tools2002"
[17:03:35] <MacFan4000>	 $log switched dns for phab
[17:03:44] <Void-bot>	 Saved item "switched dns for phab"
[17:04:39] <icinga-mhbots>	 PROBLEM - Phabricator on tools1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 6790 bytes in 0.645 second response time   
[17:08:31] <Not-9244>	 [02puppet] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/MirahezeBots/puppet/compare/1c65de8d0de4...230b0b3114ab
[17:08:32] <Not-9244>	 [02puppet] 07MacFan4000 03230b0b3 - Update tools2002.yaml
[17:11:20] <MacFan4000>	 .op
[17:11:20] <MirahezeBot>	 Attempting to OP...
[17:32:39] <icinga-mhbots>	 RECOVERY - Phabricator on tools1 is OK: HTTP OK: HTTP/1.1 200 OK - 22218 bytes in 0.644 second response time   
[17:32:55] <RhinosF1>	 Hi chrs
[17:33:05] <chrs>	 hey
[17:33:51] <RhinosF1>	 Phab is now back but read only
[17:33:55] <RhinosF1>	 Repos are also missing
[17:33:59] <RhinosF1>	 How's chrs
[17:35:17] <chrs>	 good
[17:35:18] <chrs>	 you?
[17:35:35] <RhinosF1>	 Glad it's the weekend
[17:35:39] <RhinosF1>	 I got next week off
[17:35:57] <chrs>	 nice
[17:36:11] <Not-9244>	 [02puppet] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/MirahezeBots/puppet/compare/230b0b3114ab...e6b2e13eea80
[17:36:12] <Not-9244>	 [02puppet] 07MacFan4000 03e6b2e13 - switch icinga master
[17:37:52] <Not-9244>	 [02puppet] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/MirahezeBots/puppet/compare/e6b2e13eea80...562552971c72
[17:37:54] <Not-9244>	 [02puppet] 07MacFan4000 035625529 - Update site.pp
[17:41:58] <icinga-mhbots>	 PROBLEM - puppet service on tools1 is CRITICAL: CRITICAL: Status of the systemd unit run-puppet  
[17:42:08] <icinga-mhbots>	 	from /usr/lib/nagios/plugins/check_puppet_run:154:in `<main>'  
[17:49:41] <icinga-mhbots>	 PROBLEM - Phabricator on tools1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 5419 bytes in 0.554 second response time   
[17:51:41] <icinga-mhbots>	 RECOVERY - Phabricator on tools1 is OK: HTTP OK: HTTP/1.1 200 OK - 22216 bytes in 0.574 second response time   
[18:01:20] <Not-9244>	 [02puppet] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/MirahezeBots/puppet/compare/562552971c72...05fcf6180bf0
[18:01:22] <Not-9244>	 [02puppet] 07MacFan4000 0305fcf61 - Update master-zones.conf
[18:03:47] <Not-9244>	 [02puppet] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/MirahezeBots/puppet/compare/05fcf6180bf0...176ce2ea37d5
[18:03:49] <Not-9244>	 [02puppet] 07MacFan4000 03176ce2e - Update hosts.conf
[18:10:08] <icinga-mhbots>	 PROBLEM - Puppet on tools2002 is UNKNOWN: Remote Icinga instance 'tools2002.mirahezebots.org' is not connected to '112.node1.net.fosshost.org'  
[18:10:44] <icinga-mhbots>	 PROBLEM - http on tools2002 is UNKNOWN: Remote Icinga instance 'tools2002.mirahezebots.org' is not connected to '112.node1.net.fosshost.org'  
[18:10:48] <icinga-mhbots>	 PROBLEM - apt on tools2002 is UNKNOWN: Remote Icinga instance 'tools2002.mirahezebots.org' is not connected to '112.node1.net.fosshost.org'  
[18:10:58] <icinga-mhbots>	 PROBLEM - ufw service on tools2002 is UNKNOWN: Remote Icinga instance 'tools2002.mirahezebots.org' is not connected to '112.node1.net.fosshost.org'  
[18:11:02] <icinga-mhbots>	 PROBLEM - users on tools2002 is UNKNOWN: Remote Icinga instance 'tools2002.mirahezebots.org' is not connected to '112.node1.net.fosshost.org'  
[18:11:08] <icinga-mhbots>	 PROBLEM - disk / on tools2002 is UNKNOWN: Remote Icinga instance 'tools2002.mirahezebots.org' is not connected to '112.node1.net.fosshost.org'  
[18:11:14] <icinga-mhbots>	 PROBLEM - load on tools2002 is UNKNOWN: Remote Icinga instance 'tools2002.mirahezebots.org' is not connected to '112.node1.net.fosshost.org'  
[18:11:18] <icinga-mhbots>	 PROBLEM - puppet service on tools2002 is UNKNOWN: Remote Icinga instance 'tools2002.mirahezebots.org' is not connected to '112.node1.net.fosshost.org'  
[18:11:20] <icinga-mhbots>	 PROBLEM - ping6 #page on tools2002 is UNKNOWN: Remote Icinga instance 'tools2002.mirahezebots.org' is not connected to '112.node1.net.fosshost.org'  
[18:11:26] <icinga-mhbots>	 PROBLEM - disk on tools2002 is UNKNOWN: Remote Icinga instance 'tools2002.mirahezebots.org' is not connected to '112.node1.net.fosshost.org'  
[18:11:32] <icinga-mhbots>	 PROBLEM - ssh #page on tools2002 is UNKNOWN: Remote Icinga instance 'tools2002.mirahezebots.org' is not connected to '112.node1.net.fosshost.org'  
[18:11:40] <icinga-mhbots>	 PROBLEM - Flask-site on tools2002 is UNKNOWN: Remote Icinga instance 'tools2002.mirahezebots.org' is not connected to '112.node1.net.fosshost.org'  
[18:35:47] <RhinosF1>	 You can ignore the page
[19:04:29] <MacFan4000>	 Yay!
[19:07:14] <RhinosF1>	 MacFan4000: nice!
[19:07:39] <RhinosF1>	 Might be a bit less around next 2 hours but ping if needed
[19:07:47] <RhinosF1>	 I'll keep glancing at phone
[19:11:23] <MacFan4000>	 $log switch dns for icinga
[19:11:32] <Void-bot>	 Saved item "switch dns for icinga"
[19:11:46] <RhinosF1>	 Looks like icinga web needs some config
[19:24:41] <Not-9244>	 [02puppet] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/MirahezeBots/puppet/compare/176ce2ea37d5...30e0958fd351
[19:24:42] <Not-9244>	 [02puppet] 07MacFan4000 0330e0958 - Update services.conf
[19:31:10] <RhinosF1>	 $log purging cloudflare cache for icinga
[19:31:19] <Void-bot>	 Saved item "purging cloudflare cache for icinga"
[19:32:45] <Not-9244>	 [02puppet] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/MirahezeBots/puppet/compare/30e0958fd351...0e4ac93b9181
[19:32:47] <Not-9244>	 [02puppet] 07MacFan4000 030e4ac93 - Update node-zones.conf
[19:52:24] <Not-9244>	 [02puppet] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/MirahezeBots/puppet/compare/0e4ac93b9181...7a29eafe0c5a
[19:52:26] <Not-9244>	 [02puppet] 07MacFan4000 037a29eaf - Update hosts.conf
[19:54:44] <Not-9244>	 [02puppet] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/MirahezeBots/puppet/compare/7a29eafe0c5a...21477e2cd852
[19:54:45] <Not-9244>	 [02puppet] 07MacFan4000 0321477e2 - Update services.conf
[20:22:10] <icinga-mhbots>	 PROBLEM - Icinga IRC bot on tools2002 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'irc.py'   
[21:19:48] <Not-9244>	 [02puppet] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/MirahezeBots/puppet/compare/21477e2cd852...295e0e31b26f
[21:19:49] <Not-9244>	 [02puppet] 07MacFan4000 03295e0e3 - phab: lift RO
[21:34:56] <Not-9244>	 [02puppet] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/MirahezeBots/puppet/compare/295e0e31b26f...bc9978d6f9d4
[21:34:57] <Not-9244>	 [02puppet] 07MacFan4000 03bc9978d - Update master-zones.conf
[21:36:49] <Not-9244>	 [02puppet] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/MirahezeBots/puppet/compare/bc9978d6f9d4...fcaff8a669ae
[21:36:51] <Not-9244>	 [02puppet] 07MacFan4000 03fcaff8a - Update master-zones.conf
[22:15:38] <Not-9244>	 [02puppet] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/MirahezeBots/puppet/compare/fcaff8a669ae...d9954b83d8c2
[22:15:40] <Not-9244>	 [02puppet] 07MacFan4000 03d9954b8 - Update master-zones.conf
[22:23:45] <icinga-mhbots>	 RECOVERY - mariadb service on db2002 is OK: OK: Status of the systemd unit mariadb  
[22:23:51] <icinga-mhbots>	 RECOVERY - ping6 #page on db2002 is OK: PING OK - Packet loss = 0%, RTA = 0.26 ms  
[22:24:01] <icinga-mhbots>	 RECOVERY - ssh #page on db2002 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0)   
[22:24:09] <icinga-mhbots>	 RECOVERY - procs on db2002 is OK: PROCS OK: 107 processes   
[22:24:13] <icinga-mhbots>	 RECOVERY - disk on db2002 is OK: DISK OK - free space: / 27563 MB (88% inode=97%); /boot/efi 126 MB (99% inode=-);  
[22:24:23] <icinga-mhbots>	 RECOVERY - disk / on db2002 is OK: DISK OK - free space: / 27563 MB (88% inode=97%); /boot/efi 126 MB (99% inode=-);  
[22:24:25] <icinga-mhbots>	 RECOVERY - puppet service on db2002 is OK: OK: Status of the systemd unit run-puppet  
[22:24:27] <icinga-mhbots>	 RECOVERY - load on db2002 is OK: OK - load average: 0.09, 0.06, 0.01  
[22:24:31] <icinga-mhbots>	 RECOVERY - users on db2002 is OK: USERS OK - 1 users currently logged in   
[22:25:58] <RhinosF1>	 MacFan4000: fixed?
[22:27:59] <icinga-mhbots>	 RECOVERY - disk / on bots2002 is OK: DISK OK - free space: / 22747 MB (72% inode=94%); /boot/efi 126 MB (99% inode=-);  
[22:28:03] <MacFan4000>	 Yes I had to switch the zone endpoint to bots1’s ipv6 address
[22:28:03] <icinga-mhbots>	 RECOVERY - apt on bots2002 is OK: APT OK: 0 packages available for upgrade (0 critical updates).   
[22:28:07] <icinga-mhbots>	 RECOVERY - MirahezeBots SSL on bots2002 is OK: OK - Certificate 'CloudFlare Origin Certificate' will expire on Tue Jan 29 15:23:00 2036 +0000.  
[22:28:11] <icinga-mhbots>	 RECOVERY - users on bots2002 is OK: USERS OK - 2 users currently logged in   
[22:28:13] <icinga-mhbots>	 RECOVERY - Discord-relay on bots2002 is OK: PROCS OK: 1 process with command name 'matterbridge'   
[22:28:14] <RhinosF1>	 ok
[22:28:21] <icinga-mhbots>	 RECOVERY - ssh #page on bots2002 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0)   
[22:28:23] <icinga-mhbots>	 RECOVERY - procs on bots2002 is OK: PROCS OK: 117 processes   
[22:28:29] <icinga-mhbots>	 RECOVERY - ufw service on bots2002 is OK: OK: Status of the systemd unit ufw  
[22:28:31] <icinga-mhbots>	 RECOVERY - load on bots2002 is OK: OK - load average: 0.32, 0.28, 0.27  
[22:28:34] <icinga-mhbots>	 RECOVERY - Apache on bots2002 is OK: PROCS OK: 3 processes with command name 'apache2'   
[22:28:36] <icinga-mhbots>	 RECOVERY - Flask-site on bots2002 is OK: HTTP OK: HTTP/1.1 200 OK - 5561 bytes in 0.094 second response time   
[22:28:45] <RhinosF1>	 MacFan4000: why not tools2002
[22:28:46] <icinga-mhbots>	 RECOVERY - disk on bots2002 is OK: DISK OK - free space: / 22747 MB (72% inode=94%); /boot/efi 126 MB (99% inode=-);  
[22:28:52] <icinga-mhbots>	 RECOVERY - ping6 #page on bots2002 is OK: PING OK - Packet loss = 0%, RTA = 0.28 ms  
[22:28:54] <icinga-mhbots>	 RECOVERY - streambot service on bots2002 is OK: OK: Status of the systemd unit streambot  
[22:37:04] <icinga-mhbots>	 CUSTOM - Puppet on bots2002 is OK: OK: Puppet is currently enabled, last run 5 minutes ago with 0 failures macfan test
[23:12:02] <Not-9244>	 [02puppet] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/MirahezeBots/puppet/compare/d9954b83d8c2...2a2d51e5827d
[23:12:03] <Not-9244>	 [02puppet] 07MacFan4000 032a2d51e - Update main.cf
[23:13:18] <icinga-mhbots>	 CUSTOM - Puppet on bots2002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures macfan test
[23:18:04] <icinga-mhbots>	 CUSTOM - Puppet on bots2002 is OK: OK: Puppet is currently enabled, last run 5 minutes ago with 0 failures macfan test
[23:27:19] <Not-9244>	 [02puppet] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/MirahezeBots/puppet/compare/2a2d51e5827d...f617fc6e1034
[23:27:21] <Not-9244>	 [02puppet] 07MacFan4000 03f617fc6 - Update commands.conf
[23:27:59] <icinga-mhbots>	 RECOVERY - Icinga IRC bot on tools2002 is OK: PROCS OK: 1 process with command name 'python3'