[07:21:45] <icinga-mhbots>	 PROBLEM - MirahezeBots SSL on bots1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[07:28:32] <icinga-mhbots>	 RECOVERY - MirahezeBots SSL on bots1 is OK: OK - Certificate 'CloudFlare Origin Certificate' will expire on Tue 29 Jan 2036 15:23:00 GMT +0000.  
[07:30:58] <icinga-mhbots>	 PROBLEM - MirahezeBots SSL on bots1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[07:31:58] <icinga-mhbots>	 RECOVERY - MirahezeBots SSL on bots1 is OK: OK - Certificate 'CloudFlare Origin Certificate' will expire on Tue 29 Jan 2036 15:23:00 GMT +0000.  
[11:01:38] <Not-1527>	 [02sopel] 07sario528 closed pull request 03#61: build(deps-dev): bump requests-mock from 1.9.1 to 1.9.3 - 13https://git.io/JCk95
[11:01:40] <Not-1527>	 [02sopel] 07sario528 pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JWhJy
[11:01:41] <Not-1527>	 [02sopel] 07sario528 030c95a74 - Merge pull request #61 from MirahezeBots/dependabot/pip/requests-mock-1.9.3 build(deps-dev): bump requests-mock from 1.9.1 to 1.9.3
[11:01:48] <Not-1527>	 [02sopel] 07dependabot[bot] deleted branch 03dependabot/pip/requests-mock-1.9.3
[11:01:49] <Not-1527>	 [02sopel] 07dependabot[bot] deleted branch 03dependabot/pip/requests-mock-1.9.3 - 13https://git.io/JYcoO
[11:02:05] <Not-1527>	 [02sopel] 07dependabot[bot] edited pull request 03#62: build(deps-dev): bump furo from 2021.6.24b37 to 2021.7.5b38 - 13https://git.io/JCk9x
[11:02:32] <Not-1527>	 [02sopel] 07dependabot[bot] edited pull request 03#62: build(deps-dev): bump furo from 2021.6.24b37 to 2021.7.5b38 - 13https://git.io/JCk9x
[11:02:34] <Not-1527>	 [02sopel] 07dependabot[bot] pushed 031 commit to 03dependabot/pip/furo-2021.7.5b38 [+0/-0/±1] 13https://git.io/JWhUv
[11:02:36] <Not-1527>	 [02sopel] 07dependabot[bot] 036942dc0 - build(deps-dev): bump furo from 2021.6.24b37 to 2021.7.5b38 Bumps [furo](https://github.com/pradyunsg/furo) from 2021.6.24b37 to 2021.7.5b38. - [Release notes](https://github.com/pradyunsg/furo/releases) - [Changelog](https://github.com/pradyunsg/furo/blob/main/docs/changelog.md) - [Commits](https://github.com/pradyunsg/furo/compare/2021.06.24.beta37...2021.07.05.beta38)
[11:02:36] <Not-1527>	 --- updated-dependencies: - dependency-name: furo dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
[11:02:36] <MirahezeBot>	 [url] GitHub - pradyunsg/furo: A clean customizable documentation theme for Sphinx | github.com
[11:02:37] <MirahezeBot>	 [url] Releases · pradyunsg/furo · GitHub | github.com
[11:02:37] <MirahezeBot>	 [url] furo/changelog.md at main · pradyunsg/furo · GitHub | github.com
[11:02:37] <Not-1527>	 [02sopel] 07dependabot[bot] synchronize pull request 03#62: build(deps-dev): bump furo from 2021.6.24b37 to 2021.7.5b38 - 13https://git.io/JCk9x
[11:02:38] <MirahezeBot>	 [url] Comparing 2021.06.24.beta37...2021.07.05.beta38 · pradyunsg/furo · GitHub | github.com
[11:02:39] <Not-1527>	 [02sopel] 07dependabot[bot] edited pull request 03#62: build(deps-dev): bump furo from 2021.6.24b37 to 2021.7.5b38 - 13https://git.io/JCk9x
[11:03:09] <Not-1527>	 [02sopel] 07dependabot[bot] edited pull request 03#62: build(deps-dev): bump furo from 2021.6.24b37 to 2021.7.5b38 - 13https://git.io/JCk9x
[11:12:57] <Not-1527>	 [02sopel] 07sario528 closed pull request 03#62: build(deps-dev): bump furo from 2021.6.24b37 to 2021.7.5b38 - 13https://git.io/JCk9x
[11:12:58] <Not-1527>	 [02sopel] 07sario528 pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JWhIO
[11:13:00] <Not-1527>	 [02sopel] 07sario528 0342bb998 - Merge pull request #62 from MirahezeBots/dependabot/pip/furo-2021.7.5b38 build(deps-dev): bump furo from 2021.6.24b37 to 2021.7.5b38
[11:13:05] <Not-1527>	 [02sopel] 07dependabot[bot] deleted branch 03dependabot/pip/furo-2021.7.5b38
[11:13:07] <Not-1527>	 [02sopel] 07dependabot[bot] deleted branch 03dependabot/pip/furo-2021.7.5b38 - 13https://git.io/JYcoO
[11:15:00] <Not-1527>	 [02sopel] 07sario528 commented on pull request 03#64: build(deps): bump emoji from 1.2.0 to 1.4.1 - 13https://git.io/JWhLe
[11:15:02] <Not-1527>	 [02sopel] 07dependabot[bot] edited pull request 03#64: build(deps): bump emoji from 1.2.0 to 1.4.1 - 13https://git.io/JWAty
[11:15:41] <Not-1527>	 [02sopel] 07dependabot[bot] edited pull request 03#64: build(deps): bump emoji from 1.2.0 to 1.4.1 - 13https://git.io/JWAty
[11:15:43] <Not-1527>	 [02sopel] 07dependabot[bot] pushed 031 commit to 03dependabot/pip/emoji-1.4.1 [+0/-0/±1] 13https://git.io/JWhLO
[11:15:44] <Not-1527>	 [02sopel] 07dependabot[bot] 03572a09f - build(deps): bump emoji from 1.2.0 to 1.4.1 Bumps [emoji](https://github.com/carpedm20/emoji) from 1.2.0 to 1.4.1. - [Release notes](https://github.com/carpedm20/emoji/releases) - [Changelog](https://github.com/carpedm20/emoji/blob/master/CHANGES.md) - [Commits](https://github.com/carpedm20/emoji/compare/v.1.2.0...v.1.4.1) --- updated-dependencies: - dependency-name:
[11:15:45] <Not-1527>	 emoji dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
[11:15:45] <MirahezeBot>	 [url] GitHub - carpedm20/emoji: emoji terminal output for Python | github.com
[11:15:45] <MirahezeBot>	 [url] Releases · carpedm20/emoji · GitHub | github.com
[11:15:46] <MirahezeBot>	 [url] emoji/CHANGES.md at master · carpedm20/emoji · GitHub | github.com
[11:15:46] <Not-1527>	 [02sopel] 07dependabot[bot] synchronize pull request 03#64: build(deps): bump emoji from 1.2.0 to 1.4.1 - 13https://git.io/JWAty
[11:15:47] <MirahezeBot>	 [url] Comparing v.1.2.0...v.1.4.1 · carpedm20/emoji · GitHub | github.com
[11:22:19] <Not-1527>	 [02sopel] 07sario528 closed pull request 03#64: build(deps): bump emoji from 1.2.0 to 1.4.1 - 13https://git.io/JWAty
[11:22:20] <Not-1527>	 [02sopel] 07sario528 pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JWhtA
[11:22:22] <Not-1527>	 [02sopel] 07sario528 034179f82 - Merge pull request #64 from MirahezeBots/dependabot/pip/emoji-1.4.1 build(deps): bump emoji from 1.2.0 to 1.4.1
[11:22:35] <Not-1527>	 [02sopel] 07dependabot[bot] deleted branch 03dependabot/pip/emoji-1.4.1
[11:22:36] <Not-1527>	 [02sopel] 07dependabot[bot] deleted branch 03dependabot/pip/emoji-1.4.1 - 13https://git.io/JYcoO
[13:43:17] <icinga-mhbots>	 PROBLEM - apt on bots1 is CRITICAL: APT CRITICAL: 13 packages available for upgrade (8 critical updates).   
[13:43:31] <RhinosF1>	 There we go Sario
[13:43:58] <Sario>	 Upgrade in progress
[13:44:17] <icinga-mhbots>	 PROBLEM - apt on bots1 is UNKNOWN: Remote Icinga instance '100.node4.net.fosshost.org' is not connected to '112.node1.net.fosshost.org'  
[13:44:34] <RhinosF1>	 That's good
[13:44:57] <RhinosF1>	 I guess icinga crashed
[13:45:17] <icinga-mhbots>	 RECOVERY - apt on bots1 is OK: APT OK: 0 packages available for upgrade (0 critical updates).   
[13:45:26] <RhinosF1>	 That's good
[13:45:31] <RhinosF1>	 Actually
[13:52:33] <Not-1527>	 [02puppet] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JWh5c
[13:52:34] <Not-1527>	 [02puppet] 07RhinosF1 0340b936e - users: move Sario to ops
[13:52:36] <Not-1527>	 [02puppet] 07RhinosF1 created branch 03RhinosF1-patch-1 - 13https://git.io/JJGvA
[13:52:37] <Not-1527>	 [02puppet] 07RhinosF1 opened pull request 03#192: users: move Sario to ops - 13https://git.io/JWh5W
[13:53:06] <RhinosF1>	 Sario: needs someone else to sign off and Reception|away but ^
[13:53:11] <RhinosF1>	 Reception|away: any objections
[13:53:43] <RhinosF1>	 MacFan4000: ^
[13:58:04] <RhinosF1>	 Once Reception|away or Void give green it's good to go
[13:59:46] <Reception|away>	 No objections from me, though as usual I don't feel involved enough to make a clear decision 
[14:01:01] <MacFan4000>	 RhinosF1: ^^
[14:01:35] <Not-1527>	 [02puppet] 07RhinosF1 closed pull request 03#192: users: move Sario to ops - 13https://git.io/JWh5W
[14:01:36] <Not-1527>	 [02puppet] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JWhFH
[14:01:38] <Not-1527>	 [02puppet] 07RhinosF1 03cdf7b4a - users: move Sario to ops (#192)
[14:01:39] <Not-1527>	 [02puppet] 07RhinosF1 deleted branch 03RhinosF1-patch-1
[14:01:41] <Not-1527>	 [02puppet] 07RhinosF1 deleted branch 03RhinosF1-patch-1 - 13https://git.io/JJGvA
[14:01:46] <RhinosF1>	 MacFan4000: ty
[14:01:54] <RhinosF1>	 Sario: merged
[14:02:02] <RhinosF1>	 You can sync own access
[14:02:20] <Sario>	 ack, thanks
[14:02:50] <RhinosF1>	 Or wait for pupppet in 8 minutes
[14:03:06] <MacFan4000>	 $log removed Sario from icinga sopel-admins and added to admins
[14:03:13] <Void-bot>	 Saved item "removed Sario from icinga sopel-admins and added to admins"
[14:03:20] <Sario>	 I'll wait, I'm multitasking anyway
[14:03:46] <MacFan4000>	 $log made Sario a GH org owner
[14:03:55] <Void-bot>	 Saved item "made Sario a GH org owner"
[14:05:42] <RhinosF1>	 $log moved Sario to github admins group
[14:05:47] <Void-bot>	 Saved item "moved Sario to github admins group"
[14:07:24] <RhinosF1>	 MacFan4000: can you give phab admin
[14:08:22] <StreamBotMH>	 [[Team Members]] on botswiki was edited by MacFan4000 with the following comment: updates Link: https://wiki.mirahezebots.org/w/index.php?diff=4603&oldid=4524&rcid=1457
[14:08:33] <StreamBotMH>	 On botswiki MacFan4000 MacFan4000 marked [[Team Members]] for translation; https://bots.miraheze.org/wiki/Special:Log/pagetranslation
[14:08:45] <MacFan4000>	 already has it
[14:09:41] <RhinosF1>	 Ack
[14:09:50] <RhinosF1>	 I can give Sario aarch access if he signs up
[14:10:00] <RhinosF1>	 console.aarch64.com
[14:12:17] <Sario>	 RhinosF1: account made, sario528@gmail.com
[14:13:00] <RhinosF1>	 Sario: done
[14:13:20] <RhinosF1>	 You should see us appear now
[14:13:45] <RhinosF1>	 There's only test2001 on that but otherwise use cli because the other 3 VMs will be decom in a few weeks
[14:14:16] <Sario>	 I see it, and ack
[14:15:48] <MacFan4000>	 yeah, there isn't really a good way to share proxmox access anyway
[14:16:01] <RhinosF1>	 I only have bots1
[14:16:18] <RhinosF1>	 No one has tools/db
[14:29:05] <Not-1527>	 [02puppet] 07sario528 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JWjej
[14:29:07] <Not-1527>	 [02puppet] 07sario528 0397d6fef - Make phab read-only
[14:29:20] <Sario>	 $log Set phab read only
[14:29:26] <Void-bot>	 Saved item "Set phab read only"
[14:29:54] <icinga-mhbots>	 PROBLEM - apt on db1 is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates).   
[14:30:39] <RhinosF1>	 Sario: when puppet has ran, you'll see a message on phab
[14:31:16] <Sario>	 ack
[14:35:03] <Sario>	 $log set MariaDB read only
[14:35:09] <Void-bot>	 Saved item "set MariaDB read only"
[14:36:59] <icinga-mhbots>	 RECOVERY - apt on db1 is OK: APT OK: 0 packages available for upgrade (0 critical updates).   
[14:37:05] <icinga-mhbots>	 PROBLEM - apt on test2001 is CRITICAL: APT CRITICAL: 8 packages available for upgrade (8 critical updates).   
[14:45:11] <Sario>	 $log reboot db1
[14:45:16] <Void-bot>	 Saved item "reboot db1"
[14:46:50] <Sario>	 $log set MariaDB read/write
[14:46:55] <Void-bot>	 Saved item "set MariaDB read/write"
[14:47:11] <Not-1527>	 [02puppet] 07sario528 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JWjkS
[14:47:12] <Not-1527>	 [02puppet] 07sario528 037a95cd9 - set phab read/write
[14:47:23] <Sario>	 $log set phab read/write
[14:47:28] <Void-bot>	 Saved item "set phab read/write"
[14:48:01] <RhinosF1>	 Sario: I'm still seeing read only messages, ensure puppet has run
[14:50:12] <Sario>	 ran puppet, phab now writeable 
[14:50:38] <RhinosF1>	 Looks good
[14:51:06] <Sario>	 That was kinda fun, running all the upgrades
[14:51:54] <RhinosF1>	 Good :)
[14:56:52] <Sario>	 T267 filed
[14:56:54] <MirahezeBot>	 https://phab.mirahezebots.org/T267 - Add MariaDB to puppet, authored by Sario528, assigned to None, Priority: Normal, Status: Open
[15:00:39] <RhinosF1>	 Cool
[21:03:36] <RhinosF1>	 Sario: if you want to do the honours, sudo reboot now
[21:04:07] <Sario>	 ack, bots1 first
[21:04:12] <RhinosF1>	 Yeah
[21:04:42] <RhinosF1>	 You only need to do those with reboot files. I always do phab + mysql RO before doing tools1/db1
[21:05:04] <Sario>	 Ok, so we don't need to reboot tools1
[21:05:11] <RhinosF1>	 Normally it's ~1 minute
[21:05:19] <RhinosF1>	 To reboot
[21:05:50] <RhinosF1>	 Why didn't StreamBot kill cleanly?
[21:05:55] <Sario>	 idk
[21:06:10] <Sario>	 I'll restart it in just a moment
[21:06:20] <RhinosF1>	 Stop the service, let it quit then start
[21:06:34] <RhinosF1>	 80 seconds was the relay which is what I use
[21:06:41] <RhinosF1>	 As my marker
[21:06:48] <Sario>	 Ok
[21:07:23] <RhinosF1>	 Sario: don't forget test2001 upgrade as it's alerting
[21:07:25] <icinga-mhbots>	 PROBLEM - MirahezeBots sopel on bots1 is WARNING: PROCS WARNING: 1 process with command name 'sopel'   
[21:07:47] <RhinosF1>	 Yeah that's not good though I guess
[21:07:51] <RhinosF1>	 Beta is down
[21:08:53] <icinga-mhbots>	 ACKNOWLEDGEMENT - MirahezeBots sopel on bots1 is WARNING: PROCS WARNING: 1 process with command name 'sopel'  rhinosf1 Maint 
[21:10:14] <Sario>	 !list
[21:10:14] <StreamBotMH>	 botswiki; all;
[21:10:20] <Sario>	 Streambot is up
[21:11:23] <RhinosF1>	 Cool
[21:11:29] <RhinosF1>	 Have a look at beta
[21:11:44] <Sario>	 Looking now
[21:11:55] <RhinosF1>	 Check sudo service mirahezebottestlibera status Sario
[21:12:01] <RhinosF1>	 As that will have logs
[21:12:10] <RhinosF1>	 Even though icinga says it's fine
[21:12:14] <RhinosF1>	 Oh you're back
[21:12:17] <icinga-mhbots>	 RECOVERY - MirahezeBots sopel on bots1 is OK: PROCS OK: 2 processes with command name 'sopel'   
[21:12:39] <Sario>	 I did a service stop, gave it a sec, then a start
[21:16:16] <RhinosF1>	 Ok
[21:20:51] <Sario>	 Looking at the logs, it looks like the server failed to auto-start Beta after the reboot.
[21:21:50] <icinga-mhbots>	 ACKNOWLEDGEMENT - apt on test2001 is CRITICAL: APT CRITICAL: 8 packages available for upgrade (8 critical updates).  macfan Sario will do soon
[21:26:15] <RhinosF1>	 ok
[21:27:13] <RhinosF1>	 loading syslog
[21:30:46] <RhinosF1>	 Sario: Jul 20 21:04:58 bots1 systemd[1]: Started Sopel IRC bot (test instance).
[21:35:52] <RhinosF1>	 $log rhinosf1@bots1:~$ sudo -u sopel /srv/sopelbots/devvenv/bin/pip3 install -U "idna<4,>=3.2"
[21:35:57] <Void-bot>	 Saved item "rhinosf1@bots1:~$ sudo -u sopel /srv/sopelbots/devvenv/bin/pip3 install -U "idna<4,>=3.2""
[21:41:20] <Not-1527>	 [02puppet] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+1/-0/±0] 13https://git.io/JleEQ
[21:41:22] <Not-1527>	 [02puppet] 07RhinosF1 03dbef0f4 - Create sshd_config
[21:41:23] <Not-1527>	 [02puppet] 07RhinosF1 created branch 03RhinosF1-patch-1 - 13https://git.io/JJGvA
[21:41:26] <Not-1527>	 [02puppet] 07RhinosF1 opened pull request 03#193: Create sshd_config - 13https://git.io/JleE5
[21:45:03] <Not-1527>	 [02puppet] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/Jleun
[21:45:04] <Not-1527>	 [02puppet] 07RhinosF1 032fd1294 - Update base.pp
[21:45:06] <Not-1527>	 [02puppet] 07RhinosF1 synchronize pull request 03#193: Create sshd_config - 13https://git.io/JleE5
[21:45:21] <RhinosF1>	 MacFan4000: ^
[21:45:30] <RhinosF1>	 I can merge if you test ssh after
[21:45:35] <MacFan4000>	 ok
[21:46:56] <Not-1527>	 [02puppet] 07RhinosF1 closed pull request 03#193: Create sshd_config - 13https://git.io/JleE5
[21:46:57] <Not-1527>	 [02puppet] 07RhinosF1 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/Jleu6
[21:46:59] <Not-1527>	 [02puppet] 07RhinosF1 03592e0eb - Create sshd_config (#193) * Create sshd_config * Update base.pp
[21:47:00] <Not-1527>	 [02puppet] 07RhinosF1 deleted branch 03RhinosF1-patch-1
[21:47:02] <Not-1527>	 [02puppet] 07RhinosF1 deleted branch 03RhinosF1-patch-1 - 13https://git.io/JJGvA
[21:47:47] <MacFan4000>	 RhinosF1: you have an error
[21:47:48] <MacFan4000>	 Error: Evaluation Error: Error while evaluating a Resource Statement, Systemd::Service[ssh]: expects a value for parameter 'content' (file: /etc/puppet/code/modules/profile/manifests/base.pp, line: 18) on node bots1.mirahezebots.org
[21:48:20] <Not-1527>	 [02puppet] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jleub
[21:48:22] <Not-1527>	 [02puppet] 07RhinosF1 0372b229b - Update base.pp
[21:48:45] <Not-1527>	 [02puppet] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jleux
[21:48:47] <Not-1527>	 [02puppet] 07RhinosF1 034d539ff - Update base.pp
[21:49:40] <MacFan4000>	 ssh failed to start RhinosF1
[21:50:05] <Not-1527>	 [02puppet] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jlezf
[21:50:07] <Not-1527>	 [02puppet] 07RhinosF1 03f4d3c88 - Update base.pp
[21:50:18] <icinga-mhbots>	 	from /usr/lib/nagios/plugins/check_puppet_run:154:in `<main>'  
[21:50:26] <icinga-mhbots>	 PROBLEM - ssh on test2001 is CRITICAL: connect to address 2a09:11c0:339:ffff::2 and port 22: Connection refused  
[21:50:32] <RhinosF1>	 MacFan4000: yes i know
[21:50:38] <icinga-mhbots>	 	from /usr/lib/nagios/plugins/check_puppet_run:154:in `<main>'  
[21:50:53] <RhinosF1>	 MacFan4000: try test2001
[21:51:18] <icinga-mhbots>	 RECOVERY - Puppet on test2001 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures  
[21:51:20] <icinga-mhbots>	 	from /usr/lib/nagios/plugins/check_puppet_run:154:in `<main>'  
[21:51:26] <icinga-mhbots>	 RECOVERY - ssh on test2001 is OK: SSH OK - OpenSSH_7.9p1 Debian-10+deb10u2 (protocol 2.0)   
[21:51:38] <icinga-mhbots>	 RECOVERY - Puppet on bots1 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures  
[21:51:58] <MacFan4000>	 everything is accessible
[21:52:20] <icinga-mhbots>	 RECOVERY - Puppet on db1 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures  
[21:54:49] <RhinosF1>	 $log upgrade test2001, run puppet everywhere, ssh is happy
[21:54:55] <Void-bot>	 Saved item "upgrade test2001, run puppet everywhere, ssh is happy"
[21:55:06] <icinga-mhbots>	 RECOVERY - apt on test2001 is OK: APT OK: 0 packages available for upgrade (0 critical updates).