[00:08:52] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.28, 7.02, 6.07
[00:22:02] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.21, 7.49, 6.89
[00:24:15] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.85, 7.26, 6.89
[00:26:22] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.35, 7.10, 6.18
[00:28:32] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.41, 6.66, 6.13
[00:30:48] <icinga-miraheze>	 RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.69, 6.45, 6.70
[00:36:13] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.86, 7.95, 6.70
[00:37:46] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.31, 8.33, 7.30
[00:38:26] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.37, 7.17, 6.57
[00:39:57] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.77, 7.89, 7.29
[00:51:31] <icinga-miraheze>	 RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.52, 6.12, 6.54
[00:56:20] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.67, 7.04, 6.89
[00:56:48] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.21, 7.40, 6.88
[00:58:33] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.37, 7.21, 6.95
[00:58:34] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.01, 3.57, 3.26
[00:59:57] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.36, 8.05, 7.48
[01:00:47] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.38, 7.42, 7.08
[01:01:14] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.76, 8.49, 7.42
[01:02:09] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.89, 7.86, 7.48
[01:03:03] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.88, 3.23, 3.20
[01:04:23] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.26, 7.84, 7.49
[01:05:18] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.34, 8.14, 7.40
[01:05:38] <Not>	 [02mediawiki] 07Universal-Omega closed pull request 03#2994: Install EmbedSpotify (T7567) - 13https://git.io/JzvuJ
[01:05:39] <Not>	 [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+1/-0/±2] 13https://git.io/Jzftz
[01:05:40] <Not>	 [02miraheze/mediawiki] 07R4356th 03baa1a8a - Install EmbedSpotify (T7567) (#2994)
[01:09:54] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.63, 7.40, 7.42
[01:12:09] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.71, 6.85, 7.42
[01:13:11] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.35, 7.77, 7.99
[01:18:42] <icinga-miraheze>	 RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.83, 6.12, 6.72
[01:18:45] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.73, 5.39, 6.54
[01:28:40] <icinga-miraheze>	 RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.74, 6.07, 6.68
[01:35:30] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.53, 7.01, 6.61
[01:35:33] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.11, 7.25, 6.98
[01:37:37] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.07, 6.58, 6.50
[01:37:41] <icinga-miraheze>	 RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.25, 6.48, 6.74
[01:42:17] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.01, 6.89, 6.84
[01:44:29] <icinga-miraheze>	 RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.59, 6.35, 6.64
[02:09:35] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.67, 3.77, 3.37
[02:11:44] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.45, 3.53, 3.33
[02:13:51] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.84, 3.23, 3.25
[02:20:42] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.71, 3.35, 3.27
[02:23:02] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.76, 2.69, 3.03
[02:35:58] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.42, 7.57, 6.46
[02:38:12] <icinga-miraheze>	 PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.03, 6.91, 6.36
[02:40:22] <icinga-miraheze>	 RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.25, 6.34, 6.20
[02:42:36] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.22, 3.50, 3.30
[02:44:48] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.75, 3.20, 3.22
[05:18:21] <Reception123>	 !log begin reimage mw8 back to buster 
[05:18:25] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[05:21:55] <icinga-miraheze>	 PROBLEM - mw8 Puppet on mw8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[05:23:12] <icinga-miraheze>	 PROBLEM - Host mw8 is DOWN: PING CRITICAL - Packet loss = 100%
[05:28:52] <icinga-miraheze>	 RECOVERY - Host mw8 is UP: PING OK - Packet loss = 77%, RTA = 0.64 ms
[05:28:52] <icinga-miraheze>	 PROBLEM - mw8 HTTPS on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[05:28:52] <icinga-miraheze>	 PROBLEM - mw8 PowerDNS Recursor on mw8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[05:28:53] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[05:28:53] <icinga-miraheze>	 PROBLEM - mw8 JobRunner Service on mw8 is CRITICAL: connect to address 51.195.236.221 port 5666: Connection refusedconnect to host 51.195.236.221 port 5666: Connection refused
[05:28:53] <icinga-miraheze>	 PROBLEM - mw8 Check Gluster Clients on mw8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[05:28:53] <icinga-miraheze>	 PROBLEM - mw8 SSH on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[05:28:53] <icinga-miraheze>	 PROBLEM - mw8 Disk Space on mw8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[05:29:27] <icinga-miraheze>	 PROBLEM - mw8 APT on mw8 is CRITICAL: connect to address 51.195.236.221 port 5666: Connection refusedconnect to host 51.195.236.221 port 5666: Connection refused
[05:29:33] <icinga-miraheze>	 PROBLEM - mw8 NTP time on mw8 is CRITICAL: connect to address 51.195.236.221 port 5666: Connection refusedconnect to host 51.195.236.221 port 5666: Connection refused
[05:29:37] <icinga-miraheze>	 PROBLEM - ping4 on mw8 is CRITICAL: PING CRITICAL - Packet loss = 100%
[05:29:38] <icinga-miraheze>	 PROBLEM - ping6 on mw8 is CRITICAL: PING CRITICAL - Packet loss = 100%
[05:29:39] <icinga-miraheze>	 RECOVERY - ping6 on mw8 is OK: PING OK - Packet loss = 0%, RTA = 2.30 ms
[05:29:40] <icinga-miraheze>	 RECOVERY - ping4 on mw8 is OK: PING OK - Packet loss = 0%, RTA = 2.04 ms
[05:30:10] <icinga-miraheze>	 RECOVERY - mw8 SSH on mw8 is OK: SSH OK - OpenSSH_7.9p1 Debian-10+deb10u2 (protocol 2.0)
[05:38:20] <Not>	 [02miraheze/puppet] 07Reception123 pushed 031 commit to 03revert-1897-patch-79 [+0/-0/±1] 13https://git.io/JzJLv
[05:38:21] <Not>	 [02miraheze/puppet] 07Reception123 034816827 - Revert "mw8: update php (#1897)"
[05:38:23] <Not>	 [02puppet] 07Reception123 created branch 03revert-1897-patch-79 - 13https://git.io/vbiAS
[05:38:24] <Not>	 [02puppet] 07Reception123 opened pull request 03#1975: Revert "mw8: update php" - 13https://git.io/JzJLf
[05:38:30] <Not>	 [02puppet] 07Reception123 closed pull request 03#1975: Revert "mw8: update php" - 13https://git.io/JzJLf
[05:38:31] <Not>	 [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JzJLT
[05:38:33] <Not>	 [02miraheze/puppet] 07Reception123 031567541 - Revert "mw8: update php (#1897)" (#1975)
[05:38:34] <Not>	 [02puppet] 07Reception123 deleted branch 03revert-1897-patch-79 - 13https://git.io/vbiAS
[05:38:36] <Not>	 [02miraheze/puppet] 07Reception123 deleted branch 03revert-1897-patch-79
[05:49:08] <icinga-miraheze>	 RECOVERY - mw8 NTP time on mw8 is OK: NTP OK: Offset -0.009336829185 secs
[05:49:19] <icinga-miraheze>	 RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 1.17, 0.43, 0.16
[05:49:24] <icinga-miraheze>	 PROBLEM - mw8 Puppet on mw8 is UNKNOWN: NRPE: Unable to read output
[05:50:27] <icinga-miraheze>	 RECOVERY - mw8 PowerDNS Recursor on mw8 is OK: DNS OK: 0.218 seconds response time. miraheze.org returns 2001:41d0:800:170b::5,2001:41d0:801:2000::58af,51.38.69.175,54.38.211.199
[05:50:48] <icinga-miraheze>	 RECOVERY - mw8 Disk Space on mw8 is OK: DISK OK - free space: / 14786 MB (81% inode=92%);
[05:52:38] <icinga-miraheze>	 RECOVERY - mw8 php-fpm on mw8 is OK: PROCS OK: 3 processes with command name 'php-fpm7.3'
[05:53:21] <icinga-miraheze>	 PROBLEM - cp13 Stunnel Http for mw8 on cp13 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 338 bytes in 0.004 second response time
[05:53:45] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw8 on cp3 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 338 bytes in 0.751 second response time
[05:53:48] <icinga-miraheze>	 PROBLEM - cp14 Stunnel Http for mw8 on cp14 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 338 bytes in 0.003 second response time
[05:53:51] <icinga-miraheze>	 RECOVERY - mw8 HTTPS on mw8 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 557 bytes in 0.004 second response time
[05:54:27] <icinga-miraheze>	 PROBLEM - cp12 Stunnel Http for mw8 on cp12 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 338 bytes in 0.228 second response time
[05:57:20] <icinga-miraheze>	 RECOVERY - cp13 Stunnel Http for mw8 on cp13 is OK: HTTP OK: HTTP/1.1 200 OK - 15715 bytes in 0.090 second response time
[05:57:37] <icinga-miraheze>	 RECOVERY - cp14 Stunnel Http for mw8 on cp14 is OK: HTTP OK: HTTP/1.1 200 OK - 15707 bytes in 0.004 second response time
[05:57:45] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mw8 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15707 bytes in 0.997 second response time
[05:58:07] <icinga-miraheze>	 RECOVERY - mw8 Check Gluster Clients on mw8 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs'
[05:58:11] <icinga-miraheze>	 PROBLEM - mw8 MediaWiki Rendering on mw8 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 2560 bytes in 0.006 second response time
[05:58:21] <icinga-miraheze>	 RECOVERY - mw8 JobRunner Service on mw8 is OK: PROCS OK: 1 process with args 'redisJobRunnerService'
[05:58:26] <icinga-miraheze>	 RECOVERY - cp12 Stunnel Http for mw8 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15707 bytes in 0.330 second response time
[06:00:11] <icinga-miraheze>	 PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 771 bytes in 0.104 second response time
[06:00:25] <Reception123>	 .tell RhinosF1 ploticus and python-is-python3 are "unable to locate package"
[06:00:25] <MirahezeBot>	 Reception123: I'll pass that on when RhinosF1 is around.
[06:01:38] <Reception123>	 !log sudo -u www-data php /srv/mediawiki/w/maintenance/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki loginwiki && sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki=loginwiki on mw8
[06:01:44] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[06:02:09] <icinga-miraheze>	 RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 20005 bytes in 0.122 second response time
[06:02:45] <Reception123>	 !log repool mw8
[06:02:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[06:02:54] <icinga-miraheze>	 RECOVERY - cp12 Varnish Backends on cp12 is OK: All 9 backends are healthy
[06:03:18] <icinga-miraheze>	 RECOVERY - cp14 Varnish Backends on cp14 is OK: All 9 backends are healthy
[06:03:27] <icinga-miraheze>	 RECOVERY - cp3 Varnish Backends on cp3 is OK: All 9 backends are healthy
[06:03:47] <icinga-miraheze>	 RECOVERY - cp13 Varnish Backends on cp13 is OK: All 9 backends are healthy
[06:08:42] <Not>	 [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JzJGp
[06:08:43] <Not>	 [02miraheze/dns] 07Reception123 03a5855a7 - add jp to popira.info
[06:08:47] <Not>	 [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-1/±0] 13https://git.io/JzJGj
[06:08:49] <Not>	 [02miraheze/dns] 07Reception123 03c6bbb66 - Delete jp.popira.info
[06:11:40] <Reception123>	 RhinosF1: hi
[06:16:14] <RhinosF1>	 Hi
[06:16:14] <MirahezeBot>	 [tell] RhinosF1: 2021-09-17 - 07:00:25BST <Reception123> tell RhinosF1 ploticus and python-is-python3 are "unable to locate package"
[06:16:42] <RhinosF1>	 No idea about ploticus but the other shouldn't on buster
[06:16:44] <Reception123>	 and yes we're back on buster now and mw8 is back up, but not sure why those two packages don't ant to install
[06:16:59] <RhinosF1>	 And I believe I called that package an extremely stupid idea
[06:17:31] <Reception123>	 RhinosF1: which one? the python one?
[06:17:37] <RhinosF1>	 Yes
[06:17:54] <Reception123>	 why's that and then why was it installed?
[06:18:19] <RhinosF1>	 Because python2 != python3 and paladox did
[06:20:12] <Reception123>	 I don't think ploticus works with buster either
[06:20:46] <Reception123>	 nevermind, that's been there for a while
[06:21:48] <Reception123>	 RhinosF1: that really makes no sense as sudo apt-get install ploticus gives different results on mw9 and mw8 even though they're running the same software
[06:22:13] <RhinosF1>	 Reception123: what does it say
[06:22:24] <Reception123>	 E: Unable to locate package ploticus
[06:25:18] <RhinosF1>	 Does apt sources look right
[06:26:09] <majavah>	 ploticus is not on buster per https://packages.debian.org/bullseye/ploticus
[06:26:10] <MirahezeBot>	 [url] Debian -- Details of package ploticus in bullseye | packages.debian.org
[06:26:40] <majavah>	 v
[06:26:41] <majavah>	 https://github.com/miraheze/puppet/commit/4b76691f226ec7b358d3d2a67a01569739da4a90
[06:26:42] <MirahezeBot>	 [url] mediawiki: Install ploticus from debian rather than manually (#1891) · miraheze/puppet@4b76691 · GitHub | github.com
[06:26:49] <majavah>	 https://github.com/miraheze/puppet/commit/418c6774a63ad34b9b8545097c07023cb061cc34
[06:26:49] <MirahezeBot>	 [url] mediawiki: Remove unused ploticus package (#1902) · miraheze/puppet@418c677 · GitHub | github.com
[06:27:04] <Reception123>	 that's what I thought initially, but we have it on the other mw servers
[06:27:22] <Reception123>	 ah ok I see, so it used to be installed manually
[06:27:25] <RhinosF1>	 Yes
[06:28:29] <Not>	 [02puppet] 07Reception123 commented on pull request 03#1891: mediawiki: Install ploticus from debian rather than manually - 13https://git.io/JzJC7
[06:30:08] <RhinosF1>	 Reception123: just grab the deb
[06:30:21] <RhinosF1>	 And install from it
[06:30:35] <Reception123>	 yeah that's easier
[06:30:47] <Reception123>	 thanks majavah :)
[06:31:13] <RhinosF1>	 Reception123: copy the download link on https://github.com/miraheze/puppet/blob/483d7f1a746a2e79181f4f46a55207a16d208f78/modules/mediawiki/files/packages/ploticus/ploticus_2.42-3%2Bb4_amd64.deb
[06:31:14] <MirahezeBot>	 [url] puppet/ploticus_2.42-3+b4_amd64.deb at 483d7f1a746a2e79181f4f46a55207a16d208f78 · miraheze/puppet · GitHub | github.com
[06:31:31] <Reception123>	 yup, just did
[06:34:35] <Not>	 [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JzJWh
[06:34:37] <Not>	 [02miraheze/puppet] 07Reception123 03ebe0244 - remove python-is-python3 from packages
[06:41:20] <Not>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JzJ8Z
[06:41:21] <Not>	 [02miraheze/mw-config] 07Universal-Omega 0312a1a03 - Set `$wgCategoryCollation` to numeric for supermanwiki
[06:41:23] <Not>	 [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3
[06:41:40] <Not>	 [02mw-config] 07Universal-Omega opened pull request 03#4101: T8035: set `$wgCategoryCollation` to numeric for supermanwiki - 13https://git.io/JzJ88
[06:42:42] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[06:44:46] <Not>	 [02miraheze/ssl] 07Reception123 pushed 034 commits to 03master [+5/-0/±4] 13https://git.io/JzJ4T
[06:44:47] <Not>	 [02miraheze/ssl] 07Reception123 037a5ed4a - add jp.popira.info cert
[06:44:49] <Not>	 [02miraheze/ssl] 07Reception123 033c41bc9 - add central.ballmedia.org (+redirects)
[06:44:50] <Not>	 [02miraheze/ssl] 07Reception123 034083fff - add commons.ballmedia.org cert
[06:44:52] <Not>	 [02miraheze/ssl] 07Reception123 032c6e88d - add small.polandballwiki.com cert
[06:45:59] <miraheze-github>	 miraheze/ssl - Reception123 the build has errored.
[06:46:45] <Reception123>	 CosmicAlpha: heh well at least we know it works now, though I don't think a new line is completely necessary
[06:47:03] <Not>	 [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JzJ4K
[06:47:04] <Not>	 [02miraheze/ssl] 07Reception123 03bc4a745 - newline
[06:47:27] <RhinosF1>	 Reception123: it's called style
[06:47:40] <RhinosF1>	 When you do good code, you'll understand
[06:47:51] <RhinosF1>	 Compared to poor style
[06:49:29] <CosmicAlpha>	 Reception123: yeah that rule could probably be excluded if you really want. But yeah it is style, wouldn't effect anything other then that. I do prefer not to disable any unless absolutely necessary, but we can if you really want to.
[06:50:25] <RhinosF1>	 Let's not disable any
[06:51:09] <Reception123>	 that's fine yeah
[06:53:08] <CosmicAlpha>	 RhinosF1, Reception123: lol I just realised when I set up IRC notifications on the workflows, I added cancelled to it, but it will never run anyways if its cancelled since it would even cancel the IRC notifications as well...
[06:54:16] <Reception123>	 heh
[06:56:44] <Not>	 [02ssl] 07Universal-Omega opened pull request 03#430: Remove `cancelled` from IRC workflow notifications - 13https://git.io/JzJR0
[06:58:17] <CosmicAlpha>	 Reception123: ^
[07:00:53] <Not>	 [02ssl] 07Reception123 closed pull request 03#430: Remove `cancelled` from IRC workflow notifications - 13https://git.io/JzJR0
[07:00:54] <Not>	 [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JzJ0z
[07:00:56] <Not>	 [02miraheze/ssl] 07Universal-Omega 03e0f22d2 - Remove `cancelled` from IRC workflow notifications (#430)
[07:01:08] <Not>	 [02puppet] 07Reception123 closed pull request 03#1974: Add myself to `author_map` for the IRC logbot - 13https://git.io/JuAKu
[07:01:10] <Not>	 [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JzJ0w
[07:01:11] <Not>	 [02miraheze/puppet] 07Universal-Omega 03d6dffdd - Add myself to `author_map` for the IRC logbot (#1974)
[07:05:04] <icinga-miraheze>	 RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[07:05:14] <icinga-miraheze>	 RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures
[07:06:29] <icinga-miraheze>	 RECOVERY - mw12 Puppet on mw12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[07:06:35] <icinga-miraheze>	 RECOVERY - mw13 Puppet on mw13 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures
[07:06:47] <icinga-miraheze>	 RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[07:30:23] <Reception123>	 !log renamed polandballslessthan100kpopulationwiki to polandsmallswiki
[07:30:29] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[07:32:20] <Not>	 [02mw-config] 07Reception123 closed pull request 03#4101: T8035: set `$wgCategoryCollation` to numeric for supermanwiki - 13https://git.io/JzJ88
[07:32:21] <Not>	 [02miraheze/mw-config] 07Reception123 deleted branch 03Universal-Omega-patch-1
[07:32:23] <Not>	 [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JzJVN
[07:32:24] <Not>	 [02miraheze/mw-config] 07Universal-Omega 033087d92 - Set `$wgCategoryCollation` to numeric for supermanwiki (#4101)
[07:32:25] <Not>	 [02mw-config] 07Reception123 deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3
[07:33:29] <miraheze-github>	 miraheze/mw-config - Reception123 the build passed.
[07:33:54] <MirahezeLSBot>	 !log [@test3] starting deploy of {'config': True} to skip
[07:33:56] <MirahezeLSBot>	 !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 1s
[07:33:58] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[07:34:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[07:35:56] <MirahezeLSBot>	 !log [reception@mw9] sudo -u www-data php /srv/mediawiki/w/maintenance/updateCollation.php --wiki=supermanwiki (END - exit=0)
[07:35:59] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[07:56:58] <RhinosF1>	 Reception123: should we rollout my jquery PR
[07:57:14] <RhinosF1>	 I can test this evening but I did TestWiki too so we should know if it breaks
[07:58:25] <Reception123>	 RhinosF1: sure, feel free to test this evening and then roll it out if it works
[08:00:11] <RhinosF1>	 Reception123: I'm pushing to test3 and test then I might try @Agent's wikis
[08:00:23] <RhinosF1>	 It's default in .37 but I expect it to be a big change
[08:00:28] <Reception123>	 ok, sounds good
[08:02:06] <RhinosF1>	 So I don't want to just roll out
[08:02:10] <Not>	 [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JzJ18
[08:02:11] <Not>	 [02miraheze/mw-config] 07Reception123 037dbaa60 - set wgCategoryCollation to uca-fr for academiadesusarduwiki per T8036
[08:02:18] <RhinosF1>	 We have until .38 if needed to get wikis working
[08:02:35] <Not>	 [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JzJ1E
[08:02:37] <Not>	 [02miraheze/mw-config] 07Reception123 03362507d - fix
[08:02:46] <Reception123>	 RhinosF1: yeah that's enough time then
[08:02:54] <RhinosF1>	 Yeah
[08:02:59] <MirahezeLSBot>	 !log [@test3] starting deploy of {'config': True} to skip
[08:03:00] <MirahezeLSBot>	 !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s
[08:03:02] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[08:03:05] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[08:03:18] <miraheze-github>	 miraheze/mw-config - Reception123 the build has errored.
[08:03:33] <Reception123>	 uh that's not good
[08:03:35] <RhinosF1>	 I'd like to make it default false for .37 but maybe give wikis that need it time
[08:03:46] <miraheze-github>	 miraheze/mw-config - Reception123 the build passed.
[08:03:57] <RhinosF1>	 Reception123: it can't be that bad
[08:04:01] <Reception123>	 make up your mind miraheze-github
[08:04:01] <RhinosF1>	 As test3 didn't fail
[08:04:20] <Reception123>	 yeah I don't know why it said it errored
[08:04:47] <RhinosF1>	 Reception123: it was git failing
[08:04:56] <RhinosF1>	 To push the style
[08:04:59] <Reception123>	 ah
[08:05:01] <RhinosF1>	 Even though nothing to push
[08:05:17] <RhinosF1>	 But if test3 goes down after a config change it will log an error to SAL
[08:05:27] <RhinosF1>	 So you can see it's not that bad
[08:06:20] <Reception123>	 yeah, I would imagine that
[08:06:31] <MirahezeLSBot>	 !log [reception@mw9] sudo -u www-data php /srv/mediawiki/w/maintenance/updateCollation.php --wiki=academiadesusarduwiki (END - exit=0)
[08:06:33] <RhinosF1>	 'DEPLOY ABORTED: Canary check failed for localhost'
[08:06:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[08:06:42] <RhinosF1>	 Reception123: ^ that's the message that gets logged
[08:07:00] <RhinosF1>	 Even though on test3 you're only deploying to 1 server so it doesn't abort much
[08:07:02] <CosmicAlpha>	 Reception123: I need to fix something in the CI, the CI is trying to push changes to master, but is rejected because its trying to push to the PR and master simultaneously. Which it shouldn't be doing. I'll look into it tomorrow.
[08:07:12] <Reception123>	 ah
[08:07:20] <Reception123>	 and ok, thanks CosmicAlpha 
[08:08:27] <CosmicAlpha>	 No problem.
[09:17:34] <MH-Discord>	 <Joseph> I've gotten several Fatal Exemption of type "Error"  on FAMEPedia
[10:00:30] <RhinosF1>	 Can you give codes
[10:00:32] <RhinosF1>	 Reception123: ^
[10:00:48] <RhinosF1>	 If they're not consistent then Reception123 has done mw8 broke
[10:19:12] <MH-Discord>	 <Joseph> RhinosF1  MediaWiki internal error.  Original exception: [c62109aaa90b957912bb6521] 2021-09-17 09:16:22: Fatal exception of type "Error"  Exception caught inside exception handler.  Set $wgShowExceptionDetails = true; at the bottom of LocalSettings.php to show detailed debugging information.
[10:20:24] <MH-Discord>	 <Joseph> not just saving edits, it happens anytime.
[10:36:28] <Reception123>	 RhinosF1: I really hope not since I won't have access until the evening. I used MH debug and got no errors whatsoever on mw8
[10:37:37] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 54.38.211.199/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb
[10:38:09] <icinga-miraheze>	 PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.73, 6.46, 4.14
[10:39:35] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[10:40:07] <icinga-miraheze>	 RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.88, 5.58, 4.09
[10:45:36] <icinga-miraheze>	 PROBLEM - wiki.creaturathegame.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.creaturathegame.com' expires in 15 day(s) (Sun 03 Oct 2021 10:37:52 GMT +0000).
[11:50:05] <MH-Discord>	 <Redmin> !sre [ebf71e918e92db12284798e4] 2021-09-17 11:49:20: Fatal exception of type "Error" - Trying to view a non-existent page
[12:24:28] <MH-Discord>	 <Void> anyone with access simply needs to run composer on mw8
[12:24:53] <MH-Discord>	 <DarkMatterMan4500> You're up early, it seems.
[12:39:52] <MH-Discord>	 <Redmin> 🤦‍♂️
[12:40:25] <MH-Discord>	 <DarkMatterMan4500> Which one were you directing the facepalm at?
[12:54:49] <MH-Discord>	 <Redmin> Void's message and Sysadmins not being online?
[12:55:03] <MH-Discord>	 <DarkMatterMan4500> Oh.
[13:02:50] <MH-Discord>	 <Wiki: dreamverse.mira> What is reception's time zone?
[13:03:56] <Reception123>	 Unfortunately I don't currently have access to my PC in order to connect to the server
[13:04:17] <Reception123>	 paladox: ^
[13:08:02] <MH-Discord>	 <Wiki: dreamverse.mira> Oh, ok  Because my wiki gets these errors too bow
[13:08:08] <MH-Discord>	 <Wiki: dreamverse.mira> *now
[13:13:17] <Reception123>	 Sorry about that, when I tested out the server (mw8) I saw no errors.
[13:17:05] <paladox>	 hi
[13:17:07] <paladox>	 Reception123: ?
[13:17:39] <paladox>	 we'll need to resuse mw8 as a form of tester to test the memory again once the sessions are fixed
[13:19:28] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.71, 3.08, 2.78
[13:21:27] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.33, 2.86, 2.74
[13:24:16] <paladox>	 composer should be fixed on mw8 now
[13:35:58] <icinga-miraheze>	 RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[13:48:28] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.76, 3.35, 3.08
[13:52:31] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.21, 3.11, 3.07
[14:24:31] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.25, 3.71, 3.25
[14:28:31] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.09, 3.64, 3.35
[14:36:30] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.08, 3.63, 3.46
[14:37:44] <RhinosF1>	 looking at errors
[14:38:29] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.07, 3.37, 3.38
[14:48:15] <RhinosF1>	 oh paladox fixed
[15:03:42] <Not>	 [02puppet] 07RhinosF1 opened pull request 03#1976: mwdeploy-rsync: drop progress; add --update by default - 13https://git.io/JzTGG
[15:03:55] <RhinosF1>	 paladox: ^
[15:04:20] <Not>	 [02puppet] 07paladox closed pull request 03#1976: mwdeploy-rsync: drop progress; add --update by default - 13https://git.io/JzTGG
[15:04:21] <Not>	 [02miraheze/puppet] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JzTGu
[15:04:23] <Not>	 [02miraheze/puppet] 07RhinosF1 030127e4b - mwdeploy-rsync: drop progress; add --update by default
[15:04:24] <Not>	 [02miraheze/puppet] 07paladox 034906a26 - Merge pull request #1976 from RhinosF1/patch-92
[15:05:05] <RhinosF1>	 paladox: if you pull i'll test
[15:05:50] <paladox>	 done
[15:08:26] <MirahezeLSBot>	 !log [rhinos@test3] starting deploy of {'config': True, 'world': True, 'l10n': True, 'gitinfo': True} to skip
[15:08:31] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:09:51] <Not>	 [02puppet] 07RhinosF1 opened pull request 03#1977: mwdeploy: fix sp - 13https://git.io/JzTnZ
[15:09:55] <RhinosF1>	 !log failed
[15:09:58] <RhinosF1>	 paladox: ^
[15:09:59] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:10:15] <Not>	 [02puppet] 07paladox closed pull request 03#1977: mwdeploy: fix sp - 13https://git.io/JzTnZ
[15:10:17] <Not>	 [02miraheze/puppet] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JzTnl
[15:10:18] <Not>	 [02miraheze/puppet] 07RhinosF1 031356e21 - mwdeploy: fix sp
[15:10:20] <Not>	 [02miraheze/puppet] 07paladox 0321ac024 - Merge pull request #1977 from RhinosF1/patch-93
[15:14:33] <MirahezeLSBot>	 !log [rhinos@test3] starting deploy of {'config': True, 'world': True, 'l10n': True, 'gitinfo': True} to skip
[15:14:37] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:19:47] <RhinosF1>	 Need to make tea
[15:19:51] <RhinosF1>	 Will deal with later
[15:50:39] <icinga-miraheze>	 PROBLEM - robloxapi.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for robloxapi.wiki could not be found
[15:52:38] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.43, 3.16, 3.09
[15:54:38] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.93, 3.01, 3.04
[16:04:07] <icinga-miraheze>	 RECOVERY - robloxapi.wiki - reverse DNS on sslhost is OK: SSL OK - robloxapi.wiki reverse DNS resolves to cp13.miraheze.org - CNAME FLAT
[17:42:30] <icinga-miraheze>	 PROBLEM - robloxapi.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for robloxapi.wiki could not be found
[19:24:25] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.60, 3.56, 3.09
[19:26:23] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.11, 3.57, 3.14
[19:28:24] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.82, 3.29, 3.09
[20:07:52] <icinga-miraheze>	 PROBLEM - wiki.wysteria.ga - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.wysteria.ga could not be found
[20:09:21] <icinga-miraheze>	 PROBLEM - wiki.wysteria.ga - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:23:33] <icinga-miraheze>	 PROBLEM - ping6 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 158.66 ms
[23:25:34] <icinga-miraheze>	 RECOVERY - ping6 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 119.63 ms