[01:07:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[03:04:13] <icinga-miraheze>	 PROBLEM - mon181 Backups Grafana on mon181 is WARNING: FILE_AGE WARNING: /var/log/grafana-backup.log is 864231 seconds old and 92 bytes
[03:16:00] <icinga-miraheze>	 PROBLEM - ff8.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ff8.wiki' expires in 15 day(s) (Sun 07 Jul 2024 02:45:57 AM GMT +0000).
[03:16:13] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/317691144b6e...66c3a39a2b9b
[03:16:15] <botifico>	 [02ssl] 07WikiTideSSLBot 0366c3a39 - Bot: Update SSL cert for ff8.wiki
[03:26:21] <icinga-miraheze>	 PROBLEM - www.satepedia.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.satepedia.com' expires in 15 day(s) (Sun 07 Jul 2024 03:00:18 AM GMT +0000).
[03:26:33] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/66c3a39a2b9b...2d0cdb83b7d9
[03:26:34] <botifico>	 [02ssl] 07WikiTideSSLBot 032d0cdb8 - Bot: Update SSL cert for www.satepedia.com
[03:45:00] <icinga-miraheze>	 RECOVERY - ff8.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ff8.wiki' will expire on Thu 19 Sep 2024 02:16:08 AM GMT +0000.
[03:55:13] <icinga-miraheze>	 RECOVERY - www.satepedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.satepedia.com' will expire on Thu 19 Sep 2024 02:26:26 AM GMT +0000.
[05:07:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[07:37:25] <icinga-miraheze>	 [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[08:15:24] <botifico>	 [02IncidentReporting] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/braces-3.0.3 [+0/-0/±1] 13https://github.com/miraheze/IncidentReporting/commit/cd7a3dec033a
[08:15:25] <botifico>	 [02IncidentReporting] 07dependabot[bot] 03cd7a3de - Bump braces from 3.0.2 to 3.0.3
[08:15:27] <botifico>	 [02IncidentReporting] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/braces-3.0.3 - 13https://github.com/miraheze/IncidentReporting
[08:15:30] <botifico>	 [02IncidentReporting] 07dependabot[bot] opened pull request 03#89: Bump braces from 3.0.2 to 3.0.3 - 13https://github.com/miraheze/IncidentReporting/pull/89
[08:15:33] <botifico>	 [02IncidentReporting] 07dependabot[bot] labeled pull request 03#89: Bump braces from 3.0.2 to 3.0.3 - 13https://github.com/miraheze/IncidentReporting/pull/89
[08:15:36] <botifico>	 [02IncidentReporting] 07dependabot[bot] labeled pull request 03#89: Bump braces from 3.0.2 to 3.0.3 - 13https://github.com/miraheze/IncidentReporting/pull/89
[08:15:39] <botifico>	 [02IncidentReporting] 07coderabbitai[bot] commented on pull request 03#89: Bump braces from 3.0.2 to 3.0.3 - 13https://github.com/miraheze/IncidentReporting/pull/89#issuecomment-2182243627
[08:16:46] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[08:20:02] <miraheze-github>	 miraheze/IncidentReporting - dependabot[bot] the build passed.
[08:44:45] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[08:46:58] <botifico>	 [02IncidentReporting] 07RhinosF1 closed pull request 03#89: Bump braces from 3.0.2 to 3.0.3 - 13https://github.com/miraheze/IncidentReporting/pull/89
[08:46:59] <botifico>	 [02IncidentReporting] 07RhinosF1 deleted branch 03dependabot/npm_and_yarn/braces-3.0.3
[08:47:03] <botifico>	 [02IncidentReporting] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/IncidentReporting/compare/cb24f455c3dd...53975bcb9b22
[08:47:05] <botifico>	 [02IncidentReporting] 07dependabot[bot] 0353975bc - Bump braces from 3.0.2 to 3.0.3 (#89)
[08:47:07] <botifico>	 [02IncidentReporting] 07RhinosF1 deleted branch 03dependabot/npm_and_yarn/braces-3.0.3 - 13https://github.com/miraheze/IncidentReporting
[08:51:39] <miraheze-github>	 miraheze/IncidentReporting - RhinosF1 the build passed.
[09:33:38] <botifico>	 [02ssl] 07BlankEclair opened pull request 03#785: T12223: Redirect www.lebork.info.pl to lebork.info.pl - 13https://github.com/miraheze/ssl/pull/785
[09:33:44] <botifico>	 [02ssl] 07coderabbitai[bot] commented on pull request 03#785: T12223: Redirect www.lebork.info.pl to lebork.info.pl - 13https://github.com/miraheze/ssl/pull/785#issuecomment-2182387096
[09:36:19] <botifico>	 [02ssl] 07coderabbitai[bot] edited a comment on pull request 03#785: T12223: Redirect www.lebork.info.pl to lebork.info.pl - 13https://github.com/miraheze/ssl/pull/785#issuecomment-2182387096
[09:44:55] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[09:47:53] <botifico>	 [02puppet] 07redbluegreenhat pushed 031 commit to 03pywikibot-logrotate [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/78812d49e549
[09:47:56] <botifico>	 [02puppet] 07redbluegreenhat 0378812d4 - pywikibot: setup proper logging and logrotate
[09:47:57] <botifico>	 [02puppet] 07redbluegreenhat created branch 03pywikibot-logrotate - 13https://github.com/miraheze/puppet
[09:48:02] <botifico>	 [02puppet] 07redbluegreenhat opened pull request 03#3871: pywikibot: setup proper logging and logrotate - 13https://github.com/miraheze/puppet/pull/3871
[09:48:07] <botifico>	 [02puppet] 07coderabbitai[bot] commented on pull request 03#3871: pywikibot: setup proper logging and logrotate - 13https://github.com/miraheze/puppet/pull/3871#issuecomment-2182410288
[09:48:38] <botifico>	 [02puppet] 07github-actions[bot] synchronize pull request 03#3871: pywikibot: setup proper logging and logrotate - 13https://github.com/miraheze/puppet/pull/3871
[09:48:41] <botifico>	 [02puppet] 07github-actions[bot] pushed 031 commit to 03pywikibot-logrotate [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/78812d49e549...ac256b15814f
[09:48:44] <botifico>	 [02puppet] 07github-actions 03ac256b1 - CI: lint puppet code to standards
[09:55:43] <botifico>	 [02puppet] 07redbluegreenhat pushed 031 commit to 03pywikibot-logrotate [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ac256b15814f...d44470f999e6
[09:55:46] <botifico>	 [02puppet] 07redbluegreenhat 03d44470f - fix name
[09:55:49] <botifico>	 [02puppet] 07redbluegreenhat synchronize pull request 03#3871: pywikibot: setup proper logging and logrotate - 13https://github.com/miraheze/puppet/pull/3871
[10:11:08] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/2d0cdb83b7d9...456f881a9073
[10:11:09] <botifico>	 [02ssl] 07WikiTideSSLBot 03456f881 - Bot: Add SSL cert for wiki.chevrine.com
[10:12:10] <botifico>	 [02dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/3e522254a576...32acc1615ba1
[10:12:11] <botifico>	 [02dns] 07Reception123 0332acc16 - Create maxcapacity.wiki
[10:12:28] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/456f881a9073...33c29f432b16
[10:12:31] <botifico>	 [02ssl] 07WikiTideSSLBot 0333c29f4 - Bot: Add SSL cert for wiki.eastsidegamers.no
[10:13:44] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/33c29f432b16...bc4613f0b90e
[10:13:46] <botifico>	 [02ssl] 07WikiTideSSLBot 03bc4613f - Bot: Add SSL cert for wiki.minigamemadness.net
[10:15:14] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[10:21:29] <botifico>	 [02puppet] 07redbluegreenhat pushed 031 commit to 03pywikibot-logrotate [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d44470f999e6...45c27a3fcb75
[10:21:32] <botifico>	 [02puppet] 07redbluegreenhat 0345c27a3 - use copytruncate
[10:21:34] <botifico>	 [02puppet] 07redbluegreenhat synchronize pull request 03#3871: pywikibot: setup proper logging and logrotate - 13https://github.com/miraheze/puppet/pull/3871
[10:25:24] <botifico>	 [02puppet] 07redbluegreenhat closed pull request 03#3871: pywikibot: setup proper logging and logrotate - 13https://github.com/miraheze/puppet/pull/3871
[10:25:27] <botifico>	 [02puppet] 07redbluegreenhat deleted branch 03pywikibot-logrotate
[10:25:30] <botifico>	 [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4566ba74897f...773238518fe1
[10:25:33] <botifico>	 [02puppet] 07redbluegreenhat 037732385 - pywikibot: setup proper logging and logrotate (#3871)
[10:25:35] <botifico>	 [02puppet] 07redbluegreenhat deleted branch 03pywikibot-logrotate - 13https://github.com/miraheze/puppet
[10:31:24] <icinga-miraheze>	 PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query.
[10:32:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[10:33:54] <botifico>	 [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/773238518fe1...2f001ae25211
[10:33:57] <botifico>	 [02puppet] 07redbluegreenhat 032f001ae - pywikibot: add missing comma
[10:35:10] <icinga-miraheze>	 PROBLEM - bots171 Puppet on bots171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[10:41:10] <icinga-miraheze>	 RECOVERY - bots171 Puppet on bots171 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures
[10:46:47] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[10:49:28] <botifico>	 [02puppet] 07redbluegreenhat pushed 031 commit to 03pywikibot-pil [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/20c59fd8c0ab
[10:49:30] <botifico>	 [02puppet] 07redbluegreenhat 0320c59fd - pywikibot: ensure python3-pil is installed instead
[10:49:32] <botifico>	 [02puppet] 07redbluegreenhat created branch 03pywikibot-pil - 13https://github.com/miraheze/puppet
[10:49:48] <botifico>	 [02puppet] 07redbluegreenhat opened pull request 03#3872: pywikibot: ensure python3-pil is installed instead - 13https://github.com/miraheze/puppet/pull/3872
[10:49:56] <botifico>	 [02puppet] 07coderabbitai[bot] commented on pull request 03#3872: pywikibot: ensure python3-pil is installed instead - 13https://github.com/miraheze/puppet/pull/3872#issuecomment-2182511339
[10:50:39] <botifico>	 [02puppet] 07redbluegreenhat closed pull request 03#3872: pywikibot: ensure python3-pil is installed instead - 13https://github.com/miraheze/puppet/pull/3872
[10:50:40] <botifico>	 [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2f001ae25211...9e0a684bd8aa
[10:50:42] <botifico>	 [02puppet] 07redbluegreenhat 039e0a684 - pywikibot: ensure python3-pil is installed instead (#3872)
[10:50:43] <botifico>	 [02puppet] 07redbluegreenhat deleted branch 03pywikibot-pil
[10:50:45] <botifico>	 [02puppet] 07redbluegreenhat deleted branch 03pywikibot-pil - 13https://github.com/miraheze/puppet
[11:00:50] <icinga-miraheze>	 RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp37.wikitide.net - CNAME OK
[11:15:34] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures
[11:42:47] <MirahezeLSBot>	 !log [alex@test151] starting deploy of {'versions': '1.42', 'upgrade_skins': 'Mirage'} to test151
[11:42:48] <MirahezeLSBot>	 !log [alex@test151] finished deploy of {'versions': '1.42', 'upgrade_skins': 'Mirage'} to test151 - SUCCESS in 0s
[11:42:52] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:42:56] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:47:25] <icinga-miraheze>	 [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[12:02:05] <icinga-miraheze>	 PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query.
[12:07:49] <MirahezeLSBot>	 !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/purgeExpiredBlocks.php --wiki=holidayswiki (END - exit=0)
[12:07:53] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:31:42] <icinga-miraheze>	 RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK
[13:13:03] <MirahezeLSBot>	 !log [alex@test151] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/graph.json /srv/mediawiki/1.42/extensions/ManageWiki/maintenance/toggleExtension.php --disable graph
[13:13:08] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:16:02] <Orange_Star>	 This extension does have config variables, but these seem to not have been configured in either LocalSettings.php or ManageWikiSettings.php
[13:23:28] <MirahezeLSBot>	 !log [alex@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/graph.json /srv/mediawiki/1.41/extensions/ManageWiki/maintenance/toggleExtension.php --disable graph (interrupted)
[13:23:32] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:30:27] <MirahezeLSBot>	 !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/ManageWiki/maintenance/toggleExtension.php --wiki=snowballaspediawiki --disable graph (END - exit=0)
[13:30:32] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:30:35] <MirahezeLSBot>	 !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/ManageWiki/maintenance/toggleExtension.php --wiki=snowballaspediawiki --disable graph (END - exit=0)
[13:30:39] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:31:08] <Orange_Star>	 lovely
[14:00:13] <botifico>	 [02ManageWiki] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/commit/546ce467cd90
[14:00:16] <botifico>	 [02ManageWiki] 07RhinosF1 03546ce46 - fix logic error when using --disable
[14:00:18] <botifico>	 [02ManageWiki] 07RhinosF1 created branch 03RhinosF1-patch-1 - 13https://github.com/miraheze/ManageWiki
[14:00:20] <botifico>	 [02ManageWiki] 07RhinosF1 opened pull request 03#470: fix logic error when using --disable - 13https://github.com/miraheze/ManageWiki/pull/470
[14:00:23] <botifico>	 [02ManageWiki] 07coderabbitai[bot] commented on pull request 03#470: fix logic error when using --disable - 13https://github.com/miraheze/ManageWiki/pull/470#issuecomment-2182813302
[14:00:32] <botifico>	 [02ManageWiki] 07RhinosF1 commented on pull request 03#470: fix logic error when using --disable - 13https://github.com/miraheze/ManageWiki/pull/470#issuecomment-2182813533
[14:00:36] <botifico>	 [02ManageWiki] 07coderabbitai[bot] edited a comment on pull request 03#470: fix logic error when using --disable - 13https://github.com/miraheze/ManageWiki/pull/470#issuecomment-2182813302
[14:00:39] <botifico>	 [02ManageWiki] 07coderabbitai[bot] commented on pull request 03#470: fix logic error when using --disable - 13https://github.com/miraheze/ManageWiki/pull/470#issuecomment-2182813658
[14:00:42] <botifico>	 [02ManageWiki] 07RhinosF1 assigned pull request 03#470: fix logic error when using --disable - 13https://github.com/miraheze/ManageWiki/pull/470
[14:01:40] <botifico>	 [02ManageWiki] 07github-actions[bot] pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/546ce467cd90...14788f661037
[14:01:42] <botifico>	 [02ManageWiki] 07github-actions 0314788f6 - CI: lint code to MediaWiki standards
[14:01:45] <botifico>	 [02ManageWiki] 07github-actions[bot] synchronize pull request 03#470: fix logic error when using --disable - 13https://github.com/miraheze/ManageWiki/pull/470
[14:02:17] <botifico>	 [02ManageWiki] 07coderabbitai[bot] edited a comment on pull request 03#470: fix logic error when using --disable - 13https://github.com/miraheze/ManageWiki/pull/470#issuecomment-2182813302
[14:02:19] <botifico>	 [02ManageWiki] 07coderabbitai[bot] edited a comment on pull request 03#470: fix logic error when using --disable - 13https://github.com/miraheze/ManageWiki/pull/470#issuecomment-2182813302
[14:04:10] <botifico>	 [02ManageWiki] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/c1688c3d70dd...18909f0a13ae
[14:04:12] <botifico>	 [02ManageWiki] 07RhinosF1 0318909f0 - fix logic error when using --disable (#470)
[14:04:15] <botifico>	 [02ManageWiki] 07redbluegreenhat closed pull request 03#470: fix logic error when using --disable - 13https://github.com/miraheze/ManageWiki/pull/470
[14:04:16] <botifico>	 [02ManageWiki] 07RhinosF1 deleted branch 03RhinosF1-patch-1
[14:04:18] <botifico>	 [02ManageWiki] 07RhinosF1 deleted branch 03RhinosF1-patch-1 - 13https://github.com/miraheze/ManageWiki
[14:04:46] <miraheze-github>	 miraheze/ManageWiki - RhinosF1 the build passed.
[14:05:14] <MirahezeLSBot>	 !log [alex@test151] starting deploy of {'versions': '1.42', 'upgrade_extensions': 'ManageWiki'} to test151
[14:05:15] <MirahezeLSBot>	 !log [alex@test151] finished deploy of {'versions': '1.42', 'upgrade_extensions': 'ManageWiki'} to test151 - SUCCESS in 1s
[14:05:18] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:05:22] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:05:46] <MirahezeLSBot>	 !log [alex@test151] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/extensions/ManageWiki/maintenance/toggleExtension.php --wiki=testwikibeta --disable graph (END - exit=0)
[14:05:50] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:06:35] <MirahezeLSBot>	 !log [alex@test151] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/extensions/ManageWiki/maintenance/toggleExtension.php --wiki=testwikibeta graph (END - exit=0)
[14:06:40] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:06:47] <MirahezeLSBot>	 !log [alex@test151] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/extensions/ManageWiki/maintenance/toggleExtension.php --wiki=testwikibeta --disable graph (END - exit=0)
[14:06:52] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:06:58] <MirahezeLSBot>	 !log [alex@test151] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/extensions/ManageWiki/maintenance/toggleExtension.php --wiki=testwikibeta --disable graph (END - exit=0)
[14:07:02] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:08:40] <miraheze-github>	 miraheze/ManageWiki - redbluegreenhat the build passed.
[14:10:02] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[14:11:18] <MirahezeLSBot>	 !log [alex@test151] starting deploy of {'files': 'extensions/ManageWiki/maintenance/toggleExtension.php'} to test151
[14:11:22] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:11:33] <MirahezeLSBot>	 !log [alex@test151] starting deploy of {'files': '1.42/extensions/ManageWiki/maintenance/toggleExtension.php'} to test151
[14:11:34] <MirahezeLSBot>	 !log [alex@test151] finished deploy of {'files': '1.42/extensions/ManageWiki/maintenance/toggleExtension.php'} to test151 - SUCCESS in 0s
[14:11:37] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:11:42] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:11:57] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0001830458641 secs
[14:12:05] <MirahezeLSBot>	 !log [alex@test151] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/extensions/ManageWiki/maintenance/toggleExtension.php --wiki=testwikibeta --disable graph (END - exit=0)
[14:12:09] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:12:11] <MirahezeLSBot>	 !log [alex@test151] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/extensions/ManageWiki/maintenance/toggleExtension.php --wiki=testwikibeta --disable graph (END - exit=0)
[14:12:16] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:13:10] <MirahezeLSBot>	 !log [alex@test151] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/extensions/ManageWiki/maintenance/toggleExtension.php --wiki=testwikibeta --disable graph (END - exit=0)
[14:13:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:15:12] <botifico>	 [02ManageWiki] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/18909f0a13ae...5c3202a3eeb1
[14:15:14] <botifico>	 [02ManageWiki] 07redbluegreenhat 035c3202a - fix logic for enabling an extension
[14:16:14] <MirahezeLSBot>	 !log [alex@test151] starting deploy of {'force_upgrade': True, 'versions': '1.42', 'upgrade_extensions': 'ManageWiki'} to test151
[14:16:15] <MirahezeLSBot>	 !log [alex@test151] finished deploy of {'force_upgrade': True, 'versions': '1.42', 'upgrade_extensions': 'ManageWiki'} to test151 - SUCCESS in 1s
[14:16:19] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:16:23] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:17:10] <MirahezeLSBot>	 !log [alex@test151] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/extensions/ManageWiki/maintenance/toggleExtension.php --wiki=testwikibeta --disable graph (END - exit=0)
[14:17:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:17:30] <MirahezeLSBot>	 !log [alex@test151] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/extensions/ManageWiki/maintenance/toggleExtension.php --wiki=testwikibeta graph (END - exit=0)
[14:17:34] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:17:36] <MirahezeLSBot>	 !log [alex@test151] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/extensions/ManageWiki/maintenance/toggleExtension.php --wiki=testwikibeta --disable graph (END - exit=0)
[14:17:40] <MirahezeLSBot>	 !log [alex@test151] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/extensions/ManageWiki/maintenance/toggleExtension.php --wiki=testwikibeta --disable graph (END - exit=0)
[14:17:41] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:17:45] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:19:47] <miraheze-github>	 miraheze/ManageWiki - redbluegreenhat the build passed.
[16:07:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[16:22:25] <icinga-miraheze>	 [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:16:48] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_dns]
[17:17:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:26:02] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[17:27:57] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0002329051495 secs
[17:45:16] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[18:15:04] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 120.82 ms
[18:15:50] <MirahezeLSBot>	 !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/resetUserEmail.php --wiki=metawiki Miraheze_maintenance_script sre@wikitide.org (END - exit=0)
[18:15:55] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:18:27] <MirahezeRelay>	 <rhinosf1, replying to MirahezeLSBot> Isn't that a system account?
[18:18:31] <MirahezeRelay>	 <rhinosf1> Rather than a human
[18:21:08] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 121.16 ms
[18:26:33] <MirahezeRelay>	 <bluemoon0332> it's an actual account
[18:44:46] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 120.78 ms
[18:44:46] <icinga-miraheze>	 PROBLEM - lostidols.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'lostidols.wiki' expires in 15 day(s) (Sun 07 Jul 2024 06:42:13 PM GMT +0000).
[18:48:48] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 119.94 ms
[19:00:14] <icinga-miraheze>	 PROBLEM - www.lostidols.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'lostidols.wiki' expires in 15 day(s) (Sun 07 Jul 2024 06:42:13 PM GMT +0000).
[20:15:01] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.38, 20.00, 16.30
[20:21:00] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.51, 20.35, 17.68
[20:29:00] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.52, 22.93, 19.79
[20:31:00] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.24, 22.21, 19.86
[20:35:01] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 14.40, 19.39, 19.40
[21:17:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[21:38:08] <MirahezeLSBot>	 !log [void@mwtask181] ran cleanupTitles on thewhiteroomwiki
[21:38:13] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:38:22] <MirahezeLSBot>	 !log [void@mwtask181] ran cleanupTitles on voidstrangerwiki
[21:38:26] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:29:36] <MirahezeLSBot>	 !log [alex@puppet181] Upgraded packages php8.2-zip, php8.2-xml, php8.2-readline, php8.2-opcache, php8.2-mysql, php8.2-mbstring, php8.2-ldap, php8.2-intl, php8.2-gmp, php8.2-gd, php8.2-dba, php8.2-curl, php8.2-cli, php8.2-bz2, php8.2-bcmath, and php8.2-common on jobchron171
[22:29:40] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:29:51] <icinga-miraheze>	 RECOVERY - jobchron171 APT on jobchron171 is OK: APT OK: 26 packages available for upgrade (0 critical updates).
[22:32:10] <MirahezeLSBot>	 !log [alex@puppet181] Upgraded packages php8.2-xml, php8.2-readline, php8.2-opcache, php8.2-mysql, php8.2-mbstring, php8.2-ldap, php8.2-intl, php8.2-gmp, php8.2-gd, php8.2-fpm, php8.2-curl, php8.2-cli, and php8.2-common on matomo151
[22:32:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:33:05] <MirahezeLSBot>	 !log [alex@puppet181] Upgraded packages php8.2-opcache, php8.2-readline, php8.2-cli, php8.2-xml, php8.2-soap, php8.2-pgsql, php8.2-mysql, php8.2-mbstring, php8.2-ldap, php8.2-gmp, php8.2-gd, php8.2-fpm, php8.2-curl, php8.2-intl, php8.2-common, and php8.2 on mon181
[22:33:09] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:33:32] <icinga-miraheze>	 RECOVERY - mon181 APT on mon181 is OK: APT OK: 31 packages available for upgrade (0 critical updates).
[22:33:37] <icinga-miraheze>	 RECOVERY - matomo151 APT on matomo151 is OK: APT OK: 26 packages available for upgrade (0 critical updates).
[22:33:54] <MirahezeLSBot>	 !log [alex@puppet181] Upgraded packages php8.2-zip, php8.2-xml, php8.2-readline, php8.2-opcache, php8.2-mysql, php8.2-mbstring, php8.2-ldap, php8.2-intl, php8.2-gmp, php8.2-gd, php8.2-cli, php8.2-fpm, php8.2-dba, php8.2-curl, php8.2-bz2, php8.2-bcmath, and php8.2-common on mw151
[22:33:57] <icinga-miraheze>	 RECOVERY - mw151 APT on mw151 is OK: APT OK: 26 packages available for upgrade (0 critical updates).
[22:33:58] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:34:43] <MirahezeLSBot>	 !log [alex@puppet181] Upgraded packages php8.2-zip, php8.2-xml, php8.2-readline, php8.2-opcache, php8.2-mysql, php8.2-mbstring, php8.2-ldap, php8.2-intl, php8.2-gmp, php8.2-gd, php8.2-cli, php8.2-fpm, php8.2-dba, php8.2-curl, php8.2-bz2, php8.2-bcmath, and php8.2-common on mwtask171
[22:34:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:34:53] <icinga-miraheze>	 RECOVERY - mwtask171 APT on mwtask171 is OK: APT OK: 1 packages available for upgrade (0 critical updates).
[22:35:27] <MirahezeLSBot>	 !log [alex@puppet181] Upgraded packages php8.2-zip, php8.2-xml, php8.2-readline, php8.2-opcache, php8.2-mysql, php8.2-mbstring, php8.2-ldap, php8.2-intl, php8.2-gmp, php8.2-gd, php8.2-cli, php8.2-fpm, php8.2-dba, php8.2-curl, php8.2-bz2, php8.2-bcmath, and php8.2-common on mw162
[22:35:32] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:36:14] <icinga-miraheze>	 RECOVERY - mw152 APT on mw152 is OK: APT OK: 26 packages available for upgrade (0 critical updates).
[22:36:18] <icinga-miraheze>	 RECOVERY - mw162 APT on mw162 is OK: APT OK: 26 packages available for upgrade (0 critical updates).
[22:36:27] <MirahezeLSBot>	 !log [alex@puppet181] Upgraded packages php8.2-zip, php8.2-xml, php8.2-readline, php8.2-opcache, php8.2-mysql, php8.2-mbstring, php8.2-ldap, php8.2-intl, php8.2-gmp, php8.2-gd, php8.2-cli, php8.2-fpm, php8.2-dba, php8.2-curl, php8.2-bz2, php8.2-bcmath, and php8.2-common on mw152
[22:36:31] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:37:14] <MirahezeLSBot>	 !log [alex@puppet181] Upgraded packages php8.2-zip, php8.2-xml, php8.2-readline, php8.2-opcache, php8.2-mysql, php8.2-mbstring, php8.2-ldap, php8.2-intl, php8.2-gmp, php8.2-gd, php8.2-cli, php8.2-fpm, php8.2-dba, php8.2-curl, php8.2-bz2, php8.2-bcmath, and php8.2-common on mwtask181
[22:37:19] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:37:35] <icinga-miraheze>	 RECOVERY - mw171 APT on mw171 is OK: APT OK: 26 packages available for upgrade (0 critical updates).
[22:37:40] <icinga-miraheze>	 RECOVERY - mwtask181 APT on mwtask181 is OK: APT OK: 1 packages available for upgrade (0 critical updates).
[22:38:00] <MirahezeLSBot>	 !log [alex@puppet181] Upgraded packages php8.2-zip, php8.2-xml, php8.2-readline, php8.2-opcache, php8.2-mysql, php8.2-mbstring, php8.2-ldap, php8.2-intl, php8.2-gmp, php8.2-gd, php8.2-cli, php8.2-fpm, php8.2-dba, php8.2-curl, php8.2-bz2, php8.2-bcmath, and php8.2-common on mw171
[22:38:05] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:38:46] <MirahezeLSBot>	 !log [alex@puppet181] Upgraded packages php8.2-zip, php8.2-xml, php8.2-readline, php8.2-opcache, php8.2-mysql, php8.2-mbstring, php8.2-ldap, php8.2-intl, php8.2-gmp, php8.2-gd, php8.2-cli, php8.2-fpm, php8.2-dba, php8.2-curl, php8.2-bz2, php8.2-bcmath, and php8.2-common on mw172
[22:38:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:38:55] <icinga-miraheze>	 RECOVERY - mw172 APT on mw172 is OK: APT OK: 26 packages available for upgrade (0 critical updates).
[22:39:41] <MirahezeLSBot>	 !log [alex@puppet181] Upgraded packages php8.2-zip, php8.2-xml, php8.2-readline, php8.2-opcache, php8.2-mysql, php8.2-mbstring, php8.2-ldap, php8.2-intl, php8.2-gmp, php8.2-gd, php8.2-cli, php8.2-fpm, php8.2-dba, php8.2-curl, php8.2-bz2, php8.2-bcmath, and php8.2-common on mw181
[22:39:45] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:40:28] <icinga-miraheze>	 RECOVERY - mw181 APT on mw181 is OK: APT OK: 26 packages available for upgrade (0 critical updates).
[22:40:33] <MirahezeLSBot>	 !log [alex@puppet181] Upgraded packages php8.2-zip, php8.2-xml, php8.2-readline, php8.2-opcache, php8.2-mysql, php8.2-mbstring, php8.2-ldap, php8.2-intl, php8.2-gmp, php8.2-gd, php8.2-cli, php8.2-fpm, php8.2-dba, php8.2-curl, php8.2-bz2, php8.2-bcmath, and php8.2-common on mw161
[22:40:34] <icinga-miraheze>	 RECOVERY - mw161 APT on mw161 is OK: APT OK: 26 packages available for upgrade (0 critical updates).
[22:40:38] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:41:10] <MirahezeLSBot>	 !log [alex@puppet181] Upgraded packages php8.2-zip, php8.2-xml, php8.2-readline, php8.2-opcache, php8.2-mysql, php8.2-mbstring, php8.2-ldap, php8.2-intl, php8.2-gmp, php8.2-gd, php8.2-fpm, php8.2-curl, php8.2-cli, and php8.2-common on phorge171
[22:41:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:41:19] <icinga-miraheze>	 RECOVERY - phorge171 APT on phorge171 is OK: APT OK: 28 packages available for upgrade (0 critical updates).
[22:42:19] <MirahezeLSBot>	 !log [alex@puppet181] Upgraded packages php8.2-zip, php8.2-xml, php8.2-readline, php8.2-opcache, php8.2-mysql, php8.2-mbstring, php8.2-ldap, php8.2-intl, php8.2-gmp, php8.2-gd, php8.2-cli, php8.2-fpm, php8.2-dba, php8.2-curl, php8.2-bz2, php8.2-bcmath, and php8.2-common on mw182
[22:42:23] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:42:37] <icinga-miraheze>	 RECOVERY - mw182 APT on mw182 is OK: APT OK: 26 packages available for upgrade (0 critical updates).
[22:42:54] <MirahezeLSBot>	 !log [alex@puppet181] Upgraded packages php8.2-xml, php8.2-readline, php8.2-opcache, php8.2-mysql, php8.2-mbstring, php8.2-ldap, php8.2-intl, php8.2-gmp, php8.2-gd, php8.2-fpm, php8.2-curl, php8.2-cli, and php8.2-common on reports171
[22:42:58] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:43:48] <icinga-miraheze>	 RECOVERY - reports171 APT on reports171 is OK: APT OK: 28 packages available for upgrade (0 critical updates).
[22:44:22] <MirahezeLSBot>	 !log [alex@puppet181] Upgraded packages php8.2-zip, php8.2-xml, php8.2-readline, php8.2-opcache, php8.2-mysql, php8.2-mbstring, php8.2-ldap, php8.2-intl, php8.2-gmp, php8.2-gd, php8.2-cli, php8.2-fpm, php8.2-dba, php8.2-curl, php8.2-bz2, php8.2-bcmath, and php8.2-common on test151
[22:44:26] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:45:03] <icinga-miraheze>	 RECOVERY - test151 APT on test151 is OK: APT OK: 26 packages available for upgrade (0 critical updates).
[22:52:57] <MirahezeRelay>	 <bluemoon0332> !log rebooted test151
[22:53:01] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:53:42] <MirahezeRelay>	 <bluemoon0332> didn't mean to reboot, did an oopsie with systemctl reboot instead of systemctl restart while trying to restart php-fpm