[00:00:46] <icinga-miraheze>	 PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN
[00:01:44] <icinga-miraheze>	 PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 50%
[00:02:46] <icinga-miraheze>	 PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is CRITICAL: CRITICAL - NGINX Error Rate is 100%
[00:03:44] <icinga-miraheze>	 PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 67%
[00:05:44] <icinga-miraheze>	 RECOVERY - cp36 Disk Space on cp36 is OK: DISK OK - free space: / 20435MiB (23% inode=98%);
[00:06:32] <icinga-miraheze>	 RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 20473MiB (23% inode=98%);
[01:03:43] <icinga-miraheze>	 PROBLEM - storytime.jdstroy.cf - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for storytime.jdstroy.cf could not be found
[01:32:26] <icinga-miraheze>	 RECOVERY - storytime.jdstroy.cf - reverse DNS on sslhost is OK: SSL OK - storytime.jdstroy.cf reverse DNS resolves to cp36.wikitide.net - CNAME OK
[02:01:04] <botifico>	 [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/415496f2d901...5f26e7a8f72e
[02:01:05] <botifico>	 [02miraheze/mw-config] 07AgentIsai 035f26e7a - Apply global blocks on Meta
[02:01:32] <MirahezeLSBot>	 !log [agent@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[02:01:35] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:01:42] <MirahezeLSBot>	 !log [agent@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s
[02:01:44] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:02:00] <miraheze-github>	 miraheze/mw-config - AgentIsai the build passed.
[02:07:34] <botifico>	 [02miraheze/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/cb290c7063b7...cc95307ccd7f
[02:07:36] <botifico>	 [02miraheze/puppet] 07AgentIsai 03cc95307 - T11876, T11877: Remove access for Owen and Paladox
[02:23:09] <MirahezeLSBot>	 !log [@test151] starting deploy of {'config': True} to test151
[02:23:10] <MirahezeLSBot>	 !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s
[02:23:11] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:23:13] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:28:09] <botifico>	 [02miraheze/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/cc95307ccd7f...5b809bdfe87d
[02:28:11] <botifico>	 [02miraheze/puppet] 07AgentIsai 035b809bd - icinga2: Remove owen and paladox
[04:44:51] <icinga-miraheze>	 PROBLEM - wiki.altilunium.my.id - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.altilunium.my.id' expires in 15 day(s) (Thu 07 Mar 2024 04:36:53 AM GMT +0000).
[04:45:05] <botifico>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/5dce45f9c8b9...aac3be2e64f7
[04:45:07] <botifico>	 [02miraheze/ssl] 07MirahezeSSLBot 03aac3be2 - Bot: Update SSL cert for wiki.altilunium.my.id
[04:45:54] <miraheze-github>	 miraheze/ssl - MirahezeSSLBot the build has errored.
[05:00:39] <MirahezeLSBot>	 !log [macfan@test151] starting deploy of {'versions': '1.41', 'upgrade_extensions': 'DiscordNotifications'} to test151
[05:00:40] <MirahezeLSBot>	 !log [macfan@test151] finished deploy of {'versions': '1.41', 'upgrade_extensions': 'DiscordNotifications'} to test151 - SUCCESS in 0s
[05:00:42] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[05:00:44] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[05:00:54] <MirahezeLSBot>	 !log [macfan@test151] starting deploy of {'force_upgrade': True, 'versions': '1.41', 'upgrade_extensions': 'DiscordNotifications'} to test151
[05:00:55] <MirahezeLSBot>	 !log [macfan@test151] finished deploy of {'force_upgrade': True, 'versions': '1.41', 'upgrade_extensions': 'DiscordNotifications'} to test151 - SUCCESS in 0s
[05:00:57] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[05:01:00] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[05:10:59] <botifico>	 [02miraheze/mw-config] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/5f26e7a8f72e...b07b11f48a02
[05:11:02] <botifico>	 [02miraheze/mw-config] 07MacFan4000 03b07b11f - rm paladox and Owen access to staffwiki
[05:11:52] <miraheze-github>	 miraheze/mw-config - MacFan4000 the build passed.
[05:13:05] <MirahezeLSBot>	 !log [macfan@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[05:13:08] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[05:13:15] <MirahezeLSBot>	 !log [macfan@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s
[05:13:18] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[05:23:47] <MirahezeLSBot>	 !log [@test151] starting deploy of {'config': True} to test151
[05:23:48] <MirahezeLSBot>	 !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s
[05:23:50] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[05:23:52] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[05:34:06] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[05:44:32] <icinga-miraheze>	 RECOVERY - wiki.altilunium.my.id - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.altilunium.my.id' will expire on Mon 20 May 2024 03:45:00 AM GMT +0000.
[05:50:23] <icinga-miraheze>	 PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 148.36 ms
[05:52:25] <icinga-miraheze>	 RECOVERY - ping6 on cp28 is OK: PING OK - Packet loss = 0%, RTA = 152.90 ms
[06:14:07] <botifico>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/aac3be2e64f7...9390c46f83bb
[06:14:08] <botifico>	 [02miraheze/ssl] 07MirahezeSSLBot 039390c46 - Bot: Add SSL cert for eros.locksteel.net
[10:02:14] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK
[10:05:46] <botifico>	 [02mw-config] 07OAuthority closed pull request 03#5480: Add Snap! Blocks to ManageWikiExtensions.php - 13https://github.com/miraheze/mw-config/pull/5480
[10:05:49] <botifico>	 [02miraheze/mw-config] 07OAuthority pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/b07b11f48a02...3cadfce6dac6
[10:05:50] <botifico>	 [02miraheze/mw-config] 07pixDeVl 033cadfce - Add Snap! Blocks to ManageWikiExtensions.php (#5480)
[10:06:40] <miraheze-github>	 miraheze/mw-config - OAuthority the build passed.
[10:08:07] <MirahezeLSBot>	 !log [oa@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/refreshLinks.php --wiki=321nailswiki
[10:08:10] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:10:24] <MirahezeLSBot>	 !log [oa@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/initSiteStats.php --update --wiki=321nailswiki
[10:10:26] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:23:38] <MirahezeLSBot>	 !log [@test151] starting deploy of {'config': True} to test151
[10:23:39] <MirahezeLSBot>	 !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s
[10:23:42] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:23:45] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:32:10] <MirahezeLSBot>	 !log [@mwtask181] starting deploy of {'config': True} to all
[10:32:13] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:32:19] <MirahezeLSBot>	 !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 8s
[10:32:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:35:37] <botifico>	 [02miraheze/WikiDiscover] 07OAuthority pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/WikiDiscover/compare/99f500bd3268...8c7ddbb92e19
[10:35:40] <botifico>	 [02miraheze/WikiDiscover] 07OAuthority 038c7ddbb - drop 1.40 from CI pipeline (T11809)
[10:39:07] <miraheze-github>	 miraheze/WikiDiscover - OAuthority the build passed.
[12:12:19] <icinga-miraheze>	 PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is WARNING: WARNING - NGINX Error Rate is 59%
[12:13:43] <MirahezeRelay>	 <rhinosf1> @originalauthority update the comment too please 🙂  it says 1.41 is next
[12:18:19] <icinga-miraheze>	 PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is CRITICAL: CRITICAL - NGINX Error Rate is 60%
[12:20:16] <icinga-miraheze>	 PROBLEM - www.johanloopmans.nl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.johanloopmans.nl' expires in 15 day(s) (Thu 07 Mar 2024 12:05:44 PM GMT +0000).
[12:20:27] <botifico>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9390c46f83bb...1d120d2a682f
[12:20:29] <botifico>	 [02miraheze/ssl] 07MirahezeSSLBot 031d120d2 - Bot: Update SSL cert for www.johanloopmans.nl
[12:26:36] <MirahezeRelay>	 <originalauthority> what?
[12:31:27] <icinga-miraheze>	 PROBLEM - puppet181 Puppet on puppet181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_ssl]
[12:35:36] <icinga-miraheze>	 PROBLEM - johanloopmans.nl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.johanloopmans.nl' expires in 15 day(s) (Thu 07 Mar 2024 12:05:44 PM GMT +0000).
[12:38:56] <MirahezeRelay>	 <rhinosf1, replying to originalauthority> look at https://github.com/miraheze/WikiDiscover/compare/99f500bd3268...8c7ddbb92e19
[12:39:14] <MirahezeRelay>	 <originalauthority> okay you could've made that clear. That is a very ambigious message you have sent.
[12:41:23] <MirahezeRelay>	 <originalauthority> i don't really know what you're on about, though
[12:41:38] <botifico>	 [02miraheze/WikiDiscover] 07OAuthority pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/WikiDiscover/compare/8c7ddbb92e19...a90607c8a3c7
[12:41:40] <botifico>	 [02miraheze/WikiDiscover] 07OAuthority 03a90607c - mark 1.41 as not experimental
[12:44:07] <miraheze-github>	 miraheze/WikiDiscover - OAuthority the build has errored.
[12:45:52] <MirahezeRelay>	 <originalauthority> oh ik niw
[12:47:14] <botifico>	 [02miraheze/WikiDiscover] 07OAuthority pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/WikiDiscover/compare/a90607c8a3c7...f88a7d0bafec
[12:47:15] <botifico>	 [02miraheze/WikiDiscover] 07OAuthority 03f88a7d0 - fix comment
[12:49:44] <miraheze-github>	 miraheze/WikiDiscover - OAuthority the build has errored.
[12:49:55] <MirahezeRelay>	 <originalauthority> hmmm it doesn't like being marked as not experimental
[12:59:27] <icinga-miraheze>	 RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures
[13:05:02] <icinga-miraheze>	 RECOVERY - johanloopmans.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'www.johanloopmans.nl' will expire on Mon 20 May 2024 11:20:22 AM GMT +0000.
[13:05:10] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[13:10:02] <MirahezeRelay>	 <rhinosf1> that's not good
[13:10:10] <MirahezeRelay>	 <rhinosf1> that means CI isn't passing on our deployed code
[13:18:25] <icinga-miraheze>	 RECOVERY - www.johanloopmans.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'www.johanloopmans.nl' will expire on Mon 20 May 2024 11:20:22 AM GMT +0000.
[13:38:41] <MirahezeRelay>	 <originalauthority> It passes when its marked as experimental, but not when marked as not
[13:39:38] <RhinosF1>	 that's cause experimental doesn't fail
[13:39:45] <RhinosF1>	 it just lists problems
[14:04:25] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query.
[14:34:02] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK
[14:38:11] <botifico>	 [02WikiDiscover] 07AgentIsai closed pull request 03#109: Update mediawiki/mediawiki-phan-config requirement from 0.13.0 to 0.14.0 - 13https://github.com/miraheze/WikiDiscover/pull/109
[14:38:14] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 100%
[14:38:14] <botifico>	 [02miraheze/WikiDiscover] 07AgentIsai pushed 032 commits to 03master [+0/-0/±2] 13https://github.com/miraheze/WikiDiscover/compare/f88a7d0bafec...ab421f53da67
[14:38:17] <botifico>	 [02miraheze/WikiDiscover] 07AgentIsai 03ab421f5 - Merge pull request #109 from miraheze/dependabot/composer/mediawiki/mediawiki-phan-config-0.14.0
[14:38:19] <botifico>	 [02miraheze/WikiDiscover] 07dependabot[bot] deleted branch 03dependabot/composer/mediawiki/mediawiki-phan-config-0.14.0
[14:38:21] <botifico>	 [02WikiDiscover] 07dependabot[bot] deleted branch 03dependabot/composer/mediawiki/mediawiki-phan-config-0.14.0 - 13https://github.com/miraheze/WikiDiscover
[14:38:32] <botifico>	 [02WikiDiscover] 07dependabot[bot] edited pull request 03#108: Update mediawiki/mediawiki-codesniffer requirement from 42.0.0 to 43.0.0 - 13https://github.com/miraheze/WikiDiscover/pull/108
[14:39:15] <botifico>	 [02WikiDiscover] 07dependabot[bot] edited pull request 03#108: Update mediawiki/mediawiki-codesniffer requirement from 42.0.0 to 43.0.0 - 13https://github.com/miraheze/WikiDiscover/pull/108
[14:39:16] <botifico>	 [02miraheze/WikiDiscover] 07dependabot[bot] pushed 031 commit to 03dependabot/composer/mediawiki/mediawiki-codesniffer-43.0.0 [+0/-0/±1] 13https://github.com/miraheze/WikiDiscover/compare/17f53fad4648...729dbf76109c
[14:39:19] <botifico>	 [02miraheze/WikiDiscover] 07dependabot[bot] 03729dbf7 - Update mediawiki/mediawiki-codesniffer requirement from 42.0.0 to 43.0.0
[14:39:19] <icinga-miraheze>	 PROBLEM - cp41 SSH on cp41 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[14:39:21] <botifico>	 [02WikiDiscover] 07dependabot[bot] synchronize pull request 03#108: Update mediawiki/mediawiki-codesniffer requirement from 42.0.0 to 43.0.0 - 13https://github.com/miraheze/WikiDiscover/pull/108
[14:39:22] <botifico>	 [02WikiDiscover] 07dependabot[bot] edited pull request 03#108: Update mediawiki/mediawiki-codesniffer requirement from 42.0.0 to 43.0.0 - 13https://github.com/miraheze/WikiDiscover/pull/108
[14:39:26] <botifico>	 [02WikiDiscover] 07AgentIsai closed pull request 03#108: Update mediawiki/mediawiki-codesniffer requirement from 42.0.0 to 43.0.0 - 13https://github.com/miraheze/WikiDiscover/pull/108
[14:39:28] <botifico>	 [02miraheze/WikiDiscover] 07AgentIsai pushed 032 commits to 03master [+0/-0/±2] 13https://github.com/miraheze/WikiDiscover/compare/ab421f53da67...597b9f79e3ba
[14:39:29] <botifico>	 [02miraheze/WikiDiscover] 07AgentIsai 03597b9f7 - Merge pull request #108 from miraheze/dependabot/composer/mediawiki/mediawiki-codesniffer-43.0.0
[14:39:33] <botifico>	 [02miraheze/WikiDiscover] 07dependabot[bot] deleted branch 03dependabot/composer/mediawiki/mediawiki-codesniffer-43.0.0
[14:39:36] <botifico>	 [02WikiDiscover] 07dependabot[bot] deleted branch 03dependabot/composer/mediawiki/mediawiki-codesniffer-43.0.0 - 13https://github.com/miraheze/WikiDiscover
[14:39:46] <icinga-miraheze>	 PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds
[14:40:33] <icinga-miraheze>	 PROBLEM - Host cp41 is DOWN: PING CRITICAL - Packet loss = 100%
[14:41:10] <miraheze-github>	 miraheze/WikiDiscover - dependabot[bot] the build has errored.
[14:41:19] <miraheze-github_>	 miraheze/WikiDiscover - AgentIsai the build has errored.
[15:00:30] <icinga-miraheze>	 PROBLEM - cp36 Disk Space on cp36 is WARNING: DISK WARNING - free space: / 9707MiB (10% inode=98%);
[15:02:17] <icinga-miraheze>	 RECOVERY - Host cp41 is UP: PING OK - Packet loss = 0%, RTA = 131.58 ms
[15:02:34] <icinga-miraheze>	 RECOVERY - cp41 HTTPS on cp41 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3610 bytes in 0.839 second response time
[15:02:39] <icinga-miraheze>	 RECOVERY - cp41 SSH on cp41 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0)
[15:03:29] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 128.17 ms
[15:30:03] <icinga-miraheze>	 PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 9703MiB (10% inode=98%);
[15:35:25] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[15:43:26] <MirahezeLSBot>	 !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=1d6chanwiki /home/agent/images --comment=[[phab:T11781\|Requested]] --user=Maintenance_script (START)
[15:43:26] <wm-bot>	 https://meta.miraheze.org/wiki/phab:T11781%5c
[15:43:29] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:43:30] <MirahezeLSBot>	 !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=lgbtawiki --no-updates lgbtqiawiki_xml_3095cc1edb334194a72f.xml.gz (START)
[15:43:33] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:44:19] <MirahezeLSBot>	 !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=1d6chanwiki /home/agent/images --comment=[[phab:T11781\|Requested]] --user=Maintenance_script (END - exit=0)
[15:44:19] <wm-bot>	 https://meta.miraheze.org/wiki/phab:T11781%5c
[15:44:19] <icinga-miraheze>	 PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is WARNING: WARNING - NGINX Error Rate is 48%
[15:44:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:44:31] <MirahezeLSBot>	 !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=1d6chanwiki --no-updates 1d4chan.org-20230927-history.xml (START)
[15:44:33] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:46:27] <botifico>	 [02miraheze/WikiDiscover] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/WikiDiscover/compare/597b9f79e3ba...14fc3a798e98
[15:46:30] <botifico>	 [02miraheze/WikiDiscover] 07AgentIsai 0314fc3a7 - Add phpcodesniffer
[15:49:04] <miraheze-github>	 miraheze/WikiDiscover - AgentIsai the build has errored.
[15:50:42] <MirahezeLSBot>	 !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CreateWiki/maintenance/setContainersAccess.php --wiki=projektregnumpoloniaewiki (END - exit=0)
[15:50:45] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:51:07] <MirahezeLSBot>	 !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CreateWiki/maintenance/setContainersAccess.php --wiki=racwiki (END - exit=0)
[15:51:10] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:51:25] <MirahezeLSBot>	 !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CreateWiki/maintenance/setContainersAccess.php --wiki=pfpwiki (END - exit=0)
[15:51:28] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:59:57] <icinga-miraheze>	 PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 155.13 ms
[16:03:55] <icinga-miraheze>	 RECOVERY - ping6 on cp28 is OK: PING OK - Packet loss = 0%, RTA = 153.00 ms
[16:32:06] <botifico>	 [02MirahezeMagic] 07amire80 synchronize pull request 03#465: Improve requestssl-text - 13https://github.com/miraheze/MirahezeMagic/pull/465
[16:38:20] <miraheze-github>	 miraheze/MirahezeMagic - amire80 the build passed.
[16:38:25] <botifico>	 [02MirahezeMagic] 07songnguxyz commented on pull request 03#465: Improve requestssl-text - 13https://github.com/miraheze/MirahezeMagic/pull/465#issuecomment-1954610063
[17:04:22] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK
[17:24:23] <botifico>	 [02MirahezeMagic] 07Universal-Omega closed pull request 03#465: Improve requestssl-text - 13https://github.com/miraheze/MirahezeMagic/pull/465
[17:24:26] <botifico>	 [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/79c682d998cf...4066ff585259
[17:24:29] <botifico>	 [02miraheze/MirahezeMagic] 07amire80 034066ff5 - Improve requestssl-text (#465)
[17:30:52] <miraheze-github>	 miraheze/MirahezeMagic - Universal-Omega the build passed.
[17:32:38] <MirahezeLSBot>	 !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all
[17:32:40] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:32:46] <MirahezeLSBot>	 !log [@mwtask181] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 8s
[17:32:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:33:06] <MirahezeRelay>	 <cosmicalpha, replying to rhinosf1> All extensions can be updated at once also btw but can we try and maintain 1.40 support for a couple more days? I'm fine if you would rather not, just my local test I haven't upgraded to 1.41 yet lol. Additionally 1.41 doesn't work for our CI which is why I didn't bump it yet, it doesn't like the way I had to do CreateWiki for it (@originalauthority)
[17:38:03] <MirahezeRelay>	 <originalauthority, replying to cosmicalpha> I'm fine with that.
[17:52:43] <icinga-miraheze>	 PROBLEM - wiki.seamly.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.seamly.net All nameservers failed to answer the query.
[17:53:58] <MirahezeLSBot>	 !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151
[17:53:59] <MirahezeLSBot>	 !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s
[17:54:00] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:54:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:59:21] <icinga-miraheze>	 PROBLEM - wiki.seamly.io - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.seamly.io All nameservers failed to answer the query.
[18:06:46] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[18:17:37] <botifico>	 [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/dc9200188b4b...e3270e03036e
[18:17:38] <botifico>	 [02miraheze/dns] 07Universal-Omega 03e3270e0 - Add acme challenges
[18:21:55] <icinga-miraheze>	 PROBLEM - wiki.seamly.net - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns2.dreamhost.com.', 'ns1.dreamhost.com.', 'ns3.dreamhost.com.'], 'CNAME': 'wiki.seamly.io.'}
[18:28:51] <icinga-miraheze>	 RECOVERY - wiki.seamly.io - reverse DNS on sslhost is OK: SSL OK - wiki.seamly.io reverse DNS resolves to cp37.wikitide.net - CNAME OK
[18:36:25] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK
[18:56:01] <botifico>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03trustandsafety [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/4a80e59ed863
[18:56:02] <botifico>	 [02miraheze/mw-config] 07Universal-Omega 034a80e59 - Allow meta trustandsafety to add/remove trustandsafety
[18:56:05] <botifico>	 [02mw-config] 07Universal-Omega created branch 03trustandsafety - 13https://github.com/miraheze/mw-config
[18:56:07] <botifico>	 [02mw-config] 07Universal-Omega opened pull request 03#5485: Allow meta trustandsafety to add/remove trustandsafety - 13https://github.com/miraheze/mw-config/pull/5485
[18:57:08] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[18:58:27] <botifico>	 [02mw-config] 07Universal-Omega closed pull request 03#5485: Allow meta trustandsafety to add/remove trustandsafety - 13https://github.com/miraheze/mw-config/pull/5485
[18:58:30] <botifico>	 [02miraheze/mw-config] 07Universal-Omega deleted branch 03trustandsafety
[18:58:31] <botifico>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/3cadfce6dac6...9cf5a634eba7
[18:58:32] <botifico>	 [02miraheze/mw-config] 07Universal-Omega 039cf5a63 - Allow meta trustandsafety to add/remove trustandsafety (#5485)
[18:58:34] <botifico>	 [02mw-config] 07Universal-Omega deleted branch 03trustandsafety - 13https://github.com/miraheze/mw-config
[18:59:19] <MirahezeLSBot>	 !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[18:59:22] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:59:24] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[18:59:29] <MirahezeLSBot>	 !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s
[18:59:32] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:23:24] <MirahezeLSBot>	 !log [@test151] starting deploy of {'config': True} to test151
[19:23:25] <MirahezeLSBot>	 !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s
[19:23:29] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:23:31] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:29:01] <botifico>	 [02RemovePII] 07Universal-Omega created branch 03fix-rename - 13https://github.com/miraheze/RemovePII
[19:29:03] <botifico>	 [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03fix-rename [+0/-0/±1] 13https://github.com/miraheze/RemovePII/commit/49095d9bd830
[19:29:05] <botifico>	 [02miraheze/RemovePII] 07Universal-Omega 0349095d9 - Fix GlobalRenameUser
[19:29:08] <botifico>	 [02RemovePII] 07Universal-Omega opened pull request 03#80: Fix GlobalRenameUser - 13https://github.com/miraheze/RemovePII/pull/80
[19:30:50] <botifico>	 [02RemovePII] 07Universal-Omega closed pull request 03#80: Fix GlobalRenameUser - 13https://github.com/miraheze/RemovePII/pull/80
[19:30:52] <botifico>	 [02miraheze/RemovePII] 07Universal-Omega deleted branch 03fix-rename
[19:30:54] <botifico>	 [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/RemovePII/compare/6701a37f7cf2...03d300a46b81
[19:30:55] <botifico>	 [02miraheze/RemovePII] 07Universal-Omega 0303d300a - Fix GlobalRenameUser (#80)
[19:30:57] <botifico>	 [02RemovePII] 07Universal-Omega deleted branch 03fix-rename - 13https://github.com/miraheze/RemovePII
[19:31:21] <MirahezeLSBot>	 !log [universalomega@mwtask181] starting deploy of {'versions': '1.41', 'upgrade_extensions': 'RemovePII'} to all
[19:31:24] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:31:30] <MirahezeLSBot>	 !log [universalomega@mwtask181] finished deploy of {'versions': '1.41', 'upgrade_extensions': 'RemovePII'} to all - SUCCESS in 9s
[19:31:33] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:36:39] <miraheze-github>	 miraheze/RemovePII - Universal-Omega the build passed.
[19:38:45] <miraheze-github>	 miraheze/RemovePII - Universal-Omega the build passed.
[19:45:55] <MirahezeLSBot>	 !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=lgbtawiki --no-updates lgbtqiawiki_xml_3095cc1edb334194a72f.xml.gz (END - exit=0)
[19:45:58] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:52:30] <icinga-miraheze>	 PROBLEM - cp36 Disk Space on cp36 is CRITICAL: DISK CRITICAL - free space: / 5308MiB (5% inode=98%);
[21:01:09] <MirahezeLSBot>	 !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=geografenstestamentewiki --no-updates --username-prefix=wikia:geografens-testamente /home/reception/geografenstestamentewiki-20240220170046.xml (START)
[21:01:11] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:02:29] <MirahezeLSBot>	 !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=geografenstestamentewiki --no-updates --username-prefix=wikia:geografens-testamente /home/reception/geografenstestamentewiki-20240220170046.xml (END - exit=0)
[21:02:29] <MirahezeLSBot>	 !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=geografenstestamentewiki (START)
[21:02:31] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:02:34] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:02:48] <MirahezeLSBot>	 !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=geografenstestamentewiki (END - exit=0)
[21:02:49] <MirahezeLSBot>	 !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/initSiteStats.php --wiki=geografenstestamentewiki --active --update (END - exit=0)
[21:02:51] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:02:54] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:14:03] <icinga-miraheze>	 PROBLEM - cp37 Disk Space on cp37 is CRITICAL: DISK CRITICAL - free space: / 5288MiB (5% inode=98%);
[21:14:55] <icinga-miraheze>	 PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 144.55 ms
[21:18:53] <icinga-miraheze>	 RECOVERY - ping6 on cp28 is OK: PING OK - Packet loss = 0%, RTA = 152.71 ms
[21:22:03] <icinga-miraheze>	 PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 5347MiB (6% inode=98%);
[21:26:03] <icinga-miraheze>	 PROBLEM - cp37 Disk Space on cp37 is CRITICAL: DISK CRITICAL - free space: / 5300MiB (5% inode=98%);
[21:35:43] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/OATHAuth/maintenance/disableOATHAuthForUser.php --wiki=ldapwikiwiki Macfan (END - exit=256)
[21:35:46] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:37:50] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/OATHAuth/maintenance/disableOATHAuthForUser.php --wiki=ldapwikiwiki Macfan (END - exit=256)
[21:37:53] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:49:21] <MirahezeRelay>	 <cosmicalpha> MacFan4000: do you mind doing that for my ldap account as well? If not I can in a few hours so no worry.
[22:31:25] <MirahezeLSBot>	 !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/DataDump/maintenance/insertMissingDumps.php --wiki=kanrikyarawiki (END - exit=0)
[22:31:27] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:31:50] <MirahezeLSBot>	 !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/DataDump/maintenance/deleteUnknownDumps.php --wiki=kanrikyarawiki (END - exit=0)
[22:31:53] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:36:15] <MirahezeLSBot>	 !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/DataDump/maintenance/migrateCompletedAndFailedToStatusColumn.php --wiki=kanrikyarawiki (END - exit=0)
[22:36:18] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:50:02] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/OATHAuth/maintenance/disableOATHAuthForUser.php --wiki=ldapwikiwiki Universalomega (END - exit=256)
[22:50:05] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:50:34] <MacFan4000>	 @cosmicalpha done
[22:50:37] <MacFan4000>	 User Universalomega doesn't have OATHAuth enabled!
[22:51:25] <MacFan4000>	 I did have to reapply the OATHauth awl earlier, so it may be that it isn’t enabled anymore for any ldap accounts
[22:51:36] <MacFan4000>	 s/awl/sql
[23:32:19] <icinga-miraheze>	 PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is CRITICAL: CRITICAL - NGINX Error Rate is 60%
[23:37:07] <MirahezeRelay>	 <cosmicalpha> ah
[23:37:12] <MirahezeRelay>	 <cosmicalpha> thanks though