[01:30:12] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [01:32:12] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [170 system event log (SEL) entries present] [03:25:52] PROBLEM - wiki.altilunium.my.id - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.altilunium.my.id' expires in 15 day(s) (Fri 02 Aug 2024 03:04:17 AM GMT +0000). [03:26:09] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/7c4287ec4a63...394f84998209 [03:26:10] [02ssl] 07WikiTideSSLBot 03394f849 - Bot: Update SSL cert for wiki.altilunium.my.id [03:55:44] RECOVERY - wiki.altilunium.my.id - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.altilunium.my.id' will expire on Tue 15 Oct 2024 02:25:58 AM GMT +0000. [05:14:10] PROBLEM - eros.locksteel.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'eros.locksteel.net' expires in 15 day(s) (Fri 02 Aug 2024 04:44:29 AM GMT +0000). [05:14:21] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/394f84998209...9acd7f3f4149 [05:14:22] [02ssl] 07WikiTideSSLBot 039acd7f3 - Bot: Update SSL cert for eros.locksteel.net [05:42:50] RECOVERY - eros.locksteel.net - LetsEncrypt on sslhost is OK: OK - Certificate 'eros.locksteel.net' will expire on Tue 15 Oct 2024 04:14:15 AM GMT +0000. [05:58:13] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [06:00:12] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [173 system event log (SEL) entries present] [06:57:53] PROBLEM - wiki.themarkside.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.themarkside.com All nameservers failed to answer the query. [07:27:53] PROBLEM - wiki.themarkside.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.themarkside.com could not be found [07:44:40] RECOVERY - mwtask181 Disk Space on mwtask181 is OK: DISK OK - free space: / 49067MiB (22% inode=94%); [07:50:40] PROBLEM - mwtask181 Disk Space on mwtask181 is CRITICAL: DISK CRITICAL - free space: / 7134MiB (3% inode=94%); [07:56:40] RECOVERY - mwtask181 Disk Space on mwtask181 is OK: DISK OK - free space: / 38337MiB (17% inode=94%); [08:02:11] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [08:02:40] PROBLEM - mwtask181 Disk Space on mwtask181 is CRITICAL: DISK CRITICAL - free space: / 5547MiB (2% inode=94%); [08:04:12] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [174 system event log (SEL) entries present] [08:04:40] RECOVERY - mwtask181 Disk Space on mwtask181 is OK: DISK OK - free space: / 47189MiB (21% inode=94%); [09:49:11] mwdeploy --upgrade-world --versions=1.42 in progress [09:49:17] didn't log for some reason [09:50:24] do you want to !log that? [09:50:30] nah [09:50:41] actually good point because it'll make a wiki edit, oops [09:51:10] the actual upgrade will be when I run changeMediaWikiVersion.php with mwscript on all wikis [09:51:15] until then we're still on 1.41 [09:53:06] [02ImportDump] 07redbluegreenhat closed pull request 03#99: Update for MediaWiki 1.42 and use virtual domain - 13https://github.com/miraheze/ImportDump/pull/99 [09:53:09] [02ImportDump] 07redbluegreenhat pushed 031 commit to 03master [+2/-0/±18] 13https://github.com/miraheze/ImportDump/compare/269385816d02...fd9ab0e69be8 [09:53:11] [02ImportDump] 07Universal-Omega 03fd9ab0e - Update for MediaWiki 1.42 and use virtual domain (#99) [09:53:12] [02ImportDump] 07redbluegreenhat edited pull request 03#108: Move ImportDumpRequestManager to Services namespace - 13https://github.com/miraheze/ImportDump/pull/108 [09:53:13] [02ImportDump] 07redbluegreenhat deleted branch 03REL1_42 [09:53:16] [02ImportDump] 07redbluegreenhat deleted branch 03REL1_42 - 13https://github.com/miraheze/ImportDump [10:02:13] miraheze/ImportDump - redbluegreenhat the build passed. [10:13:58] oops [10:14:35] !log [alex@bots171] test [10:14:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:14:44] logbot is back [10:15:03] just had to do a little restart [10:21:56] [02mw-config] 07Reception123 closed pull request 03#5607: T12143: Undo the original changes - 13https://github.com/miraheze/mw-config/pull/5607 [10:21:59] [02mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/e6cad6a02b1d...8ed474e93a00 [10:22:01] [02mw-config] 07Honglan233 038ed474e - T12143: Undo the original changes (#5607) [10:23:12] miraheze/mw-config - Reception123 the build passed. [10:23:28] !log [@test151] starting deploy of {'config': True} to test151 [10:23:29] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [10:23:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:23:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:30:05] !log [alex@mwtask181] finished deploy of {'upgrade_world': True, 'versions': '1.42', 'upgrade_extensions': 'all', 'upgrade_skins': 'all'} to all - SUCCESS in 1162s [10:30:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:31:07] !log [alex@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/MirahezeMagic/maintenance/changeMediaWikiVersion.php (START) [10:31:10] !log [alex@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/MirahezeMagic/maintenance/changeMediaWikiVersion.php (END - exit=2) [10:31:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:31:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:31:41] !log [alex@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/MirahezeMagic/maintenance/changeMediaWikiVersion.php --mwversion 1.42 (START) [10:31:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:31:54] wikis are upgrading now [10:32:18] now is when stuff can go weird with exceptions due to not running SQL patches and maint scripts [10:32:41] !log [@mwtask181] starting deploy of {'config': True} to all [10:32:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:32:47] Once that's done I move on to Puppet and MirahezeFunctions, then apply SQL patches [10:32:53] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 12s [10:32:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:37:37] !log [@mwtask171] starting deploy of {'config': True} to all [10:37:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:37:47] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 10s [10:37:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:39:44] [02mw-config] 07redbluegreenhat opened pull request 03#5609: Mark 1.42 as the stable version in MirahezeFunctions - 13https://github.com/miraheze/mw-config/pull/5609 [10:40:33] miraheze/mw-config - redbluegreenhat the build passed. [10:42:09] [02puppet] 07redbluegreenhat pushed 031 commit to 031.42-default-version [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/811b71228fee [10:42:12] [02puppet] 07redbluegreenhat 03811b712 - Make 1.42 the default version [10:42:14] [02puppet] 07redbluegreenhat created branch 031.42-default-version - 13https://github.com/miraheze/puppet [10:42:44] [02puppet] 07redbluegreenhat opened pull request 03#3884: Make 1.42 the default version - 13https://github.com/miraheze/puppet/pull/3884 [10:43:34] [02puppet] 07coderabbitai[bot] commented on pull request 03#3884: Make 1.42 the default version - 13https://github.com/miraheze/puppet/pull/3884#issuecomment-2233000345 [10:47:51] PROBLEM - johanloopmans.nl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.johanloopmans.nl' expires in 15 day(s) (Fri 02 Aug 2024 10:33:02 AM GMT +0000). [10:49:46] Orange_Star: do u need me to purge cloudflare [10:49:52] not yet [10:50:20] Orange_Star: aren't all wikis running the new code though [10:50:25] no [10:51:01] Oh you're change script hasn't finished yet [10:51:04] I'd like to rely on the fallback that would make them automatically run 1.42 when I change MirahezeFunctions [10:51:22] but some wikis have 1.41 set in mhglobal, instead of NULL [10:51:29] How many [10:51:29] so I have to run changeMediaWikiVersion [10:51:38] I don't remember [10:51:43] I remember metawiki was one of them [10:51:56] Weird [10:52:17] Ping me [10:52:30] I'm around like all of the afternoon [10:52:33] RhinosF1: ping ;) [10:52:39] Ok [10:53:35] !log purged cache-tag "mediawiki" from cloudflare [10:53:37] Orange_Star: ^ [10:53:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:54:06] I was joking with the ping [10:54:11] the script isn't done yet [10:54:15] sorry [10:54:23] Oh well [10:54:27] Won't matter much anyway [10:54:29] I do it whenever [10:54:58] It will only affect around 350k requests an hour [10:55:55] Cache hit is only about 50% when you exclude static [10:55:59] Static is like 98% [10:56:03] or maybe higher [10:59:16] PROBLEM - www.johanloopmans.nl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.johanloopmans.nl' expires in 15 day(s) (Fri 02 Aug 2024 10:33:02 AM GMT +0000). [11:10:45] I've decided to apply the SQL patches in parallel to the upgrade to speed up things a bit. There's no danger with this procedure, since the SQL patch process will not catch up to the upgrade script. [11:12:45] Patching the global database now [11:13:02] !log [alex@mwtask171] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php --wiki=loginwiki sql.php --wikidb=mhglobal /srv/mediawiki/1.42/extensions/GlobalBlocking/sql/mysql/patch-add-gb_target_central_id.sql (END - exit=256) [11:13:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:13:35] Orange_Star: good idea [11:14:01] Although maybe avoid any that drop data [11:14:47] I don't think any of them drop data, I'll be checking them before running anyway [11:16:39] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:16:44] !log [alex@mwtask171] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php --wiki=loginwiki sql.php --wikidb=mhglobal /srv/mediawiki/1.42/extensions/GlobalBlocking/sql/mysql/patch-add-gb_target_central_id.sql (END - exit=256) [11:16:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:17:08] ahh [11:17:17] well the global db is going to have to wait a bit [11:19:59] !log [alex@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php /srv/mediawiki/1.42/maintenance/archives/patch-block_target.sql (START) [11:20:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:25:39] !log [alex@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php /srv/mediawiki/1.42/extensions/CheckUser/schema/mysql/cu_useragent.sql (START) [11:25:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:28:06] !log [alex@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php /srv/mediawiki/1.42/extensions/CheckUser/schema/mysql/patch-cu_changes-add-cuc_agent_id.sql (START) [11:28:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:31:18] !log [alex@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php /srv/mediawiki/1.42/extensions/CheckUser/schema/mysql/patch-cu_changes-modify-cuc_id-bigint.sql (START) [11:31:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:36:32] !log [alex@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/sql.php /srv/mediawiki/1.42/extensions/CheckUser/schema/mysql/patch-cu_log-add-cul_result_id.sql (START) [11:36:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:37:51] !log [alex@mwtask171] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php --wiki=loginwiki --wikidb=mhglobal /srv/mediawiki/1.42/extensions/GlobalBlocking/sql/mysql/patch-add-gb_target_central_id.sql (END - exit=0) [11:37:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:40:09] !log [alex@mwtask171] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php --wiki=loginwiki --wikidb=mhglobal /srv/mediawiki/1.42/extensions/GlobalBlocking/sql/mysql/patch-modify-gb_by-default.sql (END - exit=256) [11:40:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:44:16] !log [alex@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php /srv/mediawiki/1.42/extensions/CheckUser/schema/mysql/patch-cu_log_event-add-cule_agent_id.sql (START) [11:44:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:44:46] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:46:41] Orange_Star: what's the exit 256 for? [11:47:36] Wikimedia\Rdbms\DBQueryError from line 1203 of /srv/mediawiki/1.42/includes/libs/rdbms/database/Database.php: Error 1265: Data truncated for column 'gb_by_central_id' at row 1 [11:47:47] hmm [11:47:49] Weird [11:51:10] Orange_Star: what about the first one [11:53:09] !log [alex@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php /srv/mediawiki/1.42/maintenance/archives/patch-drop-cl_collation_ext.sql (START) [11:53:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:53:53] I have no clue why the second causes an error [11:54:45] !log [alex@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/sql.php /srv/mediawiki/1.42/extensions/CheckUser/schema/mysql/patch-cu_private_event-add-cupe_agent_id.sql (START) [11:54:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:57:03] !log [alex@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php /srv/mediawiki/1.42/extensions/CheckUser/schema/mysql/patch-cu_private_event-modify-cupe_actor-nullable.sql (START) [11:57:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:58:22] !log [alex@mwtask171] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php --wiki=loginwiki --wikidb=mhglobal /srv/mediawiki/1.42/extensions/CentralAuth/schema/mysql/patch-add-global_user_autocreate_serial.sql (END - exit=256) [11:58:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:58:48] table already exists [11:59:06] !log [alex@mwtask171] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php --wiki=loginwiki --wikidb=mhglobal /srv/mediawiki/1.42/extensions/CentralAuth/schema/mysql/patch-add-uas_year.sql (END - exit=0) [11:59:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:09:04] !log [alex@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php /srv/mediawiki/1.42/extensions/GlobalBlocking/sql/mysql/patch-add-gbw_target_central_id.sql (START) [12:09:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:13:02] !log [alex@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php /srv/mediawiki/1.42/maintenance/archives/patch-iwlinks-drop-iwl_prefix_from_title.sql (START) [12:13:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:14:19] !log [alex@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php /srv/mediawiki/1.42/maintenance/archives/patch-sites-drop_indexes.sql (START) [12:14:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:14:38] this last one is going to take a long time to finish compared to the rest [12:19:43] !log [alex@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php /srv/mediawiki/1.42/maintenance/archives/patch-user_autocreate_serial-uas_year.sql (START) [12:19:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:23:25] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.88, 21.44, 17.96 [12:24:11] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [12:25:21] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.47, 22.66, 18.81 [12:26:12] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [176 system event log (SEL) entries present] [12:29:13] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.65, 23.49, 20.06 [12:31:09] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.39, 23.49, 20.44 [12:33:04] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.88, 22.67, 20.47 [12:38:52] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.94, 24.38, 21.70 [12:42:43] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.47, 23.71, 22.13 [12:45:09] !log [alex@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/MirahezeMagic/maintenance/changeMediaWikiVersion.php --mwversion 1.42 (END - exit=0) [12:45:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:45:17] 😂 [12:45:23] wait, wrong emoji [12:45:26] :Partyheze: [12:46:32] [02mw-config] 07redbluegreenhat closed pull request 03#5609: Mark 1.42 as the stable version in MirahezeFunctions - 13https://github.com/miraheze/mw-config/pull/5609 [12:46:34] [02mw-config] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/8ed474e93a00...8b87358c821f [12:46:35] [02mw-config] 07redbluegreenhat 038b87358 - Mark 1.42 as the stable version in MirahezeFunctions (#5609) [12:46:59] [02puppet] 07redbluegreenhat closed pull request 03#3884: Make 1.42 the default version - 13https://github.com/miraheze/puppet/pull/3884 [12:47:02] [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/938dd53dfdb9...d989b30946d2 [12:47:03] [02puppet] 07redbluegreenhat 03d989b30 - Make 1.42 the default version (#3884) [12:47:06] [02puppet] 07redbluegreenhat deleted branch 031.42-default-version [12:47:07] [02puppet] 07redbluegreenhat deleted branch 031.42-default-version - 13https://github.com/miraheze/puppet [12:48:01] miraheze/mw-config - redbluegreenhat the build passed. [12:48:32] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.95, 18.03, 20.05 [12:48:46] \o/ [12:52:37] !log [alex@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/migrateBlocks.php (START) [12:52:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:53:11] ipblocks migration in progress [12:53:57] !log [@test151] starting deploy of {'config': True} to test151 [12:53:58] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [12:54:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:54:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:02:54] !log [@mwtask181] starting deploy of {'config': True} to all [13:03:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:03:07] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 13s [13:03:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:07:20] !log [@mwtask171] starting deploy of {'config': True} to all [13:07:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:07:30] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 10s [13:07:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:09:34] !log sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/extensions/CentralAuth/maintenance/createLocalAccount.php --wiki=dnatptrgwiki Waki285 [13:09:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:21:33] !log [alex@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/sql.php /srv/mediawiki/1.42/extensions/CheckUser/schema/mysql/patch-cu_log-add-cul_result_id.sql (END - exit=65280) [13:21:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:22:58] !log [alex@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/alex/TimedMediaHandler.json /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/extensions/TimedMediaHandler/maintenance/rebuildPlaylists.php (START) [13:23:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:24:25] Running this script fixes the problem described in https://github.com/wikimedia/mediawiki-extensions-TimedMediaHandler/commit/081c701c7e3d461680384bd6e71b11e35f6147a5 [13:26:20] failed due to not running with 1.42 mediawiki [13:26:25] my bad [13:26:32] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.15, 19.45, 17.33 [13:26:49] !log [alex@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php /srv/mediawiki/1.42/extensions/CheckUser/schema/mysql/patch-cu_log-add-cul_result_id.sql (START) [13:26:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:27:19] running it again is no issue because it will skip databases where the column already exists [13:28:32] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.62, 18.13, 17.12 [13:29:21] !log [alex@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/sql.php /srv/mediawiki/1.42/extensions/CheckUser/schema/mysql/patch-cu_private_event-add-cupe_agent_id.sql (END - exit=65280) [13:29:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:30:40] PROBLEM - wiki.theatlas.live - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.theatlas.live could not be found [13:32:09] same as the other one [13:32:15] !log [alex@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php /srv/mediawiki/1.42/extensions/CheckUser/schema/mysql/patch-cu_private_event-add-cupe_agent_id.sql (START) [13:32:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:33:57] !log [alex@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php /srv/mediawiki/1.42/maintenance/archives/patch-block_target.sql (END - exit=0) [13:34:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:34:57] !log [alex@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php /srv/mediawiki/1.42/extensions/CheckUser/schema/mysql/cu_useragent.sql (END - exit=0) [13:35:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:35:53] PROBLEM - wiki.theatlas.live - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [13:36:25] !log [alex@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php /srv/mediawiki/1.42/extensions/CheckUser/schema/mysql/patch-cu_changes-add-cuc_agent_id.sql (END - exit=0) [13:36:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:44:15] !log [alex@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php /srv/mediawiki/1.42/extensions/CheckUser/schema/mysql/patch-cu_changes-modify-cuc_id-bigint.sql (END - exit=0) [13:44:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:52:28] !log [alex@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php /srv/mediawiki/1.42/extensions/CheckUser/schema/mysql/patch-cu_log_event-add-cule_agent_id.sql (END - exit=0) [13:52:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:54:19] [02mw-config] 07redbluegreenhat opened pull request 03#5610: Disable RightFunctions - 13https://github.com/miraheze/mw-config/pull/5610 [13:55:09] miraheze/mw-config - redbluegreenhat the build passed. [13:56:39] [02mw-config] 07redbluegreenhat closed pull request 03#5610: Disable RightFunctions - 13https://github.com/miraheze/mw-config/pull/5610 [13:56:42] [02mw-config] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/8b87358c821f...a711120e357d [13:56:45] [02mw-config] 07redbluegreenhat 03a711120 - Disable RightFunctions (#5610) [13:57:40] miraheze/mw-config - redbluegreenhat the build passed. [14:02:17] !log [alex@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/alex/TimedMediaHandler.json /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/extensions/TimedMediaHandler/maintenance/rebuildPlaylists.php (END - exit=256) [14:02:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:02:44] !log [@mwtask181] starting deploy of {'config': True} to all [14:02:49] "failed" due to https://issue-tracker.miraheze.org/T12236 [14:02:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:02:57] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 12s [14:03:00] but it ran on all wikis with TimedMediaHandler enabled [14:03:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:06:02] !log [alex@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/alex/WikibaseLexeme.json /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/extensions/WikibaseLexeme/maintenance/FixPagePropsSortkey.php (START) [14:06:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:07:06] !log [@mwtask171] starting deploy of {'config': True} to all [14:07:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:07:12] !log [alex@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/alex/WikibaseLexeme.json /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/extensions/WikibaseLexeme/maintenance/FixPagePropsSortkey.php (END - exit=0) [14:07:16] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 9s [14:07:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:07:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:08:22] !log [alex@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php /srv/mediawiki/1.42/extensions/CheckUser/schema/mysql/patch-cu_private_event-modify-cupe_actor-nullable.sql (END - exit=0) [14:08:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:16:20] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:16:36] !log [alex@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php /srv/mediawiki/1.42/extensions/GlobalBlocking/sql/mysql/patch-add-gbw_target_central_id.sql (END - exit=0) [14:16:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:24:03] !log [@test151] starting deploy of {'config': True} to test151 [14:24:04] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [14:24:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:24:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:26:32] !log [alex@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php /srv/mediawiki/1.42/maintenance/archives/patch-drop-cl_collation_ext.sql (END - exit=0) [14:26:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:41:37] !log [alex@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php /srv/mediawiki/1.42/maintenance/archives/patch-user_autocreate_serial-uas_year.sql (END - exit=0) [14:41:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:46:32] !log [alex@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php /srv/mediawiki/1.42/maintenance/archives/patch-iwlinks-drop-iwl_prefix_from_title.sql (END - exit=0) [14:46:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:54:11] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [14:56:12] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [177 system event log (SEL) entries present] [14:56:30] !log [alex@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/alex/Translate.json /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/extensions/Translate/maintenance/refresh-translation-progress-stats.php (START) [14:56:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:04:52] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.32, 20.46, 17.88 [15:05:31] !log [alex@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/migrateBlocks.php (END - exit=0) [15:05:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:06:48] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.04, 18.68, 17.58 [15:12:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [15:14:25] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:22:24] !log [reception@mwtask181] starting deploy of {'files': '../mediawiki/1.42/extensions/SemanticMediaWiki/.smw.json'} to all [15:22:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:22:35] !log [reception@mwtask181] finished deploy of {'files': '../mediawiki/1.42/extensions/SemanticMediaWiki/.smw.json'} to all - SUCCESS in 11s [15:22:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:25:55] !log [alex@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php /srv/mediawiki/1.42/extensions/CheckUser/schema/mysql/patch-cu_log-add-cul_result_id.sql (END - exit=0) [15:26:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:27:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [15:30:56] !log [alex@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php /srv/mediawiki/1.42/extensions/CheckUser/schema/mysql/patch-cu_private_event-add-cupe_agent_id.sql (END - exit=0) [15:31:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:40:28] PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:42:26] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0005670189857 secs [15:44:05] !log remove some cf mitigations for a DDoS that are too many false positives [15:44:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:06:34] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.27, 19.00, 14.97 [16:08:34] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 15.50, 17.73, 15.02 [16:14:28] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.21, 20.94, 17.14 [16:20:28] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 21.30, 22.69, 19.51 [16:22:28] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.46, 23.07, 19.99 [16:24:34] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.35, 20.07, 17.57 [16:32:34] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.53, 22.69, 19.72 [16:38:34] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.50, 23.93, 21.47 [16:40:03] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.43, 20.04, 18.05 [16:41:59] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 20.35, 20.19, 18.35 [16:46:34] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.48, 22.72, 21.62 [16:52:25] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.08, 22.59, 22.11 [16:53:49] [02mw-config] 07redbluegreenhat opened pull request 03#5611: T12344: Disable RatePage - 13https://github.com/miraheze/mw-config/pull/5611 [16:55:06] [02mw-config] 07redbluegreenhat synchronize pull request 03#5611: T12344: Disable RatePage - 13https://github.com/miraheze/mw-config/pull/5611 [16:56:03] miraheze/mw-config - redbluegreenhat the build passed. [17:00:10] [02mw-config] 07redbluegreenhat closed pull request 03#5611: T12344: Disable RatePage - 13https://github.com/miraheze/mw-config/pull/5611 [17:00:12] [02mw-config] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a711120e357d...2e4a6614e735 [17:00:13] [02mw-config] 07redbluegreenhat 032e4a661 - T12344: Disable RatePage (#5611) [17:01:10] miraheze/mw-config - redbluegreenhat the build passed. [17:02:08] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 9.36, 16.14, 19.57 [17:02:16] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [17:02:23] !log [@mwtask181] starting deploy of {'config': True} to all [17:02:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:02:35] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 12s [17:02:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:04:16] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [178 system event log (SEL) entries present] [17:06:00] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.59, 21.31, 19.61 [17:07:24] !log [@mwtask171] starting deploy of {'config': True} to all [17:07:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:07:38] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 13s [17:07:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:07:59] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.25, 22.04, 20.09 [17:08:08] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.44, 22.91, 21.23 [17:12:08] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.62, 23.54, 21.90 [17:14:50] !log [alex@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/alex/SemanticMediaWiki.json /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/extensions/SemanticMediaWiki/maintenance/setupStore.php (START) [17:14:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:15:55] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.86, 21.61, 20.52 [17:18:33] !log [alex@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/alex/SemanticMediaWiki.json /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/extensions/SemanticMediaWiki/maintenance/setupStore.php (END - exit=0) [17:18:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:18:47] !log [alex@mwtask171] starting deploy of {'files': '../mediawiki/1.42/extensions/SemanticMediaWiki/.smw.json'} to all [17:18:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:19:01] !log [alex@mwtask171] finished deploy of {'files': '../mediawiki/1.42/extensions/SemanticMediaWiki/.smw.json'} to all - SUCCESS in 13s [17:19:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:19:54] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.99, 23.03, 21.52 [17:20:08] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.96, 23.46, 22.20 [17:22:08] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 18.11, 21.67, 21.72 [17:24:07] !log [@test151] starting deploy of {'config': True} to test151 [17:24:08] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [17:24:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:24:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:28:08] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.16, 22.66, 22.00 [17:30:08] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.39, 22.01, 21.81 [17:31:49] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.17, 23.19, 22.06 [17:32:08] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.11, 23.92, 22.51 [17:33:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.32, 22.22, 21.81 [17:35:47] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.00, 22.59, 21.95 [17:42:08] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.94, 23.63, 23.32 [17:42:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [17:43:44] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.59, 22.86, 22.86 [17:44:08] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.04, 24.53, 23.69 [17:47:38] !log [alex@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php /srv/mediawiki/1.42/maintenance/archives/patch-sites-drop_indexes.sql (END - exit=0) [17:47:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:48:02] and that's the last SQL patch [17:52:25] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [17:53:40] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.17, 21.98, 22.16 [17:55:38] woooooooo [17:57:39] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.99, 23.58, 22.88 [18:11:00] PROBLEM - 9b9t.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [18:11:33] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 19.18, 18.53, 20.21 [18:12:25] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [18:17:08] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:19:29] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.85, 21.74, 20.74 [18:21:28] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.86, 23.52, 21.48 [18:27:25] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.33, 22.91, 22.01 [18:37:21] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.43, 23.50, 22.21 [18:45:05] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:47:25] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [19:15:06] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.97, 22.23, 23.85 [19:19:57] PROBLEM - 9b9t.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for 9b9t.wiki could not be found [19:22:25] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [19:25:02] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.37, 21.88, 22.22 [19:29:00] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.99, 21.95, 22.25 [19:38:08] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 18.79, 21.91, 23.88 [19:40:08] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.91, 23.08, 24.03 [19:41:50] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 19.22, 21.02, 23.72 [19:42:08] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 17.82, 20.67, 23.02 [19:42:54] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.69, 18.82, 20.26 [19:43:50] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.12, 22.37, 23.88 [19:44:08] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.03, 22.54, 23.39 [19:45:50] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.93, 21.95, 23.56 [19:46:08] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.28, 21.40, 22.85 [19:47:50] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.87, 23.85, 24.03 [19:49:50] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.17, 23.15, 23.76 [19:52:49] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.74, 20.61, 20.24 [19:53:50] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 28.96, 25.64, 24.58 [19:54:08] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.32, 22.81, 22.75 [19:54:48]