[00:00:03] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [00:02:04] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [00:02:45] PROBLEM - cp36 Disk Space on cp36 is CRITICAL: DISK CRITICAL - free space: / 4821MiB (5% inode=98%); [00:04:25] RECOVERY - cp41 Disk Space on cp41 is OK: DISK OK - free space: / 21695MiB (23% inode=98%); [00:04:45] RECOVERY - cp36 Disk Space on cp36 is OK: DISK OK - free space: / 20124MiB (22% inode=98%); [00:05:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [00:05:42] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 19500MiB (22% inode=98%); [00:18:52] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03job [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/1f29b2b892b6 [00:18:53] [02miraheze/mw-config] 07Universal-Omega 031f29b2b - Switch loginwiki, iowiki, and staffwiki to EventBus JobQueue [00:18:54] [02mw-config] 07Universal-Omega created branch 03job - 13https://github.com/miraheze/mw-config [00:18:56] [02mw-config] 07Universal-Omega opened pull request 03#5513: Switch loginwiki, iowiki, and staffwiki to EventBus JobQueue - 13https://github.com/miraheze/mw-config/pull/5513 [00:19:30] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03job [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/1f29b2b892b6...9abed29de66e [00:19:31] [02miraheze/mw-config] 07github-actions 039abed29 - CI: lint code to MediaWiki standards [00:19:33] [02mw-config] 07github-actions[bot] synchronize pull request 03#5513: Switch loginwiki, iowiki, and staffwiki to EventBus JobQueue - 13https://github.com/miraheze/mw-config/pull/5513 [00:20:00] [02mw-config] 07Universal-Omega synchronize pull request 03#5513: Switch loginwiki, iowiki, and staffwiki to EventBus JobQueue - 13https://github.com/miraheze/mw-config/pull/5513 [00:20:01] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03job [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/9abed29de66e...2a2db04e08e9 [00:20:02] miraheze/mw-config - Universal-Omega the build passed. [00:20:04] [02miraheze/mw-config] 07Universal-Omega 032a2db04 - Fix [00:20:56] miraheze/mw-config - Universal-Omega the build passed. [00:21:23] [02mw-config] 07Universal-Omega closed pull request 03#5513: Switch loginwiki, iowiki, and staffwiki to EventBus JobQueue - 13https://github.com/miraheze/mw-config/pull/5513 [00:21:26] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/cdbaa55c08c1...179b70c6f6b1 [00:21:27] [02miraheze/mw-config] 07Universal-Omega 03179b70c - Switch loginwiki, iowiki, and staffwiki to EventBus JobQueue (#5513) [00:21:30] !log [universalomega@mwtask171] starting deploy of {'pull': 'config', 'config': True} to all [00:21:30] [02miraheze/mw-config] 07Universal-Omega deleted branch 03job [00:21:31] [02mw-config] 07Universal-Omega deleted branch 03job - 13https://github.com/miraheze/mw-config [00:21:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:21:39] !log [universalomega@mwtask171] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s [00:21:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:22:13] miraheze/mw-config - Universal-Omega the build passed. [00:23:42] !log [@test151] starting deploy of {'config': True} to test151 [00:23:43] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [00:23:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:23:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:32:48] !log [@mwtask181] starting deploy of {'config': True} to all [00:32:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:32:57] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 9s [00:33:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:55:33] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/179b70c6f6b1...793407b2d522 [00:55:34] [02miraheze/mw-config] 07Universal-Omega 03793407b - Switch all wikis using SMW to EventBus JobQueue [00:56:25] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [00:56:27] miraheze/mw-config - Universal-Omega the build passed. [00:56:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:56:35] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s [00:56:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:07:40] !log [@mwtask171] starting deploy of {'config': True} to all [01:07:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:07:48] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s [01:07:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:23:12] !log [@test151] starting deploy of {'config': True} to test151 [01:23:13] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [01:23:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:23:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:25:25] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.402 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [02:54:24] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03cpjobqueue [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/5a010e78984b [02:54:27] [02miraheze/puppet] 07Universal-Omega 035a010e7 - mwtask181: enable use of cpjobqueue [02:54:30] [02puppet] 07Universal-Omega created branch 03cpjobqueue - 13https://github.com/miraheze/puppet [02:54:33] [02puppet] 07Universal-Omega opened pull request 03#3795: mwtask181: enable use of cpjobqueue - 13https://github.com/miraheze/puppet/pull/3795 [02:54:57] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [02:56:48] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03jobqueue [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/58a86d832191 [02:56:50] [02miraheze/mw-config] 07Universal-Omega 0358a86d8 - - [02:56:51] [02mw-config] 07Universal-Omega created branch 03jobqueue - 13https://github.com/miraheze/mw-config [02:56:54] [02mw-config] 07Universal-Omega opened pull request 03#5514: - - 13https://github.com/miraheze/mw-config/pull/5514 [02:57:15] [02mw-config] 07Universal-Omega edited pull request 03#5514: Enable EventBus JobQueue on all wikis - 13https://github.com/miraheze/mw-config/pull/5514 [02:57:18] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03jobqueue [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/58a86d832191...3db36f2db4ff [02:57:20] [02miraheze/mw-config] 07github-actions 033db36f2 - CI: lint code to MediaWiki standards [02:57:23] [02mw-config] 07github-actions[bot] synchronize pull request 03#5514: Enable EventBus JobQueue on all wikis - 13https://github.com/miraheze/mw-config/pull/5514 [02:57:50] miraheze/mw-config - Universal-Omega the build passed. [02:59:15] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03jobqueue [+1/-0/±0] 13https://github.com/miraheze/mw-config/compare/3db36f2db4ff...b12ed8717699 [02:59:17] [02miraheze/mw-config] 07Universal-Omega 03b12ed87 - Add [02:59:20] [02mw-config] 07Universal-Omega synchronize pull request 03#5514: Enable EventBus JobQueue on all wikis - 13https://github.com/miraheze/mw-config/pull/5514 [02:59:47] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03jobqueue [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/b12ed8717699...b31820b7c34e [02:59:48] [02miraheze/mw-config] 07Universal-Omega 03b31820b - Add [02:59:50] [02mw-config] 07Universal-Omega synchronize pull request 03#5514: Enable EventBus JobQueue on all wikis - 13https://github.com/miraheze/mw-config/pull/5514 [03:00:09] miraheze/mw-config - Universal-Omega the build passed. [03:00:39] miraheze/mw-config - Universal-Omega the build passed. [03:00:54] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03jobqueue [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/b31820b7c34e...55494731d518 [03:00:55] [02miraheze/mw-config] 07Universal-Omega 035549473 - Add [03:00:58] [02mw-config] 07Universal-Omega synchronize pull request 03#5514: Enable EventBus JobQueue on all wikis - 13https://github.com/miraheze/mw-config/pull/5514 [03:01:31] [02mw-config] 07Universal-Omega synchronize pull request 03#5514: Enable EventBus JobQueue on all wikis - 13https://github.com/miraheze/mw-config/pull/5514 [03:01:34] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03jobqueue [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/55494731d518...f9003c8b9b37 [03:01:35] [02miraheze/mw-config] 07Universal-Omega 03f9003c8 - +classes [03:01:47] miraheze/mw-config - Universal-Omega the build passed. [03:02:00] RECOVERY - db151 Backups SQL on db151 is OK: FILE_AGE OK: /var/log/sql-backup.log is 118 seconds old and 0 bytes [03:02:27] miraheze/mw-config - Universal-Omega the build passed. [03:03:47] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03mwtask181-jobrunner [+0/-0/±1] 13https://github.com/miraheze/dns/commit/21ba4f2a8b14 [03:03:49] [02miraheze/dns] 07Universal-Omega 0321ba4f2 - Add mwtask181 to jobrunner pool [03:03:51] [02dns] 07Universal-Omega created branch 03mwtask181-jobrunner - 13https://github.com/miraheze/dns [03:03:54] [02dns] 07Universal-Omega opened pull request 03#503: Add mwtask181 to jobrunner pool - 13https://github.com/miraheze/dns/pull/503 [03:05:16] [02dns] 07Universal-Omega synchronize pull request 03#503: Add mwtask181 to jobrunner pool - 13https://github.com/miraheze/dns/pull/503 [03:05:19] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03mwtask181-jobrunner [+0/-0/±1] 13https://github.com/miraheze/dns/compare/21ba4f2a8b14...e1c040175bb6 [03:05:21] [02miraheze/dns] 07Universal-Omega 03e1c0401 - Update IP [03:55:51] [02mw-config] 07Universal-Omega commented on pull request 03#5512: Setup wgPrivilegedGroups - 13https://github.com/miraheze/mw-config/pull/5512#issuecomment-2019336449 [03:59:59] [02mw-config] 07songnguxyz synchronize pull request 03#5511: Adding function for UnlinkedWikibase - 13https://github.com/miraheze/mw-config/pull/5511 [04:00:55] miraheze/mw-config - songnguxyz the build passed. [04:02:56] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03jobqueue [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f9003c8b9b37...e739e7ada5c2 [04:02:58] [02miraheze/mw-config] 07Universal-Omega 03e739e7a - - [04:03:01] [02mw-config] 07Universal-Omega synchronize pull request 03#5514: Enable EventBus JobQueue on all wikis - 13https://github.com/miraheze/mw-config/pull/5514 [04:03:52] miraheze/mw-config - Universal-Omega the build passed. [04:05:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [04:05:41] [02mw-config] 07songnguxyz synchronize pull request 03#5511: Adding function for UnlinkedWikibase - 13https://github.com/miraheze/mw-config/pull/5511 [04:06:35] miraheze/mw-config - songnguxyz the build passed. [04:56:24] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.408 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [05:25:59] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [07:21:39] [02mw-config] 07Universal-Omega closed pull request 03#5511: Adding function for UnlinkedWikibase - 13https://github.com/miraheze/mw-config/pull/5511 [07:21:41] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/793407b2d522...6beacd789414 [07:21:44] [02miraheze/mw-config] 07songnguxyz 036beacd7 - Adding function for UnlinkedWikibase (#5511) [07:22:35] miraheze/mw-config - Universal-Omega the build passed. [07:23:54] !log [@test151] starting deploy of {'config': True} to test151 [07:23:54] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [07:24:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:24:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:32:41] !log [@mwtask181] starting deploy of {'config': True} to all [07:32:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:32:51] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 10s [07:32:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:37:09] !log [@mwtask171] starting deploy of {'config': True} to all [07:37:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:37:17] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s [07:37:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:05:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [09:04:23] [02CreateWiki] 07redbluegreenhat opened pull request 03#493: Rename query_wiki_request to wiki_request - 13https://github.com/miraheze/CreateWiki/pull/493 [09:07:54] [02CreateWiki] 07redbluegreenhat synchronize pull request 03#493: Rename query_wiki_request to wiki_request - 13https://github.com/miraheze/CreateWiki/pull/493 [09:10:46] [02CreateWiki] 07redbluegreenhat closed pull request 03#493: Rename query_wiki_request to wiki_request - 13https://github.com/miraheze/CreateWiki/pull/493 [09:10:48] [02miraheze/CreateWiki] 07redbluegreenhat pushed 031 commit to 03master [+1/-1/±1] 13https://github.com/miraheze/CreateWiki/compare/533855c76862...818e7a897415 [09:10:50] [02miraheze/CreateWiki] 07redbluegreenhat 03818e7a8 - Rename query_wiki_request to wiki_request (#493) [09:14:34] !log [alex@test151] starting deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to test151 [09:14:35] !log [alex@test151] finished deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to test151 - SUCCESS in 1s [09:14:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:14:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:15:41] miraheze/CreateWiki - redbluegreenhat the build passed. [09:19:02] miraheze/CreateWiki - redbluegreenhat the build passed. [09:41:25] PROBLEM - wiki.sofurry.games - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.sofurry.games' expires in 15 day(s) (Thu 11 Apr 2024 09:13:00 AM GMT +0000). [09:50:46] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=sagan4alphawiki --search-recursively /home/reception/sagan4alpha --summary=Imported from sagan4alpha.wikitide.org (START) [09:50:46] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=sagan4alphawiki --search-recursively /home/reception/sagan4alpha --summary=Imported from sagan4alpha.wikitide.org (END - exit=0) [09:50:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:51:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:53:01] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=sagan4alphawiki --search-recursively /home/reception/sagan4alpha --summary=Imported from sagan4alpha.wikitide.org (START) [09:53:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:27:33] PROBLEM - wiki.closai.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.closai.com' expires in 15 day(s) (Thu 11 Apr 2024 10:06:52 AM GMT +0000). [10:38:33] !log [alex@test151] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/sql.php --wiki=metawikibeta (END - exit=2) [10:38:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:35:56] [02miraheze/CreateWiki] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/818e7a897415...61b1de583ede [11:35:58] [02miraheze/CreateWiki] 07redbluegreenhat 0361b1de5 - Merge pull request from GHSA-8wjf-mxjg-j8p9 [11:37:14] !log [alex@test151] starting deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to test151 [11:37:15] !log [alex@test151] finished deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to test151 - SUCCESS in 1s [11:37:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:37:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:38:45] [02miraheze/CreateWiki] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/61b1de583ede...0c7c4f938343 [11:38:47] [02miraheze/CreateWiki] 07redbluegreenhat 030c7c4f9 - Fix [11:38:58] !log [alex@test151] starting deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to test151 [11:38:59] !log [alex@test151] finished deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to test151 - SUCCESS in 1s [11:39:06] miraheze/CreateWiki - redbluegreenhat the build has errored. [11:39:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:39:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:40:31] !log [alex@mwtask181] starting deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to all [11:40:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:40:41] !log [alex@mwtask181] finished deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to all - SUCCESS in 10s [11:40:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:44:47] miraheze/CreateWiki - redbluegreenhat the build has errored. [11:48:04] [02miraheze/CreateWiki] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/0c7c4f938343...38c989ddf02b [11:48:07] [02miraheze/CreateWiki] 07redbluegreenhat 0338c989d - Fix indentation [11:48:08] !log [alex@test151] starting deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to test151 [11:48:09] !log [alex@test151] finished deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to test151 - SUCCESS in 1s [11:48:10] !log [alex@mwtask181] starting deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to all [11:48:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:48:21] !log [alex@mwtask181] finished deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to all - SUCCESS in 10s [11:48:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:48:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:48:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:56:38] miraheze/CreateWiki - redbluegreenhat the build passed. [12:05:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [12:06:11] it's sagan's import [15:32:11] https://meta.miraheze.org/wiki/Special:Log?logid=1066263 doesn't seem like this rename quite worked [15:32:38] https://meta.miraheze.org/wiki/Special:CentralAuth/Cielobot the global account is not attached to any wiki at all [15:33:01] https://meta.miraheze.org/wiki/Special:CentralAuth/Astrono-chan the account that was renamed has some leftover local accounts still [15:37:45] https://meta.miraheze.org/wiki/Special:GlobalRenameProgress/Cielobot that's taking an abnormal amount of time... [15:41:07] @bluemoon0332 want a lesson in Magic? [15:41:15] sure [15:41:54] @bluemoon0332 https://wikitech.wikimedia.org/wiki/Stuck_global_renames [15:42:08] 👀 [15:42:41] seems for now there's nothing to do but be patient [15:43:16] mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=loginwiki --logwiki=metawiki 'Astrono-chan' 'Cielobot' [15:43:24] Will do if I've got the usernames right [15:43:47] @bluemoon0332 you can check for jobs on loginwiki [15:43:50] Or any errors [15:44:10] `In case of doubt, ask the maintainers of CentralAuth.` Those exist? 😂 [15:44:38] @bluemoon0332 no [15:45:46] ik, just having a little fun [15:49:42] No one actually knows why renames sometimes get stuck [15:50:12] Sometimes somewhere it just forgets what it's doing [15:51:43] I'll take a look in a sec, see if the jobs are just not running or what's going on [16:05:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [16:06:26] !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/showJobs.php --wiki=loginwiki --type LocalRenameUserJob (END - exit=0) [16:06:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:07:25] !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/showJobs.php --wiki=loginwiki --type RenameUserJob (END - exit=0) [16:07:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:08:37] !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/showJobs.php --wiki=metawiki --type LocalRenameUserJob (END - exit=0) [16:08:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:09:16] !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/showJobs.php --wiki=metawiki --type RenameUserJob (END - exit=0) [16:09:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:11:06] @bluemoon0332 use --no-log for ones that don't write [16:12:43] Did @bluemoon0332 learn something new? [16:12:50] nothing at all [16:12:55] zero jobs on both wikis [16:13:03] @bluemoon0332 try the fix script [16:13:11] It's probably got amnesia [16:13:38] CentralAuth renames sometimes just forget to work [16:14:02] I have no other explanation [16:15:15] If the fix script doesn't work, then there's other more aggressive hammers [16:15:19] a quick query of localnames on mhglobal also reveals no modifications there [16:15:23] I'll try the fix script [16:15:29] ye it's probably just forgot to run [16:15:56] I got to go in a minute or 2 for like half an hour [16:16:53] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/initSiteStats.php --wiki=bugfableswiki --update (END - exit=0) [16:17:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:17:37] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/refreshLinks.php --wiki=bugfableswiki (START) [16:17:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:18:00] hmmm, I think I'll wait for a while before doing that just in case [16:19:05] the job should eventually error out if it's really stuck [16:20:04] then again, according to showJobs there's no job at all to begin with... [16:20:37] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/refreshLinks.php --wiki=bugfableswiki (END - exit=0) [16:20:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:38:21] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b3da2bceb00e...eeed877b2f58 [16:38:23] [02miraheze/puppet] 07Universal-Omega 03eeed877 - mediawiki: remove jobrunner_composer unconditionally [16:41:01] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/eeed877b2f58...55d20a50d3fc [16:41:04] [02miraheze/puppet] 07Universal-Omega 0355d20a5 - mediawiki: remove absented package [16:41:55] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/55d20a50d3fc...2a8840485148 [16:41:57] [02miraheze/puppet] 07Universal-Omega 032a88404 - mediawiki: remove unnecessary check [16:47:54] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03defined [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/4c9dda5b004b [16:47:55] [02miraheze/puppet] 07Universal-Omega 034c9dda5 - mediawiki::extensionsetup: add file defined check [16:47:58] [02puppet] 07Universal-Omega created branch 03defined - 13https://github.com/miraheze/puppet [16:48:00] [02puppet] 07Universal-Omega opened pull request 03#3796: mediawiki::extensionsetup: add file defined check - 13https://github.com/miraheze/puppet/pull/3796 [16:48:49] [02miraheze/puppet] 07github-actions[bot] pushed 031 commit to 03defined [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4c9dda5b004b...ee6ec918004e [16:48:50] [02miraheze/puppet] 07github-actions 03ee6ec91 - CI: lint puppet code to standards [16:48:52] [02puppet] 07github-actions[bot] synchronize pull request 03#3796: mediawiki::extensionsetup: add file defined check - 13https://github.com/miraheze/puppet/pull/3796 [16:50:15] [02puppet] 07Universal-Omega synchronize pull request 03#3796: mediawiki::extensionsetup: add file defined check - 13https://github.com/miraheze/puppet/pull/3796 [16:50:18] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03defined [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ee6ec918004e...ac6cc6a64592 [16:50:19] [02miraheze/puppet] 07Universal-Omega 03ac6cc6a - More checks [16:54:38] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03defined [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ac6cc6a64592...da24e82cd545 [16:54:40] [02miraheze/puppet] 07Universal-Omega 03da24e82 - Fix [16:54:42] [02puppet] 07Universal-Omega synchronize pull request 03#3796: mediawiki::extensionsetup: add file defined check - 13https://github.com/miraheze/puppet/pull/3796 [16:55:13] [02miraheze/puppet] 07github-actions[bot] pushed 031 commit to 03defined [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/da24e82cd545...0a6444268fb0 [16:55:16] [02miraheze/puppet] 07github-actions 030a64442 - CI: lint puppet code to standards [16:55:18] [02puppet] 07github-actions[bot] synchronize pull request 03#3796: mediawiki::extensionsetup: add file defined check - 13https://github.com/miraheze/puppet/pull/3796 [17:02:13] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03defined [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0a6444268fb0...9fd50adba4e8 [17:02:14] [02miraheze/puppet] 07Universal-Omega 039fd50ad - Fix [17:02:15] [02puppet] 07Universal-Omega synchronize pull request 03#3796: mediawiki::extensionsetup: add file defined check - 13https://github.com/miraheze/puppet/pull/3796 [17:02:51] [02miraheze/puppet] 07github-actions[bot] pushed 031 commit to 03defined [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9fd50adba4e8...5d7bb7682e03 [17:02:54] [02miraheze/puppet] 07github-actions 035d7bb76 - CI: lint puppet code to standards [17:02:56] [02puppet] 07github-actions[bot] synchronize pull request 03#3796: mediawiki::extensionsetup: add file defined check - 13https://github.com/miraheze/puppet/pull/3796 [17:04:19] PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:05:09] something seems to be wrong with loginwiki [17:05:50] `CentralAuthCreateLocalAccount` jobs have been piling up there [17:06:06] Back @bluemoon0332 [17:06:29] Try running them [17:07:25] !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=loginwiki (START) [17:07:26] !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=loginwiki (END - exit=0) [17:07:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:07:35] about that [17:07:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:07:46] as far as mwtask is concerned, loginwiki has no jobs in the queue [17:07:58] I suspect this is related to the wiki being switched over to Kafka [17:08:07] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03defined [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5d7bb7682e03...092d864af449 [17:08:10] [02miraheze/puppet] 07Universal-Omega 03092d864 - Ignore file [17:08:11] [02puppet] 07Universal-Omega synchronize pull request 03#3796: mediawiki::extensionsetup: add file defined check - 13https://github.com/miraheze/puppet/pull/3796 [17:08:23] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03defined [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/092d864af449...de948937f5ed [17:08:25] [02miraheze/puppet] 07Universal-Omega 03de94893 - Fix [17:08:26] [02puppet] 07Universal-Omega synchronize pull request 03#3796: mediawiki::extensionsetup: add file defined check - 13https://github.com/miraheze/puppet/pull/3796 [17:09:30] miraheze/puppet - Universal-Omega the build passed. [17:09:50] miraheze/puppet - Universal-Omega the build passed. [17:11:06] [02puppet] 07Universal-Omega synchronize pull request 03#3796: mediawiki::extensionsetup: add file defined check - 13https://github.com/miraheze/puppet/pull/3796 [17:11:07] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03defined [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/de948937f5ed...094004c2f397 [17:11:09] [02miraheze/puppet] 07Universal-Omega 03094004c - Ignore [17:12:21] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2a8840485148...7552acf870e2 [17:12:22] [02miraheze/puppet] 07Universal-Omega 037552acf - Fix parameter_order check [17:12:50] miraheze/puppet - Universal-Omega the build passed. [17:13:07] [02miraheze/puppet] 07Universal-Omega pushed 0310 commits to 03defined [+0/-0/±10] 13https://github.com/miraheze/puppet/compare/094004c2f397...9958a28cf7f7 [17:13:09] [02miraheze/puppet] 07Universal-Omega 033f7d141 - mediawiki::extensionsetup: add file defined check [17:13:10] [02miraheze/puppet] 07github-actions 038c6c1eb - CI: lint puppet code to standards [17:13:13] [02miraheze/puppet] 07Universal-Omega 0330185a1 - More checks [17:13:15] [02miraheze/puppet] ... and 7 more commits. [17:13:17] [02puppet] 07Universal-Omega synchronize pull request 03#3796: mediawiki::extensionsetup: add file defined check - 13https://github.com/miraheze/puppet/pull/3796 [17:15:23] miraheze/puppet - Universal-Omega the build passed. [17:26:25] Ah [17:26:27] Hmm [17:26:29] @cosmicalpha [17:26:52] @bluemoon0332 I don't think showJobs works with Kafka [17:27:25] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [17:32:13] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:35:17] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/WikiSEO/maintenance/GenerateDescription.php --wiki=bugfableswiki --force 0 (END - exit=0) [17:35:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:57:01] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03changeprop-parsoidCachePrewarm [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/12ec6c4e288e [17:57:01] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [17:57:04] [02miraheze/puppet] 07Universal-Omega 0312ec6c4 - changeprop: add parsoidCachePrewarm to high_traffic_jobs_config [17:57:05] [02puppet] 07Universal-Omega created branch 03changeprop-parsoidCachePrewarm - 13https://github.com/miraheze/puppet [17:57:06] [02puppet] 07Universal-Omega opened pull request 03#3797: changeprop: add parsoidCachePrewarm to high_traffic_jobs_config - 13https://github.com/miraheze/puppet/pull/3797 [18:21:58] [02miraheze/puppet] 07Universal-Omega deleted branch 03changeprop-parsoidCachePrewarm [18:22:00] [02puppet] 07Universal-Omega closed pull request 03#3797: changeprop: add parsoidCachePrewarm to high_traffic_jobs_config - 13https://github.com/miraheze/puppet/pull/3797 [18:22:03] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7552acf870e2...7494ad0bb4e1 [18:22:04] [02miraheze/puppet] 07Universal-Omega 037494ad0 - changeprop: add parsoidCachePrewarm to high_traffic_jobs_config (#3797) [18:22:07] [02puppet] 07Universal-Omega deleted branch 03changeprop-parsoidCachePrewarm - 13https://github.com/miraheze/puppet [18:43:22] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03cp [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/66e2d1c96653 [18:43:23] [02miraheze/puppet] 07Universal-Omega 0366e2d1c - changeprop: use statsd_exporter [18:43:25] [02puppet] 07Universal-Omega created branch 03cp - 13https://github.com/miraheze/puppet [18:43:26] [02puppet] 07Universal-Omega opened pull request 03#3798: changeprop: use statsd_exporter - 13https://github.com/miraheze/puppet/pull/3798 [18:44:03] [02miraheze/puppet] 07github-actions[bot] pushed 031 commit to 03cp [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/66e2d1c96653...7b0633012641 [18:44:05] [02miraheze/puppet] 07github-actions 037b06330 - CI: lint puppet code to standards [18:44:07] [02puppet] 07github-actions[bot] synchronize pull request 03#3798: changeprop: use statsd_exporter - 13https://github.com/miraheze/puppet/pull/3798 [18:44:12] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03cp [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7b0633012641...996285434304 [18:44:14] [02miraheze/puppet] 07Universal-Omega 039962854 - Update [18:44:15] [02puppet] 07Universal-Omega synchronize pull request 03#3798: changeprop: use statsd_exporter - 13https://github.com/miraheze/puppet/pull/3798 [18:44:53] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03cp [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/996285434304...7f395adcfaa4 [18:44:56] [02miraheze/puppet] 07Universal-Omega 037f395ad - Fix [18:44:58] [02puppet] 07Universal-Omega synchronize pull request 03#3798: changeprop: use statsd_exporter - 13https://github.com/miraheze/puppet/pull/3798 [18:45:05] [02puppet] 07Universal-Omega closed pull request 03#3798: changeprop: use statsd_exporter - 13https://github.com/miraheze/puppet/pull/3798 [18:45:06] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/7494ad0bb4e1...288bdf98b6a8 [18:45:08] [02miraheze/puppet] 07Universal-Omega 03288bdf9 - changeprop: use statsd_exporter (#3798) [18:45:10] [02puppet] 07Universal-Omega deleted branch 03cp - 13https://github.com/miraheze/puppet [18:45:12] [02miraheze/puppet] 07Universal-Omega deleted branch 03cp [19:05:18] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/288bdf98b6a8...ae532db2dac3 [19:05:19] [02miraheze/puppet] 07Universal-Omega 03ae532db - changeprop: add statsd exporter mappings [19:50:39] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ae532db2dac3...dc8ced3f0b22 [19:50:41] [02miraheze/puppet] 07Universal-Omega 03dc8ced3 - -defaults [19:52:38] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/dc8ced3f0b22...67209d37cb64 [19:52:40] [02miraheze/puppet] 07Universal-Omega 0367209d3 - Add default [20:00:56] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=sagan4alphawiki --search-recursively /home/reception/sagan4alpha --summary=Imported from sagan4alpha.wikitide.org (END - exit=0) [20:01:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:01:33] \o/ [20:05:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [20:14:11] !log ran initSiteStats.php for gundamfrwiki [20:14:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:41:42] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 9709MiB (10% inode=98%); [21:12:51] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/67209d37cb64...c96a396a10d8 [21:12:52] [02miraheze/puppet] 07Universal-Omega 03c96a396 - Add ignore [21:13:35] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03cp-cluster [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/bb11cf40aa4d [21:13:37] [02miraheze/puppet] 07Universal-Omega 03bb11cf4 - changeprop: add support for clustering [21:13:39] [02puppet] 07Universal-Omega created branch 03cp-cluster - 13https://github.com/miraheze/puppet [21:13:40] [02puppet] 07Universal-Omega opened pull request 03#3799: changeprop: add support for clustering - 13https://github.com/miraheze/puppet/pull/3799 [21:14:10] miraheze/puppet - Universal-Omega the build passed. [21:18:48] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c96a396a10d8...d29b9e96655c [21:18:49] [02miraheze/puppet] 07Universal-Omega 03d29b9e9 - Remove mediawiki.job.ThumbnailRender from exclude [21:20:04] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d29b9e96655c...404b1538e13c [21:20:06] [02miraheze/puppet] 07Universal-Omega 03404b153 - changeprop: set low_traffic_concurrency to 5 [21:29:24] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.402 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [21:32:45] PROBLEM - cp36 Disk Space on cp36 is WARNING: DISK WARNING - free space: / 9691MiB (10% inode=98%); [21:57:40] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/404b1538e13c...52a185e1dd8c [21:57:41] [02miraheze/puppet] 07Universal-Omega 0352a185e - Eventgate: update config [21:59:02] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [22:00:42] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/52a185e1dd8c...d797f2174218 [22:00:43] [02miraheze/puppet] 07Universal-Omega 03d797f21 - - [22:39:29] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d797f2174218...caab624d27b0 [22:39:31] [02miraheze/puppet] 07Universal-Omega 03caab624 - prometheus::exporter::statsd_exporter: add option to disable defaults [22:40:34] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/caab624d27b0...9cd4fd18793d [22:40:35]