[00:03:42] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 19557MiB (22% inode=98%); [00:04:25] RECOVERY - cp41 Disk Space on cp41 is OK: DISK OK - free space: / 21508MiB (23% inode=98%); [00:04:45] RECOVERY - cp36 Disk Space on cp36 is OK: DISK OK - free space: / 21411MiB (24% inode=98%); [00:05:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [00:27:00] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 58% [00:49:05] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [01:23:06] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/6beacd789414...a502c24af9ee [01:23:08] [02miraheze/mw-config] 07Universal-Omega 03a502c24 - Switch all Wikibase wikis to use EventBus jobqueue [01:23:26] !log [@test151] starting deploy of {'config': True} to test151 [01:23:27] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [01:23:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:23:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:23:51] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [01:23:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:24:01] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 10s [01:24:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:24:20] miraheze/mw-config - Universal-Omega the build passed. [01:32:56] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a502c24af9ee...22e4b6ee85a0 [01:32:57] [02miraheze/mw-config] 07Universal-Omega 0322e4b6e - Add jobrunner timeout [01:33:50] miraheze/mw-config - Universal-Omega the build passed. [01:34:34] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [01:34:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:34:44] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 10s [01:34:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:37:43] !log [@mwtask171] starting deploy of {'config': True} to all [01:37:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:37:51] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s [01:37:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:53:51] !log [@test151] starting deploy of {'config': True} to test151 [01:53:52] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [01:53:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:54:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:08:15] [02miraheze/puppet] 07Universal-Omega pushed 0314 commits to 03defined [+0/-0/±20] 13https://github.com/miraheze/puppet/compare/9958a28cf7f7...6f757e6443e2 [02:08:16] [02miraheze/puppet] 07Universal-Omega 036f757e6 - Merge branch 'master' into defined [02:08:18] [02puppet] 07Universal-Omega synchronize pull request 03#3796: mediawiki::extensionsetup: add file defined check - 13https://github.com/miraheze/puppet/pull/3796 [02:08:30] [02puppet] 07Universal-Omega closed pull request 03#3796: mediawiki::extensionsetup: add file defined check - 13https://github.com/miraheze/puppet/pull/3796 [02:08:32] [02miraheze/puppet] 07Universal-Omega deleted branch 03defined [02:08:33] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/facc5ba45d13...eaa4aae88e2d [02:08:35] [02miraheze/puppet] 07Universal-Omega 03eaa4aae - mediawiki::extensionsetup: add file defined check (#3796) [02:08:37] [02puppet] 07Universal-Omega deleted branch 03defined - 13https://github.com/miraheze/puppet [02:09:40] [02miraheze/puppet] 07Universal-Omega pushed 0310 commits to 03cp-cluster [+0/-0/±14] 13https://github.com/miraheze/puppet/compare/bb11cf40aa4d...3456f99fd2ee [02:09:42] [02miraheze/puppet] 07Universal-Omega 033456f99 - Merge branch 'master' into cp-cluster [02:09:43] [02puppet] 07Universal-Omega synchronize pull request 03#3799: changeprop: add support for clustering - 13https://github.com/miraheze/puppet/pull/3799 [02:11:38] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03revert-3796-defined [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/74e2b96817cc [02:11:40] [02miraheze/puppet] 07Universal-Omega 0374e2b96 - Revert "mediawiki::extensionsetup: add file defined check (#3796)" [02:11:41] [02puppet] 07Universal-Omega created branch 03revert-3796-defined - 13https://github.com/miraheze/puppet [02:11:43] [02puppet] 07Universal-Omega opened pull request 03#3800: Revert "mediawiki::extensionsetup: add file defined check" - 13https://github.com/miraheze/puppet/pull/3800 [02:11:47] [02puppet] 07Universal-Omega closed pull request 03#3800: Revert "mediawiki::extensionsetup: add file defined check" - 13https://github.com/miraheze/puppet/pull/3800 [02:11:48] [02miraheze/puppet] 07Universal-Omega deleted branch 03revert-3796-defined [02:11:49] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/eaa4aae88e2d...4b1cd9cc11fd [02:11:51] [02miraheze/puppet] 07Universal-Omega 034b1cd9c - Revert "mediawiki::extensionsetup: add file defined check" (#3800) [02:11:53] [02puppet] 07Universal-Omega deleted branch 03revert-3796-defined - 13https://github.com/miraheze/puppet [02:14:26] [02puppet] 07Universal-Omega synchronize pull request 03#3799: changeprop: add support for clustering - 13https://github.com/miraheze/puppet/pull/3799 [02:14:28] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03cp-cluster [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3456f99fd2ee...99678a6d8da8 [02:14:30] [02miraheze/puppet] 07Universal-Omega 0399678a6 - Update [02:17:53] [02puppet] 07Universal-Omega synchronize pull request 03#3799: changeprop: add support for clustering - 13https://github.com/miraheze/puppet/pull/3799 [02:17:54] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03cp-cluster [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/99678a6d8da8...072912773145 [02:17:56] [02miraheze/puppet] 07Universal-Omega 030729127 - Fix [02:20:19] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03cp-cluster [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/072912773145...9aa4a50ffa01 [02:20:20] [02miraheze/puppet] 07Universal-Omega 039aa4a50 - Add [02:20:22] [02puppet] 07Universal-Omega synchronize pull request 03#3799: changeprop: add support for clustering - 13https://github.com/miraheze/puppet/pull/3799 [02:21:40] [02puppet] 07Universal-Omega synchronize pull request 03#3799: changeprop: add support for clustering - 13https://github.com/miraheze/puppet/pull/3799 [02:21:42] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03cp-cluster [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9aa4a50ffa01...02c7c0e7ce21 [02:21:44] [02miraheze/puppet] 07Universal-Omega 0302c7c0e - - [02:29:53] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [02:34:13] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4b1cd9cc11fd...dc9410793538 [02:34:13] [02miraheze/puppet] 07Universal-Omega 03dc94107 - changeprop: add ThumbnailRender to high traffic jobs [02:37:45] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/dc9410793538...39eb05cf2dc6 [02:37:48] [02miraheze/puppet] 07Universal-Omega 0339eb05c - changeprop: add smw.changePropagationClassUpdate to high traffic jobs [02:42:37] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/39eb05cf2dc6...090f17bfa711 [02:42:40] [02miraheze/puppet] 07Universal-Omega 03090f17b - changeprop: add smw.changePropagationDispatch to high traffic jobs [02:54:09] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/090f17bfa711...cab373fca6ea [02:54:11] [02miraheze/puppet] 07Universal-Omega 03cab373f - changeprop: add cirrusSearchElasticaWrite to high traffic jobs [02:57:10] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/22e4b6ee85a0...1acdceaf1f1d [02:57:11] [02miraheze/mw-config] 07Universal-Omega 031acdcea - Use EventBus JobQueue on all CirrusSearch wikis except metawiki [02:57:28] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [02:57:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:57:38] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 10s [02:57:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:58:05] miraheze/mw-config - Universal-Omega the build passed. [02:59:29] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [03:07:04] !log [@mwtask171] starting deploy of {'config': True} to all [03:07:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:07:11] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s [03:07:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:10:03] PROBLEM - mw162 Puppet on mw162 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[libboost-regex1.74.0-icu72] [03:13:28] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/cab373fca6ea...4df0fb8d7660 [03:13:29] [02miraheze/puppet] 07Universal-Omega 034df0fb8 - changeprop: add refreshLinks to high traffic jobs [03:16:25] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/1acdceaf1f1d...f9538f18b90e [03:16:26] [02miraheze/mw-config] 07Universal-Omega 03f9538f1 - Enable EventBus job queue on all wikis except metawiki [03:16:55] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [03:17:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:17:05] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 10s [03:17:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:17:25] miraheze/mw-config - Universal-Omega the build passed. [03:23:05] !log [@test151] starting deploy of {'config': True} to test151 [03:23:06] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [03:23:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:23:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:37:24] !log [@mwtask171] starting deploy of {'config': True} to all [03:37:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:37:33] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s [03:37:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:38:03] RECOVERY - mw162 Puppet on mw162 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [03:46:17] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4df0fb8d7660...6cc2787714c6 [03:46:19] [02miraheze/puppet] 07Universal-Omega 036cc2787 - changeprop: add priority queues [03:47:57] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6cc2787714c6...207b411862ae [03:47:58] [02miraheze/puppet] 07Universal-Omega 03207b411 - changeprop: add LocalGlobalUserPageCacheUpdateJob to high traffic jobs [03:53:47] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/207b411862ae...7728616c7582 [03:53:48] [02miraheze/puppet] 07Universal-Omega 037728616 - changeprop: add Translate queues [03:57:58] PROBLEM - changeprop151 NTP time on changeprop151 is WARNING: NTP WARNING: Offset 0.2866491079 secs [03:58:23] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f9538f18b90e...4a7899e99afa [03:58:24] [02miraheze/mw-config] 07Universal-Omega 034a7899e - Enable EventBus JobQueue on metawiki [03:58:52] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [03:59:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:59:02] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s [03:59:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:59:15] miraheze/mw-config - Universal-Omega the build has errored. [04:05:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [04:07:02] !log [@mwtask171] starting deploy of {'config': True} to all [04:07:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:07:10] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s [04:07:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:15:35] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [04:20:02] @bluemoon0332 I fixed the global account issues btw, the issue was they happened to get stuck in the old redis job queue when I switched over so were still there so I re-enabled the redis job queue on mwtask1i1 and cleared it with runJobs then switched back to Kafka and the old redis queue is empty now. [04:21:33] I assume it may have been with metawiki and loginwiki being on different queues but now everything is switched to Kafka [04:23:33] !log [@test151] starting deploy of {'config': True} to test151 [04:23:34] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [04:23:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:23:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:29:58] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7728616c7582...ed2fdc02c727 [04:30:00] [02miraheze/puppet] 07Universal-Omega 03ed2fdc0 - changeprop: lower some concurrency [04:39:18] PROBLEM - changeprop151 NTP time on changeprop151 is CRITICAL: NTP CRITICAL: Offset 0.5005663335 secs [04:48:31] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/4a7899e99afa...a1fc0299a1d8 [04:48:34] [02miraheze/mw-config] 07Universal-Omega 03a1fc029 - Add JobExecutor logging [04:49:30] miraheze/mw-config - Universal-Omega the build has errored. [04:49:38] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a1fc0299a1d8...899e28e68c5c [04:49:41] [02miraheze/mw-config] 07Universal-Omega 03899e28e - Enable $wgShowExceptionDetails on mwtask171 also [04:50:34] miraheze/mw-config - Universal-Omega the build has errored. [04:53:14] !log [@test151] starting deploy of {'config': True} to test151 [04:53:15] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [04:53:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:53:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:02:43] !log [@mwtask181] starting deploy of {'config': True} to all [05:02:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:02:52] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 9s [05:03:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:04:20] PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/local/sbin/reload-vcl] [05:07:38] !log [@mwtask171] starting deploy of {'config': True} to all [05:07:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:07:47] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s [05:07:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:29:34] [02jobrunner-service] 07Universal-Omega closed pull request 03#14: Update jobrunner-service - 13https://github.com/miraheze/jobrunner-service/pull/14 [05:29:37] [02miraheze/jobrunner-service] 07Universal-Omega pushed 031 commit to 03miraheze [+9/-3/±7] 13https://github.com/miraheze/jobrunner-service/compare/4f1bc6cb7100...8592ee0b17ad [05:29:40] [02miraheze/jobrunner-service] 07Universal-Omega 038592ee0 - Update jobrunner-service (#14) [05:29:41] [02miraheze/jobrunner-service] 07Universal-Omega deleted branch 03update [05:29:43] [02jobrunner-service] 07Universal-Omega deleted branch 03update - 13https://github.com/miraheze/jobrunner-service [05:30:20] miraheze/jobrunner-service - Universal-Omega the build passed. [05:32:15] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:45:33] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [06:30:48] [02miraheze/mw-config] 07Universal-Omega pushed 039 commits to 03jobqueue [+0/-0/±12] 13https://github.com/miraheze/mw-config/compare/e739e7ada5c2...4d7a1f8b826b [06:30:51] [02miraheze/mw-config] 07Universal-Omega 034d7a1f8 - Merge branch 'master' into jobqueue [06:30:54] [02mw-config] 07Universal-Omega synchronize pull request 03#5514: Enable EventBus JobQueue on all wikis - 13https://github.com/miraheze/mw-config/pull/5514 [06:31:22] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03jobqueue [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/4d7a1f8b826b...a79118edd84b [06:31:24] [02miraheze/mw-config] 07github-actions 03a79118e - CI: lint code to MediaWiki standards [06:31:25] [02mw-config] 07github-actions[bot] synchronize pull request 03#5514: Enable EventBus JobQueue on all wikis - 13https://github.com/miraheze/mw-config/pull/5514 [06:32:00] miraheze/mw-config - Universal-Omega the build passed. [06:35:16] [02mw-config] 07Universal-Omega edited pull request 03#5514: Unconditionally enable EventBus JobQueue on all wikis - 13https://github.com/miraheze/mw-config/pull/5514 [06:35:27] [02mw-config] 07Universal-Omega closed pull request 03#5514: Unconditionally enable EventBus JobQueue on all wikis - 13https://github.com/miraheze/mw-config/pull/5514 [06:35:28] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±4] 13https://github.com/miraheze/mw-config/compare/899e28e68c5c...f8cdbda3417d [06:35:31] [02miraheze/mw-config] 07Universal-Omega 03f8cdbda - Unconditionally enable EventBus JobQueue on all wikis (#5514) [06:35:33] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [06:35:33] [02miraheze/mw-config] 07Universal-Omega deleted branch 03jobqueue [06:35:34] [02mw-config] 07Universal-Omega deleted branch 03jobqueue - 13https://github.com/miraheze/mw-config [06:35:40] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [06:35:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:35:51] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 10s [06:35:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:36:30] miraheze/mw-config - Universal-Omega the build passed. [06:37:04] !log [@mwtask171] starting deploy of {'config': True} to all [06:37:06] [02puppet] 07Universal-Omega closed pull request 03#3795: mwtask181: enable use of cpjobqueue - 13https://github.com/miraheze/puppet/pull/3795 [06:37:07] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ed2fdc02c727...61aa2cacba3d [06:37:08] [02miraheze/puppet] 07Universal-Omega 0361aa2ca - mwtask181: enable use of cpjobqueue (#3795) [06:37:11] [02miraheze/puppet] 07Universal-Omega deleted branch 03cpjobqueue [06:37:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:37:13] [02puppet] 07Universal-Omega deleted branch 03cpjobqueue - 13https://github.com/miraheze/puppet [06:37:13] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 9s [06:37:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:37:47] [02dns] 07Universal-Omega closed pull request 03#503: Add mwtask181 to jobrunner pool - 13https://github.com/miraheze/dns/pull/503 [06:37:50] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/aa5a335a3b20...840278a9ff44 [06:37:51] [02miraheze/dns] 07Universal-Omega 03840278a - Add mwtask181 to jobrunner pool (#503) [06:37:53] [02miraheze/dns] 07Universal-Omega deleted branch 03mwtask181-jobrunner [06:37:55] [02dns] 07Universal-Omega deleted branch 03mwtask181-jobrunner - 13https://github.com/miraheze/dns [06:40:54] PROBLEM - mwtask181 JobRunner Service on mwtask181 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [06:42:24] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f8cdbda3417d...854544120367 [06:42:27] [02miraheze/mw-config] 07Universal-Omega 038545441 - FileBackend: use rdb151 for redisLockManager [06:42:44] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [06:42:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:42:54] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 10s [06:43:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:43:20] miraheze/mw-config - Universal-Omega the build passed. [06:53:15] !log [@test151] starting deploy of {'config': True} to test151 [06:53:16] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [06:53:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:54:36] PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw152.wikitide.net port 443 after 0 ms: Couldn't connect to server [06:54:39] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 7 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw182 [06:54:40] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 7 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw182 [06:54:45] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 9 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mediawiki [06:54:58] PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw151.wikitide.net port 443 after 0 ms: Couldn't connect to server [06:54:59] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 9 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mediawiki [06:55:12] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: connect to address 10.0.15.115 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [06:55:24] PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw162.wikitide.net port 443 after 0 ms: Couldn't connect to server [06:55:29] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: connect to address 10.0.16.133 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [06:55:31] @cosmicalpha everything alerting [06:55:42] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: connect to address 10.0.16.132 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [06:55:45] RECOVERY - test151 ferm_active on test151 is OK: OK ferm input default policy is set [06:55:51] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: connect to address 10.0.15.114 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [06:55:54] !log [@mwtask171] starting deploy of {'config': True} to all [06:55:55] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw161.wikitide.net port 443 after 0 ms: Couldn't connect to server [06:56:01] PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 9 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mediawiki [06:56:07] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: connect to address 10.0.17.122 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [06:56:15] PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [06:56:18] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw172.wikitide.net port 443 after 0 ms: Couldn't connect to server [06:56:30] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 9 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mediawiki [06:56:38] PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw171.wikitide.net port 443 after 0 ms: Couldn't connect to server [06:56:44] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [06:56:50] PROBLEM - cp27 HTTPS on cp27 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [06:56:52] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [06:56:55] PROBLEM - mw151 Puppet on mw151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [06:56:57] PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw181.wikitide.net port 443 after 0 ms: Couldn't connect to server [06:57:01] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: connect to address 10.0.18.105 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [06:57:01] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [06:57:01] PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw182.wikitide.net port 443 after 1 ms: Couldn't connect to server [06:57:04] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: connect to address 10.0.18.104 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [06:57:11] PROBLEM - mwtask181 HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mwtask181.wikitide.net port 443 after 0 ms: Couldn't connect to server [06:57:14] PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: connect to address 10.0.18.106 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [06:57:23] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: connect to address 10.0.17.123 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [06:57:26] PROBLEM - mwtask171 HTTPS on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mwtask171.wikitide.net port 443 after 0 ms: Couldn't connect to server [06:57:30] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [06:57:53] PROBLEM - mw152 Puppet on mw152 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [06:58:03] PROBLEM - mw162 Puppet on mw162 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [06:58:09] PROBLEM - mw172 Puppet on mw172 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [06:58:09] PROBLEM - mw161 Puppet on mw161 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [06:58:09] PROBLEM - test151 HTTPS on test151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to test151.wikitide.net port 443 after 4 ms: Couldn't connect to server [06:58:17] PROBLEM - mw182 Puppet on mw182 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 32 seconds ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [06:58:22] PROBLEM - test151 JobChron Service on test151 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobChronService' [06:58:30] PROBLEM - mwtask171 MediaWiki Rendering on mwtask171 is CRITICAL: connect to address 10.0.17.144 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [06:58:42] PROBLEM - mw171 Puppet on mw171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [06:58:50] PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: connect to address 10.0.15.118 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [06:59:49] PROBLEM - mw181 Puppet on mw181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [07:25:28] PROBLEM - changeprop151 Puppet on changeprop151 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 5 minutes ago with 0 failures [07:25:45] PROBLEM - changeprop151 changeprop on changeprop151 is CRITICAL: connect to address 10.0.15.148 and port 7200: Connection refused [07:27:45] RECOVERY - changeprop151 changeprop on changeprop151 is OK: TCP OK - 0.000 second response time on 10.0.15.148 port 7200 [07:39:45] PROBLEM - changeprop151 changeprop on changeprop151 is CRITICAL: connect to address 10.0.15.148 and port 7200: Connection refused [07:47:45] RECOVERY - changeprop151 changeprop on changeprop151 is OK: TCP OK - 0.000 second response time on 10.0.15.148 port 7200 [07:55:45] PROBLEM - changeprop151 changeprop on changeprop151 is CRITICAL: connect to address 10.0.15.148 and port 7200: Connection refused [07:57:45] RECOVERY - changeprop151 changeprop on changeprop151 is OK: TCP OK - 0.000 second response time on 10.0.15.148 port 7200 [08:05:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [08:09:28] RECOVERY - changeprop151 Puppet on changeprop151 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [08:10:09] PROBLEM - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is WARNING: WARNING - NGINX Error Rate is 40% [08:16:09] RECOVERY - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is OK: OK - NGINX Error Rate is 22% [08:20:09] PROBLEM - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is WARNING: WARNING - NGINX Error Rate is 42% [08:26:09] RECOVERY - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is OK: OK - NGINX Error Rate is 29% [08:32:09] PROBLEM - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is WARNING: WARNING - NGINX Error Rate is 57% [08:36:02] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/037c844bc9f1...febd3ed57037 [08:36:04] [02miraheze/ssl] 07Universal-Omega 03febd3ed - Remove cert causing nginx to fail starting [08:37:00] miraheze/ssl - Universal-Omega the build has errored. [08:38:32] RECOVERY - mwtask171 MediaWiki Rendering on mwtask171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.739 second response time [08:39:05] RECOVERY - mw161 Puppet on mw161 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [08:39:08] RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3615 bytes in 6.513 second response time [08:39:26] RECOVERY - mwtask171 HTTPS on mwtask171 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 285 bytes in 0.087 second response time [08:39:33] PROBLEM - puppet181 Puppet on puppet181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_srv-ssl] [08:39:45] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.567 second response time [08:39:56] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 285 bytes in 1.160 second response time [08:40:09] RECOVERY - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is OK: OK - NGINX Error Rate is 32% [08:40:53] RECOVERY - cp26 HTTPS on cp26 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3613 bytes in 6.358 second response time [08:41:24] RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 285 bytes in 0.211 second response time [08:41:24] RECOVERY - mw151 Puppet on mw151 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [08:41:25] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.688 second response time [08:41:30] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.490 second response time [08:41:33] RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:41:52] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.804 second response time [08:41:59] RECOVERY - mw152 Puppet on mw152 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [08:42:03] RECOVERY - mw162 Puppet on mw162 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [08:42:07] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.247 second response time [08:42:09] RECOVERY - mw172 Puppet on mw172 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [08:42:15] RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3615 bytes in 0.063 second response time [08:42:17] RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [08:42:18] RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 285 bytes in 0.069 second response time [08:42:30] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 19 backends are healthy [08:42:34] RECOVERY - cp51 HTTPS on cp51 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3637 bytes in 1.173 second response time [08:42:36] RECOVERY - mw152 HTTPS on mw152 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 285 bytes in 0.066 second response time [08:42:38] RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 285 bytes in 0.064 second response time [08:42:39] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy [08:42:39] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 19 backends are healthy [08:42:42] RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:42:45] RECOVERY - cp41 HTTPS on cp41 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3637 bytes in 0.828 second response time [08:42:50] RECOVERY - cp27 HTTPS on cp27 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3635 bytes in 1.059 second response time [08:42:56] RECOVERY - mw181 Puppet on mw181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:42:57] RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 285 bytes in 0.070 second response time [08:42:58] RECOVERY - mw151 HTTPS on mw151 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 285 bytes in 0.066 second response time [08:42:59] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [08:43:01] RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 285 bytes in 0.067 second response time [08:43:01] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.274 second response time [08:43:04] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.292 second response time [08:43:11] RECOVERY - mwtask181 HTTPS on mwtask181 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 285 bytes in 0.254 second response time [08:43:12] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.249 second response time [08:43:14] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [08:43:16] RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.851 second response time [08:44:01] RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy [08:53:45] RECOVERY - test151 HTTPS on test151 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 285 bytes in 0.146 second response time [08:54:49] RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.175 second response time [09:06:22] I'm not sure why this ping didn't alert me.. [09:06:33] It was my emails that did... [09:14:52] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/61aa2cacba3d...a82f2ed363f4 [09:14:54] [02miraheze/puppet] 07Universal-Omega 03a82f2ed - changeprop: temp lower parsoidCachePrewarm concurrency to 1 [09:41:00] !log mhglobal: INSERT INTO localnames (ln_wiki, ln_name) VALUES ("kylariswiki", "BobOrson"); [09:41:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:44:15] !log [alex@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CentralAuth/maintenance/migraceAccount.php --wiki=kylariswiki --username=BobOrson --auto --attachmissing --attachbroken --homewiki=metawiki (END - exit=256) [09:44:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:44:36] !log [alex@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CentralAuth/maintenance/migrateAccount.php --wiki=kylariswiki --username=BobOrson --auto --attachmissing --attachbroken --homewiki=metawiki (END - exit=0) [09:44:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:45:49] !log [alex@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CentralAuth/maintenance/attachAccount.php --wiki=--wiki=kylariswiki --userlist=/home/alex/BobOrson.txt (END - exit=65280) [09:45:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:46:08] !log [alex@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CentralAuth/maintenance/attachAccount.php --wiki=kylariswiki --userlist=/home/alex/BobOrson.txt (END - exit=0) [09:46:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:34:09] PROBLEM - changeprop151 APT on changeprop151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:36:21] RECOVERY - changeprop151 APT on changeprop151 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [11:26:05] PROBLEM - changeprop151 APT on changeprop151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:28:00] RECOVERY - changeprop151 APT on changeprop151 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [11:40:32] [02CreateWiki] 07redbluegreenhat opened pull request 03#494: Allow users with permission to query suppressed wiki requests through… - 13https://github.com/miraheze/CreateWiki/pull/494 [11:40:44] [02CreateWiki] 07redbluegreenhat edited pull request 03#494: Allow users with permission to query suppressed wiki requests through the API - 13https://github.com/miraheze/CreateWiki/pull/494 [11:40:52] [02CreateWiki] 07redbluegreenhat edited pull request 03#494: Allow users with permission to query suppressed wiki requests through the API - 13https://github.com/miraheze/CreateWiki/pull/494 [11:47:57] miraheze/CreateWiki - redbluegreenhat the build passed. [11:54:25] [02CreateWiki] 07redbluegreenhat synchronize pull request 03#494: Allow users with permission to query suppressed wiki requests through the API - 13https://github.com/miraheze/CreateWiki/pull/494 [12:00:47] miraheze/CreateWiki - redbluegreenhat the build passed. [12:05:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [12:08:05] [02CreateWiki] 07redbluegreenhat closed pull request 03#494: Allow users with permission to query suppressed wiki requests through the API - 13https://github.com/miraheze/CreateWiki/pull/494 [12:08:08] [02miraheze/CreateWiki] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/38c989ddf02b...9d0dcca93e10 [12:08:10] [02miraheze/CreateWiki] 07redbluegreenhat 039d0dcca - Allow users with permission to query suppressed wiki requests through the API (#494) [12:08:42] !log [alex@test151] starting deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to test151 [12:08:43] !log [alex@test151] finished deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to test151 - SUCCESS in 1s [12:08:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:08:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:15:23] miraheze/CreateWiki - redbluegreenhat the build passed. [12:21:09] [02CreateWiki] 07redbluegreenhat opened pull request 03#495: Switch timestamps to TS_ISO_8601 - 13https://github.com/miraheze/CreateWiki/pull/495 [12:21:14] [02CreateWiki] 07redbluegreenhat closed pull request 03#495: Switch timestamps to TS_ISO_8601 - 13https://github.com/miraheze/CreateWiki/pull/495 [12:21:17] [02miraheze/CreateWiki] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/9d0dcca93e10...0c95d0d8659e [12:21:18] !log [alex@test151] starting deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to test151 [12:21:19] [02miraheze/CreateWiki] 07redbluegreenhat 030c95d0d - Switch timestamps to TS_ISO_8601 (#495) [12:21:20] !log [alex@test151] finished deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to test151 - SUCCESS in 1s [12:21:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:21:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:27:44] miraheze/CreateWiki - redbluegreenhat the build passed. [12:28:34] miraheze/CreateWiki - redbluegreenhat the build passed. [12:32:39] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [12:43:48] PROBLEM - changeprop151 APT on changeprop151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:49:18] RECOVERY - changeprop151 APT on changeprop151 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [12:59:03] [02CreateWiki] 07redbluegreenhat opened pull request 03#496: Fix permissions check - 13https://github.com/miraheze/CreateWiki/pull/496 [12:59:16] [02CreateWiki] 07redbluegreenhat closed pull request 03#496: Fix permissions check - 13https://github.com/miraheze/CreateWiki/pull/496 [12:59:18] [02miraheze/CreateWiki] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/0c95d0d8659e...6bc0685c65bc [12:59:20] [02miraheze/CreateWiki] 07redbluegreenhat 036bc0685 - Fix permissions check (#496) [13:00:09] !log [alex@test151] starting deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to test151 [13:00:10] !log [alex@test151] finished deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to test151 - SUCCESS in 1s [13:00:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:00:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:01:52] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [13:05:19] miraheze/CreateWiki - redbluegreenhat the build passed. [13:06:00] miraheze/CreateWiki - redbluegreenhat the build passed. [13:13:48] PROBLEM - changeprop151 APT on changeprop151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:18:39] RECOVERY - changeprop151 APT on changeprop151 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:25:27] PROBLEM - changeprop151 APT on changeprop151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:27:22] RECOVERY - changeprop151 APT on changeprop151 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:57:42] [02miraheze/ImportDump] 07dependabot[bot] pushed 031 commit to 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/commit/1acfe57d6b8c [13:57:43] [02miraheze/ImportDump] 07dependabot[bot] 031acfe57 - Update php-parallel-lint/php-parallel-lint requirement [13:57:46] [02ImportDump] 07dependabot[bot] created branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 - 13https://github.com/miraheze/ImportDump [13:57:49] [02ImportDump] 07dependabot[bot] opened pull request 03#91: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/ImportDump/pull/91 [13:57:52] [02ImportDump] 07dependabot[bot] labeled pull request 03#91: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/ImportDump/pull/91 [13:57:55] [02ImportDump] 07dependabot[bot] labeled pull request 03#91: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/ImportDump/pull/91 [14:05:52] miraheze/ImportDump - dependabot[bot] the build passed. [14:17:46] [02mw-config] 07redbluegreenhat opened pull request 03#5515: T11999: Disable RequestWikiQueue outside of the meta wikis - 13https://github.com/miraheze/mw-config/pull/5515 [14:18:44] miraheze/mw-config - redbluegreenhat the build passed. [14:19:37] [02mw-config] 07redbluegreenhat closed pull request 03#5515: T11999: Disable RequestWikiQueue outside of the meta wikis - 13https://github.com/miraheze/mw-config/pull/5515 [14:19:39] [02miraheze/mw-config] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/854544120367...e968fc2c7161 [14:19:40] [02miraheze/mw-config] 07redbluegreenhat 03e968fc2 - T11999: Disable RequestWikiQueue outside of the meta wikis (#5515) [14:20:35] miraheze/mw-config - redbluegreenhat the build passed. [14:20:36] !log [alex@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [14:20:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:20:46] !log [alex@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s [14:20:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:21:25] [02miraheze/mw-config] 07redbluegreenhat pushed 031 commit to 03revert-5515-T11999 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/5f602bcbf419 [14:21:26] [02miraheze/mw-config] 07redbluegreenhat 035f602bc - Revert "T11999: Disable RequestWikiQueue outside of the meta wikis (#5515)" [14:21:28] [02mw-config] 07redbluegreenhat created branch 03revert-5515-T11999 - 13https://github.com/miraheze/mw-config [14:21:29] [02mw-config] 07redbluegreenhat opened pull request 03#5516: Revert "T11999: Disable RequestWikiQueue outside of the meta wikis" - 13https://github.com/miraheze/mw-config/pull/5516 [14:21:34] [02mw-config] 07redbluegreenhat closed pull request 03#5516: Revert "T11999: Disable RequestWikiQueue outside of the meta wikis" - 13https://github.com/miraheze/mw-config/pull/5516 [14:21:36] [02miraheze/mw-config] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/e968fc2c7161...40896dd9574e [14:21:37] !log [alex@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [14:21:37] [02miraheze/mw-config] 07redbluegreenhat 0340896dd - Revert "T11999: Disable RequestWikiQueue outside of the meta wikis" (#5516) [14:21:40] [02miraheze/mw-config] 07redbluegreenhat deleted branch 03revert-5515-T11999 [14:21:41] [02mw-config] 07redbluegreenhat deleted branch 03revert-5515-T11999 - 13https://github.com/miraheze/mw-config [14:21:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:21:46] !log [alex@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s [14:21:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:22:24] miraheze/mw-config - redbluegreenhat the build passed. [14:22:33] miraheze/mw-config - redbluegreenhat the build passed. [14:23:41] [02miraheze/mw-config] 07dependabot[bot] pushed 031 commit to 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/fa35aa41d5c7 [14:23:44] [02miraheze/mw-config] 07dependabot[bot] 03fa35aa4 - Update php-parallel-lint/php-parallel-lint requirement [14:23:45] [02mw-config] 07dependabot[bot] created branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 - 13https://github.com/miraheze/mw-config [14:23:46] [02mw-config] 07dependabot[bot] labeled pull request 03#5517: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/mw-config/pull/5517 [14:23:49] [02mw-config] 07dependabot[bot] opened pull request 03#5517: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/mw-config/pull/5517 [14:23:50] [02mw-config] 07dependabot[bot] labeled pull request 03#5517: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/mw-config/pull/5517 [14:24:33] miraheze/mw-config - dependabot[bot] the build passed. [14:27:44] PROBLEM - kinnikupedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'kinnikupedia.org' expires in 15 day(s) (Fri 12 Apr 2024 02:08:19 PM GMT +0000). [14:34:30] [02mw-config] 07redbluegreenhat opened pull request 03#5518: T11999: Disable Special:RequestWikiQueue outside of metawiki - 13https://github.com/miraheze/mw-config/pull/5518 [14:35:10] [02mw-config] 07redbluegreenhat closed pull request 03#5518: T11999: Disable Special:RequestWikiQueue outside of metawiki - 13https://github.com/miraheze/mw-config/pull/5518 [14:35:13] [02miraheze/mw-config] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/40896dd9574e...fb3e68bcef45 [14:35:15] [02miraheze/mw-config] 07redbluegreenhat 03fb3e68b - T11999: Disable Special:RequestWikiQueue outside of metawiki (#5518) [14:35:16] !log [alex@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [14:35:22] miraheze/mw-config - redbluegreenhat the build passed. [14:35:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:35:26] !log [alex@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s [14:35:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:36:16] miraheze/mw-config - redbluegreenhat the build passed. [14:36:51] !log [@mwtask171] starting deploy of {'config': True} to all [14:36:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:36:59] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 7s [14:37:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:53:19] !log [@test151] starting deploy of {'config': True} to test151 [14:53:20] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [14:53:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:53:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:08:12] [02miraheze/GlobalNewFiles] 07dependabot[bot] pushed 031 commit to 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 [+0/-0/±1] 13https://github.com/miraheze/GlobalNewFiles/commit/e32490a5c0e2 [15:08:14] [02miraheze/GlobalNewFiles] 07dependabot[bot] 03e32490a - Update php-parallel-lint/php-parallel-lint requirement [15:08:16] [02GlobalNewFiles] 07dependabot[bot] created branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 - 13https://github.com/miraheze/GlobalNewFiles [15:08:18] [02GlobalNewFiles] 07dependabot[bot] opened pull request 03#55: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/GlobalNewFiles/pull/55 [15:08:21] [02GlobalNewFiles] 07dependabot[bot] labeled pull request 03#55: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/GlobalNewFiles/pull/55 [15:08:22] [02GlobalNewFiles] 07dependabot[bot] labeled pull request 03#55: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/GlobalNewFiles/pull/55 [15:10:08] [02miraheze/PDFEmbed] 07dependabot[bot] pushed 031 commit to 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 [+0/-0/±1] 13https://github.com/miraheze/PDFEmbed/commit/a30430f5bfdc [15:10:09] [02miraheze/PDFEmbed] 07dependabot[bot] 03a30430f - Update php-parallel-lint/php-parallel-lint requirement [15:10:11] [02PDFEmbed] 07dependabot[bot] created branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 - 13https://github.com/miraheze/PDFEmbed [15:10:12] [02PDFEmbed] 07dependabot[bot] opened pull request 03#20: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/PDFEmbed/pull/20 [15:10:13] [02PDFEmbed] 07dependabot[bot] labeled pull request 03#20: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/PDFEmbed/pull/20 [15:12:44] miraheze/GlobalNewFiles - dependabot[bot] the build passed. [15:14:29] miraheze/PDFEmbed - dependabot[bot] the build passed. [15:55:48] PROBLEM - changeprop151 APT on changeprop151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:57:58] RECOVERY - changeprop151 APT on changeprop151 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [16:05:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [16:33:22] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a82f2ed363f4...109771536860 [16:33:25] [02miraheze/puppet] 07Universal-Omega 031097715 - changeprop: increase parsoidCachePrewarm concurrency to 25 [16:39:53] [02miraheze/MatomoAnalytics] 07dependabot[bot] pushed 031 commit to 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 [+0/-0/±1] 13https://github.com/miraheze/MatomoAnalytics/commit/1fba869ec31e [16:39:55] [02miraheze/MatomoAnalytics] 07dependabot[bot] 031fba869 - Update php-parallel-lint/php-parallel-lint requirement [16:39:58] [02MatomoAnalytics] 07dependabot[bot] created branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 - 13https://github.com/miraheze/MatomoAnalytics [16:39:59] [02MatomoAnalytics] 07dependabot[bot] labeled pull request 03#122: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/MatomoAnalytics/pull/122 [16:40:02] [02MatomoAnalytics] 07dependabot[bot] opened pull request 03#122: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/MatomoAnalytics/pull/122 [16:40:03] [02MatomoAnalytics] 07dependabot[bot] labeled pull request 03#122: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/MatomoAnalytics/pull/122 [16:44:51] miraheze/MatomoAnalytics - dependabot[bot] the build passed. [16:47:55] PROBLEM - changeprop151 Current Load on changeprop151 is CRITICAL: LOAD CRITICAL - total load average: 6.75, 10.01, 5.56 [16:49:55] PROBLEM - changeprop151 Current Load on changeprop151 is WARNING: LOAD WARNING - total load average: 3.63, 7.62, 5.21 [16:51:55] RECOVERY - changeprop151 Current Load on changeprop151 is OK: LOAD OK - total load average: 2.14, 5.65, 4.77 [17:31:02] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03kafka-configs [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/ba9dee80bdef [17:31:05] [02miraheze/puppet] 07Universal-Omega 03ba9dee8 - kafka: set some configs [17:31:08] [02puppet] 07Universal-Omega created branch 03kafka-configs - 13https://github.com/miraheze/puppet [17:31:11] [02puppet] 07Universal-Omega opened pull request 03#3801: kafka: set some configs - 13https://github.com/miraheze/puppet/pull/3801 [17:41:33] [02puppet] 07Universal-Omega closed pull request 03#3801: kafka: set some configs - 13https://github.com/miraheze/puppet/pull/3801 [17:41:35] [02miraheze/puppet] 07Universal-Omega deleted branch 03kafka-configs [17:41:36] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/109771536860...392260e9f2c1 [17:41:39] [02miraheze/puppet] 07Universal-Omega 03392260e - kafka: set some configs (#3801) [17:41:40] [02puppet] 07Universal-Omega deleted branch 03kafka-configs - 13https://github.com/miraheze/puppet [17:59:20] [02miraheze/ManageWiki] 07dependabot[bot] pushed 031 commit to 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/commit/a2b1d0a301a5 [17:59:22] [02miraheze/ManageWiki] 07dependabot[bot] 03a2b1d0a - Update php-parallel-lint/php-parallel-lint requirement [17:59:24] [02ManageWiki] 07dependabot[bot] created branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 - 13https://github.com/miraheze/ManageWiki [17:59:25] [02ManageWiki] 07dependabot[bot] labeled pull request 03#456: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/ManageWiki/pull/456 [17:59:26] [02ManageWiki] 07dependabot[bot] labeled pull request 03#456: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/ManageWiki/pull/456 [17:59:27] [02ManageWiki] 07dependabot[bot] opened pull request 03#456: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/ManageWiki/pull/456 [18:04:02] miraheze/ManageWiki - dependabot[bot] the build passed. [18:04:10] PROBLEM - wikitide.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikitide.net' expires in 15 day(s) (Fri 12 Apr 2024 05:51:56 PM GMT +0000). [18:28:39] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03changeprop-concurrency [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/746f07ca9fd9 [18:28:40] [02miraheze/puppet] 07Universal-Omega 03746f07c - changeprop: tweak some concurrency [18:28:41] [02puppet] 07Universal-Omega created branch 03changeprop-concurrency - 13https://github.com/miraheze/puppet [18:28:46] [02puppet] 07Universal-Omega opened pull request 03#3802: changeprop: tweak some concurrency - 13https://github.com/miraheze/puppet/pull/3802 [18:40:17] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/392260e9f2c1...a8a1fc2bfbce [18:40:19] [02miraheze/puppet] 07Universal-Omega 03a8a1fc2 - mediawiki: remove redisJobRunnerService monitoring if not using [18:45:34] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a8a1fc2bfbce...d9d1cf27e0bb [18:45:35] [02miraheze/puppet] 07Universal-Omega 03d9d1cf2 - mediawiki: also disable redisJobChronService if using cpjobqueue [18:57:18] PROBLEM - changeprop151 Current Load on changeprop151 is CRITICAL: LOAD CRITICAL - total load average: 52.59, 49.77, 26.94 [18:59:13] RECOVERY - changeprop151 Current Load on changeprop151 is OK: LOAD OK - total load average: 9.79, 35.35, 24.29 [19:16:32] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d9d1cf27e0bb...a318457ba372 [19:16:34] [02miraheze/puppet] 07Universal-Omega 03a318457 - Remove mediawiki::jobqueue::chron from test151 [19:17:52] [02puppet] 07Universal-Omega closed pull request 03#3802: changeprop: tweak some concurrency - 13https://github.com/miraheze/puppet/pull/3802 [19:17:55] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a318457ba372...0fb80522de26 [19:17:58] [02miraheze/puppet] 07Universal-Omega 030fb8052 - changeprop: tweak some concurrency (#3802) [19:18:00] [02miraheze/puppet] 07Universal-Omega deleted branch 03changeprop-concurrency [19:18:01] [02puppet] 07Universal-Omega deleted branch 03changeprop-concurrency - 13https://github.com/miraheze/puppet [19:26:59] [02miraheze/CreateWiki] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/CreateWiki/compare/6bc0685c65bc...b2fd86c0ffa6 [19:27:00] [02miraheze/CreateWiki] 07redbluegreenhat 03b2fd86c - Merge pull request from GHSA-4rcf-3cj2-46mq [19:29:26] [02miraheze/RottenLinks] 07dependabot[bot] pushed 031 commit to 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/commit/792a3b199efd [19:29:27] [02miraheze/RottenLinks] 07dependabot[bot] 03792a3b1 - Update php-parallel-lint/php-parallel-lint requirement [19:29:29] [02RottenLinks] 07dependabot[bot] created branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 - 13https://github.com/miraheze/RottenLinks [19:29:32] [02RottenLinks] 07dependabot[bot] labeled pull request 03#78: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/RottenLinks/pull/78 [19:29:33] [02RottenLinks] 07dependabot[bot] opened pull request 03#78: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/RottenLinks/pull/78 [19:30:09] miraheze/CreateWiki - redbluegreenhat the build has errored. [19:30:26] [02miraheze/mw-config] 07redbluegreenhat pushed 031 commit to 03revert-5518-T11999-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/02241b58b752 [19:30:29] [02miraheze/mw-config] 07redbluegreenhat 0302241b5 - Revert "T11999: Disable Special:RequestWikiQueue outside of metawiki (#5518)" [19:30:31] [02mw-config] 07redbluegreenhat created branch 03revert-5518-T11999-2 - 13https://github.com/miraheze/mw-config [19:30:32] [02mw-config] 07redbluegreenhat opened pull request 03#5519: Revert "T11999: Disable Special:RequestWikiQueue outside of metawiki" - 13https://github.com/miraheze/mw-config/pull/5519 [19:30:44] [02mw-config] 07redbluegreenhat closed pull request 03#5519: Revert "T11999: Disable Special:RequestWikiQueue outside of metawiki" - 13https://github.com/miraheze/mw-config/pull/5519 [19:30:45] [02miraheze/mw-config] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/fb3e68bcef45...1212bf6d1e90 [19:30:47] [02miraheze/mw-config] 07redbluegreenhat 031212bf6 - Revert "T11999: Disable Special:RequestWikiQueue outside of metawiki" (#5519) [19:30:50] [02mw-config] 07redbluegreenhat deleted branch 03revert-5518-T11999-2 - 13https://github.com/miraheze/mw-config [19:30:52] !log [alex@test151] starting deploy of {'pull': 'config', 'config': True} to test151 [19:30:53] !log [alex@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 0s [19:30:53] [02miraheze/mw-config] 07redbluegreenhat deleted branch 03revert-5518-T11999-2 [19:31:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:31:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:31:32] miraheze/mw-config - redbluegreenhat the build passed. [19:31:41] miraheze/mw-config - redbluegreenhat the build passed. [19:31:52] !log [alex@test151] starting deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to test151 [19:31:53] !log [alex@test151] finished deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to test151 - SUCCESS in 1s [19:32:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:32:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:32:41] !log [@mwtask181] starting deploy of {'config': True} to all [19:32:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:32:50] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 9s [19:32:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:33:04] !log [alex@mwtask181] starting deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to all [19:33:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:33:14] !log [alex@mwtask181] finished deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to all - SUCCESS in 10s [19:33:21] !log [alex@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [19:33:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:33:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:33:31] !log [alex@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s [19:33:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:34:09] miraheze/RottenLinks - dependabot[bot] the build passed. [19:36:53] !log [@mwtask171] starting deploy of {'config': True} to all [19:37:01] [02miraheze/CreateWiki] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/b2fd86c0ffa6...23415c17ffb4 [19:37:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:37:01] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 7s [19:37:04] [02miraheze/CreateWiki] 07redbluegreenhat 0323415c1 - Fix [19:37:06] !log [alex@mwtask181] starting deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to all [19:37:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:37:13] !log [alex@test151] starting deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to test151 [19:37:14] !log [alex@test151] finished deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to test151 - SUCCESS in 1s [19:37:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:37:16] !log [alex@mwtask181] finished deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to all - SUCCESS in 10s [19:37:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:37:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:37:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:37:48] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/1212bf6d1e90...34ce07836e06 [19:37:50] [02miraheze/mw-config] 07Universal-Omega 0334ce078 - PoolCounter: use rdb151 [19:38:41] miraheze/mw-config - Universal-Omega the build passed. [19:39:21] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0fb80522de26...3fb3047adc90 [19:39:22] [02miraheze/puppet] 07Universal-Omega 033fb3047 - Move poolcounter to rdb151 [19:44:18] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03mw-changeprop [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/84f6ebf97a82 [19:44:21] [02miraheze/puppet] 07Universal-Omega 0384f6ebf - Grant mediawiki-admins access to changeprop151 and revoke jobchron171 [19:44:24] [02puppet] 07Universal-Omega created branch 03mw-changeprop - 13https://github.com/miraheze/puppet [19:44:25] [02puppet] 07Universal-Omega opened pull request 03#3803: Grant mediawiki-admins access to changeprop151 and revoke jobchron171 - 13https://github.com/miraheze/puppet/pull/3803 [19:45:07] miraheze/CreateWiki - redbluegreenhat the build passed. [19:45:24] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03mw-changeprop [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/84f6ebf97a82...be23f43ec10e [19:45:27] [02miraheze/puppet] 07Universal-Omega 03be23f43 - Add [19:45:30] [02puppet] 07Universal-Omega synchronize pull request 03#3803: Grant mediawiki-admins access to changeprop151 and revoke jobchron171 - 13https://github.com/miraheze/puppet/pull/3803 [19:45:57] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03mw-changeprop [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/be23f43ec10e...5818cf5cf819 [19:45:59] [02miraheze/puppet] 07Universal-Omega 035818cf5 - Remove [19:46:01] [02puppet] 07Universal-Omega synchronize pull request 03#3803: Grant mediawiki-admins access to changeprop151 and revoke jobchron171 - 13https://github.com/miraheze/puppet/pull/3803 [19:53:54] !log [@test151] starting deploy of {'config': True} to test151 [19:53:55] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [19:54:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:54:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:00:06] PROBLEM - changeprop151 Current Load on changeprop151 is CRITICAL: LOAD CRITICAL - total load average: 90.87, 74.75, 37.57 [20:02:36] !log [@mwtask181] starting deploy of {'config': True} to all [20:02:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:02:45] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 9s [20:02:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:04:06] RECOVERY - changeprop151 Current Load on changeprop151 is OK: LOAD OK - total load average: 1.84, 33.56, 29.04 [20:05:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [20:07:22] !log [@mwtask171] starting deploy of {'config': True} to all [20:07:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:07:30] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 7s [20:07:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:11:40] Hello there! I've been encountering a weird issue lately. I wanted to create a wiki, but each time I submit my request, it keeps telling me to login, over and over again, even after asking to keep me logged in. Do you perhaps know the source of this issue? [20:12:08] Huh [20:12:25] Does the rest of meta work? [20:13:34] Not really, practically any pages that require to be logged in does the same. It's as if I get somehow logged out constantly [20:13:50] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/3fb3047adc90...c6df4c3336cc [20:13:53] [02miraheze/puppet] 07Universal-Omega 03c6df4c3 - mediawiki: add mediawiki::periodic_job [20:14:59] What’s your account? [20:15:19] username's Shiv [20:25:50] PROBLEM - changeprop151 Current Load on changeprop151 is CRITICAL: LOAD CRITICAL - total load average: 108.89, 40.03, 28.16 [20:27:47] RECOVERY - changeprop151 Current Load on changeprop151 is OK: LOAD OK - total load average: 21.22, 29.48, 25.70 [20:45:17] [02miraheze/CreateWiki] 07redbluegreenhat pushed 031 commit to 03redbluegreenhat-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/commit/e1700f6c3533 [20:45:19] [02miraheze/CreateWiki] 07redbluegreenhat 03e1700f6 - rm read check [20:45:20] [02CreateWiki] 07redbluegreenhat created branch 03redbluegreenhat-patch-1 - 13https://github.com/miraheze/CreateWiki [20:46:44] [02miraheze/CreateWiki] 07redbluegreenhat pushed 031 commit to 03redbluegreenhat-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/e1700f6c3533...8b1878376989 [20:46:45] [02miraheze/CreateWiki] 07redbluegreenhat 038b18783 - rm read check [20:48:09] [02CreateWiki] 07redbluegreenhat opened pull request 03#497: Remove checks for `(read)` - 13https://github.com/miraheze/CreateWiki/pull/497 [20:50:51] [02CreateWiki] 07redbluegreenhat closed pull request 03#497: Remove checks for `(read)` - 13https://github.com/miraheze/CreateWiki/pull/497 [20:50:52] [02miraheze/CreateWiki] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/CreateWiki/compare/23415c17ffb4...8f8442ed5299 [20:50:54] [02miraheze/CreateWiki] 07redbluegreenhat 038f8442e - Remove checks for `(read)` (#497) [20:50:54] !log [alex@mwtask181] starting deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to all [20:50:56] [02miraheze/CreateWiki] 07redbluegreenhat deleted branch 03redbluegreenhat-patch-1 [20:50:58] !log [alex@test151] starting deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to test151 [20:50:58] [02CreateWiki] 07redbluegreenhat deleted branch 03redbluegreenhat-patch-1 - 13https://github.com/miraheze/CreateWiki [20:50:59] !log [alex@test151] finished deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to test151 - SUCCESS in 1s [20:51:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:51:04] PROBLEM - swiftproxy161 HTTP on swiftproxy161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:51:04] !log [alex@mwtask181] finished deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to all - SUCCESS in 10s [20:51:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:51:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:51:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:53:03] RECOVERY - swiftproxy161 HTTP on swiftproxy161 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.010 second response time [20:55:58] miraheze/CreateWiki - redbluegreenhat the build passed. [20:59:20] miraheze/CreateWiki - redbluegreenhat the build passed. [21:05:15] PROBLEM - changeprop151 Current Load on changeprop151 is CRITICAL: LOAD CRITICAL - total load average: 369.46, 120.36, 51.93 [21:11:15] PROBLEM - changeprop151 Current Load on changeprop151 is WARNING: LOAD WARNING - total load average: 3.75, 47.11, 41.22 [21:12:43] [02miraheze/DataDump] 07dependabot[bot] pushed 031 commit to 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 [+0/-0/±1] 13https://github.com/miraheze/DataDump/commit/b21e81b89b6d [21:12:46] [02miraheze/DataDump] 07dependabot[bot] 03b21e81b - Update php-parallel-lint/php-parallel-lint requirement [21:12:49] [02DataDump] 07dependabot[bot] created branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 - 13https://github.com/miraheze/DataDump [21:12:50] [02DataDump] 07dependabot[bot] labeled pull request 03#85: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/DataDump/pull/85 [21:12:53] [02DataDump] 07dependabot[bot] opened pull request 03#85: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/DataDump/pull/85 [21:13:15] RECOVERY - changeprop151 Current Load on changeprop151 is OK: LOAD OK - total load average: 0.65, 31.56, 36.23 [21:13:46] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c6df4c3336cc...d6283ae4bc62 [21:13:48] [02miraheze/puppet] 07Universal-Omega 03d6283ae - eventgate: tweak config [21:14:16] [02miraheze/WikiDiscover] 07dependabot[bot] pushed 031 commit to 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 [+0/-0/±1] 13https://github.com/miraheze/WikiDiscover/commit/4bcfb0d029cf [21:14:18] [02miraheze/WikiDiscover] 07dependabot[bot] 034bcfb0d - Update php-parallel-lint/php-parallel-lint requirement [21:14:21] [02WikiDiscover] 07dependabot[bot] created branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 - 13https://github.com/miraheze/WikiDiscover [21:14:23] [02WikiDiscover] 07dependabot[bot] opened pull request 03#114: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/WikiDiscover/pull/114 [21:14:25] [02WikiDiscover] 07dependabot[bot] labeled pull request 03#114: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/WikiDiscover/pull/114 [21:14:26] [02WikiDiscover] 07dependabot[bot] labeled pull request 03#114: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/WikiDiscover/pull/114 [21:16:57] miraheze/DataDump - dependabot[bot] the build passed. [21:18:49] miraheze/WikiDiscover - dependabot[bot] the build passed. [21:21:27] [02miraheze/MirahezeMagic] 07dependabot[bot] pushed 031 commit to 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/commit/1035d0079bce [21:21:30] [02miraheze/MirahezeMagic] 07dependabot[bot] 031035d00 - Update php-parallel-lint/php-parallel-lint requirement [21:21:31] [02MirahezeMagic] 07dependabot[bot] created branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 - 13https://github.com/miraheze/MirahezeMagic [21:21:32] [02MirahezeMagic] 07dependabot[bot] labeled pull request 03#486: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/MirahezeMagic/pull/486 [21:21:35] [02MirahezeMagic] 07dependabot[bot] opened pull request 03#486: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/MirahezeMagic/pull/486 [21:21:37] [02MirahezeMagic] 07dependabot[bot] labeled pull request 03#486: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/MirahezeMagic/pull/486 [21:22:38] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03cp-conf [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/1c2783e200e1 [21:22:41] [02miraheze/puppet] 07Universal-Omega 031c2783e - changeprop: tweak some config [21:22:44] [02puppet] 07Universal-Omega created branch 03cp-conf - 13https://github.com/miraheze/puppet [21:22:46] [02puppet] 07Universal-Omega opened pull request 03#3804: changeprop: tweak some config - 13https://github.com/miraheze/puppet/pull/3804 [21:23:42] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d6283ae4bc62...cfe13637903d [21:23:45] [02miraheze/puppet] 07Universal-Omega 03cfe1363 - eventgate: use error logging [21:25:02] miraheze/MirahezeMagic - dependabot[bot] the build has errored. [21:27:27] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03cp-conf [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/1c2783e200e1...5b539e12729b [21:27:30] [02miraheze/puppet] 07Universal-Omega 035b539e1 - Set some disable_delayed_execution [21:27:32] [02puppet] 07Universal-Omega synchronize pull request 03#3804: changeprop: tweak some config - 13https://github.com/miraheze/puppet/pull/3804 [21:28:07] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 70%, RTA = 186.74 ms [21:28:51] [02puppet] 07Universal-Omega closed pull request 03#3804: changeprop: tweak some config - 13https://github.com/miraheze/puppet/pull/3804 [21:28:52] [02miraheze/puppet] 07Universal-Omega deleted branch 03cp-conf [21:28:55] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/cfe13637903d...540d383d0359 [21:28:56] [02miraheze/puppet] 07Universal-Omega 03540d383 - changeprop: tweak some config (#3804) [21:28:58] [02puppet] 07Universal-Omega deleted branch 03cp-conf - 13https://github.com/miraheze/puppet [21:30:09] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 122.84 ms [21:35:33] PROBLEM - cp36 Disk Space on cp36 is WARNING: DISK WARNING - free space: / 9704MiB (10% inode=98%); [21:44:39] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/540d383d0359...c5adc97fc5eb [21:44:42] [02miraheze/puppet] 07Universal-Omega 03c5adc97 - jobrunner: add port 9005 to firewall [21:52:46] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [21:54:01] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 9701MiB (10% inode=98%); [22:01:15] PROBLEM - changeprop151 Current Load on changeprop151 is CRITICAL: LOAD CRITICAL - total load average: 102.76, 98.45, 48.35 [22:05:15] PROBLEM - changeprop151 Current Load on changeprop151 is WARNING: LOAD WARNING - total load average: 4.98, 45.24, 37.75 [22:06:12] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03jobrunner-apache [+1/-0/±0] 13https://github.com/miraheze/puppet/commit/81fb48905a41 [22:06:13] [02miraheze/puppet] 07Universal-Omega 0381fb489 - jobrunner: add some apache config [22:06:16] [02puppet] 07Universal-Omega created branch 03jobrunner-apache - 13https://github.com/miraheze/puppet [22:06:17] [02puppet] 07Universal-Omega opened pull request 03#3805: jobrunner: add some apache config - 13https://github.com/miraheze/puppet/pull/3805 [22:07:15] RECOVERY - changeprop151 Current Load on changeprop151 is OK: LOAD OK - total load average: 6.89, 33.46, 34.38 [22:08:38] [02puppet] 07Universal-Omega synchronize pull request 03#3805: jobrunner: add some apache config - 13https://github.com/miraheze/puppet/pull/3805 [22:08:41] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03jobrunner-apache [+1/-1/±0] 13https://github.com/miraheze/puppet/compare/81fb48905a41...393d791b60da [22:08:43] [02miraheze/puppet] 07Universal-Omega 03393d791 - Move [22:09:33] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03jobrunner-apache [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/393d791b60da...c21887fb3739 [22:09:34] [02miraheze/puppet] 07Universal-Omega 03c21887f - Add worker config [22:09:37] [02puppet] 07Universal-Omega synchronize pull request 03#3805: jobrunner: add some apache config - 13https://github.com/miraheze/puppet/pull/3805 [22:11:15] PROBLEM - changeprop151 Current Load on changeprop151 is CRITICAL: LOAD CRITICAL - total load average: 181.01, 94.70, 58.02 [22:15:15] PROBLEM - changeprop151 Current Load on changeprop151 is WARNING: LOAD WARNING - total load average: 12.81, 45.46, 45.87 [22:15:18] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03jobrunner-apache [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c21887fb3739...5136cdde53bc [22:15:21] [02miraheze/puppet] 07Universal-Omega 035136cdd - Update [22:15:23] [02puppet] 07Universal-Omega synchronize pull request 03#3805: jobrunner: add some apache config - 13https://github.com/miraheze/puppet/pull/3805 [22:15:53] [02miraheze/puppet] 07github-actions[bot] pushed 031 commit to 03jobrunner-apache [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/5136cdde53bc...132e8c090b3c [22:15:56] [02miraheze/puppet] 07github-actions 03132e8c0 - CI: lint puppet code to standards [22:15:57] [02puppet] 07github-actions[bot] synchronize pull request 03#3805: jobrunner: add some apache config - 13https://github.com/miraheze/puppet/pull/3805 [22:17:15] RECOVERY - changeprop151 Current Load on changeprop151 is OK: LOAD OK - total load average: 1.75, 30.43, 40.31 [22:17:31] [02puppet] 07Universal-Omega synchronize pull request 03#3805: jobrunner: add some apache config - 13https://github.com/miraheze/puppet/pull/3805 [22:17:32] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03jobrunner-apache [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/132e8c090b3c...d91bb9fa907e [22:17:33] [02miraheze/puppet] 07Universal-Omega 03d91bb9f - Update [22:21:41] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [22:26:31] [02miraheze/IncidentReporting] 07dependabot[bot] pushed 031 commit to 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 [+0/-0/±1] 13https://github.com/miraheze/IncidentReporting/commit/79ccce66d963 [22:26:32] [02miraheze/IncidentReporting] 07dependabot[bot] 0379ccce6 - Update php-parallel-lint/php-parallel-lint requirement [22:26:34] [02IncidentReporting] 07dependabot[bot] created branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 - 13https://github.com/miraheze/IncidentReporting [22:26:35] [02IncidentReporting] 07dependabot[bot] labeled pull request 03#78: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/IncidentReporting/pull/78 [22:26:38] [02IncidentReporting] 07dependabot[bot] labeled pull request 03#78: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/IncidentReporting/pull/78 [22:26:39] [02IncidentReporting] 07dependabot[bot] opened pull request 03#78: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/IncidentReporting/pull/78 [22:31:08] miraheze/IncidentReporting - dependabot[bot] the build passed. [22:35:05] PROBLEM - cp41 Disk Space on cp41 is WARNING: DISK WARNING - free space: / 10234MiB (10% inode=98%); [22:46:36] [02miraheze/CreateWiki] 07dependabot[bot] pushed 031 commit to 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/commit/560e04998ee7 [22:46:38] [02miraheze/CreateWiki] 07dependabot[bot] 03560e049 - Update php-parallel-lint/php-parallel-lint requirement [22:46:41] [02CreateWiki] 07dependabot[bot] created branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 - 13https://github.com/miraheze/CreateWiki [22:46:44] [02CreateWiki] 07dependabot[bot] opened pull request 03#498: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/CreateWiki/pull/498 [22:46:45] [02CreateWiki] 07dependabot[bot] labeled pull request 03#498: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/CreateWiki/pull/498 [22:46:46] [02CreateWiki] 07dependabot[bot] labeled pull request 03#498: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/CreateWiki/pull/498 [22:55:05] miraheze/CreateWiki - dependabot[bot] the build passed. [23:06:18] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c5adc97fc5eb...ed242795b558 [23:06:19] [02miraheze/puppet] 07Universal-Omega 03ed24279 - changeprop: set linger.ms to 20 and raise default concurrency [23:22:46] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [23:25:38] PROBLEM - mwtask171 ferm_active on mwtask171 is CRITICAL: ERROR ferm input drop default policy not set, ferm might not have been started correctly [23:27:04] PROBLEM - mwtask181 ferm_active on mwtask181 is CRITICAL: ERROR ferm input drop default policy not set, ferm might not have been started correctly [23:33:27] PROBLEM - mwtask171 MediaWiki Rendering on mwtask171 is CRITICAL: connect to address 10.0.17.144 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [23:35:26] RECOVERY - mwtask171 MediaWiki Rendering on mwtask171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.264 second response time [23:37:58] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ed242795b558...868b4ce4ced7 [23:38:01] [02miraheze/puppet] 07Universal-Omega 03868b4ce - php: increase max_execution_time to 180 [23:47:50] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/868b4ce4ced7...ebf8db55cd4d [23:47:52] [02miraheze/puppet] 07Universal-Omega 03ebf8db5 - mediawiki::php: make apc.shm_size configurable [23:51:20] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03increase_open_files [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/e4332397011b [23:51:22] [02miraheze/puppet] 07Universal-Omega 03e433239 - mediawiki: add increase_open_files option for PHP [23:51:25] [02puppet] 07Universal-Omega created branch 03increase_open_files - 13https://github.com/miraheze/puppet [23:51:26] [02puppet] 07Universal-Omega opened pull request 03#3806: mediawiki: add increase_open_files option for PHP - 13https://github.com/miraheze/puppet/pull/3806 [23:52:48] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03increase_open_files [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e4332397011b...1683c82a9077 [23:52:50] [02miraheze/puppet] 07Universal-Omega 031683c82 - Add [23:52:53] [02puppet] 07Universal-Omega synchronize pull request 03#3806: mediawiki: add increase_open_files option for PHP - 13https://github.com/miraheze/puppet/pull/3806 [23:53:18] [02puppet] 07Universal-Omega closed pull request 03#3806: mediawiki: add increase_open_files option for PHP - 13https://github.com/miraheze/puppet/pull/3806 [23:53:20] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/ebf8db55cd4d...24089a020180 [23:53:22] [02miraheze/puppet] 07Universal-Omega 0324089a0 - mediawiki: add increase_open_files option for PHP (#3806) [23:53:25] [02miraheze/puppet] 07Universal-Omega deleted branch 03increase_open_files [23:53:28] [02puppet] 07Universal-Omega deleted branch 03increase_open_files - 13https://github.com/miraheze/puppet [23:53:34] [02miraheze/puppet] 07github-actions[bot] pushed 033 commits to 03increase_open_files [+0/-0/±4] 13https://github.com/miraheze/puppet/compare/e4332397011b^...3d4dfb4dc519 [23:53:36] [02miraheze/puppet] 07Universal-Omega 03e433239 - mediawiki: add increase_open_files option for PHP [23:53:39] [02miraheze/puppet] 07Universal-Omega 031683c82 - Add [23:53:42] [02miraheze/puppet] 07github-actions 033d4dfb4 - CI: lint puppet code to standards [23:53:43] [02puppet] 07github-actions[bot] created branch 03increase_open_files - 13https://github.com/miraheze/puppet [23:54:57] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/24089a020180...889bea5f5d90 [23:54:58] [02miraheze/puppet] 07Universal-Omega 03889bea5 - mwtask171: tweak php config [23:56:08] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/889bea5f5d90...56cd33308c61 [23:56:09] [02miraheze/puppet] 07Universal-Omega 0356cd333 - mwtask181: tweak php config