[02:10:54] [02mw-config] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/66d1da5107c5...cca33fc01abc [02:10:56] [02mw-config] 07MacFan4000 03cca33fc - add agent access to staffwiki [02:11:24] !log [macfan@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [02:11:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:11:37] !log [macfan@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 13s [02:11:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:11:52] miraheze/mw-config - MacFan4000 the build passed. [02:22:55] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/5a5c3f08be56...f433fd437bfa [02:22:57] [02ssl] 07WikiTideSSLBot 03f433fd4 - Bot: Add SSL cert for weavefarers.wiki [02:24:07] !log [@test151] starting deploy of {'config': True} to test151 [02:24:08] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [02:24:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:24:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:28:02] [02ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f433fd437bfa...4f7672afcb83 [02:28:04] [02ssl] 07MacFan4000 034f7672a - add kagagawiki redirect [02:30:22] [02dns] 07MacFan4000 opened pull request 03#610: add www.kagaga.wiki - 13https://github.com/miraheze/dns/pull/610 [02:36:53] !log [@mwtask171] starting deploy of {'config': True} to all [02:36:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:37:04] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 10s [02:37:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:38:45] [02ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/4f7672afcb83...f13a65b2c758 [02:38:47] [02ssl] 07MacFan4000 03f13a65b - +www.ssbuniverses.com [02:45:06] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is CRITICAL: CRITICAL - NGINX Error Rate is 70% [02:47:06] RECOVERY - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is OK: OK - NGINX Error Rate is 11% [03:01:08] [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/a562763229a9...8468342f8b0f [03:01:09] [02dns] 07MacFan4000 038468342 - add www.kagaga.wiki (#610) [03:01:12] [02dns] 07Universal-Omega closed pull request 03#610: add www.kagaga.wiki - 13https://github.com/miraheze/dns/pull/610 [03:04:50] [02puppet] 07Universal-Omega pushed 032 commits to 03mattermost [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/f31d70078222...80bf7adacc99 [03:04:52] [02puppet] 07Universal-Omega 0380bf7ad - Merge branch 'master' into mattermost [03:04:54] [02puppet] 07Universal-Omega synchronize pull request 03#3913: Add mattermost module - 13https://github.com/miraheze/puppet/pull/3913 [03:44:36] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/deleteWiki.php --wiki=loginwiki --delete --deletewiki themanyworldswikiwiki (END - exit=256) [03:44:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:45:15] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/extensions/CreateWiki/maintenance/deleteWiki.php --wiki=loginwiki --delete --deletewiki themanyworldswikiwiki (END - exit=0) [03:45:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:45:27] [02ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f13a65b2c758...16ce5d67cf25 [03:45:29] [02ssl] 07MacFan4000 0316ce5d6 - add ssbuniversewiki redirect [03:50:48] [02ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/16ce5d67cf25...199b7a1f7c82 [03:50:49] [02ssl] 07MacFan4000 03199b7a1 - add kagagawiki redirect (2nd) [03:53:37] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/extensions/ManageWiki/maintenance/removeSettings.php --wiki=snapwikiwiki --setting wgRCMaxAge (END - exit=256) [03:53:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:53:46] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/extensions/ManageWiki/maintenance/removeSettings.php --wiki=snapwikiwiki wgRCMaxAge (END - exit=0) [03:53:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:54:02] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/rebuildrecentchanges.php --wiki=snapwikiwiki (END - exit=0) [03:54:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:00:10] RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [04:00:30] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/cleanupTitles.php --wiki=zamnhackingwiki (END - exit=0) [04:00:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:06:17] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/cleanupTitles.php --wiki=allthetropeswiki (END - exit=0) [04:06:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:17:40] RECOVERY - mon181 Check correctness of the icinga configuration on mon181 is OK: Icinga configuration is correct [04:19:18] !log [agent@mwtask181] starting deploy of {'versions': '1.42', 'upgrade_extensions': ['MyVariables', 'TimezoneConverter', 'VariablesLua']} to all [04:19:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:19:47] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/cleanupTitles.php --wiki=alphaverwiki (END - exit=0) [04:19:49] !log [agent@mwtask181] finished deploy of {'versions': '1.42', 'upgrade_extensions': ['MyVariables', 'TimezoneConverter', 'VariablesLua']} to all - SUCCESS in 31s [04:19:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:19:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:03:03] !log [agent@mwtask181] starting deploy of {'versions': '1.42', 'upgrade_extensions': 'InputBox'} to all [05:03:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:03:17] !log [agent@mwtask181] finished deploy of {'versions': '1.42', 'upgrade_extensions': 'InputBox'} to all - SUCCESS in 13s [05:03:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:42:25] [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [08:08:08] !log [@mwtask171] starting deploy of {'config': True} to all [08:08:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:08:24] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 16s [08:08:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:16:16] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_dns] [10:16:32] [02dns] 07RhinosF1 opened pull request 03#611: Delete zones/wiki.surf - 13https://github.com/miraheze/dns/pull/611 [10:44:16] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [11:30:25] [02puppet] 07redbluegreenhat edited pull request 03#3914: DNM: T12537: Migrate to BIND - 13https://github.com/miraheze/puppet/pull/3914 [11:39:18] PROBLEM - cp27 Puppet on cp27 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:42:29] [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [12:07:15] RECOVERY - cp27 Puppet on cp27 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:20:42] !log [agent@mwtask181] running runJobs on all wikis (90,000+ unclaimed jobs) [13:20:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:23:50] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/runJobs.php --wiki=dominionstrategywiki --procs 64 (START) [13:23:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:30:00] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/runJobs.php --wiki=dominionstrategywiki --procs 64 (END - exit=0) [13:30:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:37:13] [02ssl] 07redbluegreenhat created branch 03redirects-origin-cert - 13https://github.com/miraheze/ssl [13:37:16] [02ssl] 07redbluegreenhat pushed 031 commit to 03redirects-origin-cert [+0/-0/±1] 13https://github.com/miraheze/ssl/commit/944e6ee3e066 [13:37:17] [02ssl] 07redbluegreenhat 03944e6ee - Move redirects from domains we control to the origin cert [13:37:21] [02ssl] 07redbluegreenhat opened pull request 03#798: Move redirects from domains we control to the origin cert - 13https://github.com/miraheze/ssl/pull/798 [13:37:29] [02ssl] 07coderabbitai[bot] commented on pull request 03#798: Move redirects from domains we control to the origin cert - 13https://github.com/miraheze/ssl/pull/798#issuecomment-2335190424 [13:39:26] [02ssl] 07coderabbitai[bot] edited a comment on pull request 03#798: Move redirects from domains we control to the origin cert - 13https://github.com/miraheze/ssl/pull/798#issuecomment-2335190424 [13:40:51] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/runJobs.php --wiki=dominionstrategywiki --procs 64 --memory-limit max (START) [13:40:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:41:07] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/runJobs.php --wiki=dominionstrategywiki --procs 64 --memory-limit max (END - exit=0) [13:41:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:45:25] [02ssl] 07redbluegreenhat created branch 03T12554-authenticated-origin-pull - 13https://github.com/miraheze/ssl [13:45:26] [02ssl] 07redbluegreenhat pushed 031 commit to 03T12554-authenticated-origin-pull [+1/-0/±0] 13https://github.com/miraheze/ssl/commit/570c359c2964 [13:45:27] [02ssl] 07redbluegreenhat 03570c359 - T12554: Add the Cloudflare Authenticated Origin Pull CA cert [13:45:35] [02ssl] 07redbluegreenhat opened pull request 03#799: T12554: Add the Cloudflare Authenticated Origin Pull CA cert - 13https://github.com/miraheze/ssl/pull/799 [13:45:40] [02ssl] 07coderabbitai[bot] commented on pull request 03#799: T12554: Add the Cloudflare Authenticated Origin Pull CA cert - 13https://github.com/miraheze/ssl/pull/799#issuecomment-2335192365 [13:47:32] [02ssl] 07coderabbitai[bot] edited a comment on pull request 03#799: T12554: Add the Cloudflare Authenticated Origin Pull CA cert - 13https://github.com/miraheze/ssl/pull/799#issuecomment-2335192365 [14:09:11] [02ssl] 07redbluegreenhat pushed 031 commit to 03T12554-authenticated-origin-pull [+1/-0/±0] 13https://github.com/miraheze/ssl/compare/570c359c2964...b8899de7baaf [14:09:14] [02ssl] 07redbluegreenhat 03b8899de - T12554: Add the Cloudflare Authenticated Origin Pull CA cert [14:09:15] [02ssl] 07redbluegreenhat synchronize pull request 03#799: T12554: Add the Cloudflare Authenticated Origin Pull CA cert - 13https://github.com/miraheze/ssl/pull/799 [14:09:20] [02ssl] 07coderabbitai[bot] edited a comment on pull request 03#799: T12554: Add the Cloudflare Authenticated Origin Pull CA cert - 13https://github.com/miraheze/ssl/pull/799#issuecomment-2335192365 [14:12:18] [02ssl] 07coderabbitai[bot] edited a comment on pull request 03#799: T12554: Add the Cloudflare Authenticated Origin Pull CA cert - 13https://github.com/miraheze/ssl/pull/799#issuecomment-2335192365 [14:13:56] [02ssl] 07RhinosF1 commented on pull request 03#799: T12554: Add the Cloudflare Authenticated Origin Pull CA cert - 13https://github.com/miraheze/ssl/pull/799#issuecomment-2335200186 [14:14:02] [02ssl] 07coderabbitai[bot] edited a comment on pull request 03#799: T12554: Add the Cloudflare Authenticated Origin Pull CA cert - 13https://github.com/miraheze/ssl/pull/799#issuecomment-2335192365 [14:14:12] daft ai [14:16:36] [02ssl] 07coderabbitai[bot] edited a comment on pull request 03#799: T12554: Add the Cloudflare Authenticated Origin Pull CA cert - 13https://github.com/miraheze/ssl/pull/799#issuecomment-2335192365 [14:25:14] [02puppet] 07redbluegreenhat created branch 03T12554 - 13https://github.com/miraheze/puppet [14:25:17] [02puppet] 07redbluegreenhat pushed 031 commit to 03T12554 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/287819c0d499 [14:25:20] [02puppet] 07redbluegreenhat 03287819c - T12554: Enable mandatory authenticated origin pull on domains using Cloudflare [14:25:43] [02puppet] 07redbluegreenhat opened pull request 03#3916: T12554: Enable mandatory authenticated origin pull on domains using Cloudflare - 13https://github.com/miraheze/puppet/pull/3916 [14:25:51] [02puppet] 07coderabbitai[bot] commented on pull request 03#3916: T12554: Enable mandatory authenticated origin pull on domains using Cloudflare - 13https://github.com/miraheze/puppet/pull/3916#issuecomment-2335239168 [14:26:51] [02puppet] 07redbluegreenhat pushed 031 commit to 03T12554 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/287819c0d499...f88c671db9ad [14:26:53] [02puppet] 07redbluegreenhat 03f88c671 - T12554: Enable mandatory authenticated origin pull on domains using Cloudflare [14:26:54] [02puppet] 07redbluegreenhat synchronize pull request 03#3916: T12554: Enable mandatory authenticated origin pull on domains using Cloudflare - 13https://github.com/miraheze/puppet/pull/3916 [14:29:03] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 55.34, 24.78, 9.92 [14:31:38] [02puppet] 07redbluegreenhat pushed 031 commit to 03T12554 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f88c671db9ad...5ddbc1703f63 [14:31:40] [02puppet] 07redbluegreenhat 035ddbc17 - T12554: Enable mandatory authenticated origin pull on domains using Cloudflare [14:31:42] [02puppet] 07redbluegreenhat synchronize pull request 03#3916: T12554: Enable mandatory authenticated origin pull on domains using Cloudflare - 13https://github.com/miraheze/puppet/pull/3916 [14:35:03] PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 1.08, 11.14, 9.16 [14:37:02] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 0.62, 7.63, 8.11 [14:43:46] [02puppet] 07RhinosF1 commented on pull request 03#3916: T12554: Enable mandatory authenticated origin pull on domains using Cloudflare - 13https://github.com/miraheze/puppet/pull/3916#issuecomment-2335305338 [14:45:10] [02puppet] 07coderabbitai[bot] commented on pull request 03#3916: T12554: Enable mandatory authenticated origin pull on domains using Cloudflare - 13https://github.com/miraheze/puppet/pull/3916#issuecomment-2335310565 [14:45:13] [02puppet] 07coderabbitai[bot] commented on pull request 03#3916: T12554: Enable mandatory authenticated origin pull on domains using Cloudflare - 13https://github.com/miraheze/puppet/pull/3916#issuecomment-2335310937 [14:46:01] [02puppet] 07coderabbitai[bot] edited pull request 03#3916: T12554: Enable mandatory authenticated origin pull on domains using Cloudflare - 13https://github.com/miraheze/puppet/pull/3916 [14:46:37] [02puppet] 07coderabbitai[bot] edited a comment on pull request 03#3916: T12554: Enable mandatory authenticated origin pull on domains using Cloudflare - 13https://github.com/miraheze/puppet/pull/3916#issuecomment-2335310937 [14:48:44] [02puppet] 07redbluegreenhat edited pull request 03#3916: T12554: Enable mandatory authenticated origin pull on domains using Cloudflare - 13https://github.com/miraheze/puppet/pull/3916 [15:11:35] [02ssl] 07redbluegreenhat closed pull request 03#799: T12554: Add the Cloudflare Authenticated Origin Pull CA cert - 13https://github.com/miraheze/ssl/pull/799 [15:11:38] [02ssl] 07redbluegreenhat pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/ssl/compare/199b7a1f7c82...d06e4708d12e [15:11:40] [02ssl] 07redbluegreenhat 03d06e470 - T12554: Add the Cloudflare Authenticated Origin Pull CA cert (#799) [15:11:43] [02ssl] 07redbluegreenhat deleted branch 03T12554-authenticated-origin-pull [15:11:46] [02ssl] 07redbluegreenhat deleted branch 03T12554-authenticated-origin-pull - 13https://github.com/miraheze/ssl [15:12:10] [02mw-config] 07waki285 opened pull request 03#5652: T12553: Add edittemplateprotected to cricketnepalwiki - 13https://github.com/miraheze/mw-config/pull/5652 [15:13:11] miraheze/mw-config - waki285 the build passed. [15:21:47] !log turned on authenticated origin pulls on miraheze.org [15:22:42] is the bot gone again? [15:22:59] @MirahezeRelay [15:23:04] Says online [15:23:09] not that one [15:25:21] @bluemoon0332 shouldn't we set https://developers.cloudflare.com/ssl/origin-configuration/ssl-modes/ssl-only-origin-pull/ too [15:25:24] it is receiving my messages [15:25:35] !log test [15:25:36] !log test [15:25:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:28:31] [1/8] ``` [15:28:31] [2/8] DEBUG: command: pubmsg, source: MirahezeRelay_!~MirahezeR@2602:294:0:b23::113, target: #miraheze-tech-ops, arguments: [' !log test'], tags: None [15:28:32] [3/8] Sep 07 15:25:35 bots171 logbot[566]: 2024-09-07 15:25:35,287 DEBUG: _dispatcher: pubmsg [15:28:32] [4/8] Sep 07 15:25:36 bots171 logbot[566]: 2024-09-07 15:25:36,645 DEBUG: FROM SERVER: :RhinosF1!sid339563@user/RhinosF1 PRIVMSG #miraheze-tech-ops :!log test [15:28:32] [5/8] Sep 07 15:25:36 bots171 logbot[566]: 2024-09-07 15:25:36,645 DEBUG: _dispatcher: all_raw_messages [15:28:32] [6/8] Sep 07 15:25:36 bots171 logbot[566]: 2024-09-07 15:25:36,645 DEBUG: command: pubmsg, source: RhinosF1!sid339563@user/RhinosF1, target: #miraheze-tech-ops, arguments: ['!log test'] [15:28:33] [7/8] ``` [15:28:33] [8/8] :squint: [15:28:47] @bluemoon0332 doesn't like you [15:29:17] !log raise cloudflare tls mode to strict (tls only) [15:29:40] doesn't like you either 😂 [15:30:04] !log raise cloudflare tls mode to strict (tls only) on miraheze.org [15:32:42] it _has_ received that message, maybe it just doesn't consider it worthy of being in the SAL [15:33:35] [02ssl] 07RhinosF1 opened pull request 03#800: Update certs.yaml - 13https://github.com/miraheze/ssl/pull/800 [15:33:43] [02ssl] 07coderabbitai[bot] commented on pull request 03#800: Update certs.yaml - 13https://github.com/miraheze/ssl/pull/800#issuecomment-2335544906 [15:34:31] !log turned on authenticated origin pull on miraheze.org [15:34:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:35:03] it has developed a conscience and is deciding on it's own which messages to log [15:35:06] !log raise cloudflare tls mode to strict (tls only) on miraheze.org [15:35:10] it's the only logical conclusion [15:35:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:35:55] Fuck the singularity is here already [15:36:02] [02ssl] 07coderabbitai[bot] edited a comment on pull request 03#800: Update certs.yaml - 13https://github.com/miraheze/ssl/pull/800#issuecomment-2335544906 [15:36:22] [02ssl] 07redbluegreenhat closed pull request 03#800: Update certs.yaml - 13https://github.com/miraheze/ssl/pull/800 [15:36:23] [02ssl] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d06e4708d12e...28569b43051a [15:36:25] [02ssl] 07RhinosF1 0328569b4 - Update certs.yaml (#800) [15:38:35] [02puppet] 07redbluegreenhat closed pull request 03#3916: T12554: Enable mandatory authenticated origin pull on domains using Cloudflare - 13https://github.com/miraheze/puppet/pull/3916 [15:38:38] [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c42d430e294a...262f644a4741 [15:38:41] [02puppet] 07redbluegreenhat 03262f644 - T12554: Enable mandatory authenticated origin pull on domains using Cloudflare (#3916) [15:38:42] [02puppet] 07redbluegreenhat deleted branch 03T12554 - 13https://github.com/miraheze/puppet [15:38:45] [02puppet] 07redbluegreenhat deleted branch 03T12554 [15:42:25] [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [15:43:24] [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/262f644a4741...e462765d1cb9 [15:43:26] [02puppet] 07redbluegreenhat 03e462765 - Turn off client cert verification entirely outside of Cloudflare [15:47:12] PROBLEM - puppet181 Puppet on puppet181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_puppet] [15:49:08] yeah... CA is testing his mattermost PR so that's why this is failing [16:03:36] [02ssl] 07redbluegreenhat created branch 03ssl-182 - 13https://github.com/miraheze/ssl [16:03:37] [02ssl] 07redbluegreenhat pushed 031 commit to 03ssl-182 [+0/-0/±1] 13https://github.com/miraheze/ssl/commit/002e19b96452 [16:03:40] [02ssl] 07redbluegreenhat 03002e19b - Add wiki.graalmilitary.com [16:03:55] [02ssl] 07redbluegreenhat opened pull request 03#801: Add wiki.graalmilitary.com - 13https://github.com/miraheze/ssl/pull/801 [16:04:04] [02ssl] 07coderabbitai[bot] commented on pull request 03#801: Add wiki.graalmilitary.com - 13https://github.com/miraheze/ssl/pull/801#issuecomment-2335708079 [16:04:35] @bluemoon0332 I really think mediawiki should be the default site [16:04:50] But we need to think about all the sites we have that should be serving landing [16:05:01] It's at least the apex and www of every domain in cloudflare [16:05:25] [02ssl] 07redbluegreenhat closed pull request 03#801: Add wiki.graalmilitary.com - 13https://github.com/miraheze/ssl/pull/801 [16:05:28] [02ssl] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/28569b43051a...75b888ed79db [16:05:31] [02ssl] 07redbluegreenhat 0375b888e - Add wiki.graalmilitary.com (#801) [16:05:32] [02ssl] 07redbluegreenhat deleted branch 03ssl-182 [16:05:33] [02ssl] 07redbluegreenhat deleted branch 03ssl-182 - 13https://github.com/miraheze/ssl [16:05:37] [02ssl] 07coderabbitai[bot] edited a comment on pull request 03#801: Add wiki.graalmilitary.com - 13https://github.com/miraheze/ssl/pull/801#issuecomment-2335708079 [16:32:58] [02ssl] 07RhinosF1 opened pull request 03#802: Update certs.yaml - 13https://github.com/miraheze/ssl/pull/802 [16:33:04] [02ssl] 07coderabbitai[bot] commented on pull request 03#802: Update certs.yaml - 13https://github.com/miraheze/ssl/pull/802#issuecomment-2335852410 [16:33:21] [02ssl] 07RhinosF1 edited pull request 03#802: +armoredpatrolremasteredcom - 13https://github.com/miraheze/ssl/pull/802 [16:35:42] [02ssl] 07coderabbitai[bot] edited a comment on pull request 03#802: +armoredpatrolremasteredcom - 13https://github.com/miraheze/ssl/pull/802#issuecomment-2335852410 [17:58:09] [02ssl] 07Universal-Omega closed pull request 03#802: +armoredpatrolremasteredcom - 13https://github.com/miraheze/ssl/pull/802 [17:58:11] [02ssl] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/75b888ed79db...0222cfbbc2c0 [17:58:13] [02ssl] 07RhinosF1 030222cfb - +armoredpatrolremasteredcom (#802) [18:10:53] @cosmicalpha can you do ManageWiki too? [18:17:48] Sure! [18:18:51] @rhinosf1 what wiki? I see no task [18:21:34] [02puppet] 07Universal-Omega pushed 033 commits to 03mattermost [+0/-0/±3] 13https://github.com/miraheze/puppet/compare/80bf7adacc99...0455490cd4c0 [18:21:36] [02puppet] 07Universal-Omega 030455490 - Merge branch 'master' into mattermost [18:21:36] [02puppet] 07Universal-Omega synchronize pull request 03#3913: Add mattermost module - 13https://github.com/miraheze/puppet/pull/3913 [18:23:09] @cosmicalpha they emailed [18:23:21] Cause we took request ssl of the instructions for unknown reason [18:23:34] Oh right [18:25:24] PROBLEM - mw181 MediaWiki Rendering on mw181 is WARNING: HTTP WARNING: HTTP/1.1 400 Bad Request - 3958 bytes in 0.010 second response time [18:25:27] @cosmicalpha for the record, requestssl is 100% staying [18:25:37] I want to fully automate it [18:25:37] PROBLEM - test151 MediaWiki Rendering on test151 is WARNING: HTTP WARNING: HTTP/1.1 400 Bad Request - 3938 bytes in 0.010 second response time [18:25:41] Cause it's easy enough [18:25:50] PROBLEM - mw151 MediaWiki Rendering on mw151 is WARNING: HTTP WARNING: HTTP/1.1 400 Bad Request - 3958 bytes in 0.009 second response time [18:25:53] PROBLEM - mw161 MediaWiki Rendering on mw161 is WARNING: HTTP WARNING: HTTP/1.1 400 Bad Request - 3958 bytes in 0.010 second response time [18:26:03] what happened? [18:26:12] PROBLEM - mw162 MediaWiki Rendering on mw162 is WARNING: HTTP WARNING: HTTP/1.1 400 Bad Request - 3958 bytes in 0.010 second response time [18:26:18] So requestssl can use the CF API, update certs.yaml (probably simplified) and add the domain in ManageWiki [18:26:32] PROBLEM - mw171 MediaWiki Rendering on mw171 is WARNING: HTTP WARNING: HTTP/1.1 400 Bad Request - 3958 bytes in 0.009 second response time [18:26:39] MacFan took RequestSSL off the steps on the custom domain page [18:26:42] I restored them [18:26:45] PROBLEM - mwtask171 MediaWiki Rendering on mwtask171 is WARNING: HTTP WARNING: HTTP/1.1 400 Bad Request - 3962 bytes in 0.010 second response time [18:26:45] Cause it's needed [18:26:51] No I mean with icinga [18:26:53] PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is WARNING: HTTP WARNING: HTTP/1.1 400 Bad Request - 3962 bytes in 0.013 second response time [18:26:56] PROBLEM - mw152 MediaWiki Rendering on mw152 is WARNING: HTTP WARNING: HTTP/1.1 400 Bad Request - 3958 bytes in 0.010 second response time [18:27:06] PROBLEM - mw172 MediaWiki Rendering on mw172 is WARNING: HTTP WARNING: HTTP/1.1 400 Bad Request - 3958 bytes in 0.009 second response time [18:27:19] PROBLEM - mw182 MediaWiki Rendering on mw182 is WARNING: HTTP WARNING: HTTP/1.1 400 Bad Request - 3958 bytes in 0.009 second response time [18:27:35] Oh [18:27:59] @cosmicalpha can we add redirects to ManageWiki or RequestSSL too [18:30:05] I can maybe explain [18:30:35] RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:30:38] mediawiki is working, however there's probably some unintended side effects with Icinga regarding my latest change [18:31:41] Done [18:31:46] Oh the origin pulls [18:31:47] Oh ye [18:31:51] Icinga won't like that [18:32:09] I mean it proves it works [18:32:19] Oh yeah I just pulled that to puppet [18:32:22] We could just update icinga to see if cloudflare thinks the origin is up [18:32:23] gotta see the positive side of things [18:32:36] It's working exactly as intended [18:32:37] It may be possible [18:32:45] I'd like to consider it [18:34:44] @cosmicalpha when is puppet181 going to pull the master branch again? [18:34:48] just for curiosity [18:35:22] yep, domains with it disabled like allthetropes.org are also working [18:35:30] other than Icinga it is working [18:35:53] ye the icinga alerts aren't a bug [18:36:00] it's behaving perfectly [18:36:37] @bluemoon0332 what's the best fix here? [18:36:54] well first I want to see what wiki that check is using [18:36:59] then we can go from there [18:38:13] https://github.com/miraheze/puppet/blob/master/modules/monitoring/files/commands.conf#L201 [18:38:26] Oh right I guess I could merge the mattermost one for now since it all works (installed on the unused cp26 for now, but this will be a new VPS soon). I am mostly just running functionality tests for it now making sure it will behave good and has everything we need. [18:38:50] @bluemoon0332 https://github.com/miraheze/puppet/blob/master/hieradata/common.yaml#L21 [18:38:52] loginwiki [18:38:59] could move to a legacy wiki for now [18:39:17] well anything on miraheze.org requires a client cert from cloudflare so that's not gonna work [18:39:32] allthetropes _is_ an option [18:40:14] which cps are turned off cause I'll delete them from the old mw-lb in cloudflare [18:40:31] which still exists in case we want to retain it for any reason when we immediately switch [18:40:47] as (mw-lb.wikitide.net and mw-lb-legacy.miraheze.org) [18:41:04] only cp3 are on* [18:41:59] done [18:42:07] also the custom domain user emailed and said thanks [18:44:01] !log depooled and deleted non-us cps from legacy cp load balancer in cloudflare [18:44:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:44:24] Thanks! [18:45:20] [02puppet] 07Universal-Omega closed pull request 03#3913: Add mattermost module - 13https://github.com/miraheze/puppet/pull/3913 [18:45:21] [02puppet] 07Universal-Omega pushed 031 commit to 03master [+71/-0/±3] 13https://github.com/miraheze/puppet/compare/e462765d1cb9...264c67ffaca8 [18:45:24] [02puppet] 07Universal-Omega 03264c67f - Add mattermost module [18:45:25] [02puppet] 07Universal-Omega deleted branch 03mattermost - 13https://github.com/miraheze/puppet [18:45:26] [02puppet] 07Universal-Omega deleted branch 03mattermost [18:46:15] we can use cp as VPS? [18:46:16] Goodnight cp*'s [18:46:25] You have served us well [18:46:26] @bluemoon0332 puppet is now back on master [18:46:31] nice [18:46:33] It is a regular VPS after all [18:46:57] we could wipe them and use them for miraheze toolforge [18:47:01] lmao [18:47:16] I’m down [18:47:28] long way off [18:47:41] we could use nexttide for the toolforge domain [18:47:42] I did think about something similar during the provisioning of bots171 [18:47:50] @agentisai we have no migration plan for people using our dns [18:48:01] we kind of do [18:48:21] waiting an unspecified amount of time for browsers to implement SVCB AliasMode is the current plan [18:48:31] yeah, I was just kidding about the fact that CA said that cp3x are the only servers on right now [18:49:09] that's not a plan [18:49:11] just send a patch upstream [18:49:14] eazy [18:49:16] that could take decades [18:49:36] our best is probably wait until i'm back from holiday and chase our program manager [18:49:55] to get us IPs we can use [18:50:17] it's anycast anyway i think [18:58:48] !log turned on authenticated origin pulls for mirabeta.org [18:58:54]