[00:02:33] 10PAWS: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T376556#10236281 (10LibUp-bot) A new upstream version of Pywikibot is now available: 9.4.1. * https://gerrit.wikimedia.org/g/pywikibot/core/+/refs/tags/9.4.1 * https://doc.wikimedia.org/pywikibot/stable/changelog.html [00:02:34] 06cloud-services-team, 10Toolforge: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T370115#10236282 (10LibUp-bot) A new upstream version of Pywikibot is now available: 9.4.1. * https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Pywikibot_image * https://gerrit.wikimedia.org/g/py... [01:21:28] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [01:41:16] 10Quarry: Quarry shows error: This web service cannot be reached - https://phabricator.wikimedia.org/T375988#10236338 (10GTrang) [01:41:17] 10Quarry: worker nodes issue with garbage collection - https://phabricator.wikimedia.org/T375997#10236339 (10GTrang) [01:43:16] 10Quarry: Quarry shows error: This web service cannot be reached - https://phabricator.wikimedia.org/T375988#10236341 (10GTrang) 05Resolved→03Open >>! In T375988#10186586, @rook wrote: > Quarry is working again. Though I didn't have time to investigate what is happening so this may happen again. Opening T375... [02:07:20] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [02:18:21] 10Tool-ldap: Shut down ldap-beta tool - https://phabricator.wikimedia.org/T377409 (10Legoktm) 03NEW [02:35:12] 10Tool-video-answer-tool, 06Future-Audiences: Display true 1920x1080 video resolution for the video tool - https://phabricator.wikimedia.org/T375296#10236393 (10etz) @Maryana yes it is [02:35:21] 10Tool-video-answer-tool, 06Future-Audiences: Display true 1920x1080 video resolution for the video tool - https://phabricator.wikimedia.org/T375296#10236394 (10etz) 05Open→03Resolved [03:54:50] 10wikitech.wikimedia.org, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: Wikitech showing Wikipedia CentralNotice banners - https://phabricator.wikimedia.org/T377030#10236454 (10jeremyb) a:03Ejegg @Ejegg looks good to me. I saw no banner there and then adde... [05:21:28] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:07:20] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [09:21:04] 06cloud-services-team, 10Toolforge: Introduce health checks for Toolforge Jobs Framework - https://phabricator.wikimedia.org/T377420 (10Urbanecm) 03NEW [09:21:28] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:21:42] 06cloud-services-team, 10Toolforge: Introduce health checks for Toolforge Jobs Framework - https://phabricator.wikimedia.org/T377420#10236883 (10Urbanecm) Related IRC conversation from -cloud: `lang=irc 10:53 hey! any idea why a toolforge job would just freeze for hours/days? Is there a way to defi... [09:21:59] 06cloud-services-team, 10Toolforge: Introduce health checks for Toolforge Jobs Framework cronjobs - https://phabricator.wikimedia.org/T377420#10236885 (10aborrero) [09:24:41] 06cloud-services-team, 10Toolforge: Introduce health checks for Toolforge Jobs Framework cronjobs - https://phabricator.wikimedia.org/T377420#10236887 (10aborrero) change would be: * move healthcheck template into the pod template, see: ** https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/blob/mai... [09:25:57] 06cloud-services-team, 10Toolforge (Toolforge iteration 16): Introduce health checks for Toolforge Jobs Framework cronjobs - https://phabricator.wikimedia.org/T377420#10236901 (10aborrero) [09:26:53] 06cloud-services-team, 10Toolforge (Toolforge iteration 16): Introduce health checks for Toolforge Jobs Framework cronjobs - https://phabricator.wikimedia.org/T377420#10236902 (10aborrero) p:05Triage→03Medium [09:38:20] (03CR) 10D3r1ck01: Use a CSP policy to reduce risk of XSS (031 comment) [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1080836 (https://phabricator.wikimedia.org/T377168) (owner: 10Brian Wolff) [09:49:16] FIRING: [3x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:54:16] FIRING: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:59:16] RESOLVED: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:07:20] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [10:41:26] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10237161 (10CanonNi) >>! In T376267#10235546, @Ladsgroup wrote: > @CanonNi I can't find your user in wikitech. Are you sure you have a user there? Uh... turns out I didn't. I have a Develope... [11:29:26] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS, 13Patch-For-Review: openstack: develop a script to migrate a VM instance from the old network setting (vlan) to the new (vxlan, IPv6) - https://phabricator.wikimedia.org/T377346#10237311 (10aborrero) I found a couple of nice shortcuts in the openstack C... [11:58:18] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS, 13Patch-For-Review: openstack: develop a script to migrate a VM instance from the old network setting (vlan) to the new (vxlan, IPv6) - https://phabricator.wikimedia.org/T377346#10237412 (10aborrero) problem seems to be this file: `lang=shell-session $... [12:06:05] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10237447 (10Ladsgroup) If you want to be able to edit wikitech, we can make it happen. If you don't want to, we can wait until end of November. [12:25:15] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1081132 (owner: 10L10n-bot) [12:32:00] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS, 13Patch-For-Review: openstack: develop a script to migrate a VM instance from the old network setting (vlan) to the new (vxlan, IPv6) - https://phabricator.wikimedia.org/T377346#10237516 (10aborrero) Manually editing the file `/etc/netplan/50-cloud-init... [12:45:06] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS, 13Patch-For-Review: openstack: develop a script to migrate a VM instance from the old network setting (vlan) to the new (vxlan, IPv6) - https://phabricator.wikimedia.org/T377346#10237569 (10aborrero) It seems not all VMs can be configured that way. Usi... [12:50:37] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: Migrate Cloud VPS instances to VXLAN based networks - https://phabricator.wikimedia.org/T364725#10237596 (10aborrero) the script being developed in {T377346} is proving to be challenging for a number of reasons. I wonder if instead of force-migrating VMs... [13:21:28] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:07:20] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [14:43:22] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: cloudgw1002: network interface problem - https://phabricator.wikimedia.org/T376589#10238164 (10aborrero) 05In progress→03Resolved thanks! [14:51:12] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: Decision Request - How to do the Cloud VPS VXLAN/IPv6 migration - https://phabricator.wikimedia.org/T377467 (10aborrero) 03NEW [14:52:53] 10Cloud Services Proposals, 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: Decision Request - How to do the Cloud VPS VXLAN/IPv6 migration - https://phabricator.wikimedia.org/T377467#10238220 (10aborrero) p:05Triage→03Medium [14:54:27] 10wikitech.wikimedia.org, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: Wikitech showing Wikipedia CentralNotice banners - https://phabricator.wikimedia.org/T377030#10238217 (10Pcoombe) 05Open→03Resolved Looks good to me as well. Thanks @Ejegg, and th... [14:59:31] 10Cloud Services Proposals, 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: Decision Request - How to do the Cloud VPS VXLAN/IPv6 migration - https://phabricator.wikimedia.org/T377467#10238249 (10aborrero) [15:08:50] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10238274 (10Dominic3203) 05Open→03Resolved a:03Dominic3203 Thanks, I have connected my Wikitech account to my global account successfully. [15:09:11] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10238281 (10Dominic3203) 05Resolved→03In progress [15:11:55] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10238286 (10Dominic3203) 05In progress→03Open a:05Dominic3203→03None [15:18:16] 06cloud-services-team, 10Toolforge (Toolforge iteration 16): Introduce health checks for Toolforge Jobs Framework cronjobs - https://phabricator.wikimedia.org/T377420#10238316 (10Raymond_Ndibe) [15:52:09] 10Cloud Services Proposals, 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: Decision Request - How to do the Cloud VPS VXLAN/IPv6 migration - https://phabricator.wikimedia.org/T377467#10238497 (10aborrero) [17:21:28] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:57:46] 06cloud-services-team, 10Toolforge (Toolforge iteration 16): Introduce health checks for Toolforge Jobs Framework cronjobs - https://phabricator.wikimedia.org/T377420#10239144 (10bd808) I wonder if adding support for declaring `concurrencyPolicy: Replace` for a scheduled job would also be helpful? Something li... [18:06:40] (03PS2) 10Brian Wolff: Use a CSP policy to reduce risk of XSS [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1080836 (https://phabricator.wikimedia.org/T377168) [18:07:20] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [18:07:47] (03CR) 10Brian Wolff: Use a CSP policy to reduce risk of XSS (031 comment) [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1080836 (https://phabricator.wikimedia.org/T377168) (owner: 10Brian Wolff) [18:07:56] (03CR) 10Brian Wolff: Use a CSP policy to reduce risk of XSS (031 comment) [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1080836 (https://phabricator.wikimedia.org/T377168) (owner: 10Brian Wolff) [19:27:40] 10Tool-refill: https://refill.toolforge.org/ returns 403 - https://phabricator.wikimedia.org/T369439#10239445 (10Curb_Safe_Charmer) 05In progress→03Declined I'm closing this, as the old version of reFill is now considered dead. Long live the /ng version. [19:30:21] 10Tool-refill: https://refill.toolforge.org/ returns 403 - https://phabricator.wikimedia.org/T369439#10239449 (10Pppery) https://refill.toolforge.org/ still shouldn't be a 403. It should at the very least redirect to https://refill.toolforge.org/ng if you want to declare the old version dead [19:31:47] 10Tool-refill: https://refill.toolforge.org/ returns 403 - https://phabricator.wikimedia.org/T369439#10239455 (10Curb_Safe_Charmer) >>! In T369439#10239449, @Pppery wrote: > https://refill.toolforge.org/ still shouldn't be a 403. It should at the very least redirect to https://refill.toolforge.org/ng if you... [19:32:47] 10Tool-refill: https://refill.toolforge.org/ returns 403 - https://phabricator.wikimedia.org/T369439#10239457 (10Pppery) No. [20:08:04] 10Tool-refill: https://refill.toolforge.org/ returns 403 - https://phabricator.wikimedia.org/T369439#10239558 (10Novem_Linguae) Is https://refill.toolforge.org/ running the PHP image? If so maybe just move all the current files to a subdirectory, then add an index.php with the following code? `php (03PS1) 10Bking: analytics_test_cluster: add secret [labs/private] - 10https://gerrit.wikimedia.org/r/1081261 (https://phabricator.wikimedia.org/T374948) [20:38:31] (03PS2) 10Bking: analytics_test_cluster: add secret [labs/private] - 10https://gerrit.wikimedia.org/r/1081261 (https://phabricator.wikimedia.org/T374948) [20:38:39] (03CR) 10Bking: "check experimental" [labs/private] - 10https://gerrit.wikimedia.org/r/1081261 (https://phabricator.wikimedia.org/T374948) (owner: 10Bking) [20:43:19] 10Tool-refill: https://refill.toolforge.org/ returns 403 - https://phabricator.wikimedia.org/T369439#10239869 (10SD0001) 05Declined→03Resolved >>! In T369439#10239558, @Novem_Linguae wrote: > Is https://refill.toolforge.org/ running the PHP image? If so maybe just move all the current files to a subdirec... [21:21:28] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:07:20] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [22:24:05] FIRING: ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_toolserver_org_redirects_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:27:06] RESOLVED: ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_toolserver_org_redirects_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:47:35] FIRING: PuppetCertificateAboutToExpire: Puppet CA certificate mwv-builder-03.mediawiki-vagrant.eqiad.wmflabs is about to expire in 7d 23h 58m 34s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire