[00:16:45] 10wikitech.wikimedia.org, 10Wikidata: Enable interwiki links to/from Wikitech - https://phabricator.wikimedia.org/T290147#10205516 (10Bugreporter) This requires making Wikitech a Wikidata client. [02:03:49] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [02:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:03:49] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [07:18:23] FIRING: [3x] ProbeDown: Service virt.cloudgw.eqiad1.wikimediacloud.org:0 has failed probes (icmp_virt_cloudgw_eqiad1_wikimediacloud_org_ip4) - https://wikitech.wikimedia.org/wiki/Network_monitoring#ProbeDown - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [07:18:55] 06cloud-services-team: ProbeDown - https://phabricator.wikimedia.org/T376589 (10phaultfinder) 03NEW [07:18:55] PROBLEM - Host checker.tools.wmflabs.org is DOWN: PING CRITICAL - Packet loss = 100% [07:18:55] RECOVERY - Host checker.tools.wmflabs.org is UP: PING WARNING - Packet loss = 33%, RTA = 111.24 ms [07:23:15] FIRING: [4x] ProbeDown: Service virt.cloudgw.eqiad1.wikimediacloud.org:0 has failed probes (icmp_virt_cloudgw_eqiad1_wikimediacloud_org_ip4) - https://wikitech.wikimedia.org/wiki/Network_monitoring#ProbeDown - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [07:23:24] 06cloud-services-team: ProbeDown - https://phabricator.wikimedia.org/T376589#10205684 (10phaultfinder) [07:43:15] RESOLVED: [2x] ProbeDown: Service virt.cloudgw.eqiad1.wikimediacloud.org:0 has failed probes (icmp_virt_cloudgw_eqiad1_wikimediacloud_org_ip4) - https://wikitech.wikimedia.org/wiki/Network_monitoring#ProbeDown - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [07:55:38] 06cloud-services-team: ProbeDown - https://phabricator.wikimedia.org/T376589#10205754 (10aborrero) cloudgw1002 had a kernel problem related to the NIC driver: `lines=10 [Mon Oct 7 07:17:27 2024] ------------[ cut here ]------------ [Mon Oct 7 07:17:27 2024] NETDEV WATCHDOG: enp101s0f0np0 (bnxt_en): transmit q... [08:03:27] 06cloud-services-team: cloudgw1002: ProbeDown - https://phabricator.wikimedia.org/T376589#10205762 (10aborrero) 05Open→03In progress p:05Triage→03High a:03aborrero [08:06:27] FIRING: SystemdUnitDown: The service unit cadvisor.service is in failed status on host cloudgw1002. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudgw1002 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [08:11:27] RESOLVED: SystemdUnitDown: The service unit cadvisor.service is in failed status on host cloudgw1002. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudgw1002 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [08:58:21] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10205917 (10hashar) >>! In T376267#10202260, @Ladsgroup wrote: > @hashar Please try https://wikitech.wikimedia.org/wiki/Special:PasswordReset and reset your password. https://idm.wikimedia.or... [09:13:26] 06cloud-services-team: cloudgw1002: ProbeDown - https://phabricator.wikimedia.org/T376589#10205951 (10aborrero) [09:14:11] 06cloud-services-team: cloudgw1002: network interface problem - https://phabricator.wikimedia.org/T376589#10205952 (10aborrero) [09:31:33] 06cloud-services-team, 10Toolforge: Support jdk21 on toolforge - https://phabricator.wikimedia.org/T346477#10206006 (10dcaro) >>! In T346477#10204756, @Benjavalero wrote: > I don't think this issue is resolved, at least until java-21 is one of the options when running `webservices --help`. > To clarify,... [09:31:41] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad: cloudgw1002: network interface problem - https://phabricator.wikimedia.org/T376589#10206010 (10aborrero) [09:33:46] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad: cloudgw1002: network interface problem - https://phabricator.wikimedia.org/T376589#10206023 (10aborrero) hey @VRiley-WMF or @Jclark-ctr have you seen this error before on any network card or related? rings any bell? Do you think that upgrading the NIC... [09:34:53] (03approved) 10fnegri: projects: refactor back into a single file [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/88 (https://phabricator.wikimedia.org/T375283) (owner: 10aborrero) [09:40:15] (03approved) 10dcaro: api: add auth [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/16 (owner: 10sstefanova) [09:47:33] (03Abandoned) 10Slyngshede: labsauth: Allow OATH to be disabled for all users. [labs/striker] - 10https://gerrit.wikimedia.org/r/1073742 (owner: 10Slyngshede) [09:51:14] (03update) 10sstefanova: api: add auth [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/16 [09:51:21] (03merge) 10sstefanova: api: add auth [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/16 [09:53:22] (03CR) 10FNegri: [C:03+1] "LGTM" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1076980 (owner: 10David Caro) [09:54:24] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: DONOTMERGE components-api: bump to 0.0.29-20241002095441-cd2060f1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/544 [10:01:08] (03CR) 10FNegri: [C:03+1] "LGTM" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1075042 (owner: 10David Caro) [10:03:49] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [10:09:32] (03merge) 10aborrero: projects: refactor back into a single file [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/88 (https://phabricator.wikimedia.org/T375283) [10:11:00] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [10:11:29] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [10:55:54] 06cloud-services-team, 10Toolforge: Support jdk21 on toolforge - https://phabricator.wikimedia.org/T346477#10206311 (10Benjavalero) >>! El T346477#10206006, @dcaro escribió: > To clarify, what you are requesting is to have support for a pre-built template of java-21, or to advertise the availability of jav... [11:24:17] (03open) 10aborrero: codfw1dev: add external IPv6 to main router [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/89 (https://phabricator.wikimedia.org/T375847) [11:27:09] (03merge) 10aborrero: codfw1dev: add external IPv6 to main router [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/89 (https://phabricator.wikimedia.org/T375847) [11:27:28] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [11:28:14] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [11:38:08] (03open) 10aborrero: codfw1dev: fix external IPv6 router setting [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/90 (https://phabricator.wikimedia.org/T375847) [11:46:02] 10Tool-Global-user-contributions, 10Special:GlobalContributions, 06Stewards-and-global-tools, 10Temporary accounts (Blockers to minor pilot wiki deployment): Implement Global Contributions as a central page on Meta and implement redirects from other projects - https://phabricator.wikimedia.org/T376612 (10Ni... [11:57:09] (03update) 10aborrero: codfw1dev: fix external IPv6 router setting [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/90 (https://phabricator.wikimedia.org/T375847) [12:05:38] (03merge) 10aborrero: codfw1dev: fix external IPv6 router setting [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/90 (https://phabricator.wikimedia.org/T375847) [12:05:43] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [12:06:13] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [12:06:17] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [12:17:43] 06cloud-services-team, 10Cloud-VPS, 07Epic: tofu-infra: introduce additional gitlab-ci automation - https://phabricator.wikimedia.org/T370652#10206515 (10dcaro) >>! In T370652#10196313, @aborrero wrote: >>>! In T370652#10196276, @aborrero wrote: >> See: >> >> * https://gitlab.wikimedia.org/bd808/deployment-... [12:17:56] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [12:32:20] 06cloud-services-team, 10Toolforge: Support jdk21 on toolforge - https://phabricator.wikimedia.org/T346477#10206534 (10dcaro) >>! In T346477#10206311, @Benjavalero wrote: >>>! El T346477#10206006, @dcaro escribió: >> To clarify, what you are requesting is to have support for a pre-built template of java-21... [12:33:44] (03open) 10aborrero: eqiad1: delete wmdeanalytics project [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/91 (https://phabricator.wikimedia.org/T371696) [12:34:25] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), 10Release-Engineering-Team (Seen): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org - https://phabricator.wikimedia.org/T374830#10206549 (10aborrero) Today... [12:34:46] (03PS1) 10Brouberol: Provision dummy secrets for ceph-csi users [labs/private] - 10https://gerrit.wikimedia.org/r/1078401 [12:35:16] (03PS2) 10Brouberol: Provision dummy secrets for ceph-csi users [labs/private] - 10https://gerrit.wikimedia.org/r/1078401 (https://phabricator.wikimedia.org/T376407) [12:36:36] (03approved) 10sstefanova: eqiad1: delete wmdeanalytics project [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/91 (https://phabricator.wikimedia.org/T371696) (owner: 10aborrero) [12:47:48] (03merge) 10aborrero: eqiad1: delete wmdeanalytics project [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/91 (https://phabricator.wikimedia.org/T371696) [12:48:06] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [12:48:36] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [12:49:18] 06cloud-services-team, 10Cloud-VPS, 10Wikidata, 10Wikidata Analytics, 13Patch-For-Review: Delete the wmdeanalytics Cloud VPS project - https://phabricator.wikimedia.org/T371696#10206592 (10aborrero) 05Open→03Resolved [12:55:30] 06cloud-services-team, 10Cloud-VPS, 10Wikidata, 10Wikidata Analytics, 13Patch-For-Review: Delete the wmdeanalytics Cloud VPS project - https://phabricator.wikimedia.org/T371696#10206595 (10AndrewTavis_WMDE) Thanks so much for the support here, @aborrero! [12:57:56] (03update) 10dcaro: storage: add k8s storage [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/14 [13:10:17] (03open) 10raymond-ndibe: Draft: [lima-kilo] test k8s Installation [repos/cloud/toolforge/lima-kilo] (configure_high_availability) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/193 (https://phabricator.wikimedia.org/T362867) [13:10:32] (03update) 10dcaro: storage: add k8s storage [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/14 [13:20:41] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:26:32] 06cloud-services-team, 10Toolforge, 10Elasticsearch: Deploy multi-tentant OpenSearch cluster as replacement for Elasticsearch - https://phabricator.wikimedia.org/T348943#10206714 (10bd808) [13:42:37] (03update) 10raymond-ndibe: Draft: [lima-kilo] test k8s Installation [repos/cloud/toolforge/lima-kilo] (configure_high_availability) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/193 (https://phabricator.wikimedia.org/T362867) [13:52:44] (03update) 10sstefanova: storage: add k8s storage [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/14 (owner: 10dcaro) [13:59:33] (03approved) 10sstefanova: storage: add k8s storage [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/14 (owner: 10dcaro) [13:59:58] (03update) 10sstefanova: storage: add k8s storage [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/14 (owner: 10dcaro) [14:00:09] 10Tool-Global-user-contributions, 10Special:GlobalContributions, 06Stewards-and-global-tools, 10Temporary accounts (Blockers to minor pilot wiki deployment): Implement Global Contributions as a central page on Meta and implement redirects from other projec... - https://phabricator.wikimedia.org/T376612#10206840 [14:00:10] (03update) 10sstefanova: storage: add k8s storage [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/14 (owner: 10dcaro) [14:03:49] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [14:30:50] (03open) 10aborrero: tofu-infra: delete test projects [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/92 [15:02:19] 10wikitech.wikimedia.org: OAuth consumers registered locally at Wikitech are no longer configured to be used - https://phabricator.wikimedia.org/T376188#10207047 (10Ladsgroup) >>! In T376188#10197509, @Umherirrender wrote: > Maybe empty the local oauth tables when no longer used to avoid orphaned rows (maybe as... [15:22:42] 10Tool-Global-user-contributions, 10Special:GlobalContributions, 06Stewards-and-global-tools, 10Temporary accounts (Blockers to minor pilot wiki deployment): Implement Global Contributions as a central page on Meta and implement redirects from other projec... - https://phabricator.wikimedia.org/T376612#10207118 [15:29:41] 10Tool-Global-user-contributions, 10Special:GlobalContributions, 06Stewards-and-global-tools, 10Temporary accounts (Blockers to minor pilot wiki deployment): Implement Global Contributions as a central page on Meta and implement redirects from other projec... - https://phabricator.wikimedia.org/T376612#10207151 [15:37:25] RECOVERY - Host cloudcephosd1025 is UP: PING WARNING - Packet loss = 80%, RTA = 30.69 ms [15:43:49] PROBLEM - Host cloudcephosd1025 is DOWN: PING CRITICAL - Packet loss = 100% [16:32:33] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install cloudlb2004-dev - https://phabricator.wikimedia.org/T370678#10207365 (10Jhancock.wm) [16:32:41] (03open) 10fnegri: Draft: test new project module [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93 [16:46:34] 10Tool-Global-user-contributions, 10Special:GlobalContributions, 06Stewards-and-global-tools, 10Temporary accounts (Blockers to minor pilot wiki deployment): Implement Global Contributions as a central page on Meta and implement redirects from other projec... - https://phabricator.wikimedia.org/T376612#10207404 [17:15:55] 10Cloud-VPS (Quota-requests): Request floating IP for wikiwho project - https://phabricator.wikimedia.org/T376637 (10MusikAnimal) 03NEW [17:17:25] RECOVERY - Host cloudcephosd1025 is UP: PING WARNING - Packet loss = 90%, RTA = 0.25 ms [17:20:41] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:23:49] PROBLEM - Host cloudcephosd1025 is DOWN: PING CRITICAL - Packet loss = 100% [18:03:49] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [18:16:23] 10Tool-video-answer-tool, 06Future-Audiences, 07Spike: Image layout adjustment for lower-res images - https://phabricator.wikimedia.org/T375690#10207717 (10derenrich) i accidentally merged a "fix" for this at https://gitlab.wikimedia.org/repos/future-audiences/video-answer-tool/-/commit/bf30d9f0a416c9df76ee8... [18:20:58] 10VPS-project-Wikistats: Add ibawiki to wikistats - https://phabricator.wikimedia.org/T376576#10207789 (10Dzahn) 05Open→03Stalled [18:21:48] 10VPS-project-Wikistats: Add ibawiki to wikistats - https://phabricator.wikimedia.org/T376576#10207796 (10Dzahn) Shouldn't be created until T376568 is resolved since it is blocked on that. [19:15:17] 06cloud-services-team, 10Toolforge: [infra, k8s, webservice] remove deprecated kubectl --wait flag before k8s 1.29 upgrade - https://phabricator.wikimedia.org/T373866#10208001 (10LucasWerkmeister) The warning still appears, by the way – I guess someone™ still needs to deploy the new version? (I might techn... [19:20:13] 10wikitech.wikimedia.org: Consider enabling local thumbnail cache on wikitech-static - https://phabricator.wikimedia.org/T376650 (10Krinkle) 03NEW [19:23:03] 10wikitech.wikimedia.org: Consider enabling local thumbnail cache on wikitech-static - https://phabricator.wikimedia.org/T376650#10208040 (10Krinkle) [20:11:37] (03CR) 10Eugene233: [C:03+2] Use tasks for updating images when a campaign is updated (031 comment) [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1055949 (https://phabricator.wikimedia.org/T367397) (owner: 10Sebastian Berlin (WMSE)) [20:11:59] (03Merged) 10jenkins-bot: Use tasks for updating images when a campaign is updated [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1055949 (https://phabricator.wikimedia.org/T367397) (owner: 10Sebastian Berlin (WMSE)) [20:12:47] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install cloudlb2004-dev - https://phabricator.wikimedia.org/T370678#10208231 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host cloudlb2004-dev.codfw.wmnet with OS bookworm [20:38:19] 10Tool-video-answer-tool, 06Future-Audiences: Add ability to edit attribution and swap image in video tool - https://phabricator.wikimedia.org/T376659 (10Maryana) 03NEW [20:40:41] 10Tool-video-answer-tool, 06Future-Audiences: Create video for arbitrary topic - https://phabricator.wikimedia.org/T376660#10208418 (10Maryana) [20:41:03] 10Tool-video-answer-tool, 06Future-Audiences: Create video for arbitrary topic - https://phabricator.wikimedia.org/T376660#10208422 (10derenrich) a:03derenrich [20:42:20] 10Tool-video-answer-tool, 06Future-Audiences: Ensure Image Attribution Is Readable - https://phabricator.wikimedia.org/T375830#10208434 (10derenrich) 05Open→03Resolved [20:42:30] 10Tool-video-answer-tool, 06Future-Audiences: Add element to video preview to allow for text editing - https://phabricator.wikimedia.org/T375933#10208431 (10derenrich) 05Open→03Resolved a:03derenrich [20:42:56] 10Tool-video-answer-tool, 06Future-Audiences: Improvements to video server-side rendering - https://phabricator.wikimedia.org/T375408#10208438 (10derenrich) [20:45:15] 10Tool-video-answer-tool, 06Future-Audiences: Improvements to video server-side rendering - https://phabricator.wikimedia.org/T375408#10208443 (10derenrich) quota increase was requested at https://phabricator.wikimedia.org/T376505 [21:00:16] 10Tool-video-answer-tool, 06Future-Audiences: Add ability to edit attribution and swap image in video tool - https://phabricator.wikimedia.org/T376659#10208458 (10etz) a:03etz [21:01:21] 10Tool-video-answer-tool, 06Future-Audiences: Generate first batch of videos with latest styling - https://phabricator.wikimedia.org/T375888#10208459 (10etz) a:03etz [21:02:33] 10Tool-video-answer-tool, 06Future-Audiences: Generate first batch of videos with latest styling - https://phabricator.wikimedia.org/T375888#10208461 (10etz) 05Open→03Resolved [21:03:52] 10Tool-video-answer-tool, 06Future-Audiences: Implement attribution requirements for demo video - https://phabricator.wikimedia.org/T374376#10208463 (10etz) 05Open→03Resolved [21:14:51] 10wikitech.wikimedia.org: Consider enabling local thumbnail cache on wikitech-static - https://phabricator.wikimedia.org/T376650#10208485 (10JJMC89) [21:20:41] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:25:23] 10Tool-toolwatch: Implementing alert system to notify maintainers of downtime - https://phabricator.wikimedia.org/T368816#10208506 (10Nux) I wouldn't mind getting notifications, but keep in mind that this might also be a per-tool preference (not only per-dev). # Some of the tools I'm assigned to are used dail... [21:33:06] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install cloudlb2004-dev - https://phabricator.wikimedia.org/T370678#10208555 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host cloudlb2004-dev.codfw.wmnet with OS bookworm executed... [22:03:49] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [22:21:46] 10Tool-nfp: Allow filtering by user edit count - https://phabricator.wikimedia.org/T364698#10208803 (10JayCubby) I think it's fair to assume that relatively high-age accounts with several edits are experienced users, but there could be several categories (5 uploads, 30 days; 10 uploads 1mo; 6mo 20; etc). [22:44:22] 10cloud-services-team (FY2024/2025-Q1-Q2): [openstack object storage] deleted files still occupying space - https://phabricator.wikimedia.org/T376673 (10Raymond_Ndibe) 03NEW [22:47:32] FIRING: PuppetCertificateAboutToExpire: Puppet CA certificate mwv-builder-03.mediawiki-vagrant.eqiad.wmflabs is about to expire in 17d 23h 58m 34s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire