[00:07:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:12:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:25:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:30:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:02:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:07:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:52:42] 10Tool-adminscore, 06Moderator-Tools-Team, 10XTools: Redirect XTools adminscore URLs to adminscore.toolforge.org - https://phabricator.wikimedia.org/T431880#12435073 (10Legoktm) Anything I can help with here? [02:15:44] (03update) 10zhaofjx: T440420: Add line clamp to top story extract [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/29 (owner: 10chimaifeanyi29) [02:15:45] (03update) 10zhaofjx: T440420: Add line clamp to top story extract [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/29 (owner: 10chimaifeanyi29) [02:15:47] 10Tool-wikinewsie, 07good first task: Wikinewsie: Character limit issue on top story - https://phabricator.wikimedia.org/T440420#12435083 (10ZhaoFJx) p:05Triage→03Unbreak! [02:16:09] (03approved) 10zhaofjx: T440420: Add line clamp to top story extract [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/29 (owner: 10chimaifeanyi29) [02:16:24] (03merge) 10zhaofjx: T440420: Add line clamp to top story extract [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/29 (owner: 10chimaifeanyi29) [02:17:53] 10Tool-wikinewsie: 'See also' mode for Wikinewsie - https://phabricator.wikimedia.org/T434468#12435090 (10ZhaoFJx) 05Open→03In progress [02:17:54] 10Tool-wikinewsie, 07good first task: Wikinewsie: Character limit issue on top story - https://phabricator.wikimedia.org/T440420#12435088 (10ZhaoFJx) 05Open→03Resolved [02:18:24] 10Tool-wikinewsie: 'See also' mode for Wikinewsie - https://phabricator.wikimedia.org/T434468#12435093 (10ZhaoFJx) Thanks @Akash_A ! [02:21:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:26:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:27:25] FIRING: SystemdUnitFailed: remove_dangling_cinder_snapshots.service on cloudbackup2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:42:25] RESOLVED: SystemdUnitFailed: remove_dangling_cinder_snapshots.service on cloudbackup2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:17:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:22:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:23:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:33:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:58:37] (03merge) 10sadrettin: Kurulum düzeltmeleri ve saatlik tarama işi [toolforge-repos/kategori] - 10https://gitlab.wikimedia.org/toolforge-repos/kategori/-/merge_requests/1 (owner: 10project_5175_bot_1dc5cd2b4a7cadb6d1af8e29974ce1a7) [04:26:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:29:19] (03open) 10project_5175_bot_1dc5cd2b4a7cadb6d1af8e29974ce1a7: İl, ilçe ve anıt sayfaları (sadece okuma) [toolforge-repos/kategori] - 10https://gitlab.wikimedia.org/toolforge-repos/kategori/-/merge_requests/2 [04:31:01] (03merge) 10sadrettin: İl, ilçe ve anıt sayfaları (sadece okuma) [toolforge-repos/kategori] - 10https://gitlab.wikimedia.org/toolforge-repos/kategori/-/merge_requests/2 (owner: 10project_5175_bot_1dc5cd2b4a7cadb6d1af8e29974ce1a7) [04:31:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:50:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:55:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:09:21] 06cloud-services-team, 10Data-Services, 06tools-platform-team: [wikireplicas] Create views for new wiki mlwikivoyage - https://phabricator.wikimedia.org/T441176#12435220 (10Marostegui) Wiki sanitized (T441172), this can proceed [05:25:21] (03merge) 10countcount: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/54 (owner: 10renovatebot) [05:26:29] (03merge) 10countcount: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/67 (owner: 10renovatebot) [05:26:58] (03merge) 10countcount: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/28 (owner: 10renovatebot) [05:27:56] (03merge) 10countcount: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/85 (owner: 10renovatebot) [05:28:00] FIRING: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [05:28:54] (03merge) 10countcount: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/31 (owner: 10renovatebot) [05:29:10] (03merge) 10countcount: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/35 (owner: 10renovatebot) [05:29:27] (03merge) 10countcount: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/39 (owner: 10renovatebot) [05:36:32] (03open) 10countcount: Update rustls to 0.23.46 [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/32 [05:36:41] (03open) 10countcount: Update rustls to 0.23.46 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/86 [05:39:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:43:52] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [webservice-cli] use jobs-api to start the webservice - https://phabricator.wikimedia.org/T431421#12435252 (10Raymond_Ndibe) # Webservices migration To Jobs-api And Webservice-cli Deprecation Plan ## [] Add support for reading jobs-api created webservices... [05:44:08] (03update) 10countcount: Update rustls to 0.23.46 [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/32 [05:44:58] (03merge) 10countcount: Update rustls to 0.23.46 [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/32 [05:46:45] (03update) 10countcount: Update rustls to 0.23.46 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/86 [05:47:45] (03update) 10renovatebot: Update dependency happy-dom to v21 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/29 [05:47:57] (03update) 10renovatebot: Update dependency happy-dom to v21 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/29 [05:50:43] (03merge) 10countcount: Update dependency happy-dom to v21 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/29 (owner: 10renovatebot) [05:52:34] (03merge) 10countcount: Update rustls to 0.23.46 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/86 [05:54:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:03:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:08:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:01:56] 10Tool-wsindex, 10Wikisource Reader App (Android), 13Patch-For-Review: WS Index thumbnail URLs are broken if the file is redirected - https://phabricator.wikimedia.org/T432367#12435314 (10Bodhisattwa) @RCodes , please note that this task has not been assigned for Outreachy. [07:11:00] 10Tool-wsindex, 10Wikisource Reader App (Android), 13Patch-For-Review: WS Index thumbnail URLs are broken if the file is redirected - https://phabricator.wikimedia.org/T432367#12435345 (10RCodes) Hi @Bodhisattwa, thanks for clarifying. I chose this task because the microtasks listed on the Outreachy page we... [07:13:00] FIRING: [2x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [07:14:00] 10Tool-wsindex, 10Wikisource Reader App (Android): Add name_native_language to People - https://phabricator.wikimedia.org/T430705#12435350 (10RCodes) Hi @Saiphani02, @bodhisattwa, @muguro can I work on this task for my Outreachy contribution? [07:14:38] 10Tool-wsindex, 10Wikisource Reader App (Android), 13Patch-For-Review: WS Index thumbnail URLs are broken if the file is redirected - https://phabricator.wikimedia.org/T432367#12435351 (10RCodes) a:05RCodes→03None [07:18:00] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [07:31:21] 10Tool-wikicrawler: Wiki Crawler: Add Wiki Article Race page with split-screen layout - https://phabricator.wikimedia.org/T441109#12435405 (10Adityakumar0545) 05Open→03In progress [07:31:50] 10Tool-wsindex, 10Wikisource Reader App (Android), 13Patch-For-Review: WS Index thumbnail URLs are broken if the file is redirected - https://phabricator.wikimedia.org/T432367#12435407 (10Bodhisattwa) No issue. Please wait for some time until the review of your PRs are done by @Saiphani02 and @Muguro before... [07:33:25] (03update) 10rabuma: Fix thumbnail URLs for renamed Commons files (T432367) [toolforge-repos/wsindex] - 10https://gitlab.wikimedia.org/toolforge-repos/wsindex/-/merge_requests/5 (https://phabricator.wikimedia.org/T432367) [07:35:02] (03update) 10rabuma: Add Wikipedia descriptions for books (T430400) [toolforge-repos/wsindex] - 10https://gitlab.wikimedia.org/toolforge-repos/wsindex/-/merge_requests/3 (https://phabricator.wikimedia.org/T430400) [07:35:05] 10Tool-wsindex, 10Wikisource Reader App (Android): Add name_native_language to People - https://phabricator.wikimedia.org/T430705#12435413 (10Bodhisattwa) Please wait for some time until the review of your PRs are done by @Saiphani02 and @Muguro [07:36:21] (03update) 10rabuma: Add catalog summary statistics to analytics page (T430399) [toolforge-repos/wsindex] - 10https://gitlab.wikimedia.org/toolforge-repos/wsindex/-/merge_requests/4 (https://phabricator.wikimedia.org/T430399) [07:40:42] 10Tool-wsindex, 10Wikisource Reader App (Android), 13Patch-For-Review: WS Index thumbnail URLs are broken if the file is redirected - https://phabricator.wikimedia.org/T432367#12435430 (10RCodes) okay @Bodhisattwa, Thank you, i will wait. [07:41:30] 10Tool-wsindex, 10Wikisource Reader App (Android): Add name_native_language to People - https://phabricator.wikimedia.org/T430705#12435431 (10RCodes) Okay @Bodhisattwa, Thank you [08:04:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:09:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:16:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:21:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:23:32] 10Tools, 10Lift-Wing, 06Machine-Learning-Team, 06Research: FactDiff: an editor worklist of conflicting dates and numbers across Wikipedia language editions (WIP, feedback welcome) - https://phabricator.wikimedia.org/T441279 (10Devdoingdev) 03NEW [08:23:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:28:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:40:22] 10Toolforge, 10Tools, 06tools-platform-team: Mix'n'match: Temporary DB size increase to reduce DB size - https://phabricator.wikimedia.org/T440708#12435640 (10fnegri) @magnusmanske all done: `lang=mysql,lines=10 MariaDB [s51434__mixnmatch_p]> -- 7.1 MariaDB [s51434__mixnmatch_p]> SET SESSION lock_wait_timeo... [08:40:35] (03merge) 10dcaro: package: add debian packaging bits [repos/cloud/toolforge/replica-cnf] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/replica-cnf/-/merge_requests/2 (https://phabricator.wikimedia.org/T441108) [08:40:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:45:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:46:49] 06cloud-services-team (Hardware), 06tools-infrastructure-team, 06DC-Ops, 06SRE: How to configure storage on new dumps servers - https://phabricator.wikimedia.org/T441202#12435665 (10dcaro) >>! In T441202#12434211, @Andrew wrote: > Claude points out, reasonably, that taking an hour to write 700GB means we a... [08:50:09] (03update) 10rabuma: Fix thumbnail URLs for renamed Commons files (T432367) [toolforge-repos/wsindex] - 10https://gitlab.wikimedia.org/toolforge-repos/wsindex/-/merge_requests/5 (https://phabricator.wikimedia.org/T432367) [08:50:29] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component replica-cnf [08:50:46] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component replica-cnf [08:52:49] (03merge) 10dcaro: toolforge_get_versions: don't fail on missing package [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1466 [08:52:53] (03update) 10dcaro: replica_cnf: add to versions and empty tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1467 (https://phabricator.wikimedia.org/T441108) [08:55:28] (03update) 10rabuma: Add catalog summary statistics to analytics page (T430399) [toolforge-repos/wsindex] - 10https://gitlab.wikimedia.org/toolforge-repos/wsindex/-/merge_requests/4 (https://phabricator.wikimedia.org/T430399) [08:57:30] (03update) 10rabuma: Add Wikipedia descriptions for books (T430400) [toolforge-repos/wsindex] - 10https://gitlab.wikimedia.org/toolforge-repos/wsindex/-/merge_requests/3 (https://phabricator.wikimedia.org/T430400) [08:57:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:00:18] (03approved) 10fnegri: replica_cnf: add to versions and empty tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1467 (https://phabricator.wikimedia.org/T441108) (owner: 10dcaro) [09:00:18] (03update) 10fnegri: replica_cnf: add to versions and empty tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1467 (https://phabricator.wikimedia.org/T441108) (owner: 10dcaro) [09:02:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:09:49] 06tools-infrastructure-team, 06tools-platform-team: Evaluate ToolforgeKubernetesNodeNotReady page-ability - https://phabricator.wikimedia.org/T441285 (10fgiunchedi) 03NEW [09:11:05] 06tools-infrastructure-team, 06tools-platform-team: Evaluate ToolforgeKubernetesNodeNotReady page-ability - https://phabricator.wikimedia.org/T441285#12435769 (10fgiunchedi) [09:15:03] RESOLVED: PuppetAgentFailure: Puppet agent failure detected on instance toolsbeta-nfs-5 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [09:15:32] 06tools-infrastructure-team: cloudlb haproxy names have diverged between codfw and eqiad - https://phabricator.wikimedia.org/T441286 (10fgiunchedi) 03NEW [09:16:23] 10Tool-leximap: Add Malayalam language to Leximap - https://phabricator.wikimedia.org/T433956#12435794 (10Gnoeee) 05Open→03Resolved a:05Mohammed_Sadat_WMDE→03Gnoeee Added the feature to list Malayalam language. [09:18:02] 06tools-infrastructure-team: cloudlb haproxy keystone proxy names have diverged between codfw and eqiad - https://phabricator.wikimedia.org/T441286#12435798 (10fgiunchedi) [09:19:41] 10Tool-leximap, 10Wikidata, 10Wikidata Integration in Wikimedia projects: Create the frontend for leximap - https://phabricator.wikimedia.org/T432250#12435802 (10Gnoeee) 05Open→03Resolved a:03Gnoeee [09:20:07] 10Tool-leximap, 10Wikidata, 10Wikidata Integration in Wikimedia projects: Create the frontend for leximap - https://phabricator.wikimedia.org/T432250#12435809 (10Gnoeee) 05Resolved→03Open a:05Gnoeee→03Dagmawi-M [09:20:23] 10Tool-leximap, 10Wikidata, 10Wikidata Integration in Wikimedia projects: Create the frontend for leximap - https://phabricator.wikimedia.org/T432250#12435814 (10Gnoeee) 05Open→03Resolved [09:31:01] 06tools-infrastructure-team: Improve CloudVirtDown runbook and drain cookbook when an host is hard-down - https://phabricator.wikimedia.org/T441289 (10fgiunchedi) 03NEW [09:32:35] 06tools-infrastructure-team: Improve CloudVirtDown runbook and drain cookbook when an host is hard-down - https://phabricator.wikimedia.org/T441289#12435903 (10fgiunchedi) [09:35:25] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team: [wmcs-cookbooks] Improve cloudvirt.drain and cloudvirt.set_maintenance - https://phabricator.wikimedia.org/T375867#12435911 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi Resolving in favor of {T441289} instead, the set_maintenance c... [09:37:35] 10Toolforge, 10Tools, 06tools-platform-team: Mix'n'match: Temporary DB size increase to reduce DB size - https://phabricator.wikimedia.org/T440708#12435932 (10magnusmanske) Thanks! I have switched over the old and new tables, and it seems to work fine. I'll wait a bit and then drop the old table, which shoul... [09:38:28] (03update) 10dcaro: replica_cnf: add to versions and empty tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1467 (https://phabricator.wikimedia.org/T441108) [09:40:13] (03approved) 10dcaro: replica_cnf: add to versions and empty tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1467 (https://phabricator.wikimedia.org/T441108) [09:40:18] (03update) 10dcaro: replica_cnf: add to versions and empty tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1467 (https://phabricator.wikimedia.org/T441108) [09:40:19] (03merge) 10dcaro: replica_cnf: add to versions and empty tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1467 (https://phabricator.wikimedia.org/T441108) [09:42:02] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component replica-cnf [09:42:19] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component replica-cnf [09:42:33] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component replica-cnf [09:42:48] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component replica-cnf [09:44:25] 10Toolforge, 06tools-platform-team: [functional-tests,deploy] Lately got ~50% errors due to not finding the branch - https://phabricator.wikimedia.org/T440224#12435954 (10dcaro) Got a log: ` @@@@@@@@ Configuring toolforge-deploy for toolsbeta.test + git branch -a + grep -qwE 'ma... [09:45:52] (03CR) 10David Caro: "Tested in toolsbeta:" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1355062 (https://phabricator.wikimedia.org/T441108) (owner: 10David Caro) [09:47:32] (03CR) 10David Caro: toolforge.deploy: add replica-cnf (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1355062 (https://phabricator.wikimedia.org/T441108) (owner: 10David Caro) [09:48:31] (03PS3) 10David Caro: toolforge.deploy: add replica-cnf [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1355062 (https://phabricator.wikimedia.org/T441108) [09:48:41] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component replica-cnf [09:49:03] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component replica-cnf [09:59:50] (03PS4) 10David Caro: toolforge.deploy: add replica-cnf [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1355062 (https://phabricator.wikimedia.org/T441108) [09:59:50] (03CR) 10David Caro: toolforge.deploy: add replica-cnf (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1355062 (https://phabricator.wikimedia.org/T441108) (owner: 10David Caro) [10:00:00] !log dcaro@cloudcumin1001 paws START - Cookbook wmcs.toolforge.component.deploy for component replica-cnf [10:00:00] !log dcaro@cloudcumin1001 paws END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component replica-cnf [10:02:11] (03PS5) 10David Caro: toolforge.deploy: add replica-cnf [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1355062 (https://phabricator.wikimedia.org/T441108) [10:02:15] !log dcaro@cloudcumin1001 paws START - Cookbook wmcs.toolforge.component.deploy for component replica-cnf [10:02:20] !log dcaro@cloudcumin1001 paws END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component replica-cnf [10:04:16] (03PS6) 10David Caro: toolforge.deploy: add replica-cnf [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1355062 (https://phabricator.wikimedia.org/T441108) [10:04:25] !log dcaro@cloudcumin1001 paws START - Cookbook wmcs.toolforge.component.deploy for component replica-cnf [10:04:39] !log dcaro@cloudcumin1001 paws END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component replica-cnf [10:07:28] (03CR) 10CI reject: [V:04-1] toolforge.deploy: add replica-cnf [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1355062 (https://phabricator.wikimedia.org/T441108) (owner: 10David Caro) [10:16:55] 10Toolforge, 10Tools, 06tools-platform-team: Mix'n'match: Temporary DB size increase to reduce DB size - https://phabricator.wikimedia.org/T440708#12436048 (10fnegri) `lang=shell-session root@tools-db-6:~# du -hs /srv/labsdb/data/s51434__mixnmatch_p/ 408G /srv/labsdb/data/s51434__mixnmatch_p/ root@tools-db-6... [10:18:29] (03PS7) 10David Caro: toolforge.deploy: add replica-cnf [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1355062 (https://phabricator.wikimedia.org/T441108) [10:18:43] !log dcaro@cloudcumin1001 paws START - Cookbook wmcs.toolforge.component.deploy for component replica-cnf [10:19:01] !log dcaro@cloudcumin1001 paws END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component replica-cnf [10:21:02] (03CR) 10CI reject: [V:04-1] toolforge.deploy: add replica-cnf [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1355062 (https://phabricator.wikimedia.org/T441108) (owner: 10David Caro) [10:21:12] (03PS8) 10David Caro: toolforge.deploy: add replica-cnf [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1355062 (https://phabricator.wikimedia.org/T441108) [10:21:39] (03CR) 10David Caro: "Tested with paws also, added some extra checks just in case (only allow replica-cnf component when using paws)." [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1355062 (https://phabricator.wikimedia.org/T441108) (owner: 10David Caro) [10:31:33] 10Striker, 06tools-platform-team, 06Infrastructure-Foundations, 06SRE, 07LDAP: Test Striker against the new LDAP server - https://phabricator.wikimedia.org/T441296 (10fnegri) 03NEW [10:43:49] 06tools-infrastructure-team, 13Patch-For-Review: cloudlb haproxy keystone proxy names have diverged between codfw and eqiad - https://phabricator.wikimedia.org/T441286#12436192 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi done! [10:43:55] 10Data-Services, 06tools-platform-team, 06DBA: Decommission clouddb1013-clouddb1020 - https://phabricator.wikimedia.org/T434048#12436195 (10Marostegui) [10:44:05] 10Data-Services, 06tools-platform-team, 06DBA: Decommission clouddb1013-clouddb1020 - https://phabricator.wikimedia.org/T434048#12436197 (10Marostegui) [11:08:06] 06tools-infrastructure-team: Deploy metricsinfra amtool config to prod alertmanager - https://phabricator.wikimedia.org/T441303 (10fgiunchedi) 03NEW [11:18:00] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [11:24:17] FIRING: JobUnavailable: Reduced availability for job openstack in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [11:29:17] RESOLVED: JobUnavailable: Reduced availability for job openstack in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [12:09:52] (03update) 10taavi: eqiad1: cloudinfra: Add security group for WAF (requestctl) nodes [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/365 (https://phabricator.wikimedia.org/T439418) [12:10:35] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/365 [12:10:49] !log taavi@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/365 [12:11:32] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/365 [12:11:36] (03update) 10taavi: eqiad1: cloudinfra: Add security group for WAF (requestctl) nodes [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/365 (https://phabricator.wikimedia.org/T439418) [12:11:47] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/365 [12:12:44] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [12:12:47] (03merge) 10taavi: eqiad1: cloudinfra: Add security group for WAF (requestctl) nodes [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/365 (https://phabricator.wikimedia.org/T439418) [12:13:25] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [12:13:48] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'waf' [12:13:50] !log taavi@cloudcumin1001 cloudinfra END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'waf' [12:14:33] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'waf' [12:22:33] !log taavi@cloudcumin1001 cloudinfra END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'waf' [12:23:49] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.vps.refresh_puppet_certs on waf-1.cloudinfra.eqiad1.wikimedia.cloud [12:26:49] !log taavi@cloudcumin1001 cloudinfra END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on waf-1.cloudinfra.eqiad1.wikimedia.cloud [12:47:17] 10Toolforge, 10Tools, 06tools-platform-team: Mix'n'match: Temporary DB size increase to reduce DB size - https://phabricator.wikimedia.org/T440708#12436649 (10magnusmanske) I have dropped the old table. Please check the difference. This concludes the Phabricator task, it can be closed now (if you agree). [12:56:02] (03PS4) 10Majavah: etcd: Support managing a SRV DNS record for etcd cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341953 [12:56:02] (03PS1) 10Majavah: etcd: Update settings for production cloudinfra cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1355624 (https://phabricator.wikimedia.org/T410721) [12:58:47] (03CR) 10CI reject: [V:04-1] etcd: Support managing a SRV DNS record for etcd cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341953 (owner: 10Majavah) [12:59:47] (03CR) 10CI reject: [V:04-1] etcd: Update settings for production cloudinfra cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1355624 (https://phabricator.wikimedia.org/T410721) (owner: 10Majavah) [13:02:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:05:17] FIRING: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [13:05:19] 10Cloud-VPS, 06tools-infrastructure-team, 07Security: spectre and MDS vulnerabilities on cloud-vps? - https://phabricator.wikimedia.org/T440615#12436712 (10MoritzMuehlenhoff) We definitely applied all these mitigations to the cloudvirt servers and VMs back then, e.g. https://phabricator.wikimedia.org/T228870... [13:07:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:08:44] 10Toolforge, 10Tools, 06tools-platform-team: Mix'n'match: Temporary DB size increase to reduce DB size - https://phabricator.wikimedia.org/T440708#12436727 (10fnegri) 05In progress→03Resolved `lang=shell-session root@tools-db-6:~# du -hs /srv/labsdb/data/s51434__mixnmatch_p/ 176G /srv/labsdb/data/s51... [13:10:17] RESOLVED: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [13:13:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:16:10] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component replica-cnf [13:16:39] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component replica-cnf [13:18:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:20:07] (03PS9) 10David Caro: toolforge.deploy: add replica-cnf [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1355062 (https://phabricator.wikimedia.org/T441108) [13:20:19] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component replica-cnf [13:20:42] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component replica-cnf [13:21:14] (03CR) 10David Caro: [C:03+2] "Deployed now toolsbeta, paws and tools" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1355062 (https://phabricator.wikimedia.org/T441108) (owner: 10David Caro) [13:23:51] (03Merged) 10jenkins-bot: toolforge.deploy: add replica-cnf [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1355062 (https://phabricator.wikimedia.org/T441108) (owner: 10David Caro) [13:24:29] 10Tool-wsindex, 10Wikisource Reader App (Android), 13Patch-For-Review: Add descriptions for books where Wikipedia articles are available [API] - https://phabricator.wikimedia.org/T430400#12436824 (10RCodes) Hi @Bodhisattwa , @Saiphani02, @Muguro While waiting for the review, I tested the MR against some rea... [13:35:31] 10Toolforge, 06tools-platform-team: [replica-cnf] find a way to install/run the functional tests now that they are packaged - https://phabricator.wikimedia.org/T441320 (10dcaro) 03NEW [13:35:46] 10Tool-gerrit-reviewer-bot, 06Project-Admins: Request new Phabricator tag: gerrit-reviewer-bot - https://phabricator.wikimedia.org/T441297#12436880 (10siebrand) 05Open→03Resolved a:03siebrand >>! In T441297#12436136 schreef @Peachey88: > As this is a toolforge tool, iirc you should be able to self ma... [13:36:52] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,heat [13:37:42] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,heat [13:41:43] 10Toolforge, 10Tools, 06tools-platform-team: Mix'n'match: Temporary DB size increase to reduce DB size - https://phabricator.wikimedia.org/T440708#12436908 (10magnusmanske) Cheers. FYI MariaDB is using the "hole punch" mechanism, but it doesn't seem to reflect on the file on disk. If you feel like invest... [13:42:02] (03approved) 10fnegri: helpers: add replica-cnf package support [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/346 (https://phabricator.wikimedia.org/T441108) (owner: 10dcaro) [13:43:37] (03merge) 10dcaro: helpers: add replica-cnf package support [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/346 (https://phabricator.wikimedia.org/T441108) [13:46:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:48:45] 06cloud-services-team, 10Data-Services, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [replica-cnf] Package the new component - https://phabricator.wikimedia.org/T441108#12436937 (10dcaro) [13:48:51] 06cloud-services-team, 10Data-Services, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [replica-cnf] Package the new component - https://phabricator.wikimedia.org/T441108#12436938 (10dcaro) 05Open→03Resolved [13:49:50] 06cloud-services-team, 10Data-Services, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Move replica_cnf_api out of the Puppet repo - https://phabricator.wikimedia.org/T340754#12436950 (10dcaro) [13:50:02] 06cloud-services-team, 10Data-Services, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Move replica_cnf_api out of the Puppet repo - https://phabricator.wikimedia.org/T340754#12436962 (10dcaro) 05In progress→03Resolved [13:55:10] 10Toolforge, 10Tools, 06tools-platform-team: Mix'n'match: Temporary DB size increase to reduce DB size - https://phabricator.wikimedia.org/T440708#12436976 (10fnegri) Looks like for `FTS_*.ibd` files the `apparent` size but `entry.ibd` does not: `lang=shell-session root@tools-db-6:/srv/labsdb/data/s5143... [13:56:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:57:31] 10Tool-gerrit-reviewer-bot, 06Collaboration-Services: gerrit reviewer-bot should trust global SSH known_hosts - https://phabricator.wikimedia.org/T434133#12436981 (10siebrand) [13:58:31] 10Tool-gerrit-reviewer-bot: Reviewer-bot does not add reviewers when leaving initial WIP mode - https://phabricator.wikimedia.org/T267787#12436999 (10siebrand) [13:59:49] 10Tool-gerrit-reviewer-bot, 10Gerrit: Gerrit creates duplicate changes when pushing same commit twice in quick succession - https://phabricator.wikimedia.org/T242590#12437016 (10siebrand) [14:01:35] 10Tool-gerrit-reviewer-bot, 10Gerrit, 07TestMe: Gerrit creates duplicate changes when pushing same commit twice in quick succession - https://phabricator.wikimedia.org/T242590#12437032 (10siebrand) Test if the condition can still be created. Maybe it's fixed in Gerrit. [14:03:34] 10Tool-wsindex, 10Wikisource Reader App (Android): Add name_native_language to People - https://phabricator.wikimedia.org/T430705#12437057 (10RCodes) Hi @Bodhisattwa , @Saiphani02, @Muguro While testing WSIndex I noticed something that's related to this task. I checked the existing Tool-wsindex tasks and coul... [14:27:12] 10Tools, 06Commons, 07Community-Wishlist: A set of tools for image editing directly in Wikimedia Commons - https://phabricator.wikimedia.org/T428062#12437214 (10mikez-WMF) > Apply filter, crop, and rotate would all go to CropTool2 as it provides all three functions. However, there's no deep linking mechanism... [14:47:26] 10Striker, 06tools-platform-team, 07OKR-Work: Draft: Measure adoption of reliability views - https://phabricator.wikimedia.org/T441328 (10TLepage-WMF) 03NEW [14:47:53] 10Striker, 06tools-platform-team, 07OKR-Work: Draft: Measure adoption of reliability views - https://phabricator.wikimedia.org/T441328#12437364 (10TLepage-WMF) [14:47:54] 10Toolforge, 06tools-platform-team, 07Epic, 07OKR-Work: ST5.4.3 Implement Toolforge UI MVP - https://phabricator.wikimedia.org/T440589#12437365 (10TLepage-WMF) [14:56:40] 10Striker, 06tools-platform-team, 07OKR-Work: Draft: Measure adoption of reliability views - https://phabricator.wikimedia.org/T441328#12437429 (10TLepage-WMF) [15:02:02] 10Tool-gerrit-reviewer-bot, 06Collaboration-Services: gerrit reviewer-bot should trust global SSH known_hosts - https://phabricator.wikimedia.org/T434133#12437451 (10siebrand) 05Open→03Resolved a:03siebrand This was fixed in https://github.com/valhallasw/gerrit-reviewer-bot/commit/f2a5173dde2d7427ea2... [15:04:31] 10Toolforge, 10Tools, 06tools-platform-team: Mix'n'match: Temporary DB size increase to reduce DB size - https://phabricator.wikimedia.org/T440708#12437473 (10fnegri) I did some more digging but I couldn't find an explanation for the "punch hole" not working on `entry.ibd`. I think for the moment I'm ha... [15:13:24] 10Tool-delintbot: Mention what was fixed in the edit summary - https://phabricator.wikimedia.org/T416068#12437512 (10Redmin) 05Open→03In progress p:05Low→03Medium [15:18:12] 10Tool-gerrit-reviewer-bot: Reviewer-bot does not add reviewers when leaving initial WIP mode - https://phabricator.wikimedia.org/T267787#12437559 (10siebrand) >>! In T267787#6658893 schreef @hashar: > Had the bot listened for Gerrit event stream: I'd like to explore this, @hashar. Is there already a place wher... [15:32:17] (03open) 10nichesports: Main: use 'last' level 3 section before the Cleanup section [toolforge-repos/nichesportsbot] - 10https://gitlab.wikimedia.org/toolforge-repos/nichesportsbot/-/merge_requests/2 [15:42:09] 10Tool-gerrit-reviewer-bot: Reviewer-bot not adding reviewers due to SSH key change - https://phabricator.wikimedia.org/T434075#12437689 (10Aklapper) [15:43:04] 10Tool-gerrit-reviewer-bot: Reviewer-bot option to be added as CC instead of reviewer - https://phabricator.wikimedia.org/T334118#12437690 (10Aklapper) [15:43:14] 10Tool-gerrit-reviewer-bot: Gerrit reviewer bot should have an option to add people as CC instead of reviewers - https://phabricator.wikimedia.org/T363290#12437691 (10Aklapper) [15:43:30] 10Tool-gerrit-reviewer-bot, 07Developer Productivity: gerrit-reviewer-bot got stuck due to non-existing account - https://phabricator.wikimedia.org/T342066#12437692 (10Aklapper) [15:43:58] 10Tool-gerrit-reviewer-bot, 10VPS-Projects: gerrit-reviewer-bot stopped working - https://phabricator.wikimedia.org/T141390#12437694 (10Aklapper) [15:44:02] 10Cloud-VPS, 10Tool-gerrit-reviewer-bot, 06tools-infrastructure-team: Gerrit reviewer bot dosent seem to be working - https://phabricator.wikimedia.org/T153719#12437695 (10Aklapper) [15:44:07] 10Tool-gerrit-reviewer-bot: Gerrit Reviewer Bot should support gerrit's new inline edit screen - https://phabricator.wikimedia.org/T142338#12437699 (10Aklapper) [15:44:25] 10Tool-gerrit-reviewer-bot: Gerrit Reviewer Bot is unreliable - https://phabricator.wikimedia.org/T290905#12437700 (10Aklapper) [15:44:28] 10Tool-gerrit-reviewer-bot, 10Toolforge, 06tools-platform-team: SGE Job gerrit_reviewer_bot (9999704) stuck in delete state - https://phabricator.wikimedia.org/T278645#12437702 (10Aklapper) [15:44:36] 10Tool-gerrit-reviewer-bot: reviewer-bot sometimes not working - https://phabricator.wikimedia.org/T222983#12437705 (10Aklapper) [15:44:42] 10Tool-gerrit-reviewer-bot: Reviewer-bot does not add reviewers when leaving initial WIP mode - https://phabricator.wikimedia.org/T267787#12437706 (10siebrand) a:03siebrand [15:45:20] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2010-dev.codfw.wmnet' [15:45:29] (03open) 10rluts: Adopt MediaWiki message format for translatewiki.net [toolforge-repos/findsmith] - 10https://gitlab.wikimedia.org/toolforge-repos/findsmith/-/merge_requests/5 [15:45:54] (03merge) 10rluts: Adopt MediaWiki message format for translatewiki.net [toolforge-repos/findsmith] - 10https://gitlab.wikimedia.org/toolforge-repos/findsmith/-/merge_requests/5 [15:46:03] (03merge) 10nichesports: Main: use 'last' level 3 section before the Cleanup section [toolforge-repos/nichesportsbot] - 10https://gitlab.wikimedia.org/toolforge-repos/nichesportsbot/-/merge_requests/2 [15:49:37] 10Tool-gerrit-reviewer-bot, 06Collaboration-Services: gerrit reviewer-bot should trust global SSH known_hosts - https://phabricator.wikimedia.org/T434133#12437747 (10Dzahn) Thank you !:) [15:57:06] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2010-dev.codfw.wmnet' [15:57:49] 10Toolforge, 10Tools, 06tools-platform-team: Mix'n'match: Temporary DB size increase to reduce DB size - https://phabricator.wikimedia.org/T440708#12437794 (10fnegri) Possibly related upstream issue: https://jira.mariadb.org/browse/MDEV-38425 [15:57:55] (03open) 10dcaro: rotationg_file_flag: allow creating a file to flag for cred rotation [repos/cloud/toolforge/replica-cnf] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/replica-cnf/-/merge_requests/3 (https://phabricator.wikimedia.org/T440952) [15:58:54] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2005-dev.codfw.wmnet' [15:59:59] 10Cloud-VPS (Debian Bullseye Deprecation), 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 10Event-Platform: Upgrade (replace) beta|deployment-prep event platform VPS instances - https://phabricator.wikimedia.org/T429497#12437812 (10Ahoelzl) [16:00:24] 10Toolforge, 06tools-platform-team: [alerts-api] Tools that opt out still get a resolved email - https://phabricator.wikimedia.org/T441338 (10fnegri) 03NEW [16:03:25] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [maintain-harbor] delete untagged artifacts too - https://phabricator.wikimedia.org/T440343#12437985 (10Raymond_Ndibe) 05Open→03In progress [16:10:22] 10Tool-gerrit-reviewer-bot, 10Toolforge: SGE Job gerrit_reviewer_bot (9999704) stuck in delete state - https://phabricator.wikimedia.org/T278645#12438044 (10aputhin) [16:11:48] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2005-dev.codfw.wmnet' [16:13:17] FIRING: JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:13:34] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2006-dev.codfw.wmnet' [16:16:38] 10Striker, 06tools-platform-team, 07OKR-Work: Draft: Measure adoption of reliability views - https://phabricator.wikimedia.org/T441328#12438100 (10Aruotes) a:03Aruotes Hey, I would like to work on this task. Thank you!! [16:17:47] (03PS1) 10Aruotes: tools: Measure adoption of reliability views with pseudonymised weekly visits [labs/striker] - 10https://gerrit.wikimedia.org/r/1355729 (https://phabricator.wikimedia.org/T441328) [16:18:17] RESOLVED: JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:20:18] (03update) 10dcaro: rotationg_file_flag: allow creating a file to flag for cred rotation [repos/cloud/toolforge/replica-cnf] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/replica-cnf/-/merge_requests/3 (https://phabricator.wikimedia.org/T440952) [16:24:38] (03update) 10dcaro: rotationg_file_flag: allow creating a file to flag for cred rotation [repos/cloud/toolforge/replica-cnf] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/replica-cnf/-/merge_requests/3 (https://phabricator.wikimedia.org/T440952) [16:24:57] (03open) 10r4356th: Run delinter on page fragments with lint errors only; make lint category an optional argument [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/40 (https://phabricator.wikimedia.org/T416792 https://phabricator.wikimedia.org/T440911) [16:25:46] (03update) 10dcaro: rotationg_file_flag: allow creating a file to flag for cred rotation [repos/cloud/toolforge/replica-cnf] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/replica-cnf/-/merge_requests/3 (https://phabricator.wikimedia.org/T440952) [16:25:53] (03update) 10r4356th: Run delinter on page fragments with lint errors only; make lint category an optional argument [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/40 (https://phabricator.wikimedia.org/T416792 https://phabricator.wikimedia.org/T440911) [16:26:22] 10Tool-delintbot, 13Patch-For-Review: Make lint category an optional argument - https://phabricator.wikimedia.org/T416792#12438201 (10Redmin) 05Open→03In progress [16:26:22] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2006-dev.codfw.wmnet' [16:26:35] 10Tool-gerrit-reviewer-bot: Reviewer-bot does not add reviewers when leaving initial WIP mode - https://phabricator.wikimedia.org/T267787#12438205 (10siebrand) I think this is fixed. I changed from email to REST and implemented action on un-WIP. Considering this test phase for now. [16:27:59] (03update) 10r4356th: Run delinter on page fragments with lint errors only; make lint category an optional argument [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/40 (https://phabricator.wikimedia.org/T416792 https://phabricator.wikimedia.org/T440911) [16:29:36] 10Tool-gerrit-reviewer-bot: Reviewer-bot does not add reviewers when leaving initial WIP mode - https://phabricator.wikimedia.org/T267787#12438246 (10siebrand) 05Open→03Resolved [16:36:14] 10Tool-wikinewsie: Wikinewsie elements fail to translate upon reload from language selector - https://phabricator.wikimedia.org/T439940#12438282 (10Chimaifeanyi29) a:03Chimaifeanyi29 [16:37:58] (03CR) 10Dzahn: [C:03+2] "being bold because:" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1354897 (https://phabricator.wikimedia.org/T268199) (owner: 10Dzahn) [16:39:22] (03Merged) 10jenkins-bot: frontend: make the public Hound URL configurable [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1354897 (https://phabricator.wikimedia.org/T268199) (owner: 10Dzahn) [16:40:30] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1011.eqiad.wmnet' (T441318) [16:41:48] 10Toolforge, 10Tools, 06tools-platform-team: Mix'n'match: Temporary DB size increase to reduce DB size - https://phabricator.wikimedia.org/T440708#12438314 (10magnusmanske) Looks like that might be it. Maybe the next MariaDB update will fix it. [16:47:24] FIRING: SystemdUnitFailed: nova-fullstack.service on cloudcontrol1007:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:50:22] FIRING: [15x] HAProxyBackendUnavailable: HAProxy service cinder-api_backend backend cloudcontrol1011.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [16:50:37] PROBLEM - Host cloudcontrol1011 is DOWN: PING CRITICAL - Packet loss = 100% [16:51:10] FIRING: [2x] GaleraClusterSizeMismatch: Galera in eqiad1 has 2 nodes - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraClusterSizeMismatch - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraClusterSizeMismatch [16:51:58] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1011.eqiad.wmnet' (T441318) [16:52:05] RECOVERY - Host cloudcontrol1011 is UP: PING OK - Packet loss = 0%, RTA = 1.81 ms [16:53:04] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1007.eqiad.wmnet' (T441318) [16:58:03] 10Tool-gerrit-reviewer-bot, 10Gerrit, 13Patch-For-Review, 07TestMe: Gerrit creates duplicate changes when pushing same commit twice in quick succession - https://phabricator.wikimedia.org/T242590#12438420 (10siebrand) I wasn't able to reproduce creating a two changesets with the same change ID. Proposing t... [16:59:16] (03update) 10dcaro: rotationg_file_flag: allow creating a file to flag for cred rotation [repos/cloud/toolforge/replica-cnf] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/replica-cnf/-/merge_requests/3 (https://phabricator.wikimedia.org/T440952) [17:03:58] 10Toolforge, 06tools-platform-team: [alerts-api] Count the restarts for each pod in ToolFrequentRestarts - https://phabricator.wikimedia.org/T441348 (10fnegri) 03NEW [17:04:25] (03CR) 10Dzahn: [C:03+2] "being bold. because:" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1354897 (https://phabricator.wikimedia.org/T268199) (owner: 10Dzahn) [17:04:53] (03CR) 10Dzahn: "being bold. because:" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1354853 (https://phabricator.wikimedia.org/T268199) (owner: 10Dzahn) [17:05:13] (03CR) 10Dzahn: [C:03+2] frontend: build and publish a frontend image with blubber [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1354853 (https://phabricator.wikimedia.org/T268199) (owner: 10Dzahn) [17:06:27] (03Merged) 10jenkins-bot: frontend: build and publish a frontend image with blubber [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1354853 (https://phabricator.wikimedia.org/T268199) (owner: 10Dzahn) [17:07:17] FIRING: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [17:07:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:07:40] RESOLVED: GaleraClusterSizeMismatch: Galera in eqiad1 has 2 nodes - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraClusterSizeMismatch - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraClusterSizeMismatch [17:08:29] PROBLEM - Host cloudcontrol1007 is DOWN: PING CRITICAL - Packet loss = 100% [17:09:07] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1007.eqiad.wmnet' (T441318) [17:09:32] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1006.eqiad.wmnet' (T441318) [17:09:44] (03update) 10dcaro: rotationg_file_flag: allow creating a file to flag for cred rotation [repos/cloud/toolforge/replica-cnf] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/replica-cnf/-/merge_requests/3 (https://phabricator.wikimedia.org/T440952) [17:09:54] RESOLVED: SystemdUnitFailed: nova-fullstack.service on cloudcontrol1007:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:10:05] RECOVERY - Host cloudcontrol1007 is UP: PING OK - Packet loss = 0%, RTA = 0.28 ms [17:10:52] RESOLVED: [15x] HAProxyBackendUnavailable: HAProxy service cinder-api_backend backend cloudcontrol1011.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [17:11:20] 10Toolforge, 06tools-platform-team: Release Alerting System and remove Beta phase labels - https://phabricator.wikimedia.org/T440602#12438483 (10fnegri) [17:11:27] 10Toolforge, 06tools-platform-team, 07Epic, 07OKR-Work: ST5.4.2 Toolforge Alerting System - https://phabricator.wikimedia.org/T432860#12438484 (10fnegri) [17:11:57] 10Toolforge, 06tools-platform-team: Release Alerting System and remove Beta phase labels - https://phabricator.wikimedia.org/T440602#12438485 (10fnegri) [17:11:59] 10Toolforge, 06tools-platform-team, 07Epic, 07OKR-Work: ST5.4.2 Toolforge Alerting System - https://phabricator.wikimedia.org/T432860#12438486 (10fnegri) [17:12:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:14:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:14:46] 10Toolforge, 06tools-platform-team: Release Alerting System and remove Beta phase labels - https://phabricator.wikimedia.org/T440602#12438491 (10fnegri) [17:15:16] 10Toolforge, 06tools-platform-team: Release Alerting System and remove Beta phase labels - https://phabricator.wikimedia.org/T440602#12438492 (10fnegri) [17:15:17] 10Toolforge, 06tools-platform-team: [alerts-api] Reduce flapping for ToolFrequentRestarts - https://phabricator.wikimedia.org/T440570#12438493 (10fnegri) [17:19:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:22:23] PROBLEM - Host cloudcontrol1006 is DOWN: PING CRITICAL - Packet loss = 100% [17:22:38] 06cloud-services-team (Hardware), 06tools-infrastructure-team, 06Data-Platform-SRE, 06DC-Ops, 06SRE: How to configure storage on new dumps servers - https://phabricator.wikimedia.org/T441202#12438508 (10BLiviero-WMF) [17:22:40] FIRING: GaleraClusterSizeMismatch: Galera in eqiad1 has 2 nodes - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraClusterSizeMismatch - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraClusterSizeMismatch [17:22:45] 06cloud-services-team (Hardware), 06tools-infrastructure-team, 06Data-Platform-SRE, 06DC-Ops, 06SRE: How to configure storage on new dumps servers - https://phabricator.wikimedia.org/T441202#12438510 (10BLiviero-WMF) Adding Data Platform team to weigh in - as this may need to be their call to make- [17:22:52] FIRING: [15x] HAProxyBackendUnavailable: HAProxy service cinder-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [17:23:36] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1006.eqiad.wmnet' (T441318) [17:24:05] RECOVERY - Host cloudcontrol1006 is UP: PING OK - Packet loss = 0%, RTA = 1.39 ms [17:25:55] RESOLVED: [2x] GaleraClusterSizeMismatch: Galera in eqiad1 has 2 nodes - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraClusterSizeMismatch - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraClusterSizeMismatch [17:26:07] FIRING: [30x] HAProxyBackendUnavailable: HAProxy service cinder-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [17:27:52] RESOLVED: [30x] HAProxyBackendUnavailable: HAProxy service cinder-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [17:29:47] RESOLVED: [2x] JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [17:38:00] 10Tool-delintbot: Store data about blocked edit attempts - https://phabricator.wikimedia.org/T440299#12438632 (10Redmin) `ALTER TABLE blocked_attempts ADD lintcat VARCHAR(2) NOT NULL AFTER fixed_num_errors; Query OK, 0 rows affected (0.084 sec) Records: 0 Duplicates: 0 Warnings: 0 ` ` DESCRIBE blocked_attempts... [18:24:17] (03update) 10r4356th: Run delinter on page fragments with lint errors only; make lint category an optional argument [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/40 (https://phabricator.wikimedia.org/T416792 https://phabricator.wikimedia.org/T440911) [18:26:31] (03open) 10chimaifeanyi29: T439940: Fix untranslated labels after switching language [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/30 [18:27:53] (03update) 10chimaifeanyi29: T439940: Fix untranslated labels after switching language [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/30 [18:28:09] (03update) 10r4356th: Run delinter on page fragments with lint errors only; make lint category an optional argument [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/40 (https://phabricator.wikimedia.org/T416792 https://phabricator.wikimedia.org/T440911) [18:29:13] (03CR) 10Andrea Denisse: [V:03+2 C:03+2] alert: add a dummy Klaxon managers incident URL [labs/private] - 10https://gerrit.wikimedia.org/r/1345218 (owner: 10Andrea Denisse) [18:29:53] (03update) 10chimaifeanyi29: T439940: Fix untranslated labels after switching language [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/30 [18:32:27] (03update) 10chimaifeanyi29: T439940: Fix untranslated labels after switching language [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/30 [18:43:06] (03update) 10r4356th: Run delinter on page fragments with lint errors only; make lint category an optional argument [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/40 (https://phabricator.wikimedia.org/T416792 https://phabricator.wikimedia.org/T440911) [18:46:19] (03update) 10r4356th: Run delinter on page fragments with lint errors only; make lint category an optional argument [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/40 (https://phabricator.wikimedia.org/T416792 https://phabricator.wikimedia.org/T440911) [18:47:36] (03update) 10r4356th: Run delinter on page fragments with lint errors only; make lint category an optional argument [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/40 (https://phabricator.wikimedia.org/T416792 https://phabricator.wikimedia.org/T440911) [18:47:48] (03merge) 10r4356th: Run delinter on page fragments with lint errors only; make lint category an optional argument [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/40 (https://phabricator.wikimedia.org/T416792 https://phabricator.wikimedia.org/T440911) [18:48:05] 10Tool-delintbot, 13Patch-For-Review: Make lint category an optional argument - https://phabricator.wikimedia.org/T416792#12438910 (10Redmin) 05In progress→03Resolved a:03Redmin [18:48:47] 10Tool-delintbot, 13Patch-For-Review: Explore working with specific parts of page Wikitext - https://phabricator.wikimedia.org/T440911#12438915 (10Redmin) 05In progress→03Resolved a:03Redmin [19:03:32] 10Tool-delintbot: Mention what was fixed in the edit summary - https://phabricator.wikimedia.org/T416068#12438990 (10Redmin) 05In progress→03Open [19:37:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:42:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:46:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:51:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:55:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:00:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:19:26] (03open) 10project_5175_bot_1dc5cd2b4a7cadb6d1af8e29974ce1a7: Yeni anıt kategorileri için üst kategori zinciri [toolforge-repos/kategori] - 10https://gitlab.wikimedia.org/toolforge-repos/kategori/-/merge_requests/3 [20:22:21] 10Tool-wikinewsie, 07good first task: Specify Wikinewsie interface language in url - https://phabricator.wikimedia.org/T441366 (10Pharos) 03NEW Thank you for tagging this task with #good_first_task for Wikimedia newcomers! Newcomers often may not be aware of things that may seem obvious to seasoned contribu... [20:23:48] (03merge) 10sadrettin: Yeni anıt kategorileri için üst kategori zinciri [toolforge-repos/kategori] - 10https://gitlab.wikimedia.org/toolforge-repos/kategori/-/merge_requests/3 (owner: 10project_5175_bot_1dc5cd2b4a7cadb6d1af8e29974ce1a7) [20:28:56] (03open) 10project_5175_bot_1dc5cd2b4a7cadb6d1af8e29974ce1a7: Yeni ara kategorileri ilçe kategorisinin adıyla aç [toolforge-repos/kategori] - 10https://gitlab.wikimedia.org/toolforge-repos/kategori/-/merge_requests/4 [20:31:17] (03merge) 10sadrettin: Yeni ara kategorileri ilçe kategorisinin adıyla aç [toolforge-repos/kategori] - 10https://gitlab.wikimedia.org/toolforge-repos/kategori/-/merge_requests/4 (owner: 10project_5175_bot_1dc5cd2b4a7cadb6d1af8e29974ce1a7) [20:41:27] 10Tool-wikinewsie: 'See also' mode for Wikinewsie - https://phabricator.wikimedia.org/T434468#12439374 (10Pharos) I guess the inherited "See also" items will be different on English Wikipedia vs. French Wikipedia vs. Chinese Wikipedia, so you may have to specify which languages you would like to do an import fro... [21:34:03] FIRING: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [21:39:03] RESOLVED: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [21:50:25] 10Tool-wikinewsie: Multimedia focus original reporting on Wikinewsie - https://phabricator.wikimedia.org/T435264#12439746 (10E_mln_e) Following on our discussion on Signal, my suggestion was that commons category are oftentimes related to the news - e.g., festivals, cultural events, protests, etc. We could, perh... [22:51:13] 10Cloud-VPS, 06tools-infrastructure-team, 06Infrastructure-Foundations, 10Puppet CI, 13Patch-For-Review: PCC: no facts for Cloud VPS hosts on the cloudinfra puppetserver (codesearch10) - https://phabricator.wikimedia.org/T441067#12439931 (10Dzahn) Is T433980 also this? [22:54:08] 10Toolforge, 06tools-platform-team, 13Patch-For-Review, 07Sustainability (Incident Followup): [toolforge.infra] run and monitor our own sample tools - https://phabricator.wikimedia.org/T357977#12439939 (10BLiviero-WMF) I am removing the sustainability (Incident followup) tag from this task as the most impo... [22:54:30] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [toolforge.infra] run and monitor our own sample tools - https://phabricator.wikimedia.org/T357977#12439940 (10BLiviero-WMF) [22:57:23] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 07Sustainability (Incident Followup): [toolsdb] pt-heartbeat service should automatically follow the primary - https://phabricator.wikimedia.org/T409890#12439968 (10BLiviero-WMF) I am removing the sustainability (Incident followup) tag from this ta... [22:57:53] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [toolsdb] pt-heartbeat service should automatically follow the primary - https://phabricator.wikimedia.org/T409890#12439973 (10BLiviero-WMF) [23:21:42] 10Tool-Pageviews: Add clickable pie slice in the Traffic graphic - https://phabricator.wikimedia.org/T440726#12440030 (10MusikAnimal) 05Open→03Resolved a:03MusikAnimal