[00:06:20] FIRING: PrometheusK8sCertExpirySoon: Prometheus k8s certificate is about to expire - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/PrometheusK8sCertExpirySoon - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusK8sCertExpirySoon [00:24:23] (03approved) 10pepepiton: Closes T393153 - no reasults page error fixed [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/5 (owner: 10piracalamina) [00:25:09] (03merge) 10pepepiton: Closes T393153 - no reasults page error fixed [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/5 (owner: 10piracalamina) [00:38:41] 10Cloud-VPS (Project-requests): Request a free server - https://phabricator.wikimedia.org/T393294 (10Zxydk) 03NEW [02:00:29] 10Cloud-VPS (Project-requests): Request a free server - https://phabricator.wikimedia.org/T393294#10789291 (10JJMC89) 05Open→03Invalid [04:16:02] 10Cloud-VPS (Project-requests): Request creation of osmviews VPS project with 20 GiB object storage quota - https://phabricator.wikimedia.org/T393013#10789299 (10dcaro) p:05Triage→03Medium +1 approved [04:24:10] 10Cloud-VPS (Project-requests): Request creation of MissSearch VPS project - https://phabricator.wikimedia.org/T392903#10789302 (10dcaro) Yep, currently there's no easy on demand storage on toolforge side besides NFS (it's on our minds, so keep an eye!). Fyi. there's an elasticsearch instance for toolforge user... [04:24:31] 10Cloud-VPS (Project-requests): Request creation of MissSearch VPS project - https://phabricator.wikimedia.org/T392903#10789304 (10dcaro) p:05Triage→03Medium [04:38:57] 10Cloud-VPS (Quota-requests), 06Moderator-Tools-Team, 10Wikilink-Tool: Request to increase object storage capacity - Wikilink project - https://phabricator.wikimedia.org/T392870#10789307 (10dcaro) Thanks for the numbers, they are very useful for us to be able to plan and manage the service, LGTM +1 [07:09:07] (03update) 10aborrero: gitlab-ci: introduce tofu-provisioning code [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/51 [07:09:15] (03update) 10aborrero: gitlab-ci: replace local logic with included one [repos/cloud/cloud-vps/networktests-tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/23 [07:11:23] (03update) 10aborrero: gitlab-ci: introduce tofu-provisioning code [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/51 [07:14:20] (03update) 10aborrero: gitlab-ci: introduce tofu-provisioning code [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/51 [07:23:09] (03update) 10aborrero: gitlab-ci: introduce tofu-provisioning code [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/51 [07:23:58] (03update) 10aborrero: gitlab-ci: introduce tofu-provisioning code [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/51 [07:57:10] (03merge) 10gopavasanth: Fix placeholder message for username field [toolforge-repos/yearinreview] - 10https://gitlab.wikimedia.org/toolforge-repos/yearinreview/-/merge_requests/9 (owner: 10waldir) [09:38:42] 10Tool-wdrecentchanges: Feature Request: Make markers less tiny - https://phabricator.wikimedia.org/T393312 (10Gnoeee) 03NEW [09:39:48] 10Tool-wdrecentchanges, 03Wikimedia-Hackathon-2025: Feature Request: Make markers less tiny - https://phabricator.wikimedia.org/T393312#10789748 (10Gnoeee) [09:46:20] 10Tool-paulina, 03Wikimedia-Hackathon-2025: Improve the Paulina tool through different tasks - https://phabricator.wikimedia.org/T393096#10789767 (10SGautam_WMF) |Author page| |{F59664010 size=full}| |Work List page| |{F59664077 size=full}| |Work page| |{F59664090 size=full}| |Countries list page| |{F59664... [10:08:11] 10Tool-wdrecentchanges, 03Wikimedia-Hackathon-2025: Add the edit damaging score - https://phabricator.wikimedia.org/T393317 (10Gnoeee) 03NEW [10:10:59] 10Tool-wdrecentchanges, 03Wikimedia-Hackathon-2025: Add the edit damaging score - https://phabricator.wikimedia.org/T393317#10789864 (10Gnoeee) 05Open→03In progress [10:14:45] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [toolforge] Investigate authentication - https://phabricator.wikimedia.org/T363983#10789865 (10dcaro) Quick update, I've been looking into enabling CAS or OIDC (openid connect) into nginx, and it seems it's not supported unless you have the paid "plus"... [10:15:47] 10Tool-wdrecentchanges, 03Wikimedia-Hackathon-2025: Add the edit damaging score - https://phabricator.wikimedia.org/T393317#10789866 (10Gnoeee) [10:47:40] 10Tool-techcontribs: Show admin groups in tech-contribs - https://phabricator.wikimedia.org/T393322 (10taavi) 03NEW [11:06:52] 10Toolforge-standards-committee (Maintainer needed), 10Tools, 10Wikidata, 03Wikimedia-Hackathon-2025: The EditGroups tool needs new maintainer(s) - https://phabricator.wikimedia.org/T376072#10789945 (10Pintoch) {F59665586} [11:18:28] FIRING: PuppetStaleCertificates: Found non-revoked Puppet certificates for 1 deleted instances on tools-puppetserver-01 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [11:27:01] 10Tool-wdactle, 10Wikidata, 03Wikimedia-Hackathon-2025: Create Wikidata Redactle game - https://phabricator.wikimedia.org/T393291#10789986 (10LucasWerkmeister) The game is live here: **** – it still needs more polish, but it is playable. By default, it presents a random item... [11:35:07] 10Tool-techcontribs: Show admin groups in tech-contribs - https://phabricator.wikimedia.org/T393322#10790013 (10Chlod) p:05Triage→03Medium [11:38:53] 10Tool-techcontribs: Emojis used in Phabricator project description causes extra space along with the emoji not being displayed - https://phabricator.wikimedia.org/T393324 (10Dreamy_Jazz) 03NEW [11:44:33] 10Tool-techcontribs: Make sections explainers - https://phabricator.wikimedia.org/T393326 (10Rexogamer) 03NEW [11:46:48] 10Tool-techcontribs: Make section explainers collapsible - https://phabricator.wikimedia.org/T393326#10790054 (10Rexogamer) [11:50:41] (03open) 10bd808: README: add usage [toolforge-repos/gitlab-content] - 10https://gitlab.wikimedia.org/toolforge-repos/gitlab-content/-/merge_requests/3 [11:50:46] (03merge) 10bd808: README: add usage [toolforge-repos/gitlab-content] - 10https://gitlab.wikimedia.org/toolforge-repos/gitlab-content/-/merge_requests/3 [11:51:20] 10Tool-techcontribs: Too little spacing between columns in Phabricator projects list - https://phabricator.wikimedia.org/T393328 (10Dreamy_Jazz) 03NEW [12:03:42] 10Tool-gitlab-content, 07Documentation: Add maintainer docs and user usage instructions to wikitech:Tool:Gitlab-content - https://phabricator.wikimedia.org/T393330 (10bd808) 03NEW [12:04:03] 10Tool-gitlab-content, 07Documentation: Add maintainer docs and user usage instructions to wikitech:Tool:Gitlab-content - https://phabricator.wikimedia.org/T393330#10790131 (10bd808) a:03bd808 [12:05:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:46:30] 10Tool-inteGraality, 03Wikimedia-Hackathon-2025: Support for 'mul' in integraality - https://phabricator.wikimedia.org/T393166#10790189 (10JeanFred) [12:50:29] (03open) 10samwilson: Check that uploads are enabled before uploading [toolforge-repos/mwcli] - 10https://gitlab.wikimedia.org/toolforge-repos/mwcli/-/merge_requests/4 [13:07:34] 10Tool-inteGraality, 03Wikimedia-Hackathon-2025: pywikibot.exceptions.ServerError: HTTPSConnectionPool(host='query.wikidata.org', port=443): Read timed out. (read timeout=45) - https://phabricator.wikimedia.org/T384882#10790209 (10JeanFred) [15:59:19] (03update) 10eliza189: Eliza data update [toolforge-repos/miss-search] (linkhere_branch) - 10https://gitlab.wikimedia.org/toolforge-repos/miss-search/-/merge_requests/3 [16:05:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:05:11] (03open) 10ilanen1: Ilanapi [toolforge-repos/miss-search] (linkhere_branch) - 10https://gitlab.wikimedia.org/toolforge-repos/miss-search/-/merge_requests/4 [20:03:56] (03update) 10ilanen1: Ilanapi [toolforge-repos/miss-search] (linkhere_branch) - 10https://gitlab.wikimedia.org/toolforge-repos/miss-search/-/merge_requests/4 [20:05:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:30:18] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.undrain_node [21:30:19] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) [21:30:52] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.undrain_node [21:31:05] !log andrew@cloudcumin1001 admin END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97) [21:31:09] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.undrain_node [21:31:10] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) [21:31:29] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.undrain_node [21:31:39] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) [22:36:40] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.drain_node [22:37:28] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) [22:38:32] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.drain_node [22:39:20] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) [22:40:45] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.drain_node [22:41:26] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) [22:41:51] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.drain_node [22:42:33] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) [22:43:07] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.drain_node [22:43:49] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) [22:53:04] 06cloud-services-team, 10decommission-hardware: decommission cloudcephosd100[1-3] - https://phabricator.wikimedia.org/T393196#10790598 (10Andrew) [23:02:30] 06cloud-services-team, 10decommission-hardware: decommission cloudcephosd100[1-3] - https://phabricator.wikimedia.org/T393196#10790599 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by andrew@cumin1002 for hosts: `cloudcephosd1001.eqiad.wmnet` - cloudcephosd1001.eqiad.wmnet (**PASS**) - Dow... [23:15:30] 06cloud-services-team, 10decommission-hardware: decommission cloudcephosd100[1-3] - https://phabricator.wikimedia.org/T393196#10790600 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by andrew@cumin1002 for hosts: `cloudcephosd1002.eqiad.wmnet` - cloudcephosd1002.eqiad.wmnet (**PASS**) - Dow... [23:27:36] 06cloud-services-team, 10decommission-hardware, 13Patch-For-Review: decommission cloudcephosd100[1-3] - https://phabricator.wikimedia.org/T393196#10790605 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by andrew@cumin1002 for hosts: `cloudcephosd1003.eqiad.wmnet` - cloudcephosd1003.eqiad.wm... [23:29:17] 06cloud-services-team, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, and 2 others: decommission cloudcephosd100[1-3] - https://phabricator.wikimedia.org/T393196#10790607 (10Andrew) a:05Andrew→03None [23:53:18] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.undrain_node [23:53:28] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)