[00:04:22] FIRING: [2x] HAProxyBackendUnavailable: HAProxy service nova-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [00:07:56] RESOLVED: SystemdUnitDown: The service unit nova-api.service is in failed status on host cloudcontrol1011. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1011 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [00:09:22] RESOLVED: [2x] HAProxyBackendUnavailable: HAProxy service nova-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [00:33:47] 06cloud-services-team, 10Cloud-VPS: Cloud init and unattended upgrades while bootstrapping Trixie VMs - https://phabricator.wikimedia.org/T422509#11822458 (10Andrew) 05Open→03Resolved a:03Andrew [00:48:45] 10Toolforge, 06tools-platform-team: [Toolforge]: User Stats Baseline (Outreach Channels, Accounts, Activity) - https://phabricator.wikimedia.org/T422783#11822484 (10bd808) >>! In T422783#11820755, @bd808 wrote: > Linked "Channels" google doc seems to be private access only. This has been fixed. [00:52:14] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Tool-etherpad-backup: Increase object storage object count for etherpads3 project - https://phabricator.wikimedia.org/T423354#11822485 (10bd808) [01:33:55] 10Toolforge, 06tools-platform-team: [Toolforge]: User Stats Baseline (Outreach Channels, Accounts, Activity) - https://phabricator.wikimedia.org/T422783#11822539 (10komla) >>! In T422783#11822484, @bd808 wrote: >>>! In T422783#11820755, @bd808 wrote: >> Linked "Channels" google doc seems to be private access o... [02:37:42] 06cloud-services-team, 10Cloud-VPS: Consider enabling static website support in the WMCS radosgw implementation of S3 buckets - https://phabricator.wikimedia.org/T423194#11822596 (10Andrew) p:05Triage→03Low [02:39:09] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Puppet fail to create volume group for ephemeral disk space when it is sda (instead of sdb) - https://phabricator.wikimedia.org/T422258#11822597 (10Andrew) p:05Triage→03Medium a:03Andrew [03:17:50] 10Tool-community-activity-alerts-system: Extending the dashboard to report abonormalities in editor counts - https://phabricator.wikimedia.org/T398999#11822612 (10ecoholic) 05In progress→03Resolved [05:11:53] (03open) 10r4356th: Make sure all attributes on font tags are preserved [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/17 [05:18:16] (03update) 10r4356th: Make sure all attributes on font tags are preserved [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/17 [05:24:41] (03update) 10r4356th: Make sure all attributes on font tags are preserved [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/17 [05:40:59] (03update) 10r4356th: Make sure all attributes on font tags are preserved [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/17 [05:46:45] (03update) 10r4356th: Make sure all attributes on font tags are preserved [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/17 [05:50:39] (03update) 10r4356th: Make sure all attributes on font tags are preserved [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/17 [06:02:43] (03update) 10r4356th: Make sure all attributes on font tags are preserved [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/17 [06:03:27] (03merge) 10r4356th: Make sure all attributes on font tags are preserved [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/17 [06:05:22] (03open) 10r4356th: Add ty for checking types [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/18 [06:08:47] (03update) 10r4356th: Add ty for checking types [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/18 [06:08:55] (03update) 10r4356th: Add ty for checking types [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/18 [07:07:21] 10Toolforge, 06tools-platform-team: [Toolforge]: User Activity (based on successful SSH logins) - https://phabricator.wikimedia.org/T423228#11822825 (10komla) Counting on Bastion14 was much quicker(lower activity) [07:08:00] FIRING: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [07:52:23] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS: Do not share shm files for openstack uwsgi processes - https://phabricator.wikimedia.org/T423378 (10fgiunchedi) 03NEW [08:18:13] 10Tool-delintbot: Fix all sorts of bogus-image-options errors - https://phabricator.wikimedia.org/T421530#11823046 (10Redmin) [09:39:53] 06cloud-services-team, 10Cloud-VPS: [tofuinfratest] Magnum cluster creation is failing - https://phabricator.wikimedia.org/T423393 (10fnegri) 03NEW [10:44:21] 10Data-Services, 06tools-infrastructure-team, 10Datasets-General-or-Unknown, 06Traffic, 13Patch-For-Review: Migrate clouddumps https/rsync interfaces behind LVS - https://phabricator.wikimedia.org/T422040#11823716 (10taavi) 05Open→03Resolved [10:46:56] 06tools-infrastructure-team, 10Datasets-General-or-Unknown: Reclaim public IPs from individual dumps distribution (clouddumps) hosts - https://phabricator.wikimedia.org/T417028#11823732 (10taavi) [10:57:38] (03update) 10fnegri: [jobs-cli] refactor job payload [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/98 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) (owner: 10raymond-ndibe) [11:08:15] FIRING: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [11:40:05] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Tool-etherpad-backup: Increase object storage object count for etherpads3 project - https://phabricator.wikimedia.org/T423354#11823817 (10Raymond_Ndibe) **Before:** ` sudo radosgw-admin user info --uid 84baa6f9fe8d41afb4b7ca99891161f3\$84baa6f9fe8d41afb4... [11:40:27] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Tool-etherpad-backup: Increase object storage object count for etherpads3 project - https://phabricator.wikimedia.org/T423354#11823818 (10Raymond_Ndibe) a:03Raymond_Ndibe [11:40:41] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Tool-etherpad-backup: Increase object storage object count for etherpads3 project - https://phabricator.wikimedia.org/T423354#11823820 (10Raymond_Ndibe) 05Open→03Resolved [11:55:41] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge, 06tools-platform-team: [jobs-api] support exposing continuous jobs to the internet - https://phabricator.wikimedia.org/T423408 (10Raymond_Ndibe) 03NEW [12:00:44] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge, 06tools-platform-team: [jobs-cli] support publishing continuous job to the internet - https://phabricator.wikimedia.org/T423410 (10Raymond_Ndibe) 03NEW [12:09:37] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Improve linting - unify example validation - https://phabricator.wikimedia.org/T423411 (10KBach) 03NEW [12:11:13] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team: Improve linting - requestBody and response examples - https://phabricator.wikimedia.org/T421375#11823924 (10KBach) 05Open→03Declined Superseded by {T423411}. [12:15:26] 06tools-platform-team: [jobs-cli] refactor to use job_type argument that is already available in jobs-api - https://phabricator.wikimedia.org/T423412 (10Raymond_Ndibe) 03NEW [12:25:54] (03update) 10raymond-ndibe: [jobs-cli] refactor job payload [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/98 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) [12:26:45] 06tools-platform-team: [jobs-cli] refactor to use job_type argument that is already available in jobs-api - https://phabricator.wikimedia.org/T423412#11823999 (10Raymond_Ndibe) [12:27:41] (03update) 10raymond-ndibe: [jobs-cli] refactor job payload [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/98 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136 https://phabricator.wikimedia.org/T423412) [12:29:21] (03update) 10raymond-ndibe: [jobs-cli] refactor job payload [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/98 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136 https://phabricator.wikimedia.org/T423412) [12:29:43] (03update) 10raymond-ndibe: refactor to use job_type argument that is already available in jobs-api [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/98 (https://phabricator.wikimedia.org/T423412) [12:29:54] 10Tool-delintbot: Fix all sorts of bogus-image-options errors - https://phabricator.wikimedia.org/T421530#11824011 (10Redmin) [12:31:52] 10Tool-delintbot: Fix all sorts of bogus-image-options errors - https://phabricator.wikimedia.org/T421530#11824019 (10Redmin) [12:44:35] (03update) 10raymond-ndibe: refactor to use job_type argument that is already available in jobs-api [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/98 (https://phabricator.wikimedia.org/T423412) [12:48:29] 10Tool-campwiz-nxt, 10Google-Summer-of-Code (2026): GSoC 2026: CampWiz NxT Redesign - https://phabricator.wikimedia.org/T414269#11824084 (10Dipanshu1223) @Nokib_Sarkar,@Tiven2240 and @LGoto yesterday I got email of interview but not see any time slot for interview in google calendar. I send the email about th... [12:50:31] (03update) 10raymond-ndibe: refactor to use job_type argument that is already available in jobs-api [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/98 (https://phabricator.wikimedia.org/T423412) [12:57:58] (03update) 10raymond-ndibe: refactor to use job_type argument that is already available in jobs-api [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/98 (https://phabricator.wikimedia.org/T423412) [13:05:34] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: [SPIKE] Verify info.license in non-generated specs - https://phabricator.wikimedia.org/T422911#11824135 (10KBach) [13:10:58] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Improve linter rules for info.license validation - https://phabricator.wikimedia.org/T422912#11824166 (10KBach) 05Open→03Declined Closing as duplicate of {T422911} [13:11:06] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Improve linting - info.license - https://phabricator.wikimedia.org/T422908#11824170 (10KBach) [13:11:07] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Improve linter rules for info.license validation - https://phabricator.wikimedia.org/T422912#11824169 (10KBach) [13:11:45] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Improve linting - info.license - https://phabricator.wikimedia.org/T422908#11824171 (10KBach) [13:12:53] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Improve linting - info.license - https://phabricator.wikimedia.org/T422908#11824176 (10KBach) [13:12:54] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Improve linter rules for info.license validation - https://phabricator.wikimedia.org/T422912#11824175 (10KBach) [13:16:28] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team, 07Essential-Work: Linter: Split rules into more logical groupings. - https://phabricator.wikimedia.org/T415914#11824194 (10KBach) [13:16:30] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team, 07Epic, 07Essential-Work: OpenAPI linter - performance and dev-ex improvements - https://phabricator.wikimedia.org/T415388#11824195 (10KBach) [13:16:43] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team, 07Essential-Work: Linter: Split rules into more logical groupings. - https://phabricator.wikimedia.org/T415914#11824198 (10KBach) This task was originally a subtask of {T415388}. I removed that parent task to simplify the task structure... [13:17:07] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team, 07Essential-Work: Spec caching: Cache validation results for identical OpenAPI specs - https://phabricator.wikimedia.org/T415916#11824205 (10KBach) [13:17:10] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team, 07Epic, 07Essential-Work: OpenAPI linter - performance and dev-ex improvements - https://phabricator.wikimedia.org/T415388#11824206 (10KBach) [13:17:30] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team, 07Essential-Work: Spec caching: Cache validation results for identical OpenAPI specs - https://phabricator.wikimedia.org/T415916#11824208 (10KBach) This task was originally a subtask of {T415388}. I removed that parent task to simplify... [13:17:54] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team, 07Essential-Work: [SPIKE] Figure out best approach for spec result caching - https://phabricator.wikimedia.org/T415917#11824213 (10KBach) [13:17:55] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team, 07Essential-Work: Spec caching: Cache validation results for identical OpenAPI specs - https://phabricator.wikimedia.org/T415916#11824214 (10KBach) [13:18:00] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team, 07Essential-Work: [SPIKE] Figure out best approach for spec result caching - https://phabricator.wikimedia.org/T415917#11824218 (10KBach) This task was originally a subtask of {T415388}. I removed that parent task to simplify the task s... [13:18:36] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team, 07Essential-Work: Lazy loading of ruleset: Implement caching layer for Spectral ruleset - https://phabricator.wikimedia.org/T415920#11824221 (10KBach) [13:18:39] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team, 07Epic, 07Essential-Work: OpenAPI linter - performance and dev-ex improvements - https://phabricator.wikimedia.org/T415388#11824222 (10KBach) [13:18:41] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team, 07Essential-Work: Lazy loading of ruleset: Implement caching layer for Spectral ruleset - https://phabricator.wikimedia.org/T415920#11824224 (10KBach) This task was originally a subtask of {T415388}. I removed that parent task to simpli... [13:22:55] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team, 07Epic, 07Essential-Work: OpenAPI linter - performance and dev-ex improvements - https://phabricator.wikimedia.org/T415388#11824252 (10KBach) [13:24:40] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team, 07Essential-Work: [SPIKE] Figure out best approach for spec result caching - https://phabricator.wikimedia.org/T415917#11824272 (10KBach) [13:24:46] 10Tool-wmf-openapi-linter, 03[MWI] FY2025-26 Q4, 07Epic, 07OKR-Work: [5.2.5b Epic] Create a proof of concept of a CI workflow with the spec linter - https://phabricator.wikimedia.org/T422483#11824273 (10KBach) [13:24:47] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team, 07Essential-Work: Spec caching: Cache validation results for identical OpenAPI specs - https://phabricator.wikimedia.org/T415916#11824274 (10KBach) [14:02:33] 06cloud-services-team, 10Cloud-VPS: [tofuinfratest] Magnum cluster creation is failing - https://phabricator.wikimedia.org/T423393#11824466 (10Andrew) p:05Triage→03Medium [14:03:04] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS: memcache is a SPOF for designate/tooz coordination - https://phabricator.wikimedia.org/T422646#11824468 (10Andrew) p:05Triage→03Medium [14:28:00] 06cloud-services-team, 10Toolforge: lighttpd: starting a webserver logs "WARNING: unknown config-key: server.dir-listing (ignored)" to error.log - https://phabricator.wikimedia.org/T423019#11824651 (10aputhin) p:05Triage→03Low Tagging as "Low" prio for now as it doesn't seem to be blocking (only a Warning). [14:28:36] 10Toolforge, 06tools-platform-team: lighttpd: starting a webserver logs "WARNING: unknown config-key: server.dir-listing (ignored)" to error.log - https://phabricator.wikimedia.org/T423019#11824654 (10aputhin) [14:31:17] 10Toolforge, 06tools-platform-team: [logs-api,jobs-cli] `toolforge jobs logs` has inconsistent ordering - https://phabricator.wikimedia.org/T401552#11824662 (10aputhin) TODO: double check if it's still reproducible - if it is, we'll work on a fix [14:41:43] 10Tool-etherpad-backup: display a list of backed up etherpads somewhere - https://phabricator.wikimedia.org/T423014#11824716 (10Tkarcher) 05Open→03Resolved a:03Tkarcher Done: I created a pad_data.json from the output of the directory listing and an index.html file to present the data in a searchable ta... [14:43:07] 10Toolforge, 06tools-platform-team: Cannot output "Ερευνητής Αλήθειας" - https://phabricator.wikimedia.org/T367663#11824728 (10aputhin) p:05Triage→03Medium TODO: Double check if this is still an issue with latest images / buildpacks [14:45:42] 10Toolforge, 06tools-platform-team: [components-api,beta] Config not updated from remote source - https://phabricator.wikimedia.org/T401868#11824733 (10aputhin) [14:47:53] 06cloud-services-team, 10Toolforge: ChieBot: Intermittent connection reset by peer errors - https://phabricator.wikimedia.org/T356163#11824747 (10aputhin) 05Open→03Declined Closing this for now, as we haven't heard back on whether the issue still persists in 1,5 years. Please reach out again if it's st... [14:53:01] RESOLVED: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [15:45:21] 06cloud-services-team, 10Cloud-VPS, 07Puppet: Repeated Puppet failures for PetScan - https://phabricator.wikimedia.org/T366141#11825251 (10A_smart_kitten) (belatedly triaging into the #cloud-vps project, but @magnus I assume that it might be good if you could confirm whether or not this remains an issue `:)`) [15:47:18] 10Tool-etherpad-backup: Archive etherpads mentioned in hackathon/techconf tasks - https://phabricator.wikimedia.org/T417207#11825267 (10bd808) >>! In T417207#11745229, @Tkarcher wrote: > Sounds good. Just as a heads up: If you plan to copy / mirror all NFS content to S3, you'll quickly hit the storage limit of 4... [15:52:45] 10Tool-etherpad-backup: display a list of backed up etherpads somewhere - https://phabricator.wikimedia.org/T423014#11825298 (10bd808) >>! In T423014#11824717, @Tkarcher wrote: > (Pinging @bd808 to let him know about the update) Thanks. Some slightly different solution will be needed when we get the object... [15:56:48] (03update) 10apaskulin: Copyedits [toolforge-repos/wikimedia-attribution] - 10https://gitlab.wikimedia.org/toolforge-repos/wikimedia-attribution/-/merge_requests/2 [15:57:21] (03update) 10apaskulin: Copyedits [toolforge-repos/wikimedia-attribution] - 10https://gitlab.wikimedia.org/toolforge-repos/wikimedia-attribution/-/merge_requests/2 [16:05:40] 10Toolforge, 06tools-platform-team: [Toolforge]: User Activity (based on successful SSH logins) - https://phabricator.wikimedia.org/T423228#11825422 (10aputhin) 05Open→03In progress [16:21:17] 10VPS-project-Codesearch: Codesearch stuck at Feb 12th? - https://phabricator.wikimedia.org/T421147#11825506 (10Krinkle) There are no `.git/shallow.lock` files this time. We instead have a case of `.git/index.lock`. Last time, ref T421147#11784788 (and the three comments after that). ` krinkle@codesearch9:/srv... [17:19:19] 06cloud-services-team, 10Cloud-VPS, 07Puppet: Repeated Puppet failures for PetScan - https://phabricator.wikimedia.org/T366141#11826138 (10bd808) 05Open→03Declined petscan4 was replaced by petscan5 in mid-2024. If and when something like this happens again the next step in investigating is to ssh int... [17:40:05] 06cloud-services-team, 07artificial-intelligence: Supporting AI, LLM, and data models on WMCS - https://phabricator.wikimedia.org/T336905#11826393 (10Fnielsen) One possibility could be to set up a GPU supported embedding service that would be available from tools to Toolforge and WMCloud only. Initially this c... [17:57:29] (03PS1) 10Akoopal: ignoring some more fields [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1271840 (https://phabricator.wikimedia.org/T55271) [18:42:49] PROBLEM - Host cloudvirt1054 is DOWN: PING CRITICAL - Packet loss = 100% [18:44:14] FIRING: [2x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server toolsbeta-test-k8s-ingress-11.toolsbeta.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [18:44:49] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1054 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [18:47:32] FIRING: InstanceDown: Project gitlab-runners instance runner-1031 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [18:47:32] FIRING: InstanceDown: Project toolsbeta instance toolsbeta-test-k8s-ingress-11 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [18:47:32] FIRING: [4x] InstanceDown: Project tools instance tools-docker-registry-7 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [18:47:41] FIRING: [2x] InstanceDown: Project cloudinfra instance enc-3 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [18:47:45] FIRING: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [18:47:50] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [18:48:10] FIRING: CloudVirtDown: Cloudvirt node cloudvirt1054 is down. #page - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CloudVirtDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudvirt1054 - https://alerts.wikimedia.org/?q=alertname%3DCloudVirtDown [18:52:23] FIRING: ToolforgeKubernetesNodeNotReady: Kubernetes node toolsbeta-test-k8s-ingress-11 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [18:53:23] FIRING: ToolforgeKubernetesNodeNotReady: Kubernetes node tools-k8s-worker-nfs-35 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [18:57:03] RECOVERY - Host cloudvirt1054 is UP: PING OK - Packet loss = 0%, RTA = 0.28 ms [18:57:15] PROBLEM - ensure kvm processes are running on cloudvirt1054 is CRITICAL: PROCS CRITICAL: 0 processes with regex args qemu-system-x86_64 https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting [18:58:10] RESOLVED: CloudVirtDown: Cloudvirt node cloudvirt1054 is down. #page - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CloudVirtDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudvirt1054 - https://alerts.wikimedia.org/?q=alertname%3DCloudVirtDown [18:58:15] RECOVERY - ensure kvm processes are running on cloudvirt1054 is OK: PROCS OK: 7 processes with regex args qemu-system-x86_64 https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting [18:59:50] RESOLVED: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1054 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [19:00:04] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1054 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [19:00:19] RESOLVED: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1054 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [19:02:23] RESOLVED: ToolforgeKubernetesNodeNotReady: Kubernetes node toolsbeta-test-k8s-ingress-11 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [19:02:32] RESOLVED: InstanceDown: Project gitlab-runners instance runner-1031 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [19:02:32] RESOLVED: InstanceDown: Project toolsbeta instance toolsbeta-test-k8s-ingress-11 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [19:02:32] RESOLVED: [2x] InstanceDown: Project cloudinfra instance enc-3 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [19:02:39] RESOLVED: [4x] InstanceDown: Project tools instance tools-docker-registry-7 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [19:02:43] RESOLVED: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [19:02:46] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [19:03:23] RESOLVED: ToolforgeKubernetesNodeNotReady: Kubernetes node tools-k8s-worker-nfs-35 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [19:04:14] RESOLVED: [2x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server toolsbeta-test-k8s-ingress-11.toolsbeta.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDo [19:06:32] (03PS1) 10Lokal Profil: Pass progress arg to process_country in multi-country branch [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1271874 [19:07:57] 06cloud-services-team, 10Cloud-VPS: cloudvirt1054 crash - https://phabricator.wikimedia.org/T423517 (10Andrew) 03NEW [19:13:56] (03CR) 10Akoopal: [C:03+2] "oops, overlooked that one, thanks for the quick patch, looks good." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1271874 (owner: 10Lokal Profil) [19:16:23] (03Merged) 10jenkins-bot: Pass progress arg to process_country in multi-country branch [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1271874 (owner: 10Lokal Profil) [19:24:34] 06cloud-services-team, 10Cloud-VPS: cloudvirt1054 crash - https://phabricator.wikimedia.org/T423517#11826728 (10Andrew) syslog is unhelpful: ` 2026-04-15T18:40:49.207083+00:00 cloudvirt1054 neutron-openvswitch-agent: 2026-04-15 18:40:49.206 2328871 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutr... [19:38:41] 06cloud-services-team, 10Toolforge: Automatically !log webservice related commands - https://phabricator.wikimedia.org/T229173#11826796 (10LucasWerkmeister) > This really sounds like an ask for deployment tool functionality (something like scap3) in a tool that is a service control instead I think at that poi... [20:16:52] (03PS1) 10Lokal Profil: Recreate venv on build-python when pip is missing [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1271893 [20:23:36] (03CR) 10Akoopal: [C:03+2] "all makes sense, code looks clean" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1271893 (owner: 10Lokal Profil) [20:26:22] (03Merged) 10jenkins-bot: Recreate venv on build-python when pip is missing [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1271893 (owner: 10Lokal Profil) [20:33:55] (03PS1) 10Lokal Profil: Truncate muni on character boundary to avoid invalid UTF-8 [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1271900 [20:47:51] 10Tool-etherpad-backup: display a list of backed up etherpads somewhere - https://phabricator.wikimedia.org/T423014#11827049 (10Novem_Linguae) Looks great. Thanks! [20:51:19] 10Tool-paulina: Allow for users to filter out authors by field of work - https://phabricator.wikimedia.org/T393242#11827054 (10Pepe_piton) Done, using @marfossatti's work on the taxonomy of creative areas: T393157 Here's the commit: https://gitlab.wikimedia.org/toolforge-repos/paulina/-/commit/5fae5e357619a026b... [20:51:32] 10Tool-paulina: Allow for users to filter out authors by field of work - https://phabricator.wikimedia.org/T393242#11827056 (10Pepe_piton) 05In progress→03Resolved [20:53:14] (03open) 10vriaa: feat: support character level text styling in banner editor [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/54 (https://phabricator.wikimedia.org/T420943) [21:00:41] 10Tool-paulina: Taxonomía de áreas creativas - https://phabricator.wikimedia.org/T393157#11827112 (10Pepe_piton) 05Open→03Resolved Thank you @marfossatti @Soylacarli @Mind_Booster_Noori! Your help in creating this taxonomy has been super useful for the implementation a new search filter in Paulina, to se... [21:01:51] 10Tool-paulina: Create API call examples to create and edit Wikidata works and authors items. - https://phabricator.wikimedia.org/T405635#11827124 (10Pepe_piton) 05Open→03Resolved a:03Pepe_piton [21:03:14] (03PS1) 10Lokal Profil: Port cl_to lookups to linktarget join [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1271919 [21:03:33] (03PS1) 10Lokal Profil: Avoid fill_table_monuments_all timeout on large monuments_all [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1271920 [21:03:46] 10Tool-paulina: Proof of concept: add work form, with autocomplete - https://phabricator.wikimedia.org/T407558#11827144 (10Pepe_piton) 05Open→03Resolved a:03Pepe_piton [21:04:20] 10Tool-paulina: Proof of concept: Add author form - https://phabricator.wikimedia.org/T407551#11827147 (10Pepe_piton) 05Open→03Resolved a:03Pepe_piton [21:04:29] 10Tool-paulina: Create new generic page - https://phabricator.wikimedia.org/T407385#11827149 (10Pepe_piton) 05Open→03Resolved a:03Pepe_piton [21:04:41] 10Tool-paulina: Proof of concept for adding new statements - https://phabricator.wikimedia.org/T406989#11827151 (10Pepe_piton) 05Open→03Resolved a:03Pepe_piton [21:04:50] 10Tool-paulina: Proof of concept for editing statements in Paulina - https://phabricator.wikimedia.org/T405412#11827154 (10Pepe_piton) 05Open→03Resolved [21:14:13] (03PS1) 10Lokal Profil: Port ContestStatsBuilder cl_to lookups to linktarget join [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1271928 [21:24:37] (03PS1) 10Lokal Profil: Silence curl progress meter in create_toolforge_job [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1271933 [21:36:43] (03PS2) 10Lokal Profil: Silence curl progress meter and drop response body in create_toolforge_job [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1271933 [22:09:45] 10Cloud-VPS (Project-requests), 10VPS-project-voterlists: Request creation of voterlists VPS project - https://phabricator.wikimedia.org/T399418#11827478 (10Novem_Linguae) Did this project end up having a publicly accessible URL? If so please let me know and I will bookmark it. [22:40:06] 10Tools: Image Checker throwing 'Unknown column 'cl_to' in 'ON'' error - https://phabricator.wikimedia.org/T423532 (10Engrigg22) 03NEW [23:02:51] 10Tools: Image Checker throwing 'Unknown column 'cl_to' in 'ON'' error - https://phabricator.wikimedia.org/T423532#11827623 (10Zabe) No, they apparently missed T299951. Also the filtering of `il_to` will no work due to T299953. [23:24:34] (03merge) 10vriaa: feat: support character level text styling in banner editor [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/54 (https://phabricator.wikimedia.org/T420943) [23:31:51] 10Tool-centralnotice-banner-editor: Allow bold to be applied to individual words or phrases within a text element - https://phabricator.wikimedia.org/T420943#11827708 (10Oyelola_Victoria) 05In progress→03Resolved [23:36:38] (03open) 10vriaa: fix: viewport menu group description [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/55 [23:36:57] (03merge) 10vriaa: fix: viewport menu group description [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/55