[00:53:56] FIRING: ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:58:56] RESOLVED: ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:05:56] FIRING: ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:10:56] RESOLVED: ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:13:56] FIRING: [2x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:18:56] RESOLVED: [4x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:17:18] 06cloud-services-team, 10PAWS, 10Lingua-Libre: Allows request to all endpoints in wikidata allowlist - https://phabricator.wikimedia.org/T423969#11848479 (10Yug) [07:18:13] 06cloud-services-team, 10PAWS, 10Lingua-Libre: Allows request to all endpoints in wikidata allowlist - https://phabricator.wikimedia.org/T423969#11848480 (10Yug) [08:16:25] (03PS1) 10Alien4444: Use tabs not spaces in twig files and SQL queries [labs/xtools] - 10https://gerrit.wikimedia.org/r/1276362 [09:25:47] (03PS1) 10Alien4444: SimpleEditCounter: add latest edit and log [labs/xtools] - 10https://gerrit.wikimedia.org/r/1276371 (https://phabricator.wikimedia.org/T198872) [09:49:13] 10Tool-ldap: HTTP 500 'filter parse error' when visiting `/cn/Andrew%20McAllister%20(WMDE)` - https://phabricator.wikimedia.org/T424129 (10A_smart_kitten) 03NEW [12:38:41] (03CR) 10Jean-Frédéric: Sync scheduled Toolforge jobs from jobs.yml.template (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1274108 (https://phabricator.wikimedia.org/T423771) (owner: 10Lokal Profil) [13:23:40] (03CR) 10Jean-Frédéric: [C:03+2] Fix removed (in 9+) pywikibot APIs in bot scripts [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1270569 (https://phabricator.wikimedia.org/T409003) (owner: 10Lokal Profil) [13:27:04] (03Merged) 10jenkins-bot: Fix removed (in 9+) pywikibot APIs in bot scripts [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1270569 (https://phabricator.wikimedia.org/T409003) (owner: 10Lokal Profil) [13:56:02] (03open) 10vriaa: refactor: make text run styling reusable [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/57 [13:56:41] (03merge) 10vriaa: refactor: make text run styling reusable [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/57 [13:58:18] (03update) 10pepepiton: Draft: initial commit for pipeline [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/182 (owner: 10davidcoronel) [15:36:50] 06cloud-services-team, 10superset.wmcloud.org: Sunset superset.wmcloud.org - https://phabricator.wikimedia.org/T416373#11849187 (10Andrew) Great! The two of you should now have full access to the project -- please follow up here if you find otherwise, and don't hesitate to reach out if there's anything I/we ca... [15:37:30] 06cloud-services-team: Advice for Hybrid Architecture for AI Tool - https://phabricator.wikimedia.org/T416590#11849190 (10Odinaldo) This is extremely useful, thanks @bd808. We will migrate to a fully compliant RDBMS and both MariaDB and PostgreSQL seem like good candidates. We are prioritising faster access and... [15:41:12] 06cloud-services-team, 10superset.wmcloud.org: Sunset superset.wmcloud.org - https://phabricator.wikimedia.org/T416373#11849195 (10Andrew) Oh, one other thing, are you happy to continue to use this [[ https://phabricator.wikimedia.org/tag/superset.wmcloud.org/ | phab tag ]] for coordinating with users, or do y... [15:44:57] (03update) 10danyya: Draft: SQLite CLI [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/12 [15:50:52] 10Tool-humaniki-2: Use dumps instead of live QLever/MediaWiki queries - https://phabricator.wikimedia.org/T424147 (10Danya) 03NEW [15:51:51] 10Tool-humaniki-2: Use dumps instead of live QLever/MediaWiki queries - https://phabricator.wikimedia.org/T424147#11849211 (10Danya) p:05Triage→03High a:03Danya [15:51:52] 10Tool-humaniki-2: Use dumps instead of live QLever/MediaWiki queries - https://phabricator.wikimedia.org/T424147#11849214 (10Danya) 05Open→03In progress [15:52:18] (03update) 10danyya: SQLite CLI [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/12 [15:52:30] (03merge) 10danyya: SQLite CLI [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/12 [16:36:42] 10Tool-humaniki-2: Scheduler does not take birth year snapshots - https://phabricator.wikimedia.org/T423025#11849266 (10Danya) 05Open→03In progress [16:36:59] 10Tool-humaniki-2: Scheduler does not take birth year snapshots - https://phabricator.wikimedia.org/T423025#11849267 (10Danya) a:03Danya [16:46:07] 10Tool-humaniki-2: "unknown" gender QID gets saved to database - https://phabricator.wikimedia.org/T424150 (10Danya) 03NEW [16:46:18] 10Tool-humaniki-2: "unknown" gender QID gets saved to database - https://phabricator.wikimedia.org/T424150#11849283 (10Danya) p:05Triage→03Low [16:46:56] 10Tool-humaniki-2: Use dumps instead of live QLever/MediaWiki queries? - https://phabricator.wikimedia.org/T424147#11849285 (10Danya) [17:21:52] 10Tool-wikimonitor: [WikiStreamService] High-Efficiency Event Broadcasting via DTO - https://phabricator.wikimedia.org/T424152 (10Gerges) 03NEW [17:22:05] 10Tool-wikimonitor: [WikiStreamService] High-Efficiency Event Broadcasting via DTO - https://phabricator.wikimedia.org/T424152#11849339 (10Gerges) a:03Gerges [17:22:35] 10Tool-wikimonitor: [WikiStreamService] High-Efficiency Event Broadcasting via DTO - https://phabricator.wikimedia.org/T424152#11849340 (10Gerges) p:05Triage→03High [17:23:05] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Improve linting - detect examples in nested schema.properties - https://phabricator.wikimedia.org/T424002#11849341 (10Atieno) p:05Triage→03Medium [17:32:04] 10Tool-wikimonitor: [WikiStreamService] High-Efficiency Event Broadcasting via DTO - https://phabricator.wikimedia.org/T424152#11849355 (10Gerges) 05Open→03In progress [18:25:29] 10Tool-wikimonitor: [WikiStreamService] 30-Minute SseEmitter Timeout with Event Resumption - https://phabricator.wikimedia.org/T422194#11849404 (10Gerges) [19:36:19] (03CR) 10Lokal Profil: Fix removed (in 9+) pywikibot APIs in bot scripts (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1270569 (https://phabricator.wikimedia.org/T409003) (owner: 10Lokal Profil) [20:42:20] (03CR) 10Jean-Frédéric: Tighten monuments_config schema and validation tests (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1275532 (owner: 10Lokal Profil) [21:28:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [22:03:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [23:30:14] (03update) 10davidcoronel: Draft: initial commit for pipeline [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/182 [23:37:26] (03update) 10bd808: Draw the rest of the owl [toolforge-repos/etherpad-backup] - 10https://gitlab.wikimedia.org/toolforge-repos/etherpad-backup/-/merge_requests/1 (https://phabricator.wikimedia.org/T422958 https://phabricator.wikimedia.org/T423014)