[07:11:18] 10Phabricator, 06Infrastructure-Foundations: Removing an offboarded user from privileged Phab ACL projects should not require admins - https://phabricator.wikimedia.org/T392111#10767413 (10MoritzMuehlenhoff) There's a central offboarding process, which covers - shell access - NDA-relevant LDAP groups - NDA-rel... [07:11:31] 10Phabricator, 06Infrastructure-Foundations: Removing an offboarded user from privileged Phab ACL projects should not require admins - https://phabricator.wikimedia.org/T392111#10767414 (10MoritzMuehlenhoff) p:05Triageโ†’03Medium [07:17:46] 10Phabricator, 06Security-Team, 07SecTeam-Processed: Audit Phabricator security policies and groups membership - https://phabricator.wikimedia.org/T391150#10767422 (10MoritzMuehlenhoff) >>! In T391150#10718489, @sbassett wrote: > I think we can have both. #security-team does this during our own offboarding... [08:54:12] (03open) 10jnuche: spiderpig: notify backport operator of prompts [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/771 (https://phabricator.wikimedia.org/T392487) [08:55:04] (03update) 10jnuche: spiderpig: notify backport operator of prompts [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/771 (https://phabricator.wikimedia.org/T392487) [08:57:10] (03update) 10jnuche: Draft: spiderpig: notify backport operator of prompts [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/771 (https://phabricator.wikimedia.org/T392487) [09:04:38] (03update) 10jnuche: Draft: spiderpig: notify backport operator of prompts [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/771 (https://phabricator.wikimedia.org/T392487) [09:05:14] (03update) 10jnuche: Draft: spiderpig: notify backport operator of prompts [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/771 (https://phabricator.wikimedia.org/T392487) [09:08:03] 10Gerrit, 06Release-Engineering-Team: Enable and explore Google search console for Gerrit - https://phabricator.wikimedia.org/T392669 (10hashar) 03NEW [09:08:18] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services: Enable and explore Google search console for Gerrit - https://phabricator.wikimedia.org/T392669#10767617 (10hashar) [09:08:40] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services: Enable and explore Google search console for Gerrit - https://phabricator.wikimedia.org/T392669#10767618 (10hashar) [09:26:31] 10Release-Engineering-Team (Yak Shaving ๐Ÿƒ๐Ÿช’), 10Scap (SpiderPig ๐Ÿ•ธ๏ธ), 13Patch-For-Review: Add browser notification when deployment is awaiting user interaction - https://phabricator.wikimedia.org/T392487#10767634 (10jnuche) I've created a draft with only the user that triggered a backport job getting notificat... [10:09:34] 10Scap (SpiderPig ๐Ÿ•ธ๏ธ): Format links in commit messages using Codex link styling - https://phabricator.wikimedia.org/T392673 (10taavi) 03NEW [11:19:48] 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 3 others: Migrate gitlab storage to apus (also: backups from S3?) - https://phabricator.wikimedia.org/T378922#10767820 (10Jelto) I've done some more tests with gitlab-replica-b.wikimedia.org and the download speed,... [11:20:04] 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 3 others: Migrate gitlab storage to apus (also: backups from S3?) - https://phabricator.wikimedia.org/T378922#10767822 (10Jelto) [13:18:41] hullo, just a heads up that my home IP got caught up in the blocks for the beta cluster โ€” I raised T392684 and can proxy around the block (for now) [13:23:02] shit isp is shit [13:24:05] OFNL is *awful* [13:25:21] (or at least our ISP who use OFNL is.. no IPv6 in 2025?!) [13:27:27] I too can now say "which ridiculous ISP would not have IPv6 in 2025??" [13:38:22] * Lucas_WMDE glances at recent cloud-l announcements ;) [13:41:33] :-) [14:06:32] (03update) 10dancy: spiderpig: Revise /api/searchPatch interface [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/768 (https://phabricator.wikimedia.org/T392591) [14:08:23] (03merge) 10dancy: spiderpig: Revise /api/searchPatch interface [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/768 (https://phabricator.wikimedia.org/T392591) [14:11:38] (03update) 10dancy: web: Bump Codex from 1.15.0 to 1.23.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/770 [14:13:35] (03merge) 10dancy: web: Bump Codex from 1.15.0 to 1.23.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/770 [14:14:33] 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 3 others: Migrate gitlab storage to apus (also: backups from S3?) - https://phabricator.wikimedia.org/T378922#10768322 (10MatthewVernon) This is good to know, thanks :) [14:16:27] (03open) 10dancy: Release 4.156.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/772 [14:17:39] (03merge) 10dancy: Release 4.156.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/772 [14:47:53] 10MediaWiki-Releasing, 10MediaWiki-Installer, 06MediaWiki-Stakeholders-Group, 05MW-1.45-release: Expand the set of bundled extensions and skins in MediaWiki 1.45 - https://phabricator.wikimedia.org/T392080#10768392 (10A_smart_kitten) (Tagging #mediawiki-stakeholders-group for parity with previous tasks) [14:49:28] 10Scap: Drop support for Debian Buster - https://phabricator.wikimedia.org/T392693 (10dancy) 03NEW [14:49:56] ping for T387060 again, could you resolve this? (: [14:49:57] T387060: Create new GitLab project group: wmgmc - https://phabricator.wikimedia.org/T387060 [14:51:14] xtex: I can take care of that. [14:51:22] Sorry for the delay [14:54:22] thanks dancy. I just saw it too. Poke me if you need better instructions on things. [14:56:11] bd808: I'll look into getting you the right privs too [14:56:55] dancy: thanks! [14:59:46] dancy: ha. I misread who you were responding to the first time. I was looking at TheresNoTime's beta cluster firewall hole problem. But yes please to getting whatever hat I'm missing right now in gitlab for setting up new projects. [15:01:05] 10GitLab (Project Migration), 06Release-Engineering-Team: Create new GitLab project group: wmgmc - https://phabricator.wikimedia.org/T387060#10768454 (10dancy) 05Openโ†’03Resolved a:03dancy https://gitlab.wikimedia.org/groups/repos/wmgmc is ready now. I added `xtex` as the owner. [15:06:12] xtex: Done [15:06:32] thank you dancy! [15:07:29] bd808: You are a gitlab admin, so you should have been able to create the subgroup. I'll note that it did take me a while to figure out the right place to go to do it. [15:15:38] 10Phabricator, 06Security-Team, 07SecTeam-Processed: Audit Phabricator security policies and groups membership - https://phabricator.wikimedia.org/T391150#10768490 (10sbassett) >>! In T391150#10767422, @MoritzMuehlenhoff wrote: > Our offboarding process already covers both cases. The process is only document... [15:17:15] (03open) 10dancy: Revert: pyproject.toml: Modernize license specification [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/773 (https://phabricator.wikimedia.org/T392693) [15:17:17] (03update) 10dancy: Revert: pyproject.toml: Modernize license specification [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/773 (https://phabricator.wikimedia.org/T392693) [15:18:11] 10Scap, 13Patch-For-Review: Drop support for Debian Buster - https://phabricator.wikimedia.org/T392693#10768502 (10dancy) p:05Triageโ†’03Low [15:19:28] (03merge) 10dancy: Revert: pyproject.toml: Modernize license specification [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/773 (https://phabricator.wikimedia.org/T392693) [15:20:07] (03open) 10dancy: Release 4.157.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/774 [15:20:25] 10Scap, 13Patch-For-Review: Drop support for Debian Buster - https://phabricator.wikimedia.org/T392693#10768509 (10taavi) [15:21:57] (03merge) 10dancy: Release 4.157.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/774 [15:30:53] 10Release-Engineering-Team (Yak Shaving ๐Ÿƒ๐Ÿช’), 10Scap (SpiderPig ๐Ÿ•ธ๏ธ): Support pasting Gerrit URLs into SpiderPig - https://phabricator.wikimedia.org/T392591#10768584 (10dancy) 05In progressโ†’03Resolved Update deployed. Pasting a Gerrit URL should work more reasonably now. [15:37:51] 10Scap (SpiderPig ๐Ÿ•ธ๏ธ): Format links in commit messages using Codex link styling - https://phabricator.wikimedia.org/T392673#10768603 (10dancy) @taavi Can you explain your request in terms of a representative code diff please? [15:38:12] 10Scap, 06serviceops: Drop support for Debian Buster - https://phabricator.wikimedia.org/T392693#10768605 (10thcipriani) Tagging in #serviceops since they're working to remove these hosts entirely and are best positioned to say where this lands in that roadmap. [15:42:19] (03open) 10taavi: spiderpig: JobCard: Apply Codex link styles to message body [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/775 (https://phabricator.wikimedia.org/T392673) [15:42:22] (03update) 10taavi: spiderpig: JobCard: Apply Codex link styles to message body [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/775 (https://phabricator.wikimedia.org/T392673) [15:43:31] FIRING: [2x] ProbeDown: Service gerrit2002:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [15:43:40] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T392699 (10phaultfinder) 03NEW [15:44:02] 10Release-Engineering-Team (Yak Shaving ๐Ÿƒ๐Ÿช’), 10Scap (SpiderPig ๐Ÿ•ธ๏ธ): Support pasting Gerrit URLs into SpiderPig - https://phabricator.wikimedia.org/T392591#10768625 (10Lucas_Werkmeister_WMDE) Seems to work great, thanks! \o/ [15:44:07] 10Scap (SpiderPig ๐Ÿ•ธ๏ธ), 13Patch-For-Review: Format links in commit messages using Codex link styling - https://phabricator.wikimedia.org/T392673#10768626 (10taavi) >>! In T392673#10768603, @dancy wrote: > @taavi Can you explain your request in terms of a representative code diff please? Sure: https://gitlab.wi... [15:47:05] (03merge) 10dancy: spiderpig: JobCard: Apply Codex link styles to message body [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/775 (https://phabricator.wikimedia.org/T392673) (owner: 10taavi) [15:47:47] (03open) 10dancy: Release 4.157.1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/776 [15:48:31] RESOLVED: [2x] ProbeDown: Service gerrit2002:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [15:49:02] (03merge) 10dancy: Release 4.157.1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/776 [15:49:31] FIRING: [2x] ProbeDown: Service gerrit2002:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [15:49:35] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T392699#10768631 (10phaultfinder) [15:50:22] 10Release-Engineering-Team (Yak Shaving ๐Ÿƒ๐Ÿช’), 10Scap (SpiderPig ๐Ÿ•ธ๏ธ), 13Patch-For-Review: Format links in commit messages using Codex link styling - https://phabricator.wikimedia.org/T392673#10768632 (10dancy) [15:51:38] 10Release-Engineering-Team (Yak Shaving ๐Ÿƒ๐Ÿช’), 10Scap (SpiderPig ๐Ÿ•ธ๏ธ), 13Patch-For-Review: Format links in commit messages using Codex link styling - https://phabricator.wikimedia.org/T392673#10768636 (10dancy) p:05Triageโ†’03Medium a:03taavi [15:54:31] 10Release-Engineering-Team (Yak Shaving ๐Ÿƒ๐Ÿช’), 10Scap (SpiderPig ๐Ÿ•ธ๏ธ): Support pasting Gerrit URLs into SpiderPig - https://phabricator.wikimedia.org/T392591#10768671 (10Lucas_Werkmeister_WMDE) > Also โ€“ though this might be a separate issue โ€“ the Home and End keys donโ€™t work in the input field either, making... [15:54:53] 10Release-Engineering-Team (Yak Shaving ๐Ÿƒ๐Ÿช’), 10Scap (SpiderPig ๐Ÿ•ธ๏ธ), 13Patch-For-Review: Format links in commit messages using Codex link styling - https://phabricator.wikimedia.org/T392673#10768673 (10dancy) 05Openโ†’03Resolved Deployed in scap 4.157.1 [15:59:31] RESOLVED: [2x] ProbeDown: Service gerrit2002:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [16:04:10] 10Continuous-Integration-Infrastructure, 10Testing Support, 10ci-test-error (WMF-deployed Build Failure), 10MW-1.44-notes (1.44.0-wmf.23; 2025-04-01), and 2 others: Selenium timeouts can cause the job to remain stuck until the build times out - https://phabricator.wikimedia.org/T389536#10768702 (10Umherirre... [16:09:58] 10GitLab (Pipeline Services Migration๐Ÿค), 06collaboration-services, 13Patch-For-Review: move os-reports.wikimedia.org to kubernetes - https://phabricator.wikimedia.org/T350794#10768718 (10Arnoldokoth) @Jelto Thank you for this. I had tried some blubber combinations of `lives` and `runs` but none of them worke... [16:31:31] FIRING: ProbeDown: Service gerrit2002:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [16:31:42] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T392699#10768811 (10phaultfinder) [16:36:31] FIRING: [2x] ProbeDown: Service gerrit2002:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [16:36:40] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T392699#10768845 (10phaultfinder) [16:41:31] RESOLVED: [2x] ProbeDown: Service gerrit2002:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:03:01] 10GitLab (Upstream pit of despair ๐Ÿ•ณ๏ธ), 10DPE-Mediawiki-Content, 10Data-Engineering (Q4 2025 April 1st - June 30th): Investigate artifact mismatch error when running mw_content_merge_events_to_mw_content_history_daily - https://phabricator.wikimedia.org/T391123#10768947 (10thcipriani) [17:10:31] FIRING: [2x] ProbeDown: Service gerrit2002:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:13:03] 10Release-Engineering-Team (Yak Shaving ๐Ÿƒ๐Ÿช’), 10Scap (SpiderPig ๐Ÿ•ธ๏ธ): Automatically select first backport search match - https://phabricator.wikimedia.org/T392508#10768968 (10bd808) >>! In T392591#10768663, @Lucas_Werkmeister_WMDE wrote: >> Also โ€“ though this might be a separate issue โ€“ the Home and End keys don... [17:20:31] RESOLVED: [2x] ProbeDown: Service gerrit2002:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:41:31] FIRING: ProbeDown: Service gerrit2002:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:41:38] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T392699#10769069 (10phaultfinder) [17:46:31] RESOLVED: ProbeDown: Service gerrit2002:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:48:31] FIRING: [2x] ProbeDown: Service gerrit2002:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:48:40] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T392699#10769090 (10phaultfinder) [17:53:31] RESOLVED: [2x] ProbeDown: Service gerrit2002:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:55:31] FIRING: [2x] ProbeDown: Service gerrit2002:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:00:31] RESOLVED: [2x] ProbeDown: Service gerrit2002:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:05:42] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T392699#10769118 (10thcipriani) Hrm, Gerrit's JVM looks fine. Apache seems like it's not getting hammered any more than it is usually being hammeredโ€”internal traffic seems like the most prevalent thing at the mo... [18:18:47] (03PS2) 10Jforrester: Zuul: [operations/software/gerrit] Switch to Node 20 CI [integration/config] - 10https://gerrit.wikimedia.org/r/1138385 [18:18:56] (03PS3) 10Jforrester: Zuul: [operations/software/gerrit] Switch to Node 20 CI [integration/config] - 10https://gerrit.wikimedia.org/r/1138385 [18:20:33] 10Continuous-Integration-Config: Create a non-voting CI job to flag deprecated code - https://phabricator.wikimedia.org/T388948#10769160 (10Michael) >>! In T388948#10638982, @Jdforrester-WMF wrote: > Loathe as I am to add Yet Another CI Job at a time when CI is massively congested and we're trying to make fewer... [18:25:58] (03CR) 10Jforrester: "@hashar, you reverted this last time without comment, so waiting on your approval. This is the last Node 18 image use in CI, finally." [integration/config] - 10https://gerrit.wikimedia.org/r/1138385 (owner: 10Jforrester) [19:09:19] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Enable and explore Google search console for Gerrit - https://phabricator.wikimedia.org/T392669#10769291 (10Dzahn) I really like the idea to let code comments be indexed! Regarding the access to search console itself thoug... [19:17:13] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - gerrit2002 - https://phabricator.wikimedia.org/T392699#10769301 (10Dzahn) [19:18:09] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - gerrit2002 - https://phabricator.wikimedia.org/T392699#10769307 (10Dzahn) I looked at our Gerrit Grafana dashboard and availability, error rate, HTTP responses, SSH failures.. they all look like nothing happened here. [19:22:31] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - gerrit2002 - https://phabricator.wikimedia.org/T392699#10769331 (10Dzahn) Looking further back at the last 24 hours and noticed the number of open connections and the nftables throttling tracking list size has increased. This would be T392467 a... [19:31:41] dancy: testing [21:05:31] FIRING: [2x] ProbeDown: Service gerrit2002:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:05:39] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T392727 (10phaultfinder) 03NEW [21:10:31] RESOLVED: [2x] ProbeDown: Service gerrit2002:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:38:59] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - gerrit2002 - https://phabricator.wikimedia.org/T392727#10769640 (10Dzahn) [21:40:31] FIRING: [2x] ProbeDown: Service gerrit2002:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:40:39] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T392732 (10phaultfinder) 03NEW [21:45:31] RESOLVED: [2x] ProbeDown: Service gerrit2002:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [22:05:31] FIRING: [2x] ProbeDown: Service gerrit2002:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [22:05:42] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T392732#10769676 (10phaultfinder) [22:10:31] RESOLVED: [2x] ProbeDown: Service gerrit2002:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [23:15:36] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - gerrit2002 - https://phabricator.wikimedia.org/T392732#10769842 (10Dzahn)