[00:05:03] <wmcs-alerts>	 (InstanceDown) firing: Project tools instance tools-harbor-1 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[00:10:03] <wmcs-alerts>	 (InstanceDown) resolved: Project tools instance tools-harbor-1 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[00:37:03] <wmcs-alerts>	 (InstanceDown) firing: Project tools instance tools-harbor-1 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[00:39:38] <jinxer-wm>	 (SystemdUnitDown) firing: (2) The service unit wikitech_run_jobs.service is in failed status on host cloudweb1003. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown  - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown
[00:44:38] <jinxer-wm>	 (SystemdUnitDown) resolved: (2) The service unit wikitech_run_jobs.service is in failed status on host cloudweb1003. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown  - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown
[00:47:35] <wmcs-alerts>	 (HarborDown) firing: Harbor is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborDown  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborDown
[00:49:38] <jinxer-wm>	 (SystemdUnitDown) firing: The service unit wikitech_run_jobs.service is in failed status on host cloudweb1004. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudweb1004 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown
[00:52:03] <wmcs-alerts>	 (InstanceDown) resolved: Project tools instance tools-harbor-1 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[00:52:35] <wmcs-alerts>	 (HarborDown) resolved: Harbor is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborDown  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborDown
[00:54:38] <jinxer-wm>	 (SystemdUnitDown) resolved: The service unit wikitech_run_jobs.service is in failed status on host cloudweb1004. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudweb1004 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown
[01:04:38] <jinxer-wm>	 (SystemdUnitDown) firing: (2) The service unit nova-fullstack.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown  - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown
[01:19:38] <jinxer-wm>	 (SystemdUnitDown) resolved: The service unit nova-fullstack.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown
[02:52:44] <jinxer-wm>	 (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse
[04:06:03] <wmcs-alerts>	 (InstanceDown) firing: Project tools instance tools-harbor-1 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[04:13:35] <wmcs-alerts>	 (HarborDown) firing: Harbor is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborDown  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborDown
[04:16:03] <wmcs-alerts>	 (InstanceDown) resolved: Project tools instance tools-harbor-1 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[04:18:35] <wmcs-alerts>	 (HarborDown) resolved: Harbor is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborDown  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborDown
[05:36:03] <wmcs-alerts>	 (InstanceDown) firing: Project tools instance tools-harbor-1 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[05:46:03] <wmcs-alerts>	 (InstanceDown) resolved: Project tools instance tools-harbor-1 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[06:35:03] <wmcs-alerts>	 (InstanceDown) firing: Project tools instance tools-harbor-1 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[06:43:35] <wmcs-alerts>	 (HarborDown) firing: Harbor is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborDown  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborDown
[06:52:44] <jinxer-wm>	 (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse
[06:53:35] <wmcs-alerts>	 (HarborDown) resolved: Harbor is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborDown  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborDown
[06:55:03] <wmcs-alerts>	 (InstanceDown) resolved: Project tools instance tools-harbor-1 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[07:07:30] <jinxer-wm>	 (OpenstackAPIResponse) resolved: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse
[07:36:03] <wmcs-alerts>	 (InstanceDown) firing: Project tools instance tools-harbor-1 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[07:41:35] <wmcs-alerts>	 (HarborDown) firing: Harbor is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborDown  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborDown
[07:51:03] <wmcs-alerts>	 (InstanceDown) resolved: Project tools instance tools-harbor-1 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[07:51:35] <wmcs-alerts>	 (HarborDown) resolved: Harbor is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborDown  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborDown
[08:35:03] <wmcs-alerts>	 (InstanceDown) firing: Project tools instance tools-harbor-1 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[08:47:35] <wmcs-alerts>	 (HarborDown) firing: Harbor is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborDown  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborDown
[08:55:03] <wmcs-alerts>	 (InstanceDown) resolved: Project tools instance tools-harbor-1 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[08:57:35] <wmcs-alerts>	 (HarborDown) resolved: Harbor is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborDown  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborDown
[10:05:03] <wmcs-alerts>	 (InstanceDown) firing: Project tools instance tools-harbor-1 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[10:10:35] <wmcs-alerts>	 (HarborDown) firing: Harbor is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborDown  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborDown
[10:20:03] <wmcs-alerts>	 (InstanceDown) resolved: Project tools instance tools-harbor-1 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[10:20:35] <wmcs-alerts>	 (HarborDown) resolved: Harbor is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborDown  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborDown
[10:41:42] <wikibugs>	 (03PS1) 10Majavah: cli: Simplify code [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/986804
[10:41:44] <wikibugs>	 (03PS1) 10Majavah: task 5: week numbering sucks [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/986805
[10:42:57] <wikibugs>	 (03CR) 10Majavah: [C: 03+2] cli: Simplify code [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/986804 (owner: 10Majavah)
[10:43:03] <wikibugs>	 (03CR) 10Majavah: [C: 03+2] task 5: week numbering sucks [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/986805 (owner: 10Majavah)
[10:44:02] <wikibugs>	 (03Merged) 10jenkins-bot: cli: Simplify code [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/986804 (owner: 10Majavah)
[10:44:05] <wikibugs>	 (03Merged) 10jenkins-bot: task 5: week numbering sucks [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/986805 (owner: 10Majavah)
[11:05:03] <wmcs-alerts>	 (InstanceDown) firing: Project tools instance tools-harbor-1 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[11:10:03] <wmcs-alerts>	 (InstanceDown) resolved: Project tools instance tools-harbor-1 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[12:35:03] <wmcs-alerts>	 (InstanceDown) firing: Project tools instance tools-harbor-1 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[12:40:03] <wmcs-alerts>	 (InstanceDown) resolved: Project tools instance tools-harbor-1 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[15:07:03] <wmcs-alerts>	 (InstanceDown) firing: Project tools instance tools-harbor-1 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[15:12:35] <wmcs-alerts>	 (HarborDown) firing: Harbor is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborDown  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborDown
[15:27:03] <wmcs-alerts>	 (InstanceDown) resolved: Project tools instance tools-harbor-1 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[15:27:35] <wmcs-alerts>	 (HarborDown) resolved: Harbor is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborDown  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborDown
[15:35:03] <wmcs-alerts>	 (InstanceDown) firing: Project tools instance tools-harbor-1 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[15:42:35] <wmcs-alerts>	 (HarborDown) firing: Harbor is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborDown  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborDown
[15:53:37] <wikibugs>	 10Toolforge Build Service, 10cloud-services-team: tools-harbor-1.tools.eqiad1.wikimedia.cloud overloaded - https://phabricator.wikimedia.org/T354151 (10Andrew)
[15:57:35] <wmcs-alerts>	 (HarborDown) resolved: Harbor is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborDown  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborDown
[16:00:03] <wmcs-alerts>	 (InstanceDown) resolved: Project tools instance tools-harbor-1 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[16:06:03] <wmcs-alerts>	 (InstanceDown) firing: Project tools instance tools-harbor-1 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[16:26:03] <wmcs-alerts>	 (InstanceDown) resolved: Project tools instance tools-harbor-1 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[20:56:02] <wikibugs>	 10Striker, 10wikitech.wikimedia.org, 10MediaWiki-extensions-OATHAuth, 10TestMe: Wikitech 2FA does not appear to allow recovery with recovery codes - https://phabricator.wikimedia.org/T204682 (10Reedy)
[21:49:16] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate redirtalkdeleter from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319998 (10Andriy.v) >>! In T319998#9423736, @Lejonel wrote: > https://wikitech.wikimedia.org/wiki/Help:Toolforge/Running_Pywikibot_scripts_(advanced) might be better for...
[23:56:28] <wikibugs>	 10Toolforge, 10Elasticsearch: Add access control for Toolforge Elasticsearch - https://phabricator.wikimedia.org/T348943 (10RoySmith) Some time ago, when I first started looking at a project which would use ES, I decided I was unable to use the Toolforge ES for exactly the reason @SD0001 describes.  I didn't w...