[01:09:01] <jinxer-wm>	 (NodeTextfileStale) firing: Stale textfile for cloudcontrol2004-dev:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale
[03:24:59] <jinxer-wm>	 (PuppetConstantChange) firing: Puppet performing a change on every puppet run on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange
[05:09:16] <jinxer-wm>	 (NodeTextfileStale) firing: Stale textfile for cloudcontrol2004-dev:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale
[06:24:20] <jinxer-wm>	 (HAProxyBackendUnavailable) firing: HAProxy service neutron-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable
[06:29:19] <jinxer-wm>	 (HAProxyBackendUnavailable) resolved: HAProxy service neutron-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable
[07:24:59] <jinxer-wm>	 (PuppetConstantChange) firing: Puppet performing a change on every puppet run on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange
[09:09:16] <jinxer-wm>	 (NodeTextfileStale) firing: Stale textfile for cloudcontrol2004-dev:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale
[09:45:19] <jinxer-wm>	 (HAProxyBackendUnavailable) firing: HAProxy service neutron-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable
[09:50:19] <jinxer-wm>	 (HAProxyBackendUnavailable) resolved: HAProxy service neutron-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable
[11:25:00] <jinxer-wm>	 (PuppetConstantChange) firing: Puppet performing a change on every puppet run on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange
[12:50:45] <wmcs-alerts>	 (ProbeDown) firing: Service tools-k8s-haproxy-3:30000 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-3:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[12:55:45] <wmcs-alerts>	 (ProbeDown) resolved: Service tools-k8s-haproxy-3:30000 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-3:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[13:07:19] <jinxer-wm>	 (HAProxyBackendUnavailable) firing: HAProxy service neutron-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable
[13:09:16] <jinxer-wm>	 (NodeTextfileStale) firing: Stale textfile for cloudcontrol2004-dev:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale
[13:12:19] <jinxer-wm>	 (HAProxyBackendUnavailable) resolved: HAProxy service neutron-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable
[13:25:00] <jinxer-wm>	 (PuppetConstantChange) resolved: Puppet performing a change on every puppet run on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange
[14:32:56] <wmcs-alerts>	 (ToolsToolsDBReplicationMissing) firing: ToolsDB replication is not running on tools-db-1 (errno 0) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationMissing
[14:32:56] <wmcs-alerts>	 (ToolsToolsDBReplicationError) firing: ToolsDB replication is broken on tools-db-2 (errno 1595) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationError
[15:19:02] <jinxer-wm>	 (NodeTextfileStale) resolved: Stale textfile for cloudcontrol2004-dev:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale
[15:22:56] <wmcs-alerts>	 (ToolsToolsDBReplicationMissing) resolved: ToolsDB replication is not running on tools-db-1 (errno 0) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationMissing
[15:22:56] <wmcs-alerts>	 (ToolsToolsDBReplicationError) resolved: ToolsDB replication is broken on tools-db-2 (errno 1595) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationError
[15:36:03] <wmcs-alerts>	 (InstanceDown) firing: Project toolsbeta instance toolsbeta-bastion-6 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[17:51:00] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate wahrani from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320133 (10wahrani) Sorry for this late response.  I already migrated my jobs from GridEngine (jsub) to Kubernetes (toolforge-jobs). My "crontab.grid_stopped" file is empty with n...
[18:30:14] <wikibugs>	 10PAWS, 10Pywikibot, 10Documentation, 10Pywikibot-Documentation, 10good first task: Move Pywikibot PAWS tutorial out of Jupyter notebook onto wiki - https://phabricator.wikimedia.org/T342397 (10Enag2000) Moved page out of drafts. New link is [[ https://www.mediawiki.org/wiki/Manual:Pywikibot/Pywikibot_an...
[18:36:03] <wmcs-alerts>	 (InstanceDown) firing: Project toolsbeta instance toolsbeta-bastion-6 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[19:03:45] <wmcs-alerts>	 (ProbeDown) firing: Service tools-k8s-haproxy-4:30000 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-4:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[19:08:45] <wmcs-alerts>	 (ProbeDown) resolved: Service tools-k8s-haproxy-4:30000 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-4:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[19:16:03] <wmcs-alerts>	 (InstanceDown) resolved: Project toolsbeta instance toolsbeta-bastion-6 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[19:28:29] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate wahrani from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320133 (10fnegri) @wahrani your Kubernetes jobs are still active, as you can verify with the command `toolforge jobs list`. Only the GridEngine part of Toolforge was disabled for...