[00:09:03] (TfInfraTestDestroyFailed) resolved: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [00:10:03] (InstanceDown) firing: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:15:03] (InstanceDown) resolved: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:19:17] 10Tool-global-search: Extend the OAuth duration for global-search - https://phabricator.wikimedia.org/T350521 (10Billinghurst) [01:09:24] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [01:56:58] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.drain_node (348643) [03:28:00] (PuppetConstantChange) firing: Puppet performing a change on every puppet run on cloudcumin1001:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [04:09:24] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [04:24:00] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643) [07:09:24] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [07:28:00] (PuppetConstantChange) firing: Puppet performing a change on every puppet run on cloudcumin1001:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [09:47:07] 10VPS-project-Wikistats, 10Patch-For-Review, 10User-RhinosF1: wikia was renamed to fandom - https://phabricator.wikimedia.org/T221537 (10RhinosF1) Migration is around 1/3rd complete. Found 15k wikis so far. Around 50k left to check. Probably looking at about 2 days to check all. [10:09:24] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [11:28:00] (PuppetConstantChange) firing: Puppet performing a change on every puppet run on cloudcumin1001:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [11:53:14] 10Tools: Move Wiki random tool to toolforge - https://phabricator.wikimedia.org/T283205 (10Ainali) 05Open→03Resolved Fixed! Now available at https://shuffle.toolforge.org/ [12:16:26] (ProbeDown) firing: Service toolserver-proxy-01:443 has failed probes (http_toolserver_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#toolserver-proxy-01:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:21:26] (ProbeDown) resolved: Service toolserver-proxy-01:443 has failed probes (http_toolserver_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#toolserver-proxy-01:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:51:19] 10Tool-nlwikibots: archiver on nlwikibots checks only if linked, not if it is a template. - https://phabricator.wikimedia.org/T350534 (10Akoopal) [12:54:22] 10Tool-nlwikibots: archiver on nlwikibots checks only if linked, not if it is a template. - https://phabricator.wikimedia.org/T350534 (10Akoopal) a:03Akoopal On line 476 of archiving.py it calls getReferences. Here we should probably set the flag only_template_inclusion to true. [12:56:13] 10Tool-nlwikibots: archiver on nlwikibots checks only if linked, not if it is a template. - https://phabricator.wikimedia.org/T350534 (10Akoopal) 05Open→03In progress p:05Triage→03Low [13:09:24] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [13:13:36] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.drain_node (348643) [13:31:23] 10Tool-nlwikibots: Prevent archiving topics that have not been resolved yet - https://phabricator.wikimedia.org/T350537 (10FrankGeerlings) [13:38:47] 10Tool-nlwikibots: Prevent archiving topics that have not been resolved yet - https://phabricator.wikimedia.org/T350537 (10FrankGeerlings) 05Open→03Declined p:05Triage→03Low It is deemed to be beyond the scope of the archive bot. I would suggest a dedicated admin's noticeboard bot script. That said, pull... [13:55:34] 10Tool-nlwikibots: archiver on nlwikibots doesn't check on nobots, but crashes if set. - https://phabricator.wikimedia.org/T350536 (10Akoopal) [15:01:30] 10Tool-nlwikibots: archiver on nlwikibots checks only if linked, not if it is a template. - https://phabricator.wikimedia.org/T350534 (10FrankGeerlings) 05In progress→03Resolved Fixed by https://github.com/frankgeerlings/nlwikibots/commit/d94f77544aaa7b2dc07be8940f6e2e4b57c3c51d which is now in production. [15:03:17] 10Tool-nlwikibots: archiver on nlwikibots doesn't check on nobots, but crashes if set. - https://phabricator.wikimedia.org/T350536 (10FrankGeerlings) 05Open→03Resolved a:03FrankGeerlings Fixed by https://github.com/frankgeerlings/nlwikibots/commit/ebce8c95c99f7d619e4e821f7cb18fc46a726761 which is now in pr... [15:05:59] (PuppetConstantChange) firing: Puppet performing a change on every puppet run on cloudcontrol2006-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [15:28:15] (PuppetConstantChange) firing: Puppet performing a change on every puppet run on cloudcumin1001:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [16:00:17] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.drain_node (348643) [16:09:24] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [16:10:00] 10Tool-nlwikibots: archivingbot on nlwikibots should not reference user:Erwin anymore. - https://phabricator.wikimedia.org/T350540 (10Akoopal) [16:10:30] 10Tool-nlwikibots: archivingbot on nlwikibots should not reference user:Erwin anymore. - https://phabricator.wikimedia.org/T350540 (10Akoopal) p:05Triage→03Low [16:19:54] 10Tool-nlwikibots: archivingbot on nlwikibots should not reference user:Erwin anymore. - https://phabricator.wikimedia.org/T350540 (10Akoopal) The bot is changed by Frank that it can use multiple templates, we have choosen the simple {{Archiveer}} This gives an issue, the bot simply searches for the template on... [16:39:18] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643) [19:05:59] (PuppetConstantChange) firing: Puppet performing a change on every puppet run on cloudcontrol2006-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [19:14:24] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [19:28:15] (PuppetConstantChange) firing: Puppet performing a change on every puppet run on cloudcumin1001:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [20:12:52] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.drain_node (348643) [22:14:24] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [22:36:25] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643) [23:05:38] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.drain_node (348643) [23:05:59] (PuppetConstantChange) firing: Puppet performing a change on every puppet run on cloudcontrol2006-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [23:28:15] (PuppetConstantChange) firing: Puppet performing a change on every puppet run on cloudcumin1001:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange