[00:43:57] PROBLEM - SSH on wdqs2001.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [03:46:25] RECOVERY - SSH on wdqs2001.mgmt is OK: SSH OK - OpenSSH_7.0 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [07:00:05] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20210613T0700) [11:21:56] (03Abandoned) 10Urbanecm: eswiki: Add Abuse Filter blocks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/699489 (owner: 10ZAR) [11:49:20] urbanecm: can you disable that account [11:49:34] i think it already should be disabled [11:49:47] I thought we'd sweeped them up before [11:49:52] It's an old email [11:52:54] RhinosF1, urbanecm: the account shows as disabled in ldap [11:53:08] but it might have alive sessions in gerrit or something like that [11:53:19] majavah: how old is the account? [11:54:05] RhinosF1: created on 2021-06-12 13:56:52Z [11:54:35] majavah: lovely. I wonder if there's a way to ban emails. He's been using that since day 1. [11:54:50] I could guess the IP too and have a 50% chance at being right [11:58:05] https://meta.wikimedia.org/wiki/Email_blacklist does exist [11:58:47] Might be worth it [11:58:50] urbanecm: ^ [12:00:37] useless as it's public [12:13:05] ma: yeah that's the hardest part with these lists [13:37:21] (03CR) 10Urbanecm: [C: 04-1] enwiki: Remove 'collectionsaveascommunitypage' from the 'autoconfirmed' and 'confirmed' user groups (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/698041 (https://phabricator.wikimedia.org/T283523) (owner: 10MarcoAurelio) [14:47:14] (03PS1) 10Mbch331: Remove special configurations for Dagbani in Wikibase code [mediawiki-config] - 10https://gerrit.wikimedia.org/r/699540 (https://phabricator.wikimedia.org/T283168) [14:49:18] (03CR) 10Amire80: Remove special configurations for Dagbani in Wikibase code (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/699540 (https://phabricator.wikimedia.org/T283168) (owner: 10Mbch331) [14:54:58] (03PS2) 10Mbch331: Remove special configurations for Dagbani in Wikibase code [mediawiki-config] - 10https://gerrit.wikimedia.org/r/699540 (https://phabricator.wikimedia.org/T283168) [14:56:51] (03CR) 10Mbch331: Remove special configurations for Dagbani in Wikibase code (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/699540 (https://phabricator.wikimedia.org/T283168) (owner: 10Mbch331) [16:21:30] (03CR) 10Zabe: [C: 03+1] Update $wgNamespacesToBeSearchedDefault for wikimania [mediawiki-config] - 10https://gerrit.wikimedia.org/r/699290 (https://phabricator.wikimedia.org/T284793) (owner: 10DannyS712) [17:20:38] (03PS2) 10Sahilgrewalhere: selenium: Upgrade WebdriverIO to v7 [phabricator/deployment] (wmf/stable) - 10https://gerrit.wikimedia.org/r/697069 (https://phabricator.wikimedia.org/T274579) [17:47:07] RECOVERY - BGP status on cr3-eqsin is OK: BGP OK - up: 317, down: 3, shutdown: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BGP_status [18:37:26] (03PS7) 10MarcoAurelio: enwiki: Remove 'collectionsaveascommunitypage' from the 'autoconfirmed' and 'confirmed' user groups [mediawiki-config] - 10https://gerrit.wikimedia.org/r/698041 (https://phabricator.wikimedia.org/T283523) [18:37:50] (03PS8) 10MarcoAurelio: enwiki: Remove 'collectionsaveascommunitypage' from the 'autoconfirmed' user group [mediawiki-config] - 10https://gerrit.wikimedia.org/r/698041 (https://phabricator.wikimedia.org/T283523) [18:38:19] (03CR) 10MarcoAurelio: enwiki: Remove 'collectionsaveascommunitypage' from the 'autoconfirmed' user group (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/698041 (https://phabricator.wikimedia.org/T283523) (owner: 10MarcoAurelio) [19:16:39] PROBLEM - Prometheus prometheus1004/global -or a Prometheus it scrapes- was restarted: beware possible monitoring artifacts on prometheus1004 is CRITICAL: instance=127.0.0.1 job=prometheus https://wikitech.wikimedia.org/wiki/Prometheus%23Prometheus_was_restarted https://grafana.wikimedia.org/d/000000271/prometheus-stats?var-datasource=eqiad+prometheus/global [19:40:34] (03CR) 10Zabe: [C: 03+1] enwiki: Remove 'collectionsaveascommunitypage' from the 'autoconfirmed' user group [mediawiki-config] - 10https://gerrit.wikimedia.org/r/698041 (https://phabricator.wikimedia.org/T283523) (owner: 10MarcoAurelio) [19:40:39] RECOVERY - Prometheus prometheus1004/global -or a Prometheus it scrapes- was restarted: beware possible monitoring artifacts on prometheus1004 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Prometheus%23Prometheus_was_restarted https://grafana.wikimedia.org/d/000000271/prometheus-stats?var-datasource=eqiad+prometheus/global [19:59:21] PROBLEM - SSH on wdqs2001.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [21:00:07] RECOVERY - SSH on wdqs2001.mgmt is OK: SSH OK - OpenSSH_7.0 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [21:52:23] (03CR) 10Zabe: "this is a dupe of https://gerrit.wikimedia.org/r/c/operations/puppet/+/692370" [puppet] - 10https://gerrit.wikimedia.org/r/698649 (owner: 10Jforrester) [23:27:35] PROBLEM - SSH on contint2001.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook