[07:09:56] (VarnishTrafficDrop) firing: (2) 62% GET drop in text@codfw during the past 30 minutes - https://alerts.wikimedia.org [07:14:56] (VarnishTrafficDrop) resolved: (4) 68% GET drop in text@codfw during the past 30 minutes - https://alerts.wikimedia.org [07:37:47] godog: the alert seems to be working fine ^ [07:38:22] godog: it would be nice to include a link to the dashboard in the irc notification though [07:44:47] 10Traffic, 10SRE, 10ops-eqiad: cp1087 down with hardware issues - https://phabricator.wikimedia.org/T278729 (10ema) [08:04:13] ema: \o/ yeah that and/or the runbook [09:10:01] 10Traffic, 10netops, 10SRE, 10User-jbond: varnish filtering: should we automatically update public_cloud_nets - https://phabricator.wikimedia.org/T270391 (10jbond) > I personally don't see Netbox as the right place for those, at least as prefixes Ack, I think in thtat case the best way forward for now is t... [09:38:13] 10netops, 10Data-Persistence-Backup, 10SRE: Understand (and mitigate) the backup speed differences between backup1002->backup2002 and backup2002->backup1002 - https://phabricator.wikimedia.org/T274234 (10cmooney) Thanks for the info @jcrespo that should help. I did a lot of tests yesterday in relation to th... [10:12:43] 10Traffic, 10netops, 10SRE, 10User-jbond: varnish filtering: should we automatically update public_cloud_nets - https://phabricator.wikimedia.org/T270391 (10ayounsi) I still think filtering public clouds on their ASN (with MaxMind DB) is the most sustainable path until T270618. Having to maintain multiple... [12:11:12] 10Traffic, 10SRE, 10User-ArielGlenn, 10User-MoritzMuehlenhoff, 10User-jbond: Migrate to nginx-light - https://phabricator.wikimedia.org/T164456 (10jbond) I have updated puppetdb to use nginx-light. further It should now be fairly simple to switch other services to nginx-light. 1. Add `profile::nginx` [... [16:10:37] 10Traffic, 10SRE, 10decommission-hardware: decommission cescout1001.eqiad.wmnet - https://phabricator.wikimedia.org/T275696 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by sukhe@cumin1001 for hosts: `cescout1001.eqiad.wmnet` - cescout1001.eqiad.wmnet (**PASS**) - Downtimed host on Icing... [16:30:52] 10Traffic, 10SRE, 10decommission-hardware, 10Patch-For-Review: decommission cescout1001.eqiad.wmnet - https://phabricator.wikimedia.org/T275696 (10ssingh) Puppet roles/profiles/hiera configurations have not been removed as they will be needed for the cescout VM that will be provisioned later. [16:42:45] 10netops, 10Data-Persistence-Backup, 10SRE, 10bacula: Understand (and mitigate) the backup speed differences between backup1002->backup2002 and backup2002->backup1002 - https://phabricator.wikimedia.org/T274234 (10jcrespo) [19:37:23] 10Traffic, 10SRE, 10vm-requests, 10Patch-For-Review: Please create two Ganeti VMs for Wikidough in esams - https://phabricator.wikimedia.org/T283852 (10Dzahn) a:03Dzahn [19:37:32] 10Traffic, 10SRE, 10vm-requests, 10Patch-For-Review: Please create two Ganeti VMs for Wikidough in esams - https://phabricator.wikimedia.org/T283852 (10Dzahn) p:05Medium→03High [20:58:49] 10Traffic, 10SRE, 10vm-requests, 10Patch-For-Review: Please create two Ganeti VMs for Wikidough in esams - https://phabricator.wikimedia.org/T283852 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: `doh3001.wikimedia.org` - doh3001.wikimedia.org (**WARN**) -... [22:41:42] 10Traffic, 10SRE, 10vm-requests, 10Patch-For-Review: Please create two Ganeti VMs for Wikidough in esams - https://phabricator.wikimedia.org/T283852 (10Dzahn) We ran out of resources on esams ganeti (disk). Creating machines with just 10G instead after discussion. [23:05:09] 10Traffic, 10SRE, 10vm-requests, 10Patch-For-Review: Please create two Ganeti VMs for Wikidough in esams - https://phabricator.wikimedia.org/T283852 (10Dzahn) VMs have been created. doh3001 and doh3002.wikimedia.org are up and running with "insetup". Feel free to use them now. They have about 1.8GB used... [23:06:52] 10Traffic, 10SRE, 10vm-requests, 10Patch-For-Review: Please create two Ganeti VMs for Wikidough in esams - https://phabricator.wikimedia.org/T283852 (10Dzahn) ` [doh3001:~] $ df -h Filesystem Size Used Avail Use% Mounted on .. /dev/vda1 8.9G 1.6G 6.8G 19% / [doh3002:~] $ df -h Filesystem... [23:10:34] 10Traffic, 10SRE, 10vm-requests: Please create two Ganeti VMs for Wikidough in esams - https://phabricator.wikimedia.org/T283852 (10Dzahn) 05Open→03Resolved [23:11:09] 10Traffic, 10SRE, 10vm-requests: Please create two Ganeti VMs for Wikidough in esams - https://phabricator.wikimedia.org/T283852 (10Dzahn)