[09:01:00] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 4 others: cloudservices2004-dev: reimage into new network setup - https://phabricator.wikimedia.org/T338778 (10aborrero) [09:36:32] 10Traffic: port 80 paging on scheduled single host maintenance in text@esams - https://phabricator.wikimedia.org/T339898 (10Vgutierrez) [09:36:45] 10Traffic: port 80 paging on scheduled single host maintenance in text@esams - https://phabricator.wikimedia.org/T339898 (10Vgutierrez) p:05Triage→03High [09:42:49] 10Traffic: port 80 paging on scheduled single host maintenance in text@esams - https://phabricator.wikimedia.org/T339898 (10Fabfur) My log message contained a typo: should've been "rebooting cp3050 and cp3051 for kernel upgrade (T335835)" [10:09:14] 10Traffic, 10DNS, 10Patch-For-Review: add wikimedia.social to WMF DNS (was: Update DNS records for mastodon.wikimedia.org) - https://phabricator.wikimedia.org/T337586 (10taavi) 05Resolved→03Open Name servers for the `wikimedia.social` domain still need to be updated to `ns[0-2].wikimedia.org`: `lang=shel... [10:15:39] 10Traffic, 10SRE: port 80 paging on scheduled single host maintenance in text@esams - https://phabricator.wikimedia.org/T339898 (10Vgutierrez) [[ https://grafana.wikimedia.org/goto/0JYX92u4z?orgId=1 | During the issue ]] text@esams never went higher than ~400 rps on port 80 per instance: {F37110208} [[ https:... [10:18:57] 10Traffic, 10SRE: port 80 paging on scheduled single host maintenance in text@esams - https://phabricator.wikimedia.org/T339898 (10Vgutierrez) pybal on lvs3005 and lvs3007 didn't report any healthcheck failures during the issue (besides the expected one for cp3050/cp3051 under maintenance at that moment) [10:31:02] 10Traffic, 10SRE: port 80 paging on scheduled single host maintenance in text@esams - https://phabricator.wikimedia.org/T339898 (10Vgutierrez) for both IPv4 and IPv6 the alert reports "context deadline exceeded": ` target=http://[91.198.174.192]:80/wiki/Special:BlankPage msg="Error for HTTP request" err="Get \... [11:16:45] 10Traffic, 10Data-Engineering: Webrequest x_analtics `wprov` value is incorrectly formatted - https://phabricator.wikimedia.org/T339910 (10JAllemandou) [11:31:00] 10Traffic, 10DNS, 10Patch-For-Review: add wikimedia.social to WMF DNS (was: Update DNS records for mastodon.wikimedia.org) - https://phabricator.wikimedia.org/T337586 (10ssingh) Yes, checking this with Chuck in private email as there was a notification sent to dns-admin@ and this ticket is public. Will updat... [11:37:49] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 3 others: Move cloud vps ns-recursor IPs to host/row-independent addressing - https://phabricator.wikimedia.org/T307357 (10aborrero) [12:10:16] 10Traffic, 10Data-Engineering, 10SRE: Webrequest x_analtics `wprov` value is incorrectly formatted - https://phabricator.wikimedia.org/T339910 (10JAllemandou) [12:12:21] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 4 others: cloudservices2004-dev: reimage into new network setup - https://phabricator.wikimedia.org/T338778 (10aborrero) [12:12:37] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 4 others: cloudservices2004-dev: reimage into new network setup - https://phabricator.wikimedia.org/T338778 (10aborrero) 05In progress→03Resolved [12:12:47] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 3 others: Move cloud vps ns-recursor IPs to host/row-independent addressing - https://phabricator.wikimedia.org/T307357 (10aborrero) [12:18:49] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 4 others: cloudservices2004-dev: reimage into new network setup - https://phabricator.wikimedia.org/T338778 (10aborrero) Run: ` update domains set master="185.15.57.25:5354 185.15.57.26:5354 172.20.5.8:5354 172.20.5.9:5354"; ` In both servers. [12:32:14] 10Traffic, 10SRE: port 80 paging on scheduled single host maintenance in text@esams - https://phabricator.wikimedia.org/T339898 (10Vgutierrez) pybal timeout for ProxyFetch is set to 5s while prometheus blackbox http probe timeouts at 3s. this could explain the gap mentioned on https://phabricator.wikimedia.org... [15:53:44] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 3 others: Move cloud vps ns-recursor IPs to host/row-independent addressing - https://phabricator.wikimedia.org/T307357 (10aborrero) [16:16:55] 10Domains, 10SRE: Mark Monitor administration panel (redirects for wikimedia.pl) - https://phabricator.wikimedia.org/T333827 (10BCornwall) [17:52:22] 10Traffic, 10DNS, 10SRE, 10Patch-For-Review: Additional DNS entries for WikiLearn - https://phabricator.wikimedia.org/T339942 (10BCornwall) 05Open→03In progress p:05Triage→03Low a:03ssingh [17:53:48] 10Traffic, 10DNS, 10Patch-For-Review: Additional DNS entries for WikiLearn - https://phabricator.wikimedia.org/T339942 (10BCornwall) [17:57:44] 10Traffic, 10DNS, 10Patch-For-Review: Additional DNS entries for WikiLearn - https://phabricator.wikimedia.org/T339942 (10ssingh) 05In progress→03Resolved DNS records updated. Please wait for at least an hour to let the current records expiry (if they are cached anywhere). Thanks!