[09:04:47] 10Traffic, 10Data-Engineering, 10Event-Platform Value Stream, 10SRE, and 2 others: Incident: 2022-03-4 Banner sampling leading to a relatively wide site outage (mostly esams) - https://phabricator.wikimedia.org/T303036 (10Marostegui) @lmata what should we do with this follow up task? [09:07:25] 10Traffic, 10Phabricator, 10SRE, 10Wikimedia-Incident: Phabricator was logging out users repeatedly (2022-08-26) - https://phabricator.wikimedia.org/T316337 (10Marostegui) What should we do with this task? Anything left? [09:08:18] 10Traffic, 10SRE, 10affects-Kiwix-and-openZIM: HTTP 500 against api.php?action=parse API on tr.wikipedia.org - https://phabricator.wikimedia.org/T317011 (10Marostegui) 05Open→03Resolved a:03Marostegui I am going to tentatively close this as fixed per T317011#8212217. Please reopen if it is not the case. [09:09:08] 10Traffic, 10SRE, 10affects-Kiwix-and-openZIM: HTTP 500 against api.php?action=parse API on tr.wikipedia.org - https://phabricator.wikimedia.org/T317011 (10Marostegui) a:05Marostegui→03None [09:18:46] 10Traffic, 10Phabricator, 10SRE, 10Wikimedia-Incident: Phabricator was logging out users repeatedly (2022-08-26) - https://phabricator.wikimedia.org/T316337 (10jcrespo) As soon as I finish the wikitech description I intend to resolve it. [09:40:26] 10Traffic, 10Phabricator, 10SRE, 10Wikimedia-Incident: Phabricator was logging out users repeatedly (2022-08-26) - https://phabricator.wikimedia.org/T316337 (10jcrespo) [09:40:36] 10Traffic, 10SRE: strip non session cookies before cache lookup in ATS - https://phabricator.wikimedia.org/T316338 (10jcrespo) [09:40:54] 10Traffic, 10Phabricator, 10SRE, 10Wikimedia-Incident: Phabricator was logging out users repeatedly (2022-08-26) - https://phabricator.wikimedia.org/T316337 (10jcrespo) 05Open→03Resolved a:03Vgutierrez @hashar @Vgutierrez Please review my summary of the incident at: https://wikitech.wikimedia.org/wik... [09:41:02] 10Traffic, 10Data-Engineering, 10Event-Platform Value Stream, 10SRE, and 2 others: Incident: 2022-03-4 Banner sampling leading to a relatively wide site outage (mostly esams) - https://phabricator.wikimedia.org/T303036 (10BTullis) I'm not sure that there's much more to do, is there? From a technical perspe... [09:45:28] 10Traffic, 10SRE, 10Patch-For-Review: Deprecate and disable port 80 for one-off sites under canonical domains - https://phabricator.wikimedia.org/T238720 (10Vgutierrez) [09:48:04] 10Traffic, 10SRE, 10Patch-For-Review: Deprecate and disable port 80 for one-off sites under canonical domains - https://phabricator.wikimedia.org/T238720 (10Vgutierrez) [09:48:45] 10Traffic, 10SRE, 10Patch-For-Review: Deprecate and disable port 80 for one-off sites under canonical domains - https://phabricator.wikimedia.org/T238720 (10Vgutierrez) [09:53:13] 10Traffic, 10SRE, 10Patch-For-Review: Deprecate and disable port 80 for one-off sites under canonical domains - https://phabricator.wikimedia.org/T238720 (10hashar) For Gerrit, I have made the announcement on [[ https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/thread/WGIDWKB4YN3DM7K... [10:31:52] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Move WMCS servers to 1 NIC - https://phabricator.wikimedia.org/T319184 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1001 for host cloudvirt1043.eqiad.wmnet with O... [10:48:46] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Move WMCS servers to 1 NIC - https://phabricator.wikimedia.org/T319184 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1001 for host cloudvirt1043.eqiad.wmnet with OS bu... [10:49:13] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Move WMCS servers to 1 NIC - https://phabricator.wikimedia.org/T319184 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1001 for host cloudvirt1043.eqiad.wmnet with O... [11:30:54] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Move WMCS servers to 1 NIC - https://phabricator.wikimedia.org/T319184 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1001 for host cloudvirt1043.eqiad.wmnet with OS bu... [12:35:48] 10Traffic, 10Data-Engineering, 10Event-Platform Value Stream, 10SRE, and 2 others: Incident: 2022-03-4 Banner sampling leading to a relatively wide site outage (mostly esams) - https://phabricator.wikimedia.org/T303036 (10lmata) 05Open→03Resolved a:03lmata Thank you @BTullis for T303036#8423773. I th... [13:31:11] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Move WMCS servers to 1 NIC - https://phabricator.wikimedia.org/T319184 (10aborrero) [15:27:27] 10Traffic, 10SRE, 10affects-Kiwix-and-openZIM: HTTP 500 against api.php?action=parse API on tr.wikipedia.org - https://phabricator.wikimedia.org/T317011 (10Kelson) The reported bug seems indeed to have "vanished". Thank you for the good work. [17:36:32] 10Wikimedia-Apache-configuration, 10SRE, 10Wikibase Product Platform, 10serviceops: Incorrect handling of ETags taking precedence over timestamps in conditional requests - https://phabricator.wikimedia.org/T320241 (10jijiki) @Silvan_WMDE sorry for not replying sooner, I will take a look at this when I find... [17:36:47] 10Wikimedia-Apache-configuration, 10SRE, 10Wikibase Product Platform, 10serviceops: Incorrect handling of ETags taking precedence over timestamps in conditional requests - https://phabricator.wikimedia.org/T320241 (10jijiki) a:03jijiki [18:57:26] 10Traffic: haproxy: work on systemd unit hardening (cp hosts) - https://phabricator.wikimedia.org/T323944 (10ssingh) [18:57:40] 10Traffic: haproxy: work on systemd unit hardening (cp hosts) - https://phabricator.wikimedia.org/T323944 (10ssingh) p:05Triage→03Medium [19:41:35] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install/decom eqsin: unified decommission task - https://phabricator.wikimedia.org/T323830 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by sukhe@cumin2002 for hosts: `cp[5002,5007].eqsin.wmnet` - cp5002.eqsin.w... [19:44:13] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install/decom eqsin: unified decommission task - https://phabricator.wikimedia.org/T323830 (10ssingh) [21:23:14] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install/decom eqsin: unified decommission task - https://phabricator.wikimedia.org/T323830 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by sukhe@cumin2002 for hosts: `cp[5003,5008].eqsin.wmnet` - cp5003.eqsin.w... [21:25:12] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install/decom eqsin: unified decommission task - https://phabricator.wikimedia.org/T323830 (10ssingh) [23:23:18] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install/decom eqsin: unified decommission task - https://phabricator.wikimedia.org/T323830 (10ssingh)