[00:15:54] (03CR) 10Jdlrobson: [C: 03+2] Update web_ab_test_enrollment group property [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/735993 (https://phabricator.wikimedia.org/T292587) (owner: 10Clare Ming) [00:17:37] (03Merged) 10jenkins-bot: Update web_ab_test_enrollment group property [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/735993 (https://phabricator.wikimedia.org/T292587) (owner: 10Clare Ming) [03:09:50] 10Analytics-Radar, 10Data-Engineering, 10Event-Platform, 10The-Wikipedia-Library: Store page-links-change data in a database table and make available through a Special page - https://phabricator.wikimedia.org/T221397 (10Harej) [03:14:28] 10Analytics-Radar, 10Data-Engineering, 10Event-Platform, 10Internet-Archive, 10The-Wikipedia-Library: Store page-links-change data in a database table and make available through a Special page - https://phabricator.wikimedia.org/T221397 (10Harej) [03:36:51] 10Analytics-Radar, 10Analytics-Wikistats: Feedback on Wikistats 2 new edits pages - https://phabricator.wikimedia.org/T210306 (10Harej) [14:54:35] 10Analytics, 10Data-Engineering, 10Event-Platform, 10Platform Engineering, 10tech-decision-forum: MediaWiki Events as Source of Truth - Decision Statement Overview - https://phabricator.wikimedia.org/T291120 (10fkaelin) A lot of thoughtful comments have been made, and like others I find it difficult to s... [15:49:42] deployment train boarding, I only see one sanitize allow list change, all aboard! (I'll wait until after meetings) [15:52:44] milimetric: I haven't anything to deploy, thanks. :-) [16:02:08] 10Analytics, 10DBA, 10Event-Platform, 10WMF-Architecture-Team: Consistent MediaWiki state change events | MediaWiki events as source of truth - https://phabricator.wikimedia.org/T120242 (10Nuria) New developments in this are of interest: watermark change data capture framework from netflix that aims to do... [16:02:26] btullis: o/ is it ok if I give the green light to Rob to reimage an-db100[1,2] within the analytics vlan? [16:36:10] It's ottomata's baby really, so I think I'd hold off until tomorrow, just in case. [16:44:08] sure sure I wanted to avoid slowdowns to Andrew since the hosts are basically empty now [16:44:26] so it would be safe to reimage them ( and Robh has time today) [16:45:19] OK, you've convinved me. It's a 👍 from me then. [16:49:55] done :) [16:50:49] 😬 Thanks. [16:51:57] I'll get the blame if Andrew will not be happy :D [16:52:25] 10Analytics, 10Analytics-Kanban, 10Data-Engineering, 10Data-Engineering-Kanban, and 3 others: Migrate analytics cluster alerts from Icinga to AlertManager - https://phabricator.wikimedia.org/T293399 (10BTullis) [16:54:06] I'm sure there will be enough to go around :-) [17:07:02] I think that didn't sound the way I'd intended. I made it sound like he'd be really cross, which wasn't what I meant at all. [17:17:56] nono I completely get it :) [17:18:02] *got it [17:58:40] 10Analytics-Clusters, 10DC-Ops, 10Data-Engineering, 10SRE, 10ops-eqiad: Q1:(Need By: ASAP) rack/setup/install an-db100[12].eqiad.wmnet - https://phabricator.wikimedia.org/T289632 (10RobH) 05Open→03In progress irc update chatted with @elukey and these do indeed need to shift to analtyics vlan. [18:16:02] 10Analytics-Clusters, 10DC-Ops, 10Data-Engineering, 10SRE, 10ops-eqiad: Q1:(Need By: ASAP) rack/setup/install an-db100[12].eqiad.wmnet - https://phabricator.wikimedia.org/T289632 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by robh@cumin1001 for hosts: `an-db1001.eqiad.wmnet` - an-db1... [18:32:27] 10Analytics-Clusters, 10DC-Ops, 10Data-Engineering, 10SRE, 10ops-eqiad: Q1:(Need By: ASAP) rack/setup/install an-db100[12].eqiad.wmnet - https://phabricator.wikimedia.org/T289632 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by robh@cumin1001 for host an-db1001.eqiad.wmnet with... [19:08:46] 10Analytics-Clusters, 10DC-Ops, 10Data-Engineering, 10SRE, 10ops-eqiad: Q1:(Need By: ASAP) rack/setup/install an-db100[12].eqiad.wmnet - https://phabricator.wikimedia.org/T289632 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by robh@cumin1001 for host an-db1001.eqiad.wmnet with OS b... [19:35:57] 10Analytics-Clusters, 10DC-Ops, 10Data-Engineering, 10SRE, 10ops-eqiad: Q1:(Need By: ASAP) rack/setup/install an-db100[12].eqiad.wmnet - https://phabricator.wikimedia.org/T289632 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by robh@cumin1001 for hosts: `an-db1002.eqiad.wmnet` - an-db1... [20:13:47] hey quick question here, in the Hive webrequest and derived tables, dt is described as "Timestamp at cache". That is the time when the initial request from the client is processed by the front-end cache, correct? and so that value would be completely unaffected by any network slowness in transmitting the response to the client, no? [20:13:55] mgerlach ^ [20:28:33] 10Analytics-Clusters, 10DC-Ops, 10Data-Engineering, 10SRE, 10ops-eqiad: Q1:(Need By: ASAP) rack/setup/install an-db100[12].eqiad.wmnet - https://phabricator.wikimedia.org/T289632 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by robh@cumin1001 for host an-db1001.eqiad.wmnet with... [20:52:47] 10Analytics-Clusters, 10DC-Ops, 10Data-Engineering, 10SRE, 10ops-eqiad: Q1:(Need By: ASAP) rack/setup/install an-db100[12].eqiad.wmnet - https://phabricator.wikimedia.org/T289632 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by robh@cumin1001 for host an-db1001.eqiad.wmnet with OS b... [20:55:45] 10Analytics-Radar, 10SRE, 10Traffic, 10WMF-General-or-Unknown, and 2 others: Requests for /static get an invalid WMF-Last-Access cookie for wikipedia.org on non-Wikipedia requests - https://phabricator.wikimedia.org/T261803 (10Krinkle) [21:04:05] 10Analytics-Clusters, 10DC-Ops, 10Data-Engineering, 10SRE, 10ops-eqiad: Q1:(Need By: ASAP) rack/setup/install an-db100[12].eqiad.wmnet - https://phabricator.wikimedia.org/T289632 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by robh@cumin1001 for host an-db1002.eqiad.wmnet with... [21:05:49] 10Analytics-Radar, 10SRE, 10Traffic, 10WMF-General-or-Unknown, and 2 others: Requests for /static get an invalid WMF-Last-Access cookie for wikipedia.org on non-Wikipedia requests - https://phabricator.wikimedia.org/T261803 (10Krinkle) The URLs that use w/extensions, w/skins, and w/resources are also used... [21:32:37] 10Analytics-Clusters, 10DC-Ops, 10Data-Engineering, 10SRE, 10ops-eqiad: Q1:(Need By: ASAP) rack/setup/install an-db100[12].eqiad.wmnet - https://phabricator.wikimedia.org/T289632 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by robh@cumin1001 for host an-db1002.eqiad.wmnet with OS b... [21:50:43] 10Analytics-Clusters, 10DC-Ops, 10Data-Engineering, 10SRE, 10ops-eqiad: Q1:(Need By: ASAP) rack/setup/install an-db100[12].eqiad.wmnet - https://phabricator.wikimedia.org/T289632 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by robh@cumin1001 for host an-db1002.eqiad.wmnet with... [22:14:16] 10Analytics-Clusters, 10DC-Ops, 10Data-Engineering, 10SRE, 10ops-eqiad: Q1:(Need By: ASAP) rack/setup/install an-db100[12].eqiad.wmnet - https://phabricator.wikimedia.org/T289632 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by robh@cumin1001 for host an-db1002.eqiad.wmnet with OS b... [22:15:40] 10Analytics-Clusters, 10DC-Ops, 10Data-Engineering, 10SRE, 10ops-eqiad: Q1:(Need By: ASAP) rack/setup/install an-db100[12].eqiad.wmnet - https://phabricator.wikimedia.org/T289632 (10RobH) 05In progress→03Resolved >>! In T289632#7474421, @elukey wrote: > Hi everybody, I noticed that the two hosts are... [22:15:55] 10Analytics-Clusters, 10DC-Ops, 10Data-Engineering, 10SRE, 10ops-eqiad: Q1:(Need By: ASAP) rack/setup/install an-db100[12].eqiad.wmnet - https://phabricator.wikimedia.org/T289632 (10RobH) a:05Cmjohnson→03RobH