[04:02:27] 06Data-Engineering, 06Data-Engineering-Radar, 10CampaignEvents, 06Connection-Team, and 17 others: Add namespace descriptions for Special:NamespaceInfo in WMF-deployed extensions - https://phabricator.wikimedia.org/T373070#11301184 (10phuedx) [05:29:01] 06Data-Engineering, 06Experimentation Lab, 10Experimentation Lab Roadmap, 10MediaWiki-extensions-EventLogging, 07Epic: [EPIC] Deprecate mw.eventLog.logEvent() - https://phabricator.wikimedia.org/T317874#11301280 (10phuedx) [05:32:11] 06Data-Engineering, 06Experimentation Lab, 10Experimentation Lab Roadmap, 10MediaWiki-extensions-EventLogging, 07Epic: [EPIC] Deprecate EventLogging::schemaValidate() - https://phabricator.wikimedia.org/T317793#11301284 (10phuedx) [05:33:14] 06Data-Engineering, 06Experimentation Lab, 10Experimentation Lab Roadmap, 10MediaWiki-extensions-EventLogging, 07Epic: [EPIC] Deprecate EventLogging::schemaValidate() - https://phabricator.wikimedia.org/T317793#11301286 (10phuedx) See T351879#9437753 for notes on localising validation errors thrown by ju... [05:35:08] 06Data-Engineering, 06Data-Engineering-Radar, 06Experimentation Lab, 10MediaWiki-extensions-EventLogging, and 2 others: Deprecate and remove MetricsClient#dispatch() - https://phabricator.wikimedia.org/T352969#11301289 (10phuedx) [05:35:13] 06Data-Engineering, 06Experimentation Lab, 10Experimentation Lab Roadmap, 06Growth-Team, and 4 others: [EPIC] Deprecate EventLogging::logEvent() - https://phabricator.wikimedia.org/T318263#11301290 (10phuedx) [07:00:54] 06Data-Engineering, 06Data-Engineering-Icebox, 10MediaWiki-extensions-EventLogging, 13Patch-Needs-Improvement: Deprecate/delete the mw.eventLog.Schema class - https://phabricator.wikimedia.org/T305491#11301388 (10phuedx) Tempus fugit! There's a mention of it in the Popups/Page Previews codebase in the ins... [07:34:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [07:44:17] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [07:50:23] (03CR) 10Joal: [C:03+1] "LGTM!" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1198152 (https://phabricator.wikimedia.org/T401022) (owner: 10Xcollazo) [07:54:17] RESOLVED: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [08:35:17] 06Data-Engineering, 06Data-Engineering-Icebox, 10MediaWiki-extensions-EventLogging: Deprecate/delete the mw.eventLog.Schema class - https://phabricator.wikimedia.org/T305491#11301647 (10phuedx) [08:55:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [09:00:17] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [09:15:02] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Create a new gitlab repository for use with dbt - https://phabricator.wikimedia.org/T406765#11301822 (10JMonton-WMF) Yesterday during a meeting, @amastilovic suggested `dbt-models` as name, I believe we agreed it is a good name. Is everyone else ok wit... [09:16:08] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Create a new gitlab repository for use with dbt - https://phabricator.wikimedia.org/T406765#11301823 (10BTullis) >>! In T406765#11301821, @JMonton-WMF wrote: > Yesterday during a meeting, @amastilovic suggested `dbt-models` as name, I believe we agreed... [09:25:17] RESOLVED: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [09:39:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [09:44:17] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [10:14:17] RESOLVED: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [10:18:17] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.10.17 - 2025.11.07), 07OKR-Work: Add dbt related packages to conda-analytics - https://phabricator.wikimedia.org/T406766#11302061 (10BTullis) I'm deploying version 0.0.39 to production now with: ` btullis@cumin1003:~$ sudo... [10:24:36] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE, 06Movement-Insights, 07Epic: Explore the use of dbt-core and appropriate adapters in the data-platform environment - https://phabricator.wikimedia.org/T406764#11302069 (10JAllemandou) Pasting here the [[ https://wikimedia.sla... [10:38:01] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Create a new gitlab repository for use with dbt - https://phabricator.wikimedia.org/T406765#11302130 (10JAllemandou) >>! In T406765#11301821, @JMonton-WMF wrote: > Yesterday during a meeting, @amastilovic suggested `dbt-models` as name, I believe we ag... [10:46:31] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.10.17 - 2025.11.07), 07OKR-Work: Add dbt related packages to conda-analytics - https://phabricator.wikimedia.org/T406766#11302176 (10BTullis) The update was applied successfully. ` conda-analytics was updated: 0.0.38 -> 0.... [10:47:17] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [10:47:48] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.10.17 - 2025.11.07), 07OKR-Work: Add dbt related packages to conda-analytics - https://phabricator.wikimedia.org/T406766#11302181 (10BTullis) 05Open→03Resolved All looks good. ` btullis@stat1010:~$ source conda-ana... [10:50:30] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Platform-SRE (2025.10.17 - 2025.11.07), 07OKR-Work: Set up a working, usable dbt installation on stat boxes - https://phabricator.wikimedia.org/T406634#11302185 (10BTullis) [10:52:17] RESOLVED: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [11:07:17] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [11:22:17] RESOLVED: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [11:56:17] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [12:01:17] RESOLVED: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [12:10:38] (03CR) 10Xcollazo: [C:03+2] Add utility to create SHA256 fingerprints of the files of a particular HDFS folder. [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1198152 (https://phabricator.wikimedia.org/T401022) (owner: 10Xcollazo) [12:16:22] (03CR) 10CI reject: [V:04-1] Add utility to create SHA256 fingerprints of the files of a particular HDFS folder. [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1198152 (https://phabricator.wikimedia.org/T401022) (owner: 10Xcollazo) [12:24:33] (03PS1) 10Joal: Update referer classification patterns [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1198313 (https://phabricator.wikimedia.org/T406531) [12:40:48] 06Data-Engineering: Technical work for SDS1.3.7 Incorporate Edge Signal - https://phabricator.wikimedia.org/T407893#11302658 (10mforns) [12:42:23] 06Data-Engineering: Technical work for SDS1.3.7 Incorporate Edge Signal - https://phabricator.wikimedia.org/T407893#11302670 (10mforns) [12:42:49] 06Data-Engineering: Technical work for SDS1.3.7 Incorporate Edge Signal - https://phabricator.wikimedia.org/T407893#11302674 (10mforns) [13:00:18] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [13:03:53] (03CR) 10Snwachukwu: [C:03+1] Update referer classification patterns [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1198313 (https://phabricator.wikimedia.org/T406531) (owner: 10Joal) [13:05:17] RESOLVED: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [13:09:17] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [13:19:17] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [13:20:52] btullis: dse-k8s-worker2002.codfw.wmnet [13:21:03] > k8s-ingress-dse_30443: Servers dse-k8s-worker2002.codfw.wmnet are marked down but pooled [13:21:06] any ideas on this? [13:21:13] if the server is down, it should not be marked as pooled [13:21:55] sukhe: This has not been on my radar. I'm not aware of the server being down for any reason, but I can check. [13:22:04] sorry, I am going strictly by last here [13:22:21] There is no prod workload on this cluster, so it's not critical, but let's fix it. [13:22:23] thank you and let me know if this is better directed at someone else [13:23:30] The only thing that occurs to me is that it might be related to a failed cluster reboot cookbook. It failed when there were too few nodes to support typha's pod disruption budget of 3. [13:24:17] RESOLVED: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [13:28:53] I don't think it's that, as `kubectl get nodes` looks fine. I've asked around, to see if anyone else is working on it. [13:29:56] thank you [13:30:12] this has been alerting for quite a bit and the only concern is that we try to keep the pybal logs clean on such alerts [13:30:41] Ack, thanks for context. [13:38:21] sukhe: Could you point me to the health check that is failing, please? I can't see anything wrong with the host itself, so I'm interested in why pybal has marked it down. Thanks. [13:39:01] just a sec [13:39:28] sukhe@lvs2014:~$ curl localhost:9090/pools/k8s-ingress-dse_30443 [13:39:28] dse-k8s-worker2001.codfw.wmnet: enabled/down/not pooled [13:39:28] dse-k8s-worker2002.codfw.wmnet: enabled/down/pooled [13:39:39] and [13:39:52] Sep 04 05:22:30 lvs2014 pybal[504134]: [k8s-ingress-dse_30443 IdleConnection] WARN: dse-k8s-worker2002.codfw.wmnet (enabled/down/pooled): Connection to 10.192.42.14:30443 failed. [13:42:12] OK, thanks. So dse-k8s-worker2003 is OK? [13:42:39] btullis: that doesn't seem to be in the pool at least [13:42:57] even if we look at conftool data [13:43:01] conftool-data/node/codfw.yaml [13:43:13] that's in aux-k8s [13:44:35] checking [13:45:49] OK, so I see that we haven't added dse-k8s-worker2003 to `conftool-data/node/codfw.yaml` in the section `codfw:dse-k8s:` [13:45:54] yeah [13:46:45] When were were setting up the similar cluster in eqiad (which has now got a lot of services) we had a similar alert. Ingress didn't respond to health checks. [13:47:02] happy to help debug that [13:47:12] this is not straight forward if you don't do it regulary enough :) [13:47:21] But this was because (if I remember correctly) we didn't have any services actually behind ingress. So we had to install the `echoserver` service before it would respond. [13:47:55] We have deployed an echoserver service to dse-k8s-codfw, but perhaps it is not wired up correctly. [13:48:56] Shall we depool dse-k8s-worker2002 now to clear the alert? [13:51:37] Here is a patch to add the third host to the conftool-data: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1198329 [13:53:32] yeah depooling dse-k8s-worker should help [13:53:46] but the depool threshold might kick in anyway [13:53:57] but yeah let's do that and then take it from there, adding worker2003 should help [13:58:30] This is odd both servers are shown as pooled at the moment. [13:58:35] https://www.irccloud.com/pastebin/xw2e2nM0/ [13:59:04] yeah, hence the pybal alert [13:59:07] that it is pooled but down [13:59:09] Didn't your snippet show that dse-k8s-worker2001 as not pooled? [13:59:46] hmm indeed, 2002 is pooled but down but 2001 is not pooled [13:59:56] looking [14:00:25] sukhe@puppetserver1001:~$ sudo confctl select 'name=dse-k8s-worker2001.codfw.wmnet' get [14:00:26] I've just merged the patch above. Should I run puppet on any lvs servers, or wait for you? [14:00:28] {"dse-k8s-worker2001.codfw.wmnet": {"weight": 1, "pooled": "yes"}, "tags": "dc=codfw,cluster=dse-k8s,service=kubesvc"} [14:01:56] btullis: running [14:02:09] and yeah, something weird is up for sure [14:02:55] we will probably need to start from the ground up and revisit service definitions and all thta [14:03:21] Sep 04 05:23:16 lvs2014 pybal[504134]: [k8s-ingress-dse_30443 IdleConnection] WARN: dse-k8s-worker2001.codfw.wmnet (enabled/down/not pooled): Connection to 10.192.28.20:30443 failed. [14:03:25] 2001 as well [14:03:36] OK, that's fine by me. There is nothing on this cluster yet that uses ingress. [14:04:20] anyway thanks for checking and the rubber duck debugging. I will take a deeper look later [14:04:26] and will check here as required [14:04:31] But, as I say, the `echoserver` service is what we put in place that *should* be listening and responding. Maybe we have botched the configu for that service, somehow. [14:04:35] (03PS2) 10Joal: Update referer classification patterns [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1198313 (https://phabricator.wikimedia.org/T406531) [14:04:56] Cool. I also have to disappear for a while for a school run. Can check back later. [14:05:07] yeah that seems to be the most likely thing but I will first make sure everything is good on the service side [14:05:10] take care <3 [14:05:50] Cheers 👍 [14:11:02] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11303076 (10Eevans) [14:12:27] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Data-Persistence, 10Data-Persistence-Design-Review, 06Growth-Team, and 3 others: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task - https://phabricator.wikimedia.org/T401021#11303082 (10Eevans) [14:44:53] sukhe: Having a look at the echoserver setup https://github.com/wikimedia/operations-deployment-charts/blob/master/helmfile.d/dse-k8s-services/echoserver/values-dse-k8s-codfw.yaml [14:47:50] We did have some pending changes from the recent envoy work, but that didn't seem to be the cause [15:01:17] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [15:07:11] (03PS3) 10Joal: Update referer classification patterns [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1198313 (https://phabricator.wikimedia.org/T406531) [15:26:17] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [15:31:17] RESOLVED: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [15:34:27] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10Data-Platform, 06Movement-Insights (FY25-26 H1), 13Patch-For-Review: NEWFEATURE REQUEST: Add new referral sources to pageview data - https://phabricator.wikimedia.org/T406531#11303462 (10JAllemandou) I have a question as I'm changing this: in t... [15:46:55] (03PS4) 10Joal: Update referer classification patterns [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1198313 (https://phabricator.wikimedia.org/T406531) [15:52:17] FIRING: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [15:53:31] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10Data-Platform, 06Movement-Insights (FY25-26 H1), 13Patch-For-Review: NEWFEATURE REQUEST: Add new referral sources to pageview data - https://phabricator.wikimedia.org/T406531#11303547 (10Isaac) Thanks for the ping! A few thoughts but don't let... [15:56:10] 07Analytics-Data-Problem, 06Data-Engineering, 06Movement-Insights, 06Product-Analytics, 13Patch-For-Review: webrequest dataset sets referer_class "unknown" instead of "external (search engine)" for origin-based referer values - https://phabricator.wikimedia.org/T383088#11303557 (10Isaac) Just commenting... [16:01:31] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Change RefererClassifier to capture malformed but internal referer strings as internal (wmf.pageview_actor) - https://phabricator.wikimedia.org/T400799#11303585 (10JAllemandou) →14Duplicate dup:03T383088 [16:01:37] 07Analytics-Data-Problem, 06Data-Engineering, 06Movement-Insights, 06Product-Analytics, 13Patch-For-Review: webrequest dataset sets referer_class "unknown" instead of "external (search engine)" for origin-based referer values - https://phabricator.wikimedia.org/T383088#11303587 (10JAllemandou) [16:04:19] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10Data-Platform, 06Movement-Insights (FY25-26 H1), 13Patch-For-Review: NEWFEATURE REQUEST: Add new referral sources to pageview data - https://phabricator.wikimedia.org/T406531#11303595 (10JAllemandou) @Mayakp.wiki can I get your perspective on I... [16:05:50] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th): Technical work for SDS1.3.7 Incorporate Edge Signal - https://phabricator.wikimedia.org/T407893#11303613 (10Ahoelzl) [16:12:23] (03PS2) 10Xcollazo: Add utility to create SHA256 fingerprints of the files of a particular HDFS folder. [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1198152 (https://phabricator.wikimedia.org/T401022) [16:13:08] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10Event-Platform: Fix mediawiki event enrichment to work with newest version of Blubber - https://phabricator.wikimedia.org/T406872#11303647 (10Ahoelzl) [16:17:06] (03CR) 10Xcollazo: "(There was an intermittent test failure that I just fixed by making the call to get file status objects deterministic.)" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1198152 (https://phabricator.wikimedia.org/T401022) (owner: 10Xcollazo) [16:19:37] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 06Java-Scala-Standardization, 06Discovery-Search (2025.09.26 - 2025.10.17), 07Essential-Work: Ignore MacOS .DS_Store in parent pom - https://phabricator.wikimedia.org/T407514#11303673 (10Ahoelzl) [16:20:17] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10Event-Platform: mediawiki_event_enrichment - update default params and tests to use mediawiki/page_change 1.3.0 (latest) schema - https://phabricator.wikimedia.org/T407779#11303676 (10Ahoelzl) [16:22:17] RESOLVED: [2x] EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [16:24:44] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10AbuseFilter, 06DBA, 07Schema-change-in-production: Drop the afl_ip column and the afl_ip_timestamp index from the abuse_filter_log table - https://phabricator.wikimedia.org/T407997#11303685 (10Ahoelzl) [16:28:48] 06Data-Engineering, 06Data-Engineering-Radar, 10CampaignEvents, 06DBA, and 2 others: Apply ce_event_contributions schema changes in production (x1) - https://phabricator.wikimedia.org/T407587#11303709 (10Ahoelzl) [16:49:05] 06Data-Engineering, 06Experimentation Lab, 10MediaWiki-extensions-WikimediaEvents, 05MW-1.45-notes (1.45.0-wmf.24; 2025-10-21): Prepare mediawiki-client-error Logstash dashboards for mobile subdomain sunsetting - https://phabricator.wikimedia.org/T400852#11303766 (10Jdlrobson-WMF) 05Open→03Resolved... [17:06:48] (03CR) 10Xcollazo: [C:03+2] Add utility to create SHA256 fingerprints of the files of a particular HDFS folder. [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1198152 (https://phabricator.wikimedia.org/T401022) (owner: 10Xcollazo) [17:36:58] (03CR) 10CI reject: [V:04-1] Add utility to create SHA256 fingerprints of the files of a particular HDFS folder. [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1198152 (https://phabricator.wikimedia.org/T401022) (owner: 10Xcollazo) [18:10:05] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10Data-Platform, 06Movement-Insights (FY25-26 H1), 13Patch-For-Review: NEWFEATURE REQUEST: Add new referral sources to pageview data - https://phabricator.wikimedia.org/T406531#11304009 (10Mayakp.wiki) hey @Isaac, more and more search engines are... [18:12:23] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10Data-Platform, 06Movement-Insights (FY25-26 H1), 13Patch-For-Review: NEWFEATURE REQUEST: Add new referral sources to pageview data - https://phabricator.wikimedia.org/T406531#11304010 (10Mayakp.wiki) @JAllemandou , +1 on changing the IP referer... [18:53:00] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 07OKR-Work: SDS 1.3.2 Conduct Analysis on Alerting for changes in automated traffic distribution - https://phabricator.wikimedia.org/T406882#11304196 (10Hghani) @Snwachukwu Based on our discussion it sounds like using March 20-th to October 15 2025 t... [18:58:19] (03CR) 10Xcollazo: [C:03+2] "recheck" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1198152 (https://phabricator.wikimedia.org/T401022) (owner: 10Xcollazo) [19:12:02] (03Merged) 10jenkins-bot: Add utility to create SHA256 fingerprints of the files of a particular HDFS folder. [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1198152 (https://phabricator.wikimedia.org/T401022) (owner: 10Xcollazo) [19:27:41] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 10Data-Platform, 06Movement-Insights (FY25-26 H1), 13Patch-For-Review: NEWFEATURE REQUEST: Add new referral sources to pageview data - https://phabricator.wikimedia.org/T406531#11304307 (10Isaac) > so the way I see it, it means we may not need a... [19:28:45] 06Data-Engineering (Q2 FY25/26 October 1st - December 31th), 13Patch-For-Review: Implement the data layout, UI, and documentation for the XML file export - https://phabricator.wikimedia.org/T401022#11304309 (10xcollazo) >>! In T401022#11054851, @xcollazo wrote: > Parking this here before I forget: one thing we... [20:20:27] 06Data-Engineering, 10CirrusSearch, 06Data-Platform-SRE, 10DPE-Mediawiki-Content, and 3 others: Source the CirrusSearch index dumps from hadoop instead of a MW maintenance script - https://phabricator.wikimedia.org/T366248#11304450 (10EBernhardson) It looks like DE are going to move forward with the existi... [21:48:00] 06Data-Engineering: Requesting Kerberos access for Jmoore111 - https://phabricator.wikimedia.org/T408165 (10JMoore-WMF) 03NEW [22:26:17] FIRING: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [23:06:17] RESOLVED: EventgateProduceRateAnomaly: Significant produce rate deviation (+-25%) on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-dc=eqiad%2Bprometheus/k8s&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateProduceRateAnomaly [23:22:43] (03CR) 10Aleksandar Mastilovic: "LGTM apart from one nitpick" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1198313 (https://phabricator.wikimedia.org/T406531) (owner: 10Joal)