[01:32:59] 06Traffic, 10Huggle: Huggle is getting rate-limited when working on multiple wikis in parallel - https://phabricator.wikimedia.org/T415141#11540052 (10Framawiki) [04:11:43] FIRING: [10x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [04:16:43] FIRING: [14x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [04:21:43] FIRING: [15x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [04:26:43] FIRING: [15x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [04:31:43] RESOLVED: [15x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [14:40:30] 10netops, 06Infrastructure-Foundations, 06SRE, 06Data-Platform-SRE (2026.01.05 - 2026.01.23), 07Essential-Work: Socket leaking on some dse-k8s row C & D hosts - https://phabricator.wikimedia.org/T414460#11541938 (10ops-monitoring-bot) Roll-reboot of nodes in dse-eqiad cluster started by btullis: * dse-k8... [14:40:45] 10netops, 06Infrastructure-Foundations, 06SRE, 06Data-Platform-SRE (2026.01.05 - 2026.01.23), 07Essential-Work: Socket leaking on some dse-k8s row C & D hosts - https://phabricator.wikimedia.org/T414460#11541963 (10ops-monitoring-bot) Roll-reboot of nodes in dse-eqiad cluster started by btullis: * dse-k8... [14:41:04] 10netops, 06Infrastructure-Foundations, 06SRE, 06Data-Platform-SRE (2026.01.05 - 2026.01.23), 07Essential-Work: Socket leaking on some dse-k8s row C & D hosts - https://phabricator.wikimedia.org/T414460#11541988 (10ops-monitoring-bot) Roll-reboot of nodes in dse-eqiad cluster started by btullis: * dse-k8... [15:07:53] 10netops, 06Infrastructure-Foundations, 06SRE, 06Data-Platform-SRE (2026.01.05 - 2026.01.23), 07Essential-Work: Socket leaking on some dse-k8s row C & D hosts - https://phabricator.wikimedia.org/T414460#11542145 (10ops-monitoring-bot) Host dse-k8s-worker1008.eqiad.wmnet rebooted by btullis@cumin1003 with... [15:18:18] 06Traffic, 10DNS, 06Infrastructure-Foundations, 10SRE-tools, 13Patch-Needs-Improvement: DNS repo: add Jenkins job to ensure there are no duplicates - https://phabricator.wikimedia.org/T155761#11542183 (10BCornwall) 05Stalled→03Resolved a:03BCornwall I believe this has been solved with the lates... [15:26:34] 10netops, 06Infrastructure-Foundations, 06SRE, 06Data-Platform-SRE (2026.01.05 - 2026.01.23), 07Essential-Work: Socket leaking on some dse-k8s row C & D hosts - https://phabricator.wikimedia.org/T414460#11542216 (10ops-monitoring-bot) Roll-reboot of nodes in dse-eqiad cluster started by btullis: * dse-k8... [15:40:36] 06Traffic, 10Prod-Kubernetes, 06ServiceOps new, 07Kubernetes, 13Patch-For-Review: Handling inbound IPIP traffic on low traffic LVS k8s based realservers - https://phabricator.wikimedia.org/T352956#11542340 (10Vgutierrez) > It's not clear why the net.ipv4.conf.default.rp_filter would need to change to 0 f... [15:43:43] 06Traffic, 13Patch-For-Review: purged package cannot be built due to failing test - https://phabricator.wikimedia.org/T354712#11542348 (10BCornwall) 05Open→03Resolved I think the merged MRs means this can be closed. Please reopen if this was closed in error! [15:59:16] 06Traffic, 06Data-Engineering, 06Infrastructure-Foundations, 13Patch-For-Review: Export development_network_probe data to Puppet servers for CDN deployment - https://phabricator.wikimedia.org/T402512#11542398 (10elukey) Thanks a lot for the detailed explanation Ben! I tried to work on the Puppet part in ht... [17:51:25] FIRING: SystemdUnitFailed: bird.service on dns7001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:56:25] RESOLVED: SystemdUnitFailed: bird.service on dns7001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:55:29] 06Traffic, 10Huggle: Huggle is getting rate-limited when working on multiple wikis in parallel - https://phabricator.wikimedia.org/T415141#11543348 (10Framawiki) I was able to get rate-limited while patrolling on a single wiki at once. @Joe could it be possible to take a quick look in the logs to confirm you i... [21:04:59] 06Traffic, 10Huggle: Huggle is getting rate-limited when working on multiple wikis in parallel - https://phabricator.wikimedia.org/T415141#11543380 (10Joe) @Framawiki the error message you get seems to indicate that your tool is not authenticated, but the [[ https://github.com/wikimedia/operations-puppet/blob/... [21:15:03] 06Traffic, 10Huggle: Huggle is getting rate-limited when working on multiple wikis in parallel - https://phabricator.wikimedia.org/T415141#11543395 (10Joe) Ah sorry, you're saying you're using bot-password above. I missed that. Uhm, this means that probably the hotfix I deployed to try to hotpatch the issue wh... [21:19:32] 06Traffic, 10Huggle: Huggle is getting rate-limited when working on multiple wikis in parallel - https://phabricator.wikimedia.org/T415141#11543399 (10Joe) I should add - it's also possible that actually huggle doesn't send auth cookies with every request; in that case, you get rate-limited to 100 requests ove...