[06:48:37] <ryankemper>	 We're in the middle of a big spike in WDQS queries. Spike started 20 mins after wdqs deploy finished so I think it's unrelated to the deploy
[06:57:14] <ryankemper>	 https://grafana.wikimedia.org/d/000000489/wikidata-query-service?orgId=1&from=1743481823320&to=1743490608846 here's the qps graph from approximate time of deploy concluding
[06:59:33] <ryankemper>	 I'm also confused why maxlag is firing because I've depooled the only two hosts that are experiencing high lag (wdqs2016 / wdqs2017)
[07:01:48] <ryankemper>	 Oh nevermind it's resolved right now. Weird it fired again though
[07:04:47] <gehel>	 ryankemper: go get some sleep, we'll keep an eye on things 
[07:05:01] <ryankemper>	 ack!
[07:05:01] <gehel>	 Thanks for having a look !
[07:45:21] <dcausse>	 sigh... we nearly tripled the query rate on wdqs, from 10k/min to 30k/min
[07:45:55] <gehel>	 it looks like there is an NPE on 2016-2017, which seems to block the updater.
[07:46:12] <gehel>	 I'm opening a phab task, but I suspect we'll need you dcausse to have a look
[07:48:35] <dcausse>	 sure
[07:49:46] <gehel>	 dcausse: T390665
[07:49:47] <stashbot>	 T390665: wdqs2016 and 2017 not consuming updates - https://phabricator.wikimedia.org/T390665
[07:53:25] <dcausse>	 looking
[07:54:36] <gehel>	 we might need help from brouberol / stevemunene to identify this additional traffic and maybe block it.
[07:55:03] <brouberol>	 how can I do that?
[07:56:54] <gehel>	 We have some documentation in https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Identifying_Abusive_Traffic. Basically, using Turnilo to see if we can identify an offender.
[07:57:30] <gehel>	 I suspect that a 3x increase in traffic in such a short timeframe comes from a single user (maybe from multiple IPs).
[07:57:50] <gehel>	 Depending on what we learn, we might use requestctl to block that traffic.
[07:58:05] <gehel>	 brouberol: ping dcausse for additional help if needed.
[07:58:30] <gehel>	 This isn't UBN yet, the problematic servers have been depooled, and the rest of the cluster seems to be able to handle the load.
[07:58:39] <brouberol>	 ack thanks
[07:58:45] <dcausse>	 gehel, brouberol could we get https://gerrit.wikimedia.org/r/c/operations/puppet/+/1131940 with depooled host not being updated we should not trigger wikidata max lag but we do at the moment
[08:00:28] <brouberol>	 AS number 14618 seems to be over-represented in the hits
[08:00:33] <brouberol>	 dcausse looking
[08:01:57] <brouberol>	 so you want me to merge this patch but avoid deploying it on wdqs201{6,7} ?
[08:02:50] <dcausse>	 brouberol: no it's fine deploying everything, esp on wdqs20[67] :)
[08:03:29] <dcausse>	 blazegraph is in bad shape there...
[08:03:44] <brouberol>	 ok, I'm going to trust you on the patch and delpoy
[08:04:08] <dcausse>	 it should just touch the nginx condif
[08:04:12] <dcausse>	 *config
[08:05:10] <brouberol>	 are we good about running
[08:05:10] <brouberol>	 brouberol@cumin2002:~$ sudo cumin 'wdqs2*.codfw.wmnet' run-puppet-agent 
[08:05:10] <brouberol>	 ?
[08:05:23] <brouberol>	 aka deploying the patch to the whole codfw wdqs fleet?
[08:06:25] <dcausse>	 brouberol: yes
[08:06:36] <brouberol>	 :spinner:
[08:07:18] <dcausse>	 what we to see is this graph https://grafana-rw.wikimedia.org/d/000000489/wikidata-query-service?orgId=1&refresh=1m&viewPanel=41&from=now-3h&to=now being flat
[08:07:29] <dcausse>	 *we want to see
[08:08:28] <brouberol>	 I'm seeing multiple IPs from AS 14618 hitting wdqs starting ~midnight
[08:09:41] <dcausse>	 brouberol: the bump started around ~5am: https://grafana-rw.wikimedia.org/d/000000489/wikidata-query-service?orgId=1&refresh=1m&viewPanel=44&from=now-24h&to=now
[08:10:45] <dcausse>	 and it should from ips hitting codfw only
[08:11:05] <gehel>	 Let's ban AWS!
[08:13:18] <brouberol>	 oof indeed, that is AWS
[08:13:41] <brouberol>	 > and it should from ips hitting codfw only
[08:13:41] <brouberol>	 trying to figure out how to do this is turnillo, which is <insert rant>
[08:14:30] <gehel>	 We might have a user agent for those requests, and we might be able to ban from that user agent.
[08:17:17] <brouberol>	 I'm seeing the UA `axios/0.22.0` being the source of many requests, starting from 0:00 UTC and peeking again between 5 and 6 am utc
[08:18:25] <dcausse>	 brouberol: see in #sec I found an UA mentionning AWS
[08:18:46] <brouberol>	 let's move to #sec 
[08:43:52] <dcausse>	 going to revert what was deployed this night
[08:44:49] <dcausse>	 the new deletion query appears to be way too slow
[08:45:40] <dcausse>	 running scap
[08:56:17] <gmodena>	 o/
[08:56:21] <dcausse>	 o/
[08:57:15] <gmodena>	 Not urgent, but I got a bit lost in the thread (just back from a long weekend :P). Are good with the mjolnir internal range as implemented in https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1186. Can I merge and test?
[08:58:21] <dcausse>	 gmodena: sure, but that might need a rebase, we've fixed a bunch of fixtures while fixing execution_date in unit tests
[08:58:52] <dcausse>	 brouberol: I might your help still, could run "systemctl restart wdqs-updater.service" on the whole wdqs fleet?
[08:58:55] <dcausse>	 *need
[08:58:57] <gmodena>	 dcausse ack. I saw there's a few changes.
[08:59:49] <brouberol>	 dcausse: I can do that
[08:59:53] <brouberol>	 one by one I take it?
[09:00:03] <brouberol>	 with a bit of delay between each host?
[09:00:29] <dcausse>	 brouberol: no not really, you batch multiple ones at once with no delay
[09:00:38] <dcausse>	 *can batch
[09:01:48] <brouberol>	 alright then
[09:03:20] <gehel>	 dcausse: 1:1 or are you on the WDQS issues?
[09:03:29] <brouberol>	 dcausse: brouberol@cumin2002:~$ sudo cumin -b 3 -s 15 'wdqs*.wmnet' 'systemctl restart wdqs-updater.service'
[09:03:36] <dcausse>	 gehel: I can join
[09:03:47] <dcausse>	 brouberol: +1
[09:04:59] <gmodena>	 merginig/rebasing in this mono repo feels always a bit sketchy :|. Let's see what CI says.
[09:15:18] <brouberol>	 dcausse: the rolling restart is all done
[09:15:44] <dcausse>	 brouberol: thanks!
[09:15:51] <brouberol>	 np
[10:03:31] <dcausse>	 lunch
[10:19:33] <dcausse>	 an extra restart of blazegraph was required...
[10:20:20] <dcausse>	 we should be back to normal operations in a couple minutes
[10:23:39] <dcausse>	 ok, all alerts should resolve
[10:24:19] <dcausse>	 going back to lunch
[11:56:11] <dcausse>	 wow, there's a big mistake in the new query to delete lexemes... could have deleted all lexeme data...
[11:56:30] <dcausse>	 explains why blazegraph got stuck...
[13:13:28] <inflatador>	 <o/
[13:14:35] <dcausse>	 o/
[13:14:46] <dcausse>	 quick errand
[13:14:50] <inflatador>	 Sounds like WDQS is having some issues...LMK if I can do anything to help
[13:52:41] <dcausse>	 inflatador: yes some troubles today, a serious bug I introduced causing the updates to stall and a bot from amazon tripling the query rate, was hard to figure out what was going on but it's back to normal now
[14:03:26] <inflatador>	 Trey314159 got a minute to help with a regex? ref https://gerrit.wikimedia.org/r/c/operations/puppet/+/1132772/3/hieradata/regex.yaml#2
[14:09:07] <inflatador>	 not urgent BTW
[14:10:24] <ebernhardson>	 \o
[14:10:40] <dcausse>	 going to a snapshot build on wdqs2025 (test machine) to make sure I got that delete query right this time
[14:10:42] <dcausse>	 o/
[14:10:45] <dcausse>	 *to test
[14:10:54] <inflatador>	 It works according to https://regexp.online/ but just wanted to make sure
[14:24:59] <Trey314159>	 inflatador: that regex looks straightforward enough. Matches 6 elastic hosts and 1 cirrus host. The parens around 055 aren't strictly required, but they make sense to have if you expect to possibly add more cirrus hosts in the future. Looks good!
[14:25:21] <inflatador>	 Trey314159 excellent, thanks for taking a look!
[14:32:55] <dcausse>	 deployed a test build to wdqs2025:/srv/deployment/wdqs/wdqs/lib/streaming-updater-consumer-0.3.150-jar-with-dependencies.jar
[14:58:30] <dcausse>	 ok seems to work ok this time
[15:25:08] <inflatador>	 ryankemper can you join our pairing
[15:25:20] <ryankemper>	 yeah
[15:36:50] <ebernhardson>	 random thoughts, is there any harm to always running Exec[nginx-reload] on a puppet run, instead of only when refreshed?
[15:37:12] <ebernhardson>	 thinking about the TLS cert that expired, and how the Exec[nginx-reload] wasn't run beause of a transient error. Ideally it seems that would have been run in a future run of puppet
[15:37:51] <ebernhardson>	 Otherwise i suppose something could try and compare the TLS cert nginx responds with against the one on disk and refresh if there is a mismatch, but always reloading seems easier
[15:41:09] <dcausse>	 I have no clue... what was this transient error?
[15:41:19] <ebernhardson>	  Failed to open TCP connection to puppetserver1003.eqiad.wmnet:8140 (Connection refused - connect(2) for "puppetserver1003.eqiad.wmnet" port 8140)
[15:41:44] <ebernhardson>	 basically while it was running the nginx module there was a temporary problem connecting to puppetserver, which caused the Exec[nginx-reload] to be unscheduled due to dependencies
[15:41:54] <ebernhardson>	 so the cert got replaced, but the reload never ran
[15:42:35] <dcausse>	 weird... should puppet have failed altogether and mark the host an unhealthy?
[15:43:00] <ebernhardson>	 puppet did fail, but the next puppet run was fine. The problem is the cert was already replaced by then, so it didn't know it needed to reload nginx on future runs
[15:43:13] <ebernhardson>	 Basically cert replaced -> puppet failed -> no nginx-reload
[15:43:39] <ebernhardson>	 sampling of logs: https://phabricator.wikimedia.org/T390599#10695916
[15:44:32] <dcausse>	 ah, puppet being non transactionnal it did not revert to the old cert so did not detect that change on the second run
[15:45:06] <ebernhardson>	 i suppose in my ideal world, puppet would have collected all the file metadata with the initial catalog and wouldn't change anything on the server until it has all the things it needs, but apparently it doesn't work that way:P
[15:46:25] <dcausse>	 I have no clue how bad that'd be to run nginx-reload every 30mins? the few times I used it it was pretty fast
[15:46:26] <ebernhardson>	 yea, non-transactional is a good description of the problem
[15:47:00] <dcausse>	 but it was on hosts with what I'd consider low traffic like a wdqs node
[15:47:29] <dcausse>	 in theory nginx-reload should be a noop most of the time no?
[15:48:10] <ebernhardson>	 i think it will still gracefull restart regardless, IIRC it essentiall stops accepting new work then restarts itself while keeping the sockets open and passing them on to the new process
[15:48:31] <ebernhardson>	 (at least, thats how servers did it 25 years ago, i might be out of date :P)
[15:48:39] <ebernhardson>	 but nginx is also 20-ish years old
[15:49:05] <ebernhardson>	 i dunno, maybe not worth worrying about...but seems like something that should be solvable with automation
[15:51:51] <dcausse>	 reading the reload doc you're right it does all this gracefully but involves closing connections
[15:53:16] <ebernhardson>	 maybe the right answer then is something that compares the tls cert from nginx against the tls cert on disk...will see if thats hard
[15:53:42] <ebernhardson>	 but might require extra knowledge about how the certs are used, likely the cert renewal doesn't know what port for example
[15:54:19] <dcausse>	 all the liveness sensors are targetting the plain http port?
[15:54:50] <ebernhardson>	 not sure, i'll have to poke around puppet. I never fully understood how the tls bits interact
[15:55:41] <inflatador>	 we do have cert expiration alerts BTW...my team got them on Mar 27th, we just didn't act in time
[15:57:26] <ebernhardson>	 inflatador: sure, but i don't like having to react on alerts, it's failure prone :)   I've found a command that will give us a fingerprint of the running cert based on port + certname, can maybe do something with that. A small bash script that takes port+name+cert path and exits with 0 or 1
[16:23:52] <inflatador>	 workout, back in ~45
[16:51:40] <ebernhardson>	 have a script that does the comparisson now, but not sure where it fits in puppet :P
[16:53:21] <dcausse>	 dinner
[17:31:51] <inflatador>	 quick patch for fixing up the cirrus profile if anyone has a chance to look: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1133191
[17:34:40] <inflatador>	 lunch, back in 45
[18:20:35] <inflatador>	 back
[18:42:52] <inflatador>	 one more puppet patch for fixing hiera if anyone has time to look: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1133211 
[18:42:56] <inflatador>	 probably won't be the last ;(
[18:44:11] <ebernhardson>	 inflatador: looks to match the role name, +1
[18:46:23] <inflatador>	 ebernhardson thanks for the +1, let's see how far we get this time ;)
[19:22:21] <inflatador>	 ¡Uno mas! https://gerrit.wikimedia.org/r/c/operations/puppet/+/1133220
[19:24:01] <ebernhardson>	 lol
[19:24:39] <ebernhardson>	 fun, but i suppose there is no easy way around that
[19:30:07] <inflatador>	 yeah, I guess the relative lack of problems with cloudelastic/relforge made me a bit complacent
[19:30:59] <inflatador>	 and yeah, I agree re: your comment on CR. We'll have to migrate at some point...I started to look at until i realized it was touching LVS ;P
[19:34:26] <ebernhardson>	 migrating live things is always fun...will see :)
[20:06:12] <inflatador>	 One more that should hopefully get us to the next error: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1133234 cc ryankemper 
[20:06:32] <ryankemper>	 +1'd
[20:11:12] <inflatador>	 ACK, thanks!
[20:17:45] * ebernhardson notices that ttmserver is configured to only create a single replica
[20:18:09] <ebernhardson>	 although curiously, the prod indices have 2
[20:18:25] <ebernhardson>	 but only in eqiad, and not codfw...fun :P
[20:21:19] <inflatador>	 whoops! We should probably bump that up
[20:24:27] <inflatador>	 BTW, we're getting puppet errors like `Function lookup() did not find a value for the name 'profile::opensearch::cirrus::enable_remote_search'` now...it looks like we're going to have some problems if we use `role::cirrus::opensearch` with `profile::opensearch::cirrus`
[20:24:53] <ebernhardson>	 yea i'll make a patch, i'm noticing various config things while looking over configs to migrate cirrus read traffic to search.discovery.wmnet.  Probably worth a ticket to review multiple things (unused config vars, low replicas in ttmserver, maybe more)
[20:26:54] <ebernhardson>	 inflatador: isn't that just missing the appropriate hiera config in the role file? seems like it should work
[20:30:48] <inflatador>	 ebernhardson you might be right...checking. So this is the role file https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/role/manifests/cirrus/opensearch.pp . which pulls in https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/profile/manifests/opensearch/cirrus/server.pp
[20:32:08] <inflatador>	 which points to a typo: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/hieradata/role/common/cirrus/opensearch.yaml#76
[20:32:33] <ebernhardson>	 ahha, yea that will do it
[20:35:39] <inflatador>	 CR to fix this: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1133242 cc: ryankemper 
[20:36:43] * ebernhardson is finding the deployment-prep vs prod configuration variations in wmf-config tedious...
[20:51:52] <inflatador>	 Ibid. https://gerrit.wikimedia.org/r/c/operations/puppet/+/1133243
[22:08:26] <inflatador>	 we got the puppet catalog to comile
[22:08:30] <inflatador>	 or compile, even
[22:17:56] <inflatador>	 ryankemper cirrussearch2005 is reachable thru SSH if you wanna check it out. Looks like opensearch isn't able to start
[22:18:05] <inflatador>	 eh...cirrussearch2055 that is
[22:18:19] <ryankemper>	 ack, will look
[22:18:45] <inflatador>	 cool, headed out for the day. keep us posted!
[22:28:39] <ryankemper>	 whole bunch of errors. hard to tell which are independent versus occurring because other stuff is broken https://usercontent.irccloud-cdn.com/file/JFHIoKUV/opensearch_errors.log
[22:31:18] <ryankemper>	 Starting with the first error `OpenSearchJsonLayout contains invalid attributes "compact", "complete"`. Might be related to this config: `modules/opensearch/templates/log4j2_1.properties.erb:appender.ship_to_logstash.layout.compact=true`