[08:47:00] FYI, in 15 mins the IDPs will be moved to new servers [08:48:32] ack [16:26:37] !log ganeti5003 firmware updates in progress via T308238 [16:26:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:26:41] T308238: Upgrade ganeti/esams to Bullseye - https://phabricator.wikimedia.org/T308238 [16:31:10] !log ganeti5003 reboot accidental by rob, fixing [16:31:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:31:18] its 3003, sigh. [16:31:54] moritzm: So yeah, i messed up and rebooted ganeti5003 just now without downtiming or draining it [16:32:00] cuz im dumb and meant to work on ganeti3003 [16:32:11] and used the idrac so it didnt force me to retype the hostname [16:32:28] basically i bypassed all the checks to prevent this by doing it via ilom manually... sorry about that [16:32:34] robh: no big deal, let me check what's there [16:32:40] its rebooting, i killed the firmware updates cuz they are already done on that host [16:32:58] too many open tabs, i know this mistake. [16:33:04] work on one thing at a time. [16:33:46] its still booting [16:34:11] ok, that is my mistake for this week, not allowed anymore ; D [16:34:12] or not enough automation :) [16:34:31] yeah, once we automate bios updates this kinda thing wont happen [16:34:39] but i caused a false trail on that thinking it could tftp [16:34:43] easier to look at icinga to see what was there [16:34:44] but NOPE, only the idrac can tftp update [16:34:54] rest of firmware requires good old ftp [16:35:21] looks like doh5002 (cc sukhe) and netflow5002 [16:35:25] and prometheus5001 [16:35:48] yep (cc herron) [16:36:01] I don't think anything is critical here [16:36:23] when you say check icinga you mean just check for all down items cuz i dont see a list of whats on ganeti5003? [16:36:28] 2 are monitoring, the last one should failover as it uses bird [16:36:42] robh: yeah exactly [16:36:47] robh: all down items in eqsin, if you look at https://icinga.wikimedia.org/alerts they are right there [16:36:56] I just ctrl-f'd for '500' [16:37:22] heh, ok cool [16:37:28] so yeah... sorry about that =P [16:37:48] I was all riding high on figuring out wtf happened with relined [16:37:51] reality didnt like that [16:37:53] robh: might be worth opening a task to figure out how to improve automation there [16:38:08] detailing the steps needed, etc [16:38:14] XioNoX: So i think the overall thing is manual firmware updating is bad and eventually we need to automate, and yeah [16:38:22] i think there is a task already, i'll add in this event as reason to move that along [16:38:34] nice, I think John was working on it? [16:39:36] Ah if you're going to break Ganeti let me know, so I can test my Ganeti Prometheus client :-) [16:40:27] yeah https://phabricator.wikimedia.org/T283771 [16:40:32] i'll append my bork to that today [16:40:33] slyngs: I'm curious, what will that do/export? [16:41:44] Metrics for capacity management, CPU/RAM availability, number of offline node, distribution of primary and secondary instances [16:41:47] so those down items are gone now [16:41:55] are they all doin their services as expected again? [16:42:05] slyngs: nice! [16:42:26] dunno how the hell i put in ganeti5003.eqsin.wmnet when i had the tab open for ganeti3003.esams.wmnet... those arent at all the same irght? ; D [16:42:42] 5 characters different even! [16:43:02] sorry just catching up [16:43:06] I was going to say a large number of the characters are the same [16:43:35] yeah, when you reboot a host in the OS, it clearly makes you type the hostname [16:43:47] and via script it outputs the entire hostname and makes you confirm that what you are about to do will break it [16:44:04] * cdanis forcing icinga re-check on all the purple UNKNOWNs for 500x hosts [16:44:04] but via manual idrac buttons... relies on me not being inattentive. [16:44:29] !log ganeti5003 returned to service after accidental reboot [16:44:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:44:36] sukhe: is doh5002 back to normal? [16:44:48] !log ganeti3003 (already depooled) coming down for firmware update and reimage via T308238 [16:44:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:44:51] T308238: Upgrade ganeti/esams to Bullseye - https://phabricator.wikimedia.org/T308238 [16:45:04] this isn't really a big deal tbh -- this is an event that could literally happen on its own at any time [16:45:30] Best to view it as a failover test :-) [16:45:36] yeah, if a server going down is a big deal, then something should be designed better [16:45:46] unless it's a DB master :) [16:45:52] eheh [16:45:56] all the purples cleared [16:46:45] https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=doh5002 looks good too [16:47:02] oh! [16:47:05] one last teachable moment here [16:47:34] the next time that anyone, robh or otherwise, has an "oh crap I did something Bad in production" moment, when you post you should also cc the current oncalls [16:47:44] oh, true, didnt even think of that [16:48:02] should we keep oncall names in topic maybe or just rely on wikitech? [16:48:18] the topic isn't a bad idea [16:48:29] we have ops clinic listed in -operations [16:48:47] so not first time we do such a thing heh [16:48:48] and speaking of being oncall, looking at shellbox now [16:48:49] I think the only reason it's not in the topic is it changes 3x a day, and we haven't yet figured out how to give everybody ops easily [16:48:59] in -operations I mean [16:49:02] XioNoX: seems to be OK, checking running DNS tests [16:49:07] hrmm, maybe just link to the wikitech page in topci [16:49:09] all good [16:49:27] rzl: IRC integration 😇 [16:49:28] we could also just name both NA and both EU oncalls, and then it only changes weekly [16:49:39] then you're on your own for figuring out what time it is, but that's not so bad [16:51:03] rzl: do you know anything about shellbox? who in ustz does? [16:51:10] https://grafana.wikimedia.org/d/RKogW1m7z/shellbox?orgId=1&var-dc=eqiad%20prometheus%2Fk8s&var-service=shellbox&var-namespace=shellbox&var-release=main&viewPanel=28 [16:51:20] k.unal :( let me see what I can figure out though [16:51:20] I am going to guess it is either underprovisioned or overcapacity, given that 'throttled' line [16:51:44] hmm yeah that'd make sense [16:51:54] I am also fine with simply throwing more replicas at the problem and asking intelligent questions later [16:51:58] that smells right, looks like it's regularly spiky but this last spike is much higher [16:52:05] and yeah was about to suggest the same [16:52:18] already going down too [16:52:38] Yes, But For How Long™ [16:53:11] of course, was saying that to frame the urgency [16:53:30] yeah :) [16:53:36] yeah [16:53:46] godog: Looking into fixing the daily mails of pontoon being quite out of date on puppet - went to apply and I'm getting "Could not request certificate: The certificate retrieved from the master does not match the agent's private key." - Do you have any advice/caution? [16:53:47] current traffic saturated the cpu reservation and 100% of the php-fpm workers [16:53:53] so ofc the blackbox prober paged [16:53:57] (tagging you because it says so in https://wikitech.wikimedia.org/wiki/Puppet/Pontoon) [16:54:34] yeah, this looks like it was just regular old spiky traffic -- I'll be interested in where it came from, whether it's legit or should be blocked, whether we should expect it long-term, etc [16:54:43] but in the meantime if we can just provision for it, let's do that [16:54:45] looks like we have only 5 replicas? [16:55:46] no, sorry, 8 [16:56:26] I suggest just doubling that? [16:56:35] https://gerrit.wikimedia.org/g/operations/deployment-charts/+/c12d4dea22aca8178fda4242fce04ee95362cb3b/helmfile.d/services/shellbox/values.yaml#12 [16:56:44] SGTM [16:58:10] sounds right, just double-checking to make sure it's the right shellbox [16:58:55] it is port 4008 robh [16:58:58] s/robh/rzl/ [16:59:06] yep that's the one [16:59:32] and doubling to 16 sounds good, are you doing that or am I? [16:59:44] if it is easy for you i'd appreciate it [16:59:58] I think it is, and if it's not that'll be a valuable experience :) doing [17:00:22] ha I was about to volunteer for the same reasons [17:00:38] I'm about to hop into a 1:1 [17:00:46] herron: you can get the next one :D [17:00:48] cdanis: ack [17:00:56] cheers cdanis thanks [17:01:10] ha fair enough rzl [17:02:12] mailed https://gerrit.wikimedia.org/r/803953, any stamps? [17:02:49] thanks! [17:02:56] np :) [17:03:04] slownp :) [17:03:28] * rzl await(jenkins) [17:04:09] It looks like this was kind of a request spike maybe? [17:04:19] from https://grafana.wikimedia.org/d/3SiE86Nnz/mediawiki-shellouts?orgId=1 [17:04:20] yeah agreed [17:04:44] there is really a huge spike f lilypond requests [17:04:55] lilypond spike is consistent with this being shellbox (rather than shellbox-something) since that's the score instance [17:05:10] yup [17:05:26] the pdfhandler stuff is shellbox-media afaik [17:12:09] mm, there were already diffs with production :/ so this "helmfile apply" is going to double the replicas but also upgrade from shellbox 1.0.0 to 1.0.3 [17:12:39] probably nbd but not ideal -- I'm going to start with staging first, even though I'm not changing the replicas there [17:13:18] surprise upgrade! [17:15:48] I think jayme and I chatted at one point about how it'd be nice to have a process for keeping these diffs clean, or at least warning us if they're dirty for too long [17:17:15] looks like that did not directly lead somewhere :) [17:17:35] shellbox staging is updated and hasn't burned to the ground, although I don't offhand know a way to test it [17:20:07] the wikitech page has nothing more then the healthz check - that has been done by kubernetes for you already [17:21:09] yeah, ideally I'd like to give it some test traffic, later I'll see about how to do that and try to document it [17:21:14] for now I'm going ahead with codfw [17:22:44] we get a steady trickle of requests there, I'm watching https://grafana.wikimedia.org/d/RKogW1m7z/shellbox?orgId=1&var-dc=codfw+prometheus%2Fk8s&var-service=shellbox&var-namespace=shellbox&var-release=main&from=1654705347482&to=1654708947482 [17:24:33] hm, ProbeDown again, looks like it stopped answering briefly during the rollout [17:25:11] it looks like more saturation tbh [17:25:33] rps picked back up since 17:15 and apache workers busy picked way up at 17:21 [17:26:05] the downprobe was from eqiad aiui [17:26:05] oh man yeah, I didn't even see the traffic in eqiad because I was watching the rollout in codfw [17:28:37] codfw looks recovered -- I was surprised to see the request rate is doubled, but of course it is, it's just health checks and there are twice as many [17:29:32] going ahead in eqiad now [17:30:19] heh, Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress [17:30:24] I lost the race with mwdebug it looks like [17:33:06] nah, that does not make sense [17:33:43] those operations are on a "per release" basis. So more like per chart [17:34:01] "per release" with release being the "helm release" [17:34:23] hm okay, so there's some phantom operation with shellbox eqiad then [17:34:31] yeah [17:34:34] let me take a look [17:35:04] thanks! [17:35:42] uh... [17:35:58] 👀 [17:36:16] try "kube_env admin eqiad" [17:36:21] (sudo -i) [17:36:35] helm3 -n shellbox list --all [17:36:47] without --all it looks frightening :) [17:37:12] 2022-01-12, eh [17:37:34] yeah...no idea [17:37:43] is that when someone started an upgrade that was never finished? [17:37:48] "helm3 -n shellbox history main" gives you the history for the main release [17:37:59] cdanis: yes. ^C maybe [17:38:01] cdanis: evidently yeah, the status is listed as pending-upgrade [17:38:01] I think we've found at least two things we should have alerts for 😅 [17:38:14] agreed [17:38:40] so, way out of this mess is roll back to revision 2 of release main, then re-run the helmfile deployment [17:39:03] ah, cool [17:39:04] luckily revision 2 is the same chart version than the one currently "pending upgrade" [17:39:16] can I roll back via helmfile, or is that still done through helm3 directly? [17:40:20] that needs to be done with helmfile [17:40:24] äh [17:40:29] helm3, sorry [17:40:46] okay cool - I see https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency but it's still instructions for helm rather than helm3 [17:41:02] is it just "helm3 rollback" though? [17:41:29] "helm3 -n shellbox main 2" [17:41:52] amazing thank you [17:41:53] there is a (non-ideal) doc at https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency [17:42:39] and I guess I don't have to say eqiad anywhere because that's set through kube_env [17:43:06] yes, correct. If you're in doubt run "kubectl get nodes" [17:43:12] 👍 [17:44:15] > Rollback was a success! Happy Helming! [17:44:21] nice [17:44:27] and list --all now shows revision 4, status deployed [17:44:40] backing out and rerunning the helmfile apply, then [17:44:54] 👍 [17:45:05] ahhh there we go [17:45:08] thanks for the handholding! [17:45:18] sure thing, yw! [17:45:56] so summarizing followup AIs here [17:46:13] * have an alert for a helm service that has diffs between what's deployed and what is head-of-tree in git [17:46:31] * have an alert for a helm deployment/upgrade/other mutation that has been open for "too long" (days? a week?) [17:46:53] #1 is https://phabricator.wikimedia.org/T265979 [17:47:54] I guess another question in my head is -- shellbox is not active/active, correct? [17:48:43] it is [17:48:48] or at least it should be [17:48:50] ah okay cool [17:49:11] so if had to do something more invasive with the eqiad deployment, we could have depooled it there [17:49:20] yes [17:49:50] cool, thanks! [17:50:00] np [17:50:01] I'm going to take a little break and then I'll file #2 [17:51:17] janis: check me? https://wikitech.wikimedia.org/w/index.php?title=Kubernetes/Deployments&diff=1987861&oldid=1987818 [17:51:28] uh, jayme: I mean, the one you actually highlight on [17:51:55] I hightlight on both ;) [17:52:00] haha oh good [17:52:05] looks good, thanks [17:53:04] and thanks for filing a task cdanis - I'll drop off. Have a nice day o/ [17:53:13] good night! thanks again for the help [18:39:55] 13:17:35 shellbox staging is updated and hasn't burned to the ground, although I don't offhand know a way to test it <-- live hack $wgShellboxUrls on mwdebug to point to staging, then modify one of the Score pages on testwiki (you need to change the content in tags to bypass cache) and see that it renders properly [18:41:15] I still don't have a good sense of how much Shellbox should be resistant to load spikes vs having spare replicas running that won't get used 99% of the time [18:41:49] ahh okay cool! thanks for that [18:42:16] and yeah, I know we've discussed that question before [20:04:30] Hey cdanis: herron: we are currently performing a rolling restart of memcache in eqiad with an hour between hosts. [20:06:42] ok! sounds good [20:08:24] rgr that [20:27:00] > [#wikimedia-releng] <•wikibugs> Deployments, serviceops, Wikimedia-production-error: mw1415 fatals due to serving responses from 1.39.0-wmf.10 (was DBQueryError: Unknown column page_restrictions) [20:27:29] > dancy: well, at least it's not serving prod I think? Last sal entry says its depooled. [20:28:04] I see an empty line at https://config-master.wikimedia.org/pybal/eqiad/appservers-https where it would presumably be [20:28:07] not even pooled=false [20:30:25] Krinkle: when there is an empty line that means it is pooled=inactive [20:30:29] like more off than false [20:30:44] usually that is done for things like hardware repair [20:31:21] mutante: ok, does pooled=inactive also means it doesn't receive scap deploy but remains serving apache traffic to pyball and health checks? [20:31:30] it's problematic to have 5 week old code running zombie in production [20:31:45] no, it should mean "not even in config" as in "no traffic AND no scap deploys" [20:31:55] that's why there is another alert about mw versions not matching [20:32:00] because it didnt get the deploy [20:32:11] this is what used to be "not in dsh groups" [20:32:49] https://alerts.wikimedia.org/?q=%40state%3Dactive&q=instance%3Dmw1415 [20:32:59] > Host mw1415 is not in mediawiki-installation dsh group [20:33:05] > WARNING: Missing 1 sites from wikiversions. 982 mismatched wikiversions [20:33:43] yea, that happens when it's not in scap (dsh) groups [20:33:46] so.. assuming the alert has not been ignored for a long time, what does that mean exactly. the host came back to live [20:33:47] no deployments to that host [20:34:02] and alert started firing just now? [20:34:42] I can't say much about alertmanager, unlike icinga [20:34:55] but I would think it can be an expired downtime [20:35:15] or someone/something deployed to "only canary hosts" specifically [20:35:23] outside the "dsh groups" [20:35:33] which translates to confctl settings [20:36:13] a 'scap pull' on the host should fix the "mismatching mw versions" alert separate from getting traffic or not [20:36:48] checking SAL for entries about that host [20:37:21] I don't thnk the MW version warning is something we should ack. like I said, we can't be running 5-week old code in production. If this is normal, I thikn we need to make it so that this state results in apache being turned off or healthcecks being skipped or something. This intermediary state of the host clearly being up and serving apache pings whilst not getting code updates seems like a state that should be impossible. [20:37:53] but then again, I think that's mostly true already, just something went wrong here [20:38:31] a host that is not in confctl should not get any traffic [20:38:46] but it does not mean monitoring is tied to it [20:39:11] alertmanager/icinga probably has no idea whether a host is in dsh groups (getting traffic). it just keeps checking [20:39:37] I assume `/w/api.php?action=query&meta=siteinfo&format=json&formatversion=2` requests are coming from pyball, right? [20:39:46] these are part of the healthchecks? [20:39:47] I don't know why this host was removed. Usually that would happen just when hardware breaks [20:40:05] - /wiki/Main_Page as well [20:40:18] those two basically are taking turns every 5 second triggering a fatal error [20:40:30] this _seems_ to indicate it just happened today https://icinga.wikimedia.org/cgi-bin/icinga/history.cgi?host=mw1415&service=mediawiki-installation+DSH+group [20:40:36] but there is nothing in SAL [20:41:15] modules/nagios_common/files/check_commands/check_etcd_mw_config_lastindex.py: url = 'http://{host}/w/api.php?action=query&meta=siteinfo&format=json&formatversion=2'.format( [20:41:38] seems to be this 'check_etcd_mw_config" thing [20:42:37] Krinkle: yes, I believe so [20:42:57] so I've run `scap pull` by hand for now to silence the logspam [20:43:06] thing is.. I don't know why it was set to inactive [20:43:11] Krinkle: did you find this in apache logs? there should be a user-agent [20:43:23] I was about to ask if you want me to just scap pull [20:43:29] then we can set it to pooled=no again [20:43:36] which will assure it will get scap deploys [20:43:48] and clear more alerts [20:43:49] but I think it's worth following up here so that we can figure out a strategy that does not involve it being part of the normal runbook to leave a host such that it is alive but not receiving code updates and actively serving old MW code connecting to production memc/mysql in response to health checks. [20:44:07] the fatal errors are actually a good thing, could've been worse if it e.g. silentely corrupted stuff in memc or mysql. [20:45:14] do we have a defined backwards compatibility window for that? [20:45:28] It doesn't make sense that the icinga alert history said it never alerted about "not in dsh groups" until today.. while we also say it was actively serving weeks old code. [20:45:43] it sounds like pooled=inactive consistently creates this result, which presumalby has a valid use case for being a third state, but we'll need to handle that better in some sense. [20:45:51] either it was in that state or not [20:46:32] I don't think it's actively serving old code.. except that monitoring checks won't stop checking [20:46:49] https://phabricator.wikimedia.org/T307755 [20:46:54] it was broken? [20:46:56] cdanis: for trains, it's 1 week, but there are also regular depoyments that backport changes to all branches and then flip a config flag with the expectation that within ~5min things propagate. the same for e.g. schema changes and maintenane jobs, after a few minutes we expect no new processes to start with the old code. [20:46:58] more concering to me is _why_ it was set to inactive without SAL [20:47:01] and looks like it got fixed today [20:47:32] prometheus data backs this up too [20:47:36] https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=mw1415&var-datasource=thanos&var-cluster=appserver&from=now-90d&to=now [20:47:44] it looks like it has been dead since the 6th of May [20:47:46] pooled=inactive has always meant "no scap AND no traffic" [20:47:54] ooh.. of course.. that is the host that we called Dell for.. duh [20:48:00] you filed the ticket :) [20:48:01] the Dell tech was there [20:48:21] ack, if a host is unresponsive or broken, it makes sense to take it out of dsh as otherwise scap will prompt deployers with something to respond to, separation of concerns etc. [20:48:35] yea, there are many tickets. doesn't mean I knew the tech would fix that today [20:48:55] I don't think we can realistically do what you're saying Krinkle [20:48:56] yea, Krinkle. that is exactly why. because otherwise deployers will get errors every time [20:49:12] the hardware of a host breaks suddenly, no warning [20:49:40] I think we would need a workflow where dcops changes pool state. [20:49:40] what do we do once the hardware is fixed -- or even to test that the hardware is fixed? do we firewall off its IP from the memcacheds and the mysql servers? [20:49:49] or has to schedule everything with us [20:50:26] this is kind of the opposite of unexpected failure. unexpected fix [20:50:43] Haha yes [20:50:52] Damn fixes [20:51:01] cdanis: ack, reminds me a bit of what we do with mysql, where afaik we after reboot start with service off and/or read-only. [20:51:19] assuming a reboot happened as part of this unexpected fix, perhaps that's reasonable to start with, until an SRE can repool it. [20:51:33] the runbooks already say that we need to run scap pull by hand before repooling it [20:51:46] we could do something like that, sure, but in general I think serviceops has been of the belief that if a host isn't receiving production traffic, it isn't going to perform any mutations against datastores [20:52:12] I guess Main_Page can cause parsercache writes and memcached writes? [20:52:23] indeed [20:52:41] and as part of filling in caches we sometimes do db writes e.g. link table migration and actor migration result in lazy populating db rows when things are absent [20:52:48] which can go on for months while we migrate schemas [20:53:46] the good news is that the healthcheck URLs are naturally so common that anything they do will have been done already presumably [20:53:55] naturally [20:54:19] so perhaps it's good enough that we 1) get the alert soon enough about outdated MW version and lack of dsh, and 2) then respond to that by doing at least a scap pull and pooled=no [20:54:26] so, I don't think that we should have mediawiki not start up upon reboot [20:54:34] in this case that did not happen within ~2h though [20:55:14] here's a hot take: I don't think this issue is worth worrying about, and all of these warts we have around scap and code versions and how Mediawiki runs will go away once it is on k8s [20:55:34] :) [20:55:55] this is a small one of many steps that would have to happen to cause some sort of disaster corruption in mysql or memcached [20:56:27] (and, if that did happen, we had better be able to handle it anyway!) [20:56:51] zooming out: the immediate issue was logspam and confusion for developers running deploymentes and monitoring their components in prod. the response was to file a prod error, and ~1h of investigation to figure out that it is specific to a host that came back to life [20:57:40] so for me it was very obvious what had likely happened as soon as I looked at https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=mw1415&var-datasource=thanos&var-cluster=appserver&from=now-24h&to=now [20:57:44] would the normal procedure had been that e.g. within the next hour an SRE would have seen the alert and run scap pull and set pooled=no as part of a documented runbook? That might be a good solution in the interim. [21:01:12] what was the alert that fired? [21:01:51] four alerts are firing: https://alerts.wikimedia.org/?q=%40state%3Dactive&q=instance%3Dmw1415 Etcd status, MW version, PHP HTTP 500, Apache HTTP 500. [21:02:17] as of 1 hour ago, since ~30min since the host came online it seems [21:02:24] wikiversions? [21:02:51] yeah that one too I think [21:03:05] I don't see it now though [21:03:14] they should be recovering soon since I ran scap pull [21:03:30] https://icinga.wikimedia.org/cgi-bin/icinga/history.cgi?host=mw1415&service=Ensure+local+MW+versions+match+expected+deployment [21:03:32] I have not changed any pooling though. [21:03:33] so it fired in an odd way [21:03:46] ahh okay [21:03:49] "Missing 1 sites from wikiversions. 982 mismatched wikiversions" [21:04:11] so it had been so long we added a new wiki, and, we had all of the rest mismatched [21:04:41] that will be noticed eventually, but [21:04:48] * only during business hours, with no guarantees [21:05:06] * it's a very noisy alert, and last I saw often fires a lot around deployment time anyway, so it is often ignored [21:05:42] rescheduling the alerts in icinga, hold on [21:05:53] one should clear just because you ran scap pull [21:07:30] "often fires a lot around deployment time" - that's a problem :) [21:07:37] I see it also lacks a runbook page [21:07:39] yes [21:07:44] alert points to https://wikitech.wikimedia.org/wiki/Application_servers [21:07:54] also, when it fires, it generally fires on many, many hosts at once and floods the channel [21:07:56] :) [21:08:04] but "Host mw1415 is not in mediawiki-installation dsh group" is a better one I think [21:08:17] mw versions is fairly weak as it doesn't consider intra-week changes or config changes etc. [21:08:21] dsh group will capture everything [21:08:37] how, if at all, would that one have been responded to? [21:09:17] "serviceops will likely look at it eventually" is the best answer I have :) [21:09:26] it would have been ACKed until the hardware repair ticket gets updated [21:09:38] there's also that, yes [21:13:36] !log mw1415 - scap pull, restart apache, /usr/local/sbin/restart-php7.2-fpm (INFO: The server is depooled from all services. Restarting the service directly) [21:13:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:14:04] I need to go afk but happy to continue this conversation later [21:16:00] Apache alert recovered [21:16:02] I've boldly updated https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups for now. The last part of that is probably wrong as I'm not sure what the procedure is around hardware repairs etc. maybe that is redundant. Edit at all :) [21:16:03] now CRITICAL - degraded: The following units failed: prometheus-phpfpm-statustext-textfile.service [21:16:16] trying to clear that up as well [21:16:39] Edit at will* [21:17:22] setting to pooled=no after apache/php is green [21:17:37] waiting for "dsh groups" alert to recover after that [21:17:53] 21:14 <+icinga-wm> RECOVERY - PHP7 rendering on mw1415 is OK: HTTP OK: HTTP/1.1 302 Found - 559 bytes in 0.076 second response time https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_rendering [21:17:57] 21:14 <+icinga-wm> RECOVERY - Apache HTTP on mw1415 is OK: HTTP OK: HTTP/1.1 302 Found - 546 bytes in 0.103 second response time https://wikitech.wikimedia.org/wiki/Application_servers [21:20:18] Krinkle: it's back on https://config-master.wikimedia.org/pybal/eqiad/appservers-https [21:20:22] no more missing line [21:20:52] ack, thanks! [21:21:04] https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=mw1415&scroll=178 is all GREEN again [21:21:26] that means to me we can now pooled=yes [21:21:30] just like before it broke [21:22:08] and call the hardware repair ticket resolved. separate from workflow optimization questions [21:22:27] My feeling is that the way I connected the dots here from happening to see the Phab ticket to jumping in -sre is the part that was off-script and something others likely would not have done. I don't know how long it would have otherwise blocked or confused train/releng/scap. [21:23:46] what is the first thing you noticed? [21:23:58] how did it start for you [21:24:27] 21:20 <+icinga-wm> RECOVERY - mediawiki-installation DSH group on mw1415 is OK: OK https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups [21:29:51] mutante: https://phabricator.wikimedia.org/T310225 is how it started for me, or rather the spike in Logstash via dancy [21:30:07] fatal db error about missing db column [21:37:13] ok.. so ... one might argue it's a monitoring notification issue though.. I _did_ see the monitoring alert and that made me ping. and that in return caused that ticket [21:37:25] doesn't mean I would always watch IRC though of course [21:38:43] so imho it comes down to.. either notification methods of alerting ..or workflow change where dcops can set a status [21:38:54] but even that wouldn't help alone because there would be no "correct" status for it [21:39:46] let me actually pool that now [21:51:02] https://phabricator.wikimedia.org/T310225#7990630 [21:51:15] closed the hardware repair ticket, left summary on the other one [21:51:35] server pooled like before it broke