[00:00:17] Yeah becuase paladox quit which makes my decision more likely. But if you must know John, it's you. Your criticism for everything we do (and I do understand, but it isn't all that necessary at some of it was misunderstanding) and the threat today to do the vote of no confidence with the community. [00:00:36] There is no respect for what we do anymore. [00:00:43] I won't be apart of that. [00:02:34] There is a lot of respect, there is just none among SRE. There’s no discipline, no respect for the community, no respect for peers, no respect for experience. SRE had no respect, and it’s clear here when every time critisms is raised you brush it off, blame others and choose to ignore and not attempt to work or understand with others. [00:02:46] i left because SRE had no respect and was toxic [00:02:56] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:03:58] And the threat of, “if I don’t get my way I’m leaving” is the problem. Rather than stay, work, and build, people wish to leave because they’re challenged - not because they care. [00:05:02] I have raised the same concern for weeks now, every time SRE go ‘we will do better next time’ to just do the same shit again with no remorse or consideration [00:05:44] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.86, 6.85, 5.60 [00:06:32] And I will likely leave because of you're toxicity about this. I had the upmost respect for you John, but now, I don't at all, I valued your opinions and advice, I don't anymore. I have no faith in you whatsoever anymore and as long as you hold any position here (including Stewardship), I'm not sure I will remain. You went today and made a threat against SRE, because that is exactly what it is, a threat. I won't be apart of [00:06:32] the toxic environment Miraheze has become in the recent days. [00:07:20] I made a threat to ensure the community is respected. [00:07:34] SRE don’t care anymore, and you are displaying that perfect [00:07:42] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.33, 5.97, 5.44 [00:08:10] Don’t like it? Fix it. Don’t be a problem then complain when it’s challenged. [00:10:14] If you really cared, you’d challenge the processes rather than threaten to resign when challenged. You’re the one making the threats as well. [00:10:50] Yes I am. I won't deny that. [00:13:03] After all, you’re saying if I ask the community about this problem - you’ll resign. You’re using your position to attempt to suppress the community. That’s ridiculous and deplorable of you. [00:13:06] [02puppet] 07MacFan4000 opened pull request 03#1783: remove paladox sre access per resignation - 13https://git.io/JZhWz [00:13:56] Either grow up and fight for what you care about, or don’t suppress the community with positional play just to get your way. [00:14:17] To clarify here’s my resignation statement: [00:14:21] https://www.irccloud.com/pastebin/1Ag80abr [00:15:00] It praises what John is doing. And I leave it at that. Not resigning because he’s holding us to account but more praising that but resigning for other reasons. [00:15:04] ^ that shows a member of SRE agrees the current environment SRE have is worrying and a problem [00:17:13] I am absolutely not saying that right now. Whether you do or don't at this point I don't think matters. I don't even care if you do your no confidence RfC anymore. I did not want paladox to go, but maybe a point is made by you John. But I don't give a shit anymore. I tried to be here to help, maybe I made things worse, if more people see the issue with SRE, then maybe there is one. But everything you mentioned was my doing, [00:17:13] so be against me, not SRE as a whole. I may have been wrong here and will admit that. [00:18:39] SRE has a problem as a whole, it is not just you. Don’t take it personally. It’s been a problem for months and years, one I have tried so hard to fix while in SRE, and one I continue to fight outside. [00:19:21] There is a weak, barely non existent support and leadership structure, there’s no discipline and there’s weak voices that saying ‘we won’t do that again’ before doing it the day after. [00:19:53] There is a serious cultural issue within SRE that likely stems from technical volunteer burnout, on the one hand, and leadership (or lack thereof), on the other. There may also be individual personnel issues, but those are secondary to the issues with SRE as a whole. Miraheze faces serious existential questions, which need to be addressed, ideally by the board, I think [00:20:59] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.11, 17.77, 14.52 [00:21:08] I'd like to hear what you all think needs to be changed, and how you think we'd best go around fixing it. [00:22:08] Voidwalker, that's a very good question. I think it should begin with a board review of each position to determine if we still have the best person within each role. I think we have the right volunteers, but not necessarily the right person in each role [00:24:01] I kind of have the same thoughts as Void here. [00:24:01] From there, do we still have the right structure within SRE that best suits our needs? SRE restructured recently, and I think the restructure was generally the right move, but as always, does it need some further changes? [00:24:36] Why not focus on fixing it rather than having...whatever that was. [00:24:49] Focus on resolving the problems [00:24:52] @Hypercane, yeah :) [00:24:54] I'm not concerned particularly with roles other than leadership. To start, we definitely need someone active as DSRE [00:25:00] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 12.46, 16.75, 15.06 [00:25:15] I'm sure you guys can figure something out [00:25:21] This is in progress but unlikely to be concluded before the next board meeting [00:25:36] I have raised my concerns numerous time both while inside SRE and outside, they are clear that there is no policy or guidance on a majority of things, there’s no discipline, there’s no volunteer welfare, there’s no recruitment, there’s no development. There’s no leadership that leads, only leadership that follows. [00:25:39] Voidwalker, well, yes, that's what I'm saying [00:26:46] Well, I still think things could be solved more...peacefully. Maybe I'm just a fool for thinking but that's my thoughts on the matter. [00:26:57] thinking that rather [00:27:27] Voidwalker, and Owen, I also wonder if it could be useful to increase the frequency of board meetings to every 2 months rather than every 3 months, particularly since the board needs to be more involved in issues now? [00:27:31] I think so too, but at the present, I don't want to ask them back when it's clear they need to step away, at least for a time [00:28:20] JohnLewis: I understand your points now. I am truly sorry for all that has happened. I just don't know what to do anymore. [00:28:23] Was that in response to me, Void? Sorry I lack context clues sometimes [00:28:38] yeah [00:28:43] alright [00:29:27] Universal_Omega: there’s nothing *you* can do, that’s my point. Those who can, don’t seem interested in doing so, and because it’s been so long, it’s become a norm to do these things [00:29:58] Just wanted to pop in and say I love all of you guys for all the awesome work you are doing, both past, present and future. [00:30:32] Miraheze is pretty amazing I'll admit. I cannot imagine all the work it took to create such a thing., [00:30:54] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [00:33:03] I raised with SPF the fact there were members of SRE that were struggling, I got told that’s not true. Paradox above has proven that it was true because his message states that it’s been an on going problem. [00:34:34] Yes it is true. I am so sorry to both you JohnLewis and the community that it took paladox to resign and his resignation notice for me to see the real issue here. [00:35:15] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.21, 6.60, 5.52 [00:36:18] Seeing the problem is the first step, as I said it’s nothing against you personally, you’ve always done good work - but now you see the problem we’ve both been seeing all this time, maybe you could take the first step to ensuring others also see it as you’ll have a different perspective to both myself and paladox [00:36:46] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.69, 7.56, 6.67 [00:37:15] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.85, 5.99, 5.43 [00:38:36] To be clear, I appreciate the work and efforts of all SRE volunteers. I just think there is a lack of leadership and direction, which is problematic for the team. That's why I say I think we just need to make sure we have everyone in the right roles, and likely need to play "musical chairs" as a first step. :) [00:38:43] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.57, 6.45, 6.37 [00:39:04] I appreciate everyone honestly. [00:39:18] Hopefully everything can get resolved eventually. [00:39:26] It won't happen overnight of course, but eventually [00:39:29] @Hypercane, well that, too, yeah. I'm just speaking about SRE, but yeah [00:39:38] I know [00:39:51] 👍 [00:39:54] well, I'm worried that this happened just before the update to 1.36, [00:40:07] there's still a lot of unresolved things [00:40:24] Lake: s/before/after [00:40:38] oh yeah, after, lol sorry, confused [00:40:43] MH-Discord: s/before/after [00:40:43] dmehus thinks MH-Discord meant to say: well, I'm worried that this happened just after the update to 1.36, [00:40:48] heh, np @Lake [00:47:00] Just to let everyone know, I will be stepping back on my activity, but not resigning. I now realise JohnLewis's points where valid, and I am just sorry it took paladox to resign for me to see that. This whole situation is really heartbreaking to me and I hope we can resolve the issues raised in a peaceful way. I just wish the issues could've been corrected / realised before we lost 2 extremely valuable members of SRE because [00:47:00] of them. JohnLewis has done tremendous work as SRE since the very beginning and paladox has been a huge asset to both Miraheze as a whole and myself personally. It's extremely sad to loose yet another very valuable volunteer. [00:47:36] ❤️ [00:48:51] Thank you, and yes, feel free to step away for a while, anyone. You come first. [00:49:15] I mean, as tough as it may be.. perhaps this was something that needed to happen eventually. [00:49:45] Even if nobody wished it would be like this. [00:50:22] I just don't know what we are going to do without both paladox and JohnLewis here. [00:52:19] I'm thinking of stepping up, perhaps temporarily, but to handle infrastructure tasks. It's what I do for my day job anyway. [00:53:57] Voidwalker, that would be very helpful if you're able. Any time you commit to helping in that area would be immensely appreciated. :) [00:54:04] even if just temporarily, too [00:55:15] I want to say "You'll figure something out", but there no doubt that it is a difficult situation, and there might be rough times ahead. But for you guys (not all heroes wear capes), and of course for all of us who dabble in wiki shenanigans, I hope things will sort themselves out one way or another. [00:55:25] tbh, the biggest problem on any staff role we have on MH is that we rarely find anyone interested on doing something [00:55:33] unfortunately, MH is quite of a niche thing [00:59:43] A wild 10$ donation appears. [01:01:10] ty Blackwolfe :) [01:01:41] I'd buy you all a round of beers in person if I could too. [01:05:15] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.99, 6.56, 5.67 [01:06:00] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.66, 7.27, 6.33 [01:06:08] 👍 [01:07:15] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.24, 5.75, 5.49 [01:09:59] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.07, 6.14, 6.08 [01:15:16] https://phabricator.miraheze.org/T7467 [01:18:50] Thanks for voluntering, @Void :) [01:21:37] That is, if you know @Void, @Doug and @Reception123 (Away)'s ages. [01:22:53] I mean, worst case, I buy them a glass of milk. [01:23:42] I'm assuming Void, Doug and Reception123 are in their 20s. [01:25:08] Seeing as Voidwalker is stepping up and there’s discussions regarding leadership I’ve decided to stay [01:25:41] ❤️ [01:26:22] [02mw-config] 07MacFan4000 closed pull request 03#3961: remove paladox access to staffwiki - 13https://git.io/JZhON [01:26:33] [02puppet] 07MacFan4000 closed pull request 03#1783: remove paladox sre access per resignation - 13https://git.io/JZhWz [01:26:35] paladox, oh, that's great to hear then, thank you. :) [01:30:44] We really need some changes around here. As for any no confidence vote against SRE, even as a member, I would likely weakly support that at this point, as I understand what the issue is now, if only to show things need to change around here. [01:32:12] Thank god @Void is actually showing some actual leadership, as I would consider him as a leader more than the others. John comes in second, @Doug comes in third, and finally, NDKilla (who rarely does anything, but does monitor the server and the wikis independently). [01:32:21] We need to stabilize things as they have been in freefall for quite some time (which I am only now realising). We need official policies for half the stuff SRE are responsible for for one. [01:32:33] IMO, a vote would be redundant at this point unless it actually helps us get this done [01:34:07] I think Miraheze as a whole should update their policies whenever necessary, like towards the end of the year for example. [01:34:40] Voidwalker: I think maybe it is necessary if only to get the communities side/input/ what needs to change. [01:35:17] Not a bad idea, actually, as a jumping off point [01:36:17] We can also use it to get the public opinions of current (or past) SRE members on what they would like to see [01:36:23] What policy changes did you have in mind @Universal Omega? [01:36:58] @DarkMatterMan4500: Policies need written, not changed. [01:37:08] Oh. [01:37:09] An anonymous survey would be a start. [01:37:24] An anonymous survery? Ehhhh, no. Pretty sketchy business. [01:38:10] We could ask if it's necessary, or otherwise discuss a method to privately raise concerns [01:38:41] Which ones were you referring to? The anonymous survey, or something else? [01:38:50] We do do surveys, which I think are anonymous? I could be wrong [01:38:57] anonymous survey [01:39:11] Figures. [01:40:06] Surveys would be helpful, whether with the community, or as an internal survey among the SRE team. Depends on what's being surveyed, but yeah, that is also a good idea [01:40:53] It's not like we're filling out surveys just to get a song published or something like that. [01:42:48] In this case, more focused questions relevant to the topic at hand. [01:47:59] Would be two surveys I suppose. One for the community, with questions about peoples experience with getting support, how they feel tasks are handled etc. And one in-team kind of with uh.. some other questions? (I dunno) [01:48:35] I'm not actually convinced of a survey, so much as an open ended discussion [01:49:03] Open ended discussion seems better I think [01:49:41] I think so also. [01:50:32] Hmmm, I'd say that both options should be on the table. [01:50:53] I still think a more focused community survey would hold value though. [01:51:53] Besides, it doesn't have to be one or the other. [01:51:54] Voidwalker, oh I mean for these large picture issues, yes, an open-ended discussion is better. I just meant surveys on minor points, like needed changes to SRE internal policies or procedures, could be helpful, potentially [01:54:02] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [02:00:26] I think that with all that's happened today, lessons were learnt, lessons we needed to learn. Things must change in the future. SRE must change and be more open with the community. I can make a pledge from myself, I will no longer personally make any changes that effect wikis without at least making wikis aware of the changes if it is needed/non-emergency. I do think that SRE as a whole owes an apology to the community for [02:00:26] all that's happened. But I can only offer a personal apology and personal promise for the future. [02:14:48] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [02:14:50] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.01, 6.19, 5.76 [02:16:44] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.51, 6.23, 5.82 [02:23:40] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [02:39:31] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JZj35 [02:39:32] [02miraheze/mediawiki] 07paladox 03da22f21 - Update ApprovedRevs [02:47:19] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JZjnJ [02:47:21] [02miraheze/mediawiki] 07paladox 0316588ba - Update DataTransfer [02:58:22] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [03:07:17] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [03:21:09] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [04:44:18] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.77, 1.32, 1.06 [04:46:18] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.70, 1.05, 0.99 [05:27:03] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [05:40:53] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [05:56:44] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [06:03:39] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [06:40:20] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [07:08:00] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [07:37:41] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [07:56:46] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JnvV7 [07:56:47] [02miraheze/CreateWiki] 07translatewiki 030d21ddc - Localisation updates from https://translatewiki.net. [07:56:50] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JnvVN [07:56:52] [02miraheze/MirahezeMagic] 07translatewiki 0306688d3 - Localisation updates from https://translatewiki.net. [07:57:45] miraheze/CreateWiki - translatewiki the build passed. [07:57:50] miraheze/MirahezeMagic - translatewiki the build passed. [08:12:20] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [08:21:14] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [08:55:52] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [09:04:45] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [09:39:24] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [09:55:17] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [10:02:07] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [11:16:09] hi [11:16:34] PROBLEM - ping6 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 310.70 ms [11:20:12] [02mw-config] 07Southparkfan commented on pull request 03#3926: Set WebChat to default to Libera; remove freenodeChat - 13https://git.io/JnJTg [11:22:15] [02mw-config] 07Southparkfan edited a comment on pull request 03#3926: Set WebChat to default to Libera; remove freenodeChat - 13https://git.io/JnJTg [11:34:15] Hi Southparkfan [11:43:26] PROBLEM - www.hoolehistoryheritagesociety.org.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.hoolehistoryheritagesociety.org.uk' expires in 15 day(s) (Wed 30 Jun 2021 11:38:37 GMT +0000). [11:44:58] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JnJGu [11:44:59] [02miraheze/ssl] 07MirahezeSSLBot 03f99f167 - Bot: Update SSL cert for www.hoolehistoryheritagesociety.org.uk [12:03:37] RECOVERY - www.hoolehistoryheritagesociety.org.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'www.hoolehistoryheritagesociety.org.uk' will expire on Sun 12 Sep 2021 10:44:51 GMT +0000. [12:04:02] fyi ^ my comment on the github PR [12:05:56] [02mw-config] 07Southparkfan edited a comment on pull request 03#3926: Set WebChat to default to Libera; remove freenodeChat - 13https://git.io/JnJTg [12:07:00] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03revert-3926-Universal-Omega-patch-1 [+0/-0/±2] 13https://git.io/JnJRc [12:07:01] [02miraheze/mw-config] 07Reception123 03bd4da70 - Revert "Set WebChat to default to Libera; remove freenodeChat (#3926)" [12:07:03] [02mw-config] 07Reception123 created branch 03revert-3926-Universal-Omega-patch-1 - 13https://git.io/vbvb3 [12:07:29] [02mw-config] 07Reception123 opened pull request 03#3962: Revert "Set WebChat to default to Libera; remove freenodeChat" - 13https://git.io/JnJRg [12:07:34] [02mw-config] 07Reception123 closed pull request 03#3962: Revert "Set WebChat to default to Libera; remove freenodeChat" - 13https://git.io/JnJRg [12:07:36] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JnJRa [12:07:37] [02miraheze/mw-config] 07Reception123 0333e2946 - Revert "Set WebChat to default to Libera; remove freenodeChat (#3926)" (#3962) [12:07:39] [02miraheze/mw-config] 07Reception123 deleted branch 03revert-3926-Universal-Omega-patch-1 [12:07:40] [02mw-config] 07Reception123 deleted branch 03revert-3926-Universal-Omega-patch-1 - 13https://git.io/vbvb3 [12:08:00] [02mw-config] 07Reception123 deleted branch 03Reception123-patch-3 - 13https://git.io/vbvb3 [12:08:01] [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-3 [12:08:03] [02mw-config] 07Reception123 deleted branch 03revert-3840-revert-3839-patch-161 - 13https://git.io/vbvb3 [12:08:04] [02miraheze/mw-config] 07Reception123 deleted branch 03revert-3840-revert-3839-patch-161 [12:08:06] [02miraheze/mw-config] 07Reception123 deleted branch 03revert-3839-patch-161 [12:08:07] [02mw-config] 07Reception123 deleted branch 03revert-3839-patch-161 - 13https://git.io/vbvb3 [12:08:13] Reception|away: doint forget to re-add freenode to the csp whitelist [12:08:20] (wanted to clarify my response further) [12:08:25] paladox: thanks for reminding me, will do [12:08:33] miraheze/mw-config - Reception123 the build passed. [12:08:37] miraheze/mw-config - Reception123 the build passed. [12:09:06] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JnJ0q [12:09:08] [02miraheze/puppet] 07Reception123 03cd3d3f0 - re-add freenode to the CSP whitelist per revert of the original PR [12:10:00] [02puppet] 07MacFan4000 opened pull request 03#1784: Add void to sre per T262 - 13https://git.io/JnJ0o [13:25:56] can someone from SRE deal with the email i sent to sre@ please [13:26:00] thanks and bye [13:40:59] Reception|away Southparkfan: ^^ [13:44:07] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [13:51:02] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [13:53:17] [02puppet] 07MacFan4000 edited pull request 03#1784: Add void to sre per T7647 - 13https://git.io/JnJ0o [13:53:41] [02puppet] 07MacFan4000 edited pull request 03#1784: Add void to sre per T7467 - 13https://git.io/JnJ0o [14:06:53] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [14:13:15] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.95, 6.13, 5.10 [14:15:15] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.68, 5.46, 4.97 [14:19:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.82, 6.44, 5.71 [14:22:14] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.19, 6.84, 5.80 [14:24:14] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.33, 6.32, 5.75 [14:25:25] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.04, 6.45, 6.04 [14:30:56] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.63, 7.07, 6.26 [14:32:56] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.00, 6.41, 6.10 [14:35:10] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.76, 7.63, 6.45 [14:35:44] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.53, 7.20, 6.22 [14:37:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.07, 6.92, 6.33 [14:37:41] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.83, 6.64, 6.13 [14:39:01] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.03, 6.31, 6.18 [14:55:45] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.01, 6.75, 6.28 [14:57:44] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.99, 6.42, 6.21 [15:04:14] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.82, 6.53, 5.94 [15:04:53] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.31, 6.61, 6.17 [15:06:14] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.44, 6.39, 5.96 [15:06:50] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.60, 6.33, 6.10 [15:07:56] Reception|away: Southparkfan: please remove John from stewards@ :( [15:16:39] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.75, 8.25, 7.01 [15:17:10] [02puppet] 07MacFan4000 reopened pull request 03#1783: remove paladox sre access per resignation - 13https://git.io/JZhWz [15:17:36] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.18, 7.05, 6.65 [15:17:38] [02mw-config] 07MacFan4000 reopened pull request 03#3961: remove paladox access to staffwiki - 13https://git.io/JZhON [15:18:40] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.93, 7.90, 7.02 [15:19:08] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.15, 8.23, 6.93 [15:19:31] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.06, 7.35, 6.80 [15:20:39] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.56, 9.23, 7.69 [15:21:24] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.09, 9.12, 7.29 [15:21:27] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.54, 6.74, 6.64 [15:22:40] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.86, 7.86, 7.37 [15:24:57] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.85, 7.69, 7.24 [15:25:18] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.44, 7.84, 7.16 [15:28:48] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.83, 6.35, 6.77 [15:29:15] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.16, 6.58, 6.76 [15:30:11] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.40, 7.44, 6.97 [15:32:42] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.32, 7.42, 7.27 [15:34:02] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.11, 6.33, 6.62 [15:34:39] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.00, 8.29, 7.30 [15:34:41] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.31, 6.95, 7.12 [15:35:15] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.64, 6.81, 6.76 [15:36:34] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.78, 7.56, 7.15 [15:36:59] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [15:37:15] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.49, 6.44, 6.64 [15:40:25] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.05, 6.41, 6.77 [15:42:41] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.11, 6.21, 6.74 [16:04:49] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.77, 6.64, 6.20 [16:06:45] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.76, 6.44, 6.19 [16:20:35] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [16:34:27] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [16:39:20] I got this [16:39:21] Error 429 Varnish Rate Limit Exceeded [16:39:21] Varnish Rate Limit Exceeded [16:39:21] Guru Meditation: [16:39:21] XID: 251683393 [16:39:21] Varnish cache server [16:40:42] Bukkit, thanks. Can you describe what you were doing immediately preceding that error in terms of edits, rate, etc., and which wiki? [16:42:21] I started my laptop, and went on memeswiki, then got the Varnish Rate Limit [16:43:07] hrm, no edits you tried to make, or search queries in the wiki search box? [16:43:15] no [16:44:05] Is it still persisting? If so, I'd suggest creating a High priority Phabricator task detailing this. If not, I'd still suggest creating a Normal priority Phabricator task so this can be traced and looked into [16:45:41] Reloaded and got this [16:45:42] 404 Not Found [16:45:42] nginx [16:46:15] Not sure if it's a server sided issue of client sided [16:47:59] say/of/or [16:48:05] meant/of/or [16:49:48] wait I misspelled /wiki/ nvm [16:50:14] yeah still get varnish rate limit [16:50:17] [02puppet] 07Reception123 commented on pull request 03#1783: remove paladox sre access per resignation - 13https://git.io/JnTaa [16:50:20] [02puppet] 07Reception123 closed pull request 03#1783: remove paladox sre access per resignation - 13https://git.io/JZhWz [16:50:22] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JnTaw [16:50:23] [02miraheze/puppet] 07MacFan4000 03f2453bd - remove paladox sre access per resignation (#1783) [16:50:57] !log removed paladox from github [16:51:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:51:19] Reception, can you make a timeline on phab for me if u have the time [16:51:39] Bukkit: I'm busy right now, please do create a task requesting it and I'll look when I have time [16:51:46] Alright [16:53:09] !log removed paladox from LDAP/changed password [16:53:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:53:53] Are you gonna take away his Phab administrator? [16:54:23] yes, that is part of off-boarding [16:54:31] k [16:55:02] !log removed paladox from graylog [16:55:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:55:42] !log removed paladox from grafana [16:55:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:57:02] !log removed paladox from matomo [16:57:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:02:42] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.93, 6.25, 5.64 [17:03:51] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 13.90, 8.72, 6.38 [17:04:09] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [17:04:17] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.02, 7.71, 6.12 [17:04:42] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.99, 7.63, 6.24 [17:04:49] !log added Voidwalker to GitHub [17:04:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:05:37] MacFan4000: mind resolving the conflict on Void's PR? [17:05:42] Reception|away: fyi there is an mw-config PR to remove Paladox from staffwiki [17:05:44] Ok [17:05:51] MacFan4000: will merge that thanks [17:06:05] Also John needs removing from stewards@ [17:06:15] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.62, 6.81, 5.98 [17:06:21] [02mw-config] 07Reception123 closed pull request 03#3961: remove paladox access to staffwiki - 13https://git.io/JZhON [17:06:22] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JnTiS [17:06:24] [02miraheze/mw-config] 07MacFan4000 03c259eb0 - remove paladox access to staffwiki (#3961) [17:06:40] MacFan4000: will do that too [17:06:43] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.09, 6.55, 6.01 [17:06:47] thanks again for reminding me [17:07:23] miraheze/mw-config - Reception123 the build passed. [17:07:27] !log added void to sre mails [17:07:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:08:16] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.43, 6.25, 5.87 [17:08:28] [02puppet] 07MacFan4000 synchronize pull request 03#1784: Add void to sre per T7467 - 13https://git.io/JnJ0o [17:08:34] there we go [17:09:42] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.44, 7.78, 6.70 [17:09:43] !log created graylog account for void [17:09:45] MacFan4000: thanks [17:09:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:10:04] !log removed john from stewards@ [17:10:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:10:55] also, don't forget to remove paladox from the security/security reviewers/sre projects on phabricator [17:11:12] yup, will do [17:12:02] oh, and badges too :) [17:12:11] just did that :) [17:13:22] https://phabricator.miraheze.org/project/members/2/ needs paladox removed and void added [17:13:36] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.31, 6.58, 6.47 [17:15:10] [02puppet] 07Reception123 closed pull request 03#1784: Add void to sre per T7467 - 13https://git.io/JnJ0o [17:15:12] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JnTMq [17:15:13] [02miraheze/puppet] 07MacFan4000 03e8409d0 - Add void to sre per T7467 (#1784) [17:16:41] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.74, 7.18, 6.52 [17:18:00] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: rDNS OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org [17:20:43] Reception|away: void needs flags in here + -security + wherever else [17:20:56] let me get to that [17:21:56] PROBLEM - db12 Puppet on db12 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:22:20] we should probably also remove paladox flags in -offtopic and -overflow [17:22:41] PROBLEM - cp12 Puppet on cp12 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:23:13] PROBLEM - mon2 Puppet on mon2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:23:23] PROBLEM - mem2 Puppet on mem2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:23:44] MacFan4000: ^ that seems to be an issue with your PR [17:23:53] let me check [17:24:00] PROBLEM - db13 Puppet on db13 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:24:25] hmm [17:24:37]