[12:43:49] reedy, I am asking Inductiveload to come here so you don't have a dumbwaiter in between [12:45:23] Reedy guessing that video2commons is what they were using for wikimania [12:45:54] inductiveload, https://commons.wikimedia.org/wiki/Commons:Video2commons [12:47:17] I'm honestly not sure if that will work with non video [12:47:43] Speaking off wmcloud upload tools [12:47:46] https://phabricator.wikimedia.org/T287241 [12:47:57] https://phabricator.wikimedia.org/T292096 [12:48:23] Files uploaded via the ia upload tool hang about there after failure [12:50:17] as mentioned, if we could get a pull from ia-upload.wmcloud.org that would be useful as the push has been failing or unreliable [12:51:57] * urbanecm waves to sDrewth, inductiveload and Reedy [12:52:05] Hi urbanecm [12:52:25] (restate) as mentioned, if we could get a pull from ia-upload.wmcloud.org that would be useful as the push has been failing or unreliable [12:52:46] https://ia-upload.wmcloud.org/ shows that we have the files available [12:53:01] after the djvu have been created [12:53:26] and I am guessing failed to upload [12:53:35] in my experience, URL uploads tend to be bit more reliable, so doing T287241 would likely help [12:53:36] T287241: Add https://ia-upload.wmcloud.org to the wgCopyUploadsDomains allowlist of Wikimedia Commons - https://phabricator.wikimedia.org/T287241 [12:53:38] can do that later in the day :) [12:54:40] Of course, if it's a large file, downloading from a slow source... It might end up timing out too :D [12:55:07] copying it to ia-upload.wmcloud.org will likely be needed [12:55:11] to speed it up [12:55:32] a scratch storage should be reasonably easy (&fast) with cinder-based storage [12:56:57] the vast bulk of files can go via ia-upload as we are processing them,. inductiveload any reason we cannot default that if the djvus already exist, and be pending? [12:57:29] sDrewth: the djvus in that phab ticket are homemade [12:58:05] then you can fake them into the queue [12:58:05] They're only at the IA because I need to put them somewhere to request SSU [12:58:35] I can probably do that [12:58:43] Upload to in a then prod IA-Upload [12:59:22] My upload script is getting some wierd shit in it :D [12:59:28] that gives us control and responsiblity of getting them close to the heart [12:59:54] Sure I can do that (or try to) [13:00:16] Wonder if I can drive IA-Upload from python [13:00:23] and from there we can try via the allowlist method. and if that fails we can request database pull [13:00:30] Will need to be oauthed [13:00:53] please do not put the files in a restricted location [13:01:02] that makes them impossible to SSU them [13:01:18] What's a restricted location? [13:01:28] >Length: 754984992 (720M) [image/x.djvu] [13:01:36] * Reedy wonders how many more of these files are going to be much larger than 200M [13:02:01] inductiveload: for SSU to be possible, the files need to be in a location that doesn't have any authentication [13:02:06] I think the first couple are larger before I turned on the compressor [13:02:53] 21 was > 700. 22 was ~200. 23 is > 700 [13:03:24] I think 21, 23, 24 went first for some reason [13:03:41] urbanecm, so web accessible like ia-upload is now [13:03:45] ? [13:04:09] well, yeah. We need a direct link that will immediately download the file to download [13:04:16] that makes things like Google Drive ineligible [13:04:31] (as you need to click the "skip antivirus" button for large files) [13:04:46] but if we can get them onto a WMF server that makes things easy for you [13:04:51] easier [13:04:56] IA-Upload provides that [13:05:03] we still need a web link though [13:05:20] production cannot (directly) connect to cloud servers [13:05:27] The link actually exists already on the IA upload failure list [13:05:58] You can download the djvu to your own machine, but then you'll likely hit an upload failure yourself so there's not much point [13:06:21] * sDrewth struts as he chunk uploaded wtih success [13:06:27] There is a ticket for being able to access the description for the file too somewhere [13:06:33] * inductiveload rummages [13:07:12] https://phabricator.wikimedia.org/T286702 [13:08:05] T286702 [13:08:06] T286702: IA-Upload: allow access to the file description of a task - https://phabricator.wikimedia.org/T286702 [13:10:48] The Strand Magazine (Volume 23).djvu 38%[===================================> ] 276.23M 179KB/s eta 15m 12s [13:10:50] So fast [13:11:08] It's about that speed to upload them too [13:11:23] So it took literally days to upload that list [13:11:42] seems skipping IA will be a winner [13:11:46] heh [13:11:48] Bearing in mind each of them also tried to upload to commons first [13:11:52] Or having IA pull from WMF [13:12:13] we would still have those cxn Reedy? [13:12:26] I know we used to back in the days of my stewardry [13:12:27] ? [13:12:40] contacts inside IA [13:12:42] It's all a very rube Goldberg way to skirt the upload bug though [13:13:09] Go find people to fix that then ;D [13:13:21] I mean, I have tried [13:13:49] summer coders? [13:13:50] Add "Make large uploads reliable and not suck" to the next community tech wishlist [13:14:45] I mean, if someone could vaguely indicate what the problem is, I can have a stab at fixing it [13:15:16] I'm not sure. But I'd be surprised if it was just one issue [13:15:23] I suspect it'll be numerous things working together [13:15:41] Reedy, who would we ask for the error/failure logs? [13:15:45] if there is such a thing [13:15:54] But I honestly don't even know where to start, and setting up a clone of wmf prod to play with will probably take me.... Some time [13:16:11] And even then, who knows if it would have the same problem [13:16:13] why set up a clone? [13:16:14] that...doesn't look like worth it [13:16:36] Because how else would I debug it? [13:16:36] sDrewth: there are logs, but in case of timeouts, they...mostly just tell you something timed out [13:16:42] ^ [13:16:58] There might be simultaneous other logs from other systems that could help, but correlating can be hard [13:17:03] y. e. a. [13:17:34] I don't have access to any system (and since idk wtf I am doing at the best of times, I should not be given such access) [13:17:50] I only learned PHP like a couple of months ago [13:18:04] inductiveload: beta.wmflabs.org is the best WMF-like clone that exists, but it is VERY VERY far from actual parity [13:18:08] There really should be a team at WMF that should own this stuff, but unfortuantely there isn't [13:18:13] (anymore) [13:18:21] I'd say it's impossible to create a clone [13:18:24] Ok that's a lie, I did my stepmums website's wordpress skin [13:18:44] urbanecm: miraheze mostly use our puppet stuff AIUI [13:19:06] not familiar with miraheze setup, granted [13:19:12] but i doubt it's the exact same setup [13:19:25] It's not [13:19:32] they have their own repo, with a few modules imported from us I think [13:19:32] so we accept that the upload bug is not happening anytime soon [13:21:42] well honestly, I don't really care what the solution is, I'll do whatever people tell me to do [13:21:53] 👍 [13:22:02] I like the art of what fworks [13:22:48] currently the script uploads to the IA, then files a phab ticket for an SSU, but I can ssh to the moon or print it out on microfilm and post it, as long as there's a python library [13:23:18] and actually even if there isn't I'll learn some other language and shell out to it if I have to [13:25:08] for me, the preferred solution would be to put the file to the ia-upload WMCS project itself, and put a link to that in the SSU ticket [13:25:24] to avoid downloading from IA [13:25:28] which is very slow apparently [13:26:24] ok, so I wonder if I can prod that from my scripts [13:26:49] it would make sense for the IA-Uplaod to actually accept a zip of images too [13:27:04] since that's how it works when there's no DJVU anyway [13:27:12] inductiveload, can't you pull from IA-upload shell login? [13:27:30] i don't have Ia-upload access [13:27:35] ah [13:27:39] aha! [13:27:56] then why don't we do it from wikisource-bot shell on toolforge [13:27:56] then you should likely pair with whoever maintains that part to allow local file storage [13:28:00] or that [13:28:14] You have access, and it has an html window [13:28:17] exact location doesn't matter much, as long as it's somewhere at WMCS :) [13:28:35] urbanecm: https://ia-upload.wmcloud.org/ [13:28:55] the "Download DjVu" links: are they OK? [13:29:07] appear so [13:29:15] ok cool [13:29:16] i thought they point to archive.org for some reason [13:29:41] no, those ones are DJVUs that the tool made itself [13:29:41] sounds as if we can have multiple paths, via IA-UPLOAD if fail then inhale [13:29:55] if we create externally, then via WMCS wikisource-bot account [13:30:52] urbanecm, at IA-UPLOAD we have already brought them in from IA and converted to djvu or have as pdf [13:31:01] i see [13:31:25] depending on what the user wishes to work upon [13:31:25] yeah, if the DJVU is coming direct from the IA, it copy-uploads by URL [13:31:51] that usually works, because IA DJVU are compressed with black magic and are much smaller [13:32:11] (i.e. a LuraTech MRC compressor) [13:32:44] sadly they no longer make DJVUs or we would probably not be talking about this [13:33:13] i wonder how much a copy of lura tech is [13:34:05] can the WMF stretch to a license so we can provide DJVUs that aren't just c44-ed JPEGs (at 5 times the size)? [13:35:01] inductiveload, we probably just have to do a grant proposal [13:35:12] https://meta.wikimedia.org/wiki/Grants:Project/Rapid might be a way [13:35:21] I have no idea if they fund licenses though [13:35:27] i wooner if luratech still make it at all [13:35:55] they have a PDF compressor which I assume is the same general idea [13:36:14] they got bought by foxit [13:36:35] we can take that back to enWS and prepare the grant [13:37:07] If less than $2000 we can do a rapid grant application [13:37:25] http://djvu.com/djvu-sdk/ [13:37:32] i guess it is this now [13:38:24] hmm https://www.cuminas.jp/en/downloads/download?pid=17 [13:38:41] "Product Cathegory:" [sic] does not include linux [13:38:43] wrrryyyyy [13:39:12] inductiveload, what exactly am I asking djvu.com for? [13:39:29] i'm not sure [13:39:53] the IA used to use a LizardTech compressor, which used the LuraTech MRC compressor (AIUI) [13:40:02] something that super compacts djvus? [13:40:07] both those companies are now sold to others [13:40:09] yes [13:40:14] you know how the IA does it [13:40:45] dm your email address [13:40:57] username at gmail [13:41:10] k [13:41:13] (not very secret, it's on gerrit commits too :D) [13:41:32] I think that I probably have it somewhere [13:43:27] jesus the windows SDK is 600 MB [13:43:57] !log tools cleaning up unused kubernetes ingress objects for tools.wmflabs.org urls T292105 [13:44:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:44:01] T292105: Remove deprecated ingress objects from existing web services - https://phabricator.wikimedia.org/T292105 [13:56:48] inductiveload reedy . if we were going to install a copy of that software which would we be asking for http://djvu.com/djvu-sdk/ ? [13:57:35] Install it where? [13:57:55] Also, if it's non free, cloud won't be a good place [13:58:06] WMCM [13:58:43] I am writing to them to see whether they can gift us a license or we are going to have to write a grant [13:59:17] Even if they'll give "us" a license, it's still non free software [13:59:35] if it is tucked away on ia-upload for generating djvu files? [14:00:09] https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_Introduction#Review_the_terms_and_conditions [14:00:13] >Toolforge tools must be open source software licensed under an OSI approved license. [14:00:24] wmcloud? [14:00:32] applies to all of WMCS, not just Toolforge https://wikitech.wikimedia.org/wiki/Wikitech:Cloud_Services_Terms_of_use#What_uses_of_Cloud_Services_do_we_not_like? [14:00:40] okay [14:01:01] >Exceptions to this list of prohibited uses can be requested for limited circumstances, such as testing compatibility with proprietary software. To request an exception, please submit a proposal to the Cloud Services administrators. [14:01:02] it just says toolforge [14:01:10] >You agree not to use Wikimedia Cloud Services for any of the following (“Prohibited Uses”): [14:01:13] >Proprietary software: Do not use or install any software unless the software is licensed under an Open Source license. [14:01:29] https://wikitech.wikimedia.org/wiki/Wikitech:Cloud_Services_Terms_of_use#What_uses_of_Cloud_Services_do_we_not_like? says "Wikimedia Cloud Services" [14:01:57] It's not a blanket no, you can ask for an exception. But it's not guaranteed [14:02:26] okay, so there is another step in the process [14:02:51] Yeah [14:03:26] I would argue that "there is no free software available that can do this" (I've no idea if this is true, I haven't looked) would be reasonable grounds to request an exception [14:03:39] that is fine, I will get the information from them; so back to my question, what would I be asking for? [14:03:57] BTDT when I was in public service in getting software installed [14:04:09] and refreshed without going to tender again [14:12:00] is there a central log file roll and gzip script? [14:12:12] on toolforge [14:18:42] what is the canonical way to scp a file to a toolforge tool? [14:19:22] scp login.toolforge.org:? [14:19:36] you should be able to write to tool's directory directly (iirc, it's /data/project/toolname), but that file will be owned by your personal account, not tool account [14:19:47] you can run `take ` as tool at toolforge to change ownership [14:20:08] righto, must just be perms then [14:20:17] yeah [14:20:25] you can always scp to your home [14:20:28] and copy at toolforge [14:20:32] (manually) [14:21:32] ah, there we go, missing a +w [14:21:37] sorry for being dense [14:26:17] it is /data/project/wikisource-bot [14:27:03] yep, I got it [14:27:12] <= slow [14:27:32] * inductiveload tries to turn on dir listing [14:27:52] there are some things that I haven't done for 5+ years, on the server [14:29:36] aha [14:29:41] https://wikisource-bot.toolforge.org/ssu_request/ [14:30:31] though actually that's unlikely to actually need SSU since a 75MB will probably go through [14:31:13] very likely [14:31:16] always please try :-) [14:31:22] SSU is...costy (for people's time) [14:31:32] you can say that again [14:32:00] https://en.wikisource.org/wiki/Category:The_Strand_Magazine [14:32:01] the files that exist were successful [14:32:05] i mean it takes some human time :) [14:32:09] the files that don't, failed [14:32:35] i mean sure [14:32:57] adjusting the script to catch upload failures and upload to the IA took me like an entire day :-D [14:33:10] 🙂 [14:33:16] Hey all, ldap logins were briefly broken for most cloud-vps servers. I am pretty sure the issue is resolved but please ping me if you find something that still doesn't work. [14:33:29] The outage should have had little to no effect on actual running services [14:33:38] maybe I'll retry the commons upload a few times first [14:34:21] (and by 'ldap logins' I mean 'ssh logins' because they use ldap) [14:34:32] some binoomial thingy will tell me how often I need to try a 1/8 change of success to have >95% success [14:34:36] urbanecm/reedy, you were going to set up ia-upload to be accessible directly to comons to grab files? [14:34:53] there was nothing more that we needed to do? [14:35:27] the links like https://ia-upload.wmcloud.org/dli.ministry.15654.djvu look okay to me [14:35:28] andrewbogott, my shell login to toolforge bastion is working [14:36:03] urbanecm, yes, though for direct import to commons as required [14:36:03] great, thanks for checking [14:36:40] this one: https://phabricator.wikimedia.org/T287241 [14:36:44] https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/720058 [14:36:45] had logged on, not paid attention, had logged on again so wasn't hard [14:37:24] actually if it was wikisource-bot.wmcoud as well, that would let sDrewth and I self-service via copy-upload attempts [14:38:07] via urls like https://wikisource-bot.toolforge.org/ssu_request/The%20Atlantic%20Monthly%20Volume%20135.djvu [14:42:01] wikisource-bot is toolforge, not wmcloud [14:43:02] !log admin ran sudo cumin --force --timeout 500 -o json "A:all" "ps -ef | grep nslcd && service nslcd restart" to get nslcd happy again T292202 [14:43:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:43:08] T292202: Icinga/Getent speed check (cloudstore1008/1009) - https://phabricator.wikimedia.org/T292202 [14:45:55] !log sudo cumin "cloud*" "ps -ef | grep nslcd && service nslcd restart" and sudo cumin "lab*" "ps -ef | grep nslcd && service nslcd restart" T292202 [14:45:56] andrewbogott: Unknown project "sudo" [14:50:16] !log admin sudo cumin "cloud*" "ps -ef | grep nslcd && service nslcd restart" and sudo cumin "lab*" "ps -ef | grep nslcd && service nslcd restart" T292202 [14:50:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:50:21] T292202: Icinga/Getent speed check (cloudstore1008/1009) - https://phabricator.wikimedia.org/T292202 [15:10:43] urbanecm /reedy, just asking again before I sneak off to bed, whether there is anything further that I need to do [15:30:23] i can add to a backport slot if that helps? [15:42:28] inductiveload: yeah, that's the official way to request config deployment [15:42:43] but since it's a simple patch (the allowlist), i can try to remember to just do it myself in the evening [16:44:15] having a pcc issue. Added a pseudo-secret to labs/private in https://gerrit.wikimedia.org/r/724831 and now (12+hr after merge) pcc still isn't finding that key in https://gerrit.wikimedia.org/r/724829 . Any ideas why? [16:47:38] ebernhardson: there are a few things that could be happening. The first is that the patch may need another merge step, I'm checking that now [16:48:01] nope, wasn't that [16:48:35] the other possibility is that there's a disagreement with the hiera search path [16:49:45] do you know for sure that e.g. hieradata/common.yaml:profile::query_service::oauth_settings is getting picked up by pcc? [16:50:20] andrewbogott: the problem is it isn't, pcc reports Error: Function lookup() did not find a value for the name 'profile::query_service::oauth_consumer_secret' [16:50:33] andrewbogott: oh, oauth_settings. hmm, [16:50:50] ebernhardson: yeah, just a basic 'well does any of this work?' question [16:50:50] andrewbogott: it should be, it's pulled by the same code in the same place [16:51:02] andrewbogott: and that one is run during the prod catalog [16:51:08] can you link me to the pcc run that's failing? [16:51:31] https://puppet-compiler.wmflabs.org/compiler1003/996/wcqs2001.codfw.wmnet/index.html [16:52:07] this is mostly https://gerrit.wikimedia.org/r/c/operations/puppet/+/724829/5/modules/profile/manifests/query_service/wcqs.pp in the prod catalog it's pulling oauth_settings, in change it should pull both [16:53:24] yeah, hm [16:53:46] * andrewbogott briefly thought that wcqs was a typo for wdqs [16:54:21] following the MediaWiki/WikiMedia/Wiki* tradition of naming ;) [16:55:31] I don't think this will help but I'm going to refresh the pcc facts since these are newish servers [16:57:13] normally I would address this issue by just copying that key into a thousand other files until it works [17:01:28] Hmm, which ones? commmon.yaml would usually be the fallback [17:02:18] codfw.yaml/eqiad.yaml for starters [17:02:24] and also possibly making a profile-specific file [17:02:42] but I'm not saying that's the right solution, just that I'm frequently baffled by where hiera does and doesn't look [17:04:11] * andrewbogott still waiting for facts to sync [17:18:55] ebernhardson: I am still stumped but referring the question to a higher power (giuseppe) and will follow up if we find anything [17:19:26] ok :) I can certainly try moving it around into different files, i suppose i thought common.yaml was the easiest and most likely to be found but never know :) [17:24:51] ebernhardson: I need to step away but I would start by trying that key in private hieradata/common/profile/query_service.yaml [17:25:07] I don't see a lot of evidence that private's common.yaml is ever parsed [17:25:20] if that proves true then I encourage you to open a task seeking explanation :) [17:26:21] huh, interesting! ok will try. Thanks! [18:12:12] !log tools.bridgebot Adding #wikimedia-wikicon-na bridge (T292137) [18:12:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL [18:30:38] Folks just a heads up - looks like I caused a bit of an issue as a result of T292097 [18:30:39] T292097: Netbox info missing on some WMCS elements - https://phabricator.wikimedia.org/T292097 [18:31:33] Forward DNS entry for "cloudinstances2b-gw.openstack.eqiad1.wikimediacloud.org" got changed from 185.15.56.244 to 185.15.56.238 as a result. [18:31:38] About 5 mins ago. [18:32:27] I've removed the Netbox entry I added earlier and am re-running the "sre.dns.netbox" cookbook now to try to revert. [18:46:27] On further investigation it seems that hostname was always pointing to .238, there is a manual entry for it: [18:46:29] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/dns/+/refs/heads/master/templates/wikimediacloud.org [18:47:17] I'll reach out tomorrow and see what we can do, probably want to remove that manual one, re-add the .238 in Netbox, and work out why Netbox also has an entry against 185.15.56.244 for that name. [19:50:45] Hi! There seems to be some problem with SSL-Certificates DST Root CA X3 was valid until 30.9.2021 14:01:15 GMT and exactly at that time the trounle with my mono program on wikihistory.toolforge.org starts (see in german language https://heise.de/-6201155 ) Do I have to update some files or is this a server issue [19:51:01] -trounle+trouble [19:57:17] Wur_gl: what trouble [19:57:40] An exception occurred: WebException: Error: TrustFailure (A call to SSPI failed, see inner exception.) [19:57:59] Wur_gl: what exactly is the call too [19:58:06] Full error? [19:58:17] But likely related yes [19:59:17] https://justpaste.it/2ph78 [19:59:57] Wur_gl: what exactly are you requesting? [20:00:07] What software versions? [20:00:26] Like specfic urls you're accessing [20:01:27] I do not know if this an issue of server admins or if I have to upload install some local files. Sorry, mono is just another language and I have very, very, very little experiance [20:01:56] The Urls I am accessing are just and only the wikipedia-API [20:02:03] Ok [20:02:06] Nothing outside [20:02:13] Do you know what version of mono [20:02:16] I have a feeling that many folks will be finding that T283164 is breaking old things for them [20:02:17] T283164: Let's Encrypt issuance chains update - https://phabricator.wikimedia.org/T283164 [20:02:33] Yes [20:02:43] tools.wikihistory@tools-sgebastion-08:~$ mono -V [20:02:43] Mono JIT compiler version 5.12.0.226 (tarball Thu May 3 09:49:46 UTC 2018) [20:02:43] Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com [20:02:49] Ty [20:03:15] I will message someone and get it on the broken list [20:03:50] The exe was build on August, 28th last year [20:04:47] Do you know how to update it [20:05:07] Wur_gl: I assume your mono job runs on the grid engine? [20:05:25] looks like we pull mono from their official repos, https://github.com/wikimedia/puppet/blob/aae12e6d84a8a3be310140fcc7b70c4436e18ac2/modules/aptrepo/files/updates#L140 [20:05:50] bd808: Yes [20:06:09] root cause is likely https://community.letsencrypt.org/t/openssl-client-compatibility-changes-for-let-s-encrypt-certificates/143816 [20:06:17] the fix... yeah good question [20:06:22] I wonder if when was the last time we pulled updates to apt.wm.o, afaik that needs to be done manually [20:06:26] And yes, I have the source, I can build it on some (old) virtual Windows 7 [20:06:43] I'm guessing it's similar to https://libera.chat/news/letsencrypt-ca-expiry [20:07:43] Rebuilding the .NET tool probably won't help [20:08:02] AntiComposite: yup. The wikis use multiple ssl certs for provider redundancy and one of them is a LE cert [20:08:50] moritzm mentioned elsewhere that mono uses a custom TLS library, and last time we had to specially upgrade it https://phabricator.wikimedia.org/T194665 [20:10:21] bd808: I got the link from Mortiz :) [20:10:47] Personally: I hate mono. I have already replaced a few programs and rewritten them in php. But this one does a lot of string manipulation and most important: Uses threads to speed up, so it is a little bit harder to replace [20:10:51] legoktm: but you showed it to me! :) it's turtles all the way down [20:11:33] Wur_gl: to give us a sense of urgency, how important is this bot? [20:12:02] Not very important, [20:12:21] I'm guessing there'll be other mono bots broken and just not realised yet [20:12:53] there are not numerically many of them, but I would guess all broke at the exact same moment [20:13:38] I messaged Scott Helme so it'll probably end up on Twitter soon [20:13:50] He's collating a list of broken stuff [20:14:33] Which is half the internet from cloudflare to shopify to Facebook [20:15:58] I assume if some super critical bot breaks we can help them spin up a temporary VPS and install newer mono in it [20:16:03] I think it is likely that Toolforge things running in the ancient jessie containers (php5.6, python 3.4, python 2.7) will be busted too [20:16:05] Wur_gl: do you have a link to the source code? [20:17:35] https://persondata.toolforge.org/WikiHistory_src.zip [20:23:09] !log tools.wikivoyage Switch from php5.6 to php7.4 for webservice runtime (T292243) [20:23:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikivoyage/SAL [20:24:26] RhinosF1: I think I fixed T292243 with that ^^ [20:24:27] T292243: POI/marker disappeared on Wikivoyage maps - https://phabricator.wikimedia.org/T292243 [20:24:40] Ok [20:33:51] do we have a tracker or listing of tools still using jessie containers? [20:34:50] legoktm: https://k8s-status.toolforge.org/images/ (i'm waiting for it to load now) [20:35:37] 216 copies of toolforge-php5-sssd-web are running (that's 5.6 on jessie) [20:36:12] * bd808 won't look for the ticket where legoktm promised to get everyone to move from 5.6 to 7.2 ;) [20:36:22] * legoktm hides [20:38:55] on that k8s-status page, the node6-sssd*, php5-sssd*, python2-sssd*, python34-sssd*, and ruby21-sssd* images are all Jessie based. [21:00:02] Hmm ... i am getting authentication failures trying to log on to parsing-qa-01.eqiad.wmflabs ... i was on it just couple days back ... should i reboot the server from horizon and see if it fixes itself or did something change? [21:05:05] subbu: there was an LDAP outage earlier today that might cause that. A reboot of the instance would fix it if so. [21:05:20] ok. will do. [21:05:55] folks tried to reset all the broken LDAP caches, but it is very possible that some were missed [21:07:39] no dice ... [21:07:59] * subbu has to look up what the origin of that term is [21:09:08] still failing. [21:09:10] subbu: "error: maximum authentication attempts exceeded for invalid user ssastry" in the auth.log there. [21:09:52] I also see "Invalid user ssastry" which I would guess was the LDAP problem? [21:10:06] ya ... but, not sure why ... nothing changed in my ssh keys on my end. i can still get onto scandium.eqiad.wmnet. [21:11:00] subbu: `id ssastry` is failing on that node too. It is still having LDAP lookup issues for some reason. I'll poke around a bit more. [21:11:07] ok. thx. [21:43:26] !log tools.bridgebot Updating telegram channel id for #wikimedia-wikicon-na bridge (T292137) [21:43:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL [21:58:55] subbu: I have not awesome news about your parsing-qa-01.wikitextexp.eqiad.wmflabs instance. The problem seems to be the LE root signing cert expiration. [21:59:09] That instance appears to be Debian Stretch upgraded in-place to Debian Buster with some strange stretch-backports left in place. And we don't know how to fix it at the moment. [21:59:50] subbu: so if I asked you to replace the instance with a new one, how loudly will you complain and what help will you need? [22:00:00] ok ... given that it is already on an old debian distro ... what is involved with boot up a new instance (that one is special with a lot of compute and disk space) and transitioning over data? [22:00:47] if it were a regular instance, it would be almost trivial for me to do ... but one is special and so you all might have to provision the right one for me and I can take it from there. [22:01:34] but, i'll need access to the mysql data ... so, yes, we can spin up a new instance, i can set it up, and once we are all confirmed it works, we can shut this down. [22:02:07] subbu: we can help you move data. Those of us with "root" ssh keys can still get in. [22:02:14] ok. [22:02:59] And "easy" way to move data might be with a cinder volume actually. We can make one, attach it to the old server, fill it full of goodies, and then reattach it to a new instance [22:03:29] and I can bump quotas as needed in your project to do the replacement dance [22:04:43] * bd808 is remembering things about the wikitextexp project [22:05:31] sounds good. right. this one has 12 vcpus with 32gb ram and 400gb disk ... and the project will definitely need a quota bump before a new equivalent instance can be stood up. [22:05:36] That's kind of what cinder is for. As long as the database (and ideally the instance) is stopped at the time of detach, it should all be good. [22:05:54] Sounds like it [22:06:11] subbu: can you make a phab task about the loss of access so we can track getting it all fixed? [22:06:22] will do. [22:06:58] Disk will be cinder on the new instance, but we will probably need to make a special flavor for the beefy cpu + ram [22:08:17] !log wikitextexp Added self (BryanDavis) to project to help with sad instances [22:08:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikitextexp/SAL [22:10:53] I created T292264 [22:10:54] T292264: Loss of access to parsing-qa-01.eqiad.wmflabs - https://phabricator.wikimedia.org/T292264 [22:38:18] subbu: should I build the new instance with Buster or Bullseye? Bullseye would be preferred unless there is some extra weird reason you need older software. [22:38:33] No, bullseye is good. [22:38:42] excellent [22:39:33] * bd808 apparently did ram math wrong for the quota increase [22:45:59] !log wikitextexp Attaching volume "parsing" to parsing-qa-01 (T292264) [22:46:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikitextexp/SAL [22:46:02] T292264: Loss of access to parsing-qa-01.eqiad.wmflabs - https://phabricator.wikimedia.org/T292264 [23:00:56] bd808, i am signing off for the evening ... so will pick up the threads tonight / tomorrow. [23:01:30] subbu: ack. I'm copying data now. I'll update the ticket with details when I stop. [23:01:32] tomorrow is fine for me ... so don't feel the need to work through this today. thanks for your (collective you) help with this. [23:29:17] Start [23:45:18] https://t.me/wmcloudirc [23:45:44]