[03:28:45] <MacFan4000>	 !log wm-bot updated cloak in admins config
[03:28:50] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wm-bot/SAL
[11:14:06] <taavi>	 andrewbogott: done, sorry about that
[11:14:16] <taavi>	 (next time feel free to just kill my session if I don't respond)
[11:14:58] <arturo>	 !log tools reboot tools-sgebastion-10 for T301736
[11:15:02] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[11:15:02] <stashbot>	 T301736: toolsbeta.automated-toolforge-tests membership causes "groups: cannot find name for group ID 54872" error message - https://phabricator.wikimedia.org/T301736
[11:16:07] <arturo>	 !log tools purge debian package `unscd` on tools-sgebastion-10/11 for T301736
[11:16:10] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[11:49:22] <arturo>	 !log tools invalidate sssd cache in all bastions to debug T301736
[11:49:25] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[11:49:26] <stashbot>	 T301736: toolsbeta.automated-toolforge-tests membership causes "groups: cannot find name for group ID 54872" error message - https://phabricator.wikimedia.org/T301736
[13:34:21] <wm-bot>	 !log tools.lexeme-forms <lucaswerkmeister> deployed 56e69bad1a (l10n updates)
[13:34:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL
[14:15:27] <taavi>	 !log admin [codfw1dev] added domain-wide 'designateadmin' and 'observer' roles to codfw1dev-proxy-dns-manager service account T295246
[14:15:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[14:15:31] <stashbot>	 T295246: Dynamicproxy API should be useful without the Horizon dashboard - https://phabricator.wikimedia.org/T295246
[14:21:43] <chicocvenancio>	 !log paws Deploying minesweeper 
[14:21:46] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL
[16:04:29] <mdipietro>	 !log paws updating pywikibot 2fc27c99db35515121f5b80c9f3df5e648e56b58
[16:04:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL
[16:08:24] <taavi>	 !log devtools created devtools.wmcloud.org dns zone for the devtools project T301793
[16:08:27] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL
[16:08:28] <stashbot>	 T301793: Add wmcloud.org DNS zone to WMCS project devtools - https://phabricator.wikimedia.org/T301793
[17:32:50] <musikanimal>	 hello Cloud Services! Starting around 2022-02-10, Pageviews Analysis (and it's bundled apps) has been running rather slow. It seems there's an issue with some requests to the backend timing out or failing entirely. The backend is just cheap PHP scripts that query the replicas and/or tools-db. But as far as I can tell the issue is that it's not even making it that far; the backend never receives the request at all
[17:33:11] <musikanimal>	 task is at https://phabricator.wikimedia.org/T301649 (see also the comments)
[17:33:40] <musikanimal>	 I increased the number k8s pod replicas, but that made some errors go away but the symptoms are still there as far as I can tell
[17:34:08] <musikanimal>	 is anyone familiar or heard of an issue like this before? any tips for debugging?
[17:34:51] <musikanimal>	 the last paragraph in this comment I think sums up the issue best https://phabricator.wikimedia.org/T301649#7709983
[17:36:23] <wm-bb>	 <jeremy_b> scrollback to crg at Feb 13, 01:17 for someone reporting pageviews slowness. idk if that's helpful.
[17:36:40] <wm-bb>	 <jeremy_b> (this channel)
[17:37:50] <musikanimal>	 yeah I saw that. They were referring to the Topviews tool which makes ~10 requests to the backend, so the issue is most evident there https://pageviews.toolforge.org/topviews
[17:38:27] <musikanimal>	 in short, once you see that page successfully load, you should be able to refresh and things load instantly because it pulls from APCu cache. But if you check the network log you'll see sometimes some requests just hang
[17:38:50] <wm-bb>	 <jeremy_b> so it's stateless?
[17:39:11] <wm-bb>	 <jeremy_b> aside from APC cache
[17:40:24] <musikanimal>	 yes that would be fair to say. There's some clientside caching on some tools, but regardless it's not something wrong with the backend, rather connecting to it
[17:40:41] <musikanimal>	 there were no code changes that caused this, either
[17:41:01] <musikanimal>	 my guess is there's been an increase in traffic and the k8s pod can only handle so much
[17:41:18] <taavi>	 what effects are you seeing exactly? just slow requests or are you getting some errors too?
[17:41:51] <musikanimal>	 the clientside abandons the request after ~30 secs I think. I never get a response to check, and there's nothing logging at error.log
[17:43:20] <musikanimal>	 if I check the network log and directly load the URI that timed out, it always loads no problem
[17:44:08] <musikanimal>	 and sometimes the request does successfully make it to the backend, it's just slow. It all seems like a networking issue
[17:45:45] <musikanimal>	 the errors at https://phabricator.wikimedia.org/T301649#7706613 made perfect sense at first, but I am no longer seeing those in error.log after I increased the number k8s pod replicas
[17:46:24] <wm-bb>	 <jeremy_b> have you tried querying db yourself, maybe from CLI while it's misbehaving? have you found any sequence of requests or time of day that can trigger this condition?
[17:47:05] <wm-bb>	 <jeremy_b> it's there connection pooling?
[17:47:31] <musikanimal>	 yeah if I run it on my local (with an SSH tunnel to the db replicas) it all loads fine, every time. Not seeing any issue with my other tools that talk to the replicas or tools-db, either
[17:47:38] <taavi>	 this chart shows the response timings as measured by the proxy that sits directly before the webservice pods:
[17:47:38] <taavi>	 https://tools-prometheus.wmflabs.org/tools/graph?g0.range_input=1w&g0.expr=sum(nginx_ingress_controller_request_duration_seconds_sum%7Bnamespace%3D%22tool-pageviews%22%7D)%20by%20(status)%20%2F%20sum(nginx_ingress_controller_request_duration_seconds_count%7Bnamespace%3D%22tool-pageviews%22%7D)%20by%20(status)&g0.tab=0
[17:48:12] <taavi>	 499 is a nginx-specific code for "Used when the client has closed the request before the server could send a response.", so browser timeout I think
[17:48:13] <musikanimal>	 ah! that is very telling
[17:48:56] <taavi>	 so I'm wondering if you're suddenly getting hit by some misbehaving crawler
[17:49:00] <musikanimal>	 499 sounds exactly what I'm experiencing, and I see this started on the 10th
[17:49:14] <musikanimal>	 yes, I was thinking I could turn on the access.log
[17:49:34] <musikanimal>	 see also https://phabricator.wikimedia.org/T226688 :)
[17:49:43] <taavi>	 heh
[17:49:48] <taavi>	 I'm grepping the front proxy logs atm
[17:50:42] <musikanimal>	 okay thank you! I have a meeting here in 10 mins, so I will have to depart for about an hour. I feel like we're on to something, though! thank you for the help so far :)
[17:54:34] <musikanimal>	 I just turned on the access.log. In my experience with VPS tools that have the XFF header enabled, most web crawlers have human-like UAs, so it may be difficult to infer which traffic is organic and which is not
[17:55:05] <taavi>	 nothing obvious that I can see :( even the most active individual IPs are only at couple thousand requests per day, those are obviously crawlers too per the IP addresses but don't explain the sudden jump in the graphs
[17:55:08] <musikanimal>	 also most web crawlers I don't think load JavaScript, right? that was my theory as to why Pageviews was immune to the whole web crawler problem, up until now
[17:56:33] <taavi>	 one thing you could try to reduce load on the kubernetes pods would be to move images, localization files, etc static content from tools-static, https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#Static_file_server
[17:57:57] <musikanimal>	 okay I will try that, seems like that couldn't hurt
[17:59:05] <musikanimal>	 I haven't noticed any static assets failing to load so far, only the AJAX requests. But again it can't hurt load assets via tools-static
[18:16:26] <wm-bb>	 <lucaswerkmeister> jesus CHRIST
[18:16:27] <wm-bb>	 <lucaswerkmeister> background-runner-8479db69c7-ldlnc   0/1     CrashLoopBackOff   12845      124d
[18:16:41] <wm-bb>	 <lucaswerkmeister> 12845 restarts in that pod 😬
[18:17:03] <taavi>	 that reminds me to finish off my thing to automatically kill off things that have been in CrashLoopBackOff for a while
[18:17:17] <arturo>	 @lucaswerkmeister I think taavi was working on some logic to deal with that
[18:18:12] <wm-bot>	 !log tools.quickcategories <lucaswerkmeister> updated venv (includes mwparserfromhell 0.6.4) and recreated background-runner deployment from scratch (it had been stuck in CrashLoopBackOff at 12845 restarts, let’s see if this helps)
[18:18:14] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.quickcategories/SAL
[18:19:55] <wm-bb>	 <lucaswerkmeister> nope, it’s still restarting
[18:20:23] <chicocvenancio>	 !log paws added psp for minesweeper
[18:20:25] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL
[18:21:20] <taavi>	 !log tools delete tools-package-builder-03
[18:21:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[18:21:38] <wm-bot>	 !log tools.quickcategories <lucaswerkmeister> deleted background-runner deployment again, I have a stack trace, let’s see if I can make sense of it
[18:21:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.quickcategories/SAL
[18:38:17] <bd808>	 !log tools Disabled puppet on tools-proxy-05 for manual testing of nginx config changes
[18:38:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[18:40:59] <wm-bot>	 !log tools.quickcategories <lucaswerkmeister> deployed 3db46cd631 (update for cachetools 5.0.0), reset all pending jobs to planned, recreated background runner
[18:41:01] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.quickcategories/SAL
[18:43:36] <bd808>	 !log tools Enabled puppet on tools-proxy-05
[18:43:39] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[19:04:02] <wm-bot>	 !log tools.quickcategories <lucaswerkmeister> deployed 6fc4cf0508 (recognize interwiki titles) with background runner restart, seems to be going better now
[19:04:06] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.quickcategories/SAL
[19:05:07] <wm-bot>	 !log tools.quickcategories <lucaswerkmeister> restarted webservice with `kubectl rollout restart deployment quickcategories`, the `webservice restart` apparently didn’t work? (pod age was still 125d)
[19:05:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.quickcategories/SAL
[19:06:26] <wm-bot>	 !log tools.quickcategories <lucaswerkmeister> deployed bb178f29b2 (missing template files)
[19:06:29] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.quickcategories/SAL
[19:16:36] <wm-bot>	 !log tools.quickcategories <lucaswerkmeister> deployed 3627eb9988 (fix query)
[19:16:39] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.quickcategories/SAL
[19:19:03] <wm-bb>	 <lucaswerkmeister> *slaps roof of tool* this bad boy can fit so many bugs in it
[19:20:07] <taavi>	 !bash <lucaswerkmeister> *slaps roof of tool* this bad boy can fit so many bugs in it
[19:20:08] <stashbot>	 taavi: Stored quip at https://bash.toolforge.org/quip/f7rT_n4Ba_6PSCT96NHu
[19:20:14] <wm-bb>	 <lucaswerkmeister> lol
[19:21:27] <wm-bb>	 <lucaswerkmeister> (now the question is, can I be annoyed at my users for not telling me they had issues with the tool, or is it my fault for not looking into the logs)
[20:08:27] <wm-bot>	 !log tools.wd-shex-infer <lucaswerkmeister> restarted webservice (had been crashing with “too many open files” since 2022-01-23)
[20:08:30] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-shex-infer/SAL
[20:11:22] <wm-bb>	 <lucaswerkmeister> hm, are there known issues with gridengine uwsgi-plain tools at the moment? https://wd-shex-infer.toolforge.org/ is showing the standard 503 error page but the webservice claims to be running with no errors in uwsgi.log
[20:11:50] <wm-bb>	 <lucaswerkmeister> (I would expect the tool to still target the old Stretch grid for now)
[20:12:08] <wm-bb>	 <lucaswerkmeister> aha, now it’s working
[20:12:16] <wm-bb>	 <lucaswerkmeister> guess I just forgot how long it takes to come back up 🤷
[20:33:50] <wm-bot>	 !log tools.trusty-tools <bd808> Kill webgrid-lighttpd job that should not have been running
[20:33:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.trusty-tools/SAL
[20:54:58] <mutante>	 beta seems to be down
[21:01:10] <bd808>	 mutante: I see dancy poking at some things that might be related in #wikimedia-releng
[21:01:43] <mutante>	 bd808: ACK, just pinged as well a second ago :)
[21:21:36] <wm-bb>	 <jeremy_b> @andrewbogott do you think the nfs mount changes at wikidumpparse broke https://humaniki.wmcloud.org/ ?
[21:21:58] <wm-bb>	 <jeremy_b> who's in that horizon project now?
[21:22:29] <andrewbogott>	 It might have broken it but it might also be that it just didn't survive a reboot. What in particular is broken? (url works for me)
[21:22:39] <wm-bb>	 <jeremy_b> anyone know the best way to get ahold of Max Klein?
[21:23:02] <andrewbogott>	 oh, I see, something on the api server...
[21:23:18] <wm-bb>	 <jeremy_b> idk, exactly, waiting on reply about that.
[21:23:19] <wm-bb>	 <jeremy_b> but at first glance:
[21:23:21] <wm-bb>	 <jeremy_b> NaN% and Unreachable : "https://humaniki.wmcloud.org/api/v1/gender/gap/latest/gte_one_sitelink/properties?label_lang=en" (re @wmtelegram_bot: <andrewbogott> It might have broken it but it might also be that it just didn't survive a reboot. What in particular is broken? ...)
[21:23:45] <andrewbogott>	 any idea which server serves that api?
[21:24:10] <wm-bb>	 <jeremy_b> I think it's the same domain as the website?
[21:24:13] <wm-bb>	 <jeremy_b> idk
[21:26:07] <wm-bb>	 <lucaswerkmeister> is there a service.template keyword to target the buster grid with a gridengine webservice?
[21:27:10] <andrewbogott>	 hmmmm I wonder if I clobbered an existing mount
[21:27:15] <wm-bb>	 <jeremy_b> (I told the person that reported it to join here, no reply yet) (re @jeremy_b: idk, exactly, waiting on reply about that.
[21:27:15] <wm-bb>	 <jeremy_b> but at first glance:
[21:27:16] <wm-bb>	 <jeremy_b> NaN% and Unreachable : "https://humaniki.wmcloud.org/api/v1/ge...)
[21:27:41] <andrewbogott>	 nope, I didn't
[21:28:02] <andrewbogott>	 so that api is redirected to a service at /srv/humaniki/humaniki-backend/humaniki.sock
[21:28:12] <andrewbogott>	 which is probably not running
[21:28:18] <wm-bb>	 <lucaswerkmeister> looks like it’s release judging from the webservice code :) (re @lucaswerkmeister: is there a service.template keyword to target the buster grid with a gridengine webservice?)
[21:28:46] <wm-bb>	 <jeremy_b> does it just not start up all the way on boot?
[21:28:55] <andrewbogott>	 no idea, I'm just guessing
[21:29:15] <wm-bb>	 <jeremy_b> right me too 😃
[21:29:15] <andrewbogott>	 i mean, that sock file doesn't exist.  So that suggests it did not start up :)
[21:29:23] <wm-bb>	 <jeremy_b> right
[21:29:32] <wm-bb>	 <jeremy_b> maybe a clue in bash history?
[21:29:49] <andrewbogott>	 it's started by 'gunicorn.sh'
[21:30:25] <bd808>	 @lucaswerkmeister: yeah, I think adding a `release: buster` value to service.template should work.
[21:30:26] <andrewbogott>	 there, did I fix it?
[21:30:48] <wm-bb>	 <jeremy_b> maybe we need something like chaos monkey
[21:31:44] <andrewbogott>	 jeremy_b, is it working now?
[21:31:47] <wm-bb>	 <jeremy_b> I asked. looks better to me (re @wmtelegram_bot: <andrewbogott> there, did I fix it?)
[21:31:53] <andrewbogott>	 !log wikidumpparse started 'gunicorn' service on humaniki-backend.  Apparently it doesn't start itself on reboot.
[21:31:55] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidumpparse/SAL
[21:32:20] <andrewbogott>	 ok, I'm withdrawing for now :) I suggest you open a bug or something about getting that service to auto start.
[21:32:38] <wm-bb>	 <jeremy_b> I never heard of the service before so idk what it's supposed to look like
[21:32:43] <bd808>	 @jeremy_b: mostly we need an easier to use automation system for Cloud VPS instances than "make a patch to ops/puppet.get and hope someday it gets merged" so that folks actually automate their instances config and startup
[21:32:48] <wm-bb>	 <jeremy_b> 👍 I agree. thanks
[21:33:22] <andrewbogott>	 bd808: I think in this case it already has a hand-made service module so it's probably a one-liner to make it autostart
[21:33:39] <wm-bb>	 <lucaswerkmeister> oooh, pip on the stretch bastion has fancy colorful progress bars ^^
[21:34:31] <bd808>	 andrewbogott: *nod* and good for them to be that close to automated
[21:34:49] <andrewbogott>	 agreed! I was expecting 'run in screen'
[21:35:09] <wm-bb>	 <jeremy_b> hah lol
[21:35:10] <bd808>	 I fixed one like that a month or so ago. so gross
[21:35:27] <wm-bb>	 <jeremy_b> nohup > screen?
[21:35:31] <wm-bb>	 <jeremy_b> tmux???
[21:36:02] * bd808 waves a trout near @jeremy_b
[21:36:30] <bd808>	 systemd unit provisioned by Puppet is the correct answer ;)
[21:36:41] <wm-bb>	 <jeremy_b> farmed or wild? whole, fillet, or smoked? hot or cold smoke?
[21:37:00] <wm-bb>	 <lucaswerkmeister> hmm, my webservice seems to be running on tools-sgewebgrid-generic-09092, which says Debian 9 / stretch in /etc/os-release
[21:37:15] <wm-bb>	 <lucaswerkmeister> so I think the release isn’t working right…
[21:37:19] <wm-bb>	 <lucaswerkmeister> (wd-shex-infer tool)
[21:38:38] <wm-bb>	 <lucaswerkmeister> the service.manifest, updated just now, has release: stretch
[21:39:22] <bd808>	 @lucaswerkmeister: Hmm... so the template did not work?
[21:39:31] <wm-bb>	 <lucaswerkmeister> as far as I can tell no
[21:39:40] * bd808 goes to look at code
[21:40:02] <wm-bb>	 <lucaswerkmeister> could webservice be missing a release line in the argparser.set_defaults call around line 328?
[21:40:24] <andrewbogott>	 Who wants to write a regex for me? Given a string-with-an-unknown-number-of-dashes I want to truncate at the last dash, producing 'string-with-an-unknown-number-of'
[21:40:32] <wm-bb>	 <lucaswerkmeister> it adds a --release={} arg to tmpl_argv but then doesn’t get it out of the parsed template_args again, I suspect
[21:41:02] <wm-bb>	 <jeremy_b> raises hand (re @wmtelegram_bot: <andrewbogott> Who wants to write a regex for me? Given a string-with-an-unknown-number-of-dashes I want to truncate at the last...)
[21:41:17] <bd808>	 @lucaswerkmeister: yeah. you found the bug. Do you want to send a patch or have me work on it?
[21:41:23] <wm-bb>	 <lucaswerkmeister> I can send it, sure
[21:41:32] <wm-bb>	 <lucaswerkmeister> trying to test it on a local webservice copy now
[21:41:36] <wm-bb>	 <jeremy_b> what's an example before truncation?
[21:42:06] <wm-bb>	 <jeremy_b> does it always end with not a dash before and after truncation?
[21:42:17] <wm-bb>	 <jeremy_b> does it always have at least one dash to start?
[21:42:36] <wm-bb>	 <jeremy_b> what language is this?
[21:43:00] <andrewbogott>	 ruby, and it will always contain at least one dash
[21:43:04] <bd808>	 regex is probably the wrong tool unless the language is awk :)
[21:43:37] <andrewbogott>	 the source is a group of hostnames with a count postfix, like 'cluster-member-1' and 'cluster-member-2' and what I want is to isolate the prefix 'cluster-member'
[21:43:57] <andrewbogott>	 bd808: if you know how to do it in puppet w/out a regex I'm all ears :)
[21:44:07] <wm-bb>	 <jeremy_b> perl -lpe 's/-[^-]*$//;'
[21:44:13] <wm-bb>	 <jeremy_b> idk Ruby
[21:44:17] * andrewbogott tries it
[21:45:48] <wm-bb>	 <jeremy_b> writing regex on a phone is weird:)
[21:46:02] <wm-bb>	 <lucaswerkmeister> bd808: https://gerrit.wikimedia.org/r/c/operations/software/tools-webservice/+/762949
[21:46:24] <wm-bb>	 <jeremy_b> https://www.hudsonvalleyfisheries.com/
[21:46:25] <wm-bb>	 <jeremy_b> this is farmed. some of the smoked goes for $48/lb at the farmer's market. I don't get how they get people to pay that.
[21:47:22] <wm-bot>	 !log tools.wd-shex-infer <lucaswerkmeister> deployed ef7416d9c1 (move to Buster grid); I tested one job on this grid manually and it still seemed to work, let’s hope for the best
[21:47:25] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-shex-infer/SAL
[21:49:33] <wm-bot>	 !log tools.wd-shex-infer <lucaswerkmeister> deployed 16ff031278 (remove redirect_to_https call)
[21:49:35] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-shex-infer/SAL
[21:51:19] <bd808>	 andrewbogott: In python I would do "-".join("string-with-an-unknown-number-of-dashes".split("-")[:-1]) but yeah do what you gotta do in puppet/ruby land.
[21:51:57] <bd808>	 counting with regex is not a nice thing
[21:52:34] <wm-bb>	 <jeremy_b> even in python I might just use my regex from above
[21:52:49] <andrewbogott>	 it works, btw, thanks jeremy_b!
[21:53:09] <wm-bb>	 <jeremy_b> sure. I like regexes :)
[21:53:43] <andrewbogott>	 I knew someone here would :)
[21:53:51] <wm-bb>	 <jeremy_b> good bet
[22:47:22] <Platonides>	 bd808: better to do "string-with-an-unknown-number-of-dashes".rsplit("-", 1)[0] :-)
[22:50:45] <bd808>	 !log tools Built new toollabs-webservice 0.81
[22:50:48] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[22:58:06] <bd808>	 !log tools `sudo apt-get update && sudo apt-get install toolforge-webservice` on all bastions to pick up 0.81
[22:58:08] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[22:58:33] <bd808>	 @lucaswerkmeister: your fix should be live everywhere now
[22:58:47] <wm-bb>	 <lucaswerkmeister> cool, thanks! then I’ll remove my local webservice copy again
[22:58:52] * bd808 had not done that build and deploy dance for a long time
[23:06:56] <bd808>	 !log tools Started full rebuild of Toolforge containers to pick up webservice 0.81 and other package updates in tmux session on tools-docker-imagebuilder-01
[23:06:58] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[23:17:37] <bd808>	 !log tools Image builds failed in buster php image with an apt error. The error looks transient, so starting builds over.
[23:17:40] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL