[10:58:03] <wm-bot>	 !log tools.notwikilambda <lucaswerkmeister> deployed new pygments-server that works with latest SyntaxHighlight (i.e. supports input files), and restored SyntaxHighlight to latest master; it turns out yesterday’s code worked perfectly fine, MediaWiki just cached a broken lexers list in APCu, which didn’t get purged by action=purge, and also didn’t show up in shell.php because the APCu cache is disabled in cli blaarggghhhh
[10:58:07] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.notwikilambda/SAL
[11:03:21] <lucaswerkmeister>	 oh, and that APCu cache didn’t get disabled by $wgCacheType = DB_NONE either btw
[11:03:48] <lucaswerkmeister>	 because clearly when I said “please MediaWiki don’t cache anything right now this is hard enough to debug as it is” I didn’t know what I was doing and MediaWiki knew that I really still wanted to keep some caching
[11:04:17] <lucaswerkmeister>	 and I had to restart the webservice (i.e. introduce downtime) to wipe that cache
[11:04:21] <lucaswerkmeister>	 >.<
[11:09:09] <Reedy>	 opcache?
[11:09:13] <Reedy>	 or the apc user cache?
[11:36:37] <wm-bot>	 !log tools.notwikilambda <lucaswerkmeister> deployed pygments-server 1ae6e19920 (preserve boolean status of exit code)
[11:36:42] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.notwikilambda/SAL
[11:37:05] <lucaswerkmeister>	 Reedy: APCu, SyntaxHighlight was caching some `pygmentize` results there
[11:39:27] <lucaswerkmeister>	 (well, it was using the LocalServerObjectCache service, which ended up being APCu according to some debug logs I added)
[13:41:19] <wm-bot>	 !log tools.lexeme-forms <lucaswerkmeister> deployed ea93caf2ee (l10n updates)
[13:41:23] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL
[13:53:01] <wm-bot>	 !log tools.lexeme-forms <lucaswerkmeister> deployed 6f9e530018 (mobile-friendly navbar)
[13:53:04] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL
[14:17:49] <wm-bot>	 !log tools.translate-link <lucaswerkmeister> deployed 35f2372735 (Python 3.9)
[14:17:51] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.translate-link/SAL
[14:26:05] <wm-bot>	 !log tools.speedpatrolling <lucaswerkmeister> deployed 08c0ed516a (Python 3.9)
[14:26:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.speedpatrolling/SAL
[14:30:51] <wm-bot>	 !log tools.speedpatrolling <lucaswerkmeister> deployed 165e7e936a (remove redirect_to_https call)
[14:30:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.speedpatrolling/SAL
[14:31:32] <wm-bot>	 !log tools.translate-link <lucaswerkmeister> deployed 09106e75a2 (minor change, no webservice restart)
[14:31:34] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.translate-link/SAL
[14:42:19] <wm-bot>	 !log tools.wd-image-positions <lucaswerkmeister> deployed 8797b176c6 (Python 3.9)
[14:42:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL
[14:45:44] <wm-bot>	 !log tools.lexeme-forms <lucaswerkmeister> removed old venv-3.7
[14:45:48] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL
[14:46:17] <wm-bot>	 !log tools.wd-image-positions <lucaswerkmeister> removed old venv-3.5 venv-3.7
[14:46:18] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL
[14:46:57] <wm-bot>	 !log tools.speedpatrolling <lucaswerkmeister> removed old venv-3.4 venv-3.5 venv-3.7
[14:46:59] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.speedpatrolling/SAL
[14:47:49] <wm-bot>	 !log tools.pagepile-visual-filter <lucaswerkmeister> removed old venv-python3.4 venv-python3.5 venv-python3.7
[14:47:51] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.pagepile-visual-filter/SAL
[14:48:20] <wm-bot>	 !log tools.quickcategories <lucaswerkmeister> removed old venv-3.7
[14:48:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.quickcategories/SAL
[14:48:43] <wm-bot>	 !log tools.ranker <lucaswerkmeister> removed old venv-3.7
[14:48:44] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.ranker/SAL
[14:49:17] <wm-bot>	 !log tools.translate-link <lucaswerkmeister> removed old venv-3.7
[14:49:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.translate-link/SAL
[14:49:47] <lucaswerkmeister>	 alright, I think I’m done moving all my stuff to Python 3.9 :)
[14:50:06] <lucaswerkmeister>	 (well, except for wd-shex-infer, which is still on the grid)
[14:50:19] <majavah>	 if only all tool authors would be that active upgrading their tools I would be very happy :P
[14:50:27] <lucaswerkmeister>	 ^^
[14:50:39] <lucaswerkmeister>	 is there a dashboard something? I feel like I vaguely remember one
[14:50:48] <lucaswerkmeister>	 s/something/somewhere/
[14:51:54] <majavah>	 yes! https://k8s-status.toolforge.org/images/
[14:56:07] <lucaswerkmeister>	 nice, thanks
[14:59:13] <lucaswerkmeister>	 huh, python39 has more base pods than web pods
[14:59:28] <lucaswerkmeister>	 what are the base images for again? (`webservice shell` still gives me a web pod afaict)
[15:00:19] <lucaswerkmeister>	 I guess I can just look at the list of pods and try to guess from that
[15:00:38] <lucaswerkmeister>	 for python37 it looks like various bots are using the base image for jobs or something like that
[15:03:24] <lucaswerkmeister>	 …all of these are my own tools? https://k8s-status.toolforge.org/images/docker-registry.tools.wmflabs.org/toolforge-python39-sssd-web:latest/
[15:03:26] <lucaswerkmeister>	 that’s actually unexpected
[15:34:59] <majavah>	 lucaswerkmeister: web images are essentially base images with `webservice-runner` slapped inside them
[15:35:23] <majavah>	 the base images are used by the jobs framework and ideally everything else except things managed by `webservice`
[15:38:39] <AntiComposite>	 yeah I only have AntiCompositeBot on 39 so far, haven't upgraded the rest of my tools
[15:39:42] <majavah>	 k8s-status doesn't count use in non-running cronjobs
[15:58:57] <lucaswerkmeister>	 majavah: ok thanks
[16:00:02] <lucaswerkmeister>	 hm, I should probably change the notwikilambda deployments to base then, some of them use web for probably no good reason
[16:00:19] <lucaswerkmeister>	 (they do network traffic ofc but not using `webservice-runner`)
[16:18:26] <wm-bot>	 !log tools.notwikilambda <lucaswerkmeister> added --no-save to npm install in function evaluator and orchestrator (266c550663) and manually reset the package-lock.json in both repos
[16:18:28] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.notwikilambda/SAL
[16:29:34] <wm-bot>	 !log tools.notwikilambda <lucaswerkmeister> shortened periodSeconds of JS container probes from 15 to 5 seconds (3d658c10ba)
[16:29:37] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.notwikilambda/SAL
[16:36:02] <wm-bot>	 !log tools.notwikilambda <lucaswerkmeister> switched function evaluator and orchestrator from web to base image (8424307b5d)
[16:36:04] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.notwikilambda/SAL
[16:37:49] <wm-bot>	 !log tools.notwikilambda <lucaswerkmeister> switched pygments-server from web to base image (25f8a10fa0)
[16:37:51] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.notwikilambda/SAL
[16:45:17] <wm-bot>	 !log tools.notwikilambda <lucaswerkmeister> updated pygments-server to Python 3.9 (cef7948b15)
[16:45:21] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.notwikilambda/SAL
[16:45:54] <lucaswerkmeister>	 ^ turns out I had one more Python 3.7 thing lying around
[17:13:12] <wm-bot>	 !log tools.notwikilambda <lucaswerkmeister> cleaned up old replicasets: kubectl delete rs $(kubectl get rs -o json | jq -r ".items |.[] | select(.status.replicas == 0) | .metadata.name")
[17:13:15] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.notwikilambda/SAL