[03:31:49] !log paws rebooted all paws worker nodes, which may or may not have fixed the service (it was down before and is back now!) [03:31:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [08:12:51] Do I need to do something for security group changes to take effect? [08:13:12] I edited the default security group in deployment-prep to add a port allowance for memcached [08:13:26] I've rebooted the memc servers after it still didn't work 10min later. [08:13:32] And ran puppet [08:14:27] tcpdump does show something but it doesn't look like normal memc commands and it doesn't correlate with my tests. But it's not completely quiet either. [08:15:42] I assume these firewalls exist outside the VM so it's moot to look for local iptables/ferm to have changed right? [08:16:29] moot to look / moot to wait for (I did look) [08:16:43] Kinda out of my depths. [08:45:59] Krinkle: yep outside the VM. Changes should take effect immediately, no Puppet required. [11:03:42] !log lucaswerkmeister@tools-bastion-13 tools.quickcategories deployed 060c14930c (upgrade dependencies) [11:03:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.quickcategories/SAL [11:18:13] !log lucaswerkmeister@tools-bastion-13 tools.quickcategories deployed 80ed3ddf01 (split dev requirements from prod requirements → smaller image, faster build) [11:18:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.quickcategories/SAL [11:20:22] ^ image build went down from ca. 55 s to 40 s (n=1 each time, but as anecdotal evidence that makes me happy enough ^^) [12:02:57] !log lucaswerkmeister@tools-bastion-13 tools.wd-image-positions deployed d9d2273efb (upgrade dependencies) [12:03:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL [13:45:13] !log lucaswerkmeister@tools-bastion-13 tools.wd-image-positions deployed 99489081d3 (split dev requirements from prod requirements) [13:45:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL [13:51:33] !log lucaswerkmeister@tools-bastion-13 tools.wd-image-positions deployed 451794996c (add health-check-path) [13:51:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL [14:43:26] !log lucaswerkmeister@tools-bastion-13 tools.wd-image-positions deployed b0af29d932 (change config keys to uppercase to work around T374780) [14:43:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL [14:46:10] !log tools.lexeme-forms python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open("config.yaml"))["OAUTH"]["CONSUMER_KEY"]))' | toolforge envvars create TOOL_OAUTH__CONSUMER_KEY [14:46:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [14:46:26] !log tools.lexeme-forms disregard the previous message, wrong tool 🤦 [14:46:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [14:46:33] !log tools.wd-image-positions python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open("config.yaml"))["OAUTH"]["CONSUMER_KEY"]))' | toolforge envvars create TOOL_OAUTH__CONSUMER_KEY [14:46:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL [14:46:47] !log tools.wd-image-positions python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open("config.yaml"))["OAUTH"]["CONSUMER_SECRET"]))' | toolforge envvars create TOOL_OAUTH__CONSUMER_SECRET [14:46:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL [14:47:02] !log tools.wd-image-positions python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open("config.yaml"))["SECRET_KEY"]))' | toolforge envvars create TOOL_SECRET_KEY [14:47:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL [14:48:15] !log lucaswerkmeister@tools-bastion-13 tools.wd-image-positions commented out config.yaml, should use envvars instead [14:48:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL [14:48:33] !log lucaswerkmeister@tools-bastion-13 tools.wd-image-positions deployed b27c4c2d73 (read config from envvars) [14:48:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL [15:27:07] !log tools.wd-image-positions webservice stop && mv www{,-unused-tool-now-runs-on-buildservice} && wget https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/raw/81a627821b/service.template && webservice start [15:27:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL [15:27:43] !log lucaswerkmeister@tools-bastion-13 tools.wd-image-positions deployed 81a627821b (Python 3.13 + Toolforge Build Service) [15:27:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL [15:31:02] !log lucaswerkmeister@tools-bastion-13 tools.wd-image-positions deployed ebfaeef6e0 (specify .python-version) [15:31:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL [15:39:24] !log lucaswerkmeister@tools-bastion-13 tools.wd-image-positions deployed 168a4bf7cc (upgrade m3api) [15:39:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL [15:45:30] !log soda@tools-bastion-13 tools.yapping-sodium soda built and uploaded a new version [15:45:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.yapping-sodium/SAL [15:50:48] !log lucaswerkmeister@tools-bastion-13 tools.lexeme-forms deployed 3c977ccc7b (specify .python-version) [15:50:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [18:17:28] When trying to load jobs I get this error: [18:17:28] tools.multichill@tools-bastion-12:~$ toolforge jobs load jobs.yml --job coord-from-exif-mysql [18:17:30] INFO: loading job 'coord-from-exif-mysql'... [18:17:31] ERROR: TjfCliError: Unknown error (403 Client Error: Forbidden for url: https://k8s.tools.eqiad1.wikimedia.cloud:6443/apis/batch/v1/namespaces/tool-multichill/cronjobs?dryRun=All) [18:17:33] Any idea what's happening here? [21:20:44] !log multichill@tools-bastion-12 tools.multichill Unable to add jobs, created T399417 [21:20:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.multichill/SAL