[00:02:07] hi! was there some software change today/yesterday to the nginx proxy in front of cloud services, and specifically https://patchdemo.wmflabs.org/ ? i'm currently losing my mind trying to figure out why some pages are served with broken gzip content-encoding: https://gitlab.wikimedia.org/repos/ci-tools/patchdemo/-/issues/609 [00:05:14] to reproduce: log in, then select version=REL1_42, preset=Minimal, click "Create demo" and "OK" [00:07:52] (i'm trying out some things, and i might break that reproduction, let me know if you want to try it yourself) [01:24:03] hmm, looks like Quarry has died [01:24:41] and now it's back [01:30:05] Looks like wmcs.openstack.migrate_dbinstance_to_ovs was just run for quarry a few minutes ago. Perhaps that caused it to go down for a minute [10:39:27] !log wikidumpparse force-reboot humaniki-prod, it had lost the network [10:39:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidumpparse/SAL [10:49:03] !log wikidumpparse force reschedule the humaniki-prod instance to a non-OVS hypervisor with `wmcs-openstack server migrate 0bd43b56-75ba-411b-980b-b8d8f06837a8` [10:49:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidumpparse/SAL [14:00:45] hi folks! I'd need to restart ceph on some cloudcephosd, but never done it (to pick up glibc updates). Anybody that can help? [14:00:57] next week is fine, I am just going through the process :) [14:06:47] elukey: your best bet is to ping dca.ro next week when he's back [14:32:03] super thanks! [15:25:48] anything going on with wmcs DNS ATM? I can't resolve airflow-img-build.eqiad1.wikimedia.cloud or cirrus-integ03.eqiad1.wikimedia.cloud from my desktop or cumin2002 [15:42:46] inflatador: nothing widespread but let me see what's up with those in particular [15:43:06] oh, inflatador you need a project in there [15:43:16] airflow-img-build..eqiad1.wikimedia.cloud [16:15:12] andrewbogott thanks, PEBKAC strikes again!