[08:29:08] announce: we're in the progress of changing how unified certificates are brought to cache hosts, TLDR: will be stored in a tmpfs directory created at boot. This is the last change that enables this feature on two hosts for test (text|upload): https://gerrit.wikimedia.org/r/c/operations/puppet/+/1129223 [08:29:43] this won't be merged until next week, and even then only on depooled hosts to ensure all is fine while the host reboots, etc. [08:29:54] mail and more detailed documentation on this to follow [10:18:02] hello on-callers, it seems that we finally fixed (fingers crossed) the db/import issue on maps1009 [10:26:09] very nice! [10:26:29] was is more than fixing the variable type of the wikidata counter? [10:31:06] 👍 [10:36:11] we got derailed in changing a yaml file (never change yaml files :D) [10:37:03] the table attribute change was sneaky since we changed its type from int to bigint, but since it was a SEQUENCE (part of the primary key) we also had to run ALTER SEQUENCE to move the max value from max-int-4bytes to max-int-8bytes [10:37:35] it wasn't automatic [10:37:55] after that, the imported restarted working like a charm [16:12:40] do we have a "things we might want to consider during future switchovers" task / doc / equivalent? [22:02:31] Emperor: I'm not aware of one and would be happy to start (being aware, that is :-D) [22:45:05] godog: I'm looking at T371102. Using the examples from the task description (application servers red k8s), i see that 90d, 6M work reasonably quick (5-10 seconds, workable) which is huge progress. What is the retention meant to be now? When trying Jan-Mar 2023 I get no results, Jan-Mar 2024 no results, Mar-June 2024 there is data starting from Mar 28 2024, which suggests we have 1y retention, not 5y? [22:45:05] T371102: Include long-retention Prometheus data from Thanos into Grafana queries - https://phabricator.wikimedia.org/T371102 [22:45:09] https://grafana.wikimedia.org/d/000000066/resourceloader?orgId=1&from=1709251200000&orgId=1&to=1717372800000 [22:45:21] https://grafana.wikimedia.org/d/35WSHOjVk/application-servers-red-k8s?orgId=1&from=1709251200000&orgId=1&to=1717372800000 [22:46:44] interval is approximately 3h at this range/zoom level