[02:04:47] <icinga-wm>	 PROBLEM - Check systemd state on an-launcher1002 is CRITICAL: CRITICAL - degraded: The following units failed: refinery-import-siteinfo-dumps.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
[02:10:12] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: varnishkafka on cp5004 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=eqsin%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp5004%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[02:10:47] <icinga-wm>	 PROBLEM - Check unit status of refinery-import-siteinfo-dumps on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit refinery-import-siteinfo-dumps https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[02:15:12] <jinxer-wm>	 (VarnishkafkaNoMessages) resolved: varnishkafka on cp5004 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=eqsin%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp5004%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[02:25:43] <wikibugs>	 10Data-Engineering, 10API Platform, 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0: Implement the unique devices endpoints - https://phabricator.wikimedia.org/T288298 (10BPirkle)
[02:26:16] <wikibugs>	 10Data-Engineering, 10API Platform, 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0: Unique Devices service - https://phabricator.wikimedia.org/T288298 (10BPirkle)
[02:28:59] <wikibugs>	 10Data-Engineering, 10API Platform, 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0: Unique Devices service - https://phabricator.wikimedia.org/T288298 (10BPirkle)
[02:34:33] <wikibugs>	 10Data-Engineering, 10API Platform, 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0: Implement mediarequests endpoints - https://phabricator.wikimedia.org/T288303 (10BPirkle)
[02:35:08] <wikibugs>	 10Data-Engineering, 10API Platform, 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0: Implement Mediarequests service - https://phabricator.wikimedia.org/T288303 (10BPirkle)
[02:38:38] <wikibugs>	 10Data-Engineering, 10API Platform, 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0: Implement Mediarequests service - https://phabricator.wikimedia.org/T288303 (10BPirkle)
[02:43:26] <wikibugs>	 10Data-Engineering, 10API Platform, 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0: Editors service - https://phabricator.wikimedia.org/T288305 (10BPirkle)
[02:45:43] <wikibugs>	 10Data-Engineering, 10API Platform, 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0: Editors service - https://phabricator.wikimedia.org/T288305 (10BPirkle)
[02:47:40] <wikibugs>	 10Data-Engineering, 10API Platform, 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0: Editors service - https://phabricator.wikimedia.org/T288305 (10BPirkle)
[02:55:07] <wikibugs>	 10Data-Engineering, 10API Platform, 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0: Editors service - https://phabricator.wikimedia.org/T288305 (10BPirkle)
[02:56:03] <wikibugs>	 10Data-Engineering, 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0:Wikistats 2 service - https://phabricator.wikimedia.org/T288301 (10BPirkle)
[02:59:52] <wikibugs>	 10Data-Engineering, 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0:Wikistats 2 service - https://phabricator.wikimedia.org/T288301 (10BPirkle)
[03:01:36] <wikibugs>	 10Data-Engineering, 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0:Wikistats 2 service - https://phabricator.wikimedia.org/T288301 (10BPirkle)
[03:02:44] <wikibugs>	 10Data-Engineering, 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0:Wikistats 2 service - https://phabricator.wikimedia.org/T288301 (10BPirkle)
[03:07:15] <icinga-wm>	 PROBLEM - Check unit status of refinery-import-page-history-dumps on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit refinery-import-page-history-dumps https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[03:11:23] <wikibugs>	 10Data-Engineering, 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0:Wikistats 2 service - https://phabricator.wikimedia.org/T288301 (10BPirkle)
[03:13:42] <wikibugs>	 10Data-Engineering, 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0:Wikistats 2 service - https://phabricator.wikimedia.org/T288301 (10BPirkle)
[03:33:49] <wikibugs>	 (03CR) 10DLynch: Instrument blocked account registration (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/821711 (https://phabricator.wikimedia.org/T306018) (owner: 10Sergio Gimeno)
[05:02:25] <icinga-wm>	 PROBLEM - Check unit status of refinery-import-page-current-dumps on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit refinery-import-page-current-dumps https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[05:04:43] <icinga-wm>	 PROBLEM - Check unit status of refinery-sqoop-mediawiki-production-daily on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit refinery-sqoop-mediawiki-production-daily https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[06:26:44] <wikibugs>	 (03PS3) 10DLynch: New schema: editattemptsblocked [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/820908 (https://phabricator.wikimedia.org/T310390)
[07:09:40] <joal>	 Wow - it looks like some coords got killed yesterday! Probably me not clicking on the right button??? Weird - Restarting webrequest now
[07:11:00] <joal>	 !log restart webrequest oozie bundle
[07:11:01] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[08:36:23] <btullis>	 joal: Morning, is there anything I can do to help with more cleanup from yesterday? I notice that we've had lots of SLA miss emails, but are the errors all fixed now?
[08:37:53] <milimetric>	 The siteinfo dumps systemd timer is still failed, but I think with restarting the webrequest that should clear most alerts.
[08:40:19] <btullis>	 Thanks. Should I take a look at the siteinfo dumps timer? We still have new hadoop packages to push out to 21 hosts: presto, druid, driud-public, labstore, and an-tool: https://debmonitor.wikimedia.org/packages/hadoop
[08:40:37] <btullis>	 This time I shall be more careful and seek advice.
[08:46:05] <btullis>	 I see these four failed systemd units.
[08:46:06] <btullis>	 https://usercontent.irccloud-cdn.com/file/OPtsGkKW/image.png
[08:46:10] <milimetric>	 It's ops week stuff, no worries.  Yeah, hadoop stuff was always so daunting
[08:54:20] <joal>	 Heya folks - today is kid's day, I'm not online often :S
[08:54:39] <btullis>	 Ah yes, I remember now.
[08:55:12] <joal>	 btullis: those timers are related to hdfs-rsync and sqoop - I don't really know how fix them, we'll need more investigation
[08:55:46] <joal>	 btullis: we've almost back in time in terms of webrequest, which is good
[08:56:43] <joal>	 btullis: I'm checking oozie jobs I touched yesterday, just in case I did the same mistake as with webrequest
[08:59:10] <btullis>	 OK, thanks. It looks like maybe we have to unmount and then re-mount `/mnt/hdfs` on an-launcher1002. I'm just checking now which user it should use to browse. I thought it was `analytics` but that's giving me an I/O error.
[09:03:39] <wikibugs>	 10Data-Engineering, 10Equity-Landscape: UDF to calculate the average of two or more column values - https://phabricator.wikimedia.org/T313709 (10ntsako) 05Open→03Resolved
[09:03:41] <wikibugs>	 10Data-Engineering, 10Equity-Landscape: Milestone: Ingest and Transform Input Data - https://phabricator.wikimedia.org/T305475 (10ntsako)
[09:04:30] <wikibugs>	 (03PS1) 10Kosta Harlan: accountcreation: Adjust block type value from 'auto' to 'autoblock' [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/832203 (https://phabricator.wikimedia.org/T306018)
[09:05:05] <wikibugs>	 (03CR) 10CI reject: [V: 04-1] accountcreation: Adjust block type value from 'auto' to 'autoblock' [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/832203 (https://phabricator.wikimedia.org/T306018) (owner: 10Kosta Harlan)
[09:05:09] <wikibugs>	 (03CR) 10Kosta Harlan: [C: 03+2] Instrument blocked account registration (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/821711 (https://phabricator.wikimedia.org/T306018) (owner: 10Sergio Gimeno)
[09:06:46] <btullis>	 Yeah, it's supposed to be readable to `analytics` https://github.com/wikimedia/puppet/blob/production/hieradata/role/common/analytics_cluster/launcher.yaml#L20
[09:07:06] <btullis>	 ...but it isn't.
[09:07:15] <btullis>	 https://www.irccloud.com/pastebin/nK04X87D/
[09:07:36] <btullis>	 I'm going to re-mount this on an-launcher1002, unless anyone objects.
[09:10:02] <wikibugs>	 (03PS2) 10Kosta Harlan: accountcreation: Adjust block type value from 'auto' to 'autoblock' [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/832203 (https://phabricator.wikimedia.org/T306018)
[09:10:52] <btullis>	 !log re-mounted /mnt/hdfs on an-launcher1002.
[09:10:55] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[09:11:11] <btullis>	 That has worked.
[09:11:15] <btullis>	 https://www.irccloud.com/pastebin/H6mhnrXS/
[09:14:19] <joal>	 ok - I had missed 4 hours in webrequest - I restarted a job to backfill those
[09:14:33] <joal>	 I have also restarted another job I had killed
[09:14:41] <joal>	 !log Restart oozie virtualpageview job
[09:14:42] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[09:15:16] <joal>	 I think that with that I'll have covered all oozie and therefore airflow - I'll triple check later on
[09:15:47] <joal>	 btullis: I'll have time early afternoon to help you with the rest of failing stuff - sorry for not being very available
[09:17:15] <btullis>	 No problems. Sorry for causing all of the mess in the first place. I've got more `/mnt/hdfs` volumes to re-mount. Although I know that some of them use different users from analytics. I'll proceed to do those, but I'll hold off from deploying any more updated packages to the 21 remaining hosts.
[09:35:16] <aqu>	 Hi btullis would you like some help to check/restart some jobs?
[09:35:50] <aqu>	 I can begin with the 4 failed ones an-launcher1002.
[09:36:23] <btullis>	 Yes please, I think that they might work, now that I've remounted /mnt/hdfs. Haven't checked the sqoop one yet though.
[09:38:08] <aqu>	 OK, beginning with refinery-sqoop-mediawiki-production-daily.service
[09:42:06] <wikibugs>	 (03CR) 10Gergő Tisza: [C: 03+2] accountcreation: Adjust block type value from 'auto' to 'autoblock' [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/832203 (https://phabricator.wikimedia.org/T306018) (owner: 10Kosta Harlan)
[09:42:43] <wikibugs>	 (03Merged) 10jenkins-bot: accountcreation: Adjust block type value from 'auto' to 'autoblock' [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/832203 (https://phabricator.wikimedia.org/T306018) (owner: 10Kosta Harlan)
[10:07:19] <icinga-wm>	 PROBLEM - Check unit status of check_webrequest_partitions on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit check_webrequest_partitions https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[10:14:48] <wikibugs>	 10Data-Engineering, 10SRE, 10Traffic, 10Patch-For-Review: intake-analytics is responsible for up to a 85% of varnish backend fetch errors - https://phabricator.wikimedia.org/T306181 (10Vgutierrez) This is highly related to T317051 and I think we can close this one and just blame on how Varnish reports some...
[10:22:59] <icinga-wm>	 RECOVERY - Check unit status of refinery-sqoop-mediawiki-production-daily on an-launcher1002 is OK: OK: Status of the systemd unit refinery-sqoop-mediawiki-production-daily https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[10:44:55] <mforns>	 heya a-team! btullis and joal I see there are lots of alerts, can I help with something?
[11:04:55] <btullis>	 Thanks mforns. I know that aqu was looking at the failed systemd units on an-launcher1002, but the  check_webrequest_partitions alert is new. 
[11:09:25] <btullis>	 !log remounted /mnt/hdfs on an-airflow1001
[11:09:26] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[11:12:57] <btullis>	 !log remounted /mnt/hdfs on an-coord100[1-2]
[11:12:58] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[11:34:59] <btullis>	 !log remounted all remaining /mnt/hdfs mount points, except stat1005 which is busy
[11:35:00] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[11:46:15] <icinga-wm>	 RECOVERY - Check unit status of refinery-import-page-history-dumps on an-launcher1002 is OK: OK: Status of the systemd unit refinery-import-page-history-dumps https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[11:46:39] <icinga-wm>	 RECOVERY - Check unit status of refinery-import-siteinfo-dumps on an-launcher1002 is OK: OK: Status of the systemd unit refinery-import-siteinfo-dumps https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[11:47:05] <icinga-wm>	 RECOVERY - Check unit status of refinery-import-page-current-dumps on an-launcher1002 is OK: OK: Status of the systemd unit refinery-import-page-current-dumps https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[11:47:19] <joal>	 \o/ hdfs-rsync seems back in track :) Thanks for the remount btullis, and thanks aqu for the job restarts
[11:48:40] <btullis>	 joal: Great. Phew. I still haven't looked at the failed `check_webrequest_partitions.service` on an-launcher1002 yet.
[11:48:48] <joal>	 btullis: I'm on this one
[11:49:51] <btullis>	 Thanks. I've noticed that we don't actually have icinga monitoring in place for the /mnt/hdfs readability, although puppet configuration for it is in place. It seems to be disabled.
[11:51:56] <joal>	 btullis: that error is on me - When I restarted the webrequest jobs yesterday, I accidentally killed the parent (I started coordinator instances but the coordinator itself was killed), and then this morning I made 2 mistakes in restarting jobs: first I restarted them starting hour 21 (wrong reading of past oozie runs), then restarted from hour 17 to 20, but 20 got excluded /facepalm, and now 
[11:52:02] <joal>	 hour 20 is running, which will mean all hours for yesterday fixed
[11:52:36] <joal>	 btullis: I have checked other jobs I touched, and didn't do the hours mistakes on the only one I had to restart (virtualpageview)
[11:53:04] <joal>	 btullis: so with that, I think we're good in term of webrequest and downstream jobs (phew)
[11:54:21] <btullis>	 joal: Many thanks. The incident report draft is here. I'll add more detail to it over the next couple of days: https://docs.google.com/document/d/1ZkT-QVioRBoo9QwSCcGYm2miVI-OFXGxL5ZXVjxgSts/edit#
[11:55:26] <joal>	 Thank you for this btullis - I'll take the time to read and comment later on today :)
[12:11:34] <mforns>	 oh, sorry btullis, I went afk and just saw your message :/
[12:56:57] <aqu>	 !log ~1hago sudo systemctl start refinery-sqoop-mediawiki-production-daily.service ; sudo systemctl start refinery-import-siteinfo-dumps.service ; sudo systemctl start refinery-import-page-current-dumps.service ; sudo systemctl start refinery-import-page-history-dumps.service
[12:56:58] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[12:57:59] <aqu>	 Sorry, I've forgotten to log in real time my operations on jobs.
[13:08:59] <joal>	 To keep archives happy: we've checked that gobblin imported data ysterday in volumes that seem ok
[14:15:13] <icinga-wm>	 PROBLEM - Check systemd state on stat1008 is CRITICAL: CRITICAL - degraded: The following units failed: jupyter-ntsako-singleuser.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
[14:19:47] <icinga-wm>	 RECOVERY - Check systemd state on stat1008 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
[14:33:29] <wikibugs>	 10Analytics-Radar, 10Data-Engineering-Radar, 10Event-Platform Value Stream, 10Patch-For-Review: Move Kafka Jumbo's TLS clients to the new bundle - https://phabricator.wikimedia.org/T296064 (10elukey) Getting back to the task - we have moved kafka logging-codfw to PKI (eqiad will follow soon), so all the up...
[15:20:40] <btullis>	 OK, I've filled in the bulk of the incident report. joal, aqu, milimetric please do feel free to add/amend/remove anything that you feel appropriate: https://docs.google.com/document/d/1ZkT-QVioRBoo9QwSCcGYm2miVI-OFXGxL5ZXVjxgSts/edit# - then we can publish and share.
[15:23:47] <icinga-wm>	 RECOVERY - Check systemd state on an-launcher1002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
[15:29:35] <icinga-wm>	 RECOVERY - Check unit status of check_webrequest_partitions on an-launcher1002 is OK: OK: Status of the systemd unit check_webrequest_partitions https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[16:06:29] <wikibugs>	 10Data-Engineering, 10Data Pipelines: Productionize HDFS fsimage data analysis job - https://phabricator.wikimedia.org/T261283 (10lbowmaker)
[16:15:40] <joal>	 Hi xcollazo :)
[16:24:18] <wikibugs>	 (03CR) 10Joal: [C: 03+2] "Awesome explicit file naming - We should follow this pattern everywhere :) LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/831639 (https://phabricator.wikimedia.org/T305846) (owner: 10Xcollazo)
[16:26:46] <xcollazo>	 Hello joal and everyone else, great to be here! 
[16:27:10] <wikibugs>	 (03CR) 10Joal: [V: 03+2 C: 03+2] "Merging!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/788841 (https://phabricator.wikimedia.org/T201491) (owner: 10Klein Muçi)
[16:27:40] <btullis>	 Hi xcollazo - welcome home :-)
[16:27:58] <wikibugs>	 (03CR) 10Joal: [V: 03+2] "Merging for next deploy" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/831866 (https://phabricator.wikimedia.org/T317126) (owner: 10Joal)
[16:28:09] <wikibugs>	 (03CR) 10Joal: [V: 03+2 C: 03+2] Add missing tables to drop-mediawiki-snapshots [analytics/refinery] - 10https://gerrit.wikimedia.org/r/831866 (https://phabricator.wikimedia.org/T317126) (owner: 10Joal)
[16:28:33] <joal>	 xcollazo: shall I merge you HQL patch, or do wish to test it more before?
[16:29:13] <xcollazo>	 joal: that one is ready to go.
[16:29:21] <joal>	 Ok merging
[16:29:33] <xcollazo>	 ty
[16:29:35] <wikibugs>	 (03CR) 10Joal: [V: 03+2 C: 03+2] Modify geoeditor SQL scripts to play nice with Spark3 [analytics/refinery] - 10https://gerrit.wikimedia.org/r/831639 (https://phabricator.wikimedia.org/T305846) (owner: 10Xcollazo)
[16:46:45] <wikibugs>	 10Data-Engineering, 10SRE, 10Traffic, 10Patch-For-Review: intake-analytics is responsible for up to a 85% of varnish backend fetch errors - https://phabricator.wikimedia.org/T306181 (10BTullis) >>! In T306181#8235549, @Vgutierrez wrote: > This is highly related to T317051 and I think we can close this one...
[16:47:52] <wikibugs>	 10Data-Engineering, 10SRE, 10Traffic, 10Patch-For-Review: intake-analytics is responsible for up to a 85% of varnish backend fetch errors - https://phabricator.wikimedia.org/T306181 (10Vgutierrez) 05Open→03Resolved Sorry about the fuss and thanks for your thorough investigation @BTullis
[16:48:14] <vgutierrez>	 ^ I bet that btullis hates me a lot right now
[16:49:23] <btullis>	 vgutierrez: Not a bit of it :-) It's great to get exposure to the work of you and your team. I've enjoyed digging into it immensely.
[16:52:38] <wikibugs>	 10Data-Engineering, 10SRE, 10SRE-Access-Requests: Data Engineering Access for Hannah - https://phabricator.wikimedia.org/T317545 (10BTullis) I have created the principal and sent the mail to the user, as per the instructions here: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Kerberos#Create_a_princi...
[16:54:17] <wikibugs>	 10Data-Engineering-Radar, 10Gerrit-Privilege-Requests, 10Release-Engineering-Team (Blocking 🧱): Requesting membership of the analytics group in gerrit for 'snwachukwu' and 'nokafor' - https://phabricator.wikimedia.org/T314592 (10thcipriani)
[16:58:48] <btullis>	 I'm off for now, folks. Catch you tomorrow.
[16:59:18] <joal>	 Bye btullis 
[17:01:21] <wikibugs>	 10Data-Engineering, 10API Platform, 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0: Editors service - https://phabricator.wikimedia.org/T288305 (10VirginiaPoundstone) a:05FGoodwin→03BPirkle
[17:01:31] <wikibugs>	 10Data-Engineering, 10API Platform, 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0: Implement Mediarequests service - https://phabricator.wikimedia.org/T288303 (10VirginiaPoundstone) a:05FGoodwin→03BPirkle
[17:11:07] <aqu>	 !log Sep 14 15:23:34 UTC sudo  systemctl start check_webrequest_partitions.service
[17:11:08] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[17:51:43] <wikibugs>	 10Data-Engineering, 10MediaWiki-extensions-EventLogging: [EPIC] Deprecate EventLogging::schemaValidate - https://phabricator.wikimedia.org/T317793 (10phuedx)
[19:03:50] <wikibugs>	 10Data-Engineering, 10API Platform, 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0: Implement pageviews endpoints - https://phabricator.wikimedia.org/T288296 (10BPirkle)
[19:04:59] <wikibugs>	 10Data-Engineering, 10API Platform, 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0: Pageviews: Implement Unit Tests - https://phabricator.wikimedia.org/T299735 (10BPirkle)
[19:06:12] <wikibugs>	 10Analytics, 10API Platform, 10Platform Engineering Roadmap, 10User-Eevans: Implement per-article endpoint of the pageviews API - https://phabricator.wikimedia.org/T289265 (10BPirkle) a:05codebug→03BPirkle
[19:06:45] <wikibugs>	 10Data-Engineering, 10API Platform, 10Code-Health-Objective, 10Epic, and 3 others: Implement aggregate endpoint of the pageviews API - https://phabricator.wikimedia.org/T299731 (10BPirkle) a:05codebug→03BPirkle
[19:07:15] <wikibugs>	 (03PS1) 10Xcollazo: Move most CREATE statements from hive/ to hql/ [analytics/refinery] - 10https://gerrit.wikimedia.org/r/832336 (https://phabricator.wikimedia.org/T317124)
[19:07:25] <wikibugs>	 10Data-Engineering, 10API Platform, 10Platform Engineering Roadmap, 10User-Eevans: Implement top endpoint of the pageviews API - https://phabricator.wikimedia.org/T299732 (10BPirkle) a:05codebug→03BPirkle
[19:07:43] <wikibugs>	 10Data-Engineering, 10API Platform, 10Platform Engineering Roadmap, 10User-Eevans: Implement top-by-country endpoint of the pageviews API - https://phabricator.wikimedia.org/T299733 (10BPirkle) a:05codebug→03BPirkle
[19:08:13] <wikibugs>	 10Data-Engineering, 10API Platform, 10Platform Engineering Roadmap, 10User-Eevans: Implement top-per-country endpoint of the pageviews API - https://phabricator.wikimedia.org/T299734 (10BPirkle) a:05codebug→03BPirkle
[19:16:38] <wikibugs>	 10Data-Engineering, 10API Platform, 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0: Pageviews Service - https://phabricator.wikimedia.org/T288296 (10BPirkle)
[19:16:46] <wikibugs>	 10Data-Engineering, 10API Platform, 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0: Pageviews Service - https://phabricator.wikimedia.org/T288296 (10BPirkle) a:03BPirkle
[19:17:27] <wikibugs>	 10Data-Engineering, 10API Platform, 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0: Unique Devices service - https://phabricator.wikimedia.org/T288298 (10BPirkle) a:05FGoodwin→03BPirkle
[19:30:43] <wikibugs>	 10Analytics, 10API Platform (Product Roadmap), 10Code-Health-Objective, 10Epic, and 3 others: AQS 2.0 - https://phabricator.wikimedia.org/T263489 (10BPirkle)
[19:40:14] <wikibugs>	 (03PS2) 10Xcollazo: Move most CREATE statements from hive/ to hql/ [analytics/refinery] - 10https://gerrit.wikimedia.org/r/832336 (https://phabricator.wikimedia.org/T317124)
[19:43:27] <wikibugs>	 10Data-Engineering, 10API Platform, 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0: Mediarequests Service - https://phabricator.wikimedia.org/T288303 (10BPirkle)
[19:45:33] <wikibugs>	 10Data-Engineering, 10API Platform, 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0:Wikistats 2 service - https://phabricator.wikimedia.org/T288301 (10BPirkle) a:03BPirkle
[19:56:50] <wikibugs>	 (03CR) 10Xcollazo: "Sorry for the crazy big patchset, but it was a lot of git moves!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/832336 (https://phabricator.wikimedia.org/T317124) (owner: 10Xcollazo)
[21:15:17] <wikibugs>	 10Data-Engineering, 10SRE, 10SRE-Access-Requests, 10Patch-For-Review: Data Engineering Access for Hannah - https://phabricator.wikimedia.org/T317545 (10BCornwall)
[21:20:05] <wikibugs>	 10Data-Engineering, 10SRE, 10SRE-Access-Requests, 10Patch-For-Review: Data Engineering Access for Hannah - https://phabricator.wikimedia.org/T317545 (10BCornwall) Hi, @Hokwelum ! You should now have all the access you require. Could you test out your new superpowers to confirm? Perhaps @Milimetric could pr...