[06:19:38] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Persistence (Consultation), 10Data-Services, and 2 others: Toolforge db: View 'fiwiki_p.flaggedrevs' references invalid table/column/rights to use them - https://phabricator.wikimedia.org/T302233 (10Marostegui) >>! In T302233#7783686, @razzi wrote: > I... [06:46:36] !log kill remaining hanging processes for ppche*lko and accra*ze on an-test-client1001 to allow users offboard (puppet broken) [06:46:38] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [07:02:57] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Services, 10Patch-For-Review, 10cloud-services-team (Kanban): Recreate views for globaluser table - https://phabricator.wikimedia.org/T301674 (10Majavah) [08:43:57] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Persistence (Consultation), 10Data-Services, and 2 others: Toolforge db: View 'fiwiki_p.flaggedrevs' references invalid table/column/rights to use them - https://phabricator.wikimedia.org/T302233 (10Ladsgroup) Yes. It should be run on s1, s2, s3, s5, an... [09:46:54] 10Data-Engineering: Refactor refinery-drop-mediawiki-snapshots so that it no longer uses a _SUCCESS file - https://phabricator.wikimedia.org/T303988 (10BTullis) [09:59:27] 10Data-Engineering, 10Data-Engineering-Kanban: Check home/HDFS leftovers of accraze - https://phabricator.wikimedia.org/T303928 (10BTullis) It seems that there are only a few files on some of the stat boxes. Whom to ask about these is the next question... ` ====== stat1004 ====== total 0 ====== stat1005 =====... [10:03:39] 10Data-Engineering, 10Data-Engineering-Kanban: Check home/HDFS leftovers of accraze - https://phabricator.wikimedia.org/T303928 (10BTullis) Perhaps @calbon and/or @elukey would be able to say if any of these files from stat100[6-8] should be retained, archived, or simply deleted. Thanks. [10:11:19] 10Data-Engineering, 10Data-Engineering-Kanban: Check home/HDFS leftovers of ppchelko - https://phabricator.wikimedia.org/T303924 (10BTullis) Here are the files belonging to ppchelko. Just a couple of large files named 'ksql.*' from 2017. I'm assuming it's safe to delete these. ` ====== stat1004 ====== total 63... [10:13:58] 10Data-Engineering, 10Data-Engineering-Kanban: Check home/HDFS leftovers of accraze - https://phabricator.wikimedia.org/T303928 (10elukey) Thanks for the ping! I think that it is all related to tests or experiments, nothing that it isn't committed or worth to save. I'll defer to @calbon's final call though :) [10:16:37] 10Data-Engineering, 10Data-Engineering-Kanban: Check home/HDFS leftovers of accraze - https://phabricator.wikimedia.org/T303928 (10elukey) @kevinbazira can you check if there is anything worth to save too? [10:22:54] 10Data-Engineering, 10Data-Engineering-Kanban: Check home/HDFS leftovers of ppchelko - https://phabricator.wikimedia.org/T303924 (10BTullis) All files and home directories removed. [10:28:06] 10Data-Engineering, 10Data-Engineering-Kanban: reset kerberos password - https://phabricator.wikimedia.org/T303146 (10BTullis) a:03BTullis [10:32:08] 10Data-Engineering, 10Data-Engineering-Kanban: reset kerberos password - https://phabricator.wikimedia.org/T303146 (10BTullis) Hi Effiets, Apologies for the delay in resetting your kerberos password. I've done so now, so you should have an email with a temporary password and instructions on how to choose a ne... [10:59:44] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog, 10Patch-For-Review: Define LVS load-balancing for OpenSearch cluster - https://phabricator.wikimedia.org/T301458 (10BTullis) 05Open→03Resolved [10:59:46] 10Data-Engineering, 10Data-Catalog, 10Epic: Data Catalog MVP - https://phabricator.wikimedia.org/T299910 (10BTullis) [10:59:48] 10Data-Engineering-Kanban, 10Data-Catalog, 10Patch-For-Review: Set up opensearch cluster for datahub - https://phabricator.wikimedia.org/T301382 (10BTullis) [11:00:08] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog: Complete monitoring setup of datahubsearch nodes - https://phabricator.wikimedia.org/T302818 (10BTullis) 05Open→03Resolved [11:00:11] 10Data-Engineering-Kanban, 10Data-Catalog, 10Patch-For-Review: Set up opensearch cluster for datahub - https://phabricator.wikimedia.org/T301382 (10BTullis) [11:05:05] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog: Configure MariaDB database for DataHub on an-coord1001 - https://phabricator.wikimedia.org/T301459 (10BTullis) 05Open→03Resolved [11:05:07] 10Data-Engineering, 10Data-Catalog, 10Epic: Data Catalog MVP - https://phabricator.wikimedia.org/T299910 (10BTullis) [11:05:09] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog: Configure MariaDB database for DataHub on an-coord1001 - https://phabricator.wikimedia.org/T301459 (10BTullis) I have now moved this password to the private repo, where it is in: `hieradata/role/common/deployment_server/kubernetes.yaml` The entry... [11:05:14] 10Data-Engineering, 10Data-Engineering-Kanban: Check home/HDFS leftovers of accraze - https://phabricator.wikimedia.org/T303928 (10kevinbazira) @elukey, I agree with you. I don't see anything we'd like to save. [11:06:16] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog: Update DNS for the DataHub MVP services - https://phabricator.wikimedia.org/T301460 (10BTullis) [11:59:32] 10Data-Engineering: Check home/HDFS leftovers of clarakosi - https://phabricator.wikimedia.org/T304065 (10MoritzMuehlenhoff) [12:23:59] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog: Configure CAS-SSO authentication for the DataHub frontend - https://phabricator.wikimedia.org/T301462 (10BTullis) Now that I've thought more about it, I'm not 100% sure that SSO is actually the best thing to configure at the moment. My reasoning... [12:29:29] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog: Configure CAS-SSO authentication for the DataHub frontend - https://phabricator.wikimedia.org/T301462 (10BTullis) An alternative mechanism for authentication is OIDC - which is outlined here: https://datahubproject.io/docs/how/auth/sso/configure-... [12:32:36] 10Data-Engineering, 10Data-Engineering-Kanban, 10Infrastructure-Foundations, 10SRE, and 2 others: Allow access to prometheus-pushgateway.discovery.wmnet port 80 from within Analytics VLAN - https://phabricator.wikimedia.org/T304001 (10Ottomata) Would prefer to proceed if possible, just want to finish somet... [12:58:30] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog: Configure CAS-SSO authentication for the DataHub frontend - https://phabricator.wikimedia.org/T301462 (10MoritzMuehlenhoff) If this is a temporary setup for the MVP phase and the end target is to make it publicly available, then I'd recommend to... [13:00:53] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog: Configure CAS-SSO authentication for the DataHub frontend - https://phabricator.wikimedia.org/T301462 (10BTullis) Great, thanks for that. I will proceed with the LDAP configuration for now. [13:24:19] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog: Configure LDAP authentication for the DataHub frontend - https://phabricator.wikimedia.org/T301462 (10BTullis) [13:43:39] 10Data-Engineering, 10Data-Engineering-Kanban, 10Infrastructure-Foundations, 10SRE, and 2 others: Allow access to prometheus-pushgateway.discovery.wmnet port 80 from within Analytics VLAN - https://phabricator.wikimedia.org/T304001 (10ayounsi) Sounds good, it's quite quick to apply, but first: ` prometheu... [14:03:55] 10Data-Engineering, 10SRE, 10ops-eqiad: analytics10[63,67] mgmt interfaces seem flapping from time to time - https://phabricator.wikimedia.org/T303151 (10BTullis) Shutting down the two servers now. analytics1063 and analytics1067 [14:05:08] Hi milimetric - for when you're here - would you remove your -1 from the CR about data purge? [14:05:46] I got confirmation from Erin that they're ok - And by the way, thank you and mforns for the time you spent with her and Joseph Mando :) [14:05:56] 10Data-Engineering, 10SRE, 10ops-eqiad: analytics10[63,67] mgmt interfaces seem flapping from time to time - https://phabricator.wikimedia.org/T303151 (10Cmjohnson) Thanks @btullis [14:06:10] :] [14:07:33] !log shutdown analytics1063 and analytics1067 with 120 minutes of downtime T303151 [14:07:35] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:07:36] T303151: analytics10[63,67] mgmt interfaces seem flapping from time to time - https://phabricator.wikimedia.org/T303151 [14:08:33] 10Analytics-Kanban, 10Data-Engineering, 10Data-Engineering-Kanban, 10Event-Platform, and 4 others: Determine which remaining legacy EventLogging schemas need to be migrated or decommissioned - https://phabricator.wikimedia.org/T282131 (10Tobi_WMDE_SW) @Ottomata The AdvancedSearchRequest data stream can be... [14:08:59] 10Data-Engineering, 10Data-Engineering-Kanban, 10SRE: Increase max.incremental.fetch.session.cache.slots on Kafka jumbo eqiad - https://phabricator.wikimedia.org/T303324 (10Ottomata) Hm, it flattened out at the max of 2K again, and we are seeing some evictions. I suppose in hindsight this makes sense on j... [14:14:50] 10Data-Engineering, 10Data-Engineering-Kanban, 10Infrastructure-Foundations, 10SRE, and 2 others: Allow access to prometheus-pushgateway.discovery.wmnet port 80 from within Analytics VLAN - https://phabricator.wikimedia.org/T304001 (10Ottomata) a:05Ottomata→03None [14:15:41] (03PS1) 10Ottomata: Enable gobblin metric reporting to Prometheus via Prometheus PushGateway [analytics/refinery] - 10https://gerrit.wikimedia.org/r/771609 (https://phabricator.wikimedia.org/T294420) [14:18:38] 10Data-Engineering, 10Data-Engineering-Kanban, 10Infrastructure-Foundations, 10SRE, and 2 others: Allow access to prometheus-pushgateway.discovery.wmnet port 80 from within Analytics VLAN - https://phabricator.wikimedia.org/T304001 (10fgiunchedi) For sure, these are the IPs that the pushgateway could point... [14:31:50] 10Data-Engineering, 10Data-Engineering-Kanban, 10Infrastructure-Foundations, 10SRE, and 3 others: Allow access to prometheus-pushgateway.discovery.wmnet port 80 from within Analytics VLAN - https://phabricator.wikimedia.org/T304001 (10ayounsi) 05Open→03Resolved a:03ayounsi All done. [14:32:01] 10Analytics-Kanban, 10Data-Engineering, 10Data-Engineering-Kanban, 10Patch-For-Review: Send some existing Gobblin metrics to prometheus - https://phabricator.wikimedia.org/T294420 (10ayounsi) [14:35:20] 10Data-Engineering, 10Data-Engineering-Kanban: Check home/HDFS leftovers of accraze - https://phabricator.wikimedia.org/T303928 (10BTullis) Thanks all. I have removed these home directories `/home/accraze` on all stat servers. [14:36:13] 10Data-Engineering, 10Data-Engineering-Kanban, 10SRE, 10ops-eqiad: analytics10[63,67] mgmt interfaces seem flapping from time to time - https://phabricator.wikimedia.org/T303151 (10BTullis) p:05Triage→03Medium [14:36:17] (03PS2) 10Ottomata: Enable gobblin metric reporting to Prometheus via Prometheus PushGateway in test jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/771609 (https://phabricator.wikimedia.org/T294420) [14:38:20] (03CR) 10Joal: [C: 03+1] "LGTM! Merge as you wish" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/771609 (https://phabricator.wikimedia.org/T294420) (owner: 10Ottomata) [14:41:58] 10Data-Engineering, 10Data-Engineering-Kanban, 10Airflow: [Airflow] Troubleshoot MySQL connection issues - https://phabricator.wikimedia.org/T298893 (10odimitrijevic) a:05mforns→03None [14:42:42] joal: you are so quick! [14:43:21] 10Data-Engineering, 10Data-Engineering-Kanban, 10SRE, 10ops-eqiad: analytics10[63,67] mgmt interfaces seem flapping from time to time - https://phabricator.wikimedia.org/T303151 (10Cmjohnson) 05Open→03Resolved I am able to get into the idrac for both servers, it does take a little longer than normal. I... [14:48:28] 10Data-Engineering, 10Data-Engineering-Kanban, 10SRE: Increase max.incremental.fetch.session.cache.slots on Kafka jumbo eqiad - https://phabricator.wikimedia.org/T303324 (10JAllemandou) I'm not sure either @Ottomata - I have seen warnings about cache-session evictions, and thought it could be related. My la... [14:48:37] 10Data-Engineering, 10Data-Engineering-Kanban, 10SRE, 10ops-eqiad: analytics10[63,67] mgmt interfaces seem flapping from time to time - https://phabricator.wikimedia.org/T303151 (10BTullis) Thanks @Cmjohnson - I guess we'll just keep monitoring for stability and reopen this ticket if it keeps happening. [14:49:25] ottomata: it happenned that I was checking my emails when your reviews landed :) [14:55:20] hmm uh oh joal [14:55:21] in test [14:55:23] Mar 17 14:53:42 an-test-coord1001 gobblin-event_default_test[39795]: Caused by: java.lang.NoSuchMethodError: com.google.common.collect.ImmutableList.toImmutableList()Ljava/util/stream/Collector; [14:57:00] wowo [14:57:38] at org.apache.hadoop.util.RunJar.main(RunJar.java:158) [14:58:10] ottomata: this feels like a guava version error [14:58:30] yeah [14:58:37] maybe a conflictt wth hadoop jars vs whats in the shaded jar? [14:59:05] at org.wikimedia.eventutilities.core.util.ResourceLoader.asURLs(ResourceLoader.java:249) [14:59:23] very possible ottomata - I assume you've tested with our script setting? [14:59:30] script setting? [14:59:37] you mean with gobblin wrapper script? yes [14:59:42] right [14:59:46] hm [14:59:57] i did not test on test cluster though [15:00:02] hm [15:01:57] ottomata: I need to leave for kids [15:02:24] ottomata: possibly you could test using your command-line test on the test-cluster? [15:02:30] and see if it fails the same way? [15:05:15] yeah i'm ttesting [15:05:17] exactly that [15:05:27] i'm going to do that and test with deployed jar, as well as a mvn packaged one [15:05:56] HM! it worked with thte deployed jar!!!??!? [15:06:54] OH my testt gobblin conf doesn't use EventUtitlities [15:07:39] yes different versions [15:07:50] can we upgrade the guava version used by gobblin-wmf? [15:22:29] (03PS1) 10Ottomata: Change version of guava to 25.0-jre [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/771626 (https://phabricator.wikimedia.org/T297939) [15:23:22] (03CR) 10Ottomata: "I'm going to merge and release." [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/771626 (https://phabricator.wikimedia.org/T297939) (owner: 10Ottomata) [15:24:42] (03CR) 10jerkins-bot: [V: 04-1] Change version of guava to 25.0-jre [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/771626 (https://phabricator.wikimedia.org/T297939) (owner: 10Ottomata) [15:31:05] errph [15:31:32] gehel: yt? mvn duplicate-finder:check is failing after i bumped a dependency version [15:31:44] but, in the output xml file, i don't see the dependency i bumped as a confict [15:31:55] my packaged jar works just fine [15:32:11] not sure if i should disable the check here somehow? or if i can resolve it [15:32:53] your jar is expected to work mostly fine most of the time, but if that check fails, you have duplicated class, or in more direct term: jarhell! [15:33:07] you should definitely try to understand what the issue is before ignoring the check [15:33:16] the gobblin-wmf-core/target/duplicate-finder-result.xml is pretty verbose [15:33:41] it should be in the output log as well, in a more readable format [15:33:42] but i'm not sure how to read it [15:33:44] meeting, back in a few [15:33:47] oh, it wasn't tho? [15:34:14] OH yes it was [15:34:15] thank you [15:34:17] needed to scroll up [15:35:02] uhh they are in checkerframework??? [15:35:03] Found duplicate and different classes in [org.checkerframework:checker-compat-qual:2.0.0, org.checkerframework:checker-qual:2.6.0]: [15:35:56] hm guessing this is a guava transitive dep? [15:35:57] hm [15:42:55] joal / btullis: https://gerrit.wikimedia.org/r/c/operations/puppet/+/771389 is ready to merge (removed my -1) [15:48:20] yeah okay i dunno what to do then, need java help joal :/ [15:50:51] tried using shade to relocate guava in gobbln-wmf-core [15:51:09] https://github.com/google/guava/wiki/UseGuavaInYourBuild [15:51:13] but that didn't work (and I have no idea what i'mm doing) [15:53:30] 10Data-Engineering, 10Airflow: Low Risk Oozie Migration: interlanguage - https://phabricator.wikimedia.org/T300025 (10NOkafor-WMF) a:03NOkafor-WMF [15:57:28] I'm about to roll out https://gerrit.wikimedia.org/r/c/operations/puppet/+/767607 which may initiate a restart of opensearch on datahub nodes. btullis, razzi [15:57:57] Go for it cwhite - no traffic there yet [16:00:54] ottomata: [16:00:57] https://www.irccloud.com/pastebin/ftCMI0Op/ [16:01:44] looks like you are bringing both checker-compat-qual and checker-qual which probably does not make sense [16:01:52] and different versions [16:02:04] but...neither of those are in my poms [16:02:50] i think its because gobblin upstream uses guava 15 [16:03:08] ok gehel here's the problem tell me the answer: [16:03:18] event-utilities uses guava 28 [16:03:23] gobblin upstream uses gobblin 15 [16:03:38] the uber jar that we build for gobblin-wmf-core ends up using guava 15 (i think) [16:03:49] which causes an error in event-utilities about a missing method [16:03:58] Caused by: java.lang.NoSuchMethodError: com.google.common.collect.ImmutableList.toImmutableList()Ljava/util/stream/Collector; [16:04:17] so, i was trying to fix by just upgrading guava version for gobblin-wmf [16:04:35] and am now encountering this problem [16:04:39] what's the right thing to do? [16:05:38] 10Data-Engineering, 10Data-Engineering-Kanban: Resume Webrequest Data Purge Job - https://phabricator.wikimedia.org/T303977 (10Milimetric) a:03Milimetric [16:07:19] 10Data-Engineering, 10Data-Engineering-Kanban, 10Airflow: [Airflow] Create success_file operator - https://phabricator.wikimedia.org/T303405 (10ntsako) 05Open→03Resolved [16:07:43] 10Data-Engineering, 10Data-Engineering-Kanban, 10Airflow, 10Epic: Low Risk Oozie Migration: Mediawiki Geoeditors Monthly - https://phabricator.wikimedia.org/T300282 (10ntsako) 05In progress→03Resolved [16:07:47] 10Data-Engineering, 10Data-Engineering-Kanban, 10Airflow, 10Epic, 10Patch-For-Review: Migrate Oozie jobs to Airflow - https://phabricator.wikimedia.org/T299074 (10ntsako) [16:15:26] 10Data-Engineering, 10Data-Engineering-Kanban, 10SRE: Increase max.incremental.fetch.session.cache.slots on Kafka jumbo eqiad - https://phabricator.wikimedia.org/T303324 (10Ottomata) Okay, no prob! I'm inclined to just keep slots set at 2000; it is probably nice to have a little more room in jumbo anyway?... [16:39:03] gehel: ? :D [17:06:26] Heya - I'm back! [17:06:30] ottomata: let's try to fix tha [17:06:41] joa okay woo! [17:07:05] mwarf - those jar things are really not cool [17:07:18] ottomata: I'm gonna pull your patch and some jar analysis [17:08:29] thanks milimetric for the -1 removal [17:08:31] okay, joa thanks [17:08:47] btullis or ottomata - would one of ou merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/771389 please? [17:09:08] the associated log message should be restart webrequest and pageview_actor data purge :) [17:10:12] okay will do [17:10:26] thanks ottomata [17:10:32] !log restart webrequest and pageview_actor data purge - https://gerrit.wikimedia.org/r/c/operations/puppet/+/771389 [17:10:34] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:21:56] Oh you beat me to it. I just looked away for a second... [17:22:48] btullis: ottomata is often fastest :) [17:28:09] ottomata: in the forked gobblin we had, I had changed the guava version of the overall gobblin to 0.23 - it had compatiblity with EventUtilities [17:28:31] it'll be a lot more complicated with the unforked version :( [17:28:37] I had not forseen that one [17:28:46] okay i tried 0.25, but i'm getting those duplicate-finder errors [17:28:54] right, because we have upstream too [17:28:58] yup [17:28:59] i can try with 0.23 real quick [17:29:20] wait you ean 23.0 right? [17:30:05] ottomata: 0.23 was the version I used when fully rebuiding the thing - I think changing it in our small won't change - possibly we could remove dependnecies from upstream, but my patch required some code changes :( [17:30:11] I meant 23.0 yes sorry [17:31:11] OH, duplicate checker succeeds with 23?!??! [17:31:13] trying test [17:31:18] ack [17:32:57] i guess the checker-compat-qual and checker-qual version change was introduced after 23.0 [17:33:13] joal success! [17:33:28] incredible :) I wouldn't have beet on this :) [17:33:49] (03PS2) 10Ottomata: Change version of guava to 23.0 [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/771626 (https://phabricator.wikimedia.org/T297939) [17:34:44] ottomata: waiting for jenkins before sending my +2 :) [17:34:55] aye [17:43:17] joal: jenkins happy! [17:43:20] \o/ [17:43:23] Merging [17:43:38] ty [17:43:43] (03CR) 10Joal: [C: 03+2] "Merging for deploy in test" [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/771626 (https://phabricator.wikimedia.org/T297939) (owner: 10Ottomata) [17:47:05] (03Merged) 10jenkins-bot: Change version of guava to 23.0 [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/771626 (https://phabricator.wikimedia.org/T297939) (owner: 10Ottomata) [17:47:46] ok doing release [17:47:52] Starting build #4 for job analytics-gobblin-wmf-maven-release-docker [17:56:27] Project analytics-gobblin-wmf-maven-release-docker build #4: 09SUCCESS in 8 min 35 sec: https://integration.wikimedia.org/ci/job/analytics-gobblin-wmf-maven-release-docker/4/ [17:56:59] heya a-team, anybody has 10-15 mins to brainbounce around airflow DAGs dynamic configurationnnn?? [17:57:18] ooo i want to but i have meeting starting in 2 mins [17:57:24] same mforns [17:57:40] after the meeting I'll need to leave for ~1h, but I'll gladly exchange after that :) [17:57:57] pinging people directly, since I'm not sure all of us have trackers for a-team. aqu ^^^ [17:58:20] ok, ottomata and joal, maybe in a couple hours...? [17:59:54] mforns: hope so! [18:00:12] k [18:54:09] I'm out of that same meeting, happy to meet whenever now [18:54:15] mforns: o/ i have 15 minutes! [18:54:17] hehe [18:54:25] bc? [18:58:31] ottomata: ok! [18:58:32] omw [19:48:08] mforns: back for 10 mins before 1:1 [19:48:11] if you need more chats [19:53:43] (03PS1) 10Ottomata: Add gobblin-wmf-core-1.0.1-jar-with-dependencies.jar [analytics/refinery] - 10https://gerrit.wikimedia.org/r/771693 (https://phabricator.wikimedia.org/T297939) [19:54:33] heya mforns - I assume you've sorted stuff out with ottomata ? [19:54:47] (03PS2) 10Ottomata: Add gobblin-wmf-core-1.0.1-jar-with-dependencies.jar [analytics/refinery] - 10https://gerrit.wikimedia.org/r/771693 (https://phabricator.wikimedia.org/T297939) [20:35:20] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Persistence (Consultation), 10Data-Services, and 2 others: Toolforge db: View 'fiwiki_p.flaggedrevs' references invalid table/column/rights to use them - https://phabricator.wikimedia.org/T302233 (10razzi) Ok cool. Cookbook is merged, but I didn't reali... [20:37:14] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10Cmjohnson) [20:43:29] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10Cmjohnson) [21:00:16] (03PS1) 10Clare Ming: Add new enum value to webuiscroll schema [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/771700 (https://phabricator.wikimedia.org/T303297) [21:27:04] hey joal aqu and I are meeting now, sorry I didn't see your message, I assume you're not there any more. [22:06:38] (03PS1) 10Aqu: Fix Airflow job session length daily [analytics/refinery] - 10https://gerrit.wikimedia.org/r/771730 (https://phabricator.wikimedia.org/T300029) [22:28:25] (03PS2) 10Aqu: Fix Airflow job session length daily [analytics/refinery] - 10https://gerrit.wikimedia.org/r/771730 (https://phabricator.wikimedia.org/T300029) [22:31:21] (03PS3) 10Aqu: Fix Airflow job session length daily [analytics/refinery] - 10https://gerrit.wikimedia.org/r/771730 (https://phabricator.wikimedia.org/T300029) [22:50:22] (03CR) 10Mforns: [C: 03+2] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/771730 (https://phabricator.wikimedia.org/T300029) (owner: 10Aqu)