[00:03:37] (Nonwrite HTTP requests with primary DB connections alert) firing: - https://alerts.wikimedia.org/?q=alertname%3DNonwrite+HTTP+requests+with+primary+DB+connections+alert [00:08:16] (MediaWikiLatencyExceeded) resolved: Average latency high: eqiad parsoid GET/200 - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red-dashboard?panelId=9&fullscreen&orgId=1&from=now-3h&to=now&var-datasource=eqiad%20prometheus/ops&var-cluster=parsoid&var-method=GET - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [00:18:50] PROBLEM - Check systemd state on an-web1001 is CRITICAL: CRITICAL - degraded: The following units failed: hardsync-published.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [00:23:37] (Nonwrite HTTP requests with primary DB connections alert) resolved: - https://alerts.wikimedia.org/?q=alertname%3DNonwrite+HTTP+requests+with+primary+DB+connections+alert [00:31:10] RECOVERY - Check systemd state on an-web1001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [00:38:43] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/940211 [00:38:49] (03CR) 10TrainBranchBot: [C: 03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/940211 (owner: 10TrainBranchBot) [00:53:21] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/940211 (owner: 10TrainBranchBot) [01:13:16] (MediaWikiLatencyExceeded) firing: Average latency high: eqiad parsoid GET/200 - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red-dashboard?panelId=9&fullscreen&orgId=1&from=now-3h&to=now&var-datasource=eqiad%20prometheus/ops&var-cluster=parsoid&var-method=GET - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [01:18:16] (MediaWikiLatencyExceeded) resolved: Average latency high: eqiad parsoid GET/200 - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red-dashboard?panelId=9&fullscreen&orgId=1&from=now-3h&to=now&var-datasource=eqiad%20prometheus/ops&var-cluster=parsoid&var-method=GET - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [01:23:16] (MediaWikiLatencyExceeded) firing: Average latency high: eqiad parsoid GET/200 - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red-dashboard?panelId=9&fullscreen&orgId=1&from=now-3h&to=now&var-datasource=eqiad%20prometheus/ops&var-cluster=parsoid&var-method=GET - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [01:38:16] (MediaWikiLatencyExceeded) resolved: Average latency high: eqiad parsoid GET/200 - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red-dashboard?panelId=9&fullscreen&orgId=1&from=now-3h&to=now&var-datasource=eqiad%20prometheus/ops&var-cluster=parsoid&var-method=GET - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [01:42:16] (MediaWikiLatencyExceeded) firing: Average latency high: eqiad parsoid GET/200 - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red-dashboard?panelId=9&fullscreen&orgId=1&from=now-3h&to=now&var-datasource=eqiad%20prometheus/ops&var-cluster=parsoid&var-method=GET - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [02:07:32] (JobUnavailable) firing: (2) Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:18:16] PROBLEM - Check systemd state on gitlab1003 is CRITICAL: CRITICAL - degraded: The following units failed: sync-gitlab-group-with-ldap.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [02:19:16] PROBLEM - Check systemd state on gitlab2002 is CRITICAL: CRITICAL - degraded: The following units failed: sync-gitlab-group-with-ldap.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [02:30:14] RECOVERY - Check systemd state on gitlab1003 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [02:31:18] RECOVERY - Check systemd state on gitlab2002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [02:32:32] (JobUnavailable) resolved: (2) Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [04:11:54] PROBLEM - mailman archives on lists1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [04:13:14] RECOVERY - mailman archives on lists1001 is OK: HTTP OK: HTTP/1.1 200 OK - 50276 bytes in 0.079 second response time https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [04:20:02] PROBLEM - mailman list info on lists1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [04:21:24] RECOVERY - mailman list info on lists1001 is OK: HTTP OK: HTTP/1.1 200 OK - 8571 bytes in 0.272 second response time https://wikitech.wikimedia.org/wiki/Mailman/Monitoring [05:42:16] (MediaWikiLatencyExceeded) firing: Average latency high: eqiad parsoid GET/200 - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red-dashboard?panelId=9&fullscreen&orgId=1&from=now-3h&to=now&var-datasource=eqiad%20prometheus/ops&var-cluster=parsoid&var-method=GET - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [06:46:48] (03CR) 10Andrea Denisse: [C: 03+1] "LGTM, thank you!!" [puppet] - 10https://gerrit.wikimedia.org/r/937601 (https://phabricator.wikimedia.org/T234565) (owner: 10Cwhite) [07:00:06] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20230722T0700) [08:50:16] (PHPFPMTooBusy) firing: Not enough idle php7.4-fpm.service workers for Mediawiki parsoid at eqiad #page - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red-dashboard?from=now-3h&orgId=1&to=now&var-cluster=parsoid&var-datasource=eqiad%20prometheus/ops&viewPanel=64 - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [08:55:16] (PHPFPMTooBusy) resolved: Not enough idle php7.4-fpm.service workers for Mediawiki parsoid at eqiad #page - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red-dashboard?from=now-3h&orgId=1&to=now&var-cluster=parsoid&var-datasource=eqiad%20prometheus/ops&viewPanel=64 - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [09:02:33] hey folks [09:02:37] just seen the page [09:03:36] the two extra nodes that we added yesterday didn't make a big difference [09:06:27] nothing really out of the ordinary afaics from the graphs [09:06:48] the graph is not 100% in line with the alert though [09:06:59] morning [09:07:43] this is the alert that was p.aging in the last couple of days too, isn't it? [09:09:22] exactly yes [09:09:24] https://grafana-rw.wikimedia.org/d/RIA1lzDZk/application-servers-red?forceLogin&from=now-2d&orgId=1&refresh=1m&to=now&var-cluster=parsoid&var-datasource=eqiad%20prometheus%2Fops&viewPanel=22 [09:09:31] dunno if there's anything Saturday-straightforward that could be done to stop it firing over the weekend? [09:09:33] this stands out, I think it was called out in the task [09:09:38] lemme find it [09:11:48] https://phabricator.wikimedia.org/T342085 [09:14:51] I'm going to bump that task to High, since it's p.aging on a weekend :-/ [09:16:18] I don't think I want to try and shuffle host allocations further, especially since the p.age self-resolved, but I suspect that's going to result in more alerts over the weekend :-/ [09:17:28] yeah [09:17:53] so I am reviewing the metric and alert rule for PHPFPMTooBusy in the alerts repo, it is not 100% clear to me why it fired [09:18:07] the time windows is 2minutes, so very tight [09:18:39] I think because some not-enough-workers problems represent a serious outage? [09:18:42] (03PS12) 10Winston Sung: SiteMatrix config: Add actual (non-deprecated) language code for deprecated language codes [mediawiki-config] - 10https://gerrit.wikimedia.org/r/884494 (https://phabricator.wikimedia.org/T172035) [09:19:14] [at least per https://bit.ly/wmf-fpmsat ] [09:20:42] yes yes that is understood, I tried to render the metric for the alarm and I don't see it running under 0.3 [09:21:06] the graph attached to the alert is not the same as what we have in the alert rule afaics [09:21:12] sorry, I'll let you work on that and stop asking silly questions :) [09:21:40] nono please it may be a saturday's pebcak :D [09:21:53] my line of thinking is that maybe the 2min window is too tight [09:25:47] if I'm reading the graph correctly ( https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red-dashboard?from=now-3h&orgId=1&to=now&var-cluster=parsoid&var-datasource=eqiad%20prometheus/ops&viewPanel=64 ) from the p.age we're skirting around the 0.3 window all the time ATM [09:27:08] and the graph uses, afaics [09:27:09] sum by (state) (phpfpm_statustext_processes{site=~"$site",service="php7.4-fpm.service",cluster="$cluster"}) [09:27:36] but the PHPFPMTooBusy alarm in the alerts repo uses another config [09:27:42] sum by (cluster, service) (phpfpm_statustext_processes{cluster=~"(api_appserver|appserver|parsoid)", state="idle"}) [09:27:45] / [09:27:47] sum by (cluster, service) (phpfpm_statustext_processes{cluster=~"(api_appserver|appserver|parsoid)"}) [09:27:50] <= 0.3 [09:27:54] hang on a minute - isn't that graph showing about 30% _used_ so about 70% idle? [09:28:58] e.g. at 08:50 UTC it has 984 active and 1764 idle [09:29:39] <-- confused [09:30:15] if you render the above it shows more or less the same [09:30:44] I am confused as well [09:32:34] the panel is graphing "sum by (state) (phpfpm_statustext_processes{site=~"(eqiad|codfw)",service="php7.4-fpm.service",cluster="parsoid"})" [09:33:23] [once logged in, select "explore"] [09:33:51] (03PS1) 10Jelto: vrts: enable blackbox check on active_host only [puppet] - 10https://gerrit.wikimedia.org/r/940467 (https://phabricator.wikimedia.org/T342366) [09:34:29] Emperor: right, and selecting only say eqiad gets to a different view [09:34:47] now it makes more sense, it was smoothed out between eqiad and codfw [09:35:21] and around 8:50 it crossed the mark [09:35:25] so all checks out [09:35:32] maybe we could improve the alert's text [09:36:29] in eqiad I think the threshold is about 392 idle processes? [09:36:49] [because I think there are about 1300 total, and that's 0.3 of that] [09:37:07] yes yes it checks out now [09:37:34] but maybe the time window is too short, 2 minutes is good for us to react but it may take into account small variations [09:37:45] I'd propose to bump it to 5m, at least for the moment [09:38:14] that or push the threshold to 0.25 ? [09:38:34] (03CR) 10Jelto: [V: 03+1] "PCC SUCCESS (NOOP 1 DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/42646/console" [puppet] - 10https://gerrit.wikimedia.org/r/940467 (https://phabricator.wikimedia.org/T342366) (owner: 10Jelto) [09:38:55] 0.23 would be about 300 idle, which would give us more headroom [09:39:36] (I'm a bit torn between wanting to avoid people getting p.aged all weekend for this and overly-enthusiastically silencing a real problem if one turns up) [09:41:53] (03PS1) 10Elukey: team-sre: fix graph for the PHPFPMTooBusy alert [alerts] - 10https://gerrit.wikimedia.org/r/940469 (https://phabricator.wikimedia.org/T342085) [09:42:11] Emperor: in the meantime this should fix the graph that we get --^ [09:42:16] (MediaWikiLatencyExceeded) firing: Average latency high: eqiad parsoid GET/200 - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red-dashboard?panelId=9&fullscreen&orgId=1&from=now-3h&to=now&var-datasource=eqiad%20prometheus/ops&var-cluster=parsoid&var-method=GET - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [09:43:23] ah we have the same problem in this graph sigh --^ [09:43:32] (03CR) 10CI reject: [V: 04-1] team-sre: fix graph for the PHPFPMTooBusy alert [alerts] - 10https://gerrit.wikimedia.org/r/940469 (https://phabricator.wikimedia.org/T342085) (owner: 10Elukey) [09:43:42] p95 is definitely a high [09:43:50] the baseline of the lower one increased [09:44:37] ah yes the tests [09:45:00] elukey: there are some tests in mediawiki_test.yaml which fail now [09:45:13] ah you found it, good :) [09:46:20] yes yes fixing them :) [09:47:46] (03CR) 10MVernon: [C: 04-1] "This is a good change, but I think the URL needs tweaking a bit (see inline)." [alerts] - 10https://gerrit.wikimedia.org/r/940469 (https://phabricator.wikimedia.org/T342085) (owner: 10Elukey) [09:48:17] elukey: I think your URL change wasn't quite right [09:48:46] [and I think for non-p.aging alerts lets make a task to note the graphs need improving, but don't feel we have to fix them all on a Saturday] [09:49:17] Emperor: why not? [09:49:59] ah yes yes you are right, I was fixing it [09:50:26] I think we should fix it, we spent a solid 10 minutes figuring out the issue [09:51:15] FE (I'm not sure how many alerts we're talking about needing to fix) [09:51:24] [sorry, FE = fair enough] [09:53:22] (03PS2) 10Elukey: team-sre: fix mediawiki graphs using the RED dashboard [alerts] - 10https://gerrit.wikimedia.org/r/940469 (https://phabricator.wikimedia.org/T342085) [09:53:39] (03CR) 10Elukey: team-sre: fix mediawiki graphs using the RED dashboard (031 comment) [alerts] - 10https://gerrit.wikimedia.org/r/940469 (https://phabricator.wikimedia.org/T342085) (owner: 10Elukey) [09:54:45] ok better now :) [09:55:44] and about the time window: https://grafana-rw.wikimedia.org/d/RIA1lzDZk/application-servers-red?from=1690015311474&orgId=1&to=1690016070094&var-cluster=parsoid&var-datasource=eqiad%20prometheus%2Fops&forceLogin&var-site=eqiad&var-method=POST&var-code=200&var-php_version=All&editPanel=64 [09:56:15] afaics it seems that we crossed the threshold for 3 mins, this is why I think that the 2 min window may catch false alerts [09:56:35] especially like in this case that we are close to threshold [09:56:40] it will surely fire again this weekend [09:58:03] [just checking all the revised URLs in your CR] [09:59:01] <3 [10:00:50] (03CR) 10MVernon: [C: 03+1] "The revised URLs all seem good to me, thanks for fixing these!" [alerts] - 10https://gerrit.wikimedia.org/r/940469 (https://phabricator.wikimedia.org/T342085) (owner: 10Elukey) [10:01:47] OK, let's push it to 5m and get the relevant people to review on Monday? [10:02:05] filing the change [10:02:41] (03PS1) 10Elukey: team-sre: increase the time window for PHPFPMTooBusy [alerts] - 10https://gerrit.wikimedia.org/r/940471 (https://phabricator.wikimedia.org/T342085) [10:03:04] (03CR) 10Elukey: [C: 03+2] team-sre: fix mediawiki graphs using the RED dashboard [alerts] - 10https://gerrit.wikimedia.org/r/940469 (https://phabricator.wikimedia.org/T342085) (owner: 10Elukey) [10:03:19] I think you'll need to fix the tests too [10:04:43] OK, evidently not :) [10:04:43] (03Merged) 10jenkins-bot: team-sre: fix mediawiki graphs using the RED dashboard [alerts] - 10https://gerrit.wikimedia.org/r/940469 (https://phabricator.wikimedia.org/T342085) (owner: 10Elukey) [10:05:40] (03CR) 10MVernon: [C: 03+1] "I think this is a sensible approach for the weekend at least." [alerts] - 10https://gerrit.wikimedia.org/r/940471 (https://phabricator.wikimedia.org/T342085) (owner: 10Elukey) [10:06:47] Emperor: thanks for the review, ok to merge for the weekend? [10:07:06] I'll take the blame for service ops in case [10:07:08] :D [10:08:59] (03CR) 10Elukey: [C: 03+2] team-sre: increase the time window for PHPFPMTooBusy [alerts] - 10https://gerrit.wikimedia.org/r/940471 (https://phabricator.wikimedia.org/T342085) (owner: 10Elukey) [10:09:15] done, I'll update the task [10:09:24] I think we can get back to the weekend folks [10:09:27] thanks! [10:09:28] o/ [10:09:46] have a good rest-of-weekend :) [10:10:06] (03Merged) 10jenkins-bot: team-sre: increase the time window for PHPFPMTooBusy [alerts] - 10https://gerrit.wikimedia.org/r/940471 (https://phabricator.wikimedia.org/T342085) (owner: 10Elukey) [10:11:37] you too! [13:42:16] (MediaWikiLatencyExceeded) firing: Average latency high: eqiad parsoid GET/200 - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red-dashboard?panelId=9&fullscreen&orgId=1&from=now-3h&to=now&var-site=eqiad&var-cluster=parsoid&var-method=GET - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [13:49:16] PROBLEM - Check systemd state on an-launcher1002 is CRITICAL: CRITICAL - degraded: The following units failed: produce_canary_events.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [14:01:20] RECOVERY - Check systemd state on an-launcher1002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [14:07:32] (JobUnavailable) firing: (2) Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [14:17:32] (JobUnavailable) resolved: (2) Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [17:42:16] (MediaWikiLatencyExceeded) firing: Average latency high: eqiad parsoid GET/200 - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red-dashboard?panelId=9&fullscreen&orgId=1&from=now-3h&to=now&var-site=eqiad&var-cluster=parsoid&var-method=GET - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [20:01:05] (SwiftTooManyMediaUploads) firing: (2) Too many eqiad mediawiki originals uploads - https://wikitech.wikimedia.org/wiki/Swift/How_To#mediawiki_originals_uploads - https://alerts.wikimedia.org/?q=alertname%3DSwiftTooManyMediaUploads [20:31:05] (SwiftTooManyMediaUploads) firing: (2) Too many eqiad mediawiki originals uploads - https://wikitech.wikimedia.org/wiki/Swift/How_To#mediawiki_originals_uploads - https://alerts.wikimedia.org/?q=alertname%3DSwiftTooManyMediaUploads [21:21:05] (SwiftTooManyMediaUploads) resolved: Too many eqiad mediawiki originals uploads - https://wikitech.wikimedia.org/wiki/Swift/How_To#mediawiki_originals_uploads - https://grafana.wikimedia.org/d/OPgmB1Eiz/swift?panelId=26&fullscreen&orgId=1&var-DC=eqiad - https://alerts.wikimedia.org/?q=alertname%3DSwiftTooManyMediaUploads [21:42:16] (MediaWikiLatencyExceeded) firing: Average latency high: eqiad parsoid GET/200 - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red-dashboard?panelId=9&fullscreen&orgId=1&from=now-3h&to=now&var-site=eqiad&var-cluster=parsoid&var-method=GET - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [22:05:36] (GitLabCIPipelineErrors) firing: GitLab - High pipeline error rate - https://wikitech.wikimedia.org/wiki/GitLab/Runbook - https://grafana.wikimedia.org/d/Chb-gC07k/gitlab-ci-overview - https://alerts.wikimedia.org/?q=alertname%3DGitLabCIPipelineErrors [22:10:36] (GitLabCIPipelineErrors) resolved: GitLab - High pipeline error rate - https://wikitech.wikimedia.org/wiki/GitLab/Runbook - https://grafana.wikimedia.org/d/Chb-gC07k/gitlab-ci-overview - https://alerts.wikimedia.org/?q=alertname%3DGitLabCIPipelineErrors [22:31:47] (03PS1) 10Hamish: Add botadmin group on eswiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/940486 (https://phabricator.wikimedia.org/T342484) [23:12:48] 10SRE, 10Wikimedia-Mailing-lists: Add custom footer linking to Privacy Policy in Postorious and Hyperkitty - https://phabricator.wikimedia.org/T340375 (10Ladsgroup) I wonder if UCoC is a better option here. Most mailing lists are not covered by TCoC.