Community Configuration: Measurement & Instrumentation
Open, HighPublic

Description

User story & summary:

As the Growth team, I want to better understand Community Configuration use, because then we can further improve and support this feature.

Details:

Community Configuration edits are publicly tracked on the associated history page, so instrumentation needs are fairly minimal. However we would like to be able to answer the following questions:

  • How often is the Community Configuration dashboard (Special:CommunityConfiguration) visited?
  • How often is an individual Community Configuration edit form visited? Edit form example: Special:CommunityConfiguration/GrowthSuggestedEdits
  • How often is a Community Configuration edit completed?

Ideally we can track these metrics both cumulatively and per wiki.

To answer these questions, we should be able to use the existing data contract available via the Metrics Platform (utilize the standard core interactions schemas).

https://wikitech.wikimedia.org/wiki/Metrics_Platform/How_to/Create_An_Instrument

Questions:
  • Q: Should we make Dashboard and Edit Form pageviews accessible via https://pageviews.wmcloud.org/pageviews?
    • A: ideally yes, but this can be a separate task.
  • Q: Should we instrument when a change is started, but not saved?
    • A: ideally yes, but this can be a separate task to consider in the future. The reality is that people may explore the feature without saving so this data won't really tell us if users are having trouble with the feature.
  • Q: Should we instrument when a form error is triggered?
    • A: Ideally yes, but this can be covered in a separate task if this isn't a standard part of the Metrics Platform tracking.
  • Q: How can we track configuration changes across wikis?
    • A: Stewards, Movement Coms, and other stakeholders may be interested in easily tracking when a certain Community Configuration setting is changed on any wiki. For EditGrowthConfig, we have: Community_configuration_usage. Will we maintain this for the new Community Configuration extension, or create a better cross-wiki dashboard or Watchlist option? We need to answer this, but I don't think it needs to be answered in this task.
Acceptance Criteria:
  • Utilize the Metrics Platform's core interactions schemas to establish basic instrumentation on the Community Configuration dashboard and associated forms.
  • Product Manager + Data Scientist complete Data Collection Guidelines data collection activity log form before work is merged. (In theory, this tracking should be categorized as "Low Risk", but we should follow the process outlined in the Data Collection Guidelines to double-check.)

Event Timeline

Needs more discussion: ideally we can work on this at the same time as starting the Community Updates instrumentation work

Yes @KStoller-WMF, we can even do Community Updates instrumentation first to learn and then use those learnings in this.