-
Notifications
You must be signed in to change notification settings - Fork 91
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Performance testing next steps (adoption) #1093
Comments
Thanks for kicking this discussion off, @swissspidy. I appreciate the distinction between two main objectives. For now, I'm going to limit my thoughts to the first objective, "Improve Core/GB performance testing". One of the things that I observed during the 6.5 release is that finding the source of a server timing regression was challenging when that regression was committed to the Core repo as part of a larger Gutenberg sync. I think we can improve this somewhat by updating the performance tests in the Gutenberg repo to include the same server timing metrics that we record for each commit to the WP Core repo (i.e., wpTotal, wpBeforeTemplate, and wpTemplate). While TTFB is a close proximity, the additional noise from network requests and calculating the metric in a headless browser makes pinpointing potential regressions more difficult. I also strongly agree with your suggestion to improve our demo content, and think this applies as much to the Core tests as the Gutenberg tests. Currently in Core, we are only testing the default homepage for the Twenty Twenty-One and Twenty Twenty-Three themes after importing a Theme Test Data from this commit. While this keeps test content consistent over time, there are a number of limitations to this approach including the fact that some specific use cases that we care about (e.g., measuring the effect of image optimizations on LCP) are not covered by our current test content. From a visualization point of view, our current dashboards at codevitals.run have become harder to use over time as we've added additional metrics. I'd love to investigate improving or replacing these dashboards with a system where we could more granularly filter results by metric, theme, template, object cache, etc. In doing so, we should also evaluate how we are normalizing and storing the raw data which currently gets normalized before it's saved to the dashboard's database meaning we don't have the ability to build new reports using the original unfiltered data. Last idea for now, is that when we introduced these tests we used Twenty Twenty-One and Twenty Twenty-Three as representative themes for classic and block themes, but that has proven to be overly simplistic as some performance regressions are only visible based on characteristics of a theme (e.g., how many template part variations they register). At minimum, we should add Twenty Twenty-Four to our test matrix in both repos. TL;DR:
|
That all makes sense to me. I think most of those suggestions are already mentioned in some place or another. I also previously explored a Grafana-based dashboard that would be more user friendly and could be fed raw data. Some more thoughts on adoption: Let's start with directly reaching out to some top plugins that we think could benefit from performance testing but don't leverage that yet. Find out why, figure out what's missing, and help them get started. |
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
Improve Core/GB performance testing Mentioned above:
More thoughts:
Adoption
The main points I've heard so far in various discussions:
Mentioned above:
|
I have a small backlog but I just added this to our board to make sure this gets a follow-up. |
Thanks for the updates, @swissspidy. This looks like a good list to me. I do think that separating the Core/GB work and the Adoption work into two separate epics likely makes sense, and will make it easier to track progress of the two initiatives, even if there is some overlap. From my perspective, focusing on improving and aligning the metrics that are taken and the way dashboard to display them between the WP and GB repos should be a high priority, so we're able to more quickly identify the likely cause of performance regressions based on server timing data earlier in feature development in the GB repo. We also need to fix some existing issues with the WP Performance workflows that have caused big jumps to show up in the dashboard due to the way the baseline checks were taken and add metrics for the TT4 theme in our WP dashboard view. |
@swissspidy do you mean preparing a generic GitHub action that plugin authors could use to get a performance testing link on their PRs? If yes, that sounds fantastic. There's a related Playground issue and here's some inspiration to reuse:
If, on the other hand, that's about running E2E tests locally or in CI, @WunderBart's explorations here are highly relevant:
If I misunderstood, would you share some more details about that question? |
@adamziel The idea is to provide a generic GitHub Action (https://github.com/swissspidy/wp-performance-action) that automatically runs e2e performance tests in CI. It already works (using wp-env) but blueprint support would be a killer feature. This way, projects can declaratively add test data/fixtures. @dmsnell originally suggested using Playground as we could then have full I/O and DB control, which would hopefully help with getting stable results and would be a nice side effect. WordPress/gutenberg#62692 comes close, it's just not stable enough :-) Regarding wp-now, Dennis mentioned some architectural issues that "make it less favorable for testing" I also saw https://github.com/adamziel/wordpress-php-blueprints/ but not exactly sure what it does 🤔 Could I leverage this to use blueprints with an arbitrary setup? This way, I could already use blueprints while Playground is still being improved upon. |
@joemcgill Extracted core improvements to #1380 now, we can keep this one for the adoption part |
Here's a first successful attempt at using Playground for the testing environment: swissspidy/wp-performance-action#173 |
Last year we made several big improvements in the performance testing area. To name a few:
As a reminder, the two main objectives we had:
While progress slowed down due to other priorities, there is still a lot of untapped potential this year. I am therefore opening this issue to kickstart a discussion on next steps.
Some enhancements/features that are already on our radar:
Server-Timing
, etc.Some loose ideas:
Curious to hear everyone's thoughts :)
The text was updated successfully, but these errors were encountered: