Triaging Magento 2 GitHub Issues
In my post "Magento 2 in 2017 - How's Things?", I brought up the point that the Magento 2 GitHub repo has averaged 11+ new issues per day in the past 30 days (and that includes a relatively slow period, between Christmas and New Year's). A large number of these issues are irrelevant, blank or otherwise adding noise that makes it hard for Magento to identify what the high priority/high impact issues are. If those issues can't be identified quickly, then the quick resolutions we're all hoping for certainly aren't going to happen.
From looking at Twitter, the community obviously really wants to help Magento with this issue, and Magento is working on this problem as well. So I want to help improve that debate and discussion by adding a few more thoughts and a point of view that may not have been considered yet. As always, I'm trying to bring a new angle to the conversation, so if you disagree or have a different point of view, hit me with it on Twitter or here in the comments.
First, let's do a little experiment. Go to the Magento 2 GitHub Issues page - https://github.com/magento/magento2/issues - and without reading the titles, just click blindly on one of the issues. Read through it, and assuming it's not complete gibberish, do the following:
1) Attempt to reproduce it on a clean installation of Magento 2. If you need a Magento 2 dev environment, check out MageScotch (http://github.com/joshuaswarren/magescotch/) and you'll have a clean Magento 2 install ready to experiment on with very little effort.
2) If you can reproduce it, drill down to what area of Magento you think the bug is coming from and write up a brief summary of the issue and some potential causes and the overall impact the issue has.
3) If you can't reproduce it, write up a brief summary of any reasons why you might be having trouble reproducing it and the results that you do see when you reproduce it.
Alright, now stop, and look at how much time has passed since you started this experiment. Now multiply that by the 11+ issues per day that are coming in to the Magento 2 repository - and you see the scope of the problem we're talking about.
I can hear some of you saying "Hey, I'm not a Magento 2 core developer, so of course I'm going to be slower at this because it's not what I do all day". However, you have to remember that the Magento 2 core team is very, very large. I doubt there's any single developer on the core team that truly knows the entire Magento 2 codebase, top to bottom, inside and out, to an expert level. This isn't because they aren't smart developers - it's because when you have a team of 100+ developers working on a massive framework that covers so many different functional areas, the only thing that makes sense is to have them specialize in a specific area of the codebase. So if you take any specific core developer and assign them to triage Magento 2 GitHub issues, there's a good chance that on any given issue they're looking at, they'll be looking at an area of Magento 2 that they aren't an expert on.
The problem then becomes - do you interrupt the developer that is an expert on that part of Magento 2 in order to triage each new issue that comes in for their area? Or do you have a team with general knowledge of Magento 2 just do their best to triage the issues that are coming in?
This is the same challenge that community volunteers face. No single community member knows Magento 2 inside and out, top to bottom. We all have our areas we're more and less experienced with.
So, as far as I can see, there's no solution to greatly reduce the amount of time spent triaging these issues, because almost any way you structure it, the developer(s) triaging the issues won't be experts in every area of Magento 2 that has issues opened.
There's also a psychological challenge here. Long, long ago, I worked in a job where I spent part of my day triaging and answering technical tickets opened by customers. At the top of the screen I worked from, there was a counter of how many tickets I had touched that day. Management watched that number, and used it as a measure of my work effort. Replying to a ticket and simply asking for more information, spending hours producing a patch for a client or just simply closing a ticket as "won't fix" all counted exactly the same. We also could see a count of the total number of open tickets, and felt psychological pressure when the count was increasing instead of decreasing. While I did my best to always provide excellent service, I definitely watched as some developers would pad their numbers by closing tickets. Over time, I noticed even good, honest developers would err on the side of closing tickets as "can't reproduce" more quickly than they probably should, simply because it made all the numbers look better, which made them feel better.
I definitely saw this happened on the Magento 2 GitHub Issues in early to mid-2016; I watched as issues that were important issues were closed quickly as "can't reproduce" with really no other details. I understand the pressure + when so many issues that are opened truly can't be reproduced, it's easy to fall into the habit of just closing things as can't reproduce.
So, how does Magento Inc structure things so that issues are quickly triaged, classified and prioritized? Knowing that it's going to take a lot of time, and that any metrics that try to decrease the time it takes for issues to be triaged are likely to cause issues to be handled inefficiently, what do you do?
The best thing I can think of so far - and there are aspects to this idea that I don't like - is a method by which issue reporters are measured, and issues opened by 'known good' issue reporters are treated differently than issues opened by people opening their very first issue, and. both of those groups of issues are treated different from issues opened by issue reporters that have a history of opening low quality issues.
Basically - the first time you open an issue, you have no history on the repo, so the issue is triaged exactly like it is now. As the issue is triaged, it's tagged as basically well defined, needs improvement or outright closed.
Then, as you have opened more issues over time, if your previous issues have a history of being tagged as well defined, those issues are triaged before the ones that are opened by individuals with a history of issues that are outright closed.
There's a couple of things I really don't like with this idea - first, I don't ever, ever want to discourage people from contributing to the Magento community in any way. I hate things that even give the hint of elitism. Second, sometimes, very important issues are opened by 'first timers', and I don't like that those would be treated with a lower priority.
Perhaps there could be a way for anyone that has a history of having their issues tagged as well defined to 'vouch' for an issue, lending their credibility to the issue?
This is probably a terrible idea. But it's the one I have right now, and maybe it will spark an interesting conversation or other suggestions. What do you think?