So, how are we triaging bugs/requests in this forum?

Jerud

It’s simple enough to go to bugs / requests, search to see if my issue is already reported/suggested, and then either upvote & comment on the existing issue or write up my own. But then what? The process can’t stop there or it will be no better than Slack or Trello were.

BUGS

Who applies the “severity” tags to bug reports? Moderators? Will there be any moderators aside from those on the Legend team?
How often do moderators triage the bugs category? Lots of posts have no tags at all aside from the category they are posted in.
There are “low” and “high” priorities but no “middle”. Are there only two grades of severity and no-tag means it hasn’t been graded yet?
What about non-severity tags; can we apply our own tags at all? It appears the forum categories work like google Labels – so a post could exist in multiple categories at once. For example: ‘bugs’ + ‘mobile’. Users could be more helpful by applying relevant tags such as ‘mobile’ and save the moderators from having to do that. Presumably just tagging with ‘bugs’ means mobile, desktop, and web – but what if an issue is only on desktop/web and not mobile (or n/a for mobile because the feature doesn’t exist)? What if a bug appears in both beta and stable – there’s no explicit ‘stable’ tag so how would this be made clear?
Suggestion: add a “reproducible” tag. Having a tag to identify bugs that actually have reproduction steps would let the Team quickly identify “actionable” bugs and knock them out. Yet it would still allow us to document those mystery bugs we can’t quite figure out. Someone later on might encounter it again, search on it, review our notes, and be able to figure out the pattern to actually reproduce. Then they could apply the ‘reproducible’ tag and that bug can find its way into the queue…otherwise those will just never get fixed.

It seems like having a couple more tags and a clear, public process about how they get used could not only allow users to help out the team with more detail for better sorting/triaging, but also take some of the workload off the mods and garner more community engagement.

FEATURES

Will there be any tags used with feature requests so we can see the status of individual requests? Say I have a request, search for it, and find an existing post – it’s 8 months old and has lots of upvotes but has no comments and no tags. I’m going to wonder if you’ve actually “noticed” that request and have it on your roadmap, deferred it for later, dismissed it as being impractical/too difficult, or maybe have overlooked it entirely. If you want folks to search before posting, the things they find need to show evidence that the rest of the process is actually working. If all I ever see are old posts with no upvotes or comments…then we’re back where we were with Trello and Slack. It doesn’t look like you’re paying attention or care. And you’ll probably see more ‘features’ entered as ‘bugs’ because they get more attention that way – which just makes your bug triaging harder.
Feature requests can easily get buried due to lack of interest. If a bug only happens to one person once, nobody will ever search for it, it’ll never get upvotes, and will just get buried – which is appropriate. But feature requests should not be handled the same way. Sure – if someone has the same idea as my request and searches for it, they can upvote and improve the chances of that feature happening. But this means only features that more than one person independently conceive of have any chance of implementation – and that’s not right. A suggestion can be “good” even if only one person thinks of it. If we were all reviewing the “requests” channel regularly and diligently upvoting every post that we liked, there would be no issue – but I don’t think that’s a realistic expectation. I certainly don’t “check in” with ANY forum regularly, and Legend isn’t going to be an exception. If I get an email notification or have something to submit, I’ll look around while I’m here. But I’m not going to just crawl the forum for new posts like its my job. I think you may miss a lot of great suggestions because of this, and you may end up with a big pile of requests that have no tags and no votes, just languishing in purgatory because nobody sees them – which sends the wrong message to the community.
So, suggestion: What if the moderators periodically dug out old posts and highlighted them to the community for feedback? Like once a week? Maybe there is a “weekly digest email” that the forum SW automatically generates, and this can be added to it? Just start with the oldest and surface each post for upvotes and discussion. If it doesn’t hit some threshold, tag it as ‘rejected’ and move on.

GENERAL
Finally, a general question: How will duplicate posts be handled? Can moderators merge or attach them to each other? If I discover two posts about the same thing and each has 3 upvotes, can I ask a mod to combine them into a single post with 6 upvotes?

DaveT

Hmm, Jerud I didn’t read all this in detail, but @Jay , he’s correct in identifying the need for an explainer on how you want to handle things / how ofter things should be updated. Otherwise people will lose the will to contribute.

Feature requests: Again this needs thinking about. As I have repeatedly said, having a clearly visable and clearly updated development roadmap is a very, very important hook to get new users on-board, especially corporate ones.

Can the forum labels have sub-labels? If can do, then maybe:
‘Features’ has un-triaged feature suggestions.
‘Features-ongoing’ lists those items currently being developed.
‘Features-next up’ lists those features you’ve scheduled for development.
‘Features-serious contenders’ lists those items that you think might be soon to move to next-up.
etc, etc.

(in the above, the number of sub-labels and descriptors are intended to give the idea, not in anyway to be a definitive suggestion).

Really think that these are good ideas.

Jay

A lot of really great points here. I’ve been distracted with trying to get the design update ready to launch (which it seems like it might be now) and fixing some bugs. But I will come back here after that with much deeper thoughts and start working on improving it soon. I wasn’t sure how it would end up in the beginning so I didn’t go too deep on adding lots of tags, but now that we have a good amount of users and activity here we can set up some good structure.

The forum supports one primary category and multiple secondary tags. Primary categories are Features, Bug Reports, General, Beta, etc… So far the only secondary tags are Solved, Low Severity, Urgent, and Mobile. But we can certainly add more.

I made the voting buttons more prominent than the forum default hoping to use that to raise up priority. But the number of comments can also do that. It doesn’t seem like voting is being used much so far, but maybe with more users and more guidelines it’ll be more helpful.

The forum supports merging (combining duplicate posts) as well as splitting (pulling a message out into its own thread) which I’ve used a few times already and is super helpful.

Also I would be happy to make you a mod if you would like?

DaveT

Jay Thanks for the reply - I’m much relieved to hear your thoughts on this and that you have a plan to come back and look further at this. The only other thing that I would add (but am imagining that you’ll do this anyway) is to have the flexibility to come back and modify any working structure to better suit how the system actually gets used!

I think that I’ve said before that number of comments can be misleading, but it certainly is a measure of interest in something. When I was involved in running a software dev co., we always wanted to know what features clients wanted, but also why they wanted them - turns out quite often people react in the moment, voting for something that solves an immediate need, but which is a one-off better solved elsewhere. This does nothig to help people think about the bigger features. Maybe I’m over-thinking this, but maybe each vote should come with a comment to explain why that vote has been cast for that feature/bug? And I think that limiting the number of votes you can have also ‘forces’ people to think more carefully about where to give them. Anyway, really interested to see what you’re thoughts are when you’ve time to get back to this.

As to being a mod - I’d really like to help this project in someway. Not sure I have the skills to mod though. Perhaps yes with initial backstopping by you?

Jerud

Thanks for explaining the primary/secondary tagging.

I suggest two “sets” of secondary tags: one for platform (windows | macos | web | android | ios) and one for severity (solved | low | high). I see that tags can be colored so you could use that to help make it easier to see that both types of secondary tag get applied to each post, e.g. make the platform tags all be “cool” colors and the severity tags be “warm” colors. Or maybe platforms are all just different shades of grey while severity uses the conventional “traffic light” color progression.

For bugs, a way to show that a particular post has or doesn’t have reproduction steps would be really helpful. I initially thought of this as a tag, but to avoid tag-crowding maybe this just gets added to the end of the Post title as a text tag like <reproducible> or similar?

I do feel like we need more resolution in severity tags (but of course not TOO much!), because you need a way to quickly spot major function- or data- loss – which must be “high” severity – but then that only leaves “low” to handle everything else ranging from misspelled tooltips to unexpected filter behaviors. That doesn’t seem very helpful. Instead of the nebulous <low; med; high> which creates a lot of hassle with categorizing, perhaps make the tags reflect their actual criteria, such as (just brainstorming): <cosmetic; functional; critical>. Cosmetic captures issues that don’t ‘break’ anything but nonetheless add up to a poor UX. Functional bugs are things that actually impact how Legend works, and Critical are bugs that result in loss of an entire function or any user data.

Making the platform tags more specific would streamline everything, not only by letting you sort/group by specific platform but by cutting out the “what platform is this on” question that gets asked with such regularity. Maybe those only get used for bugs, not requests (Requests can just say mobile/web/desktop in the text).

Mods should review new bug/feature posts and ensure they have both types of tag, and can respond back to submitters if they are missing the basics. Probably mods would have to edit post titles to include the “reproducible” label, since you definitely can’t count on submitters to do that.

I’d be stoked to help moderate, assuming it’s something I’m actually competent at doing. I “think” I know what moderators do in forums, but have never been one so probably don’t fully understand it. I’m good at following instructions though, so if you were to spell out your expectations I can certainly take a swing at it!

As much as I’d love to see a very structured voting/ranking system, I’m wary of that taking too long. In terms of having an effective “pipeline” for both bugs and features, I feel like we’ve gone from a “4” with Slack to a “5” with this Forum, but just spelling out a clear process and adding maybe a couple mods and tags could quickly get us to an “8”, whereas more structured voting could get us all the way to “10” but will take longer. I’d prefer to grab the low-hanging fruit and then revise later if needed.

Jorge

In addition to this it would also help to create a template for a bug report. People are extremely inconsistent in how they fill their bug reports when faced with just open text. I myself have sinned of that when just wanted to fill out a quick report because I am annoyed at something that stopped working, but I am so pressed for time that I just want to do it and get back to work and very few people are as detailed minded as @Jerud in describing their problems. Beta testers tend to be more involved, but common users tend to fill out bug reports like “Date screen not working” which is next to useless to the development team.

A bug report temlate shouldn’t be too overloaded so as to become onerous to the user but provide sufficient information to try to triage faster.

I suggest a template that includes fields like:

Platform (Web App, Windows Dekstop, Mac Desktop, Android Mobile, Apple Mobile) …
I most always forget to indicate where I am seeing the problem, and this helps the developer group by all kinds of problem sin a given platform when he is working on that code section.
Given what you were doing before the error happened.
When (the type of operation or sequence of steps that produced the error)…. helps reproduc eit
Then (the description of the undesired behavior and it’s consequence)… helps judge impact

Jerud

The template suggestion is solid.

Thinking about this a bit more, though…no amount of templates, tags or other complications will yield high quality results with just everybody out there submitting reports. If this forum will be the community hub, with a link on the app homepage into it, we can expect a LOT more users here than there were on Slack, and a large portion aren’t going to be “fully engaged”, long-term members. That’s just how forums are. There’s going to be piles of garbage to deal with.

So what about a gated process? Users submit their bugs/requests – just like now – into an open Channel that is essentially an “inbox”. Moderators then process those messages, going back to the poster to pull out missing information, applying tags, merging duplicates, and then moving those posts into a different Channel, which is visible and open for comments but not NEW posts.

This way the dev team can ignore the chaotic open channel and just focus on the quality info in the closed one. Posts can be organized in whatever way works best for the developers, without having to worry if it’s too much or too confusing for the general userbase. There will still be discussion and voting, and everybody can see a clear, consistent, and well-organized list of open issues. Might even provide a (limited) a short-term quasi-roadmap without having to build anything new – just add a few more tags.

Jorge

Jerud Agree completely with @Jerud as volume picks up the amount of real issues to be triaged has the potential of being overwhelming for @Jay . I think there have to be two levels of people triaging. The first level is kind of customer support where answers, or poitners, or guiding steps or even conversations with the creator of the error trying to reproduce the issue should be handled until a real issue is identified with reproducible steps. Before the issue is fully reproducible by someobdy that knows the system it shouldn’t make it into the dev team’s inbox.

The 2nd level is what @Jerud is mentioning, the real moderators who can assess the issue and prioritize accordingly so that the dev team can be fully productive in solving real issues of importance.

Then also comes the feedback to the users… As @Jay picks up items from the backlog produced by the moderator he could idetnify them by “In Progress”, or “In Development” to maintain the loop of info back.

Hopefully when @Jay can setup a WORM type of sharing model in Legend (Write by One person, Read by Many) I would vote to establish this triaging/roadmap process in Legend itself.

Jay

I’ve been thinking about this a lot and I have some ideas:

We could have four levels of priority: low, medium, high, urgent. Urgent is red with a warning icon while the others are just gray and used for prioritizing. Then we can view the High Priority subcategory for a well-sorted todo list to do next.

I’m hesitant to mark threads with a level of progress like “ongoing”, “next up”, etc… Priorities shift a lot, so something that was marked as “next up” might get deprioritized but stay marked as “next up” unless we take time to periodic re-tagging. But I think if we use priority tags well, it could generally be assumed that urgent things would happen first, and high priority things will be happening next, etc… An “in development” tag could be good, especially since sometimes I have already fixed the bug but the update release is delayed by another task I want to get in the update, or if I fix it at night and want to wait until morning to release it.

I like the idea of tagging whether it needs reproducing, but at least for now most things have been easily reproducible or people already shared repro steps, so I think we should bias the other way and have a “Needs reproducing” tag.

Similar to reproducing, there’s some issues where the right solution isn’t clear yet and it needs further discussion. So we could have a “Needs discussion” tag for when we need more information before we can start working on it?

Since most discussion at this point is about desktop versions which are mostly consistent across web/mac/windows, and the mobile apps are generally consistent across iOS and Android, I think we should be good with just the one “Mobile” tag. Anything that’s not mobile is assumed to be desktop. I think adding more granularity there is not worth the effort, at least at this point.

The cosmetic/functional distinction is interesting, but I wonder if it would add too much complication over simple priority levels? In theory cosmetic issues should be lower priority and functional issues should be higher priority? But maybe some examples would help me understand how that distinction could be helpful?

Another possible set of tags could be for scope, as in how long development would take. There could be tags like Scope:small, Scope:medium, Scope:big. That would be hard for mods to know, but I could add them on my daily triage. And then maybe on some days we could work on a bunch of lower priority but small ones, so they don’t get left in the dust?

And another tag idea is “Popular” - Number of votes or comments should in theory bubble up the popular posts. And comments bring them to the top so they should be in our face a lot. But mods or I could make it more official by tagging as “Popular” because a lot of people want it. So then a bug/feature that is technically low priority but a lot of people want could get bubbled up.

That got long, so a summary of tags I’m proposing:
Pri:urgent, Pri:high, Pri:medium, Pri:low
Scope:big, Scope:medium, Scope:small
Needs repro
Needs discussion
In development
Popular

What do you think?

DaveT

My quick thoughts:

Let’s try something, see how it works and not be afraid to change if needed. What ever the approach is, the tags need to be clearly defined so that occasional users can understand what’s going on.
Re: ‘Popular’ tag. Shouldn’t that just get handled by voting? And how does voting work here, by which I mean how would you like it to work?
‘Usability’ is a great thing. Removes the grit from / enhances the user experience. Promise yourself eg., 1 full day in 10 to focus only on those. Makes a big impression.
Scope is all down to you.
‘In Development’ (underway?) is also very good. More work for you, but good customer relations.
Would you want the help of others with any of this (noting that some tags can only be added by you)?
However, you skin the tags, you, or someone from your end, needs to check-in frequently enough to make sure stuff is updated.

Otherwise, let’s give it a try. 👏

Jay

DaveT

Yes, in theory popular should get handled by voting. But I don’t see a lot of votes so it might be good to at least have a way to manually flag posts as popular? Maybe as the forum gets more active we won’t really need it.
“In development” does seem verbose. Underway could be good. Any other ideas? 🙂
Yes, I would very much love help! I would be happy to make anyone a mod who wants to be. But I have tags set up currently where anyone can apply any tags. We could change it to where the thread creator can use any tags, but only mods can change tags on other threads. And we could potentially also customize permissions per tag, which I may want to do for Scope tags since it doesn’t make sense for anyone else to apply those. But I think for now let’s try keeping it open to everyone and if it gets abused we can lock it down.

I think you’re right that we should just try it. I’ll start setting up these new tags and applying them tomorrow (it’s 11pm here in Singapore). But I’m excited to hear more of what people think!

DaveT

Agreed. I’ve not voted ’cos I haven’t got my head round it yet. But it’ll all probably come out in the wash, especially as the forum gets more users.
Good plan - try, assess, change if needed, repeat.

Jay

Another meta question: I’ve just started a couple of discussions in General that I would really like a lot of discussion on so I can move forward with making those changes, but I fear they may end up drowned out by lots of other threads. Do you think there should be a top level category for “development questions” or something? I can’t think of what a name would be for that. Or maybe the “Needs discussion” tag should be colored (blue?) so they’re more prominent?

Edit:
I made “Needs discussion” a primary category (so it shows in the sidebar) and colored it blue. Let me know what you think of that?

I think I’ve made it so that only I can tag as “Needs discussion”. Could someone confirm that you aren’t able to apply that tag?

Jerud

Jay Re: “needs discussion” tag: I cannot apply it. But I CAN remove it.
At least, it will let me un-check it…I didn’t click “OK” to lock in the change so maybe it would have prevented me.

Jay

I made @DaveT @Jerud @Jorge mods. Feel free to help out as much as you’d like 🙂. And if anyone else wants to be a mod, let me know. Being a mod makes you able to:

Merge similar threads together
Split posts out into their own threads
Rename threads
Lock or delete discussions/posts if they’re bad or spam

Jerud

So much here! I’ll try to be as brief as I can:

@Jorge ’s concise spelling-out of “tier 1” and “tier 2” is spot on.
I agree the process should not be overly granular or it won’t function well
Priority Tags: I like the “all grey except urgent” idea. I would also like to see an actual written-out criteria from @Jay in terms of what constitutes each “level” of priority.
Re: my “cosmetic issues” concept … rather than a separate tag, I suppose this would just get rolled in with “low” priority – along with functionally low-priority issues. Since it’s a question of the “priority” rather than the nature of the bug, there’s no value in having a separate callout. I have dozens of examples but don’t want to crowd this space any more…but I think it’s relatively evident that a lot of cosmetic problems gives a poor impression of the app.
It’s natural to assume that high priority will be done before low…but clearly there needs to be some way that low/medium are worked into the pipeline or else those will just never get done. Presumably voting will come into play here. I think mods should occasionally call attention to threads which are languishing, to “run them up the flagpole and see who salutes”. Could be done via “needs discussion” tag? Threads that get zero traction would then remain in the archive, with a final post on them from the mod stating that they were “promoted” for voting. Future users who search and find that thread would then have clear feedback that the idea had been considered but didn’t get enough support.
I like the idea of the dev team “closing the loop” by marking threads they have picked up, and also agree there is little value in getting very granular with that. I suggest we have just three tags total:
“in queue” (or “queued”) for stuff that Jay has definitively picked up – no detail on priority necessary.
“ready” or “scheduled” for solved posts that simply haven’t been pushed out to stable yet
“solved” – this tag already exists
I agree with re-scoping the reproducability tag as “needs repro”. Hopefully it can lead to the community helping each other find reproduction steps. @Jay you say that there aren’t a lot of un-reproducible bugs posted – but that’s a kind of reverse survivor bias; I don’t generally post bugs I can’t reproduce (unless they really piss me off or lose data).
I like the “needs discussion” tag and top-level menu item. If you have major topics you want to call attention to for discussion, sticky them to the top of that page.
Re: “popular” posts – is there not a way to put a “trending” or “popular” category on the menu, that auto-populates with posts that have high activity? I would have expected that to be baked into the forum SW. The current low voting volume might just be due to the newness of the forum. There’s some inertia to overcome there. Manually applying a “popular” tag feels disingenuous.
The shortcoming of only having one tag for “mobile” bugs is that it leaves no way to distinguish between bugs that happen on BOTH desktop and mobile vs. those that are desktop-only or mobile-only. What about two tags (“mobile” and “desktop”) – apply both if it’s both? Or is that just not high enough value to the dev team to be worth the trouble?
I’m wary of the “scope” tag. As a layperson I am constantly surprised by what is “easy” and what is “hard” even after years of reading beta discussions. I think the value of a simpler tagging scheme is greater than the limited value we can bring to the devs by guessing wildly at how hard we think their jobs are 😇.
Re: Tag permissions, I like the proposed where everybody can apply all tags on their own posts (except for the developer-status ones), and mods can change tags on other posts. Simple and understandable, gets the job done.

Jerud

Pretty much all of the above is about bugs.
Any thoughts about how Feature Requests should be handled? Separate thread?

Jay

I’ll post a sticky thread soon about how posting and tagging should work with details about all the tags.
Bumping: Just comment on a post would bring it to the top. I don’t think a tag is necessarily required for that. But the original poster or a mod could just comment something like “bump” to bring it up the flagpole.
In queue: I’m also hesitant about this because people get annoyed if we say we’ll do something and then it doesn’t happen for a long time. I think ideally the priority levels should give a general idea about where they are in the queue? But let’s see how it goes with the priority system and we could come back to this idea.
Repro: I can put something about that in the sticky thread - that you can post a bug you haven’t been able to repro as long as you tag it as such, and maybe someone else will have encountered it and can provide a repro.
Popular: If you click on Latest at the top left of the thread list you can change it to Top, which sorts by number of comments. Or Trending is supposed to sort by number of votes but our vote counts are so low at this point it doesn’t quite seem to do that?
Scopes: Those should be tags I would apply myself. But I’m not 100% sure it’s the right way to go. I’ll play around with it.
Feature requests: I feel like a lot of this applies to feature requests too. But if you have more ideas about that, I think this is a good thread for it.
Needs discussion permission for mods: I think I fixed that.

DaveT

Jay I like they way tagging ideas have developed 👏.

Re: Scoping
I understand concerns about a scoping tag. But I do like the idea of scoping because it shows users / potential users that the devs have though about a feature and have a top level ‘guesstimate’ of how much effort something is.

This is useful because (a) many a-time in a past life I’ve asked for features that seem trivial to me, but were in fact much harder and (b) the opposite - some features that appear hard, might actually turn out to be relatively straightforward. And for those that you’ve not yet had time to think about you don’t add a scope tag (or add one like ‘YetToScope’). We found that users were much more receptive to things not getting done if our devs had given some though and had a reason(s) why it was hard.

Also, Scope, has some bearing on Priority. A feature/bug scoped as ‘quick’, and asked for by a fair number of people might get done. But if scoped as ‘hard’ then it might not get done.

‘grouping’
One other thing that we used to try and do in my ‘managing a s/w company days’, was to ‘group’ certain features to address in a dev round. So, for eg., it might be that when done together, apparently un-connected features x, y, and z, allows you to more effectively address a wider market area - in this context maybe it means that WorkFlowy customers no longer have any reason not to use Legend (obvs, I just made that bit up to illustrate the point). But you can’t do this without having a broader view on Legend and where it stands relative to competitors + the direction you want to push it in.