Note: This was originally a roadmap post but discussion took it in a different direction, so see the new roadmap post: https://forum.legendapp.com/d/576-roadmap

Database architecture

We are updating how data is stored in the backend database and removing technical constraints on documents. This project has some big goals:
1. Simplify the UX (user experience) to have less complexity with documents
2. Allow panes to show all data, and not be restricted to individual documents
3. Sharing of individual nodes. Sharing is currently done at the document level, so this will be much more flexible.
4. Improved version history. It currently backs up entire documents periodically and the Version History diffs the full document vs its full backup, which can be slow for large documents. In the new system it will save version history for individual items as they’re changed, so Version History will be faster and more accurate while saving less data.
5. Fewer backups. With the new version history we can do full backups less often, and they will only be one file instead of one file for every document.
6. Improved local document support
7. Improved conflict resolution of offline changes to boards and settings
8. Big step towards team features. This new database is designed for future team features like shared team workspaces, task assigning, billing by seat, team member management, etc…
9. Faster load time with better handling of deleted items

    13 days later

    One concern I have with a move to eliminate “documents”:
    My most frequent use case to create a new document is when I want a set of information with it’s own set of tags. A single document with all the tags included get’s unwieldy and starts to make tags much less useful. Auto complete get’s harder with a huge tag list; group by tag becomes almost meaningless with a huge tag list.

    So: If documents are eliminated; could we put some thought into how to prevent TAGS from getting out of hand?

    Brainstorming some ideas:

    • A setting to say, “when I’m zoomed in, only present tags in this node as tag options”? (With an option to expand the tag search if needed?
    • Some kind of tag management? or tag outline?
    • Jay replied to this.

      Yes, I have the same concern… I like documents & wouldn’t want to need to do without them.
      They add an extra level of hierarchy, which is an extra level of clarity. 🙂

      • Jay replied to this.

        Jay #2 and #3 in particular would be very useful benefits.

        LauraH they86 That’s an interesting point I’ve been thinking about and am unsure of the right way to solve. There had also been some discussion before of a possible feature of selecting different default a google calendar per document, and other document-specific options. I wonder if we should have a set of node-specific options, where all items under that parent take those options. That could be “isolated tags”, “default calendar”, “default item type”, and others?

          Jay I think you definitely need (an option for) node-specific items.

          @LauraH ’s suggestion to automatically adapt tags to the zoom-level you’re currently in sounds pretty clever… Though not completely waterproof.
          (Let’s say you zoom into a specific bullet of your “master node” and you haven’t used all tags yet inside this bullet… The tag you want won’t show as a suggestion. - It might end up in a lot of switching on and off this specific zooming setting.)

          I don’t have a better idea, though. 😉

          Jay I want to bring back my age-old point to solve @LauraH ’s issue with one huge set of #tags. We should make the tags hierarchical. That way anyone can group them in whatever way they want need and when searching it’s easy to search and visualize the subgroup they are interested in, without having to deal with a humongous list of tags to scroll through. I am already having that problem even though I have separate documents, with separate sets of tags, but the number of tags in one document is just too big and is already unwieldly and I am not about to split that document further because of the other pains that having split documents bring up.
          As I have said before my workaround for this is to simulate the hierarchy by labelling the tags using a prefix, but it’s not pretty and not complety solves the problem.

            This node specific idea is interesting, but not a complete solution to the problem. It requires further discussion to build a proper use case for it. The calendar and and tasks type are fine.. but when it comes to the tags I am hesitant to support it for the very same reason @they86 brought up. It won’t be easy to manage and it can yield more complications than it is sought to solve.

            Jay , I’m one who pushes for document-level settings/properties. I find the metaphor of “documents” extremely useful. Compartmentalization is very important to me. It is, probably, the one thing that I most desperately need Legend for. I’m not excited about the possibility of losing that. Although I would indeed like to draw from across multiple documents, even then I don’t want to draw from ALL of them. And I certainly don’t want my stuff lumped together all the time. I’d rather give up the possibility of sourcing panes from multiple documents than have to deal with a single mega-outline.

            This is relevant for organization, for faster navigation, and to aid focus. Legend is specifically useful for me because it lets me handle a bunch of disparate categories and types of information without them cluttering each other. I can’t be distracted by something if I can’t see it. If one huge Outline was the answer to my problems, I would never have come looking for Moo.do. Hell, that’s just a big to-do list.

            @LauraH mentions separate sets of tags for each document and I do leverage that aspect of the current Document structure. It is, in fact, what prompted me to start agitating for other things to be per-Document. I don’t want work-tags popping up when I’m trying to auto-complete an item related to personal topics. Same with Move-to destinations when using Alt+M. Documents let me implicitly control the context of my work, so everything can be fast and efficient. Although I will say that a “Tag manager” – requested by many others and comprising a whole range of requested features – could solve that for a “node based” solution by explicitly assigning groups of tags to be “active” under a particular node. But really, how much sense does that make? One big mega-document where users are literally applying different rules to different nodes of it? That defies pretty much every intuitive analogy of digital work. And then you’d just need managers for everything else, to portion it out to nodes. That’s a whole new tier of UI. I know I’m not being very even-handed here but the thought of a mega-document repels me in a visceral way. I promise though, I’ll still listen with an open mind to the arguments.

            Sharing/collaboration was one reason I have pushed for per-document settings in the past. It seemed like being able to specify viewing and editing permissions to a whole document would be mandatory to allow collaboration and template-sharing. Assigning this on a document level seemed intuitive and practical. After all, that’s how pretty much everything else out there is shared. By document. There’s several other settings (some you already mentioned) that also make excellent sense to assign per-document, and they’d pair well with the needs of collaboration. In contrast, the idea of sharing on a per-node basis seems, frankly, ridiculous. That’s just too fine of a resolution…and how would that even work into the interface? How would you show a top-level overview of sharing settings across all nodes? What about making changes for several nodes at once? I don’t want a bunch of sharing B.S. in my way while I’m working in a document. That should be something I can assign at a high level and then just take for granted as I work.

            Compartmentalization can also have professional implications. For example, I keep all client-specific tasks/information in a single document for that client. I can take that all offline if I stop working with them, provide it directly to the client upon request, or delete it entirely if necessary – without the risk of including something irrelevant. While tools could of course be provided to handle this by-node…why? The current metaphor works. The only exception I feel the need to mention is the Archive command, which has always struck me as something that’d make more sense at the Document level than Item level.

            As with the original/instance structure of mirrors, if all my Documents simply become top-level nodes of my account’s single database “under the hood” – but still act like discrete documents when I open the Overview drawer – then I guess I don’t really care. But, also like mirrors, they need to ACT like discrete documents on the surface without belying that true structure. And also-also like mirrors: Let’s beta test the hell out of that change before making it live!

            • Jay replied to this.

              Tags

              Hmm, per-node options are definitely fraught with lots of weird edge cases and potentially a lot of new UI… I wonder if maybe a super easy fix would be to group the results in the tag autocomplete:

              1. Tags within the currently zoomed item
              2. Other tags within the top-level item (what is now a document)
              3. The rest of the tags

              Then you kind of get the best of all worlds, a focused set of tags when you want it but all tags are still available? And we could pursue a similar concept for the move menu and other places where it makes sense?

              Documents

              Jerud I definitely hear you on the usefulness of documents for compartmentalization. In the new system instead of Documents you’ll just have a set of root items, as you said at the end of your post. It actually looks almost exactly the same in the overview, and you can still zoom panes to root items the same as you would to a Document. And while there will be a possibility of sharing any node within the tree, you (and I’m guessing most people) can just share an item at the root level which is effectively the same as sharing a Document.

              Status

              A quick update - the database change is now far enough along that I am actively running my Legend app on it. I’m hoping it will be beta testable within a couple weeks (I’m splitting it with general bug fixing).

              Done:

              • All the basics
              • Conflict resolution
              • Sharing
              • Saving version history

              Not done:

              • Version history viewer
              • Per-node encryption settings
              • Migration from Documents

                Jay Jay I am all for the benefits of a single hierarchy of nodes. I first started with a single outline and then split it into multiple documents for the following reasons:

                1. Performance. the loading was killing me. Splitting helped for a while until you improved the loading speed significantly, because I didn’t need to open all documents always, so that saved time. So please just make sure that for those of us that have huge documents, performance will be acceptable with a single outline.
                2. Sharing. I needed to share some content but didn’t want to share all my outline, the current solution you are proposing should even give us much more flexibility for sharing, and that will be great for workgroups.
                3. Security. I wanted to set a separate password for encrypting each document separately. Many people have different encryption requirements for work document than for personal documents. Although I have to admit, it is a pain in the wazoo to be entering 4 different passwords every time I sign into Legend. What are your thoughts in this regards in the new architecture?
                • Jay replied to this.

                  Jorge
                  Performance will be unchanged in most cases. It does currently initialize all items in all documents even if they’re not in an active pane. The new architecture will still initialize all items. We might be able to address that later though. It does handle deleted items much better though, which should improve load time for you quite a bit:

                  1. In the online database it previously only marked it as “deleted” but now also clears all data (which is safe because it saves it to version history first).
                  2. Deleted items are fully removed from the local database and not saved at all when syncing

                  You’ll still be able to have per-node encryption passwords

                    5 days later

                    Jay OK, if it’s like you say then it truly is just a change “under the hood” and I have no complaint.

                    I like your suggestion for grouping all auto-fill contents (tags, Move-to, etc.), with one exception: I like the way #tags are currently compartmentalized by document because it acts like a sort of “data validation” feature. I’m less likely to typo a tag or accidentally use one that doesn’t “belong” in that Document. It’s imperfect (because I can still manually type any tag, even a new one) – but it’s 98% effective. Your grouping proposal would be less effective. Perhaps you could modify it so the heading for the out-of-“document” tags group is shown but not its contents – user must click the heading to expand it. Or, handle this more robustly with a Tag Manager feature, which is sounding more and more needed as Legend evolves. For example, the “other documents” tag group could be disabled entirely for a shared document.

                    It sounds like in the new version “Document” will just be shorthand for “Root level Parent”, and presumably there will not be root-level ITEMS…or will there? Seems troublesome. Either way, I suggest keeping the user-facing “separate documents” paradigm as it is now.

                    Thinking through this some more, the “all one document” structure could open up some really neat use cases. Such as:

                    • Attaching a whole “document” to an item using the existing [[Link method.
                    • Organizing “documents” into groups by putting them into nodes that are one level above them. I assume this approach will be used to hold “Trash” Documents in the new system, why not let users see some of these groups?
                    • Grouping docs could also be used to let users “Archive” stagnant docs. I think this would be a much better use of the “Archive” command (which is already set up to apply to individual nodes, but is kinda confusing and unreliable at that resolution).
                    • Since sharing is by-node, grouping docs into parent-nodes would permit sharing whole sets of documents at once by sharing just that node.

                    That’s pretty exciting!

                    A couple details I just thought of:

                    • By-node sharing will mean we need a toggle between “inherit / override parent item’s sharing settings” to allow users to share out a single node from an unshared parent/document.
                    • If docs are just nodes, the tasks of splitting / merging docs could be much more straightforward – but there’s no interface for it currently (because we can’t “see” the underlying single-document structure). Possible pro-level feature.
                    • You mention the ability to “point” Panes at multiple “documents” because all docs are just nodes in the outline – but the current mechanism for pointing Panes involves picking a single node. If docs can be grouped under higher-level nodes, then user can just pick a container node. But if we can’t group docs like that – of if we want to source multiple docs that are not in the same group, will there be a method to multi-pick (ctrl+click or similar) docs to point a Pane to?

                    @Jay , regarding the “Next” list: mobile bugs. I agree these need to get licked soon because they hurt Legend’s usability and are probably an obstacle to new users’ adoption.

                    I recently got an iPad and now I totally “get it” about tablets. They’re not just “big phones” – they are “laptops-lite”. Suddenly I’m really wanting a better tablet experience for Legend : ) Others have pointed out it’s already pretty usable on iPad already (even though it’s not “supposed to be”) – the most obvious issue with Legend on iPad (aside from the bugs that affect all mobile installs) is that control elements don’t resize to the larger display. The buttons are t-i-n-y and there’s loads of wasted space around them. If that one thing could be tweaked for PadOS as part of your mobile fixes, it would be very impactful.

                    @Jerud Like you I’m increasingly using an ipad with a magic keyboard for more computer like tasks. I prefer running the full version of Legend in the Safari browser, rather than the mobile app. The only issue is that I can’t download email into Legend in the browser app but apart from that it’s fully functional.

                      wecanseeformiles Why didn’t I think of that!?!? Thanks for helping me see the obvious solution.

                      I would still like to see the mobile app fleshed out…eventually…because Legend is such a major app in my workflow that being able to switch directly to it – and use it in split-screen configurations – is a substantial need. Digging through my tabs to find the one running Legend is not ideal, and even if I do something like install Chrome solely to run Legend inside of, I’m still having to think of the browser instead of Legend when switching in.

                      I’ve heard that all browsers in iOS are “actually just Safari in a wrapper”. Could the PadOS version of Legend literally just be a front-end to Safari window that’s permanently pointed to legendapp.com without any nav buttons? I guess that would still miss things like swiping and multi-select…

                      • Jay replied to this.

                        Jerud The mobile app currently is actually just basically a wrapper around https://legendapp.com/mobile/. We will definitely make the mobile version much more usable on tablets. I use my ipad all the time so I would love it for myself. Just using the desktop version would be a pretty poor experience (although possibly better than now), so we will still want to have a UI designed for touch. It just needs some love and attention.

                          Jay , you mentioned in another thread doing away with the “Document” naming scheme. What’s the new plan? You’ve explained how the changes to the database will improve Legend without compromising the current experience, but I don’t get why Docs should be called something else.

                          The naming doesn’t necessarily need to change. There will not technically be any difference between a top-level node and any other node, but we can still call top-level nodes Documents. As we get closer to the new system fully working we’ll put some more thought into naming and I’ll start a thread about it.

                          Powered by: FreeFlarum.
                          (remove this footer)