New sync system

Jay

At long last it seems ready to go 🙂. But it’s after midnight here so I’ll hold off and release it in the morning. Basically the problem was that boards were saving/syncing way too much data. I had been intending to optimize board sync later but it turned out to actually be the source of a lot of problems, so I just went for it.

Collapsing all would mark every item as collapsed. In the case of huge documents of 50k+ items that could be over 15k collapses. Saving that would slow things down until it finished, and sometimes it would never finish. It now toggles a “base” collapsed state, so collapse all changes just one thing.
Any change to a pane was saving the whole pane, including the collapsed state for every view the pane had used (which could be very big). Saves are now extremely targeted and save only what changed.
A big save could get stuck and make the app slow for a while. In the worst case it could delay future saves, or if it failed because of being to big, it could even prevent any syncing. It now catches and reports when saves are too big, but the combination of the above two things and other fixes should make saves never be too big.

Long story short, I think this will be a big improvement.

Jerud

Maybe I should wait for you to push this recent update…I just tried going back to beta yesterday. I had to switch back to stable and then to beta again, plus relaunch the app several times before my boards came up. No boards lost, no zombie boards appeared. But some things are ‘off’. Skimming the above, looks like others are seeing some of this. Here’s the brief version of my first hour on the new beta:

Documents have reordered themselves in the Overview drawer and cannot be manually ordered
Sometimes panes will just be blank, especially popup panes, evidently triggered by reordering an item. Possibly aggravated by using keyboard shortcut to reorder the item.
Mirrors are unreliable; I created one (alt+drag) and the resulting item had no tally box but still reflected changes back to the original and prompted “this one” or “all” upon delete. I copied that item text, deleted both mirrors, and re-created the item and a new mirror successfully.
Keyboard use is degraded – why does this have anything to do with the backend?:
- Panes randomly lose focus in the middle of typing. This is null focus; no way to get it back without clicking somewhere.
- Arrow-navigation gets ‘stuck’ on regular items as if they were links, but no link or mirror is anywhere nearby.
- Arrowing jumps unpredictably – sometimes if I’m at the end of an item and press “down” the cursor jumps to the beginning of the next item, and sometimes it just jumps straight down. If I go back up and down again, I may get a different behavior.
- The cursor sometimes gets out ahead of text while typing.
I added a new top-level branch to one of my docs in beta and switched back to stable; it does not appear. I added new items in stable and those DO appear in beta. So for some reason I grabbed the branch that didn’t sync and ctrl+dragged to copy it to a different location in that same document. The result looked like this:

At one point I managed to turn an existing mirrored item into gibberish similar to the above too, though I don’t recall what triggered it.

I’m going to fully log out, reboot the machine, and try again tonight to see how things are after this newest update. Fingers crossed.
I guess I’m sticking with beta for now, since it looks like I may lose what I did last night if I revert to stable. While that’s not a hell of a lot, lost time is lost time.

andrew

I also would reconsider releasing this. As I have mentioned elsewhere, I lost my main document… it completely disappeared. Fortunately I had one instance where it wasn’t gone and copied the contents to a new document. It did finally reappear in my Windows desktop instance but it was a version from over a year ago.

Also I had a new document “My First Document” appear in my document list today.

Jay

I just released an update to the beta which should fix a lot of the sync problems (as I mentioned above in https://forum.legendapp.com/d/906-new-sync-system/34), and it may load faster for some of you.

It’s not possible for any update to delete anything - there’s not even code anywhere for deleting items or documents. Deleting items only marks them as deleted, but all the data remains, and they would be in the Deleted view. Similarly deleting documents only marks them as deleted and then they’d be in the Trash. So I think documents disappearing would most likely have been a bug in the stable sync system that didn’t save it properly, or it’s a bug in the beta where it’s not loading as expected and just ignoring it. But if you’ve seen in on more than one instance before it is still in the database.

If the beta is not showing any particular document that is appearing on stable please tell me the name of it so I can investigate your account in the backend and see what about the document would make it fail to load in the beta. I have been stress testing this pretty hard and haven’t been able to find any ways to break the sync, but it’s possible that some old bug from long ago put a document into a state that the beta is having trouble loading.

The bug with new My First Documents being added is still on my todo list. I saw it happening a while ago when signing into an account with a single encrypted document, but I thought I’d fixed it. I’ll keep looking into it.

Also if you’re concerned about your data I’d recommend doing a manual backup just to be safe. Using the beta in the app now saves to .json files so you can keep the folder backed up to Dropbox/Drive/git or whatever backup method you use. But for now it’s probably better to run it in an incognito window for testing.

andrew

The weirdest thing, as I mentioned elsewhere, is that when I went back from Beta to Stable, my missing document reappeared but it was in the state it would have been in approximately a year ago, as far as I can tell. And it was not in Trash. (I had made a manual copy of the whole tree on the original document into a new document as well as imported an exported copy of the original document while it still existed in one instance, so I don’t think I lost any data.)

Jay

andrew Had that document been syncing correctly in the stable version this whole time? Can you try opening the beta in an incognito window and see if it still looks like it’s a year old?

Jay

I just released an update which fixes some caching issues in the desktop app where signing out and back in did not fully clear the local cache, which might have made it not load some documents/items when signing back in. If you sign out/in again it should clear that up. andrew

andrew

Jay When I try to open beta.legendapp.com, it gets stuck on Loading… and gives me the following toast. Never gets past it.

I logged out of Windows desktop and the same thing is happening… can’t load any documents. I know it is using Stable (confirmed by looking at inspector and it is loading app.legendapp.com).

Yet in a Chrome tab, I am able to load app.legendapp.com and all documents loaded (except one local one that was created when I imported my original document locally).

Eik

Using the latest update in beta, all my boards are not valid anymore. I also have a lot of these. Recreating the boards does not help, Already tried clearing local cache, no solution.

Eik

Seems that it has something to do with grouping and agenda mode:

Also google calender selection is not persisted:

Jay

Thanks a ton for your patience and detailed reports everyone! A big change like this is always going to be bumpy, but I think we’ll come out the other side in a much better place than before 🙂.

Eik andrew I think I’ve fixed the causes of those errors. If I didn’t, I set up better error reporting so it will be easier to track down the cause in the future.

andrew Jerud Is that duplicate document error happening in the stable version? That would be because the beta version caches everything to local documents instead of the internal cache (so that you can back the files up yourself, and it’s much faster). Then the stable version doesn’t understand that so it sees the files and thinks they’re local documents, and ignores them because it already has a synced version of them. If that’s happening in beta I’d need to look into it! But if it’s in stable there’s nothing to worry about 🙂.

I’ll keep looking into the other issues you’re seeing. This should (hopefully) fix all the load errors and leave only some display issues remaining. If you see that an error you already reported is still happening, please let me know so I know it wasn’t fixed. Thank you!

andrew

I am getting these errors when it hangs on Loading in Windows desktop app:

Jerud

andrew I got these same two issues (documents never load, and duplicate documents) when switching to beta as well. I got past it by reverting back to stable, then beta, then stable, then beta again. Not kidding; I really am just a stubborn caveman. It finally loaded : 0

I dismissed the duplicate documents errors as an artifact of switching to new backend – would have saved a little heartburn to know they would appear ahead of time but it at least was explainable and I saw no missing (or zombie) documents as others have.

Jorge

So testing the beta today… I noticed than in my board of bills to pay, in which I have a pane that filters for overdue items -is:complete #pay date:overdue and shows them as a list type, there was an item missing in the beta that what I was seeing in the stable version.
I found that in the beta version the view was showing an outline type view, while the stable version was a list type view (and therefore the item I was looking for was being shown inside another item and therefore the discrepancy). When I switched the board in the stable version to be of outline type, 1 second later the board in the beta version just went blank and to the default prompt to create a new board!

webalstrom

Not a huge deal, compared to missing documents and the app not working, but when I logged into the beta (using incognito), I noticed that the Document sidebar is not showing the hierarchy below one level deep. I can see the name of of the document, click the carrots to show the first level. The carrots show next to each element on this level, clicking on them rotates them, but nothing shows. Everything displays fine in the pane.

Other than that, everything seems to be working fine for me. All my boards are there and working (although they were reordered, but rearranging them back, closing the beta and reopening seemed to keep them in the correct order). The app opened up pretty quick (about 4 seconds of the loading documents screen). Changes I make in the beta reflect in the stable and vice versa.

Just FYI: I have 5 documents. two are around 12,500 items each, one is around 1000 items, two are under 200 items each (this does not include the ones in the trash).

andrew

Jay I cleared out all Legend data on my PC and reinstalled, then got stuck on Loading, but realized it was in Beta mode, switched to Stable, then it loaded and I had duplicates of every document, plus another My First Document. Then my PC crashed while using Legend and after restarting, I started Legend again and I now have one version of each document (after trashing the My First Documents). My original main document, which had come back as the version from a year ago, is now not present again but the copy I created is still OK.

Should I stay on Stable (where it seems to work) or go back to Beta (which seems to bring on these problems)?

andrew

Actually in Windows desktop Stable, I can’t access some boards. When I click on the board, it does nothing. Other boards I can access. Can’t identify a common thread.

Also in one pane, an advanced filter got turned into a regular filter that was missing some of the advanced filter terms. Not a big deal for me, but mentioning in case it affects others.

Jay

andrew I’d suggest for now using the beta in a Private/Incognito browser window. The desktop version does some extra things (saving to files instead of in the embedded browser’s storage) that may be less stable. And if there’s any problems it will be with the local cache so you can just close the private window to clear it. I’ll keep working on testing everything reported here in the desktop app and make sure all is good. It’s been working great for me but everybody has different document data so I have to try to find the bugs that are being hit.

andrew

Jay OK I am running beta in Incognito. Observations:

Beta now shows my original main document that had been missing and it appears to be current (reflects changes up through the time I copied all items to a new document). This was not present in Stable, but then I closed and restarted Windows desktop in Stable and a document by that same name appeared as a local document with no items in it.
In Beta, a few of the boards that I can’t access in Stable (the ones that I click but nothing happens) are not present in Beta. I thought it might be because all panes refereneced the original document now present in Beta but showing as empty local in Stable, but I changed the document in all panes to refer to the new document now present in both Stable and Beta but I still can’t access in Stable. One of the other boards that uses the original doc in all panes is accessible in Stable but the panes refer to a different document.
In Beta, I also cannot expand the second level of Overview tree, only the first, as described by @webalstrom
The following Advanced filter appears correctly in Beta but in Stable shows in non-advanced mode and without the part in parentheses:
(is:star or before:tomorrow) is:task -is:complete

andrew

andrew Jay I need to correct my post above:

When saying I changed the document for the panes in Beta on the boards that were inaccessible in Stable, I was referring to the subset of boards that were inaccessible in Stable but were present in Beta. Of course, I couldn’t change the document for the panes in the boards that were not present in Beta.

The advanced filter in Beta that was converted to a non-advanced filter in Stable was not actually just the same filter without the parentheses, rather it was set to exclude complete and include high priority, or effectively:
-is:complete pri:high

Jerud

Sounds like I’m in the same boat as @andrew. I thought I had moved successfully to beta but that no longer appears true. I opened Legend last night and found total swiss cheese: randomly missing boards, some boards visible but un-clickable, randomly missing panes on the boards I could see, panes pointed or defined wrongly, randomly missing docs, and randomly missing items within docs.

I fully uninstalled Legend in all locations, then reinstalled. Things look fine in stable. I tested switching to beta once and got the same Loading… delay as before, at which point I just switched back to stable, where I shall stay for now.

A few details in case they help:

@Jay you are correct the duplicates warning only happens upon entering stable and I’ve seen no negative impact.
This is all on windows desktop, which is my non-negotiable primary use
I use the same Legend account from within two different Windows “users” (work vs. personal) on the same machine. The app is installed “for all users”. My initial switch to beta was in “work” user. The problems last night happened when I signed in as “personal” user.
The “Loading…” screen that hangs up when switching to beta has an interesting glitch: the three bouncing dots are corrupted. Sometimes they don’t appear at all but usually they appear with a “broken” animation. I find this noteworthy because I assumed this to be just an animated GIF or similar, in which case glitches might cause it to pause or move slowly, but follow the same pattern. Instead, the animation is actually “wrong”. Normally they bounce up and down with a slight stagger, so they make a “wave”. But I see the 1st dot moving on its own pattern while the 2nd and 3rd dots are moving together - slightly misaligned, but perfectly in-phase. Are they three separate single-dot animations that are just started out of phase? Is there any significance to the 2nd and 3rd not getting properly phased?

« Previous Page Next Page »