New sync system

Jorge

Jay @Jorge Can you please try this latest version in a private window and let me know if you see it go offline or get stuck on Loading Documents… again? If it does, could you send me a screenshot of the Console in the browser developer tools?

Yes it did once more. It loaded, was responsive for a little while, then froze up for a few seconds and then went offline. I think you ahve already reproduced the problem and are fixing it, but here is the screenshot anyways:

I have IM you the full log (as it is quite long)

Jorge

Still getting ghot boards in the beta version running along the stable.
ONe empty board created and another board mapped to an existing board in the stable version.

This time i am not deleting any ghost boards from the beta version.

Jorge

@Jay As you know I’ve been running the beta version at the same time as the stable version. Thestable version still has the problem that goes offline every now and then, and supposedly the sync system should have fixed that for the beta. But today I was entering some text entries in the stable version in the desktop app and it suddenly went offline. So concerned about what did make it, I switched to the beta version only to find that the text hadn’t made it and that it too was offline (it had been online an hour or so earlier today).

So then I went to the stable version in the browser (opened new) and that one was online and all my text entries from the desktop had thankfully made it to the browser based stable version.

I have refreshed the beta version in the browser and it starts online, then it starts shifting very quickly between online<>offline until it remains permanently offline.
I’ll try closing the private window and reopening it again.

Update: Closing and reopening again the beta in private window allows it to become online again and sure enough all my new text entries were there.

Jay

Jorge I have those problems fixed and am finishing testing them. The new system had some cases that were VERY slow with syncing board collapsed state so it could get stuck, but they were easily fixable. There was a lot of room for improvement so it’s taken more time than expected, but it should improve things for you a great deal. I’ve almost got it done now, and am hoping to release an update tonight (in Japan time).

Jay

I was a little ambitious and time got away from me today, but I can’t find any more issues so I think it’s good to go. I just need to test that it still works with boards from stable and hopefully can release tomorrow.

Jorge

Jay Thanks Jay. I am sure this will benefit everyone as well.

Jay

At long last it seems ready to go 🙂. But it’s after midnight here so I’ll hold off and release it in the morning. Basically the problem was that boards were saving/syncing way too much data. I had been intending to optimize board sync later but it turned out to actually be the source of a lot of problems, so I just went for it.

Collapsing all would mark every item as collapsed. In the case of huge documents of 50k+ items that could be over 15k collapses. Saving that would slow things down until it finished, and sometimes it would never finish. It now toggles a “base” collapsed state, so collapse all changes just one thing.
Any change to a pane was saving the whole pane, including the collapsed state for every view the pane had used (which could be very big). Saves are now extremely targeted and save only what changed.
A big save could get stuck and make the app slow for a while. In the worst case it could delay future saves, or if it failed because of being to big, it could even prevent any syncing. It now catches and reports when saves are too big, but the combination of the above two things and other fixes should make saves never be too big.

Long story short, I think this will be a big improvement.

Jerud

Maybe I should wait for you to push this recent update…I just tried going back to beta yesterday. I had to switch back to stable and then to beta again, plus relaunch the app several times before my boards came up. No boards lost, no zombie boards appeared. But some things are ‘off’. Skimming the above, looks like others are seeing some of this. Here’s the brief version of my first hour on the new beta:

Documents have reordered themselves in the Overview drawer and cannot be manually ordered
Sometimes panes will just be blank, especially popup panes, evidently triggered by reordering an item. Possibly aggravated by using keyboard shortcut to reorder the item.
Mirrors are unreliable; I created one (alt+drag) and the resulting item had no tally box but still reflected changes back to the original and prompted “this one” or “all” upon delete. I copied that item text, deleted both mirrors, and re-created the item and a new mirror successfully.
Keyboard use is degraded – why does this have anything to do with the backend?:
- Panes randomly lose focus in the middle of typing. This is null focus; no way to get it back without clicking somewhere.
- Arrow-navigation gets ‘stuck’ on regular items as if they were links, but no link or mirror is anywhere nearby.
- Arrowing jumps unpredictably – sometimes if I’m at the end of an item and press “down” the cursor jumps to the beginning of the next item, and sometimes it just jumps straight down. If I go back up and down again, I may get a different behavior.
- The cursor sometimes gets out ahead of text while typing.
I added a new top-level branch to one of my docs in beta and switched back to stable; it does not appear. I added new items in stable and those DO appear in beta. So for some reason I grabbed the branch that didn’t sync and ctrl+dragged to copy it to a different location in that same document. The result looked like this:

At one point I managed to turn an existing mirrored item into gibberish similar to the above too, though I don’t recall what triggered it.

I’m going to fully log out, reboot the machine, and try again tonight to see how things are after this newest update. Fingers crossed.
I guess I’m sticking with beta for now, since it looks like I may lose what I did last night if I revert to stable. While that’s not a hell of a lot, lost time is lost time.

andrew

I also would reconsider releasing this. As I have mentioned elsewhere, I lost my main document… it completely disappeared. Fortunately I had one instance where it wasn’t gone and copied the contents to a new document. It did finally reappear in my Windows desktop instance but it was a version from over a year ago.

Also I had a new document “My First Document” appear in my document list today.

Jay

I just released an update to the beta which should fix a lot of the sync problems (as I mentioned above in https://forum.legendapp.com/d/906-new-sync-system/34), and it may load faster for some of you.

It’s not possible for any update to delete anything - there’s not even code anywhere for deleting items or documents. Deleting items only marks them as deleted, but all the data remains, and they would be in the Deleted view. Similarly deleting documents only marks them as deleted and then they’d be in the Trash. So I think documents disappearing would most likely have been a bug in the stable sync system that didn’t save it properly, or it’s a bug in the beta where it’s not loading as expected and just ignoring it. But if you’ve seen in on more than one instance before it is still in the database.

If the beta is not showing any particular document that is appearing on stable please tell me the name of it so I can investigate your account in the backend and see what about the document would make it fail to load in the beta. I have been stress testing this pretty hard and haven’t been able to find any ways to break the sync, but it’s possible that some old bug from long ago put a document into a state that the beta is having trouble loading.

The bug with new My First Documents being added is still on my todo list. I saw it happening a while ago when signing into an account with a single encrypted document, but I thought I’d fixed it. I’ll keep looking into it.

Also if you’re concerned about your data I’d recommend doing a manual backup just to be safe. Using the beta in the app now saves to .json files so you can keep the folder backed up to Dropbox/Drive/git or whatever backup method you use. But for now it’s probably better to run it in an incognito window for testing.

andrew

The weirdest thing, as I mentioned elsewhere, is that when I went back from Beta to Stable, my missing document reappeared but it was in the state it would have been in approximately a year ago, as far as I can tell. And it was not in Trash. (I had made a manual copy of the whole tree on the original document into a new document as well as imported an exported copy of the original document while it still existed in one instance, so I don’t think I lost any data.)

Jay

andrew Had that document been syncing correctly in the stable version this whole time? Can you try opening the beta in an incognito window and see if it still looks like it’s a year old?

Jay

I just released an update which fixes some caching issues in the desktop app where signing out and back in did not fully clear the local cache, which might have made it not load some documents/items when signing back in. If you sign out/in again it should clear that up. andrew

andrew

Jay When I try to open beta.legendapp.com, it gets stuck on Loading… and gives me the following toast. Never gets past it.

I logged out of Windows desktop and the same thing is happening… can’t load any documents. I know it is using Stable (confirmed by looking at inspector and it is loading app.legendapp.com).

Yet in a Chrome tab, I am able to load app.legendapp.com and all documents loaded (except one local one that was created when I imported my original document locally).

Eik

Using the latest update in beta, all my boards are not valid anymore. I also have a lot of these. Recreating the boards does not help, Already tried clearing local cache, no solution.

Eik

Seems that it has something to do with grouping and agenda mode:

Also google calender selection is not persisted:

Jay

Thanks a ton for your patience and detailed reports everyone! A big change like this is always going to be bumpy, but I think we’ll come out the other side in a much better place than before 🙂.

Eik andrew I think I’ve fixed the causes of those errors. If I didn’t, I set up better error reporting so it will be easier to track down the cause in the future.

andrew Jerud Is that duplicate document error happening in the stable version? That would be because the beta version caches everything to local documents instead of the internal cache (so that you can back the files up yourself, and it’s much faster). Then the stable version doesn’t understand that so it sees the files and thinks they’re local documents, and ignores them because it already has a synced version of them. If that’s happening in beta I’d need to look into it! But if it’s in stable there’s nothing to worry about 🙂.

I’ll keep looking into the other issues you’re seeing. This should (hopefully) fix all the load errors and leave only some display issues remaining. If you see that an error you already reported is still happening, please let me know so I know it wasn’t fixed. Thank you!

andrew

I am getting these errors when it hangs on Loading in Windows desktop app:

Jerud

andrew I got these same two issues (documents never load, and duplicate documents) when switching to beta as well. I got past it by reverting back to stable, then beta, then stable, then beta again. Not kidding; I really am just a stubborn caveman. It finally loaded : 0

I dismissed the duplicate documents errors as an artifact of switching to new backend – would have saved a little heartburn to know they would appear ahead of time but it at least was explainable and I saw no missing (or zombie) documents as others have.

Jorge

So testing the beta today… I noticed than in my board of bills to pay, in which I have a pane that filters for overdue items -is:complete #pay date:overdue and shows them as a list type, there was an item missing in the beta that what I was seeing in the stable version.
I found that in the beta version the view was showing an outline type view, while the stable version was a list type view (and therefore the item I was looking for was being shown inside another item and therefore the discrepancy). When I switched the board in the stable version to be of outline type, 1 second later the board in the beta version just went blank and to the default prompt to create a new board!

webalstrom

Not a huge deal, compared to missing documents and the app not working, but when I logged into the beta (using incognito), I noticed that the Document sidebar is not showing the hierarchy below one level deep. I can see the name of of the document, click the carrots to show the first level. The carrots show next to each element on this level, clicking on them rotates them, but nothing shows. Everything displays fine in the pane.

Other than that, everything seems to be working fine for me. All my boards are there and working (although they were reordered, but rearranging them back, closing the beta and reopening seemed to keep them in the correct order). The app opened up pretty quick (about 4 seconds of the loading documents screen). Changes I make in the beta reflect in the stable and vice versa.

Just FYI: I have 5 documents. two are around 12,500 items each, one is around 1000 items, two are under 200 items each (this does not include the ones in the trash).

« Previous Page Next Page »