New sync system

SamMarrocco

I also updated all instances of Legend to the newest version today (8/17/23). I worked for a few hours with no mystery documents or boards.

When I saw Jerud’s mail suggesting that login/logout/relaunch cycles might spawn old documents, I tried it (thanks, Jerud).

I logged out of my Windows Legend Native App, but left my MacOS version logged in (cut closed).
I then logged out of the Windows native app, then logged back into it. Sure enough, a “My First Document” appeared, containing some old data that I was testing with in past days.

I then launched my MacOS Legend (which was never logged out) and it then synchronized, and now it also shows the old “My First Document” document.

I then deleted that document, signed out and in again, and it came back again.

So that bug looks repeatable.

Makes me wonder….if there was a way to empty the trash, would that prevent deleted documents from coming back. Although why that document and not others is still a mystery.

Jay

Thanks SamMarrocco and Jerud! I finally reproduced the issue with it creating new documents. It only happens in the desktop app on signing out and back in, so that was a great hint. I’m going to keep testing it and aim to release an update in my morning (Asia time).

I’m looking into the boards issue. It seems like it gets fixed by signing out and back in? I’m thinking it must be having both the old board format and the new board format in the cache and it’s sometimes loading the wrong one. That’s my theory anyway - I’ll look into that after releasing the fix to creating new documents.

@Jerud The notes getting blown away is very concerning! Are those items synced to google calendar? Or did you visit the Version History? There should be lots of precautions in place to make sure that items can’t get blown away. I’ll try to figure out how to reproduce that but any hints would be very appreciated!

The formatting one may be somewhat innocuous. One of the performance improvements in the update is that it caches formatting hints after it parses, so it doesn’t have to do a full parse on every load. It may be that it’s not working correctly if items start with formatting? Are those the only ones that are having formatting issues?

SamMarrocco

Jay Good to hear about the random document creation!

The iPhone app is definitly pretty buggy about syncing. I’ve had several instances where I’ve deleted or added items to the contents of a document, then later gone into the iPhone app and that item had not been deleted or added. It almost seems as though the iPhone app get frozen/locked up. It seems to function, but all syncing capability is lost.

I force-quit the iPhone app, then restarted it. It still did not sync.
I force-quit the iPhone App a second time, then it resynced after a few seconds.

In a similiar-related issue, there have been many times where I launch legend, then start scrolling around, only to have it sync and suddenly changed the contents of the current document. This is a bit scary, as I can start making changes, and not really know if the document into which I’m editing is current (or not).

Should there be a method of “locking” the documents when Legend becomes active, or is brought to the front of a system before it lets you make changes? Or is there a method of performing a sync myself, then seeing that it is complete (i.e. a synching rotating beach ball, or "Syncing…../Succeeded/Failed or something?

I would think a sync is performed as soon as Legend becomes active/visible, but does it sync in the background in reaction to a push notification from Legend servers?

Jerud

Jay I did find another item in a different branch with the same defect. It was the same thing – a repeating subscription – I had used the same formatting and a similar thing happened. The Notes only disappeared in a few of the items, not all of them. I haven’t been working in Legend a lot recently (vacation finally!) but haven’t encountered any other item defects.

I have seen another bug with Notes where they aren’t restored during ‘undo’ operations, but don’t have good repro on it and haven’t posted it before. Seems like the way Notes get ‘connected’ to an item is perhaps a bit fragile, maybe a few markdown characters getting jumbled is enough to send that Note into the void? I wonder, does that mean it’s still in my Document, just not visible because it’s no longer connected to an item?

Jay

Just released a fix to the “My First Document” creation as well as the old boards coming back, which were both bugs specific to the desktop apps. Let me know if you see those issues again? @Jerud @SamMarrocco

SamMarrocco The iPhone app being buggy about syncing is very surprising… It’s the exact same sync code as the web and desktop apps so it shouldn’t be that different. I have seen some crashes in the error reporter that were happening during the sync process, which are fixed in this update, so it’s possible those issues are fixed now. Let me know if it continues to get locked up like that?

I’m not sure what to do about the issue with large remote changes coming in. It currently tries to show your documents as fast as possible - it loads locally and then shows your documents from the local cache, and applies remote changes when they come in. If instead we waited for remote sync it might take a really long time if you have a bad connection. There’s a “Connecting…” indicator at the bottom above the tabs - maybe that should be more pronounced?

It does not currently use push notifications - it just syncs whenever the app comes into the foreground. And there’s not really a concept of manual sync - it’s supposed to just be always synced. The saving indicator tries to not bother you unless there’s an issue. If saves succeed within 1 second it’s considered normal - on web/desktop it says “Online” and on mobile the indicator is just hidden. If saves take longer or it detects you’re offline it’ll show Saving or Offline. And if you open the overview in the iPhone app you can see the “Last sync” time. What do you think would be a better way to handle that?

SamMarrocco

Jay
I just updated all my devices to your new version as of today (8/20/23) and rebooted all devices.

Logging out/in is no longer loading old documents/MyFirstDocument. I’ll perform logout/logins more over the next several days and keep watch for it, but the newest version seems like it fixed that issue.

Regarding remote changes……yes, this is tough to determine what the best way to keep the user updated…..Here’s my biggest concern, which is common to any thing that relies on synchronization (as opposed to live data manipulation):

1-Device 1, create a new item, and that syncs with the server.
2-Device 2, that Item appears as it syncs.
3-On device 1, you update the item (Call this change A)
4-On device 2, a sync hasn’t happened for some reason or is delayed, but you don’t know that, and you update that item with Change B without seeing Change A.

Now what happens?
At some time later, does Device 2 finally sync, then overwrite Change B with the old Change A?
Or, does Change A, because it happened later, overwrite the Change A, preventing you from ever knowing it happened on Device 2.

Obviously this is a ‘dangerous’ scenario.

Regarding what you said about Legend not using ‘push notifications’ and only ‘syncing when the app comes into the foreground’……it cannot be only syncing when the app comes into the foreground, otherwise while the app is left in the foreground I would not see changes being made from another device. Is the client (Legend App) running some type of loop that says “any changes on the server” every few seconds? What is the ‘event’ that causes a Legend App app to refresh with changes? Perhaps knowing this will help me better understand why some syncs are not happening.

Jerud

Jay Just released a fix to the “My First Document” creation as well as the old boards coming back, which were both bugs specific to the desktop apps. Let me know if you see those issues again?

Has not yet recurred, and I just implemented the latest update.

Jerud

SamMarrocco Obviously this is a ‘dangerous’ scenario.

I have had the very same experience, with the same worries/concerns. My connections are almost always poor, but in the past this never happened. Big updates sometimes happen “before your eyes” well after the app seems to have settled after a refresh. It’s only happened to me twice, but it is indeed concerning.

I understand there’s a lot for you to consider here, balancing performance with robustness. Also there is much I’m sure I do not understand, technically.

I expect most of us on poor connections are quite aware we have a bad connection. Perhaps rather than a technical fix, the solution is informational – just tell me my stuff isn’t done loading and that I’ll have to wait. The “connecting” indicator is too small, easy to miss, and ambiguous in meaning. It is not suitable for handling this scenario.

The inconvenience of waiting a little while to get into my Boards is nothing compared to the stress/worry/lost time/lost data that can happen if you let us start working before sync is done. You took the trouble to make the new, better “loading” screen – I’d be perfectly happy to hang out there with a message asking me to wait until sync is totally complete.

A compromise – probably a lot more work for you – would be to open Legend ONLY into a fully-synced Board, and if user tries to switch to a not-yet-synced one, pop a message advising them they need to wait.

This applies equally – if not moreso – to mobile, where connections can be even more inconsistent and unreliable. And double that when tethering to mobile for cell data on desktop!

Jerud

Re: syncing Boards

Since rebuilding the few Boards I lost, I haven’t lost any more. But each time an update comes out, I do see something odd (and temporarily upsetting). This has happened I’m pretty sure during every update since the 13th – it was just lumped in with other, now-resolved, issues at the time. But this one behavior remains:

After an update I wait for the (win11 desktop) app to reload everything, and generally give it a good long while in the background since my connections are often poor. When I come back to check on Legend, everything looks good, and I’ll go into a Board to start working. I’ll make a handful of changes, maybe edit one item and insert a new one – not much. Then suddenly that board will disappear entirely. A new, empty board (named “Board”) will simultaneously appear in the boardlist (usually as the first one, but not always). It feels as if my board just got nuked. After approx 30 seconds of panic, the boardlist will refresh and be correct again.

Each time this has happened, it has been with the same board (name is 🟧), because that is a very common starting-point for me when switching-in to Legend – but it may happen with others, given the chance.

SamMarrocco

Jerud

I would have to agree with Jerud about the possibility of “only opening a board/document that has been synced”. The possibility of working in an unsynced document/board is far too risky, especially if you are moving between devices frequently as I do often.

BTW, this evening, I again opened my iPhone/iOS Legend App and was missing some items in a document in an already opened document. I had updated the document previously from another device. I waited at least 20 seconds for the iPhone document to update, and it didn’t. I force-quit the Legend app, then when I restarted it, the document was updated with the changed item.

I also had my iPad app do something similiarly odd. When I tried to bring up the Documents List, it appeared, but I couldn’t select a document and make it go away. I had to force quit it as well.

I’ve pretty convinced that the iOS (both iPad and iPhone) apps are somehow freezing/stopping sync when backgrounded or the iOS device is off for a while, and when they awake the syncing process is dead until the app is restarted forcibly. I’ll keep watching for this in hopes of discovering the steps for repeating the issue.

SamMarrocco

I can confirm that syncing of documents contents is still not working correctly. All my Legend instances are updated to the current version as of today (8/21/23) and all instances have been quit and restarted.

I have a document that, in the Legend native app on Windows, I had made changes to the contents of, adding several dozen new items and making changes to others on Friday (three days ago). Today, I ran Legend OSx native on my mac laptop, and thought "hmmm, some of my items don’t seem correct. I worked for a bit on the Mac and added some items and changed others.

30 minutes later I ran Legend native on my Windows machine. While looking at both my Windows and Mac Legend instances at the same time, I determined the following:

Several items in that document within the document I had changed last week on the Windows version had not been synched before today with the Mac version. I hadn’t run the Mac version since Friday.
-Several items whose order I had changed last week on the Windows version had not been synched before today with the Mac version. The order of those items on the Windows app was correct, and the Mac version they were not correct.
Items I had added to the Mac/OSx version today HAD been synched/added to the Windows native version.
Items I had made changes on today on the Mac/OSx version HAD been synced to the Windows native version.
-If I make a change to an item’s text on the Windows version, the text in the Mac item will change, but the order remains incorrect on the Mac from the order set last week in Windows.
If I change the order of a Windows version item today, the Mac item’s order will correct itself and move it to the correct place.
If I make a change to the item’s text on the Mac version’s item, the changes will show on the Windows version, but the order remains incorrect on the Windows version from the order set an hour ago on the Mac.
If I change the order of a Mac version item today, the Windows version tries to correct itself, but doesn’t always seem correct. Some items behave differently than others.

I then fired up my Web version and iOS (iPhone) versions of Legend. Both of those are showing the same inconsistencies, including out of order items that don’t match the Windows or Mac versions of Legend. I also force-quit the iOS version just to see if that made a difference, and it did not.

It’s as if the four different instances are showing the same document, in which some events cause the individual items to sync the changes or correct the order, and some do not.

Right now I am looking at the same document on four running instances of Legend: MacOS, Iphone, Windows Native and Windows Web Browser. All four have items out of order with each other, items missing and items that don’t exist elsewhere.

UPDATE: Two hours later…..
I copied the some of the contents of the Windows app document (NOT the document itself) and pasted it into a brand new document. The out-of-order items corrected themselves between the Mac and Windows versions.

I launched the iOS/iPhone version, and the new document appeared showing the first level of hierarchy. However, any attempt to navigate in the newly pasted items was met with “there are no items in this document/zoom level….”. I force quit the iOS app, and now it lets me navigate to the items, which now appear to be in the correct order.

Ugh.

Jay

SamMarrocco I believe the latest update should fix these errors. It looks like a large save only partially went through and it got stuck in the sync queue, effectively blocking future sync. I fixed the problem with large saves so these sync errors should stop happening, and the update may fix some failed saves that are stuck in the queue too. Please let me know if that seems to have fixed things or if you still see any issues?

The way to see the actual database state is to sign in on a new browser or an incognito window. If it’s not how you expect it to be then you can fix it there. The Version History is saved using the local state in any particular client, so backups made by the Windows machine should be the correct ones, so you could find the right backup with that timestamp and restore any chunks that are missing.

If you continue to see any errors send an email to support@legendapp.com (I get those on my phone so I’ll see it faster than forum posts) and we can do a video call (if you want) to work through any issues and make sure your documents are all as they should be.

Jay

I think I’ve tracked this problem down to the way it handles pasting really large data (over 12MB) at once. I’m working on it, will try to get an update out asap.

SamMarrocco

Jay
Is there a method in which I can tell how much data I am using in Legend? I wouldn’t have thought that I was using very large data at this point since I have just been testing. The sum total of all the OPML files I have ‘moved’ into Legend was less than 200 KB.

Also, does this mean that, after updating, existing data that was out of sync will be corrected, or do I need to rebuild documents?

Jay

SamMarrocco The closest approximation is the status bar at the bottom of panes that shows the number of items in documents. Boards can be surprisingly large if you collapse a lot of items because it saves a lot of item ids. The old sync system was especially bad about that so the migration of boards from old to new may have been the large save.

But it’s also possible this is unrelated, though it exhibited all the same symptoms you reported. I’ll keep looking around for errors, and let me know if you see any more inconsistencies.

SamMarrocco

Jay
I updated all my instances. Most of the out of order items seem to have corrected themselves.

I still had a couple of items that were not synched between the documents, but the vast majority were corrected.
-My iOS/iPhone did not pick up the resynched items; it was still showing them out of order or incorrect. I then restarted Legend/iOS and rebooted the phone. My iOS/iPhone is now showing the document, and the levels of that document in the Document List, but selecting the document brings up the main view as empty, as though the documents refused to view.

I then went to the Windows version and selected the Mobile Board (which still showed all the correct contents of the doc) and made a couple changes. The iPhone/iOS then hiccupped and now shows the contents of the document.

However, the items on the iPhone are not in the correct order. The Mobile Board on the iPhone and the Mobile Board on the Windows/Mac are showing two different orders for items.

SamMarrocco

Jay

After ignoring all the ‘bonus boards’ that stayed up on my MacOS version all day, I worked in the Windows version and my iPhone, changing major portions of a document.

Tonight I checked my iPhone version: The Legend app was again stuck on “SAVING”. So was my iPAD version, but they showed my document contents.

I went to my Windows Legend Native and a huge portion of what I had done today was gone.

So I went into the iPhone and iPAD, force quit them both and restarted them. Now the iPhone and iPAD both showed that huge portion of today’s work as missing.

AAAAARRRRGGGGGGHHHHH. I’ve been trying to get through a single day without missing boards, re-appearing old boards, out of order items not syncing, iPhone and iPad apps freezing and needing to be restarted, documents not synching, items disappearing and coming back to an app and seeing empty lines or previously highlighted items no longer highlighted or missing their children. Each day and update I think something has been fixed but something else pops up…..and these are all fundamental issues involving the synching.

At this point I’m spending more time repairing issues and troubleshooting the app than actually working with it, and this is a single, relatively small document. I was able to restore my document to it’s former state by using the backups but my frustration with its reliability is obvious.

SamMarrocco

I’ve also seen the iOS/iPhone app ‘stuck’ on “SAVING” at the bottom of the screen from time to time. Restarting the app fixes that. I’ll keep watching for this, but even after restarting the synched order of items is still incorrect on the Mobile board.

I have very reliable, fast connections on all my machines, so it isn’t a networking issue.

SamMarrocco

I just started Legend native on my Mac laptop, and now I have several old boards that were deleted several days ago, and several default boards were also added (Tasks, Today, Main Board, Management Board).

I checked my iPhone, and it was stuck on “SAVING” again, so I restarted it. The newly added old boards are NOT visible on my iPhone, and are NOT visible on the web version of Legend, but ARE visible on the MacOS version.

Jay

SamMarrocco I’ve been trying to reproduce this with a copy of your data on my test account to try to reproduce exactly what you’re seeing, but I can’t find any issues. I see errors in our error reporter from your account that boards are still in the old format, but all your boards have been migrated to the new format. So based on that and what you said about old boards coming back, I think the problem must be that your local cache somehow still contains the old boards and it’s trying to bring them back. I thought I had fixed that last week but I guess it’s still not quite right. I’m investigating that now but migration problems are particularly time consuming to test so it may take a little while to fix any issues I find.

So in the meantime, I think you may be able to solve all of the problems by signing out and back in. That clears the local cache and resets it to the database version. I think the bug is specifically in the desktop apps, but doing it in web/mobile apps too wouldn’t hurt.

I’m really sorry about this! This is the top priority to fix so we will fix it ASAP.

Jay

SamMarrocco I’ve been testing migration and everything I can think of for hours and can’t find any sync problems anywhere. I can’t even figure out how a local cache could get to the point where it would cause these problems.

The only other thing I can think of is that maybe your iPhone or iPad app hadn’t been used in a while and was on an old version from last week before some sync bugs were fixed? It updates to the new version after opening and closing it, so if it hadn’t been opened in a while it may have still been an old version.

If they’re still happening after signing out and back in, would you like to do a video call with me to try to track it down? If so send me an email to support@legendapp.com and we can coordinate there.

« Previous Page Next Page »