Items disappearing

Jerud

In the last few days I’ve noticed some missing – or possibly mis-synced – items.
While it’s possible this is due to my own careless clicks or keystrokes, it’s happened twice now in as many days – compared to zero times in the preceding 6 years of using Legend, so I’m thinking this isn’t me.

The first time was yesterday: I had a 2nd-level branch that was named using (only) a OneNote link (this is an “attachment”). I renamed the section in Onenote, copied the new link, pasted it at the end of the existing Branch name, Then backspace-d to delete the old link. If you’ve never tried operations like this around ‘attachments’, then I’ll just point out: it can be tricky. The ‘attachment’ behaves sort of like text but the enclosing markdown it is also sorta-involved, so you can end up with unexpected cursor movements, deleting more than you meant, etc. But this isn’t my first rodeo and everything looked fine. Later that evening I returned to that section of my outline to find the entire branch I had previously renamed was now completely gone. Now: maybe I pressed ‘delete’ just one time too many and blew away the whole branch – but where was my “delete multiple items” Toast if that was the case? I dismissed this as my mistake and restored the branch from backup, renamed it again (same way), and had no further trouble.
Now today: I went to a small Project I know I had fleshed out last week, with several Tasks and a number of subtasks (numlisted). When I got to it, I found just two Tasks, both being old versions I had sketched in as placeholders weeks ago. None of the numlist items were to be found. I did several global searches on keywords I remembered, and of course opened it in a popup with no filters – didn’t find anything.

Both these examples happened in outline regions I’ve been working in recently, so I actually noticed them. Neither of them are crushing losses. But it’s scary to consider where else this may have happened – and I’ll simply never know – that is, not until I miss a payment or appointment, or omit an important detail I was counting on having logged.

I can’t help but wonder if this is related to mobile: I saw the recent mobile update, so in a fit of hopefulness I reinstalled the mobile app on my ipad and iphone a few days ago. Things seemed to be working okay; I was able to check my tasklist while out-and-about (I had missed that). But after this, I’ll be deleting those apps again.

@Jay, I’m not sure what useful details I can provide you, but if you DM me I can point you to the problematic items in my outline.

Windows 11 desktop app, stable.
iOS 17.0.2 - iPhone 13 mini
PadOS 17.0.1 - iPad 7th gen

Jay

I think the latest update should fix these problems. There was a window of time in which it would start a save while in the middle of another save and that could cause some strange behavior. So now that is fixed and after hours of bashing against it and trying everything I can think of, I can’t find any more sync problems anywhere.

I don’t think the mobile apps would be the problem. But that bug was exacerbated by multiple instances being online at the same time. I’ll test the latest version with mobile apps open at the same time just in case.

Jerud

OK, I’ll see if the issue reproduces, and will check using mobile as well.

It’s pretty common for me to open mobile while I still have desktop open, which maybe creates the “multiple instances” situation. I do that b/c I need to be sure the mobile app updates on my home wifi; I often don’t have reliable signal while away.

I just restarted from a full system shutdown and launched Legend; no app update appeared. I did a force-reload and then a manual check for updates – nothing. Any reason the last update isn’t pushing to me? I am on stable not beta.

Jay

It was only in beta but I just released it to stable.

It should absolutely be fine to have many devices connected at the same time, and I think it is now! I’ve been testing everything I can think of for a couple days now and can’t find any more issues.

Jerud

Marking resolved; I don’t see any obvious repeat problems. Will reopen if necessary.

Jerud

Okay definitely NOT SOLVED.

I put Legend back on my phone and synced up with a bunch of changes I had made on desktop last night. I went into town and added some notes to items as I was completing them. But while I was doing that, I noticed a project I had made last night simply wasn’t appearing. When I got home and could pay better attention I went hunting for it in the mobile app, turning off filters etc. to make sure it wasn’t hidden. Nope, not there.

I booted up my laptop and turned off wifi before logging in to prevent sync, then went to check for the missing project. Yep: it was just where I had expected it to be. So I hesitantly turned wifi back on and watched Legend sync, whereupon I watched the following things happen:

The items I had added in mobile synced back over to desktop: expected
The project that had never synced to mobile from desktop now disappeared from desktop
On the same Board, I had a Pane viewing one of my top-level branches which contained four sub-branches. Three of them just plain disappeared!!! I never even opened those branches in mobile…why?!?!
Elsewhere in that document, yet again I had a smattering of zombie items reappear
- These are items that I had just yesterday moved out of my “inbox” area at the top of the outline and filed into the body of the outline … now they’re back, but at the bottom of the outline instead of the top, where they had been.
- I saw them first when viewing my inbox in mobile, then after I synced the desktop app, they all turned into broken links – the item content is replaced with just the item ID (gibberish like: -MgS23KZdOCAenS2T_1u), which is displayed in [[link]] formatting – even though these had just been plain items, not links or mirrors.
- I’m certain that some of the zombies had been organized into one of the branches that got blown away. I’m not sure if other zombies were put into the other two vaporized branches, so it’s not clear if the mechanism that spawns zombies is the same one that blows away outline branches, of if that’s just a coincidence.
The rest of that document appears OK but here again…I wouldn’t know how many things have been quietly blown away at this point. The scope of the mis-syncing doesn’t appear to be limited to where I actually worked, which means defects could be anywhere.
The web app appears to have the same sync state as on desktop. I didn’t check it before using the mobile app so I can’t be positive that my desktop changes from last night actually synced up to the server, but I have no reason to think they didn’t, either.

This is all in STABLE.

This is the biggest and most recent step backwards from what used to be a totally solid sync experience in the app. I left beta because I couldn’t afford all the time I was losing to fixes in Legend, and now here I am in stable with the same problems. Where am I supposed to go from here?

@Jay, is this really only happening to a handful of us? Is it connected to older accounts, or documents over a certain size? I can’t get through two days without having to fix my outlines – unless I sandbox Legend to strictly local operation. Is there some feature that’s been added in the past year or two that is making sync so sketchy? Maybe mirrors? Because at this point I’d say you’re better off killing whatever “feature” is breaking your sync system, even if it means cutting back functionality – just to get the app reliable.

A productivity app without reliable sync isn’t worth dirt.

Jay

I think the cause of these issues should be fixed in the latest version. There were some edge cases around moving items between documents while offline that I believe were the cause.

And I added a new “Moved” pane view type which I believe should show some of the items that have gone missing, and you can restore them from there (like the Archived/Deleted views). So hopefully that should get all your missing items back. It’s not technically possible to delete data from the database, only to mark it as archived/deleted/moved. So I think the missing items were marked as moved and should be in that view.

Unfortunately yes, the sync errors are getting increasingly edge case, but it’s the edges that I think you in particular are more likely to hit, like being offline or on bad connections. And moving items between documents has always been fragile, and I think that was the cause of your issues based on the errors I saw in the error reporter? I am focusing my testing now on going between offline and online. This update is a big improvement and I can’t find any other problems, but I will keep looking until we’re at 100% perfect sync.

The update is just in beta right now, but I’ll test it some more tomorrow and release it to stable.

Jerud

I guess I didn’t think of myself as edge-case, but knowing that does make it easier to understand why these bugs seem more prevalent to me than they really are.

It does feel a lot like every time I use the mobile apps something bad happens to my Documents. If I just skip the mobile app I have to rely on other apps, which leads to using Legend less. But if I don’t try to use the mobile app, whatever is causing this havoc will never be found and fixed. That’s not an attractive pair of options : (

I got the “moved” Pane and tried it out – no luck. I do see remnants of a previous organizational approach in there, which were later moved out – so the Pane seems to work. But none of the recently-missing content was there. The three branches were “permanent” headings in my outline, so I know exactly what they were each named. No hits came up when searching on those names or when just scrolling through.

I am trying to restore to my last local state and will see how that goes.

I guess I never really grasped what “items can’t be deleted” means until I saw these items in the Moved Pane. Does this mean that every Document will continuously grow, even if items are deleted or moved out of it – because those items are never actually removed once they are created – just marked as deleted or moved?

Jay

The documents grow in the backend but they’re not loaded at all until you open a “Moved” or “Deleted” pane. Once we get sync fully 100% stable I have a project planned to optimize that further.

Jerud

Jay The documents grow in the backend but they’re not loaded at all until you open a “Moved” or “Deleted” pane. Once we get sync fully 100% stable I have a project planned to optimize that further.

Cool, that would be nice. I’d love to know I can fully 100% eradicate the vestiges of old “systems” that are lying around within my Documents. And, ideally, the old Documents in the trash, too.

wecanseeformiles

I’ve been getting a client up to speed on Legend. We had a critical connection error on his Macbook M1 today during an update which didn’t go away. Re-installing the app didn’t solve the problem but deleting the app with AppCleaner Pro and then re-installing got it working today. We had Legend working on another of his computers at the same time.

However I noticed just being leaving that one or two of his top-level projects had disappeared. I will investigate further tomorrow but wondered if this was a similar issue.

Jay

wecanseeformiles What was the critical connection error? That was a problem in Legend that prevented it from opening?

wecanseeformiles

@jay I found that the items that had disappeared were in your ‘Delete’ folder so I manually restored them. We might have accidentally deleted them but they were items from different projects so that seems unlikely to me.

Unfortunately I didn’t take a screen-grab of the load error message but it announced a ‘Critical Connection error…’ after which the Legend remained blank except for this error message.

This happened with a client’s Legend, not my own one. He does have intermittent internetconnection issues so that might be something.

drewpierce

Hi @Jay - First let me say, I freaking LOVE Legend!! And this bug is the only thing keeping me from buying the Lifetime license. 😉 (no kidding!)

I’ve spent countless hours searching for loads of lost data and then RCA troubleshooting what/when/where/why/how it disappeared(??) to try and prevent future loss & corruption. I kept stripping things down, reducing variables, adding flags, retracing my steps, and looking for common denominators in the [seemingly random] items that vanished almost every time I used the Mobile app.

The GOOD NEWS is, I’ve been able to isolate/characterize the problem and create a super-simple(!) minimal reproducible example for you to debug on your side. 😃 Here are the steps:

1) On Desktop, create a new Synced Document with these 3 items:

Parent A

Child

Parent B

2) Open the Document on Mobile, confirm clean sync, then close the Mobile app

3) On Desktop, move the Child under Parent B, then Delete Parent A

4) Relaunch Mobile app and notice upon sync that the Child vanishes

This happens to any/all Children moved from a Parent that’s Deleted or Archived soon thereafter (presumably before sync). If Parent A was Deleted, the Child will be Deleted too. And similarly, if Parent A was Archived, the Child will also be Archived. In either case, it’s like finding precious needles in haystacks to retrieve & restore all these beautiful orphans, especially when they’re intermingled and flattened back to root. 😔

Anyway, I really hope this helps. Please let me know if you need anything else from me at all, and I’ll be glad to tag-team with ya on getting this fixed. 🙏 Thanks so much for AMAZING work!

*My hardware/software config:
Microsoft Surface Laptop 5 running Windows 11 Home 10.0.22621 and latest Legend Beta 1703172630372
Apple iPhone 14 Pro running iOS 17.1.2 and latest Legend Beta 7.0.4

FYI @Jerud @wecanseeformiles

Jerud

drewpierce

Ooooh, THANK YOU for some repro! And also for validating that I’m not the only one seeing stuff like this. I was losing so much time recovering from these glitches that I never could sit down and try to create actual repro steps.

@Jay does this give you any new ideas?

Jay

Oh wow, yeah that reproduces for me. I’m working on it now. I’ll get an update out ASAP. Thanks a ton for tracking that down drewpierce!

Jay

Just released the fix to beta! The problem was that it was running the logic for deleting children when deleting an item even when the delete came from remote. That was mostly fine but very bad in your repro steps: if the remote delete event came in before the move event, it would delete the child. So the solution is to just not run the deleting children logic for remote deletes, since there should be a remote delete event for each of the children anyway.

Those repro steps were a huge help drewpierce! Thanks so much! I’ve been testing everything I could think of to try to find why some people had items getting deleted and couldn’t find it. I think I tested the opposite (moving a child from B to A) and that worked fine because the events came in a different order.

So this fix should be a big improvement 😀

christophe

drewpierce
We have a prize to give out today.
The year has barely started and we already have the message of the year!!! 😁

drewpierce

@Jay - YES!! I confirmed your fix just now, and true to my word, immediately purchased the Lifetime license. Amazing. Such an insanely fast turnaround. Huge props and gratitude.

Legend is my digital heaven. I have found the promised land 🙌 ha. You’ll be seeing/hearing much more from me on the forum.

P.S. I’ve adopted a fail-safe clean-up approach to quickly & confidently identify any unintentional Archives or Deletions in the future. > Basically tagging items as #archive or #delete first, and then occasionally checking root-filtered Archived/Deleted Views for anything -#archive or -#delete (though sometimes for simple deletions, I just Backspace the characters manually). Anyway, I’ll let you know if I find any more kidnapped children.

Thanks again. Sooo happy…