The size of boards is mainly in panes’ collapsed state. It saves the collapsed state for each view a pane has been on, so that if you collapse a bunch of stuff in an Outline view, then switch to Agenda and back, it’s still as you left it.
But they actually shouldn’t be able to get that big anymore. The logic was originally quite naive, just saving the id of every item that had been manually collapsed or uncollapsed. Then mirrors made that more complicated, because an item could be in a pane multiple times through different hierarchies, so the unique ids got bigger. In the past few months I’ve been progressively making that better - reducing the length of the ids, improving the algorithm so collapse all sets a base state instead of modifying every item, making more targeted updates, etc…
But panes created before those fixes would still have the old large collapse state, and cloning old panes copies its old large collapsed state. So if you wanted to optimize you could close and recreate some panes you modify often, or just toggle collapse all (that resets all item collapse state and sets a base state).