I have never downsized - or frankly, even sold anything in the past 20+ years. But I think its helpful to step back ask ourselves what we are doing this for. What is the focus of the collection? If there isn't a focus - then its more of an "accumulation" than a collection. Our focus may shift over time - and some re-thinking may be in order.
FWIW: A well known person who has been in the hobby for 35+ years once said: If you don't have it on display - its time to let it go.