I don't mind carrying it around long distances, I just don't like walking around with it through a lot of people.
I normally just carry a body with a lens attached, and a maybe one extra lens. If I'm out shooting birds I always have the 20 mm AF=D in my pocket (which I believe used to be yours...