You probably have paper wasps in the birdhouses. They commonly build there on the inside roof of the nest box. Use the end of a stick from the ground to knock off the wasp nest while wearing a pair of gloves to protect you from wasp stings. They are usually out and about during the afternoon which is a good time to do it. Generally they leave one to guard the nest …. but you may be able to find it without any guarding.
They generally build elsewhere if a nest is destroyed, but they might try again in the same spot. If the wasps repeatedly build in your nest box, coat the inside of the roof with a non-toxic substance like a thin application of Vaseline or Ivory soap: this will prevent them from building but won’t bother the birds.