The Duke of Sussex shared his concerns about the other royal family members, including the three children of Prince William and Kate Middleton