If your transport facility is a T1 or Multilink T1's you may have a defective T1 in the group. If you're Ethernet to your provider (completely Ethernet/fiber) not TDM anywhere then you're probably just subject to your internet providers network if MPLS, or just the internet and/or calling area for your party.
"Breaking up" if it's affecting 'all calls' 'all users' would point to the transport line as the possible problem area. If that's all good a router reboot may be needed just to clear our arp, mac and dhcp. If you're using poe on your switch, check the wattage/current draw on your ports and see if one or two are excessive as compared to the rest and troubleshoot the cabling and/or phone itself.