You should worry when your gutters have cracks in between. These cracks will lead to leakages from the gutters causing damage to the house exteriors. Here is the PPT about when to replace gutters and 5 key signs you should look for.