To remove groups from your Saved Groups screen, tap the three-dot menu button in the upper-right and then select "Edit".  On the next screen, tap the menu button again and select "Clear". This will clear out all of your currently selected groups and allow you to reselect only those you want to see. There isn't currently a way to deselect groups one at a time. Tap the back arrow in the upper-left when you've finished.