I personally had never given up on a game before because of lousy controls. I've always found some other reason not to play something like bad game mechanics, general lameness of the story or repetitiveness, but a few days ago I've had my first ragequit over bad controls (not so much rage more than 'wtf is this shit, what monkey programmed this')
This particular game was Fatal Frame III (you take pictures of ghosts with a special cam to exorcise them) for the PS2. After having played II remake and IV (English patch) on the Wii and finding them generally enjoyable, I wonder how people actually enjoy III. The character walks like a wooden puppet, the camera controls are terrible or forced into fixed-camera and the capturing ghosts timing is horribly inaccurate (I'd sometimes hit a ghost for full points after they had vanished and flashing them right in the kisser would miss?!). From the horrible viewpoint control to the non-intuitive layout of all the action buttons I put this game down for good. I'm glad they learned with the remake of II and IV.