I also currently own a Samsung Galaxy S3 mini & a Samsung Galaxy player, both of which have a 4 inch screen. I am very happy with the screen size and what I am able to do with the available space (which is mainly texting, skype/whatsapp, using fb/twitter/pinterest/foursquare, playing games & reading emails & news)
I have quite small hands, I find both devices very comfortable to use (the s3 mini is slightly more comfortable as it is curved), my previous phone was the HTC Sensation XE which had a 4.3 inch screen, again I found this phone very easy to handle and operate and although .3 doesn't sound like much of a difference the screen seemed huge compared to a 4 inch screen and the only difference in handling was down to different designs from different manufacturers. I think 5" is the maximum I would go to for a phone screen, although I'm not really bothered about the potential for needing to use both hands to operate it, the 'concern' I do have is the possibility of a larger device not fitting in a pocket as I really dislike carrying my phone in my bag, I like to have it where I can reach it easily.
I have a 7 inch tablet which I use for kindle & google books/magazines reading, keeping up with tech news & forums (such as this one), watching tv/movies/youtube and playing games