*sniff* My 4 year old son said "I hate you mom"..?

Do all kids say that at one time or other? We have been having a rough time lately, especially since the birth of my daughter 9 months ago. He is a good big brother, and most of the time he's a really good, sweet boy. But he has been really acting out lately and testing me to my limits!! When he is angry, frustrated, overtired, excited, etc. he will often hit me, kick me, do everything that he knows is ....I put him in time outs, I take things away, I yell, occasionally I (save that for last resort). My husband always uses shouting as a first response and that probably isn't helping. But today, he was misbehaving and he thought it was really funny that I was getting angry and frustrated so I put him in his room and turned off the tv and he said "I hate you mom":( It really hurt my feelings, even though I know he doesn't even really know what that means and a few minutes later we were giving hugs and he told me he loves me. We talked about it and I told him that "hate" is not a nice word and he shouldn't say that to people that he loves (or anyone). Some things he learns from the neighbor kids who are all a couple years older than him. What do I do? I feel like I'm constantly yelling or giving him time outs or feeling frustrated. I try to be mellow, patient mom, but lately he is making that impossible and I end up feeling bad, like I'm doing something wrong. Do all parents feel this way?? I used to feel so close to him. I feel like it's too early for our relationship to be so hard. Any helpful advice would really be appreciated, thanks:)