Is this considered verbal abuse from my manager? i'm kinda scared :(?

I recommend you always have your phone near you when you're with him, and also everytime you're near him you should make sure that their are always some other people around. He sounds like he has a short temper and that's not at all a good sign. You may want to report him for harassment and/or carry something small in your purse for defence purposes. (Or put a big rock in your purse all the time so if he tries to do anything you can crack him over the head.