you decide not to fight back, maybe you think it would get worse if you do something in that situation.
I believe this is the reason many endure their bullies. They are so afraid the violence towards them will escalate they lose calmness to realize they are already having violence forced upon them and really have little to lose at that point. The doorway to this for most is shock that someone interpreted their kindness as weakness, coupled with not being trained in how to operate from a position of pain.
You can use grabbing and locking techniques to defend yourself.
Something everyone should learn. Proper locking techniques will quickly nullify size and weight differentials when implemented fast.
or throw him on the ground.
Something else most don't understand, if an opponent is thrown correctly, the ground can hit your opponent harder than you will. Enough so that one needs to be aware of controlling how they throw the opponent to ensure that they don't accidentally kill or permanently damage their opponent if the circumstances don't justify such an outcome.
Many people associate martial arts with lots of kicking (and high kicking at that) and for most its more practical applications such as you describe here, coupled with an understanding of pressure points.
Love these posts. Your ability to express options to those with little experience using words is impressive.