Someone I care about told me I've "hit the wall" yesterday, ..

Someone I care about told me I've "hit the wall" yesterday, I guess to try and hurt me.
I guess it's not showing that I am the most confident about my appearance that I've been in my entire life. I am more happy with myself than when I weighed even less, because I still had major self esteem issues then, because I just didn't like how I looked. Now, I've hit my goal weight while accepting what I thought were flaws, I've got a haircut I love even when I don't style it, and most people that meet me believe I'm still under 30. Age has never been something I'm afraid of, only dying. Now that I like how I look, I'm just worried about my mental health and struggles with life. I made the joke when I hit 30 that I've hit the wall (literally), but I've felt old for over 4 years. The number really doesn't effect my mind, or my body to be completely honest. There are some really hot old people who take care of their bodies.
It's strange to me that someone would lie, either about being attracted to me or just to try and make me upset. I've never insulted them for their appearance. They told me many times that I was gaining weight and that I should work on it, when I never said the same, not that it really matters. But I've lost the weight now, and I know that he likes how I look. I'm not trying to be conceited, that's just a fact. So, it is hard for me to know if I should respond to that kind of hatred if isn't genuine. If I knew he really thought I was too old and getting ugly, that would make it easier to move on from. The choice to be cruel is what confuses me about human beings.
But yeah, since it doesn't actually effect me or make me sad, it's hard to know how to feel. The sadness comes from the idea that someone who claims to care wants to try and tear me down.