"...I didn't bring it up to call you selfish for it. It just felt like an overreaction, even if I was being unkind to you..."

Because he was, he knows that. Even if he feels some justification for his reactions, he still let his feelings get the better of him...

"...I'm used to being alone with my feelings, so sometimes it's easier to remove myself from the situation when I'm feeling too agitated... it doesn't mean I'll disappear forever."

At least, it might not. Though out here he doesn't have his forest home to run off to and they're confined by the city walls, so it's not like he has many places to disappear to anyway. But in any case, he just wants to make that much clear in case it happens again...

But as for the memories he's made with him...

"...If just the memories are enough to help you in the future, then I'm glad for that... but hopefully you can make better, happier ones with the people you know and love."

As unlikely as it may seem, for many reasons... but at least he's able to give him these relatively small moments of happiness and normality until then.