I need advice on why these things bother him?

I don't think a 6 year really understands the situation. If you are the only mother figure to her life then I don't see it as being a problem as long as she knows that you are not her biological mother. I think it sounds like he has commitment issues and he has every right to from his past relationships. I think by hearing his daughter call you mom is a reminder of a commitment. I think the only thing that will help is time for him and seeing that you are not going anywhere. Would it help if you and his daughter come up with some other name for you than mom, like "sissy" or something else. that way you still have a bond with her but it won't offend her father?