His hand brushed my a ss...?

Could it have been accidental? If you do not think it was, then you probably should distance yourself from him and not make a big deal about it unless you want to be a homewrecker. Think of the children.