Maz, adults do lost some of their lactase as they get older, so become able to handle less lactose over time. Sometimes this leads to total lactose intolerance and sometimes it just means they can tolerate less lactose. Your symptoms sound similar to lactose intolerance but as the others have said, you can only know for sure with a true elimination diet -where all lactose is removed and then re-introduced. And it's not just dairy that you have to avoid. It could also be cows' milk protein intolerance and there are two proteins, so after an elimination diet some people find they can reintroduce some dairy foods without an issue.

I hope you can figure it out because not being able to drink coffee would be a disaster!!