The shorter time together can wear on the heart, but the memories and knowing you made his last few years ones with love might be a little bit of a comfort for you when it's time to say goodbye.
I've had to let 4 different animals go over the last year due to age related illness, and it's tough, but the memories I have are priceless. I don't regret the pain of missing them because it comes from love. It isn't easy, but I believe it is worth it.
However this turns out, I'm glad that you two have met each other and at least have that together.
Good luck