What happens when the user refreshes the page? is the refresh token sent? Are you sending the refresh token with every request?
The key idea with a refresh token is that its securely stored on the device and used so that you can have a short expiry on the access token without forcing the user to login multiple times.
It should only be used when the access token has expired, but a naive approach with cookies will mean its sent on every request, negating the whole point of an access token.
My suggested approach is this:
You have a single page application.
Have the authentication api on a different origin.
Store the access token in Http Only cookie for the API origin and the expiry time as a javascript variable.
Normal site usage doesn't refresh the page but calls APIs and dynamically changes the DOM.
API calls have the access token sent in a cookie.
Precheck that the token hasn't expired via the javascript var
If the token has expired or the API call fails. Hit the authentication API to get a new access token. This should automatically include the HTTP Only cookie (for the Auth origin) with the refresh token.
If that fails. Force the user to log on again, getting a new refresh token.
The end result is that the refresh token is only transmitted when the access token has expired (or new login). This mitigates against man in the middle attacks on the refresh token, which would grant an attacker long lived access
The access token is used on all API calls, but has a short expiry, if its intercepted the vulnerability has a time limit.
When a refresh token is used, it should be revoked and replaced with a new refresh token. This lets you detect interception as you will see the attempted use of revoked tokens. Either by the attacker using old tokens, or the user being logged out by the attacker, forcing them to re-login.