Would this not work? Using the fact that Layouts is shared through the fram

 targetUrl = '/_layouts/MyProject/AppPage.aspx' + "?Location=" + window.location.href;

or you could try:

 targetUrl = SP.Utilities.Utility.getLayoutsPageUrl('MyProject/AppPage.aspx' + "?Location=" + window.location.href)