So this one is partly my fault. I went around my stronghold farm and gathered all the materials BEFORE accepting the quest ‘A Verdant Land’ from the Jack of All Trades Nature NPC. Without having anything left to gather on the island I couldn’t complete it.
I tried abandoning the quest thinking it would pop-up again and I’d be able to recomplete it. It never did. Now I have no way to access the stronghold farm island because there is only an exit portal on the map and no entrance since I wasn’t able to complete the quest to unlock the portal.