Lol so this happens when you go to South Vern before completing quest

[JHL]

[Anii]

Had the same issue. Can get in through Vern Castle TP to Calderia, top left.