Link to initial performance thread I made when trying to figure if settings were what was causing the frame drops.
So, in the beta notes it said disable area chat for performance. Through my 10hrs of testing the most significant performance boost I found was disabling chat notifications over anything else. Disabling other items; Area, Normal, etc, would also show miniscule improvements but not ones that were game changers. When I say noticeable, I mean I had 10-15% less CPU usages and 20-30% less GPU usage just with notifications turned off, where the other settings would only give 1-3%
So through graphical changes of my system would still experience the random lag spikes regardless of if I was set to Very High or Low settings. Once Chat became fully disabled, it all stopped. Would only experience a FPS drop when entering a new town/area when settings were over what my specs could reliably perform at. At appropriate settings frame drop would be gone completely. As stated in original post the only commonality I was originally seeing was other players entering the area. Again once chat was disabled this no longer was happening, so it had to have been something with said player joining a chat channel for that area I am assuming.
TL;DR: Disabling chat settings is netting a larger performance increase than changing Graphic settings.