Daily Updates on Disconnect Work Needed

Good afternoon @Roxx,

The disconnect thing has been an issue for a very long time, and it just seems to be getting worse. You can barely play the game anymore, with a great deal of people being disconnected not just multiple times a day, but multiple times an hour.

This is a HUGE problem, and it’s been ongoing long enough that the players (aka customers) deserve daily status updates on the progress here. Additionally, I would like to request a formal RCA be published once the problem has been actually fixed.

The constant disconnect problem needs to be driven as priority 0 (highest severity). This should be a “drop everything else and all hands on deck” situation, even if it causes the next content release date to slip. All it’ll do is take away from new content as frustrated players are disconnected while trying to learn it.

3 Likes

Honestly, realistically speaking, the disconnect issue may not be resolved until March 2023.

It’s been literally a month and apparently no significant progress has been made in identifying or resolving the disconnect issue. Their priority as of now is most likely preparing for the Brel patch. Then there’s also the rumor that most employees goes on a holiday break around 12/19/2022 till new year, too.

Once they come back, then another month or two identifying the issue and then trying to find the solution for it.

They just have to undo whatever they did on the patch that was implemented on November 9th, 2022 ( Nothing big apparently was even done on this patch according to the update report on forums ). However, it was prob some crucial thing that was added in that prevents them undoing it. They can’t exactly rollback 1 month worth of progress, too if such a save point existed anyways.

2 Likes

Funny you say that, I absolutely understand. Yes a lot of people in our industry go on vacation starting about… well some are already out now in my office. xD

But realistically, they should be holding the Brel patch back and working on stability.

Anyhow, turns out I’m a cloud server engineer with past experience in games who works primarily in native code these days. So I’m going to give a true story from, oh, about 3 or so years ago on why I view things the way I do here:

Once upon a time, the stability of our services became rather poor due to an increasing volume of failures (turns out, native code is complex yo!). Now, given how cloud works, if you have 200,000 machines out there, and 10% of them have a failure, you’re likely ok and customers won’t even see any impact unless you’re running hot on most of them (at least we have good seamless failover).

But eventually things did start getting bad enough that it was impacting customers. At that point, our entire team froze all feature development in its tracks. It became an “all hands on deck” situation, and literally everyone on the team dug in for upwards of a month debugging and fixing a mass of different issues across the board. By the end, we had reduced our failure rates by over 80% dropping us from 1000’s of failures daily to lower double digits. The impact to our customers was so immense that several contacted us directly thanking us for how much better everything was.

That wasn’t the end of it though. As we worked on fixing things, we also worked on identifying how things got into that state to begin with, and worked on ways to gate things to prevent it from happening again. Sure, feature timelines slipped and had to be pushed out, but to be frank that’s the cost of running an online service business. Also, like games, in many cases like ours there is no such thing as a rollback. Due to how things stage with things such as code and customer data, you can’t just go back to an old version of the services.

Now, things like an MMORPG isn’t anything special. It’s just an online service. The players are their customers. The biggest difference I see is lack of accountability which you see in cloud services. Perhaps it’s time that Amazon pushes AGS process to be a bit closer to AWS.

3 Likes

for your information, it’s hasnt been a-month.

it has been since July that widespread of this issue was report.

you can see the huge megathread started in July with over 5000 replies.

this issue WILL NEVER BE resolved until EAC is removed from the game. It is the culprit that causing all the disconnects.

2 Likes

Lots of good stuff here.

At this point we have to assume the CM doesn’t receive updates at all or simply isn’t asking for them. 100% not going to ever see a RCA for this either.

This is a pretty accurate estimate with the information we have currently. Brel will pop just in time for everyone to disappear for the holidays. Radio Silence continued :slight_smile:

100% Agree, but it sounds like that might take some work and frankly I don’t think Andy Lin (AGS Lost Ark Production Lead from what I can tell anyways) want’s to take on that kind of responsibility.

EAC is a huge factor in all the dc’s, but at this point it feels like something more than just that.

it is. the problem is SGR lazy coding practices doesnt play well with EAC.

so SGR needs to go back and fix the codes, which will never happen, or take a huge amount of time. or remove EAC.

this is why AGS team is at Korea to discuss options.
SGR will never ever change their codes, they negotiations are probably be on the table. the outcome is EAC is removed, or switch to another option that play. well with SGR coding.

Was this confirmed? Last thing i heard about any of it is a team was going to Korea.
I had just assumed it was to discuss the next roadmap tbh.

As far as the lazy coding goes, I can’t say anything about that. From what i’ve heard KR and RU have their own Anti-Cheat that they made specifically for their game. Afaik when it comes to our version AGS/SGR didn’t want to/have time to make a Anti-Cheat for this version so they slapped a free one on it. Feels more like old school GameGuard though :rofl:

Day 1: Worked on December livestream and patch notes for Korean version.
Day 2: Heard something about the global version being laggy.
Day 3: Worked on December livestream and polished presentation material for Korean version.
Day 4: Group from AGS arrived, we all went out for some great BBQ.

And so on.

4 Likes

LOL this sounds similar to when I had dinner with Google and Rovio folks while we were at GDC for one of our games. There’s so much dinners and beers in games. :stuck_out_tongue:

Sure, technically it hasn’t been a month since this issue existed, but it has been a month since this issue exponentially expanded to affect the majority of players + how consistent it has been.

I would argue at least 60% of concurrent players will tell you a month before this issue gotten so out of hand, they had no disconnect issues such as the Gx09 or the Wx one. Now it would be maybe down to around 10% of concurrent players having few to no disconnect issues whereas 90% of concurrent players have frequently experienced some disconnects in the past month compared to the previous 9/10 months.

I understand that there may be an industry standard/expectation, but I would say that each company in your industry has its own unique circumstances whether it’s ethics, efficiency, etc. However, AGS has such an extremely poor reputation to the point nobody should realistically expect them to get anything done effectively or quickly whether it’s quick turnarounds or failing to meet deadlines.

At least for me, it’s honestly hard for me to expect the situation in AGS to have any similarity to your story. if AGS had any sort of urgency or proper leadership/management, then they would have resolved this issue when it first came to existence than let it sit a while for a couple months before November 9th happened and it exponentially expanded to effect arguably 90% of the concurrent player base.

A business imo needs good leadership/management for effective actions to lead or redirect the company/business in a good direction. Your experience/story at that company paints that picture in my head, but AGS doesn’t have good leadership/management shown over the past 9-10 months.

This past month is a perfect example of poor to little to no leadership/management reflective of AGS from the Pheon scandal, ignoring players about SH, disconnects issues, auction house issue, bot issue, lengthy black screen issues that affected EUC Asta + NAE Karta so far, their decision to continue on releasing Brel patch on 12/14 disregarding the list of issues affecting the game so far + stability.

TLDR: AGS doesn’t care about their customers. SG doesn’t care about their customers in NA/EU. Both companies just want to milk it before the next MMO takes away the majority of their temporarily customers. Both had the opportunity to profit even more from the NA/EU of Lost Ark, but the lack of leadership, management, urgency, efficiency, accountability, and so many other lacking crucial aspects of what makes a good business/company is driving this game to the ground, hurting potential profits massively, and ruins the NA/EU experience of Lost Ark making it into just a burning waiting room until the next MMO comes out to jump ship.

Yes, I know, it’s sad but true. I was just using my example of how things should go, not how they are going. Asking for an RCA, even if I won’t get one, is a common practice across the industry by customers impacted from service issues, especially paying ones.

I wonder if this lack of management and direction has anything to do with Mike Frazzini stepping down from AGS earlier this year? Let’s be honest, AGS isn’t really that big. They make / produce very few games:

And they have only a small handful of studios across the world:

It isn’t exactly a monumental amount of things to manage for whomever took it over. It’s not like they’re the size of Activision/Blizzard, Tencent, EA, or even MGS.

However, I’m also aware of how some of these corporate structures works out too. Just because AGS is part of Amazon doesn’t mean it’s treated as such inside Amazon. Large companies like that tend to segment things into separate organization units within the company, and they cannot always draw on the other units resources. So even if AGS is in Amazon, it could be woefully underfunded, even for the small number of products it has.

Sadly, short of some actual transparency or a whistleblower dumping the beans, there’s a lot we will never know here. So long a whoever’s in charge understands what this looks like from an outside perspective, and realizes that some people (like myself) DO report things gaming site folks directly (I have several writers who I email about things when they happen in this game).

When something is posted on these forums, it gets lost, drowned out, or more importantly they can remove it if they want to. But when it’s published to things like PC Gamer, Play, Game Informer, Kotaku, etc… ignoring it is a bit harder because it’s a MUCH wider blast radius that can have far bigger consequences. I totally encourage others to do the same, most writers contact info is available on their respective sites. But don’t email them just to complain, give actual constructive information with proper timelines, counts, etc… :wink:

2 Likes