I do, but that’s called planning, and as you chose to ignore the part where I said “its not about the mess up, its about how they handle it”.
If they messed up, and then said:
hey guys, we messed up. If you want a refund please contact us here: … Were implementing a new region and we have taken your feedback on board. We will be cloning the entire region so no-one loses their purchases and progress. They won’t stay in sync, but please choose which region to play on going forward. Unused accounts on the other region will be disabled after 1 month
See look i just solved the entire forums problem. Wasn’t so hard was it.
You didn’t wolve anything though, it’s easy to just make up solutions without having to deal with pesky logistics lmao.
I mean you are suggesting that amazon dont have the ability to clone an environment? Because that would suggest they are unable to back and restore too? Which would be quite a big problem. It would be a massive worry for me if amazon are unable to do this, rather than unwilling. a really huge worry.
My comment is based on my experience that every single server environment has had for the last 10-20 years. Im not saying its easy, im saying its possible. just amazon dont care enough to do it, they already have our money and everything else is irrelevant.
For 8 years I worked for one of the largest airlines on the planet as a Regional LAN admin. Once we were copying data from our backup tapes to a different medium and it was a massive problem and even with our full breadth of engineering staff and HP themselves sending in techs it was still a nightmare.
The idea that they could just “copy over” an “environment” is just laughably simplistic.
Cloning seems like an obvious and easy solution, but:
it’ll take a long(-er) maintenance period and people are already rioting because of the 3h frames in the morning (okay this would only be a small shitstorm)
after the cloning process, you’d have to decide, if you want to use region A or B and disable/delete the other one → what if you can’t coordinate this with all of your friends/guildmates and end up in different regions (again shitstorm)
many accounts are linked to digital purchases done through Steam, that can’t easily by duplicated I guess. Just imagine it would look like a key you want to redeem and Steams says “already in use”. That’s exactly what would happen, by cloning the servers. I suspect that’s also one of the facts why they can’t easily clone all the founders packs. Maybe if one account is deleted and the other one is then persistent (by choosing region A or B) this could be done, but everything would’ need to be in sync with Steam to avoid duplication…this might even have fiscal implications and so on
because of all the “sync issues”, I believe it is not possible to run just clone and run both regions at the same time with exactly the same accounts. You’d have to implement a mechanism to let the players decide upfront, which region the want to use, then clean up the region servers, then start the servers (and fingers crossed, everything goes well)
…I guess there is a whole lot more to consider
In general your solutions sounds great, but if you’re think about it, it sadly gets more and more difficult.
Your right, the IT systems of an airline clearly match that of amazon, or for that matter, any software house worth their money. Obviously, an airline put a lot of resources into their IT… i mean just take a look at their websites… its really marvellous stuff.
I’m done with this conversation, i dont care for your opinion on the matter, your experience clearly shows your know what you are talking about. Amazon have handled this poorly, especially in the aftermath. I dont know what else there is to discuss.
Agree, but I mean, its already terrible, its only going to get worse.
True, but at least people have the option and dont lose anything. The first week might take a little time for people to settle. After that it should iron itself out.
Yes, but i doubt it would re-redeem from steam, it should already be available in inventories. If it does, that would be strange but a valid concern. Should be possible to iron out from AGS though.
Trading of items on the region not used. EUW person trades EUC person, and then they do visa versa on the other region. Also potential for AH problems with people selling cheap. However, these can also be worked out and resolved.
I’m not saying it free from problems but the alternative is everyone loses everything, including what they paid for.
Do you really look at an airlines website and take this as an example how the whole IT must be structured.
Oh well, that’s really very simplistic (and I don’t believe you really look at it this way).
I give you a very simple example:
lets assume you have a home network with just one PC and assigned a fix IP to this machine
now you clone the PC, because some family member also wants to play LA with you, but doesn’t want to install Windows, Steam, the game…you get it
now you boot up this “Clone PC” and…you got a conflict in your network, because two machines got the same IP
Well yes this is a really over simplified example. But now scale that up by…let’s say a million (which might come close to an MMO serverfarm) and you are getting close to the problems AWS might face when just simply “cloning the whole EU region”, or also taking the example of Thenatural, cloning the backend infrastructure of an airline.
In the end it is just a matter of time.
Just because it is so complicated, let’s assume Amazon says: “we found a solution, but it’ll take two month”, or just for arguments sake, we assume they are really, really quick and get it done in one month (which I think impossible, but we’re just fantasizing, right?)
How do you think the reaction will be?
They already announced, that they are working on the best possible quick(!) solution and even after 48h people are rioting, because they are not quicker.
Already they have no ETA for setting up a whole new region. Do you think they could give us an ETA of cloning and syncing everything quicker?
Cloning servers happens on millions of servers across the internet every single day. I literally have one being cloned right now so I can test something against it. Yes cloning isnt a 100% clone, certain things change, but its a known way of working. If amazon cant do this then im shocked, its technology thats been around for several years.
As for time, presuming amazon actually hired talented people, It shouldn’t take that long for collective minds to work out solutions to the problems. Actual maintenance times would only be due to the size of the data… problem wise, they should have plans in place before they start. Going in blind would obviously be very bad. Planning it probably takes more time that executing it, except maybe because of the data size.
I mean your suggesting amazon dont have the ability to clone a server, something thats been available for several years. So I can only assume, in my sample size of 1 (you), that airlines dont really have their stuff in order.
We are not talking about 1 server here I can also clone my own servers in no time hehe.
But just as an example, you might want to read this article:
Or just look at some of the diagrams showing the architecture. This is not Lost Ark, but you’ll get an idea of the scale of it. And this is without Steam involved.
If it was very simple, you need:
1 login server
1 “trade” server
1 gaming server
1 database server
some other infrastructure stuff like load balancers and so on
An as before: now scale this up several times to get on a scale of cloning a whole region.
I’m not saying it is impossible, quite the opposite. But it takes a whole lot of time and testing. Because you need to clone everything, get it up and running, without anyone being able to connect of course. Take necessary steps for the re-configuration for the new region, test, test, test and then you’ll have it read.
I think Thenatural could make a far better assumption of the time necessary to do all this, but I’d say just out of my head, that even 2 month would be a very close window to do all that.
well firstly, this is specific to a region. we already know that. And yes there will be multiple servers and services. Which is no different from bring up the new region, it will be lots of servers and services (presuming this isn’t cloud based, and i seem to remember they said it wasnt). Its the data layer for a single region, so really it isn’t that farfetched to be able to clone. Just google around:
I can’t find the post, but one of the CM’s confirmed that the game isn’t setup for AWS, so doesn’t use it. If it was using AWS this cloning wouldn’t even be a discussion tbh. It’s to do with the games architecture, which is fair enough
Edit: It could be using azure, but i’d be surprised if amazon were paying a competitor for something this size… maybe they are.
In the end, it doesn’t really matter if AWs or not (I can only think of the point, that the CM meant, LA is not using the normal AWS stuff that you and I know):
As pointed out before, your approach is a good idea and might even be possible, but it is highly complicated and would take more time than they have.
Or otherwise don’t you think the experts wouldn’t have thought about this solution already and instead of creating a bigger and bigger shitstorm every day, they wouldn’t just have announced it?
I can just repeat this in every thread I’m involved in these discussions:
Amazon wants to earn much money with the game
the lose this money, if players are not playing and thus not buying stuff in the store
do you really all think they are delaying things on purpose, to get more and more negative publicity
and do you really think they don’t have any experts in the field, coming up with the best solution for them to get back on track asap to make that money?
No, I think the experts in the field are being told by managers what to do. Because the only thing that matters is the bottom line. A fresh new server for people to spend money on is far more profitable, than satisfying your existing customers who have already paid money and aren’t getting refunds.
Fair point. While the numbers aren’t the highest in the world, when you tack them onto 1.5 million founders sold, plus whatever it made after launch. Its not like they are broke. They should be offering refunds at the very least but, i dont think thats ever coming .
I can also clone my own servers in no time hehe.