Jump to content
Sign in to follow this  
You need to play a total of 5 battles to post in this section.
KaptainNemo_1

The most broken thing in 8.0 is the pre-release testing

21 comments in this topic

Recommended Posts

80
[P2W]
Members
126 posts
8,144 battles

Obviously we've had buckets of commentary about the new CV features in 8.0, and we see that there is going to be a hot-fix coming to address some of them. I've also seen repeated commentary from both WG employees and fellow travelers that there was insufficient input on the Public Test Servers to address these issues, so they had to just channel their inner Bill O'Reilly and shout "We'll do it live!!!" I'm writing this comment to say that this indicates a really broken development and test process for the software.

By way of background, I'm a professional software engineer with roughly 40 years in the business and an advanced degree from a top university, so this isn't idle speculation about how software development might be done. This is based on solid experience developing various sorts of software, all with far more critical audiences than the ones faced by game developers. So, my opinion is that there are two core things that are broken with the WG process:

  • Not having a decent in-house test group able to identify both functional and design errors
  • Not doing enough consulting with highly-skilled experts  before releasing an update

Let me take these in order. As for the QA group, I'm astounded at the low bar they seem to have set for themselves. The issue that most affected me on release date was that the upgrade process on Macs produced a game that had what appears to be a dynamic link error - the game attempted to call code that wasn't present in the Wine wrapper and died. This wasn't specific to my machine but rather was reported across multiple vintages of hardware and OS X releases. It would appear that little or no pre-release testing was done on one of the platforms. If you're not even going to test that the damn thing starts then don't tell me you have a version that runs on my computer.

Once that problem was addressed (you have to upgrade to the WGC as far as I can tell), I now face a situation where the game routinely (more than 50% of the time) locks up hard during some portion of the load process. Some times it dies as soon as the port screen shows up, some times it dies during the pre-game countdown, and sometimes I get as far as hearing "General Quarters" before it dies. It dies so thoroughly that I have to kill the process using OS tools. Again, from looking at the forum postings, this problem is not specific to either my machine or even the Mac version. Neither of these problems requires any sort of PTS input; the software simply fails to work reliably.

Moving on to consulting experts before making a game change. Your QA and/or development teams should have a number of people who actively play the game with an eye to seeing what they can break, abuse, or otherwise use to an advantage. Having access to the source code and internal design details should make it easier for WG employees to think up exploits such as spamming the F key to save your fighters. You should also be able to test basic things like how well the AA works using simple math. Unless I'm missing something, it's virtually impossible to shoot down planes that are able to dodge flak bursts. The DPS for a T4 battleship such as the Wyoming seems to be a few hundred out to 3km if the wiki is to be believed, but that may be obsolete data. If correct, then against a Hosho sending in 3 Jean TBs, each with 930HP in a squadron of 3, gives roughly 2,800 HP. Since the aura damage is applied evenly to the entire squad then you need something like 15 seconds of continuous exposure to shoot down the squad, yet it takes no where near that long to cover the 6km of distance exposed to AA fire. Given that it's obviously possible to mostly dodge flak, somebody on the design and testing sides should have figured this out without needing live guinea pigs to demonstrate it.

Finally, I'm not convinced that even if you had 5,000 random potatoes playing on the PTS that you would have gotten the information you needed. My experience is that the vast majority of us, myself included, only partly understand the game mechanics, so we're not really able to think up exploits such as stealth torp drops followed by immediate retreat. Instead, we wait for the really good players, such as @Pulicat and @LoveBote to find them, at which point we copy with shameless abandon. So, what you really need is the basic development idea of finding your power users, convincing them via adequate compensation to really hammer on the new game, specifically looking for problems that they can abuse, and learn from that. You also probably need some clueless targets in a game for them to work over, but that's what your QA department can do. Three days of solid play-testing by real rock star players would have uncovered most of these problems prior to release.

So, I don't know what to expect from the forthcoming series of fixes, balancing changes, etc., but I have to say that the broken process that resulted in this release coming out in this state does not breed confidence that the fixes will be better, especially now that you're under greater time pressure than you were before shipping 8.0.

Edited by KaptainNemo_1
  • Cool 14

Share this post


Link to post
Share on other sites
88
[RLGN]
Beta Testers
100 posts
1,810 battles

I completely agree. Some of these issues could have been uncovered with applying some simple napkin math (torpedo spamming). They are either seriously understaffed, or lack pride in their product or respect for the player base.

Edited by Soulcaller
  • Cool 1

Share this post


Link to post
Share on other sites
848
[OPRAH]
Beta Testers
4,473 posts
14,393 battles

Agreed this should have never been put on the live server without sufficient testing rather than using the general game population as lab rats!

Share this post


Link to post
Share on other sites
3,254
[WOLF7]
Members
10,746 posts
3 minutes ago, CAPTMUDDXX said:

Agreed this should have never been put on the live server without sufficient testing rather than using the general game population as lab rats!

Have to admit it's pretty amusing that they put the Hak out in this configuration, and actually thought it would be balanced comrade....:Smile_teethhappy:

Share this post


Link to post
Share on other sites
848
[OPRAH]
Beta Testers
4,473 posts
14,393 battles

@awiggin yes it is as if they did not have a clue that some super fast fingered gaming vidiot was lurking in the shadows waiting to say Hey Yall Watch This Hold My Beer!

Share this post


Link to post
Share on other sites
2,092
[90TH]
[90TH]
Alpha Tester
5,960 posts
7,946 battles
19 minutes ago, KaptainNemo_1 said:

Three days of solid play-testing by real rock star players would have uncovered most of these problems prior to release.

Puli is a great player, I am quite honestly average. And I was heartily sick of the PTS testing, didn't even get online for the 3rd and final round. 

Besides this, most if not all the critcisms of the rework, that are being discussed, were underlined, highlighted, coated in reflective DANGER! warning stickers; and more, prior to release. Testers did do their jobs, they did raise concerns.

For whatever reason, (in part, for sure, their own dev planning and commercial reasons), the rework was released on to the Live server, as is.

  • Cool 1

Share this post


Link to post
Share on other sites
80
[P2W]
Members
126 posts
8,144 battles
2 minutes ago, LoveBote said:

Puli is a great player, I am quite honestly average. And I was heartily sick of the PTS testing, didn't even get online for it inthe 3rd and final round. 

Besides this, most if not all the critcisms of the rework, that are being discussed, were underlined, highlighted, coated in reflective DANGER! warning stickers; and more, prior to release. Testers did do their jobs, they did raise concerns.

For whatever reason, (in part, for sure, their own dev planning and commercial reasons), the rework was released on to the Live server, as is.

Thanks for the reply. I certainly didn't mean that you should be expected to do something you don't want to, but I believe that if they reached out to the top 5% of CV captains, as well as the top 5% of target captains and offered either real cash or substantial in-game rewards that they could have gotten a critical mass to do testing without relying on bots.

Share this post


Link to post
Share on other sites
350
[WOLF2]
Members
1,160 posts
8,217 battles

Damn nice write up, and much more eloquent and precise than I could have written.    And I wholly agree.

Unfortunately, many games are like this, and it's surprising to find how few recognize the common pitfalls.   In WOWS parlance, it's "potato" development.     

I think they do well in the P.R. department with the contributors and streamers.   They evangelize the product that gets us engaged.   From ship reviews, to how-to guides, etc....   I can't thank the contributors enough.

But when it comes to game play?   Why not incorporate some of that unicum experience and game knowledge into the requirements gathering?   And then into design?   

And if not in using them for development, WG could also conduct bug hunts/exploit hunts.     Then weigh every found exploit and determine if it has an impact.    If they need to conduct it on a test server, make it worth the tester's time.   Find a bug or exploit, get a reward.   Provide that feedback to the developers, who can then tune it down/eliminate the exploit prior to release.      

  • Cool 1

Share this post


Link to post
Share on other sites
2,092
[90TH]
[90TH]
Alpha Tester
5,960 posts
7,946 battles
Just now, KaptainNemo_1 said:

Thanks for the reply. I certainly didn't mean that you should be expected to do something you don't want to, but I believe that if they reached out to the top 5% of CV captains, as well as the top 5% of target captains and offered either real cash or substantial in-game rewards that they could have gotten a critical mass to do testing without relying on bots.

My suggestion, pre rework release, was that they employ the same method they used to test the waters for Submarines last year, introduce them in the Live server, but only in special scenarios (pvp and pve), keeping them out of randoms until bugs and issues sorted.

  • Cool 1

Share this post


Link to post
Share on other sites
Members
1,544 posts
13,172 battles

I have seen this a number of times, it is “ We have a deadline coming up, can’t delay any more without looking like idiots, know we have major problems  just put what we got into production and we will fix it later”.

How production turns into test, and have seen companies brought to their knees by these poor implementations.  

  • Cool 2

Share this post


Link to post
Share on other sites
117
[WPP1]
Beta Testers
395 posts
4,087 battles

I agree with all of this. I'm an IT ops guy who has worked in a few software companies, telecoms, and Internet companies. I wasn't a coder but I had to deal with internal and customer facing blowback.  When I worked for IT companies and telcos, a huge QA department was standard and code was tested for months. When it was released it was usually stable and good to go with a few minor issues. What I started seeing and hearing a few years ago when I got into a couple of software companies was a tendency to simply run some sort of burn in testing/basic functionality testing and release code into production calling it "tested". Production then served as the real test environment and, in my own experience, the devs didnt care because they could simply isolate half a server farm and roll back or hotfix, almost on the fly.  I lost count how many customer facing outages were caused by half baked software being thrown into production. This happened on a regular basis. I'm not sure if this has become SOP for software development but it seems to be the WG way. 

Share this post


Link to post
Share on other sites
425
[KERN]
Alpha Tester, In AlfaTesters
1,192 posts
5,037 battles

Don't hang this on the testers, we pointed all this crapout and TOLD THEM what would happen.  and got IGNORED.

 

I sent a message to WG BEGGING them NOT TO DO THIS, and was brushed off.  This is what THEY WANTED and now they have it.

 

Share this post


Link to post
Share on other sites
625
[-BRS-]
[-BRS-]
Members
2,045 posts
14,749 battles

Really sucks that they blew it but I'm sure they're doing their best to get it right

Edited by silverdahc

Share this post


Link to post
Share on other sites
539
[MIA]
Beta Testers, In AlfaTesters
2,290 posts
7,991 battles

you are misinformed. We've had plenty of testing on PTS, but PTS is not the same as live. It's a completely different experience. You don't have the same number of players with the same captain skills playing the same ships. Obviously balanced for PTS is not the same as balanced for live because of this.

Share this post


Link to post
Share on other sites
425
[KERN]
Alpha Tester, In AlfaTesters
1,192 posts
5,037 battles
31 minutes ago, NeoRussia said:

you are misinformed. We've had plenty of testing on PTS, but PTS is not the same as live. It's a completely different experience. You don't have the same number of players with the same captain skills playing the same ships. Obviously balanced for PTS is not the same as balanced for live because of this.

Balance isn't the issue.  There were obvious abusable exploits.

 

Which is why people in random are [edited] about OP CVs and people in coop are [edited] about grossly under powered CVs.  If you play them without the exploits... they don't work well at all.  If you play using them, it's pretty obvious why they're called exploits.

Share this post


Link to post
Share on other sites
539
[MIA]
Beta Testers, In AlfaTesters
2,290 posts
7,991 battles
16 minutes ago, thegreenbaron said:

Balance isn't the issue.  There were obvious abusable exploits.

 

Which is why people in random are [edited] about OP CVs and people in coop are [edited] about grossly under powered CVs.  If you play them without the exploits... they don't work well at all.  If you play using them, it's pretty obvious why they're called exploits.

These are not exploits but in-game mechanics. Pressing F to recall your squadron is an intended feature. Only people with sore butts about carriers call it an exploit.

  • Bad 1

Share this post


Link to post
Share on other sites
777
[WOLF6]
Members
2,299 posts
5,124 battles
8 hours ago, CAPTMUDDXX said:

Agreed this should have never been put on the live server without sufficient testing rather than using the general game population as lab rats!

AND...we paid for the “privilege”!

Share this post


Link to post
Share on other sites
610
[LUCK]
Members
1,616 posts
21,616 battles
27 minutes ago, NeoRussia said:

These are not exploits but in-game mechanics. Pressing F to recall your squadron is an intended feature. Only people with sore butts about carriers call it an exploit.

You're an [edited]clown

  • Cool 1

Share this post


Link to post
Share on other sites
80
[P2W]
Members
126 posts
8,144 battles
2 hours ago, NeoRussia said:

you are misinformed. We've had plenty of testing on PTS, but PTS is not the same as live. It's a completely different experience. You don't have the same number of players with the same captain skills playing the same ships. Obviously balanced for PTS is not the same as balanced for live because of this.

I have time to write this because the game just locked up right after General Quarters; this is about the fifth time for that. Live server not needed to find this bug. 

Share this post


Link to post
Share on other sites
425
[KERN]
Alpha Tester, In AlfaTesters
1,192 posts
5,037 battles
2 hours ago, NeoRussia said:

These are not exploits but in-game mechanics. Pressing F to recall your squadron is an intended feature. Only people with sore butts about carriers call it an exploit.

Let me just ask, what other game mechanic allows you to have a 500k damage+ match at no risk to yourself?  Go on, I'm waiting.

Share this post


Link to post
Share on other sites
905
[ARS]
Beta Testers
2,594 posts
2,426 battles
10 hours ago, KaptainNemo_1 said:

so we're not really able to think up exploits such as stealth torp drops followed by immediate retreat.

Mostly agree, but note that this is intended, not an exploit.  Hak is supposed to be able to stealth torp.  What it isn't supposed to do is saturate the enemy with quasi aimed torpedoes like iChase demonstrated.  iChase didn't even have Concealment Expert on that build.

Share this post


Link to post
Share on other sites
Sign in to follow this  

  • Recently Browsing   0 members

    No registered users viewing this page.

×