Jump to content


Vak's Fun Times With Stats

stats

  • Please log in to reply
9 replies to this topic

vak_ #1 Posted 05 May 2017 - 03:05 AM

    Captain

  • Beta Testers

  • 4,864
  • Member since:
    09-28-2012

Hey there. For quite some time now I've wanted to create a quick and simple go-to guide for stats. If you ever wondered how can personal skill affect win rate when there are 23 other players in the game, or suspected that your long string of losses cannot possibly be random, or were confused by people throwing around terms like "sigma" or "normal distribution" in threads about battleship dispersion -- this thread is for you!

 

I promise not to bore you with complicated maths. Statistics can get very technical, and I shudder simply thinking about my stats notebook from grad school, with pages upon pages of double integrals and all kinds of other crap that I didn't completely understand even back then, and sure as heck don't remember now. However, at their basic core, statistics and probability concepts are fairly intuitive, and are not hard to explain.

 

I already have three posts about stats in mind.

 

In the first one, I'll talk about the normal distribution and standard deviation -- trust me, these things aren't nearly as hard as some people make them out to be! There is a lot of talk about battleship dispersion lately, and it might be useful to  have a basic understanding of how shell dispersion is described statistically in real life (and is most likely modeled in game)

 

For the second post, I want to list some of the common statistical myths that I often see on the forums (some of them are mentioned in the beginning of the first paragraph), and discuss them in some detail.

 

Third post will discuss statistical sampling of various player populations; given how often people complain about this or that ship being OP based on the WarShipsToday stats, this might be of some use.

 

Maybe i'll think of more parts in the future, we shall see. Anyways, without further ado...



vak_ #2 Posted 05 May 2017 - 03:06 AM

    Captain

  • Beta Testers

  • 4,864
  • Member since:
    09-28-2012

Part 1: Normal distribution and its standard deviation

 

Normal distribution is one of the most widely used distributions in statistics (there are others, but we don't really need to discuss them). A picture is worth a thousand words, so here is our culprit:

 

 

You might say "gah, it's just some line, so what?". But it's not just any line; this is a line that for whatever magical reason describes all kinds of random distributions in the world around you. Intuitively you already know this from experience; think of people's height, for example. Most guys you know are probably around 5'7"-6'0", give or take, right? You also might know some guys that are shorter or taller than that, but not quite as many. And then of course there the extremely tall (think NBA players) or short guys, but they are few and far in-between. Researchers have collected heights for very large groups of people, and that data is described very nicely by a normal distribution:

 

 

But hey, who cares about height. Let's discuss something  that's much more relatable to battleship dispersion in this game: artillery fire! There is a handy USMC Field Manual (FM-40) out there, called Tactics, Techniques, and Procedures for Field Artillery Manual Cannon Gunnery. Below is a very handy illustration from Section 3-5. Causes of Dispersion, that shows the schematic distribution of shell bursts of "ammunition of the same caliber, lot, and charge that are fired from the same position with identical settings used for deflection and quadrant elevation":

 

 

As you can see, the rounds will not all impact in a single point but will fall in a scattered pattern, with most bursts being close to the point of aim, and few of them further away. This scattering of bursts is caused by all kinds of stuff: minor variations in the weight of the projectile, form of the rotating band, moisture content and temperature of the propellant grains, differences in the rate of ignition of the propellant, variations in the temperature of the bore from round to round, conditions of the carriage, variations in air resistance due to wind, etc. But the cool thing is, that the combination of these random factors still produces a burst pattern around the mean point of impact, that -- you guessed it! -- is described very well by a normal distribution curve. Well, to be more precise, by two curves. For each shell burst we can measure the perpendicular distance to the mean range line (Y axis), and to the line of fire (X axis). This way, we can describe all the bursts as two columns of values, one for perpendicular distance to X, and one for perpendicular distance to Y:

 

 

Here is another illustration of how that works:

I must stress, this is a probability curve -- if we start shooting actual shells, and then plot the percent of shells from total shells fired vs. burst distance from the aim point for a given distance, the result won't match the normal distribution curve exactly. Theoretically, if we would fire an infinite number of shells it would match ideally, but who's got time for that! Anyways, for all intents and purposes the theoretical normal distribution curve describes the actual real-life shell distribution quite well, which is why it's in this Field Manual to begin with.

 

Now, let's talk about the normal distribution curve itself in more detail. It's got a few interesting properties that must be kept in mind. For one, the combined area under it, that is the sum of all possible probabilities of all possible variable variations must add up to one. A simple illustration of this concept (though not directly related to normal curves) is this: if you flip a coin a bunch of times , making sure it lands flat every time, and then divide the number of flips by the sum of heads and tails, you'll get one.

 

A second important detail: this curve is symmetrical. If you draw a line down the middle, the shape of the curve to the left and to the right of that line will be exactly same.

 

Another much more nuanced detail is the exact shape of the curve. Here is a handy illustration:

 

 

The character μ simply denotes the average of all data points (for example, the average value for all shell burst distances from either X or Y axis that we've computed above). Character σ is sigma, or standard deviation of the normal curve. It's not all that important how it is computed (link for the curious). What's important is the fact that this value quantitatively describes the expected distribution of the observations, e.g. shell bursts. Let's say that we accurately recorded all our bursts, and measured the perpendicular distance to the mean range line for each one. Based on those measurements we've calculated sigma. For the sake of argument, let's say it's one hundred meters. The graph above tells us that we can expect about 68% of shell bursts to be within 100 meters of the mean range line, 95% of shell bursts within 200 meters of the that line, and 99% of shell bursts to be within 300 meters of the line. This is an extremely powerful concept: once we compute our sigma (standard deviation), we can mathematically predict the approximate probability of shell landing within any given distance from the aim line! And if we have sigma for both X and Y axis, then we will know the probability of landing a shell at any given distance away from the aim point. You can imagine how important this is for artillery fire.

 

It should be noted that this sigma we've computed must be a physical value, it must have units of distance. If you tell me "68% of shells will land within 100 of the mean range line", that tells me nothing. One hundred what? Meters? Inches? Football fields?

 

Okay, so now we know that the normal curve has a very unique shape, and that the sigma (standard deviation) value gives us a quantitative way of estimating the distance to the center of the curve. What is the effect of changing sigma, how will that change the way our distribution looks? In the very first graph of this post, the sigma is equal to ten. Let's see how the graph will look if we change sigma to 5 and to 20, leaving all else equal:

 

Increasing sigma (standard deviation) makes our distribution wider and shorter. Shells, on average, will be landing further away from the aim point.

 

Decreasing sigma (standard deviation) makes our distribution narrower and taller. Shells, on average, will be landing closer to the aim point.

 

Keep in mind that the area under the curve is still one for all these examples, which is why the height and width of the curves change in unison. Also, keep in mind that regardless of the exact shape of the normal distribution, each sigma will still give us a quantitative way to measure probability of some observation falling within a certain distance away from the center of the distribution (68% - 95% - 99% for multiples of one, two, and three sigmas respectively).

 

Incidentally, this means that the WG "sigma value" is not the same as sigma (standard deviation) in statistics that I've described above. From patch notes we know that increasing that "sigma value" makes guns more accurate. That means that the shell distribution around the aim point is more narrow. That means that sigma (standard deviation) has decreased! I have a few hypotheses as to what exactly that "sigma value" is and how it's related to the actual sigma (standard deviation), but I'll share them in a different thread a bit later.

 

 

Well, this concludes the first part of Vak's Fun Times With Stats installment! Hopefully this was useful. Stay tuned for the second part about WoWS statistical fallacies! :)



Wulfgarn #3 Posted 05 May 2017 - 05:42 AM

    Lieutenant Commander

  • Members

  • 2,743
  • Member since:
    09-15-2015
http://forum.worldof...__fromsearch__1

Highest. CV USN Independence BB USN Colorado CA USN New Orleans GER Yorck  RN Leander DD USN Gearing IJN Shimakaze VMF Khab

Premium. Kamikaze R. Blyskawica. Leningrad. Belfast. Graf Spee, Konig Albert. Ishizuchi. Marblehead. Texas. Arizona. Sims. Atlanta. Indianapolis. Kutuzov. Tirpitz. Gremyashchy. Murmansk. Molotov. Prinz Eugen. Scharnhorst. Warspite. Anshan. Lo Yang
Intel I5-3570K @ 3.4GHz . GTX 750 Ti (Max settings). 16 GB Ram . Win 7 Pro 64-bit . SSD 120GB (OS only) . SSD 250GB (Gaming) . Mechanical 2TB (Storage)

 


mmfullen #4 Posted 05 May 2017 - 09:29 AM

    Master Chief Petty Officer

  • Members

  • 392
  • Member since:
    05-17-2016

View PostWulfgarn, on 05 May 2017 - 12:42 AM, said:

 

Considering it was some of Vak's data used in that video, I'm sure he's well aware of its existence. 

[-K-] Kraken


vak_ #5 Posted 05 May 2017 - 01:28 PM

    Captain

  • Beta Testers

  • 4,864
  • Member since:
    09-28-2012

View PostWulfgarn, on 05 May 2017 - 05:42 AM, said:

 

As mmfullen pointed out, I'm quite aware of that video :)

 

Time permitting, I'll make a thread about dispersion over the weekend, where I'll discuss my current understanding of the dispersion model, knowledge gaps that we as the community have, few hypotheses regarding its workings, and suggestions of how we the players can find out more from analyzing empirical game data. In fact, this is the reason why I wrote the post above -- one will need to understand what normal distribution and standard deviation is in order to follow what I'll write in that thread. 



vak_ #6 Posted 15 May 2017 - 05:17 AM

    Captain

  • Beta Testers

  • 4,864
  • Member since:
    09-28-2012

Part 2: Perception Fallacies

 

Time to continue my stats installment. In this post I wanted to discuss two of the extremely typical fallacies that I see on the forums.

 

Selective Attention & Confirmation Biases

 

These are all real claims that I have seen on the forums at one point or another

 

"After the recent patch, I miss much more often. That must mean that WG changed the dispersion, but didn't tell anybody!"

"Whenever I buy premium time, I never detonate. WG must rig the detonation chances when you pay them money!"

"Warspite is set on fire much more often than Arizona!"

 

And so on, and so forth. You get the picture. But there are two very important things to keep in mind when you read these claims. Firstly, human memory is not perfect. We as species have a tendency to pay attention to some things while simultaneously ignoring others. Here is a great video that demonstrates this effect. It's not long (just a minute and a half), and is fairly amusing -- or at least it was for me, when I first saw it:

 

 

Okay, so human memory isn't perfect, our brain elect to retain only some of the information that they receive from our sensory organs. But what's even worse (well, at least in the context of this discussion), is the fact that humans also tend to favoring information that confirms previously existing beliefs or biases! For example, consider a person that believes that Warspite catches fire much, much easier than other tier 6 battleships. That person will keep a close track of instances when they were playing Warspite, and it caught on fire. Yet they might quickly forget the times when they were playing some other tier 6 BBs and got set on fire with only a few shells, or times when their Warspite ate dozens of HE shells, and didn't catch fire. Their memory will only retain times when their Warspite was easily set on fire, and when they discuss her on forums they'll state "Warspite catches on fire easily" as fact. 

 

This might seem like a very obvious thing when I put it that way -- and yet, as I've mentioned, this fallacy is extremely prevalent on the forums, especially after new patches come out. So, the next time when you see someone making a claim like that, simply ask them -- how do they know this isn't their memory playing tricks on them? Did they actually keep records of whatever it is they are claiming, for example the number of fires on Warspite vs. the number of enemy shell hits in our hypothetical example? Of course, almost always the answer is going to be no.

 

Clustering Illusion

 

"I lost fifteen games in a row, and this isn't some confirmation bias: the results are right there in the client There is NO WAY that occurred randomly, random matchmaker would produce both wins and losses!"

 

Do this for me: take a coin, and flip it twenty times, recording the result after each flip. I did this very thing as I was writing the post, and here are my results:

 

 

Note how there was a streak when I got three tails in a row, and then three heads in a row immediately after. Now, we know that the probability of heads or tails in each flip is 50%. Does that suggest that my coin throws were somehow "rigged", that "true" results should have looked something like this

 

 

Of course not! The first image is just an example of inevitable streaks arising in small samples from random distributions. In the long run, a very large amount of coin flips will yield 50% heads and 50% tails (assuming a fair flip and a fair coin). But in any short run, a wide variety of probabilities are expected, as you have just probably confirmed by flipping your coin.

 

Now imagine a hundred thousand people flipping a coin one hundred times each. Do you think that one or two of them might get, for example, fifteen tails in a row? Sure, intuitively that should seem possible, even probable (there are mathematical methods to calculate the precise probability of this happening, but we don't need to get into that).

 

However, to a person that got fifteen tails in a row it might not seem random. He might think there is something going on, simply because our brains are wired to look for patterns. That person might go on the coin flipping forum and create a thread, in which he or she will claim that perhaps their particular coin isn't a "fair" one, that it tends to produce a lot of consecutive tail flips. Maybe even they'll have a "theory", for example that flipping a coin at a certain time of the week will generate more consecutive tails. Naturally, you should now see the random coincidence for what it really is, and be able to explain that person that he or she's simply fallen victim to a clustering illusion!

 

 

That's it for now. In the next installment I'll discuss why win rate is anything but random.



Herr_Reitz #7 Posted 15 May 2017 - 11:36 AM

    Captain

  • Beta Testers

  • 5,742
  • Member since:
    08-18-2014

Gosh where to start.... you can rig coin tosses by always having it heads up or tails up then flipping it so it tumbles two or three complete times. You just messed with "odds" by modifying the basic nature of the test. 

 

The "reason" things in "real life" fall the way they do has to do with the infinite number of potential outcomes for an event. This is a game, a model, that has a finite number of outcomes. The injection of the oft-discussed random number generator is a meek attempt to insert random outcomes deliberately but in the end, that too is finite. If one truly believed the outcome of events in this "game" could convincingly fit into the wonderful world of "real life statistics" then ask WoWS to remove RNG completely. Then the random events that take place in the real world would have a real effect upon the game and the players within the game. 

 

However that would not happen, would it? If you say it would then why is "rng" present? It is needed to force the game to appear random enough to fit the models you are putting before us. 

 

It's a game. It is not real life. They can force the game to do whatever they want "statistically" by tweaking "game balance mechanics".

 

By the way... replace that artillery field manual discussion with laser weapons. Want to bet the dispersion would becomes so minimal as to be practically a moot point? Most likely the target groups would be so tight that any dispersion would in effect be no dispersion at all, practically speaking. 

 

:popcorn:



vak_ #8 Posted 16 May 2017 - 01:54 AM

    Captain

  • Beta Testers

  • 4,864
  • Member since:
    09-28-2012

View PostHerr_Reitz, on 15 May 2017 - 11:36 AM, said:

you can rig coin tosses

 

Were you under the false impression that I was trying to prove that the coin toss isn't rigged? That isn't so. You can't prove an absence of something, scientific method doesn't work that way; the hypothesis has to be falsifiable.

 

I was merely demonstrating that the string of losses that people perceive to be a proof of MM rigging can occur randomly. Perhaps I should have a short post on the scientific method, Russel's teapot, the whole nine yards...

 

View PostHerr_Reitz, on 15 May 2017 - 11:36 AM, said:

You just messed with "odds"

 

Note the part where I say "In the long run, a very large amount of coin flips will yield 50% heads and 50% tails (assuming a fair flip and a fair coin)".

 

And by the way, there are statistical tests to evaluate coin's "fairness". I'll get into that in my next post, when we discuss win rate (because "fair coin" test is actually a very convincing argument against the complete randomness of win rate).

 

View PostHerr_Reitz, on 15 May 2017 - 11:36 AM, said:

The "reason" things in "real life" fall the way they do has to do with the infinite number of potential outcomes for an event. This is a game, a model

 

Not to get into a scholastic debate, but what you call "real life" is also merely a model that humans created to represent reality. Physics, chemistry, biology -- these are all model frameworks, theories that are based on existing observation and do a decent job of predicting new observations. They are not the same thing as reality, as should be obvious from the fact that certain scientific theories are proven wrong from time to time. In fact even your own personal perception of "reality" is also a model that's constructed by the extremely complex neural network inside your noggin.

 

Now, within the context of these models are there events that do not have an infinite number of outcomes? Of course. Quantum spin number of an electron in physics, for example.

 

View PostHerr_Reitz, on 15 May 2017 - 11:36 AM, said:

The injection of the oft-discussed random number generator is a meek attempt to insert random outcomes deliberately but in the end, that too is finite

 

Yes, the real-life dispersion is has a continuous quality (unless we start getting into weird quantum spacetime stuff), and in the game it is discrete just because of the way our present-day computers work. But so what? I don't understand what does this have to do with anything. Are you saying that the game's RNG isn't truly random? I don't know all that much about actual RNG algorithms, but I know enough to realize that even deterministic random bit generators can create statistical randomness. What more would you want from dispersion RNG?



TheKrimzonDemon #9 Posted 17 May 2017 - 04:50 PM

    Commander

  • Members

  • 3,419
  • Member since:
    10-22-2015

View Postvak_, on 15 May 2017 - 08:54 PM, said:

 

Were you under the false impression that I was trying to prove that the coin toss isn't rigged? That isn't so. You can't prove an absence of something, scientific method doesn't work that way; the hypothesis has to be falsifiable.

 

I was merely demonstrating that the string of losses that people perceive to be a proof of MM rigging can occur randomly. Perhaps I should have a short post on the scientific method, Russel's teapot, the whole nine yards...

 

 

Note the part where I say "In the long run, a very large amount of coin flips will yield 50% heads and 50% tails (assuming a fair flip and a fair coin)".

 

And by the way, there are statistical tests to evaluate coin's "fairness". I'll get into that in my next post, when we discuss win rate (because "fair coin" test is actually a very convincing argument against the complete randomness of win rate).

 

 

Not to get into a scholastic debate, but what you call "real life" is also merely a model that humans created to represent reality. Physics, chemistry, biology -- these are all model frameworks, theories that are based on existing observation and do a decent job of predicting new observations. They are not the same thing as reality, as should be obvious from the fact that certain scientific theories are proven wrong from time to time. In fact even your own personal perception of "reality" is also a model that's constructed by the extremely complex neural network inside your noggin.

 

Now, within the context of these models are there events that do not have an infinite number of outcomes? Of course. Quantum spin number of an electron in physics, for example.

 

 

Yes, the real-life dispersion is has a continuous quality (unless we start getting into weird quantum spacetime stuff), and in the game it is discrete just because of the way our present-day computers work. But so what? I don't understand what does this have to do with anything. Are you saying that the game's RNG isn't truly random? I don't know all that much about actual RNG algorithms, but I know enough to realize that even deterministic random bit generators can create statistical randomness. What more would you want from dispersion RNG?

 

He is in fact saying the game's RNG isn't actually random, and he's correct. Detonations are supposed to be 100% random, for instance, yet they happen mostly to destroyers. That's not random. Detonations also happen at the godawfulest of times, match after match, no matter the player nor the ship. In that case, it's the timing of the supposed random event that isn't random. In most cases, the detonation happens to the team that is winning. Those things, alone, show that the RNG isn't random, but wait, it gets better: Ever fire your main guns and watched your shells go straight up, sideways, or down? Ever watch "dispersion" give you 2-4 full km more of firing range? Now, notice when it happens. It isn't random, something in the match caused it to kick in and make your shells visit Mars.

 

This game's RNG is built according to what WG and the game's devs think a match should look like, how it should proceed, and how quickly it should progress, therefore, it is anything BUT random, these things are built in to happen at specific times, under specific circumstances. That is the total, complete, unadulterated opposite of "random."


Edited by TheKrimzonDemon, 17 May 2017 - 04:51 PM.

I drive ships. That's right, drive. I do not sail them. They don't have sails.


vak_ #10 Posted 17 May 2017 - 07:22 PM

    Captain

  • Beta Testers

  • 4,864
  • Member since:
    09-28-2012

View PostTheKrimzonDemon, on 17 May 2017 - 04:50 PM, said:

He is in fact saying the game's RNG isn't actually random

 

Firstly, for the sake of correctness: while WoWS most likely uses a deterministic random bit generator (DRBG), he can't know that for sure unless he's seen the source code.

Secondly, DRBGs are more than capable of creating statistical randomness.

 

View PostTheKrimzonDemon, on 17 May 2017 - 04:50 PM, said:

Detonations are supposed to be 100% random, for instance, yet they happen mostly to destroyers

 

That's a pretty weak straw man. There are plenty of non-random factors at work -- magazine size relative to the ship and magazine placement (how hard is it to hit), magazine protection (how likely is it to be damaged by an incoming shell or splash), and so forth. DDs have bad armor and relatively large magazines, why would it be surprising that they detonate more often?

 

View PostTheKrimzonDemon, on 17 May 2017 - 04:50 PM, said:

Detonations also happen at the godawfulest of times, match after match

 

I don't understand your argument. You're saying that detonations aren't random because they are not spread out evenly throughout games? Re-read what I write about the clustering illusion above. If you still don't get something tell me, I'll try to do a better job of explaining it (though I thought anybody should be able to intuitively understand the coin flipping example).

 

View PostTheKrimzonDemon, on 17 May 2017 - 04:50 PM, said:

In most cases, the detonation happens to the team that is winning

 

Do you have an objective proof of that? If not, re-read what I wrote about selective attention bias and confirmation bias. Tell me if you don't get something.







Also tagged with stats

1 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users