Common Distributions

Recently, I am enrolled in an open course about Bayesian Statistics which provides a note on common distributions. I quickly reviewed them and wrote this post for summarization.

Discrete

Uniform Distribution

We randomly pick \(1, 2, \ldots, N\) each with a probability \(\frac{1}{N}\). It is easy to show the mean of this distribution is \(E[X]=\frac{\frac{1}{2}(1+N)N}{N}=\frac{N+1}{2}\). As for the variance (i.e., \(Var[X]=E[X^2]-E[X]^2\)), the first term needs to solve \(\sum_{x=1}^{N}x^2\) which seems not that trivial. Luckily, Khan provides a video showing how to solve such kind of number series with the core idea of Calculus. For \(f_n=\sum_{x=1}^{N}x^2\), \(f_{0}=0, f_{1}=1, f_{2}=5, f_{3}=14, f_{4}=30\), their differences are \(f_{1}-f_{0}=1, f_{2}-f_{1}=4, f_{3}-f_{2}=9, f_{4}-f_{3}=16\), then their differences between differences are \(3, 5, 7\). Eventually, the differences are a constant of \(2\). Regarding the difference as a derivative, the closed-form solution of \(f_x\) should be \(f_{x}=Ax^3+Bx^2+Cx+D\) whose third-order derivative is constant (actually, here we know \(A=\frac{1}{3}\) directly from the constant \(2\)). Solving an linear equation of three individual \(f_{x},x=1,2,3\) equations leads to the final results \(Var[X]=\frac{N^{2}-1}{12}\).

Binomial Distribution

Considering \(Y\sim Binom(n,p)\), this is just the sum of \(n\) i.i.d. Bernoulli random variables. For simplicity, let’s consider two RVs \(X, Y\) and solve the relationship between the mean/variance of their sum and the mean/variance of themselves. First, \(\begin{align*} E[X+Y]&=\sum_{x}\sum_{y}p(x,y)(x+y)=\sum_{x}x\sum_{y}p(x,y)+\sum_{y}y\sum_{x}p(x,y) \\ &=\sum_{x}xp(x)+\sum_{y}yp(y)=E[X]+E[Y] \\ \end{align*}\).

As you can see, other variables in the joint distribution are marginalized without any independence assumption. By definition: \(\begin{align*} Var[X+Y]&=E[((X+Y)-E[X+Y])^2]=\sum_{x}\sum_{y}p(x,y)((x+y)-E[X+Y])^{2} \\ &=\sum_{x}\sum_{y}p(x,y)((x-E[X])+(y-E[y]))^{2} \\ &=Var[X]+Var[Y]+\sum_{x}\sum_{y}2p(x,y)(x-E[X])(y-E[Y]) \\ \end{align*}\)

If \(X\) and \(Y\) are independent, say that \(p(x,y)=p(x)p(y)\), the last term can be wrote as \(2\sum_{x}p(x)(x-E[X])\sum_{y}p(y)(y-E[Y])=2\times 0\times 0=0\). Thus, \(Var[X+Y]=Var[X]+Var[Y]\) if \(X\) and \(Y\) are independent. The above analysis can be easily extend to the case of arbitrary number of variables. According to the above analysis, the mean and variance of \(Y\sim Binom(n,p)\) is \(np\) and \(np(1-p)\).

Poison Distribution

First, the rate \(\lambda$ of $X\sim Poison(\lambda)=\frac{\lambda^{x}\exp{-\lambda}}{x!}\) should be the rate of duration in question (see the example in that course material). The mean and variance can be proved with a commonly used math trick as follow.

\[\begin{align*} E[X]&=\sum_{x=0}^{\inf}\frac{x\lambda^{x}\exp(-\lambda)}{x!} \\ &=\sum_{x=0}^{\inf}\frac{x\lambda^{x}\exp(-\lambda)}{x!} \\ &=\lambda\sum_{x=1}^{\inf}\frac{\lambda^{x-1}\exp(-\lambda)}{(x-1)!} \\ &=\lambda\sum_{x=0}^{\inf}\frac{\lambda^{x}\exp(-\lambda)}{x!}=\lambda \\ \end{align*}\]

The last step is valid since the summation is exactly the pdf of the Poison distribution and thus sum up to 1. The variance is induced in a similar way.

\[\begin{align*} E[X^2]&=\sum_{x=0}^{\inf}\frac{x^{2}\lambda^{x}\exp(-\lambda)}{x!} \\ &=\lambda\sum_{x=1}^{\inf}\frac{x\lambda^{x-1}\exp(-\lambda)}{(x-1)!} \\ &=\lambda\sum_{x=0}^{\inf}\frac{(x+1)\lambda^{x}\exp(-\lambda)}{(x)!} \\ &=\lambda^2+\lambda \\ \end{align*}\]

Based on above proofs, both the mean and the variance of Poison distribution equal \(\lambda\).

(Continuing…)