CHAPTER 1: FOUNDATIONS OF PROBABILITY AND RANDOM VARIABLES
1.1 Introduction to Probability Theory
Probability theory is the mathematical framework used to analyze uncertainty and randomness. It provides methods for describing uncertain events, modelling random phenomena, and making predictions based on observed or theoretical information.
Many real-world systems contain uncertainty:
- The number of customers arriving at a service centre.
- The number of insurance claims received in a year.
- The number of genetic mutations occurring in a population.
- The number of failures occurring in a computer network.
Probability theory converts these uncertain outcomes into mathematical structures that can be analysed.
A probability model consists of three fundamental components:
- A sample space.
- Events.
- A probability measure.
1.2 Sample Spaces and Events
Definition 1.1: Sample Space
The sample space, denoted by (S) or (\Omega), is the set of all possible outcomes of a random experiment.
For example, when tossing a fair coin:
[ S={H,T} ]
where:
- (H) represents Heads.
- (T) represents Tails.
For rolling a six-sided die:
[ S={1,2,3,4,5,6} ]
The individual outcomes inside a sample space are called sample points.
Definition 1.2: Event
An event is a subset of the sample space containing outcomes of interest.
For example, when rolling a die:
[ S={1,2,3,4,5,6} ]
The event of obtaining an even number is:
[ A={2,4,6} ]
An event occurs if the observed outcome belongs to the event set.
1.3 Probability Measures
A probability measure assigns numerical values to events. The probability of an event (A) is written as:
[ P(A) ]
and satisfies three fundamental axioms.
Axiom 1: Non-negativity
For every event:
[ P(A)\geq0 ]
Probabilities cannot be negative.
Axiom 2: Total Probability
The probability of the entire sample space is:
[ P(S)=1 ]
This represents certainty.
Axiom 3: Additivity
For mutually exclusive events:
[ P(A\cap B)=0 ]
the probability of their union is:
[ P(A\cup B)=P(A)+P(B) ]
These axioms form the mathematical foundation of probability theory.
1.4 Random Variables
Definition 1.3: Random Variable
A random variable is a mathematical function that assigns a numerical value to every outcome in a sample space.
It is commonly represented by:
[ X:S\rightarrow \mathbb{R} ]
where:
- (S) is the sample space.
- (X) is the random variable.
- (\mathbb{R}) represents the real numbers.
Random variables allow probability problems to be expressed mathematically.
Example 1.1: Number of Heads
Suppose three coins are tossed.
The sample space is:
[ S={HHH,HHT,HTH,THH,HTT,THT,TTH,TTT} ]
Define:
[ X=\text{number of heads obtained} ]
The possible values are:
[ X\in{0,1,2,3} ]
Therefore, (X) is a discrete random variable.
1.5 Types of Random Variables
Random variables are classified into two major categories:
- Discrete random variables.
- Continuous random variables.
1.5.1 Discrete Random Variables
A discrete random variable takes countable values.
Examples:
- Number of children in a family.
- Number of customers arriving in an hour.
- Number of defective products.
The possible values can be listed:
[ X=0,1,2,3,\ldots ]
A discrete random variable is described by a probability mass function.
Definition 1.4: Probability Mass Function (PMF)
The probability mass function gives the probability that a discrete random variable takes a particular value.
It is defined as:
[ p(x)=P(X=x) ]
The PMF satisfies:
[ p(x)\geq0 ]
and:
[ \sum_x p(x)=1 ]
Example 1.2: Simple Discrete Distribution
Consider a random variable (X) with:
[ P(X=0)=0.2 ]
[ P(X=1)=0.5 ]
[ P(X=2)=0.3 ]
The probability table is:
| (x) | (P(X=x)) | | --- | -------- | | 0 | 0.2 | | 1 | 0.5 | | 2 | 0.3 |
Checking:
[ 0.2+0.5+0.3=1 ]
Therefore, this is a valid probability distribution.
1.6 Continuous Random Variables
A continuous random variable can take any value within an interval.
Examples:
- Height of a person.
- Temperature.
- Waiting time.
- Measurement errors.
Continuous random variables are described using probability density functions.
Definition 1.5: Probability Density Function
The probability density function (PDF) is denoted by:
[ f(x) ]
and satisfies:
[ f(x)\geq0 ]
with:
[ \int_{-\infty}^{\infty}f(x),dx=1 ]
The probability that (X) lies between (a) and (b) is:
[ P(a<X<b)=\int_a^b f(x),dx ]
1.7 Expectation and Moments
The expected value describes the average long-term value of a random variable.
For a discrete random variable:
[ E(X)=\sum_x xp(x) ]
For a continuous random variable:
[ E(X)=\int_{-\infty}^{\infty}xf(x),dx ]
The expected value is also called the first moment.
Example 1.3: Expected Value
Given:
[ X={0,1,2} ]
with probabilities:
[ P(X=0)=0.2 ]
[ P(X=1)=0.5 ]
[ P(X=2)=0.3 ]
Then:
[ E(X)=0(0.2)+1(0.5)+2(0.3) ]
[ E(X)=0+0.5+0.6 ]
[ E(X)=1.1 ]
The average value of (X) is therefore:
[ \boxed{E(X)=1.1} ]
1.8 Variance and Standard Deviation
Expectation measures the centre of a distribution, while variance measures its spread.
The variance is:
[ Var(X)=E[(X-\mu)^2] ]
where:
[ \mu=E(X) ]
An equivalent computational formula is:
[ Var(X)=E(X^2)-[E(X)]^2 ]
The standard deviation is:
[ \sigma=\sqrt{Var(X)} ]
1.9 Generating Functions: An Introduction
Generating functions transform probability distributions into algebraic functions.
They provide a powerful method for:
- Finding moments.
- Studying sums of random variables.
- Analysing branching processes.
- Solving queueing models.
For discrete random variables, the most important generating function is the Probability Generating Function (PGF).
1.10 Introduction to Probability Generating Functions
Let (X) be a discrete random variable taking values:
[ 0,1,2,\ldots ]
The Probability Generating Function is defined as:
[ G_X(t)=E(t^X) ]
Expanding the expectation:
[ G_X(t)=\sum_{x=0}^{\infty}P(X=x)t^x ]
where:
- (P(X=x)) is the probability coefficient.
- (t^x) is the generating power.
The PGF stores the entire probability distribution inside a single function.
Example 1.4: Constructing a PGF
Suppose:
[ P(X=0)=0.4 ]
[ P(X=1)=0.3 ]
[ P(X=2)=0.3 ]
The PGF is:
[ G_X(t)=0.4t^0+0.3t^1+0.3t^2 ]
Therefore:
[ \boxed{G_X(t)=0.4+0.3t+0.3t^2} ]
The coefficients represent probabilities, while the powers represent possible values of the random variable.
1.11 Importance of Probability Generating Functions
PGFs are important because they convert difficult probability calculations into simpler algebraic operations.
Major applications include:
- Finding probability distributions.
- Computing expected values and variances.
- Analysing sums of independent random variables.
- Studying branching processes.
- Modelling queues and population systems.
The following chapters develop the mathematical theory of PGFs, including proofs, applications, examples, and advanced stochastic models.
References
Feller, W. (1968). An Introduction to Probability Theory and Its Applications, Volume 1 (3rd ed.). Wiley.
Grimmett, G., & Stirzaker, D. (2020). Probability and Random Processes (4th ed.). Oxford University Press.
Ross, S. M. (2014). Introduction to Probability Models (11th ed.). Academic Press.
Billingsley, P. (2012). Probability and Measure. Wiley.
CHAPTER 2: DISCRETE PROBABILITY DISTRIBUTIONS AND GENERATING FUNCTIONS
2.1 Introduction to Discrete Probability Models
A large class of random phenomena can be represented by variables that take only separate, countable values. These variables are known as discrete random variables.
Examples include:
- Number of customers arriving at a bank.
- Number of defective items in a production batch.
- Number of accidents occurring within a year.
- Number of offspring produced by an organism.
- Number of failures in a communication system.
The mathematical description of such variables requires a probability distribution.
A probability distribution completely specifies the possible values of a random variable and their corresponding probabilities.
2.2 Probability Mass Function (PMF)
Definition 2.1: Probability Mass Function
Let (X) be a discrete random variable. The probability mass function is defined as:
[ p_X(x)=P(X=x) ]
where:
- (x) represents a possible value of (X).
- (P(X=x)) represents the probability that (X) takes the value (x).
A valid PMF must satisfy two conditions:
Condition 1: Non-negative probabilities
[ p_X(x)\geq0 ]
Condition 2: Total probability equals one
[ \sum_x p_X(x)=1 ]
The complete distribution of (X) is therefore:
[ {(x,p_X(x)):x\in S} ]
where (S) is the support of the random variable.
2.3 Support of a Discrete Random Variable
The support is the set of values that a random variable can actually take.
For example:
[ X=\text{number of customers arriving in an hour} ]
has support:
[ S={0,1,2,3,\ldots} ]
However, a six-sided dice outcome has:
[ S={1,2,3,4,5,6} ]
The support determines the limits used in probability calculations.
2.4 Cumulative Distribution Function (CDF)
The cumulative distribution function describes the probability that a random variable is less than or equal to a given value.
Definition 2.2: CDF
The CDF is:
[ F_X(x)=P(X\leq x) ]
For a discrete random variable:
[ F_X(x)=\sum_{k\leq x}p_X(k) ]
The CDF accumulates probabilities from the smallest possible value up to (x).
Example 2.1: Constructing a CDF
Suppose:
| (x) | (P(X=x)) | | --- | -------- | | 0 | 0.2 | | 1 | 0.5 | | 2 | 0.3 |
The cumulative probabilities are:
For (x=0):
[ F(0)=0.2 ]
For (x=1):
[ F(1)=0.2+0.5=0.7 ]
For (x=2):
[ F(2)=0.2+0.5+0.3=1 ]
Therefore:
[
F_X(x)=
\begin{cases}
0.2,&x=0
0.7,&x=1
1,&x=2
\end{cases}
]
2.5 Important Discrete Probability Distributions
Several probability distributions appear repeatedly in statistical modelling.
The most important are:
- Bernoulli distribution.
- Binomial distribution.
- Geometric distribution.
- Negative Binomial distribution.
- Poisson distribution.
These distributions later provide important examples of Probability Generating Functions.
2.6 Bernoulli Distribution
Definition 2.3: Bernoulli Random Variable
A Bernoulli random variable represents a single experiment with two possible outcomes:
- Success.
- Failure.
Let:
[
X=
\begin{cases}
1,&\text{success}
0,&\text{failure}
\end{cases}
]
with:
[ P(X=1)=p ]
and:
[ P(X=0)=1-p ]
The PMF is:
[ P(X=x)=p^x(1-p)^{1-x} ]
where:
[ x\in{0,1} ]
Mean of Bernoulli Distribution
Using expectation:
[ E(X)=0(1-p)+1(p) ]
Therefore:
[ \boxed{E(X)=p} ]
Variance of Bernoulli Distribution
The second moment:
[ E(X^2)=p ]
Therefore:
[ Var(X)=E(X^2)-[E(X)]^2 ]
[ =p-p^2 ]
Hence:
[ \boxed{Var(X)=p(1-p)} ]
2.7 Binomial Distribution
Definition 2.4: Binomial Random Variable
A Binomial random variable counts the number of successes in (n) independent Bernoulli trials.
Suppose:
- Number of trials = (n)
- Probability of success = (p)
Then:
[ X\sim Bin(n,p) ]
The probability mass function is:
[ P(X=x)= \binom nx p^x(1-p)^{n-x} ]
where:
[ x=0,1,2,\ldots,n ]
Mean and Variance
The expected value is:
[ E(X)=np ]
The variance is:
[ Var(X)=np(1-p) ]
2.8 Poisson Distribution
Definition 2.5: Poisson Random Variable
The Poisson distribution models the number of events occurring within a fixed interval.
Examples:
- Calls arriving at a call centre.
- Accidents occurring on a highway.
- Particles detected by a sensor.
The random variable:
[ X\sim Pois(\lambda) ]
has PMF:
[ P(X=x)=\frac{e^{-\lambda}\lambda^x}{x!} ]
where:
[ x=0,1,2,\ldots ]
and (\lambda) represents the average event rate.
Mean and Variance
For a Poisson distribution:
[ E(X)=\lambda ]
and:
[ Var(X)=\lambda ]
The equality of mean and variance is a defining feature of the Poisson model.
2.9 Introduction to Generating Functions
Generating functions provide a bridge between probability distributions and algebra.
Instead of working directly with an infinite list:
[ p_0,p_1,p_2,\ldots ]
we encode the probabilities into a single function:
[ G(t)=p_0+p_1t+p_2t^2+\cdots ]
The coefficient of each power of (t) stores a probability.
2.10 Ordinary Generating Functions
Before introducing PGFs, consider a general generating function.
For a sequence:
[ a_0,a_1,a_2,\ldots ]
the ordinary generating function is:
[ A(t)=\sum_{n=0}^{\infty}a_nt^n ]
The coefficients can be recovered by expanding the function.
For example:
[ A(t)=2+3t+5t^2 ]
represents:
[ a_0=2,\quad a_1=3,\quad a_2=5 ]
Probability generating functions are a special case where the coefficients are probabilities.
2.11 Probability Generating Function Development
Let (X) be a discrete random variable with:
[ P(X=x)=p_x ]
The Probability Generating Function is defined by:
[ G_X(t)=E(t^X) ]
Using the definition of expectation:
[ G_X(t)= \sum_{x=0}^{\infty}t^xP(X=x) ]
Therefore:
[ \boxed{ G_X(t)=\sum_{x=0}^{\infty}p_xt^x } ]
This equation is the central definition of the PGF.
2.12 PGFs of Common Distributions
2.12.1 Bernoulli PGF
For Bernoulli:
[ P(X=0)=1-p ]
[ P(X=1)=p ]
Therefore:
[ G_X(t) =(1-p)t^0+pt^1 ]
Hence:
[ \boxed{ G_X(t)=1-p+pt } ]
2.12.2 Binomial PGF
For:
[ X\sim Bin(n,p) ]
the PGF is:
[ G_X(t)= \sum_{x=0}^{n} \binom nx p^x(1-p)^{n-x}t^x ]
Combining terms:
[ G_X(t)= \sum_{x=0}^{n} \binom nx (pt)^x(1-p)^{n-x} ]
Using the binomial theorem:
[ \boxed{ G_X(t)=(1-p+pt)^n } ]
2.12.3 Poisson PGF
For:
[ X\sim Pois(\lambda) ]
the PGF is:
[ G_X(t)= \sum_{x=0}^{\infty} \frac{e^{-\lambda}\lambda^x}{x!}t^x ]
Rearranging:
[ G_X(t)= e^{-\lambda} \sum_{x=0}^{\infty} \frac{(\lambda t)^x}{x!} ]
Using:
[ e^z=\sum_{x=0}^{\infty}\frac{z^x}{x!} ]
we obtain:
[ G_X(t)=e^{-\lambda}e^{\lambda t} ]
Therefore:
[ \boxed{ G_X(t)=e^{\lambda(t-1)} } ]
2.13 Why PGFs Are Powerful
The PGF converts probability questions into calculus and algebra.
From the PGF we can obtain:
Probability values:
[ P(X=x)=\frac{G_X^{(x)}(0)}{x!} ]
Mean:
[ E(X)=G_X'(1) ]
Variance:
[ Var(X)=G_X''(1)+G_X'(1)-[G_X'(1)]^2 ]
Independent sums:
If:
[ Z=X+Y ]
then:
[ G_Z(t)=G_X(t)G_Y(t) ]
These properties will be formally proved in the next chapter.
References
Feller, W. (1968). An Introduction to Probability Theory and Its Applications, Volume 1. Wiley.
Ross, S. M. (2014). Introduction to Probability Models (11th ed.). Academic Press.
Grimmett, G., & Stirzaker, D. (2020). Probability and Random Processes (4th ed.). Oxford University Press.
Wilf, H. S. (2005). generatingfunctionology (3rd ed.). A K Peters.
Pitman, J. (1993). Probability. Springer.
Next Chapter:
CHAPTER 3: THEORY AND MATHEMATICAL PROPERTIES OF PROBABILITY GENERATING FUNCTIONS
This chapter will contain:
- Formal PGF theorems.
- Proof of convergence.
- Recovery of probabilities.
- Derivation of moments.
- Variance proofs.
- Independent sums.
- Compound distributions.
- Mathematical examples.
CHAPTER 3: THEORY AND MATHEMATICAL PROPERTIES OF PROBABILITY GENERATING FUNCTIONS
3.1 Introduction
The Probability Generating Function (PGF) is one of the most important analytical tools in discrete probability theory. It transforms a discrete probability distribution into a power series whose coefficients contain the probabilities of the random variable.
For a discrete random variable (X) taking values:
[ X=0,1,2,\ldots ]
with probability mass function:
[ P(X=x)=p_x ]
the PGF is defined as:
[ \boxed{ G_X(t)=E(t^X)=\sum_{x=0}^{\infty}p_xt^x } ]
The variable (t) is not itself a random variable; it is an auxiliary or dummy variable used to encode the probability distribution.
The PGF provides a compact representation of the entire distribution and allows important characteristics such as moments, sums, and probabilities to be calculated through differentiation and algebraic manipulation.
3.2 Mathematical Structure of a PGF
A PGF is a power series:
[ G_X(t)=p_0+p_1t+p_2t^2+p_3t^3+\cdots ]
where:
- (p_0=P(X=0))
- (p_1=P(X=1))
- (p_2=P(X=2))
- (p_x=P(X=x))
The exponent of (t) represents the value of the random variable, while the coefficient represents the probability associated with that value.
For example:
[ G_X(t)=0.2+0.5t+0.3t^2 ]
means:
[ P(X=0)=0.2 ]
[ P(X=1)=0.5 ]
[ P(X=2)=0.3 ]
3.3 Fundamental Properties of Probability Generating Functions
A PGF satisfies several important mathematical properties.
These include:
- Normalisation property.
- Probability recovery.
- Uniqueness.
- Moment generation.
- Product rule for independent variables.
- Composition rule for random sums.
3.4 The Normalisation Property
Theorem 3.1
For every probability generating function:
[ \boxed{G_X(1)=1} ]
Proof
Starting from the definition:
[ G_X(t)=\sum_{x=0}^{\infty}p_xt^x ]
Substitute:
[ t=1 ]
Therefore:
[ G_X(1)=\sum_{x=0}^{\infty}p_x(1)^x ]
Since:
[ 1^x=1 ]
we obtain:
[ G_X(1)=\sum_{x=0}^{\infty}p_x ]
The total probability of all possible outcomes is:
[ \sum_{x=0}^{\infty}p_x=1 ]
Therefore:
[ \boxed{G_X(1)=1} ]
Interpretation
The value of the PGF at (t=1) confirms that the coefficients form a valid probability distribution.
If:
[ G_X(1)\neq1 ]
then the function cannot represent a valid probability distribution.
3.5 The Value at Zero
Theorem 3.2
The PGF evaluated at zero gives the probability that the random variable is zero:
[ \boxed{G_X(0)=P(X=0)} ]
Proof
The PGF is:
[ G_X(t)=p_0+p_1t+p_2t^2+\cdots ]
Setting:
[ t=0 ]
gives:
[ G_X(0)=p_0+p_1(0)+p_2(0)^2+\cdots ]
All terms vanish except the first term:
[ G_X(0)=p_0 ]
Therefore:
[ \boxed{G_X(0)=P(X=0)} ]
3.6 Recovery of Probabilities from a PGF
The PGF contains all information about the probability distribution.
Theorem 3.3: Probability Recovery Formula
The probability:
[ P(X=n) ]
can be recovered from the (n)-th derivative of the PGF:
[ \boxed{ P(X=n)=\frac{G_X^{(n)}(0)}{n!} } ]
Proof
Start with:
[ G_X(t)= p_0+p_1t+p_2t^2+\cdots+p_nt^n+\cdots ]
Differentiate once:
[ G_X'(t)
p_1+2p_2t+3p_3t^2+\cdots ]
Differentiate (n) times:
[ G_X^{(n)}(t) ]
The only term remaining after substituting:
[ t=0 ]
is:
[ n!p_n ]
Therefore:
[ G_X^{(n)}(0)=n!p_n ]
Rearranging:
[ \boxed{ p_n=\frac{G_X^{(n)}(0)}{n!} } ]
3.7 Uniqueness Property of PGFs
Theorem 3.4
A probability generating function uniquely determines its probability distribution.
This means:
If:
[ G_X(t)=G_Y(t) ]
for all values of (t) near zero, then:
[ P(X=x)=P(Y=x) ]
for every possible value of (x).
Explanation
The PGF coefficients are the probabilities:
[ G_X(t)=p_0+p_1t+p_2t^2+\cdots ]
Therefore, if two PGFs are identical, every coefficient must be identical.
Hence:
[ p_x=q_x ]
for all (x).
3.8 Convergence of Probability Generating Functions
Because a PGF is an infinite power series, convergence must be considered.
Theorem 3.5
Every PGF converges absolutely for:
[ |t|\leq1 ]
Proof
Consider:
[ |G_X(t)|
\left| \sum_{x=0}^{\infty}p_xt^x \right| ]
Using the triangle inequality:
[ |G_X(t)| \leq \sum_{x=0}^{\infty}|p_xt^x| ]
Because:
[ |t|\leq1 ]
we have:
[ |t|^x\leq1 ]
Therefore:
[ \sum_{x=0}^{\infty}p_x|t|^x \leq \sum_{x=0}^{\infty}p_x ]
Since:
[ \sum_{x=0}^{\infty}p_x=1 ]
the series is bounded.
Hence the PGF converges absolutely.
3.9 Generation of Moments
One of the greatest advantages of PGFs is their ability to generate moments.
Unlike moment generating functions, PGFs naturally produce factorial moments.
3.10 First Moment: Mean of a Random Variable
Differentiate:
[ G_X(t)=\sum_{x=0}^{\infty}p_xt^x ]
with respect to (t):
[ G_X'(t)
\sum_{x=1}^{\infty}xp_xt^{x-1} ]
Now evaluate at:
[ t=1 ]
giving:
[ G_X'(1)
\sum_{x=1}^{\infty}xp_x ]
But:
[ E(X)=\sum xp_x ]
Therefore:
[ \boxed{ E(X)=G_X'(1) } ]
3.11 Second Factorial Moment
Differentiate again:
[ G_X''(t)
\sum_{x=2}^{\infty}x(x-1)p_xt^{x-2} ]
At:
[ t=1 ]
we obtain:
[ G_X''(1)
\sum_{x=2}^{\infty}x(x-1)p_x ]
Therefore:
[ \boxed{ G_X''(1)=E[X(X-1)] } ]
3.12 Derivation of Variance Formula
The variance is:
[ Var(X)=E(X^2)-[E(X)]^2 ]
From the factorial moment:
[ X(X-1)=X^2-X ]
Therefore:
[ X^2=X(X-1)+X ]
Taking expectations:
[ E(X^2)=E[X(X-1)]+E(X) ]
Using PGFs:
[ E(X^2)=G_X''(1)+G_X'(1) ]
Hence:
[ Var(X)
G_X''(1)+G_X'(1)-[G_X'(1)]^2 ]
Therefore:
[ \boxed{ Var(X)=G_X''(1)+G_X'(1)-[G_X'(1)]^2 } ]
3.13 Independent Sums of Random Variables
One of the most powerful properties of PGFs concerns addition.
Suppose:
[ Z=X+Y ]
where (X) and (Y) are independent.
The PGF of (Z) is:
[ G_Z(t)=E(t^Z) ]
Since:
[ Z=X+Y ]
we have:
[ G_Z(t)=E(t^{X+Y}) ]
Using exponent rules:
[ t^{X+Y}=t^Xt^Y ]
Therefore:
[ G_Z(t)=E(t^Xt^Y) ]
Because (X) and (Y) are independent:
[ E(t^Xt^Y)=E(t^X)E(t^Y) ]
Hence:
[ \boxed{ G_Z(t)=G_X(t)G_Y(t) } ]
3.14 Compound Distributions
A compound distribution occurs when the number of events is itself random.
Let:
[ N=\text{number of events} ]
and:
[ X_1,X_2,\ldots ]
represent independent event sizes.
The total amount is:
[ S=X_1+X_2+\cdots+X_N ]
The PGF is:
[ \boxed{ G_S(t)=G_N(G_X(t)) } ]
This composition rule is fundamental in:
- Insurance mathematics.
- Queueing theory.
- Reliability engineering.
- Risk analysis.
References
Feller, W. (1968). An Introduction to Probability Theory and Its Applications, Volume 1. Wiley.
Grimmett, G., & Stirzaker, D. (2020). Probability and Random Processes (4th ed.). Oxford University Press.
Ross, S. M. (2014). Introduction to Probability Models (11th ed.). Academic Press.
Wilf, H. S. (2005). generatingfunctionology (3rd ed.). A K Peters.
Athreya, K. B., & Ney, P. E. (2004). Branching Processes. Dover Publications.
Next Chapter:
CHAPTER 4: APPLICATIONS OF PROBABILITY GENERATING FUNCTIONS
Topics:
- Bernoulli, Binomial, Poisson and Geometric PGFs.
- Deriving means and variances using PGFs.
- Branching processes.
- Extinction probabilities.
- Queueing applications.
- Actuarial risk models.
CHAPTER 4: APPLICATIONS OF PROBABILITY GENERATING FUNCTIONS
4.1 Introduction
The Probability Generating Function (PGF) is not only a theoretical construction but also a practical computational tool. Its ability to transform probability distributions into algebraic functions makes it valuable in many areas of applied mathematics, statistics, operations research, actuarial science, biology, and engineering.
For a discrete random variable (X), the PGF is:
[ G_X(t)=E(t^X) ]
or:
[ G_X(t)=\sum_{x=0}^{\infty}P(X=x)t^x ]
The major applications of PGFs include:
- Deriving properties of probability distributions.
- Calculating moments.
- Combining independent random variables.
- Modelling population growth.
- Analysing queueing systems.
- Studying insurance risk.
4.2 PGF of the Bernoulli Distribution
Definition
A Bernoulli random variable represents one experiment with two possible outcomes:
[
X=
\begin{cases}
1,&\text{success}
0,&\text{failure}
\end{cases}
]
where:
[ P(X=1)=p ]
and:
[ P(X=0)=1-p ]
The PGF is:
[ G_X(t)=E(t^X) ]
Expanding:
[ G_X(t)
P(X=0)t^0+P(X=1)t^1 ]
Substituting probabilities:
[ G_X(t)
(1-p)+pt ]
Therefore:
[ \boxed{ G_X(t)=1-p+pt } ]
Mean Using the PGF
Differentiate:
[ G_X'(t)=p ]
Evaluate at:
[ t=1 ]
Therefore:
[ E(X)=G_X'(1) ]
[ \boxed{ E(X)=p } ]
Variance Using the PGF
The second derivative is:
[ G_X''(t)=0 ]
Using the variance formula:
[ Var(X)=G_X''(1)+G_X'(1)-[G_X'(1)]^2 ]
Substitute:
[ Var(X)=0+p-p^2 ]
Hence:
[ \boxed{ Var(X)=p(1-p) } ]
4.3 PGF of the Binomial Distribution
Definition
The Binomial distribution counts the number of successes in (n) independent Bernoulli trials.
[ X\sim Bin(n,p) ]
The PMF is:
[ P(X=x)= \binom nx p^x(1-p)^{n-x} ]
for:
[ x=0,1,\ldots,n ]
Derivation of the Binomial PGF
The PGF is:
[ G_X(t)= \sum_{x=0}^{n} \binom nx p^x(1-p)^{n-x}t^x ]
Rearrange:
[ G_X(t)
\sum_{x=0}^{n} \binom nx(pt)^x(1-p)^{n-x} ]
Using the binomial theorem:
[ (a+b)^n= \sum_{x=0}^{n} \binom nx a^xb^{n-x} ]
where:
[ a=pt ]
and:
[ b=1-p ]
Therefore:
[ \boxed{ G_X(t)=(1-p+pt)^n } ]
Mean Derivation
Differentiate:
[ G_X(t)=(1-p+pt)^n ]
Using the chain rule:
[ G_X'(t)=np(1-p+pt)^{n-1} ]
At (t=1):
[ G_X'(1)
np(1-p+p)^{n-1} ]
Since:
[ 1-p+p=1 ]
we obtain:
[ \boxed{ E(X)=np } ]
Variance Derivation
Differentiate again:
[ G_X''(t)
n(n-1)p^2(1-p+pt)^{n-2} ]
At:
[ t=1 ]
[ G_X''(1)=n(n-1)p^2 ]
Using:
[ Var(X)=G_X''(1)+G_X'(1)-[G_X'(1)]^2 ]
we obtain:
[ Var(X)
n(n-1)p^2+np-(np)^2 ]
Simplifying:
[ \boxed{ Var(X)=np(1-p) } ]
4.4 PGF of the Poisson Distribution
Definition
The Poisson distribution models the number of events occurring in a fixed interval.
[ X\sim Pois(\lambda) ]
Its PMF is:
[ P(X=x)= \frac{e^{-\lambda}\lambda^x}{x!} ]
Derivation of the Poisson PGF
The PGF is:
[ G_X(t)
\sum_{x=0}^{\infty} \frac{e^{-\lambda}\lambda^x}{x!}t^x ]
Take constants outside:
[ G_X(t)
e^{-\lambda} \sum_{x=0}^{\infty} \frac{(\lambda t)^x}{x!} ]
Using the exponential expansion:
[ e^z= \sum_{x=0}^{\infty} \frac{z^x}{x!} ]
we get:
[ G_X(t)
e^{-\lambda}e^{\lambda t} ]
Therefore:
[ \boxed{ G_X(t)=e^{\lambda(t-1)} } ]
Mean of Poisson Distribution
Differentiate:
[ G_X'(t)= \lambda e^{\lambda(t-1)} ]
At:
[ t=1 ]
[ E(X)= \lambda e^0 ]
Therefore:
[ \boxed{ E(X)=\lambda } ]
Variance of Poisson Distribution
Second derivative:
[ G_X''(t)= \lambda^2e^{\lambda(t-1)} ]
At:
[ t=1 ]
[ G_X''(1)=\lambda^2 ]
Using:
[ Var(X)
G_X''(1)+G_X'(1)-[G_X'(1)]^2 ]
we obtain:
[ Var(X)
\lambda^2+\lambda-\lambda^2 ]
Hence:
[ \boxed{ Var(X)=\lambda } ]
4.5 PGF of the Geometric Distribution
Definition
The geometric distribution models the number of failures before the first success.
Let:
[ P(X=x)=p(1-p)^x ]
where:
[ x=0,1,2,\ldots ]
PGF Derivation
The PGF is:
[ G_X(t)
\sum_{x=0}^{\infty} p(1-p)^xt^x ]
Factor:
[ G_X(t)
p\sum_{x=0}^{\infty} [(1-p)t]^x ]
Using the geometric series:
[ \sum_{x=0}^{\infty}r^x= \frac1{1-r} ]
we obtain:
[ \boxed{ G_X(t)= \frac{p}{1-(1-p)t} } ]
4.6 Sums of Independent Random Variables
One of the strongest applications of PGFs is simplifying sums.
Let:
[ Z=X+Y ]
where (X) and (Y) are independent.
Then:
[ G_Z(t)=G_X(t)G_Y(t) ]
Example: Sum of Independent Poisson Variables
Suppose:
[ X\sim Pois(\lambda_1) ]
and:
[ Y\sim Pois(\lambda_2) ]
The PGFs are:
[ G_X(t)=e^{\lambda_1(t-1)} ]
and:
[ G_Y(t)=e^{\lambda_2(t-1)} ]
Therefore:
[ G_Z(t)
e^{\lambda_1(t-1)} e^{\lambda_2(t-1)} ]
Combining powers:
[ G_Z(t)
e^{(\lambda_1+\lambda_2)(t-1)} ]
This is the PGF of:
[ Pois(\lambda_1+\lambda_2) ]
Hence:
[ \boxed{ X+Y\sim Pois(\lambda_1+\lambda_2) } ]
4.7 Branching Processes
A major application of PGFs is the analysis of population growth.
A branching process models systems where individuals produce random numbers of offspring.
Examples:
- Biological reproduction.
- Nuclear chain reactions.
- Spread of information.
- Epidemics.
Galton-Watson Branching Process
Let:
[ X_n ]
represent the population size at generation (n).
Suppose each individual produces offspring according to the PGF:
[ G(t) ]
The population PGF after (n) generations is obtained by composition:
[ \boxed{ G_n(t)=G(G_{n-1}(t)) } ]
Repeatedly:
[ G_n(t)= \underbrace{ G(G(\cdots G(t))) }_{n\text{ times}} ]
4.8 Extinction Probability
The extinction probability is the probability that the population eventually becomes zero.
Let:
[ q=P(\text{eventual extinction}) ]
The fundamental extinction equation is:
[ \boxed{ q=G(q) } ]
The smallest solution in:
[ 0\leq q\leq1 ]
is the extinction probability.
4.9 Actuarial Applications
Insurance companies use PGFs to model claim numbers and aggregate losses.
Let:
[ N=\text{number of claims} ]
and:
[ X_i=\text{size of each claim} ]
The total claim amount is:
[ S=X_1+X_2+\cdots+X_N ]
The PGF is:
[ \boxed{ G_S(t)=G_N(G_X(t)) } ]
This allows insurers to calculate:
- Expected claims.
- Risk exposure.
- Probability of large losses.
4.10 Queueing Applications
PGFs are widely used in queueing theory.
Examples:
- Number of customers waiting.
- Number of packets in a network.
- Number of jobs in a processor queue.
For a queue length random variable (N):
[ G_N(t)=E(t^N) ]
allows researchers to determine:
- Average queue length.
- Waiting times.
- System stability.
References
Ross, S. M. (2014). Introduction to Probability Models (11th ed.). Academic Press.
Grimmett, G., & Stirzaker, D. (2020). Probability and Random Processes (4th ed.). Oxford University Press.
Feller, W. (1968). An Introduction to Probability Theory and Its Applications, Volume 1. Wiley.
Athreya, K. B., & Ney, P. E. (2004). Branching Processes. Dover Publications.
Harris, T. E. (2002). The Theory of Branching Processes. Dover Publications.
Next Chapter
CHAPTER 5: ADVANCED THEORY OF PROBABILITY GENERATING FUNCTIONS
Topics:
- Joint PGFs for multivariate random variables.
- PGFs and covariance.
- Compound distributions.
- Relationship between PGFs, MGFs, and characteristic functions.
- Advanced branching-process mathematics.
- Queueing transforms.
CHAPTER 5: ADVANCED THEORY OF PROBABILITY GENERATING FUNCTIONS 5.1 Introduction
The previous chapters introduced the basic definition of Probability Generating Functions (PGFs), their properties, and their applications to common discrete probability distributions.
This chapter develops more advanced mathematical ideas involving PGFs, including:
Multivariate probability generating functions. Joint distributions and dependence. Covariance and correlation using PGFs. Compound distributions. Relationship between PGFs, Moment Generating Functions (MGFs), and Characteristic Functions. Advanced branching process analysis. Queueing theory applications.
These extensions allow PGFs to be applied to complex stochastic systems involving several random variables.
5.2 Multivariate Probability Generating Functions
In many applications, we deal with more than one random variable.
Suppose:
X=(X 1
,X 2
,…,X k
)
is a vector of discrete random variables.
The joint probability generating function is defined as:
G(t 1
,t 2
,…,t k
)=E(t 1 X 1
t 2 X 2
⋯t k X k
)
Expanding:
G(t 1
,t 2
,…,t k
)= x 1
∑
x 2
∑
⋯ x k
∑
P(X 1
=x 1
,…,X k
=x k
)t 1 x 1
⋯t k x k
The coefficients contain the complete joint probability distribution.
5.3 Bivariate Probability Generating Functions
For two discrete random variables:
X,Y
the joint PGF is:
G X,Y
(s,t)=E(s X t Y )
Expanding:
G X,Y
(s,t)= x ∑
y ∑
P(X=x,Y=y)s x t y
where:
The power of s represents values of X. The power of t represents values of Y. The coefficient represents the joint probability. Example 5.1
Suppose:
X Y Probability 0 0 0.3 1 0 0.2 0 1 0.1 1 1 0.4
The joint PGF is:
G(s,t)=0.3+0.2s+0.1t+0.4st
Setting:
s=1
gives the marginal PGF of Y:
G Y
(t)=G(1,t) 5.4 Marginal Distributions from Joint PGFs
A joint PGF contains information about individual distributions.
For random variables:
X,Y
the marginal PGF of X is obtained by setting:
t=1
Therefore:
G X
(s)=G X,Y
(s,1)
Similarly:
G Y
(t)=G X,Y
(1,t)
This property is extremely useful in multivariate probability modelling.
5.5 Independence and Joint PGFs
Two random variables are independent if:
P(X=x,Y=y)=P(X=x)P(Y=y)
For independent variables:
E(s X t Y )=E(s X )E(t Y )
Therefore:
G X,Y
(s,t)=G X
(s)G Y
(t)
Hence:
X,Y independent⟺G X,Y
(s,t)=G X
(s)G Y
(t)
5.6 Covariance Using PGFs
Covariance measures dependence between two random variables.
It is defined as:
Cov(X,Y)=E(XY)−E(X)E(Y)
Using the joint PGF:
G(s,t)=E(s X t Y )
Differentiate with respect to s:
∂s ∂
G(s,t)=E(Xs X−1 t Y )
Differentiate with respect to t:
∂s∂t ∂ 2
G(s,t)=E(XYs X−1 t Y−1 )
At:
s=t=1
we obtain:
E(XY)= ∂s∂t ∂ 2 G
(1,1)
Therefore:
Cov(X,Y)=G st
(1,1)−G s
(1,1)G t
(1,1)
5.7 Compound Probability Distributions
A compound distribution occurs when the number of random components is itself random.
Let:
N
be a random number of events.
Let:
X 1
,X 2
,…,X N
be independent random variables.
Define:
S=X 1
+X 2
+⋯+X N
The total quantity S has a compound distribution.
Theorem 5.1: Compound PGF Formula
The PGF of S is:
G S
(t)=G N
(G X
(t))
Proof
Condition on N:
G S
(t)=E(t S )
Since:
S=X 1
+⋯+X N
we have:
t S =t X 1
⋯t X N
For fixed N=n:
E(t S ∣N=n)=[G X
(t)] n
Therefore:
G S
(t)=E([G X
(t)] N )
Using the definition of PGF:
G S
(t)=G N
(G X
(t))
5.8 Relationship Between PGFs and Moment Generating Functions
The Moment Generating Function (MGF) is:
M X
(u)=E(e uX )
The PGF is:
G X
(t)=E(t X )
Since:
t=e u
then:
u=ln(t)
Therefore:
G X
(t)=M X
(lnt)
and:
M X
(u)=G X
(e u )
Thus, PGFs and MGFs contain equivalent information.
5.9 Relationship with Characteristic Functions
The characteristic function is:
ϕ X
(θ)=E(e iθX )
Since:
t=e iθ
the PGF becomes:
G X
(e iθ )=ϕ X
(θ)
Therefore:
ϕ X
(θ)=G X
(e iθ )
This relationship connects discrete probability theory with Fourier analysis.
5.10 Advanced Branching Processes
Branching processes describe populations where individuals reproduce randomly.
Let:
Z n
represent the population size after generation n.
The offspring PGF is:
G(t)
The population PGF satisfies:
G n
(t)=G(G n−1
(t))
Repeated composition gives:
G n
(t)=G (n) (t)
where G (n) represents repeated functional composition.
5.11 Extinction Probability in Branching Processes
Let:
q=P(eventual extinction)
The extinction probability satisfies:
q=G(q)
There are two cases:
Case 1: Subcritical process
If:
E(X)<1
then:
q=1
The population eventually disappears.
Case 2: Supercritical process
If:
E(X)>1
then:
q<1
There is a positive probability of survival.
5.12 Queueing Theory Applications
PGFs are widely used to study queue systems.
Let:
N=number of customers in a system
The queue length PGF is:
G N
(t)=E(t N )
From this:
Mean queue length:
E(N)=G N ′
(1)
Variance:
Var(N)=G N ′′
(1)+G N ′
(1)−[G N ′
(1)] 2
Applications include:
Computer networks. Communication systems. Manufacturing systems. Telephone exchanges. 5.13 Summary of Important PGF Results
For a discrete random variable:
G X
(t)=E(t X )
Important identities:
Total probability: G X
(1)=1 Probability recovery: P(X=n)= n! G X (n)
(0)
Mean: E(X)=G X ′
(1) Variance: Var(X)=G X ′′
(1)+G X ′
(1)−[G X ′
(1)] 2 Independent sums: G X+Y
(t)=G X
(t)G Y
(t) Compound distributions: G S
(t)=G N
(G X
(t)) References
Feller, W. (1968). An Introduction to Probability Theory and Its Applications. Wiley.
Ross, S. M. (2014). Introduction to Probability Models. Academic Press.
Grimmett, G., & Stirzaker, D. (2020). Probability and Random Processes. Oxford University Press.
Athreya, K. B., & Ney, P. E. (2004). Branching Processes. Dover Publications.
Harris, T. E. (2002). The Theory of Branching Processes. Dover Publications.