<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=US-ASCII">
<title>Comparing the means of two samples with the Students-t test</title>
<link rel="stylesheet" href="../../../../../../../../../../doc/src/boostbook.css" type="text/css">
<meta name="generator" content="DocBook XSL Stylesheets V1.74.0">
<link rel="home" href="../../../../../index.html" title="Math Toolkit">
<link rel="up" href="../st_eg.html" title="Student's t Distribution Examples">
<link rel="prev" href="tut_mean_size.html" title="Estimating how large a sample size would have to become in order to give a significant Students-t test result with a single sample test">
<link rel="next" href="paired_st.html" title="Comparing two paired samples with the Student's t distribution">
</head>
<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
<table cellpadding="2" width="100%"><tr>
<td valign="top"><img alt="Boost C++ Libraries" width="277" height="86" src="../../../../../../../../../../boost.png"></td>
<td align="center"><a href="../../../../../../../../../../index.html">Home</a></td>
<td align="center"><a href="../../../../../../../../../../libs/libraries.htm">Libraries</a></td>
<td align="center"><a href="http://www.boost.org/users/people.html">People</a></td>
<td align="center"><a href="http://www.boost.org/users/faq.html">FAQ</a></td>
<td align="center"><a href="../../../../../../../../../../more/index.htm">More</a></td>
</tr></table>
<hr>
<div class="spirit-nav">
<a accesskey="p" href="tut_mean_size.html"><img src="../../../../../../../../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../st_eg.html"><img src="../../../../../../../../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../../../../../index.html"><img src="../../../../../../../../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="paired_st.html"><img src="../../../../../../../../../../doc/src/images/next.png" alt="Next"></a>
</div>
<div class="section" lang="en">
<div class="titlepage"><div><div><h6 class="title">
<a name="math_toolkit.dist.stat_tut.weg.st_eg.two_sample_students_t"></a><a class="link" href="two_sample_students_t.html" title="Comparing the means of two samples with the Students-t test">
Comparing the means of two samples with the Students-t test</a>
</h6></div></div></div>
<p>
Imagine that we have two samples, and we wish to determine whether
their means are different or not. This situation often arises when
determining whether a new process or treatment is better than an old
one.
</p>
<p>
In this example, we'll be using the <a href="http://www.itl.nist.gov/div898/handbook/eda/section3/eda3531.htm" target="_top">Car
Mileage sample data</a> from the <a href="http://www.itl.nist.gov" target="_top">NIST
website</a>. The data compares miles per gallon of US cars with
miles per gallon of Japanese cars.
</p>
<p>
The sample code is in <a href="../../../../../../../../example/students_t_two_samples.cpp" target="_top">students_t_two_samples.cpp</a>.
</p>
<p>
There are two ways in which this test can be conducted: we can assume
that the true standard deviations of the two samples are equal or not.
If the standard deviations are assumed to be equal, then the calculation
of the t-statistic is greatly simplified, so we'll examine that case
first. In real life we should verify whether this assumption is valid
with a Chi-Squared test for equal variances.
</p>
<p>
We begin by defining a procedure that will conduct our test assuming
equal variances:
</p>
<pre class="programlisting"><span class="comment">// Needed headers:
</span><span class="preprocessor">#include</span> <span class="special"><</span><span class="identifier">boost</span><span class="special">/</span><span class="identifier">math</span><span class="special">/</span><span class="identifier">distributions</span><span class="special">/</span><span class="identifier">students_t</span><span class="special">.</span><span class="identifier">hpp</span><span class="special">></span>
<span class="preprocessor">#include</span> <span class="special"><</span><span class="identifier">iostream</span><span class="special">></span>
<span class="preprocessor">#include</span> <span class="special"><</span><span class="identifier">iomanip</span><span class="special">></span>
<span class="comment">// Simplify usage:
</span><span class="keyword">using</span> <span class="keyword">namespace</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">math</span><span class="special">;</span>
<span class="keyword">using</span> <span class="keyword">namespace</span> <span class="identifier">std</span><span class="special">;</span>
<span class="keyword">void</span> <span class="identifier">two_samples_t_test_equal_sd</span><span class="special">(</span>
<span class="keyword">double</span> <span class="identifier">Sm1</span><span class="special">,</span> <span class="comment">// Sm1 = Sample 1 Mean.
</span> <span class="keyword">double</span> <span class="identifier">Sd1</span><span class="special">,</span> <span class="comment">// Sd1 = Sample 1 Standard Deviation.
</span> <span class="keyword">unsigned</span> <span class="identifier">Sn1</span><span class="special">,</span> <span class="comment">// Sn1 = Sample 1 Size.
</span> <span class="keyword">double</span> <span class="identifier">Sm2</span><span class="special">,</span> <span class="comment">// Sm2 = Sample 2 Mean.
</span> <span class="keyword">double</span> <span class="identifier">Sd2</span><span class="special">,</span> <span class="comment">// Sd2 = Sample 2 Standard Deviation.
</span> <span class="keyword">unsigned</span> <span class="identifier">Sn2</span><span class="special">,</span> <span class="comment">// Sn2 = Sample 2 Size.
</span> <span class="keyword">double</span> <span class="identifier">alpha</span><span class="special">)</span> <span class="comment">// alpha = Significance Level.
</span><span class="special">{</span>
</pre>
<p>
Our procedure will begin by calculating the t-statistic, assuming equal
variances the needed formulae are:
</p>
<p>
<span class="inlinemediaobject"><img src="../../../../../../equations/dist_tutorial1.png"></span>
</p>
<p>
where Sp is the "pooled" standard deviation of the two samples,
and <span class="emphasis"><em>v</em></span> is the number of degrees of freedom of the
two combined samples. We can now write the code to calculate the t-statistic:
</p>
<pre class="programlisting"><span class="comment">// Degrees of freedom:
</span><span class="keyword">double</span> <span class="identifier">v</span> <span class="special">=</span> <span class="identifier">Sn1</span> <span class="special">+</span> <span class="identifier">Sn2</span> <span class="special">-</span> <span class="number">2</span><span class="special">;</span>
<span class="identifier">cout</span> <span class="special"><<</span> <span class="identifier">setw</span><span class="special">(</span><span class="number">55</span><span class="special">)</span> <span class="special"><<</span> <span class="identifier">left</span> <span class="special"><<</span> <span class="string">"Degrees of Freedom"</span> <span class="special"><<</span> <span class="string">"= "</span> <span class="special"><<</span> <span class="identifier">v</span> <span class="special"><<</span> <span class="string">"\n"</span><span class="special">;</span>
<span class="comment">// Pooled variance:
</span><span class="keyword">double</span> <span class="identifier">sp</span> <span class="special">=</span> <span class="identifier">sqrt</span><span class="special">(((</span><span class="identifier">Sn1</span><span class="special">-</span><span class="number">1</span><span class="special">)</span> <span class="special">*</span> <span class="identifier">Sd1</span> <span class="special">*</span> <span class="identifier">Sd1</span> <span class="special">+</span> <span class="special">(</span><span class="identifier">Sn2</span><span class="special">-</span><span class="number">1</span><span class="special">)</span> <span class="special">*</span> <span class="identifier">Sd2</span> <span class="special">*</span> <span class="identifier">Sd2</span><span class="special">)</span> <span class="special">/</span> <span class="identifier">v</span><span class="special">);</span>
<span class="identifier">cout</span> <span class="special"><<</span> <span class="identifier">setw</span><span class="special">(</span><span class="number">55</span><span class="special">)</span> <span class="special"><<</span> <span class="identifier">left</span> <span class="special"><<</span> <span class="string">"Pooled Standard Deviation"</span> <span class="special"><<</span> <span class="string">"= "</span> <span class="special"><<</span> <span class="identifier">v</span> <span class="special"><<</span> <span class="string">"\n"</span><span class="special">;</span>
<span class="comment">// t-statistic:
</span><span class="keyword">double</span> <span class="identifier">t_stat</span> <span class="special">=</span> <span class="special">(</span><span class="identifier">Sm1</span> <span class="special">-</span> <span class="identifier">Sm2</span><span class="special">)</span> <span class="special">/</span> <span class="special">(</span><span class="identifier">sp</span> <span class="special">*</span> <span class="identifier">sqrt</span><span class="special">(</span><span class="number">1.0</span> <span class="special">/</span> <span class="identifier">Sn1</span> <span class="special">+</span> <span class="number">1.0</span> <span class="special">/</span> <span class="identifier">Sn2</span><span class="special">));</span>
<span class="identifier">cout</span> <span class="special"><<</span> <span class="identifier">setw</span><span class="special">(</span><span class="number">55</span><span class="special">)</span> <span class="special"><<</span> <span class="identifier">left</span> <span class="special"><<</span> <span class="string">"T Statistic"</span> <span class="special"><<</span> <span class="string">"= "</span> <span class="special"><<</span> <span class="identifier">t_stat</span> <span class="special"><<</span> <span class="string">"\n"</span><span class="special">;</span>
</pre>
<p>
The next step is to define our distribution object, and calculate the
complement of the probability:
</p>
<pre class="programlisting"><span class="identifier">students_t</span> <span class="identifier">dist</span><span class="special">(</span><span class="identifier">v</span><span class="special">);</span>
<span class="keyword">double</span> <span class="identifier">q</span> <span class="special">=</span> <span class="identifier">cdf</span><span class="special">(</span><span class="identifier">complement</span><span class="special">(</span><span class="identifier">dist</span><span class="special">,</span> <span class="identifier">fabs</span><span class="special">(</span><span class="identifier">t_stat</span><span class="special">)));</span>
<span class="identifier">cout</span> <span class="special"><<</span> <span class="identifier">setw</span><span class="special">(</span><span class="number">55</span><span class="special">)</span> <span class="special"><<</span> <span class="identifier">left</span> <span class="special"><<</span> <span class="string">"Probability that difference is due to chance"</span> <span class="special"><<</span> <span class="string">"= "</span>
<span class="special"><<</span> <span class="identifier">setprecision</span><span class="special">(</span><span class="number">3</span><span class="special">)</span> <span class="special"><<</span> <span class="identifier">scientific</span> <span class="special"><<</span> <span class="number">2</span> <span class="special">*</span> <span class="identifier">q</span> <span class="special"><<</span> <span class="string">"\n\n"</span><span class="special">;</span>
</pre>
<p>
Here we've used the absolute value of the t-statistic, because we initially
want to know simply whether there is a difference or not (a two-sided
test). However, we can also test whether the mean of the second sample
is greater or is less (one-sided test) than that of the first: all
the possible tests are summed up in the following table:
</p>
<div class="informaltable"><table class="table">
<colgroup>
<col>
<col>
</colgroup>
<thead><tr>
<th>
<p>
Hypothesis
</p>
</th>
<th>
<p>
Test
</p>
</th>
</tr></thead>
<tbody>
<tr>
<td>
<p>
The Null-hypothesis: there is <span class="bold"><strong>no difference</strong></span>
in means
</p>
</td>
<td>
<p>
Reject if complement of CDF for |t| < significance level
/ 2:
</p>
<p>
<code class="computeroutput"><span class="identifier">cdf</span><span class="special">(</span><span class="identifier">complement</span><span class="special">(</span><span class="identifier">dist</span><span class="special">,</span>
<span class="identifier">fabs</span><span class="special">(</span><span class="identifier">t</span><span class="special">)))</span>
<span class="special"><</span> <span class="identifier">alpha</span>
<span class="special">/</span> <span class="number">2</span></code>
</p>
</td>
</tr>
<tr>
<td>
<p>
The Alternative-hypothesis: there is a <span class="bold"><strong>difference</strong></span>
in means
</p>
</td>
<td>
<p>
Reject if complement of CDF for |t| > significance level
/ 2:
</p>
<p>
<code class="computeroutput"><span class="identifier">cdf</span><span class="special">(</span><span class="identifier">complement</span><span class="special">(</span><span class="identifier">dist</span><span class="special">,</span>
<span class="identifier">fabs</span><span class="special">(</span><span class="identifier">t</span><span class="special">)))</span>
<span class="special"><</span> <span class="identifier">alpha</span>
<span class="special">/</span> <span class="number">2</span></code>
</p>
</td>
</tr>
<tr>
<td>
<p>
The Alternative-hypothesis: Sample 1 Mean is <span class="bold"><strong>less</strong></span>
than Sample 2 Mean.
</p>
</td>
<td>
<p>
Reject if CDF of t > significance level:
</p>
<p>
<code class="computeroutput"><span class="identifier">cdf</span><span class="special">(</span><span class="identifier">dist</span><span class="special">,</span>
<span class="identifier">t</span><span class="special">)</span>
<span class="special">></span> <span class="identifier">alpha</span></code>
</p>
</td>
</tr>
<tr>
<td>
<p>
The Alternative-hypothesis: Sample 1 Mean is <span class="bold"><strong>greater</strong></span>
than Sample 2 Mean.
</p>
</td>
<td>
<p>
Reject if complement of CDF of t > significance level:
</p>
<p>
<code class="computeroutput"><span class="identifier">cdf</span><span class="special">(</span><span class="identifier">complement</span><span class="special">(</span><span class="identifier">dist</span><span class="special">,</span>
<span class="identifier">t</span><span class="special">))</span>
<span class="special">></span> <span class="identifier">alpha</span></code>
</p>
</td>
</tr>
</tbody>
</table></div>
<div class="note"><table border="0" summary="Note">
<tr>
<td rowspan="2" align="center" valign="top" width="25"><img alt="[Note]" src="../../../../../../../../../../doc/src/images/note.png"></td>
<th align="left">Note</th>
</tr>
<tr><td align="left" valign="top"><p>
For a two-sided test we must compare against alpha / 2 and not alpha.
</p></td></tr>
</table></div>
<p>
Most of the rest of the sample program is pretty-printing, so we'll
skip over that, and take a look at the sample output for alpha=0.05
(a 95% probability level). For comparison the dataplot output for the
same data is in <a href="http://www.itl.nist.gov/div898/handbook/eda/section3/eda353.htm" target="_top">section
1.3.5.3</a> of the <a href="http://www.itl.nist.gov/div898/handbook/" target="_top">NIST/SEMATECH
e-Handbook of Statistical Methods.</a>.
</p>
<pre class="programlisting"> ________________________________________________
Student t test for two samples (equal variances)
________________________________________________
Number of Observations (Sample 1) = 249
Sample 1 Mean = 20.14458
Sample 1 Standard Deviation = 6.41470
Number of Observations (Sample 2) = 79
Sample 2 Mean = 30.48101
Sample 2 Standard Deviation = 6.10771
Degrees of Freedom = 326.00000
Pooled Standard Deviation = 326.00000
T Statistic = -12.62059
Probability that difference is due to chance = 5.273e-030
Results for Alternative Hypothesis and alpha = 0.0500
Alternative Hypothesis Conclusion
Sample 1 Mean != Sample 2 Mean NOT REJECTED
Sample 1 Mean < Sample 2 Mean NOT REJECTED
Sample 1 Mean > Sample 2 Mean REJECTED
</pre>
<p>
So with a probability that the difference is due to chance of just
5.273e-030, we can safely conclude that there is indeed a difference.
</p>
<p>
The tests on the alternative hypothesis show that we must also reject
the hypothesis that Sample 1 Mean is greater than that for Sample 2:
in this case Sample 1 represents the miles per gallon for Japanese
cars, and Sample 2 the miles per gallon for US cars, so we conclude
that Japanese cars are on average more fuel efficient.
</p>
<p>
Now that we have the simple case out of the way, let's look for a moment
at the more complex one: that the standard deviations of the two samples
are not equal. In this case the formula for the t-statistic becomes:
</p>
<p>
<span class="inlinemediaobject"><img src="../../../../../../equations/dist_tutorial2.png"></span>
</p>
<p>
And for the combined degrees of freedom we use the <a href="http://en.wikipedia.org/wiki/Welch-Satterthwaite_equation" target="_top">Welch-Satterthwaite</a>
approximation:
</p>
<p>
<span class="inlinemediaobject"><img src="../../../../../../equations/dist_tutorial3.png"></span>
</p>
<p>
Note that this is one of the rare situations where the degrees-of-freedom
parameter to the Student's t distribution is a real number, and not
an integer value.
</p>
<div class="note"><table border="0" summary="Note">
<tr>
<td rowspan="2" align="center" valign="top" width="25"><img alt="[Note]" src="../../../../../../../../../../doc/src/images/note.png"></td>
<th align="left">Note</th>
</tr>
<tr><td align="left" valign="top"><p>
Some statistical packages truncate the effective degrees of freedom
to an integer value: this may be necessary if you are relying on
lookup tables, but since our code fully supports non-integer degrees
of freedom there is no need to truncate in this case. Also note that
when the degrees of freedom is small then the Welch-Satterthwaite
approximation may be a significant source of error.
</p></td></tr>
</table></div>
<p>
Putting these formulae into code we get:
</p>
<pre class="programlisting"><span class="comment">// Degrees of freedom:
</span><span class="keyword">double</span> <span class="identifier">v</span> <span class="special">=</span> <span class="identifier">Sd1</span> <span class="special">*</span> <span class="identifier">Sd1</span> <span class="special">/</span> <span class="identifier">Sn1</span> <span class="special">+</span> <span class="identifier">Sd2</span> <span class="special">*</span> <span class="identifier">Sd2</span> <span class="special">/</span> <span class="identifier">Sn2</span><span class="special">;</span>
<span class="identifier">v</span> <span class="special">*=</span> <span class="identifier">v</span><span class="special">;</span>
<span class="keyword">double</span> <span class="identifier">t1</span> <span class="special">=</span> <span class="identifier">Sd1</span> <span class="special">*</span> <span class="identifier">Sd1</span> <span class="special">/</span> <span class="identifier">Sn1</span><span class="special">;</span>
<span class="identifier">t1</span> <span class="special">*=</span> <span class="identifier">t1</span><span class="special">;</span>
<span class="identifier">t1</span> <span class="special">/=</span> <span class="special">(</span><span class="identifier">Sn1</span> <span class="special">-</span> <span class="number">1</span><span class="special">);</span>
<span class="keyword">double</span> <span class="identifier">t2</span> <span class="special">=</span> <span class="identifier">Sd2</span> <span class="special">*</span> <span class="identifier">Sd2</span> <span class="special">/</span> <span class="identifier">Sn2</span><span class="special">;</span>
<span class="identifier">t2</span> <span class="special">*=</span> <span class="identifier">t2</span><span class="special">;</span>
<span class="identifier">t2</span> <span class="special">/=</span> <span class="special">(</span><span class="identifier">Sn2</span> <span class="special">-</span> <span class="number">1</span><span class="special">);</span>
<span class="identifier">v</span> <span class="special">/=</span> <span class="special">(</span><span class="identifier">t1</span> <span class="special">+</span> <span class="identifier">t2</span><span class="special">);</span>
<span class="identifier">cout</span> <span class="special"><<</span> <span class="identifier">setw</span><span class="special">(</span><span class="number">55</span><span class="special">)</span> <span class="special"><<</span> <span class="identifier">left</span> <span class="special"><<</span> <span class="string">"Degrees of Freedom"</span> <span class="special"><<</span> <span class="string">"= "</span> <span class="special"><<</span> <span class="identifier">v</span> <span class="special"><<</span> <span class="string">"\n"</span><span class="special">;</span>
<span class="comment">// t-statistic:
</span><span class="keyword">double</span> <span class="identifier">t_stat</span> <span class="special">=</span> <span class="special">(</span><span class="identifier">Sm1</span> <span class="special">-</span> <span class="identifier">Sm2</span><span class="special">)</span> <span class="special">/</span> <span class="identifier">sqrt</span><span class="special">(</span><span class="identifier">Sd1</span> <span class="special">*</span> <span class="identifier">Sd1</span> <span class="special">/</span> <span class="identifier">Sn1</span> <span class="special">+</span> <span class="identifier">Sd2</span> <span class="special">*</span> <span class="identifier">Sd2</span> <span class="special">/</span> <span class="identifier">Sn2</span><span class="special">);</span>
<span class="identifier">cout</span> <span class="special"><<</span> <span class="identifier">setw</span><span class="special">(</span><span class="number">55</span><span class="special">)</span> <span class="special"><<</span> <span class="identifier">left</span> <span class="special"><<</span> <span class="string">"T Statistic"</span> <span class="special"><<</span> <span class="string">"= "</span> <span class="special"><<</span> <span class="identifier">t_stat</span> <span class="special"><<</span> <span class="string">"\n"</span><span class="special">;</span>
</pre>
<p>
Thereafter the code and the tests are performed the same as before.
Using are car mileage data again, here's what the output looks like:
</p>
<pre class="programlisting"> __________________________________________________
Student t test for two samples (unequal variances)
__________________________________________________
Number of Observations (Sample 1) = 249
Sample 1 Mean = 20.145
Sample 1 Standard Deviation = 6.4147
Number of Observations (Sample 2) = 79
Sample 2 Mean = 30.481
Sample 2 Standard Deviation = 6.1077
Degrees of Freedom = 136.87
T Statistic = -12.946
Probability that difference is due to chance = 1.571e-025
Results for Alternative Hypothesis and alpha = 0.0500
Alternative Hypothesis Conclusion
Sample 1 Mean != Sample 2 Mean NOT REJECTED
Sample 1 Mean < Sample 2 Mean NOT REJECTED
Sample 1 Mean > Sample 2 Mean REJECTED
</pre>
<p>
This time allowing the variances in the two samples to differ has yielded
a higher likelihood that the observed difference is down to chance
alone (1.571e-025 compared to 5.273e-030 when equal variances were
assumed). However, the conclusion remains the same: US cars are less
fuel efficient than Japanese models.
</p>
</div>
<table xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" width="100%"><tr>
<td align="left"></td>
<td align="right"><div class="copyright-footer">Copyright © 2006 , 2007, 2008, 2009 John Maddock, Paul A. Bristow,
Hubert Holin, Xiaogang Zhang, Bruno Lalande, Johan Råde, Gautam Sewani
and Thijs van den Berg<p>
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at <a href="http://www.boost.org/LICENSE_1_0.txt" target="_top">http://www.boost.org/LICENSE_1_0.txt</a>)
</p>
</div></td>
</tr></table>
<hr>
<div class="spirit-nav">
<a accesskey="p" href="tut_mean_size.html"><img src="../../../../../../../../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../st_eg.html"><img src="../../../../../../../../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../../../../../index.html"><img src="../../../../../../../../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="paired_st.html"><img src="../../../../../../../../../../doc/src/images/next.png" alt="Next"></a>
</div>
</body>
</html>
