st: R: R: bootstrapped p-values

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

st: R: R: bootstrapped p-values

Carlo Lazzaro
Dear Richard,
I suppose that confusion arose because you are implicitly referring to two
different things:

- the meaning of p-value reported in bootstrap printout (and Nick's
yesterday reply still holds):

- the bootstrap p-value vs the original one in searching for the
significance of the difference of the mean with skewed data.
As widely reported in literature (please see for instance, Desgagne A,
Castilloux A, Angers J, LeLorier J. The use of the bootstrap statistical
method for the pharmacoeconomic cost analysis of skewed data.
Pharmacoeconomics 1998; 13:487-497.Barber JA and Thompson SG. Analysis of
cost data in randomized trials: an application of the non-parametric
bootstrap. Statistics in Medicine 2000; 19:3219-3236), ttest procedure needs
the fulfilments of some prerequisite before being invoked, which are
systematically violated whenever data are skewed (please, take a look at the
following example sketched via Stata 9.2/SE):
---------------------------begin example---------------------------
sysuse auto
save "C:\Documents and
Settings\carlo\Documenti\Statistiche\Stata\Richard_1.dta", replace
ttest mpg, by(foreign) unequal
scalar tobs=r(t)
sum mpg, mean
scalar omean=r(mean)
sum mpg if foreign==0, mean
replace mpg = mpg-r(mean) + scalar(omean)  if foreign==0
sum mpg if foreign==1, mean
replace mpg = mpg-r(mean) + scalar(omean)  if foreign==1
sort foreign
by foreign: sum mpg
keep mpg foreign
set seed 1
bootstrap r(t), reps(1000) nodots strata(foreign) saving(C:\Documents and
Settings\carlo\Documenti\Statistiche\Stata\Richard_2.dta, every(1)
replace)verbose : ttest mpg, by(foreign) unequal
save "C:\Documents and
Settings\carlo\Documenti\Statistiche\Stata\Richard_1.dta", replace
use "C:\Documents and
Settings\carlo\Documenti\Statistiche\Stata\Richard_2.dta", clear
generate indicator =abs(r(t))>=abs(scalar(tobs))
sum indicator, mean
display "ASLboot="r(mean)
------------------------------end example----------------------------

HTH, Kind Regards and enjoy your W-E,

Carlo
-----Messaggio originale-----
Da: Richard Harvey [mailto:[hidden email]]
Inviato: venerdì 5 dicembre 2008 23.37
A: Carlo Lazzaro
Oggetto: Re: R: bootstrapped p-values

hi,

thanks nick and carlo..

I understand your point about the p value in the statout now. I am a
bit confused about some things i have read elsewhere

for example  the following

http://compdiag.molgen.mpg.de/ngfn/docs/2003/nov/Exercises_Bootstrap.pdf

discusses non parametric bootstrap on page 4 and on page 5 it talks about

"The two-sided p-values are now calculated by determining the
percentage of situation where the
absolute value of the test statistics calculated from the groups with
permutated values is at least
as large as the absolute value of the test statistics calculated from
the original groups."..

and the following page

westfall.ba.ttu.edu/Finance%20Pres.ppt

talks about something similar on page 11 and page 12

I am not what that means and how that is related to the stata p-value?
and the following page

westfall.ba.ttu.edu/Finance%20Pres.ppt

talks about something similar on page 11 and page 12

I am not what that means and how that is related to the stata p-value?


thanks
rich

2008/12/5 Carlo Lazzaro <[hidden email]>:
> Dear Richard,
> I would refer you to Stata Archives and recommend to take a look at
Maarten
> Buis' and Martin Weiss' kind replies to one thread I have posted on 2008
Sep

> 6 (which was not so different from yours, as it would seem):
>
> st: z statistics in bootstrap output
>
> Kind Regards and enjoy your W-E,
> Carlo
> -----Messaggio originale-----
> Da: [hidden email]
> [mailto:[hidden email]] Per conto di Richard Harvey
> Inviato: venerdì 5 dicembre 2008 18.40
> A: [hidden email]
> Oggetto: st: bootstrapped p-values
>
> Hi,
>
> Could someone please explain how stata computes the bootstrap p-values?
>
> suppose i issue the following commands
>
> sysuse auto
> bootstrap t=r(t), rep(1000)  saving(bsauto, replace): ttest mpg==0
>
> i get
>
>
> Bootstrap results                                               Number
> of obs=        74
> Replications       =      1000
>
>
> Observed   Bootstrap            Normal-based
> Coef.   Std. Err.       z       P>z     [95% Conf. Interval]
>
> t    31.66644   2.900212        10.92   0.000     25.98213    37.35075
>
>
> how does stata compute the P>z?
>
> does it...
>
> look at the proportion of bootstrapped t-values which are less than or
> equal to the original t-value ( from
> ttest mpg==0) say x and the proportion of bootstrapped t-values which
> are greater than or equal to the original t-value ( from ttest mpg==0)
> say y then computes p as 2 min(x,y) ?
>
> --
> thanks for your time
> rich
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/statalist/faq
> *   http://www.ats.ucla.edu/stat/stata/
>
>
>



--
thanks for your time
rich



*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/
Reply | Threaded
Open this post in threaded view
|

Re: st: R: R: bootstrapped p-values

Richard Harvey
hi,

carlo thanks for your effort...

I am still confused by your example..
especially this bit of the code..

generate indicator =abs(r(t))>=abs(scalar(tobs))
sum indicator, mean
display "ASLboot="r(mean)

i understand what it is doing is strong the original value of t-stats
as a scalar and then after the bootrap process creating a count of the
number of bootstrapped t-values greater than the original t and taking
the mean of that number

However, this always turns out to be 1 when i run the code even when i
run it on different samples ..so what does that mean??? how is that
related to the p-value.. sorry if tis is really dumb but i have got
myself into a mental knot with this.

thanks
rich


2008/12/6 Carlo Lazzaro <[hidden email]>:

> Dear Richard,
> I suppose that confusion arose because you are implicitly referring to two
> different things:
>
> - the meaning of p-value reported in bootstrap printout (and Nick's
> yesterday reply still holds):
>
> - the bootstrap p-value vs the original one in searching for the
> significance of the difference of the mean with skewed data.
> As widely reported in literature (please see for instance, Desgagne A,
> Castilloux A, Angers J, LeLorier J. The use of the bootstrap statistical
> method for the pharmacoeconomic cost analysis of skewed data.
> Pharmacoeconomics 1998; 13:487-497.Barber JA and Thompson SG. Analysis of
> cost data in randomized trials: an application of the non-parametric
> bootstrap. Statistics in Medicine 2000; 19:3219-3236), ttest procedure needs
> the fulfilments of some prerequisite before being invoked, which are
> systematically violated whenever data are skewed (please, take a look at the
> following example sketched via Stata 9.2/SE):
> ---------------------------begin example---------------------------
> sysuse auto
> save "C:\Documents and
> Settings\carlo\Documenti\Statistiche\Stata\Richard_1.dta", replace
> ttest mpg, by(foreign) unequal
> scalar tobs=r(t)
> sum mpg, mean
> scalar omean=r(mean)
> sum mpg if foreign==0, mean
> replace mpg = mpg-r(mean) + scalar(omean)  if foreign==0
> sum mpg if foreign==1, mean
> replace mpg = mpg-r(mean) + scalar(omean)  if foreign==1
> sort foreign
> by foreign: sum mpg
> keep mpg foreign
> set seed 1
> bootstrap r(t), reps(1000) nodots strata(foreign) saving(C:\Documents and
> Settings\carlo\Documenti\Statistiche\Stata\Richard_2.dta, every(1)
> replace)verbose : ttest mpg, by(foreign) unequal
> save "C:\Documents and
> Settings\carlo\Documenti\Statistiche\Stata\Richard_1.dta", replace
> use "C:\Documents and
> Settings\carlo\Documenti\Statistiche\Stata\Richard_2.dta", clear
> generate indicator =abs(r(t))>=abs(scalar(tobs))
> sum indicator, mean
> display "ASLboot="r(mean)
> ------------------------------end example----------------------------
>
> HTH, Kind Regards and enjoy your W-E,
>
> Carlo
> -----Messaggio originale-----
> Da: Richard Harvey [mailto:[hidden email]]
> Inviato: venerdì 5 dicembre 2008 23.37
> A: Carlo Lazzaro
> Oggetto: Re: R: bootstrapped p-values
>
> hi,
>
> thanks nick and carlo..
>
> I understand your point about the p value in the statout now. I am a
> bit confused about some things i have read elsewhere
>
> for example  the following
>
> http://compdiag.molgen.mpg.de/ngfn/docs/2003/nov/Exercises_Bootstrap.pdf
>
> discusses non parametric bootstrap on page 4 and on page 5 it talks about
>
> "The two-sided p-values are now calculated by determining the
> percentage of situation where the
> absolute value of the test statistics calculated from the groups with
> permutated values is at least
> as large as the absolute value of the test statistics calculated from
> the original groups."..
>
> and the following page
>
> westfall.ba.ttu.edu/Finance%20Pres.ppt
>
> talks about something similar on page 11 and page 12
>
> I am not what that means and how that is related to the stata p-value?
> and the following page
>
> westfall.ba.ttu.edu/Finance%20Pres.ppt
>
> talks about something similar on page 11 and page 12
>
> I am not what that means and how that is related to the stata p-value?
>
>
> thanks
> rich
>
> 2008/12/5 Carlo Lazzaro <[hidden email]>:
>> Dear Richard,
>> I would refer you to Stata Archives and recommend to take a look at
> Maarten
>> Buis' and Martin Weiss' kind replies to one thread I have posted on 2008
> Sep
>> 6 (which was not so different from yours, as it would seem):
>>
>> st: z statistics in bootstrap output
>>
>> Kind Regards and enjoy your W-E,
>> Carlo
>> -----Messaggio originale-----
>> Da: [hidden email]
>> [mailto:[hidden email]] Per conto di Richard Harvey
>> Inviato: venerdì 5 dicembre 2008 18.40
>> A: [hidden email]
>> Oggetto: st: bootstrapped p-values
>>
>> Hi,
>>
>> Could someone please explain how stata computes the bootstrap p-values?
>>
>> suppose i issue the following commands
>>
>> sysuse auto
>> bootstrap t=r(t), rep(1000)  saving(bsauto, replace): ttest mpg==0
>>
>> i get
>>
>>
>> Bootstrap results                                               Number
>> of obs=        74
>> Replications       =      1000
>>
>>
>> Observed   Bootstrap            Normal-based
>> Coef.   Std. Err.       z       P>z     [95% Conf. Interval]
>>
>> t    31.66644   2.900212        10.92   0.000     25.98213    37.35075
>>
>>
>> how does stata compute the P>z?
>>
>> does it...
>>
>> look at the proportion of bootstrapped t-values which are less than or
>> equal to the original t-value ( from
>> ttest mpg==0) say x and the proportion of bootstrapped t-values which
>> are greater than or equal to the original t-value ( from ttest mpg==0)
>> say y then computes p as 2 min(x,y) ?
>>
>> --
>> thanks for your time
>> rich
>> *
>> *   For searches and help try:
>> *   http://www.stata.com/help.cgi?search
>> *   http://www.stata.com/support/statalist/faq
>> *   http://www.ats.ucla.edu/stat/stata/
>>
>>
>>
>
>
>
> --
> thanks for your time
> rich
>
>
>
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/statalist/faq
> *   http://www.ats.ucla.edu/stat/stata/
>



--
thanks for your time
rich

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/
Reply | Threaded
Open this post in threaded view
|

Re: st: R: R: bootstrapped p-values

Carlo Lazzaro
In reply to this post by Carlo Lazzaro
Dear Richard,
your concern is conceivable, since to my real shame I have misreported two
bits of the code. Sorry for this.
I do hope that everything will work fine this time.
Kind Regards and have a Nice Sunday,
Carlo

---------------------------begin example---------------------------
sysuse auto
save "C:\Documents and
Settings\carlo\Documenti\Statistiche\Stata\Richard_1.dta", replace ttest
mpg, by(foreign) unequal scalar tobs=r(t) sum mpg, mean scalar omean=r(mean)
sum mpg if foreign==0, mean replace mpg = mpg-r(mean) + scalar(omean)  if
foreign==0 sum mpg if foreign==1, mean replace mpg = mpg-r(mean) +
scalar(omean)  if foreign==1 sort foreign by foreign: sum mpg keep mpg
foreign set seed 1 bootstrap t=r(t), reps(1000) nodots strata(foreign)
saving(C:\Documents and
Settings\carlo\Documenti\Statistiche\Stata\Richard_2.dta, every(1)
replace)verbose : ttest mpg, by(foreign) unequal save "C:\Documents and
Settings\carlo\Documenti\Statistiche\Stata\Richard_1.dta", replace use
"C:\Documents and Settings\carlo\Documenti\Statistiche\Stata\Richard_2.dta",
clear generate indicator =abs(t)>=abs(scalar(tobs)) sum indicator, mean
display "ASLboot="r(mean)
------------------------------end example----------------------------


*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/