Dari perspektif probabilitas Bayesian, mengapa interval kepercayaan 95% tidak mengandung parameter sebenarnya dengan probabilitas 95%?

Dari halaman Wikipedia tentang interval kepercayaan :

... jika interval kepercayaan dibangun di banyak analisis data terpisah dari percobaan yang diulang (dan mungkin berbeda), proporsi interval tersebut yang mengandung nilai sebenarnya dari parameter akan cocok dengan tingkat kepercayaan ...

Dan dari halaman yang sama:

Interval kepercayaan tidak memprediksi bahwa nilai sebenarnya dari parameter memiliki probabilitas tertentu berada dalam interval kepercayaan mengingat data benar-benar diperoleh.

Jika saya memahaminya dengan benar, pernyataan terakhir ini dibuat dengan interpretasi probabilitas yang sering dalam pikiran. Namun, dari perspektif probabilitas Bayesian, mengapa interval kepercayaan 95% tidak mengandung parameter sebenarnya dengan probabilitas 95%? Dan jika tidak, apa yang salah dengan alasan berikut?

Jika saya memiliki proses yang saya tahu menghasilkan jawaban yang benar 95% dari waktu maka probabilitas jawaban berikutnya yang benar adalah 0,95 (mengingat bahwa saya tidak memiliki informasi tambahan mengenai proses tersebut). Demikian pula jika seseorang menunjukkan saya interval kepercayaan yang dibuat oleh suatu proses yang akan berisi parameter benar 95% dari waktu, haruskah saya tidak benar mengatakan bahwa itu berisi parameter benar dengan probabilitas 0,95, mengingat apa yang saya ketahui?

Pertanyaan ini mirip dengan, tetapi tidak sama dengan, Mengapa CI 95% tidak menyiratkan kemungkinan 95% mengandung mean? Jawaban atas pertanyaan itu berfokus pada mengapa 95% CI tidak menyiratkan peluang 95% untuk mengandung rata-rata dari perspektif frequentist. Pertanyaan saya sama, tetapi dari perspektif probabilitas Bayesian.

Pembaruan : Dengan manfaat dari beberapa tahun ke belakang, saya telah menulis perawatan yang lebih ringkas dari bahan yang sama dalam menanggapi pertanyaan serupa.

---

Cara Membangun Wilayah Percaya Diri

Mari kita mulai dengan metode umum untuk membangun wilayah kepercayaan. Ini dapat diterapkan pada parameter tunggal, untuk menghasilkan interval kepercayaan atau serangkaian interval; dan itu dapat diterapkan pada dua atau lebih parameter, untuk menghasilkan wilayah kepercayaan dimensi yang lebih tinggi.

Kami menyatakan bahwa statistik yang diamati $D$ berasal dari distribusi dengan parameter $\theta$ , yaitu distribusi sampling $s(d|\theta)$ atas statistik yang mungkin $d$ , dan mencari wilayah kepercayaan untuk $\theta$ dalam set nilai yang mungkin $\Theta$ . Definisikan Wilayah Kepadatan Tertinggi (HDR): $h$ -HDR dari PDF adalah subset terkecil dari domainnya yang mendukung probabilitas $h$ . Notasikan $h$ -HDR dari $s(d|\psi)$ sebagai $H_\psi$ , untuk setiap$\psi \in \Theta$adalah 0,95. . Kemudian,wilayah kepercayaan$h$ untuk } . Nilai khas h$\theta$ , diberikan data$D$ , adalah himpunan$C_D = \{ \phi : D \in H_\phi \}$$h$

Interpretasi Frekuensi

Dari definisi sebelumnya dari wilayah kepercayaan, ikuti

$$d \in H_\psi \longleftrightarrow \psi \in C_d$$ dengan $C_d = \{ \phi : d \in H_\phi \}$ . Sekarang bayangkan satu set besar ( imajiner ) pengamatan $\{D_i\}$ , yang diambil dalam keadaan mirip dengan $D$ . yaitu Mereka adalah sampel dari $s(d|\theta)$ . Karena $H_\theta$ mendukung probabilitas massa $h$ dari PDF$s(d|\theta)$. ,$P(D_i \in H_\theta) = h$juga h untuk semua$i$ . Oleh karena itu, fraksi$\{D_i\}$ yang$D_i \in H_\theta$ adalah$h$ . Jadi, dengan menggunakan persamaan di atas, fraksi$\{D_i\}$ untuk$\theta \in C_{D_i}$$h$

Maka, inilah yang sering diklaim oleh $h$ wilayah kepercayaan untuk $\theta$ sebesar:

Mengambil sejumlah besar pengamatan imajiner $\{D_i\}$ dari distribusi sampling $s(d|\theta)$ yang memunculkan statistik yang diamati $D$ . Kemudian, $\theta$ terletak dalam fraksi $h$ dari wilayah kepercayaan analog tapi imajiner $\{C_{D_i}\}$ .

Kepercayaan wilayah $C_D$ karena tidak membuat klaim tentang probabilitas bahwa $\theta$ kebohongan di suatu tempat! Alasannya sederhana bahwa tidak ada dalam fomulasi yang memungkinkan kita untuk berbicara tentang distribusi probabilitas lebih dari $\theta$ . Interpretasinya hanyalah suprastruktur yang rumit, yang tidak meningkatkan basis. Basis hanya $s(d | \theta)$ dan $D$ , di mana $\theta$ tidak muncul sebagai kuantitas yang didistribusikan, dan tidak ada informasi yang dapat kita gunakan untuk mengatasinya. Pada dasarnya ada dua cara untuk mendapatkan distribusi lebih dari $\theta$ :

1. Tetapkan distribusi langsung dari informasi yang tersedia: $p(\theta | I)$ .
2. Hubungkan $\theta$ dengan kuantitas terdistribusi lain: $p(\theta | I) = \int p(\theta x | I) dx = \int p(\theta | x I) p(x | I) dx$ .

Dalam kedua kasus, $\theta$ harus muncul di sebelah kiri di suatu tempat. Frequentists tidak dapat menggunakan metode mana pun, karena mereka berdua membutuhkan bidat sebelumnya.

Pandangan Bayesian

Yang paling sebuah Bayesian dapat membuat satu $h$ wilayah kepercayaan diri $C_D$ , diberikan tanpa kualifikasi, hanya interpretasi langsung: bahwa itu adalah himpunan $\phi$ yang $D$ jatuh di $h$ -HDR $H_\phi$ dari distribusi sampling $s(d|\phi)$ . Itu tidak selalu memberi tahu kita banyak tentang $\theta$ , dan inilah sebabnya.

Probabilitas bahwa $\theta \in C_D$ , diberikan $D$ dan informasi latar belakang $I$ , adalah:

$$\begin{align*} P(\theta \in C_D | DI) &= \int_{C_D} p(\theta | DI) d\theta \\ &= \int_{C_D} \frac{p(D | \theta I) p(\theta | I)}{p(D | I)} d\theta \end{align*}$$ Perhatikan bahwa, tidak seperti interpretasi frequentist, kami segera menuntut distribusi lebih dari$\theta$. Informasi latar belakang yang$I$sampaikan kepada kami, seperti sebelumnya, bahwa distribusi sampling adalah$s(d | \theta)$: $$\begin{align*} P(\theta \in C_D | DI) &= \int_{C_D} \frac{s(D | \theta) p(\theta | I)}{p(D | I)} d \theta \\ &= \frac{\int_{C_D} s(D | \theta) p(\theta | I) d\theta}{p(D | I)} \\ \text{i.e.} \quad\quad P(\theta \in C_D | DI) &= \frac{\int_{C_D} s(D | \theta) p(\theta | I) d\theta}{\int s(D | \theta) p(\theta | I) d\theta} \end{align*}$$ Now this expression does not in general evaluate to $h$, which is to say, the $h$ confidence region $C_D$ does not always contain $\theta$ with probability $h$. In fact it can be starkly different from $h$. There are, however, many common situations in which it does evaluate to $h$, which is why confidence regions are often consistent with our probabilistic intuitions.

Sebagai contoh, anggaplah bahwa gabungan PDF sebelumnya dari $d$ dan $\theta$ adalah simetris dalam $p_{d,\theta}(d,\theta | I) = p_{d,\theta}(\theta,d | I)$ . (Jelas ini melibatkan asumsi bahwa rentang PDF atas domain yang sama dalam $d$ dan $\theta$ .) Kemudian, jika sebelumnya adalah $p(\theta | I) = f(\theta)$ , kami memiliki $s(D | \theta) p(\theta | I) = s(D | \theta) f(\theta) = s(\theta | D) f(D)$. Hence

$$\begin{align*} P(\theta \in C_D | DI) &= \frac{\int_{C_D} s(\theta | D) d\theta}{\int s(\theta | D) d\theta} \\ \text{i.e.} \quad\quad P(\theta \in C_D | DI) &= \int_{C_D} s(\theta | D) d\theta \end{align*}$$ From the definition of an HDR we know that for any $\psi \in \Theta$ $$\begin{align*} \int_{H_\psi} s(d | \psi) dd &= h \\ \text{and therefore that} \quad\quad \int_{H_D} s(d | D) dd &= h \\ \text{or equivalently} \quad\quad \int_{H_D} s(\theta | D) d\theta &= h \end{align*}$$ Therefore, given that $s(d | \theta) f(\theta) = s(\theta | d) f(d)$, $C_D = H_D$ implies $P(\theta \in C_D | DI) = h$. The antecedent satisfies $$C_D = H_D \longleftrightarrow \forall \psi \; [ \psi \in C_D \leftrightarrow \psi \in H_D ]$$ Applying the equivalence near the top: $$C_D = H_D \longleftrightarrow \forall \psi \; [ D \in H_\psi \leftrightarrow \psi \in H_D ]$$ Thus, the confidence region $C_D$ contains $\theta$ with probability $h$ if for all possible values $\psi$ of $\theta$, the $h$-HDR of $s(d | \psi)$ contains $D$ if and only if the $h$-HDR of $s(d | D)$ contains $\psi$.

Now the symmetric relation $D \in H_\psi \leftrightarrow \psi \in H_D$ is satisfied for all $\psi$ when $s(\psi + \delta | \psi) = s(D - \delta | D)$ for all $\delta$ that span the support of $s(d | D)$ and $s(d | \psi)$. We can therefore form the following argument:

1. $s(d | \theta) f(\theta) = s(\theta | d) f(d)$ (premise)
2. $\forall \psi \; \forall \delta \; [ s(\psi + \delta | \psi) = s(D - \delta | D) ]$ (premise)
3. $\forall \psi \; \forall \delta \; [ s(\psi + \delta | \psi) = s(D - \delta | D) ] \longrightarrow \forall \psi \; [ D \in H_\psi \leftrightarrow \psi \in H_D ]$
4. $\therefore \quad \forall \psi \; [ D \in H_\psi \leftrightarrow \psi \in H_D ]$
5. $\forall \psi \; [ D \in H_\psi \leftrightarrow \psi \in H_D ] \longrightarrow C_D = H_D$
6. $\therefore \quad C_D = H_D$
7. $[s(d | \theta) f(\theta) = s(\theta | d) f(d) \wedge C_D = H_D] \longrightarrow P(\theta \in C_D | DI) = h$
8. $\therefore \quad P(\theta \in C_D | DI) = h$

Let's apply the argument to a confidence interval on the mean of a 1-D normal distribution $(\mu, \sigma)$, given a sample mean $\bar{x}$ from $n$ measurements. We have $\theta = \mu$ and $d = \bar{x}$, so that the sampling distribution is

$$s(d | \theta) = \frac{\sqrt{n}}{\sigma \sqrt{2 \pi}} e^{-\frac{n}{2 \sigma^2} { \left( d - \theta \right) }^2 }$$ Suppose also that we know nothing about $\theta$ before taking the data (except that it's a location parameter) and therefore assign a uniform prior: $f(\theta) = k$. Clearly we now have $s(d | \theta) f(\theta) = s(\theta | d) f(d)$, so the first premise is satisfied. Let $s(d | \theta) = g\left( (d - \theta)^2 \right)$. (i.e. It can be written in that form.) Then $$\begin{gather*} s(\psi + \delta | \psi) = g \left( (\psi + \delta - \psi)^2 \right) = g(\delta^2) \\ \text{and} \quad\quad s(D - \delta | D) = g \left( (D - \delta - D)^2 \right) = g(\delta^2) \\ \text{so that} \quad\quad \forall \psi \; \forall \delta \; [s(\psi + \delta | \psi) = s(D - \delta | D)] \end{gather*}$$ whereupon the second premise is satisfied. Both premises being true, the eight-point argument leads us to conclude that the probability that $\theta$ lies in the confidence interval $C_D$ is $h$!

We therefore have an amusing irony:

1. The frequentist who assigns the $h$ confidence interval cannot say that $P(\theta \in C_D) = h$, no matter how innocently uniform $\theta$ looks before incorporating the data.
2. The Bayesian who would not assign an $h$ confidence interval in that way knows anyhow that $P(\theta \in C_D | DI) = h$.

Final Remarks

We have identified conditions (i.e. the two premises) under which the $h$ confidence region does indeed yield probability $h$ that $\theta \in C_D$. A frequentist will baulk at the first premise, because it involves a prior on $\theta$, and this sort of deal-breaker is inescapable on the route to a probability. But for a Bayesian, it is acceptable---nay, essential. These conditions are sufficient but not necessary, so there are many other circumstances under which the Bayesian $P(\theta \in C_D | DI)$ equals $h$. Equally though, there are many circumstances in which $P(\theta \in C_D | DI) \ne h$, especially when the prior information is significant.

We have applied a Bayesian analysis just as a consistent Bayesian would, given the information at hand, including statistics $D$. But a Bayesian, if he possibly can, will apply his methods to the raw measurements instead---to the $\{x_i\}$, rather than $\bar{x}$. Oftentimes, collapsing the raw data into summary statistics $D$ destroys information in the data; and then the summary statistics are incapable of speaking as eloquently as the original data about the parameters $\theta$.

from a Bayesian probability perspective, why doesn't a 95% confidence interval contain the true parameter with 95% probability?

Two answers to this, the first being less helpful than the second

1. There are no confidence intervals in Bayesian statistics, so the question doesn't pertain.

2. In Bayesian statistics, there are however credible intervals, which play a similar role to confidence intervals. If you view priors and posteriors in Bayesian statistics as quantifying the reasonable belief that a parameter takes on certain values, then the answer to your question is yes, a 95% credible interval represents an interval within which a parameter is believed to lie with 95% probability.

If I have a process that I know produces a correct answer 95% of the time then the probability of the next answer being correct is 0.95 (given that I don't have any extra information regarding the process).

yes, the process guesses a right answer with 95% probability

Similarly if someone shows me a confidence interval that is created by a process that will contain the true parameter 95% of the time, should I not be right in saying that it contains the true parameter with 0.95 probability, given what I know?

Just the same as your process, the confidence interval guesses the correct answer with 95% probability. We're back in the world of classical statistics here: before you gather the data you can say there's a 95% probability of randomly gathered data determining the bounds of the confidence interval such that the mean is within the bounds.

With your process, after you've gotten your answer, you can't say based on whatever your guess was, that the true answer is the same as your guess with 95% probability. The guess is either right or wrong.

And just the same as your process, in the confidence interval case, after you've gotten the data and have an actual lower and upper bound, the mean is either within those bounds or it isn't, i.e. the chance of the mean being within those particular bounds is either 1 or 0. (Having skimmed the question you refer to it seems this is covered in much more detail there.)

How to interpret a confidence interval given to you if you subscribe to a Bayesian view of probability.

There are a couple of ways of looking at this

1. Technically, the confidence interval hasn't been produced using a prior and Bayes theorem, so if you had a prior belief about the parameter concerned, there would be no way you could interpret the confidence interval in the Bayesian framework.

2. Another widely used and respected interpretation of confidence intervals is that they provide a "plausible range" of values for the parameter (see, e.g., here). This de-emphasises the "repeated experiments" interpretation.

Moreover, under certain circumstances, notably when the prior is uninformative (doesn't tell you anything, e.g. flat), confidence intervals can produce exactly the same interval as a credible interval. In these circumstances, as a Bayesianist you could argue that had you taken the Bayesian route you would have gotten exactly the same results and you could interpret the confidence interval in the same way as a credible interval.

I'll give you an extreme example where they are different.

Suppose I create my 95% confidence interval for a parameter $\theta$ as follows. Start by sampling the data. Then generate a random number between $0$ and $1$. Call this number $u$. If $u$ is less than $0.95$ then return the interval $(-\infty,\infty)$. Otherwise return the "null" interval.

Now over continued repititions, 95% of the CIs will be "all numbers" and hence contain the true value. The other 5% contain no values, hence have zero coverage. Overall, this is a useless, but technically correct 95% CI.

The Bayesian credible interval will be either 100% or 0%. Not 95%.

"from a Bayesian probability perspective, why doesn't a 95% confidence interval contain the true parameter with 95% probability? "

In Bayesian Statistics the parameter is not a unknown value, it is a Distribution. There is no interval containing the "true value", for a Bayesian point of view it does not even make sense. The parameter it's a random variable, you can perfectly know the probability of that value to be between x_inf an x_max if you know the distribuition. It's just a diferent mindset about the parameters, usually Bayesians used the median or average value of the distribuition of the parameter as a "estimate". There is not a confidence interval in Bayesian Statistics, something similar is called credibility interval.

Now from a frequencist point of view, the parameter is a "Fixed Value", not a random variable, can you really obtain probability interval (a 95% one) ? Remember that it's a fixed value not a random variable with a known distribution. Thats why you past the text :"A confidence interval does not predict that the true value of the parameter has a particular probability of being in the confidence interval given the data actually obtained."

The idea of repeating the experience over and over... is not Bayesian reasoning it's a Frequencist one. Imagine a real live experiment that you can only do once in your life time, can you/should you built that confidence interval (from the classical point of view )?.

But... in real life the results could get pretty close ( Bayesian vs Frequencist), maybe thats why It could be confusing.