ロジスティック回帰と尤度を理解する

パラメーター推定/ロジスティック回帰のトレーニングは実際にどのように機能しますか？これまでに手に入れたものを入れようとします。

出力はyであり、xの値に応じた確率の形でのロジスティック関数の出力： $P (y = 1 | x) = \frac{1}{1 + e^{- ω^{T} x}} \equiv σ (ω^{T} x)$ $P(y=1|x)={1\over1+e^{-\omega^Tx}}\equiv\sigma(\omega^Tx)$ $P (y = 0 | x) = 1 - P (y = 1 | x) = 1 - \frac{1}{1 + e^{- ω^{T} x}}$ $P(y=0|x)=1-P(y=1|x)=1-{1\over1+e^{-\omega^Tx}}$
1つの次元について、いわゆるオッズは次のように定義されます $\frac{p (y = 1 | x)}{1 - p (y = 1 | x)} = \frac{p (y = 1 | x)}{p (y = 0 | x)} = e^{ω_{0} + ω_{1} x}$ ${{p(y=1|x)}\over{1-p(y=1|x)}}={{p(y=1|x)}\over{p(y=0|x)}}=e^{\omega_0+\omega_1x}$
次にlog、線形形式でW_0およびW_1を取得する関数を追加します $L o g i t (y) = l o g (\frac{p (y = 1 | x)}{1 - p (y = 1 | x)}) = ω_{0} + ω_{1} x$ $Logit(y)=log({{p(y=1|x)}\over{1-p(y=1|x)}})=\omega_0+\omega_1x$
さて問題の部分へ 尤度の使用（Big X is y） $L (X | P) = \prod_{i = 1, y_{i} = 1}^{N} P (x_{i}) \prod_{i = 1, y_{i} = 0}^{N} (1 - P (x_{i}))$ $L(X|P)=\prod^N_{i=1,y_i=1}P(x_i)\prod^N_{i=1,y_i=0}(1-P(x_i))$ y = 1の確率を2回考慮している理由は誰にもわかりますか？以来： $P (y = 0 | x) = 1 - P (y = 1 | x)$ $P(y=0|x)=1-P(y=1|x)$

そして、それからωの値をどのように取得しますか

regression logistic likelihood

— エンジン
ソース

回答:

一般的に、フォームのモデルを取ることにしたと仮定します

P (y = 1 | X = x) = h (x; Θ)

$P(y=1|X=x) = h(x;\Theta)$

いくつかのパラメータ。次に、その可能性を書き留めます。つまり、 $\Theta$

L (Θ) = \prod_{i \in {1, . . ., N}, y_{i} = 1} P (y = 1 | x = x; Θ) \cdot \prod_{i \in {1, . . ., N}, y_{i} = 0} P (y = 0 | x = x; Θ)

$L(\Theta) = \prod_{i \in \{1, ..., N\}, y_i = 1} P(y=1|x=x;\Theta) \cdot \prod_{i \in \{1, ..., N\}, y_i = 0} P(y=0|x=x;\Theta)$

と同じです

L (Θ) = \prod_{i \in {1, . . ., N}, y_{i} = 1} P (y = 1 | x = x; Θ) \cdot \prod_{i \in {1, . . ., N}, y_{i} = 0} (1 - P (y = 1 | x = x; Θ))

$L(\Theta) = \prod_{i \in \{1, ..., N\}, y_i = 1} P(y=1|x=x;\Theta) \cdot \prod_{i \in \{1, ..., N\}, y_i = 0} (1-P(y=1|x=x;\Theta))$

今、あなたは「仮定する」ことに決めました（モデル）

P (y = 1 | X = x) = σ (Θ_{0} + Θ_{1} x)

$P(y=1|X=x) = \sigma(\Theta_0 + \Theta_1 x)$

ここで、

σ (z) = 1 / (1 + e^{- z})

$\sigma(z) = 1/(1+e^{-z})$

あなただけの可能性のための式を計算し、見つけるために最適化アルゴリズムのいくつかの種類を行うよう $\text{argmax}_\Theta L(\Theta)$ 、例えば、ニュートン法または他の勾配ベースの方法を。

Notce that sometimes, people say that when they are doing logistic regression they do not maximize a likelihood (as we/you did above) but rather they minimize a loss function

l (Θ) = - \sum_{i = 1}^{N} y_{i} \log (P (Y_{i} = 1 | X = x; Θ)) + (1 - y_{i}) \log (P (Y_{i} = 0 | X = x; Θ))

$l(\Theta) = -\sum_{i=1}^N{y_i\log(P(Y_i=1|X=x;\Theta)) + (1-y_i)\log(P(Y_i=0|X=x;\Theta))}$

but notice that $-\log(L(\Theta)) = l(\Theta)$ .

This is a general pattern in Machine Learning: The practical side (minimizing loss functions that measure how 'wrong' a heuristic model is) is in fact equal to the 'theoretical side' (modelling explicitly with the $P$ -symbol, maximizing statistical quantities like likelihoods) and in fact, many models that do not look like probabilistic ones (SVMs for example) can be reunderstood in a probabilistic context and are in fact maximizations of likelihoods.

— Fabian Werner
ソース

@Werner thanks for your answer. But I still need a bit of clarification.1st can you please explain what on earth the 2

\prod

$\prod$ stay for in the definition of

L (θ)

$L(\theta)$ since as far I understood it I'm interessted in the case of

y_{i} = 1

$y_i =1$ . and how can get the values of

ω_{1}

$\omega_1$ and

ω_{0}

$\omega_0$ thanks a lot dor your help !

— Engine

@Engine: The big 'pi' is a product... like a big Sigma

Σ

$\Sigma$ is a sum... do you understand or do you need more clarification on that as well? On the second question: Lets say we want to minimize a function

f (x) = x^{2}

$f(x) = x^2$ and we start at

x = 3

$x=3$ but let us assume that we do not know/can not express / can not visualize

f

$f$ as it is to complicated. Now the derivative of

f

$f$ is

f^{'} = 2 x

$f' = 2x$ . Interestingly if we are right from the minimum

x = 0

$x=0$ it points to the right and if we are left of it it points left. Mathematically the derivative points into the direction of the 'strongest ascend'

— Fabian Werner

@Engine: In more dimensions you replace the derivative by the gradient, i.e. you start off at a random point

x_{0}

$x_0$ and compute the gradient

\partial f

$\partial f$ at

x

$x$ and if you want to maximize then your next point

x_{1}

$x_1$ is

x_{1} = x_{0} + \partial f (x_{0})

$x_1 = x_0 + \partial f(x_0)$ . Then you compute

\partial f (x_{1})

$\partial f(x_1)$ and you next

x

$x$ is

x_{2} = x_{1} + \partial f (x_{1})

$x_2 = x_1 + \partial f(x_1)$ and so forth. This is called gradient ascend/descent and is the most common technique in maximizing a function. Now you do that with

L (Θ)

$L(\Theta)$ or in your notation

L (ω)

$L(\omega)$ in order to find the

ω

$\omega$ that maxeimizes

L

$L$

— Fabian Werner

@Engine: You are not at all interested in the case

y = 1

$y=1$ ! You are interested in 'the'

ω

$\omega$ that 'best explains your data'. From thet

ω

$\omega$ aou let the model 'speak for itself' and get back to the case of

y = 1

$y=1$ but first of all you need to setup a model! Here, 'best explains' means 'having the highest likelihood' because that is what people came up with (and I think it is very natural)... however, there are other metrics (different loss functions and so on) that one could use! There are two products because we want the model to explain the

y = 1

$y=1$ as well as the

y = 0

$y=0$ 'good'!

— Fabian Werner

Your likelihood function (4) consists of two parts: the product of the probability of success for only those people in your sample who experienced a success, and the product of the probability of failure for only those people in your sample who experienced a failure. Given that each individual experiences either a success or a failure, but not both, the probability will appear for each individual only once. That is what the $, y_i=1$ and $,y_i=0$ mean at the bottom of the product signs.

The coefficients are included in the likelihood function by substituting (1) into (4). That way the likelihood function becomes a function of $\omega$ . The point of maximum likelihood is to find the $\omega$ that will maximize the likelihood.

— Maarten Buis
ソース

thanks so much for your answer, sorry but still don't get it. isn't

y_{i} = 0

$y_i = 0$ means the probability that y =0[Don't occure] for all y's of the product. and vis versa for y_i=1. And still after the subtitutiing of how can I find

ω

$\omega$ values, caclulating the 2nd derivative ? or gradient ? thanks a lot for your help !

— Engine

\prod_{i = 1, y = 1}^{N}

$\prod_{i=1, y=1}^N$ should be read as "product for persons

i = 1

$i=1$ till

N

$N$ , but only if

y = 1

$y=1$ . So the first part only applies to those persons in your data that experienced the event. Similarly, the second part only refers to persons who did not experienced the event.

— Maarten Buis

There are many possible algorithms for maximizing the likelihood function. The most common one, the Newton-Raphson method, indeed involves computing the first and second derivatives.

— Maarten Buis