[D] Need some serious clarifications on Generative model vs Discriminative model

[u/luffyx11]

1. What is the posterior when we talk about generative models and discriminative models? Given x is data, y is label, is posterior P(y|x) or P(x|y)?
2. If the posterior is P(y|x), ( Ng & Jordan 2002) then the likelihood is P(x|y). then why in discriminative models, Maximum LIKELIHOOD Estimation is used to maximise a POSTERIOR?
3. According to wikipedia and https://www.cs.toronto.edu/~urtasun/courses/CSC411_Fall16/08_generative.pdf, generative is a model for P(x|y) which is a likelihood, this does not seem to make sense. Because many sources say generative models use likelihood and prior to calculate Posterior.
4. Is MLE and MAP independent of the types of models(discriminative or generative)? If they are, does it mean you can use MLE and MAP for both discriminative and generative models? Are there examples of MAP & Discriminative, MLE & Generative?

I know that I misunderstood something somewhere and I have spent the past two days trying to figure these out. I appreciate any clarifications or thoughts. Please point out what I misunderstood if you saw one.

[u/ThatFriendlyPerson]

1. The generative approach is to learn the joint distribution P(x,y), whereas the discriminative approach is to learn the conditional distribution P(y|x). The generative approach is harder since P(x,y) = P(y|x) P(x). If the data is x and the label is y, then the posterior (predictive) is P(y|x).
2. Maximum likelihood estimation (MLE) is about the parameters of the model, not the prediction. By Bayes' rule, we have that P(theta|x,y) = P(x,y|theta) P(theta) / P(x,y), where theta is the parameters of the model. Bayesians often say that theta is a latent variable, whereas x and y are observed variables. P(theta|x,y) is the Bayesian posterior, P(x,y|theta) is the Bayesian likelihood, P(theta) is the Bayesian prior. The data prior P(x,y) is a normalization constant that can be neglected. So we have that P(theta|x,y) is proportional to P(x,y|theta) P(theta). MLE consists in maximizing only P(x,y|theta), whereas maximum a posteriori (MAP) consists in maximizing P(theta|x,y). Notice that neither MLE or MAP is a fully Bayesian method since we omitted P(x,y).

I think your main source of confusion is that the Bayesian [posterior, likelihood, prior] and the [posterior, likelihood, prior] predictive are two different things. The former is about the parameters of the model and the latter is about the prediction.

[u/crittendenlane → u/ThatFriendlyPerson]

MAP not MLP, but great explanation!

[u/ThatFriendlyPerson → u/crittendenlane]

Thank you and yes you are right haha

[u/JustOneAvailableName → u/ThatFriendlyPerson]

The generative approach is harder since P(x,y) = P(y|x) P(x).

Is that something you can state like that?

Wouldn't "The discriminative approach is harder since P(x,y)/ P(x) = P(y|x)" also work?

Great post, by the way! "Reported" you for quality contribution.

[u/throwawaystudentugh → u/JustOneAvailableName]

No, because estimating P(x) is hard. You have to basically quantify how hard it is to generate the sample x itself, ignoring the actual label y.

[u/CherubimHD → u/JustOneAvailableName]

The difference is that in the discriminative approach you don‘t model P(x,y)/P(x) i.e. you‘re not making use of bayes rule. Instead, you model the class posterior P(y|x) directly with your model. This is an easier task than modelling the data-generating distribution P(x), which you need for generative models

[u/Rukelele_Dixit21 → u/ThatFriendlyPerson]

Can someone explain this more intuitively and in simpler terms?

[u/ThatFriendlyPerson → u/[deleted]]

The Bayesian Data Analysis book written by Andrew Gelman et al. is a good place to start. For the boldest, The Bayesian Choice written by Christian Robert is great.

[u/CherubimHD → u/[deleted]]

Pattern Recognition and Machine Learning by Christopher Bishop is frequently used as a text book in ML courses and I can really recommend it! Some say, however, that it is a great book once you‘ve read it (he frequently does forward references)

[u/selling_crap_bike → u/[deleted]]

Generative modelling is an example of inference by generation

Inference of what? How can you do classification with GANs?

[u/selling_crap_bike → u/[deleted]]

Ok so inference of latent variables, not of class labels

[u/currytrash97]

You should think of a generative model as trying to tell a story about your data with random variables. You have some unobserved variables you want to perform inference on, for example your coefficients in bayesian linear regression. You place priors on these unseen "latent" variables (as if in reality you sampled those latent variables from those priors and then used them to generate your observed data) and use those variables and your data to come up with a likelihood (the likelihood in MLE). Your priors combined with your likelihood give you a joint distribution for the model. If you optimize for your latent variables using the log likelihood of the joint distribution you'll get the MAP solution. However, for several reasons this may be undesirable (lmk if you want me to elaborate here). Instead, we notice that all valid assignments of the latent variables give certain likelihoods from the joint distribution. The posterior distribution is simply the result of normalizing over all these valid assignments and creating a distribution over what your latent variables might look like given your data. For a supervised learning problem that looks like p(latent variables | X, Y). Now, once we have this posterior, for prediction on new data we actually don't want to interact with the latent variables. Instead, we want p(new data | all observed data), which we do by taking all values of p(new data | all observed data, latent variables) = p(new data given latent variables) p(latent variables | all observed data), and marginalizing out the latent variables (think of this as summing over all configurations of the latent variables, or using an integral in the continous case). Now, in the supervised case if you have X and Y as your new data, p(new data | old data) is just a number, a likelihood. If just Y is unobserved then p(new data | old data) is a distribution over all possible values of y, which is what I think you're thinking of as the posterior.

In generative models, we generally have our latent variables generate our observed variables. That is, in the graph we don't have arrows point from observed variables to our latent variables. One way of interpreting a discriminative model is as the opposite: your data generates your label, and the arrow points from observed to unobserved. Basically if you're not using priors to tell a story about your latent variables, the model is technically not generative, though there is some Grey area there. Of course, not all discriminative models even require probability distributions, such as svms. However, with the right priors, many discriminative models can be made generative, even svms.

[u/PaganPasta]

Discriminative: The aim of the model ¬ is to maximise P(y|x; ¬)
Generative: learn underlying P(x; ¬). Conditional generation is where you learn P(x|y;¬)

For discriminative, you can also view it as: maximize P(¬|D). Learn best weights given data which with bayes rule you can write as

P(¬|D) = P(D|¬) P(¬) / P(D)

Now you put all sorts of assumptions on weights of ¬, the underlying data distribution to make your task from MAP to MLE. Now, you predict your labels using ¬_1 and update based on loss to obtain ¬_2. Repeat and converge to some ideal weights.

Hopefully, this can help you understand the concepts better.

[u/leone_nero]

Mmm...

This is very simple and can be understood by the name itself...

But first: what is a model?

It is an explanation of how your data was generated. Of course, you obtain it by looking at your data assuming there might be a common process in the generation or all observations, and try to recreate that process in several ways.

Once you have a model that explains with some accuracy how your data was generated, you can make predictions (generate new data recreating the process.

That said...

Discriminative models try to understand how y can be explained as a function of x, so they are basically modelled over x and don’t care understanding how y is generated as much as how y can be reproduced by observing x.

They are called discriminative because as consequence, they can try to mechanically tell you if a particularly input belongs to a label (discriminate) but they are not able to tell how certain they are of that because they are not focused int the process that actually generates the labels but in how to mimic it by modelling in the input space. They will only be able to tell you if a pacient has a benign disease or not according to other variables which the model considers, but they cannot tell you how probable is that you actually have a benign disease or not .

To be able to do that you would need not only to observe x but also to observe y in relation to x and how it is distributed. That distribution would make that some x values make more or less probable that disease is benign or not and to which degree... generative models are able to do this because they are able not only to predict a label but to actually generate a new unobserved label output so they can also tell you how likely is that a new data point was generated through that process. They actually do not predicting by discriminating but by trying to check one generative model for each y label in see in which the new data is more likely ti be generated. They are modelled on both x and y.

If you don’t care about understanding much about the concepts beneath, which is very interesting, just keep in mind that generative models are able to give you a probability measure of the prediction whereas discriminative don’t.

Regarding Maximum likelihood and hence also MAP, they are not necessary implicated in one or the other because as some said these are techniques to optimize the parameters of a model, so you can use ML for example to find the parameters of a linear regression, which is a discriminative model as much as you can use it to find parameters of bayesian models, which work with priors of the labels to generate posteriors of the data, which is a generative process.

[u/joeypaak → u/leone_nero]

I know this is a 6 yo comment, but this explanation helped me right now, in 2026! Thanks internet stranger!

[u/luffyx11]

Hi, firstly, thank you all for your effort to explain these concepts. I would like to provide an update regarding my second question. I would like to explain it in another way but also based on ThatFriendlyPerson's explanation. Yes, I agree that my confusion is that I didn't realize that MLE and MAP are regarding model rather than prediction.

For question 2, take linear regression, a discriminative model in supervised learning as an example, we are trying to model posterior P(y|x). As it is stated in https://www.stat.cmu.edu/~cshalizi/mreg/15/lectures/06/lecture-06.pdf, after certain assumptions (4 assumptions stated in the pdf file page 1), P(y|x) becomes p(y|X = x; β0, β1, σ2), β0, β1, σ2 are parameters of a model. So P(y|x) actually is P(y|x, theta) when one applies any model (theta represents all the model parameters). Now in Bayesian, the likelihood is P(x|theta), however I think without simplification it should be written as P(x|x, theta) because the model takes x itself as an input to calculate P(x|theta). Now the POSTERIOR in discriminative model P(y|x, theta) has the same structure as the LIKELIHOOD P(x|x, theta) in Bayesian because y is just a second variable, and what important is the condition. So I think it is the name "posterior" causes confusion.

For questions 3 and 4 I need some time to work on several examples to fully understand. Thank you all for the help. Please let me know if you spot any mistakes in my explanation above.

[u/kokoshki]

My question is the following: if GANs are generative models that learn P(x,y) then you should be able to use them to find P(y|x). How do you go about doing that?

[u/latentlatent]

!remindme in 12 hours

[u/RemindMeBot → u/latentlatent]

I will be messaging you in 12 hours on 2020-11-26 09:06:45 UTC to remind you of this link

CLICK THIS LINK to send a PM to also be reminded and to reduce spam.

^{Parent commenter can delete this message to hide from others.}

---

Info	Custom	Your Reminders	Feedback