Generative VS Discriminative Models

목차

• Generative classifiers

Q. There are two images of animal(lion and elephant). Classify it!!!

 

In General, A Discriminative model ‌models the decision boundary between the classes. A Generative Model ‌explicitly models the actual distribution of each class. In final both of them is predicting the conditional probability P(Animal | Features). But Both models learn different probabilities.

 

A Generative Model ‌learns the joint probability distribution p(x,y). It predicts the conditional probability with the help of Bayes Theorem. A Discriminative model ‌learns the conditional probability distribution p(y|x). Both of these models were generally used in supervised learningproblems.

 

In Math,

‌Training classifiers involve estimating f: X -> Y, or P(Y|X)

Generative classifiers

• Assume some functional form for P(Y), P(X|Y)
• Estimate parameters of P(X|Y), P(Y) directly from training data
• Use Bayes rule to calculate P(Y |X)

Discriminative Classifiers

• Assume some functional form for P(Y|X)
• Estimate parameters of P(Y|X) directly from training data

Examples:

Generative classifiers

• ‌Naïve Bayes
• Bayesian networks
• Markov random fields
• ‌Hidden Markov Models (HMM)

Discriminative Classifiers

• ‌Logistic regression
• Scalar Vector Machine
• ‌Traditional neural networks
• ‌Nearest neighbour
• Conditional Random Fields (CRF)s

Questions:

Ask yourself the following questions to get clear understanding about both these models.

 

 

1. ‌ What are the problems these models can solve?
   Classification tasks.
2. Which model learns joint probability?
   The generative model.
3. ‌Which model learns conditional probability?
   the discriminative model.
4. What happens when we give correlated features in discriminative models?
   They will produce similar output as they are correlated?
5. What happens when we give correlated features in generative models?
   We can’t tell as we do not learn the decision boundaries?
6. Which models works very well even on less training data?
   The discriminative model will work very well as only one set of parameter has to be learned?
7. Is it possible to generate data from with the help of these models?
   Don’t know.
8. Which model will take less time to get trained?
   The discriminative model will take less time as only one set of parameter has to be learned?
9. Which model will take less time to predict output?
   The discriminative model will take less time as prediction can be performed directly by the learned conditional probability. The generative model additionaly needs to apply Bayes’ Theorem.
10. Which model fails to work well if we give a lot of features?
    The generative model?
11. Which model prone to overfitting very easily?
    The generative model?
12. ‌Which model prone to underfitting easily?
    The discriminative model?
13. ‌What happens when training data is biased over one class in Generative Model?
    The joint probability will be biased as class probabilites will be biased. Thus, the conditional probability will be biased, too.
14. What happens when training data is biased over one class in Discriminative Models?
    It happens the same like in the generative case. We will receive biased conditional probability.
15. Which model is more sensitive to outliers?
    The discriminative model is more sensitive to outliers as we model the boundaries. Each outlier will shift the class boundaries.
16. ‌Are you able to fill out the missing values in a dataset with the help of these models?
    Yes — if the dataset offers features.

 

# From https://medium.com/@mlengineer/generative-and-discriminative-models-af5637a66a3