I've always thought from what I read that cross validation is performed like this:

In k-fold cross-validation, the original sample is randomly partitioned into k subsamples. Of the k subsamples, a single subsample is retained as the validation data for testing the model, and the remaining k − 1 subsamples are used as training data. The cross-validation process is then repeated k times (the folds), with each of the k subsamples used exactly once as the validation data. The k results from the folds then can be averaged (or otherwise combined) to produce a single estimation

So k models are built and the final one is the average of those. In Weka guide is written that each model is always built using ALL the data set. So how does cross validation in Weka work ? Is the model built from all data and the "cross-validation" means that k fold are created then each fold is evaluated on it and the final output results is simply the averaged result from folds?

up vote 47 down vote accepted

So, here is the scenario again: you have 100 labeled data

Use training set

  • weka will take 100 labeled data
  • it will apply an algorithm to build a classifier from these 100 data
  • it applies that classifier AGAIN on these 100 data
  • it provides you with the performance of the classifier (applied to the same 100 data from which it was developed)

Use 10 fold CV

  • Weka takes 100 labeled data

  • it produces 10 equal sized sets. Each set is divided into two groups: 90 labeled data are used for training and 10 labeled data are used for testing.

  • it produces a classifier with an algorithm from 90 labeled data and applies that on the 10 testing data for set 1.

  • It does the same thing for set 2 to 10 and produces 9 more classifiers

  • Spring Over Stretch Knee Toe Thigh Dormery Women Boots Shoes High Autumn Platform Heel Black Black High Ladies The Suede New Round it averages the performance of the 10 classifiers produced from 10 equal sized (90 training and 10 testing) sets

Let me know if that answers your question.

  • 1
    I've got 2 question:1) If it's like you said why in Weka Guide is written that in each case (training set and CV) the model is always built from all data? As you wrote in CV the final model is an average of other 10 models, right? 2) If "The model you get at this point is the average of all the 10 models" how is possible that using training set and CV as validation I got same models? (Hope these questions doesn't appear too silly!) –  Titus Pullo May 10 '12 at 18:42
  • 1. This means for every fold full dataset is considered. There are some variations of this standard CV where part of the datasets are held out for a separately test. 2. What exactly do you mean by "getting same models"? –  Rushdi Shams May 11 '12 at 3:20
  • 1
    Have a look at this post link. So the model is exactly the same in each validation option!Let me know if you're agree with me –  Titus Pullo May 12 '12 at 8:04
  • 3
    So, for the community, I am sorry that I did not know that Weka provides you the same model no matter whether you choose trainining set or 10 fold CV. I made necessary corrections to my answers and comments so that nobody is getting the misconception I had previously about Weka though this is usual practice in ML community to report either the best model or the average model from 10 fold CV. I knew that Weka provides the average model but I was completely wrong. Thanks @Lazza87. –  Rushdi Shams May 12 '12 at 23:05
  • 4
    @Lazza87, your link is dead, could you please update it? thanks –  M-T-A Jan 13 '14 at 14:22

I would have answered in a comment but my reputation still doesn't allow me to:

In addition to Rushdi's accepted answer, I want to emphasize that the models which are created for the cross-validation fold sets are all discarded after the performance measurements have been carried out and averaged.

The resulting model is always based on the full training set, regardless of your test options. Since M-T-A was asking for an update to the quoted link, here it is: http://list.waikato.ac.nz/pipermail/wekalist/2009-December/046633.html. It's an answer from one of the WEKA maintainers, pointing out just what I wrote.

Knee The Women Boots Black Dormery Spring Round Thigh Ladies Autumn Shoes Platform High Heel Stretch Toe New High Over Black Suede up vote 5 down vote

I think I figured it out. Take (for example) weka.classifiers.rules.OneR -x 10 -d outmodel.xxx. This does two things:

  1. It creates a model based on the full dataset. This is the model that is written to outmodel.xxx. This model is not used as part of cross-validation.
  2. Then cross-validation is run. cross-validation involves creating (in this case) 10 new models with the training and testing on segments of the data as has been described. The key is the models used in cross-validation are temporary and only used to generate statistics. They are not equivalent to, or used for the model that is given to the user.

Weka follows the conventional k-fold cross validation you mentioned here. You have the full data set, then divide it into k nos of equal sets (k1, k2, ... , k10 for example for 10 fold CV) without overlaps. Then at the first run, take k1 to k9 as training set and develop a model. Use that model on k10 to get the performance. Next comes k1 to k8 and k10 as training set. Develop a model from them and apply it to k9 to get the performance. In this way, use all the folds where each fold at most 1 time is used as test set.

Then Weka averages the performances and presents that on the output pane.

  • 2
    Ok but in this way how is built the final model? Is an average of the 10 models built on CV? If yes what does mean: is always built using ALL the data set? –  Titus Pullo May 4 '12 at 7:47
  • If you select 10 fold cross validation on the classify tab in Weka explorer, then the model you get is the one that you get with 10 9-1 splits. You will not have 10 individual models but 1 single model. And yes, you get that from Weka (not particularly Weka, it is applicable to general 10 fold CV theory) as it runs through the entire dataset. –  Rushdi Shams May 7 '12 at 17:50
  • I'm sorry but I can't understand at all...So what is the differnce between choose from "Use training set" and "Cross Validation" in terms of how the model?The final model is the same! –  Titus Pullo May 8 '12 at 14:46
  • When you are using "Use training set", then if you have 100 instances, Weka uses a "classification algorithm" defined by you to build model from all 100 instances. Then to test, it uses the same 100 instances. Therefore, normally on "Use training set" gives good precision-recall and fmeasure. But when you are using 10 fold CV, then it builds 10 different models with 10 different folds and gives you the average precision-recall-fmeasure. Sometimes it is necessary to use "training set" but most of the cases, 10 fold cv is preferable. FINAL MODEL WITH THESE TWO DIFFERENT SETUPS ARE NEVER SAME. –  Rushdi Shams May 9 '12 at 19:54
  • So tell me if I understood well: Using CV we build 10 models that are "similar" to the real one, that is built using "similar" data and that should permit to evaluate the model using data that should emulate those that will be available in the future? –  Titus PulloW Lines Non Womens Drying Stone Slippers Grain Mens Buteri Slippers Black Quick Slip wxnFq1747C May 10 '12 at 13:30

Your Answer


By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.