Cross Validation - K fold practicals - Part 2

If i have 10000 rows of data, then if k=5, what will be the test and train data in each fold?

What will be the effect of increasing k in K-fold? 

The variation across the training set will decrease

The variation across the training set will increase

The variation across the training set will be zero

The variation across the training set will be maximum

The correct answer is: The variation across the training set will decrease

Here's why:

Increasing the value of k means each training set in K-Fold validation will include more data points from the original dataset.

As the size of the training set in each fold gets closer to the size of the entire dataset, the differences (variation) between these training sets will become smaller. They will all be large subsets of the same overall data.

Think of it this way:

If k=2 (two folds), one training set has half the data, and the other has the other half. There can be significant variation between these two halves.

If k=10 (ten folds), each training set has 90% of the data. These training sets will be much more similar to each other, and thus the variation across them will be smaller.

In the extreme case of k=n (LOOCV), each training set has n-1 data points, differing by only one data point. The variation across these training sets is minimal.

Therefore, as k increases, the training sets in each fold become more alike, leading to a decrease in the variation across them.

If you have 10,000 rows of data and you're using K-Fold cross-validation with k=5, here's how the data will be split into training and test sets for each of the 5 folds:

Total number of rows: 10,000

Number of folds (k): 5

Size of each fold:

Ideally, each fold will have approximately the same number of rows. You can calculate this by dividing the total number of rows by the number of folds:

10,000 rows / 5 folds = 2,000 rows per fold

Data split in each fold:

In each of the 5 iterations (folds) of the K-Fold validation:

Test set: One fold will be used as the test set. Therefore, the test set will contain approximately 2,000 rows.

Train set: The remaining k-1 folds will be used as the training set. Therefore, the training set will contain approximately:

(k - 1) * (size of each fold) = (5 - 1) * 2,000 rows = 4 * 2,000 rows = 8,000 rows.

Summary for each of the 5 folds:

Fold 1:

Test set: ~2,000 rows

Train set: ~8,000 rows

Fold 2:

Test set: ~2,000 rows (a different set of 2,000 rows from Fold 1)

Train set: ~8,000 rows (the remaining 8,000 rows not in Fold 2's test set)

Fold 3:

Test set: ~2,000 rows (a different set)

Train set: ~8,000 rows (the remaining 8,000 rows)

Fold 4:

Test set: ~2,000 rows (a different set)

Train set: ~8,000 rows (the remaining 8,000 rows)

Fold 5:

Test set: ~2,000 rows (the final set of 2,000 rows)

Train set: ~8,000 rows (the remaining 8,000 rows)

Important Note on "Approximately":

If the total number of rows is not perfectly divisible by k, the folds might have a slightly uneven number of rows. For example, if you had 10,001 rows and k=5, some folds would have 2000 rows, and one fold would have 2001 rows. Most K-Fold implementations handle this gracefully. However, for simplicity in this explanation, we assume an even split.

In essence, with k=5 and 10,000 rows, in each of the five validation rounds, you will train your model on 8,000 rows and evaluate it on a distinct set of 2,000 rows. This process is repeated five times, ensuring that every data point is used for testing exactly once.

f 90,85,78,88,85 are the cross-validated scores then what would be the average cross-validation score?

The average score would be (90 + 85 + 78 + 88 + 85)/5 = 82.5