Parameters usually means number of features or independent variables. Hyperparameter values are the algorithm constants.
How you eliminate depends on your data and your algorithm choice.
You know when you are overfitting when you are doing really well on training data but poorly on validation data. However, you shouldn't overfit if you choose L1/L2 regularization hyperparameter based on best results in 5-fold nested x-validadation.
You generally still do worse on training set than on testing set with nested x-validation, but it doesn't matter because at the end once you choose the hyperparameters you train on the whole training set.
Sorry my background is in engineering not stats so I probably confused my terms a little. I think I mixed up 'variable' with 'parameter'.
As an example of what I meant, at the plant where I work we are often concerned about yield (the ratio of input mass to output mass). We measure a number of different "terms" during our process which we have control of to varying degrees (As an engineer I call these terms parameters I guess a statistician calls them 'independent variables').
So the term I'm trying to model is yield and my input variables are the various 'terms'.
I will read up on cross validation now (hadn't heard of it before). My work uses SAS - a quick google search suggest SAS supports this test.
How you eliminate depends on your data and your algorithm choice.
You know when you are overfitting when you are doing really well on training data but poorly on validation data. However, you shouldn't overfit if you choose L1/L2 regularization hyperparameter based on best results in 5-fold nested x-validadation.
You generally still do worse on training set than on testing set with nested x-validation, but it doesn't matter because at the end once you choose the hyperparameters you train on the whole training set.