Random Forest Tutorial

This is slide deck from a lecture.  It is a good introduction to RDF's with advantages and disadvantages compared with other methods.

www.dabi.temple.edu/~hbling/8590.002/Montillo_RandomForests_4-2-2009.pdf

Decision Forests for Computer Vision and Medical Image Analysis book from Microsoft Research

This is the best resource I have found for understanding and using decision tree based machine learning algorithms.  It is very thorough on both theory and practical use with comparisons with other algorithms such as SVM's and AdaBoost.

There are also a bunch of supplemental materials available for free including a very nice PowerPoint with great explanations.  The supplemental materials include C++ and C# code.

Decision Forests - Microsoft Research

Decision Forests for Classification, Regression, Density Estimation, Manifold Learning and Semi-Supervised Learning

This technical report from Microsoft Research is an A-Z tutorial on how decision tree machine learning algorithms work.  It includes in depth explanations of random forests, extra tree classifiers, random ferns and other variations for both classification and regression.

It is in report format and compares decision forests to other types of machine learning algorithms such as SVM.  Some simple toy problems give the basics and some real life applications such as body position recognition and medical image are included.

There is also an accompanying PowerPoint with some nice animations.

http://research.microsoft.com/pubs/155552/decisionForests_MSR_TR_2011_114.pdf is not available

Nonlinear regression in environmental sciences by support vector machines combined with evolutionary strategy

A hybrid algorithm combining support vector regression with evolutionary strategy (SVR-ES) is proposed for predictive models in the environmental sciences. SVR-ES uses uncorrelated mutation with p step sizes to find the optimal SVR hyper-parameters. Three environmental forecast datasets used in the WCCI-2006 contest – surface air temperature, precipitation and sulphur dioxide concentration – were tested. We used multiple linear regression (MLR) as benchmark and a variety of machine learning techniques including bootstrap-aggregated ensemble artificial neural network (ANN), SVR-ES, SVR with hyper-parameters given by the Cherkassky–Ma estimate, the M5 regression tree, and random forest (RF). We also tested all techniques using stepwise linear regression (SLR) first to screen out irrelevant predictors. We concluded that SVR-ES is an attractive approach because it tends to outperform the other techniques and can also be implemented in an almost automatic way. The Cherkassky–Ma estimate is a useful approach for minimizing the mean absolute error and saving computational time related to the hyper-parameter search. The ANN and RF are also good options to outperform multiple linear regression (MLR). Finally, the use of SLR for predictor selection can dramatically reduce computational time and often help to enhance accuracy.

Nonlinear regression in environmental sciences by support vector machines combined with evolutionary strategy

Support Vector Machines (SVMs) organization

This site is a good compilation of everything related to SVM's.  There are links to many academic papers, tutorials, applications and much more.  There are also links to learn about all the mathematics necessary to really understand how SVM's work.  I am impressed by the fact that the site creators are not just cheerleaders for SVM's, they do a good job of stating the advantages and disadvantages of SVM's as well as comparing them to competing machine learning methods.

Support Vector Machines

LIBSVM -- A Library for Support Vector Machines

This is the library for support vector machines (SVM's).  They have a version for many different programming languages including C++, Python, R, MATLAB, Perl, Ruby, Weka, Common LISP, CLISP, Haskell, OCaml, LabVIEW, and PHP interfaces. C# .NET code and CUDA extension is available.

If you are new to SVM's there is a cool java applet and a javascript toy that will show you how they work.



LIBSVM -- A Library for Support Vector Machines

DTREG SVM - Support Vector Machines

This is a commercial machine learning package that I have not used.  The page in the link contains a very good explanation of how support vector machines (SVM) work.

SVM - Support Vector Machines

Extremely randomized trees by Pierre Geurts Damien Ernst Louis Wehenkel

This paper proposes a new tree-based ensemble method for supervised classification and regression problems. It essentially consists of randomizing strongly both attribute and cut-point choice while splitting a tree node. In the extreme case, it builds totally randomized trees whose structures are independent of the output values of the learning sample. The strength of the randomization can be tuned to problem specifics by the appropriate choice of a parameter. We evaluate the robustness of the default choice of this parameter, and we also provide insight on how to adjust it in particular situations. Besides accuracy, the main strength of the resulting algorithm is computational efficiency. A bias/variance analysis of the Extra-Trees algorithm is also provided as well as a geometrical and a kernel characterization of the models induced.

orbi.ulg.ac.be/bitstream/2268/9357/1/geurts-mlj-advance.pdf