Help file for using this testing framework ------------------------------------------ If you ever forget what you're currently reading, you can always see it again by typing make help To run all the tests until one fails, type make To run all the tests regardless of failures, type make -i Either way, you would be making the default target, named 'valid'. (Naming it 'test' would get confusing when you start referring to data train/test splits.) To clean up so that you can start over, type make clean To download and prepare all the relevant data at once, type make prepData To erase all the data, type make eraseData If you change some code so that the output of some tests changes, or you add some new tests, and you want to save the new output so that future outputs will be compared with it, type make expected You can work with individual tests by prepending their name to the make target, as in make 2.valid make t17.clean make myNewTest.expected Of course, you can make multiple targets together, as in make 2.clean 2.valid There are also pre-defined groups of tests that you can run, such as make regression_group If you want to create the correct outputs for a test without actually running it, type make $testName.pretend This can be useful if you want to run test X, which depends on the outputs of test Y, but you can't run test Y for some reason. The "valid" target for each test is actually a "run" followed by a "compare". Thus make 3.run 3.compare is equivalent to make 3.valid and it's possible to invoke either part without the other. If you want to use an executable other than ../vowpalwabbit/vw, you can supply it on the command line like this: make EXEC=myProgram or make ../vowpalwabbit/vw a sym-link to it. Other variables, such as ARCH, DIFF, TIME, WGET, and GREP, can also be set on the command line similarly. If you want to pass extra parameters to all tests, set the EP variable, like this: make EP="--random_seed 13" If you want to check speed in addition to accuracy, invoke with TIMING=y . Due to the many factors that can change execution speed, we do not expect speed results to be portable across machines. Therefore, "expected" speed files are not checked into the repository. If you'd like to do regression testing for speed, you should `make TIMING=y; make TIMING=y expected` on your particular machine before making any code changes. In fact, it's a good idea to then `make TIMING=y` a couple more times, and `make TIMING=y expected` again if the speed drops significantly. This can happen even on the same machine, if it's dividing attention between multiple processes. If you want to add or modify tests, read HOWTO.write_new_tests.txt . Working with data archives -------------------------- To save time and bandwidth, you can create and retrieve all the data used in all the tests at once. To create an archive of all the data, type make ARF= archive You can then stash the archive wherever you like, probably at a URL-accessible location in the cloud. Later, to retrieve the data again, you can type make URL= prepData This will cause the data to be downloaded and installed in prepared form, so that you don't need to run the preparation steps.