Overview

This framework provides tools for instructors to write student-run grading applications, called "graders". Graders are written by instructors and distributed to students. When run by the students, the grader will assess their work and send a report to the dropbox, which is maintained by the instructor.

There are 3 significant parts to a grading application

1. A rubric - a list of specific criteria that each student must fulfil
2. Submissions - An evaluation of a specific student's work and progress through a rubric
3. A dropbox - a collection box for submissions

Rubrics

Rubrics are exactly what you would expect; a list of criteria that describe an assignment or test. Each criteria has a point value, and the students grade is determined by the criteria they fulfil.

Submissions

A submission is a bundle of data that tracks how well the student performed, according to the rubric. Rubrics and submissions are useless without each other. Submissions are meant to be submitted (who knew?) to the instructor.

The Dropbox

The dropbox is simply a web server maintained by the instructor that recieves submissions and records them. The submissions are recorded as CSV, which is supported by Microsoft Excel. Sorting and manipulation can be performed in Excel after all the submissions are in.

Rust Setup

You'll need the Rust language installed, and it's build tool "cargo". You can read more about that here. It's pretty easy to install on any platform.

You'll also want to switch over to the nightly channel. This crate uses experimental features only available in night. Do that with

$ rustup default nightly

Grader Setup

Once you have Rust installed, you'll want to make a new project to serve as your grading application.

You can create a new Rust project with

cargo new --bin my_grader

This is a Rust crate and can be installed like any other Rust crate. In the Cargo.toml file that cargo creates for you, add this crate as a dependency.

# ...
[dependencies]
rubric = "0.14" # or latest version

Note: Also in Cargo.toml is a version number. If you distribute your grader and it has bugs, you can fix them, update the version, and redistribute. It's a good idea to update it from time to time.

Now is a good time to create a rubrics/ directory alongside your src/ directory. You'll use that later.

In your main.rs, delcare the rubric crate and import the items you need.

#[macro_use] extern crate rubric;

// import what you need
use rubric::{Rubric, Submission, dropbox};

fn main() {
    // ...
}

It's also recommended that you create a tests.rs beside main.rs to hold your criteria tests.

You may need to access certain items or functions in the rubric crate. You can reference the docs on docs.rs for specific information.

Rubrics

Rubrics are the main component of this framework. A rubric is simply a list of criteria. They are the first thing you should write down when writing a grader.

Within this framework, rubrics are represented with .yml files in the rubrics/ directory. "YAML" is a markup language that you can read about here. It's pretty common within sysadmin tools and chances are you're already familiar with it. We can use YAML to write out the details of our criteria.

A Practical Example

Rubrics are very easy to understand if you see one.

In the git_lab example, we wrote a hypothetical assignment for our hypothetical students. This lab is meant to teach our students about git and how to use it. We came up with 4 criteria that will make up our assignment. These criteria are

1. The student must install git and have it available at the command line
2. The student must initialize git in a directory
3. The student must make at least 3 commits in that repository
4. The student must push the repository to Github and have it publicly accessible

This is a great start. All we need to do is formalize this into a rubric.

# Our assignments name
name: Git Lab
# An optional description
desc: A lab to teach git

# Set a deadline
deadline: 2023-06-07 23:59:59
# Don't allow late submissions
allow_late: false


# A list of our criteria
criteria:
  "Git installed":
    func: git_installed
    desc: Git should be installed and accessible
    worth: 25
    messages: ["installed", "not installed"]

  "Git initialized in repo":
    func: git_init
    desc: Current directory should have git initialized
    worth: 25
    messages: ["initialized", "not initialized"]

  "Commits present":
    func: commits_present
    worth: 25
    desc: Current git repository should have more than 2 commits

  "Repo pushed":
    func: repo_pushed
    worth: 25
    desc: Current git repository should be pushed to github

You can see that each criteria has a name, a point value, and some other configuration options. The only required fields are the name and worth, everything else has defaults.

Lets dissect one of the criteria

# ...
criteria:
  # The criteria name
  "Git installed":
    # This is the name of the corresponding function.
    # if you omit this, the function name will be the criterion's name,
    # lower-cased and whitespace replaced with dashes
    func: git_installed
    # Just a description. Students will see this when grading
    desc: Git should be installed and accessible
    # The point value, can be any number, even negative.
    worth: 25
    # Success/failure messages. These default to "passed" and "failed".
    # The're just some extra information for the student
    messages: ["installed", "not installed"]

There are a few more options that you can provide, which you can read about on the Rubric Specification Page.

Loading a Rubric

After writing out a rubric in a .yml file, we can load it into our grader and use it.

The first step to loading a rubric is reading the file. This is done with the yaml! macro.

let rubric_yaml = yaml!("../rubrics/main.yml").expect("Couldn't load file!");

yaml! loads a file relative to the current file. It returns a Result, so we can deal with the error if we want. It's usually a good idea to call expect() so the program crashes if the file couldn't be read.

yaml! is special because it embeds the .yml in the compiled executable. This means you don't have to distribute the rubric file with the executable. Your rubric can also be kept private if you want it to be.

Next, we just have to pass the YAML data to the Rubric struct like this

let rubric = Rubric::from_yaml(&rubric_yaml).expect("Bad YAML!");

Note: We're using expect() again here. We want the program to crash at compile-time when we're working on the grader, not at run-time when the students are using it. Better for us to deal with the error than them.

Writing The Tests

We have a rubric loaded, but it has no way to actually verify that the criteria have been fulfilled. We're going to write one function for each of the criteria. The function (called a "criteria test" or just "test") has the responsibility of ensuring the criteria was actually fulfilled by the student.

Assignments obviously vary greatly in scope and material, so writing these tests is the bulk of the work to be done when writing a grader. It all depends on what the student should be doing.

Things like installing Git or running a web server are easy to verify, but other tasks might not be. Because every criteria has a function, the full force of Rust is on your side. You may have to get creative in writing your tests. You can always look at the examples on Github for inspiration.

See the Criteria Tests page for more information on this topic.

Rubric Specification

If you're looking for the syntax of the YAML language itself, then look here. This page is for the allowed items in a rubric's .yml file.

Minimal Rubric

Here is a rubric with as few items as possible. Everything here is required.

name: My rubric

criteria:
  "first and only criterion":
    worth: 10

All Items

Here is a full rubric with everything specified, with comments for more information about each key.

# -- Rubric Details --
# Required name
name: My rubric
# Optional description. Gets shown to the student when grading
desc: Description of my rubric
# Sanity check. If the sum of all criteria doesn't add to this number,
# an error message will be displayed. Just ensures that you give the correct
# worth to all criteria
total: 100





# -- Deadline verification --
# All of these are optional

# The optional submission deadline. Must be in this format
#   YYYY-MM-DD HH:MM:SS
# It will use the current local timezone.
# If a submission is created after this time, the late flag will be true.
# If you enter a deadline from the past, you will be warned during compilation, but
# compilation will continue.
deadline: 2020-05-21 23:59:59
# If this is set to false, the submission will always have a 
# grade of 0 unless manually changed. Defaults to true (allowing late submission).
# A submission *can* still be submitted if this is false, but its grade will be 0
# and the late flag will be true.
# If this is true, `deadline` functions exactly like `final_deadline`
allow_late: true
# This amount will be subtracted from the grade if the submission is late
late_penalty: 10
# This will subtract this many points per day after the deadline.
# One second after the deadline is the first late day, so -5 points.
# 24 hrs + 1 second after the deadline is the second late day, so -10.
# You probably shouldn't use both this and "late_penalty", as they will 
# both take effect if you provide both.
late_penalty_per_day: 5
# Same format as deadline.
# If this is provided, absolutely no submissions will be graded
# after this time. This is mostly used when you want to set a 
# soft deadline (with `deadline`), then late penalties per day, 
# then a hard deadline.
final_deadline: 2020-05-24 23:59:59




# -- Criteria --
criteria:
  # Each criteria has its name as the key.
  # While the quotes are strictly necessary,
  # they are good practice.
  # I usually indent YAML with 2 spaces.

  "First criterion":
    # The name of the corresponding function. This should be unique.
    # If not specified, the function name will be the criterion's name,
    # lowercased and whitespace replaced with dashes. But it's best to be
    # explicit about this.
    # Should match the function name exactly.
    # Because this is unique, it is used to find criteria within a rubric.
    func: whatever_func
    # Any number (even negative). Lowest number is run first.
    # Criteria without indices do not have consistent order,
    # and may be tested concurrently.
    index: 1
    # A description
    desc: You should do this to fulfil this criterion
    # required point value
    # can be negative
    worth: 50
    # success and failure messages
    # default to "passed" and "failed"
    messages: ["Passed!", "not passed"]
    # This will prevent the criterion from being displayed
    # to the student. Useful if you want hidden requirements 
    # or are grading a test
    hide: false

  # This criterion has all default values
  "Second criterion":
    func: second_criterion
    index: 100
    desc: ""
    worth: 0
    messages: ["passed", "failed"]
    hide: false

Criteria Tests

As stated on the Rubrics page, a criteria tests is a Rust function that verifies if a criteria was actually fulfilled. They will vary widely. This page will show how to write a test and a few examples to get you started.

A Single Test

Tests are just functions, but they must have a specific signature. They must always accept the same parameters and return the same type of value. Here's the signature

// Be sure TestData is imported
use rubric::TestData;

fn my_test(data: &TestData) -> bool {
    // Test code goes here...
}

Every test must accept a reference to a TestData struct. This TestData is stored on a Submission, which I'll cover in a different section. What you should know now is that it's an alias to HashMap<String, String>.

Sometimes you won't need TestData in a test, in which case you can just name the parameter _ and Rust won't complain.

Tests must also return a boolean. true if it passes, false otherwise. If a test returns true, then the associated criteria's worth will be added to the point total. If all the criteria tests return true, the maximum score is achieved.

Using TestData

Remember that a TestData struct is really just a HashMap. It will contains keys and values that you specify when setting up a Submission. You can use any of the methods that HashMap's have. 90% of the time, you'll just want to read a value from the TestData. There's 2 ways to do that.

fn some_test(data: &TestData) -> bool {
    // The easy but dangerous way to get a value
    // this will *crash* if the key doesn't exist
    let my_value = data["my_key"];

    // The safe way to get a value
    if let Some(value) = data.get("my_key") {
        // Key exists, now we have the value
        println!("Value is {}", value);
    } else {
        // Key doesn't exist, something went wrong, handle error
        println!("Value doesn't exist!");
    }

    // ...
}

It's important that you take precautions when writing a grader. You really don't want it to crash while your students are running it. The two examples above to the same thing, but the second method won't crash if the key doesn't exist.

Organization

I strongly recommend making a test.rs file alongside main.rs to keep your tests in. Of course, you don't have to. You could keep your tests as loose functions in main.rs, or maybe have a submodule in main.rs.

Again, I recommend making a tests.rs file and keeping them in there. Here's how I set things up.

// tests.rs
use rubric::TestData;

fn test_from_tests_rs(_: &TestData) -> bool {
    // test code goes here
    return true;
}

// more tests here...

// main.rs
extern crate rubric;

// declare tests.rs as a module
mod tests;
// bring all test functions into scope
use tests::*;
// ...

Attaching Tests

Each criteria has an associated test, but we need to tell our program which test goes with which criteria. In our rubric, each criteria should have a func key. This should exactly match the name of the test. Rust uses snake_case for function names. After we've written our tests and loaded our rubric, we can use the attach! macro to assign them.

name: Basic rubric

criteria:
  "Only criteria":
    worth: 50
    # This is the important bit
    # it allows attach!() to find the right function
    func: my_criteria_test
    # ...

fn my_criteria_test(_: &TestData) -> bool {
    // test code goes here...
    return true;
}


fn main() {
    // load rubric
    // code omitted
    // be sure it's mutable
    let mut rubric = //...

    attach!(rubric, my_criteria_test);
}

Helpers

There are a few helper modules and functions that perform some common tasks. Sometimes your tests will be one-liners from the helper modules. See the helpers module documentation on docs.rs for more info.

Examples

Some basic examples can be found in the examples directory on Github, specifically in this file in the git_lab example.

Submission

A submission represents one students work on an assignment. It contains any data you may want about the student, like name and ID, their grade, which criteria they passed and failed, information about their system, information they provide, etc.

A submission is the other half to a rubric. While rubrics are identical among all students, every student has their own unique submission. The stucture of a submission is all the same, but the values contained in it are unique to each student.

A submission is designed to be constructed during the grading process and sent back to you, the instructor. You'll end up with a list of submissions from every student containing their grade and all the data you specified.

Let's take a gander at a blank Submission struct to see what's in it:

Submission {
    time: 2020-08-04 Tue 21:13:34 -05:00,
    grade: 0,
    data: {},
    passed: [],
    failed: [],
}

There's a few default values like a timestamp, grade, and which criteria the submission passed and failed, but it's pretty empty. The important bit is the data HashMap. This is where all of the data you specify will live. All of this data will be collected when the student runs the grader, and will be sent back as part of the grade report.

The data! macro

I mentioned that the data field on a submission is a HashMap. This is sort of true. It's technically a TestData, which is an alias to HashMap<String, String>.

If you read the Criteria Tests section, you might remember TestData as the value that criteria tests must accept as a parameter. The TestData on a submission is what will be passed into these tests.

Rust does not have object literals the same way a language like Python does. Instead, this crate provides a data! macro that will create a TestData for you. Here's how to use it.

let data = data! {
    "some_key" => "some value",
    "other_key" => "other value"
};

Since TestData is an alias to HashMap<String, String>, the keys and values must be strings.

Creating a Submission

Creating a submission is easy

// creates a blank submission like the one above
let sub = Submission::new();

You will most likely want to create a submission with some data. You can do that like this.

let data = data! {
    "key" => "value"
};

let sub = Submission::from_data(data);

// This will create the following
Submission {
    time: 2020-08-04 Tue 21:13:34 -05:00,
    grade: 0,
    data: {
        "key": "value",
    },
    passed: [],
    failed: [],
}

Prompting for Data

When creating a submission, you'll almost always want to ask the student for some information. Usually their name and ID, but maybe some other information as well.

The prompt! macro does this easily.

fn main() {
    // Asks for something, enforces the correct type
    let name = prompt!("Name: ", String);
    // Will loop until they enter a number
    let age = prompt!("Age: ", usize);
    // Will loop until they enter a valid IPv4 address
    let ip = prompt!("IP Addr: ", std::net::Ipv4Addr);
}

You can combine this with the data! macro to easily collect information from the user and encapsulate it in a submission.

fn main() {
    let data = data! {
        "name" => prompt!("Name: ", String),
        "id"   => prompt!("ID: ", String)
    }
}

Note: because TestData must contain string values, you lose out on the type enforcement that prompt! provides. This is an unfortunate side effect of the TestData type; all values must be strings.

Fingerprinting and Security

Security is important for graders. We don't want students grades to be visible, and we don't want students to be able to fake submissions.

Closed Source

You should consider keeping your graders closed source. While having the source available isn't explicitly dangerous, it would probably be best to keep your grader from your students. Instead, just compile and distribute the executable.

When I write graders, I set them to open a dropbox when I run the grader with a certain command line argument. While technically your students could open the dropbox, this would only start an empty dropbox on their machine. It would be able to accept submission on the off chance someone submitted to it. I would recommend setting the argument to start the dropbox to something not easily guessable. I use open_sesame to start the dropbox.

If you want to be absolutely sure that a student doesn't open the dropbox, you can compile a separate grader that only opens the dropbox, separate from the grading bit.

Fingerprints

When creating a new submission, you can enable a Fingerprint for extra verification. You provide a "secret key" (just a random string that you keep secret) that will be attached to the fingerprint. The fingerprint will also collect some basic system information (like platform). When the submission is sent to the dropbox, the fingerprint data will also be sent.

let mut submission = Submission::new();
submission.set_fingerprint("secret key. Keep this quiet!");

If fairly straightforward to POST a web request with a JSON body. Providing a secret key in the fingerprint helps protect the dropbox from fake submissions. If you recieve a submission that does not contain the secret key, it's probably fake. If you keep the secret key in the source code, be sure the source code is private. The student will not be notified about the fingerprint, so they won't know about the secret key.

The fingerprint will also collect some system information, which is more passive protection. If half of the students submission are from a Linux machine, then it switches to Windows, that may be suspicious. If the assignment is meant to be performed on one system, this might be a red flag. It's ultimately up to you as to what you do with the data.

Dropbox

A dropbox is a place to send submissions after they're done being graded. It recieves submissions in JSON format and writes them to a CSV file for review by the instructor.

The dropbox is a web server run by you, the instructor. It should be run on a publicly available server with a static IP or DNS name. The web server comes preconfigured, all you need to do is give it an environment to run it.

You can open the dropbox by calling the dropbox::open(PORT) method, where PORT is a valid port number.

extern crate rubric;
use rubric::dropbox;

fn main() {
    // Runs the dropbox on port 8080
    dropbox::open(8080);

    // or...
    // Only opens when you run with the "open_sesame" argument,
    // otherwise does nothing
    dropbox::open_with_arg("open_sesame", 8080);
}

You'll see the following output

Dropbox is open! accepting POST requests to /submit
�🔧 Configured for development.
    => address: 0.0.0.0
    => port: 8080
    => log: normal
    => workers: 24
    => secret key: generated
    => limits: forms = 32KiB
    => keep-alive: 5s
    => tls: disabled
🛰  Mounting /:
    => GET / (return_ok)
    => POST /submit application/json (accept_submission)
��� Rocket has launched from http://0.0.0.0:8080       

The home route (/) should return an OK status, but no content. If you visit the url of your webserver, you should get a blank web page. This is good, it means everything is working properly.

Submitting to the dropbox

Submissions come with a submit() method meant to work with the dropbox.

extern crate rubric;
use rubric::Submission;

fn main() {
    let submission = Submission::new();
    
    // grade...

    // assuming your dropbox is running at this url
    let url = "http://my.dns.name.or.ip.com:8080/submit";

    // Submit and give some feedback
    match submission.submit(&url) {
        Ok(_)  => println!("Submission recorded!"),
        Err(e) => println!("Something went wrong! {}", e),
    };
}

The post_json() method in the helpers::web module is made with the dropbox in mind. After creating and grading a Submission, just pass it and the url of your dropbox to send the submission.

extern crate rubric;
use rubric::{Submission, dropbox, helpers::web};

fn main() {
    let submission = Submission::new();
    
    // grade...

    // assuming your dropbox is running at this url
    let url = "http://my.dns.name.or.ip.com:8080/submit";

    // Submit and give some feedback
    match web::post_json(&url, &sub) {
        Ok(_)  => println!("Submission recorded!"),
        Err(e) => println!("Something went wrong! {}", e),
    };
}

Error Handling

When sending a submission to the dropbox, it's very important to provide feedback to the student. They should know if the submission went through successfully.

Submitting with the Submission::submit() function returns a Result that you can use to handle any possible errors. The Err variant of the Result is a reqwest::Error.

// some code omitted
fn main() {
    // Open the dropbox somewhere else
    dropbox::open_with_arg("open_sesame", 8080);

    let mut sub = Submission::new();
    
    // Grading, etc goes here...

    // using `match` gives us basic error handling.
    // here, we're just printing the error. This can give more insight
    // as to what went wrong.
    match sub.submit("http://localhost:8080/submit") {
        Ok(_) => println!("Submission recorded!"),
        Err(e) => println!("Error! Couldn't record submission.\n{}", e);
    }
}