Fix a typo in Adam and AdamW optimizers. Updated examples. Updated comparisons.

Added examples comparing the performance off different architectures. Probes the final network by turning of skips and feedbacks, etc.

• examples/compare/*

Corresponding plotting functionality.

• documentation/comparison.py

• Fix bug related to skip connections.
• Fix bug related to validation forward pass.
• Expanded examples.
• Improve feedback block.

Add dilation to the convolution layer.

Initial implementation of the deconvolution layer. Created with the good help of the GitHub Copilot. Validated against corresponding PyTorch implementation;

• documentation/validation/deconvolution.py

Minor bug-fixes and example expansion.

Thorough expansion of the feedback module. Feedback blocks automatically handle weight coupling and skip connections.

When defining a feedback block in the network's layers, the following syntax is used:

network.feedback(
    vec![feedback::Layer::Convolution(
        1,
        activation::Activation::ReLU,
        (3, 3),
        (1, 1),
        (1, 1),
        None,
    )],
    2,
    true,
    feedback::Accumulation::Mean,
);

Add possibility of skip connections.

Limitations:

• Only works between equal shapes.
• Backward pass assumes an identity mapping (gradients are simply added).

Add possibility of selecting the scaling function.

• tensor::Scale
• feedback::Accumulation See implementations of the above for more information.

Update maxpool logic to ensure consistency wrt. other layers. Maxpool layers now return a tensor::Tensor (of shape tensor::Shape::Quintuple), instead of nested Vecs. This will lead to consistency when implementing maxpool for feedback blocks.

Minor bug fixes to feedback connections. Rename simple feedback connections to loopback connections for consistency.

Add skeleton for feedback block structure. Missing correct handling of backward pass.

How should the optimizer be handled (wrt. buffer, etc.)?

Before:

network.set_optimizer(
  optimizer::Optimizer::AdamW(
      optimizer::AdamW {
          learning_rate: 0.001,
          beta1: 0.9,
          beta2: 0.999,
          epsilon: 1e-8,
          decay: 0.01,

          // To be filled by the network:
          momentum: vec![],
          velocity: vec![],
      }
  )
);

Now:

network.set_optimizer(optimizer::RMSprop::create(
  0.001,                     // Learning rate
  0.0,                       // Alpha
  1e-8,                      // Epsilon
  Some(0.01),                // Decay
  Some(0.01),                // Momentum
  true,                      // Centered
));

Layers now automatically reshape input tensors to the correct shape. I.e., your network could be conv->dense->conv etc. Earlier versions only allowed conv/maxpool->dense connections.

Note: While this is now possible, some testing proved this to be suboptimal in terms of performance.

Combines operations to single-loop instead of repeadedly iterating over the tensor::Tensor's.

Benchmarking benches/benchmark.rs (mnist version):

v2.0.1: 16.504570304s (1.05x speedup) v2.0.0: 17.268632412s

Weight updates are now batched correctly. See network::Network::learn for details.

Benchmarking examples/example_benchmark.rs (mnist version):

batched (128): 17.268632412s (4.82x speedup) unbatched (1): 83.347593292s

Optimizer step more intuitive and easy to read. Using tensor::Tensor instead of manually handing vectors.

Network of convolutional and dense layers works.

Batched training (network::Network::learn). Parallelization of batches (rayon).

Benchmarking examples/example_benchmark.rs (iris version):

v0.3.0: 0.318811179s (6.95x speedup) v0.2.2: 2.218362758s

Convolutional layer. Improved documentation.

Initial feedback connection implementation.

Improved documentation.

Custom tensor struct. Unit tests.

Dense feedforward network. Activation functions. Objective functions. Optimization techniques.

• Dense
• Convolution
  • Forward pass
    • Padding
    • Stride
    • Dilation
  • Backward pass
    • Padding
    • Stride
    • Dilation
• Deconvolution (#22)
  • Forward pass
    • Padding
    • Stride
    • Dilation
  • Backward pass
    • Padding
    • Stride
    • Dilation
• Max pooling
• Feedback

• Linear
• Sigmoid
• Tanh
• ReLU
• LeakyReLU
• Softmax

• AE
• MAE
• MSE
• RMSE
• CrossEntropy
• BinaryCrossEntropy
• KLDivergence

• SGD
• SGDM
• Adam
• AdamW
• RMSprop
• Minibatch

• Feedforward (dubbed Network)
• Feedback loops
• Skip connections
• Feedback blocks
• Recurrent

• Feedback connection
• Selectable gradient scaling
• Selectable gradient accumulation
• Feedback block
  • Selectable weight coupling

• Dropout
• Early stopping
• Batch normalization

• Parallelization of batches
• Other parallelization?
  • NOTE: Slowdown when parallelizing everything (commit: 1f94cea56630a46d40755af5da20714bc0357146).

• Unit tests
  • Thorough testing of activation functions
  • Thorough testing of objective functions
  • Thorough testing of optimization techniques
  • Thorough testing of feedback blocks
• Integration tests
  • Network forward pass
  • Network backward pass
  • Network training (i.e., weight updates)

• XOR
• Iris
• FTIR
  • MLP
    • Plain
    • Skip
    • Looping
    • Feedback
  • CNN
    • Plain
    • Skip
    • Looping
    • Feedback
• MNIST
  • CNN
  • CNN + Skip
  • CNN + Looping
  • CNN + Feedback
• Fashion-MNIST
  • CNN
  • CNN + Skip
  • CNN + Looping
  • CNN + Feedback
• CIFAR-10
  • CNN
  • CNN + Skip
  • CNN + Looping
  • CNN + Feedback

• Documentation
• Custom random weight initialization
• Custom tensor type
• Plotting
• Data from file
  • General data loading functionality
• Custom icon/image for documentation
• Custom stylesheet for documentation
• Add number of parameters when displaying Network
• Network type specification (e.g. f32, f64)
• Serialisation (saving and loading)
  • Single layer weights
  • Entire network weights
  • Custom (binary) file format, with header explaining contents
• Logging