# arcagi
A Rust attempt at the ARC-AGI prize 2024

WARNING:
-----
This code is messy and imcomplete and is likely to change rapidly. At the
moment it is a set of experiments to guage what works well and what does not.
It is perhaps 20% complete and will answer just over 10% of the examples given.

Background
----------

The Kaggle ARC-AGI challenge 2024 is an attempt to get computers to solve
visual 2D puzzles, similar to those in many IQ tests. To solve this challenge
in a principled way requires perception of stucture and reasoning guided by 
constraints. It currently cannot be solved by LLM/GenAI based methods though
there may be a small place for them (or possibly just simple Neural Networks
for specific recognition tasks). 

This code is a first submission for the prize, it could conceivably solve all
the puzzles in the challenge, but at the moment solves just over 10% in the
public evaluation set. Much more work is required to reach the 85% or better
level. 

This prize in important because it will demonstrate reasoning abilities and
not just the memoristion with some 'light' generalisation that is currently
demonstrated by all Gen-AI models (in fact all Machine Learning models). 

Abstraction, Generalisation and Functional composition are key concepts.
These work in tandem with constraint recognition and discovery to tame the
size of the search space.

This is a pure Symbolic AI approach and hence is deterministic. There may be a
place for an LLM/Gen-AI in object recognition and/or generating candidate
solutions.
It was hoped that the author's previous experience of LLMs (see LLMClient),
would help (for code generation). It might eventually, for now there is
plenty to explore with the current Symbolic approach.
Code generation was experimented with, given some data
structures and a textual description of the problem, but the code
generated was poor quality and time consuming to debug (Rust is probably not
the best target for code generation given it's memory management as simple
next token prediction does not understand that).

See: https://www.kaggle.com/competitions/arc-prize-2024

See inet submission.

To run this the first parameter is one of 'training', 'evaluation' or 'test'. There is no solutions file for the test run, this is expected so ignore the error message. A second parameter can be supplied which is the test to run, useful
for testing individual cases. A typical call to run the training set would be
'cargo run --release training'. 

Kaggle notebook
---------------

```
from cffi import FFI
ffi = FFI()
ffi.cdef("int test();")
rust = ffi.dlopen("/kaggle/input/release/libarc_agi.so")
a = rust.test();
print(a)
```

NOTE
----

To compile the above for kaggle a docker image needs to be created and the .so
file uploaded and referenced, somewhat tedious. It will run locally for public
data test with cargo as normal. A release version should run in less that a
second on most machines, this will change as more cases are completed.

Currently there is a problem with the above, with kaggle scoring on the
private dataset. Only run locally on public data for now or join the inet team.

More investigation of the scoring on the secret test set was done, in addition
to more training and evaluation examples succeeding. There is still a problem
in that the score is always 1.00 regardless of what is done. Currently 47 
evaluation tasks are correctly answered. The code being deterministic a binary
chop over the top level rules can be done. If this is done over to distinct 
halves then on both occasions 1.0 is scored, not 0.0 and 1.0 or vice versa
as would be expected. Any halves! The submission file is correctly formatted (though lacks optional white space). A null run produces a score of 0.0 as would be expected. 
So, very confused.