% A LaTeX file that describes how to install, run and modify the sample java
%  agent for ALE 0.5.
%
% Created by Marc G. Bellemare

\documentclass[12pt]{article}

\usepackage{fullpage}

\title{ALE Java Agent Tutorial}
\author{Marc G. Bellemare\\ mgbellemare@ualberta.ca}

\begin{document}

\maketitle

\section{Requirements}

To run this agent, you will need:

\begin{itemize}
  \item{Java 1.6}
  \item{ALE}
  \item{At least one ROM file}
  \item{Apache Ant}
  \item{Perl (optional)}
\end{itemize}

Although not strictly necessary, we recommend the use of Apache Ant to build
the Java agent.

This tutorial assumes that you have installed ALE 0.5 in a directory named \verb+ale_0_5+. See
the main manual (\verb+/path/to/ale_0_5/doc/manual/manual.pdf+) for ALE installation details. 

\section{Installation/Compilation}

Installing the Java agent is simple. Assuming you have extracted the Java agent
package to the directory \verb+ale_java_agent+, you may simply use Ant to build the
jar file containing the agent code: 

\begin{verbatim}
> cd ale_java_agent
ale_java_agent> ant jar
\end{verbatim}

Voil\`a! Your Java agent is compiled and good to go.

\subsection{Perl script setup}\label{subsec:perl_script}

For your convenience, we offer a perl script that automates running ALE and
the Java agent. To use this perl script, you should copy or link the ALE 
executable, configuration file and roms directory accordingly. Assuming that
your roms are located at \verb+/path/to/atari_roms+ and that you have 
installed ALE at \verb+/path/to/ale_0_5+, you may then (on Mac OS X and *NIX system): 

\begin{verbatim}
> cd /path/to/ale_java_agent
/path/to/ale_java_agent> ln -s /path/to/ale_0_5/ale . 
/path/to/ale_java_agent> ln -s /path/to/ale_0_5/stellarc . 
/path/to/ale_0_5> ln -s /path/to/atari_roms roms
\end{verbatim}

In particular, if running the Java agent from its package path, i.e. 
\begin{center}\verb+/path/to/ale/doc/java-agent+\end{center} you may simply do
\begin{verbatim}
doc/java-agent/code> ln -s ../../../ale . 
doc/java-agent/code> ln -s ../../../stellarc .  
\end{verbatim}

Your \verb+ale_java_agent+ directory is now set up.

\section{Running}

If you did not perform the optional perl script setup (Section 
\ref{subsec:perl_script}), or do not have perl installed, skip to Section 
\ref{subsec:named_pipes} for information on running ALE via named pipes. The named pipes
interface is somewhat deprecated and we encourage you to use the perl script if possible. The
C++ fifo interface example
\begin{center}\verb+doc/examples/fifoInterfaceExample.cpp+\end{center}
can also easily be adapted to run your Java agent together with ALE.

Assuming that everything is installed correctly, you should now be able to
run the HumanAgent using the provided perl script:

\begin{verbatim}
ale_java_agent> perl run_agent.perl space_invaders 
\end{verbatim}

\subsection{Named Pipes}\label{subsec:named_pipes}

To communicate with ALE via named pipes, you need to start ALE and the Java
agent in separate consoles. First, we create two named pipes:

\begin{verbatim}
/path/to/ale_0_5> mkfifo ale_fifo_in
/path/to/ale_0_5> mkfifo ale_fifo_out
\end{verbatim}

Then we run ALE and the Java agent in separate processes. We specify the path
of both named pipes using the \verb+-named_pipes+ option:

\begin{verbatim}
Terminal 1
/path/to/ale_0_5> ./ale -game_controller fifo_named 
  /path/to/atari_roms/beam_rider.bin

Terminal 2
ale_java_agent> java -cp dist/ALEJavaAgent.jar ale.agents.HumanAgent 
  -named_pipes /path/to/ale_0_5/ale_fifo_
\end{verbatim}

\section{Recording screen data}

We provide facilities for recording received screen data to PNG files. We do
so by passing the \verb+-export_frames+ command-line argument to the Java
HumanAgent class. The perl script is already set up to pass all arguments 
beyond the first to the Java agent:

\begin{verbatim}
ale_java_agent> perl run_agent.perl beam_rider -export_frames
\end{verbatim}

Frames will be saved in the \verb+ale_java_agent/frames+ directory, starting
with \verb+frame_000000.png+ and subsequently incrementing the index. The
equivalent command with named pipes is:

\begin{verbatim}
ale_java_agent> java -cp dist/ALEJavaAgent.jar ale.agents.HumanAgent 
  -named_pipes /path/to/ale_0_5/ale_fifo_ -export_frames
\end{verbatim}

For further information, see the class \verb+ale.movie.MovieGenerator+. In
\verb+ale.agents.HumanAgent+, the variable \verb+exportFramesBasename+ defines
the pathname to which frames are saved.

\section{Code listing}

To complete this tutorial, we give a list of the classes provided in the
Java agent package, along with a short description.

\begin{enumerate}
  \item{\verb+ale.agents+}
    \begin{itemize}
      \item{\verb+AbstractAgent+ Abstract class; interfaces with ALE and the GUI.}
      \item{\verb+HumanAgent+ extends \verb+AbstractAgent+. Defines an agent controlled by the user via the keyboard.}
      \item{\verb+RLAgent+ extends \verb+AbstractAgent+. A simple learning agent that uses SARSA and $\epsilon$-greedy.}
    \end{itemize}
  \item{\verb+ale.gui+}
    \begin{itemize}
      \item{\verb+AbstractUI+ Abstract class; defines a user interface.}
      \item{\verb+AgentGUI+ extends \verb+AbstractUI+. Defines a graphical user interface.}
      \item{\verb+NullUI+ extends \verb+AbstractUI+. The \verb+/dev/null+ of uesr interfaces.}
      \item{\verb+KeyboardControl+. Receives keystrokes and converts them to ALE actions.} 
      \item{\verb+ScreenDisplay+. Responsible for displaying images on the screen.}
      \item{\verb+MessageHistory+. A helper class for display messages on-screen.}
    \end{itemize}
  \item{\verb+ale.io+}
    \begin{itemize}
      \item{\verb+ALEPipes+. Communicates with ALE via stdin/out or named pipes.}
      \item{\verb+Actions+. Helper class mapping action names to integers.}
      \item{\verb+ConsoleRAM+. Encapsulates RAM data.} 
      \item{\verb+RLData+. Encapsulates RL data.} 
    \end{itemize}
  \item{\verb+ale.movie+}
    \begin{itemize}
      \item{\verb+MovieGenerator+. Helper class to save screen data to PNG files.}
    \end{itemize}
  \item{\verb+ale.rl+}
    \begin{itemize}
      \item{\verb+FeatureMap+. Maps screen data to feature vectors.}
      \item{\verb+FrameHistory+. Stores a list of recent frames.}
      \item{\verb+LinearModel+. A linear regression predictor. Used for approximating value functions.}
      \item{\verb+SarsaLearner+. The core SARSA algorithm.}
    \end{itemize}
  \item{\verb+ale.screen+}
    \begin{itemize}
      \item{\verb+ColorPalette+. Abstract class; defines basic colour palette functionality.}
      \item{\verb+NTSCPalette+. Defines the NTSC colour palette (128 colors).}
      \item{\verb+SECAMPalette+. Defines the SECAM colour palette (8 colors).}
      \item{\verb+ScreenConverter+. Converts ScreenMatrix objects to Java image objects.}
      \item{\verb+ScreenMatrix+. Encapsulates screen data.}
    \end{itemize}
\end{enumerate}

\end{document}