# API In general, Ocypod attempts to return meaningful HTTP status codes to indicate whether an operation was successful or not. As well as the status codes detailed per endpoint, Ocypod will also return the following _5xx_ codes: * 500 - unexpected internal error * 503 - Redis connection unavailable ## Queue endpoints Used for interacting with queues, i.e. creating new queues, updating queue settings, creating jobs, and fetching jobs. --- ### `GET /queue` Get all known queues as a JSON list. #### Returns * 200 - JSON list of strings #### Example $ curl localhost:8023/queue ["my-queue", "another_queue", "queue3"] --- ### `GET /queue/{queue_name}` Get a queue's settings. #### Returns * 200 - JSON object containing queue settings * 400 - invalid queue name passed as parameter * 404 - no queue with given name was found #### Example $ curl localhost:8023/queue/example {"timeout":"10m", "heartbeat_timeout":"5m", "expires_after":"5m", "retries":5, "retry_delays":["10s","30s","5m"]} --- ### `GET /queue/{queue_name}/size` Get number of jobs currently queued for a given queue name. #### Returns * 200 - JSON integer queue size * 400 - invalid queue name passed as parameter * 404 - no queue with given name was found #### Example $ curl localhost:8023/queue/example/size 3 --- ### `PUT /queue/{queue_name}` Create a new queue, or update an existing queue with given settings. #### Request The request body must contain JSON of the form: {"timeout": , "heartbeat_timeout": , "expires_after": , "retries": , "retry_delays": [[, ...]]} where `` is a human readable string of the form `"30s"`, `"5m"`, `"1w2d7h"`, etc. Any of the fields may be omitted (e.g. `{}` is the minimum accepted input), in which case server-wide default values will be used. Set `timeout`, `heartbeat_timeout`, or `expires_after` to "0s" to disable timeout/expiry. Set `retries` to `0` to disable retries. #### Returns * 201 - new queue created * 204 - existing queue updated * 400 - invalid queue name or queue settings given #### Example $ curl -i -H 'content-type: application/json' -XPUT -d '{"timeout": "10m"}' localhost:8023/queue/example HTTP/1.1 201 Created location: /queue/example content-length: 0 date: Fri, 16 Nov 2018 17:47:55 GMT --- ### `DELETE /queue/{queue_name}` Delete an existing queue, and any jobs still queued on it. Any running or ended jobs won't be affected. If any running jobs created on this queue have retries remaining, they'll instead remain in their failed/timed out state, and be eligible for expiry if they can't be requeued due to their original queue no longer existing. #### Returns * 204 - existing queue successfully deleted * 400 - invalid queue name given * 404 - queue with given name not found #### Example $ curl -i -XDELETE localhost:8023/queue/example HTTP/1.1 204 Queue deleted date: Tue, 20 Nov 2018 18:07:47 GMT --- ### `GET /queue/{queue_name}/job` Get the next job to work from the given queue, if any, as a JSON job payload. This JSON payload is of the form: {"id": , "input": } where `id` is a unique job ID that's autogenerated by Ocypod when a job is created, and `input` is any JSON provided by the client that created the job. When a client gets a job in this way, the job is marked as running, and is removed from the queue. #### Returns * 200 - JSON payload as described above * 204 - no jobs available in this queue * 400 - invalid queue name given * 404 - queue with given name not found #### Example $ curl -i localhost:8023/queue/example/job HTTP/1.1 200 OK content-length: 58 content-type: application/json date: Tue, 20 Nov 2018 18:52:56 GMT {"id":77,"input":{"some_key": [1, 2, 3]}} ----- ### `POST /queue/{queue_name}/job` Create a new job on given queue with given JSON. #### Request The request body must contain JSON of the form: {"input": , "tags": , "timeout": , "heartbeat_timeout": , "expires_after": , "retries": , "retry_delays": } All fields are optional (i.e. `{}` is the minimum valid job that can be created). `input` represents the job's main payload, and is what clients that accept the job will use to work on. This will typically contain input data to process, and can be any simple or complex JSON value. Defaults to `null` if not specified. `tags` are an optional list of strings to attach to this job. These can be used to look up jobs by tag using the `/tag` endpoints. Uses might include attaching a username to jobs, or attaching a batch ID to a number of related jobs created by the same process. Defaults to `[]` if not specified. `timeout` is the maximum amount of time the job can run before it's marked as timed out. Default is to use the queue's setting. `heartbeat_timeout` is the maximum amount of time between heartbeats before the job is marked as timed out. Default is to use the queue's setting. `expires_after` is the amount of time the job and its metadata will remain in Ocypod after the job completes, fails, or times out (with no retries remaining). Default is to use the queue's setting. `retries` is the number of times to re-queue this job on failure or time out. Default is to use the queue's setting. `retry_delays` is a list of durations specifying the minimum time to wait before re-queuing a job on failure. This can be used to specify linear/exponential/any backoff intervals when retrying jobs. Default is to use the queue's setting. If the number of retries exceeds the length of this list, then the last value will always be used, e.g. with `retries: 4` and `retry_delays: ["10s", "1m", "5m"]`, the job won't be retried for at least 10 seconds after the 1st failure, for 1 minute after the 2nd failure, for 5 minutes after the 3rd failure, and for 5 minutes on the 4th failure. #### Returns 201 - job successfully created, response contains ID of new job, and location of job in `location` header 400 - invalid queue name or invalid job creation JSON given 404 - queue with given name not found --- ### `GET /queue/{queue_name}/job_ids` Get a list of job IDs by status for all jobs originally created in the given queue. #### Returns JSON response of the form: {: [, ...], ...} * 200 - JSON response as described above * 400 - invalid queue name given * 404 - queue with given name not found #### Example $ curl -i localhost:8023/queue/example/job_ids {"completed":[1,2],"cancelled":[],"timed_out":[],"queued":[4,5],"running":[3],"failed":[]} ## Job endpoints Endpoints for interacting with jobs in any state. --- ### `GET /job/{job_id}[?fields=]` Get metadata about given job as a JSON object. Optionally accepts a comma separated list of fields to fetch, otherwise will get all fields by default. Response JSON is an object with each field being a key, i.e. {: , : , ... } #### Returns * 200 - JSON object containing requested fields * 400 - unable to parse list of fields given, or unrecognised field given * 404 - job with given ID does not exist #### Example $ curl 'localhost:8023/job/77?fields=id,status,started_at,ended_at' {"id":77, "status":"running", "started_at":"2018-11-20T18:52:42.700853Z", "ended_at":null} --- ### `PATCH /job/{job_id}` Clients can use this to modify a job's status, and/or its `output` field. This endpoint is generally used to mark a job as completed/failed/cancelled, to write progress/error information to `output`, and to write final results to `output`. #### Request The request must be JSON of the form: {"status": ("completed"|"failed"|"cancelled"), "output": } All fields are optional, only fields that are present will cause any changes. #### Response * 204 - job successfully updated * 400 - invalid or no JSON sent * 404 - no job with given ID exists * 409 - job is in state where status or output update is not allowed #### Example Mark a job as completed, and write its results: $ curl -i -XPATCH -H 'content-type: application/json' -d '{"status": "completed", "output": {"result": 321, "progress": 100.0, "warnings": 3}}' localhost:8023/job/23 HTTP/1.1 204 OK date: Wed, 21 Nov 2018 11:13:34 GMT --- ### `DELETE /job/{job_id}` Delete a job from Ocypod, regardless of its state. Any jobs which have ended and have a non-zero `expires_after` field will eventually be cleaned up once they've ended, so this endpoint is only needed if this is something you want manual control over. While you can delete jobs that are currently running, you'll need to make sure that clients/workers gracefully handle this case. For more graceful job removal, you can set a job's status to `cancelled`, and allow clients to handle this. #### Response * 204 - job successfully deleted * 404 - job with given ID does not exist #### Example $ curl -XDELETE localhost:8023/job/123 HTTP/1.1 204 OK date: Wed, 21 Nov 2018 11:13:34 GMT --- ### `PUT /job/{job_id}/heartbeat` Used by clients that are working on a job to send a heartbeat for it. This is lets the server know that the job is still actively being worked on. This is only useful for jobs with a non-zero `heartbeat_timeout` set, and can only be used on jobs that are running. #### Response * 204 - heartbeat successfully updated * 404 - job with given ID does not exist * 409 - job is not in a state where heartbeat can be updated (e.g. job is queued/failed/cancelled etc.) #### Example $ curl -XPUT localhost:8023/job/23/heartbeat HTTP/1.1 204 OK date: Wed, 21 Nov 2018 11:13:34 GMT --- ### `GET /job/{job_id}/output` Get a job's `output` JSON field. #### Response 200 - JSON contents of the job's `output` field 404 - job with given ID does not exist #### Example $ curl localhost:8023/job/123/output {"a": 1, "b": [2, 3, 4]} --- ### `PUT /job/{job_id}/output` Set a job's `output` field to given JSON. This has the same effect as updating using the `PATCH /job/{job_id}` endpoint and setting the `output` field in the request JSON, but is often more convenient if only updating this field. #### Response * 204 - job's output field successfully set * 400 - invalid or no JSON provided * 404 - job with given ID does not exist * 409 - job is not in a state where output can be update (e.g. job is already completed/cancelled) #### Example $ curl -i -XPUT -H 'content-type: application/json' -d '[1,2,3]' localhost:8023/job/12/output HTTP/1.1 204 OK date: Wed, 21 Nov 2018 11:13:34 GMT --- ## Tag endpoints Information about jobs that were created with tags can be retreived here. Tags are typically used as a way of grouping multiple jobs in some way. Use cases include e.g.: * using a "username" tag, to track which users have submitted which jobs * using a "batch ID" tag, to track jobs related to an ETL run or other batch process * using a "host" tag, to track which hosts created jobs --- ### `GET /tag/{tag_name}` Get a JSON list of job IDs with given tag. If no jobs have the requested tag, or if the tag doesn't exist, then an empty list will be returned. #### Response * 200 - JSON list of job ID integers * 400 - invalid tag name requested #### Example $ curl localhost:8023/tag/user3 [17, 19, 225] --- ## Information endpoints These provide information about the Ocypod system as a whole. ### `GET /info` TODO ### `GET /info/version` Get the version string for the currently running Ocypod server. #### Response * 200 - JSON version string #### Example $ curl localhost:8023/info/version "0.1.2" ## Healthcheck endpoints Provides a way of checking the health of the Ocypod server. Returns JSON containing the health of the system, and optionally a message about any error detected. ### `GET /health` Get JSON indicating the health of the Ocypod server. Currently this checks whether it can successfully connect to and communicate with Redis. Returns JSON of the form: {"status": ("healthy"|"unhealthy"), "error": } The `error` field is optionally present if they status is "unhealthy", the "status" field is always present. #### Response * 200 - health check completed #### Example $ curl localhost:8023/health {"status": "healthy"}