## Portal - Full Text Search Web Service [![crates.io](https://meritbadge.herokuapp.com/portal)](https://crates.io/crates/portal) ### Purpose The purpose of this service is to be your full text search web service for any JSON client including JavaScript front-ends like React with fetch. Portal is a competitor to [ElasticSearch](https://www.elastic.co/), [MeiliSearch](https://github.com/meilisearch/MeiliSearch), [AWS CloudSearch](https://aws.amazon.com/cloudsearch/), [AWS ElasticSearch Service](https://aws.amazon.com/elasticsearch-service/), and [Algolia](https://www.algolia.com/). Portal is built to be simple and blazing fast with JWT verification, indexing, deindexing, search, and word suggest. Indexing is batch indexing by default. To use this service you need to have a running [sonic](https://crates.io/crates/sonic-server) server and a [broker](https://crates.io/crates/broker) server. ### Features * Very performant with almost no CPU and memory usage * Supports full-text search and word suggestions in 87 natural languages * Under 1000 lines of code * Supports CORS * Supports JWT authentication * Supports JWT caching with expiry checking to minimize verify API calls * Multi-tenant * Supports SSL - full end-to-end encryption * JSON API * Auto-provision and renews SSL cert via LetsEncrypt or use your own SSL cert * Uses user authorization scoping * Built on [sonic](https://crates.io/crates/sonic-server) and [broker](https://crates.io/crates/broker) ### Use 1. create a user on [broker](https://crates.io/crates/broker) with the following scopes - for full permissions `portal:full` or granular permissions `portal:create_bucket, portal:delete_bucket, portal:index, portal:deindex, portal:search, or portal:suggest` 2. login to broker and get a JWT 3. attach the JWT as an Authorization: Bearer {token} to the following JSON API endpoints 4. create a bucket #### Create Bucket ```html POST /create_bucket ``` - authenticated endpoint (Authorization: Bearer {jwt}) example: ```json { "collection": "coffee", "name": "tenant_1" } ``` will return: `200` or `500` or `400` or `401` - note: bucket names must be globally unique #### Delete Bucket ```html POST /delete_bucket ``` - authenticated endpoint (Authorization: Bearer {jwt}) example: ```json { "collection": "coffee", "name": "tenant_1" } ``` will return: `200` or `500` or `400` or `401` #### Index ```html POST /index ``` - authenticated endpoint (Authorization: Bearer {jwt}) example: ```json { "items": [{ "collection": "coffee", "bucket": "tenant_1", "id": "49e28aae-88d4-4c19-86d8-51f2c9f11039", "data": { "name": "roasted", "image": "https://img.com/bucket/123/123.jpg" }, "locale": "eng", "indexes": ["name"] }] } ``` will return: `200` or `500` or `400` or `401` - note: only buckets the user has created can be used in the index call - note: `locale` is an optional field of an [ISO 639-3 locale code](https://iso639-3.sil.org/code_tables/639/data) - if not defined the locale will be auto-detected - Supported Locales: * afr => Afrikaans * aka => Akan * amh => Amharic * ara => Arabic * azj => Azerbaijani (North) * bel => Belarusian * ben => Bengali * bho => Bhojpuri * bul => Bulgarian * cat => Catalan, Valencian * ceb => Cebuano * ces => Czech * cmn => Mandarin Chinese * dan => Danish * deu => German * ell => Greek (Modern) * eng => English * epo => Esperanto * est => Estonian * fin => Finnish * fra => French * guj => Gujarati * hat => Haitian, Haitian Creole * hau => Hausa * heb => Hebrew * hin => Hindi * hrv => Croatian * hun => Hungarian * ibo => Igbo * ilo => Iloko * ind => Indonesian * ita => Italian * jav => Javanese * jpn => Japanese * kan => Kannada * kat => Georgian * khm => Khmer, Central Khmer * kin => Kinyarwanda * kor => Korean * kur => Kurdish * lat => Latin * lav => Latvian * lit => Lithuanian * mai => Maithili * mal => Malayalam * mar => Marathi * mkd => Macedonian * mlg => Malagasy * mya => Burmese * nep => Nepali (macrolanguage) * nld => Dutch, Flemish * nno => Norwegian Nynorsk * nob => Norwegian Bokmål * nya => Chewa, Chichewa, Nyanja * ori => Oriya (macrolanguage) * orm => Oromo * pan => Panjabi, Punjabi * pes => Iranian Persian * pol => Polish * por => Portuguese * ron => Moldavian, Moldovan, Romanian * run => Rundi * rus => Russian * sin => Sinhala, Sinhalese * skr => Saraiki, Seraiki * slk => Slovak * slv => Slovenian * sna => Shona * som => Somali * spa => Castilian, Spanish * srp => Serbian * swe => Swedish * tam => Tamil * tel => Telugu * tgl => Tagalog * tha => Thai * tir => Tigrinya * tuk => Turkmen * tur => Turkish * uig => Uighur, Uyghur * ukr => Ukrainian * urd => Urdu * uzb => Uzbek * vie => Vietnamese * ydd => Eastern Yiddish * yor => Yoruba * zul => Zulu #### Search ```html POST /search ``` - authenticated endpoint (Authorization: Bearer {jwt}) ```json { "collection": "coffee", "bucket": "tenant_1", "query": "roasted", "limit": 10, "offset": 10 } ``` - `limit` and `offset` are optional fields will return: `200` or `500` or `400` or `401` 200 - will return an array of objects ```json [ { "collection": "coffee", "bucket": "tenant_1", "id": "49e28aae-88d4-4c19-86d8-51f2c9f11039", "data": { "name": "roasted", "image": "https://img.com/bucket/123/123.jpg" }, "locale": "eng", "indexes": ["name"] } ] ``` #### Word Suggest ```html POST /suggest ``` - authenticated endpoint (Authorization: Bearer {jwt}) ```json { "collection": "coffee", "bucket": "tenant_1", "query": "r", "limit": 10 } ``` - note: limit is an optional field will return: `200` or `500` or `400` or `401` ```json { "suggestions": ["roasted"] } ``` #### Deindex ```html POST /deindex ``` - authenticated endpoint (Authorization: Bearer {jwt}) ```json { "ids": ["49e28aae-88d4-4c19-86d8-51f2c9f11039"] } ``` will return: `200` or `500` or `400` or `401` #### Health Check ```html GET or HEAD / ``` - public endpoint will return: `200` ### Install ``` cargo install portal ``` - the `origin` can be passed in as a flag - default `*` - the `port` can be passed in as a flag - default `8888` - can only be set for unsecure connections - the `secure` flag for https can be true or false - default `false` - the `auto_cert` flag for an autorenewing LetsEncrypt SSL cert can be true or false - requires a resolvable domain - default `true` - the `key_path` flag when `auto_cert` is `false` to set the SSL key path for your own cert - default `certs/private_key.pem` - the `cert_path` flag when `auto_cert` is `false` to set the SSL cert path for your own cert - default `certs/chain.pem` - the `certs` flag is the storage path of LetsEncrypt certs - default `certs` - the `db` flag is the path where the embedded database will be saved - default `db` - the `domain` flag is the domain name (e.g. api.broker.com) of the domain you want to register with LetsEncrypt - must be fully resolvable - the `sonic_server` flag is the sonic domain/ip/port of the sonic server - default `localhost:1491` - the `sonic_password` flag is the sonic password set in the sonic config file - default `SecretPassword` - the `broker` flag is the broker domain/ip/port of the broker server - default `http://localhost:8080` - production example: `./portal --secure="true" --domain="index.broker.com" --sonic_server="sonic.broker.com" --sonic_password="wj34T%$Dx" --broker="https://broker.broker.com"` ### Service There is an example `systemctl` service for Ubuntu called `portal.service` in the code ### TechStack * [Tide](https://crates.io/crates/tide) * [RocksDB](https://crates.io/crates/rocksdb) ### Inspiration * [Broker](https://crates.io/crates/broker) * [Sonic](https://crates.io/crates/sonic-server) * [ElasticSearch](https://www.elastic.co/) * [MeiliSearch](https://github.com/meilisearch/MeiliSearch)