09-about-solr.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0518244f-cf18-45c3-8ee1-3cbbecfba478",
   "metadata": {},
   "source": [
    "## About the Apache Solr search engine in the Smart Village Platform\n",
    "\n",
    "> I asked my search engine for a good joke. It said, \"A man walked into a bar, tavern, pole, inn, stake, tap room, stick, pub, shaft, parlour, shank, watering hole, rail, cantina, beam, alehouse, spoke, or saloon... but most likely a bar, and said ouch.\" \n",
    "\n",
    "An open source search engine like Apache Solr is required by the Smart Village application API to serve up API stored objects as quickly as possible. APIs that are backed by a search engine have numerous additional benefits compared to an API backed by a traditional relational database. A search engine is always indexed for the fastest data retrieval possible. The advanced data and query parsing of a search engine allows for extremely fast full text search, filtering, and sorting of the data. Search engine data can also be grouped, faceted, and pivoted on, for an advanced set of analytics and statistics on specific data in your query. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d93baf20-7419-4c47-be53-2e33907fe263",
   "metadata": {},
   "source": [
    "## Deploy Solr in NERC OpenShift\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab489b93-8f0b-4080-b568-e728f14fb40d",
   "metadata": {},
   "source": [
    "Run the command below to deploy the default computate Solr ConfigSet as a Kubernetes ConfigMap to the cloud. These configsets will be loaded later, when the Solr pod is running to initialize the Solr ConfigSet which is the schema used for Solr Collections in the Smart Village Platform. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c98ce0c1-3f92-49f4-b13f-d2dde76e8159",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "oc -n smart-village-faeeb6c apply -f ~/smartvillage-operator/kustomize/overlays/nerc-ocp-prod/ansible/edgesolrs/default/configmaps/computate-configset/configmap.yaml\n",
    "echo DONE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3943fe77-0430-48b2-9a25-95ea67e3a887",
   "metadata": {},
   "source": [
    "If you are curious what the EdgeSolr custom resource definition looks like that you are deploying in NERC OpenShift, run the command below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1ef2f6bc-4e99-4e05-9cba-1dc7209117f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "cat ~/smartvillage-operator/kustomize/overlays/nerc-ocp-prod/ansible/edgesolrs/default/edgesolr.yaml\n",
    "echo DONE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ef92204-0bd9-4aea-84cf-cfb09c990fa6",
   "metadata": {},
   "source": [
    "Here are some useful things to note about the configuration of the IoT Agent JSON. \n",
    "\n",
    "- `name: solr` We can name the deployment, service, and route created by this EdgeSolr resource. \n",
    "- `route`: We create a public route for Solr if you want to interact with it from your own computer. \n",
    "- `replicas: 1` We'll only deploy 1 replica for NERC OpenShift to stay within the resource quota of NERC OpenShift. \n",
    "- The `resources:` definition defines the memory and CPU requests and limits for our Solr pod. We have to keep this lower than I would recommend for a production deployment because we have limited resources available in NERC OpenShift for the many running Smart Village services. \n",
    "- `configsets:` this is a list of Solr ConfigSets given by a name, and a Kubernetes ConfigMap, which we deployed in the step above. These are deployed to the Solr pod in the right directory to be registered as a ConfigSet. \n",
    "- `collections:` this is a list of Solr Collections where we can actually store and query search results. We create two collections, one called `computate` which is used for AI/ML code generation in the Smart Village Platform. The other collection `smartvillage` is where we store Smart Village Platform data that is returned in the REST API. Both of these collections use the same `computate` ConfigSet, because it's very reusable with several good wildcard fields to solve many problems. \n",
    "- `zookeeper:` we configure the host name for the zookeeper service, because Solr is clustered and scalable thanks to Zookeeper. \n",
    "\n",
    "For more information about the EdgeSolr custom resource definition, [see the full EdgeSolr schema here](https://github.com/smartabyar-smartvillage/smartvillage-operator/blob/main/config/crd/bases/smartvillage.computate.org_edgesolrs.yaml). "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92da8063-cb43-46fe-bf01-a3feb3158948",
   "metadata": {},
   "source": [
    "Run the Ansible Playbook below to deploy Solr to the cloud. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7a7a64d0-fe90-449a-8b65-4c4ab24f3a87",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "ansible-playbook ~/smartvillage-operator/apply-edgesolr.yaml \\\n",
    "  -e ansible_operator_meta_namespace=smart-village-faeeb6c \\\n",
    "  -e crd_path=~/smartvillage-operator/kustomize/overlays/nerc-ocp-prod/ansible/edgesolrs/default/edgesolr.yaml\n",
    "echo DONE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e8c758d-3e51-4cf2-87d6-61c53289de22",
   "metadata": {},
   "source": [
    "You may see a play recap that has failed. \n",
    "This is expected because the Solr pod is just now getting created. \n",
    "The final tasks in the playbook expect the solr pod to be running to connect to the running pod and create the Solr ConfigSets and Solr Collections used by the Smart Village application. \n",
    "\n",
    "Retry the playbook once the Solr pod is running. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65f24d70-84c7-4b8b-bc91-23146988eb36",
   "metadata": {
    "tags": []
   },
   "source": [
    "### View Solr pod details\n",
    "After running the Ansible Playbook, it will take a minute before the Solr pod is up and running. Run the command below until the Solr pod health checks are `READY 1/1` and `STATUS Running`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "394e0835-4cab-48aa-85a3-0960fcede187",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "oc -n smart-village-faeeb6c get pod -l app=solr\n",
    "oc -n smart-village-faeeb6c wait pod -l app=solr --for=condition=Ready --timeout=2m\n",
    "oc -n smart-village-faeeb6c get pod -l app=solr\n",
    "echo DONE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "002f3ccb-4a88-4ab9-b120-6b10f6fffbd2",
   "metadata": {},
   "source": [
    "When the Solr pod is running, run the Ansible Playbook again. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a6b69e1a-1390-462f-a497-15d814ec7e20",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "ansible-playbook ~/smartvillage-operator/apply-edgesolr.yaml \\\n",
    "  -e ansible_operator_meta_namespace=smart-village-faeeb6c \\\n",
    "  -e crd_path=~/smartvillage-operator/kustomize/overlays/nerc-ocp-prod/ansible/edgesolrs/default/edgesolr.yaml\n",
    "echo DONE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd198430-2694-4c59-a256-d7a08367cecd",
   "metadata": {},
   "source": [
    "### View Solr pod logs\n",
    "If your Solr pod does not reach the STATUS Running, you can run the command below to view the pod logs of Solr and check for other errors that may have occured. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c3ed9ecc-6d74-4711-a27e-242094fe05a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "oc -n smart-village-faeeb6c logs -l app=solr\n",
    "echo DONE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e50c315-685d-41e5-8dea-fec9e6861371",
   "metadata": {},
   "source": [
    "## Next...\n",
    "I hope that answers your questions about Solr in the Smart Village Platform. \n",
    "- If you have additional questions or issues, please [create an issue for the course here](https://github.com/smartabyar-smartvillage/smartabyar-smartvillage-nerc-course/issues). \n",
    "- Otherwise, please continue to the next notebook [10-about-smartvillage.ipynb](10-about-smartvillage.ipynb). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a3d68b82-e612-4147-a4bd-f4ca0c7cfd05",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}