{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "29460883",
   "metadata": {},
   "source": [
    "# Using the Logging API"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a4c271a",
   "metadata": {},
   "source": [
    "## Introduction to the Logging API\n",
    "\n",
    "The Logging API enables applications to subscribe to real-time log messages from a simulation, query previously logged messages from the MySQL database, and publish messages to their either own log or their GridAPPS-D logs.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e099b57b",
   "metadata": {},
   "source": [
    "![subscribe-publish-to-logs](images/3.8/09_subscribe_publish_logs.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b281542",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "b1d44cb7",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "eeec5709",
   "metadata": {},
   "source": [
    "### API Communication Channel\n",
    "\n",
    "As with the Simulation API, the logging API uses both static `/queue/` and dynamic `/topic/` communication channel names depending on whether the API is being used for real-time simulation logs or historic logs that have already been saved in the database.\n",
    "\n",
    "For a review of GridAPPS-D topics, see [Lesson 1.4.](Lesson%201.4.%20GridAPPS-D%20Topics.ipynb)\n",
    "\n",
    "The correct topic for each Logging API call will be provided in the corresponding section for each API task below."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f88dda1",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f82b40ef",
   "metadata": {},
   "source": [
    "### Message Structure\n",
    "\n",
    "Logging messages in the GridAPPS-D environment follow the format of a python dictionary or equivalent JSON string with the format below.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8e60ca5",
   "metadata": {},
   "source": [
    "```\n",
    "{       KEY               VALUE\n",
    "    \"source\":               filename,\n",
    "    \"processId\":            simulation_id,\n",
    "    \"timestamp\":            epoch time number,\n",
    "    \"processStatus\":        \"STARTED\" or \"STOPPED\" or \"RUNNING\" or \"ERROR\" or \"PASSED\" or \"FAILED\",\n",
    "    \"logMessage\":           string,\n",
    "    \"logLevel\":             \"INFO\" or \"DEBUG\" or \"ERROR\",\n",
    "    \"storeToDb\":            true or false\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1461b9ca",
   "metadata": {},
   "source": [
    "All of the messages from a single instantiation will have the same format, with the only difference being the logMessage. As a result, it is possible to use the shortcuts available from the GridAPPSD-Python library to build out the repetitive portions of the message and pass just the logMessage string. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c4a56fb",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "446952ee",
   "metadata": {},
   "source": [
    "## GridAPPSD-Python Logging API Extensions\n",
    "\n",
    "The GridAPPSD-Python library uses several extensions to the standard Python logging library that enable applications to easily create log messages using the same syntax. These extensions support the additional log message formatting required by GridAPPS-D, such as simulation_id, log source, and process status.\n",
    "\n",
    "The following code block enables the "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f9a3a01",
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "import os\n",
    "\n",
    "os.environ['GRIDAPPSD_APPLICATION_ID'] = 'gridappsd-sensor-simulator'\n",
    "os.environ['GRIDAPPSD_APPLICATION_STATUS'] = 'STARTED'\n",
    "os.environ['GRIDAPPSD_SIMULATION_ID'] = opts.simulation_id"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "227999db",
   "metadata": {},
   "source": [
    "__Important!__ Run this import command ___BEFORE___ creating the GridAPPS-D connection object `gapps = GridAPPSD(...)`.\n",
    "\n",
    "__Note:__ If your application is containerized in Docker and registered with the GridAPPS-D platform using the docker-compose file, these extensions will be imported automatically."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66ae7468",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70099831",
   "metadata": {},
   "source": [
    "## Subscribing to Simulation Logs\n",
    "\n",
    "Similar to the two approaches used to subscribe to simulation measurements discussed in [Comparison of Subscription Approaches](../api_usage/3.6-Controlling-Simulation-API.ipynb#Comparison-of-Subscription-Approaches), it is possible to use either a function or a class definition to subscribe to the simulation logs."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a041b175",
   "metadata": {},
   "source": [
    "### Subscription API Communication Channel\n",
    "\n",
    "This is a dynamic `/topic/` communication channel that is best implemented by importing the GriAPPSD-Python library function for generating the correct topic. \n",
    "\n",
    "* `from gridappsd.topics import simulation_log_topic`\n",
    "* `log_topic = simulation_log_topic(simulation_id)`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d728b80",
   "metadata": {},
   "source": [
    "__Note on Jupyter Notebook environment:__ In the examples below, the Jupyter Notebook environment does not update definitions of the subscription object or function definitions. As a result, it is necessary to restart the notebook kernel. The gapps connection object definition is included again for convenience in executing the notebook code blocks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "93112f74",
   "metadata": {},
   "outputs": [],
   "source": [
    "viz_simulation_id = \"paste sim id here\"\n",
    "\n",
    "# Establish connection to GridAPPS-D Platform:\n",
    "from gridappsd import GridAPPSD\n",
    "\n",
    "# Set environment variables - when developing, put environment variable in ~/.bashrc file or export in command line\n",
    "# export GRIDAPPSD_USER=system\n",
    "# export GRIDAPPSD_PASSWORD=manager\n",
    "\n",
    "import os # Set username and password\n",
    "os.environ['GRIDAPPSD_USER'] = 'tutorial_user'\n",
    "os.environ['GRIDAPPSD_PASSWORD'] = '12345!'\n",
    "\n",
    "# Connect to GridAPPS-D Platform\n",
    "gapps = GridAPPSD(viz_simulation_id)\n",
    "assert gapps.connected"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ae91605",
   "metadata": {},
   "outputs": [],
   "source": [
    "from gridappsd.topics import simulation_log_topic\n",
    "log_topic = simulation_log_topic(viz_simulation_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44ba5d29",
   "metadata": {},
   "source": [
    "__Note on Jupyter Notebook environment:__ In the examples below, the Jupyter Notebook environment does not update definitions of the subscription object or function definitions. As a result, it is necessary to restart the notebook kernel. The gapps connection object definition is included again for convenience in executing the notebook code blocks"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27098979",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25065447",
   "metadata": {},
   "source": [
    "### Subscribing using a Function Definition\n",
    "\n",
    "The first approach used to subscribe to measurements is to define a function with the correct set of arguments that is then passed to the `.subscribe()` method associated with the `GridAPPPSD()` object.\n",
    "\n",
    "The function does not require a specific name, and is somewhat easier to define and use. However, the arguments of the function need to be named correctly for the GridAPPSD-Python library to process the simulation output correctly.\n",
    "\n",
    "The format for the function definition is \n",
    "\n",
    "```\n",
    "def myLogFunction(header, message):\n",
    "    # do something when receive a log message\n",
    "    # do something else\n",
    "```\n",
    "\n",
    "That function handle is then passed as an argument to the `.subscribe(topic, function_handle)` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9911691c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def demoLogFunction(header, message):\n",
    "    timestamp = message[\"timestamp\"]\n",
    "    log_message = message[\"logMessage\"]\n",
    "    \n",
    "    print(\"Log message received at timestamp \", timestamp, \"which reads:\")\n",
    "    print(log_message)\n",
    "    print(\"........................\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e52382f",
   "metadata": {},
   "outputs": [],
   "source": [
    "gapps.subscribe(log_topic, demoLogFunction)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ef62d16",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf997880",
   "metadata": {},
   "source": [
    "### Subscribing using a Class Definition\n",
    "\n",
    "The second approach used to subscribe to simulation logs is to define add a custom method to the same class with `__init__` and `on_message` methods that was created to subscribe to measurements.\n",
    "\n",
    "Unlike the Simulation API, the Logging API does not require a specific name for the method used to subscribe to log messages. \n",
    "\n",
    "It is possible to create additional methods in the subscription class definition to enable the app to subscribe to additional topics, such as the simulation log topic, as shown in the example below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "659e9119",
   "metadata": {},
   "outputs": [],
   "source": [
    "class PlatformDemo(object):\n",
    "    # A simple class for interacting with simulation\n",
    "\n",
    "    def __init__(self, simulation_id, gapps_obj):\n",
    "        # Initialize variables and attributes\n",
    "        self._gapps = gapps_obj\n",
    "        self._simulation_id = simulation_id\n",
    "        # self.foo = bar\n",
    "        \n",
    "    def on_message(self, headers, message):\n",
    "        # Do things with measurements\n",
    "        meas_value = message[\"message\"][\"measurements\"]\n",
    "        # Do more stuff with measurements\n",
    "        \n",
    "    def my_logging_method(self, headers, message):\n",
    "        timestamp = message[\"timestamp\"]\n",
    "        log_message = message[\"logMessage\"]\n",
    "        \n",
    "        print(\"Log message received at timestamp \", timestamp, \"which reads:\")\n",
    "        print(log_message)\n",
    "        print(\"........................\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e37641d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create subscription object\n",
    "demo_obj = PlatformDemo(viz_simulation_id, gapps)\n",
    "\n",
    "# Subscribe to logs using method\n",
    "gapps.subscribe(log_topic, demo_obj.my_custom_method)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62b510c1",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93b6b01e",
   "metadata": {},
   "source": [
    "## Publishing to Simulation Logs\n",
    "\n",
    "The GridAPPSD-Python library enables use of the standard Python logging syntax to create logs, publish them to the GOSS Message Bus, and store them in the MySQL database. \n",
    "\n",
    "Documentation of the standard Python logging library is available on [Python Docs](https://docs.python.org/3/library/logging.html).\n",
    "\n",
    "\n",
    "It is possible to publish to either local app logs (which are more useful for debugging) or the GridAPPS-D logs (which can be accessed by other applications and should be used for completed applications).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aaf4f8df",
   "metadata": {},
   "source": [
    "## 5.1. Publishing to Local App Logs\n",
    "\n",
    "The first approach is to use the default Python logger to write to local app logs by importing the `logging` library and then use the `.getLogger()` method from the Python library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "84054633",
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "\n",
    "python_log = logging.getLogger(__name__)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f91d821",
   "metadata": {},
   "source": [
    "Log messages can then be published by invoking the methods \n",
    "\n",
    "* `python_log.debug(\"log message\")`\n",
    "\n",
    "* `python_log.info(\"log message\")`\n",
    "\n",
    "* `python_log.warning(\"log message\")`\n",
    "\n",
    "* `python_log.error(\"log message\")`\n",
    "\n",
    "* `python_log.fatal(\"log message\")`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2216822a",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18f40cf7",
   "metadata": {},
   "source": [
    "## 5.2. Publishing to GridAPPS-D Logs\n",
    "\n",
    "The second approach is to use the GridAPPS-D logs. Importing the python logging library is not necessary. Instead initialize a logging object using the `.get_logger()` method associated with the GridAPPS-D connection object. Note the difference in spelling of the GridAPPS-D Library and default Python Library methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28021be4",
   "metadata": {},
   "outputs": [],
   "source": [
    "gapps_log = gapps.get_logger()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22c0a519",
   "metadata": {},
   "source": [
    "Log messages can then be published by invoking the methods \n",
    "\n",
    "* `gapps_log.debug(\"log message\")`\n",
    "\n",
    "* `gapps_log.info(\"log message\")`\n",
    "\n",
    "* `gapps_log.warning(\"log message\")`\n",
    "\n",
    "* `gapps_log.error(\"log message\")`\n",
    "\n",
    "* `gapps_log.fatal(\"log message\")`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94eabb47",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba984959",
   "metadata": {},
   "source": [
    "## Querying Saved Logs\n",
    "\n",
    "Log messages published using the Logging API and the GOSS Message Bus are saved to the MySQL database. These log messages can be accessed with a Logging API query. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65ae05bc",
   "metadata": {},
   "source": [
    "### Log Query API Communication Channel\n",
    "\n",
    "The query for logs uses a static `/queue/` channel that is imported from the GridAPPS-D Topics library. \n",
    "\n",
    "This topic is used with the `.get_response(topic, message)` method associated with the GridAPPS-D connection object.\n",
    "\n",
    "* `from gridappsd import topics as t`\n",
    "* `gapps.get_response(t.LOGS, message)`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9c442c6",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2665bbe7",
   "metadata": {},
   "source": [
    "### Structure of the Log Query Message\n",
    "\n",
    "The first approach to querying with the Logging API is to use a python dictionary or equivalent JSON string that follows formatting similar to the query messages used by all the other GridAPPS-D APIs:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "beea4706",
   "metadata": {},
   "outputs": [],
   "source": [
    "from gridappsd import topics as t\n",
    "\n",
    "message = {\n",
    "    \"source\": \"ProcessEvent\",\n",
    "    \"processId\": viz_simulation_id,\n",
    "    \"processStatus\": \"INFO\",\n",
    "    \"logLevel\": \"INFO\"\n",
    "}\n",
    "\n",
    "gapps.get_reponse(t.LOGS, message)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcd6c87d",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1bb1759e",
   "metadata": {},
   "source": [
    "![GridAPPS-D-narrow.png](../images/GridAPPS-D_narrow.png)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}