{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Using the Timeseries API" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Introduction to the Timeseries API\n", "\n", "The Timeseries API is used to query the Influx Database, which stores measurement data from simulations. The API calls can be used to \n", "* obtain weather data\n", "* obtain measurements from simuation data using measurement mRIDs\n", "* obtain equipments commands and other simulation input data\n", "* obtain simulated field data from the Sensor Service" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## API Syntax Overview\n", "\n", "__Application passes query to GridAPPS-D Platform__\n", "\n", "First, the application creates a query message for requesting information about the desired power system components in the format of a JSON string or equivalant Python dictionary object. The syntax of this message is explained in detail below.\n", "\n", "The query is sent using the `gapps.get_response(topic, message)` method on the Timeseries queue channel with a response expected back from the GridAPPS-D platform within the specified timeout period.\n", "\n", "The application then passes the query through the Timeseries API to the GridAPPS-D Platform, which publishes it to the `goss.gridappsd.process.request.data.timeseries` queue channel on the GOSS Message Bus. If the app is authenticated and authorized to pass queries, the query message is delivered to the Data Managers, which obtain the desired information from the Timeseries Influx Database. \n", "\n", "__GridAPPS-D Platform responds to Application query__\n", "\n", "The Data Managers then publish the response from the Timeseries Influx Database to the appropriate queue channel. The Timeseries API then returns the desired information back to the application as a JSON message or equivalant Python dictionary object." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### API Communication Channel\n", "\n", "API calls to the timeseries databases use a static `/queue/` channel, which was covered in [Lesson 3.1](3.1-API-Communication-Channels.ipynb#3.2.-Request-Timeseries-Data). The topic can be specified as a text string or by importing the topics library:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "__Text String:__ The topic can be specified as a static string:\n", "\n", "* `topic = \"goss.gridappsd.process.request.data.timeseries\"`\n", "* `gapps.get_response(topic, message)`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "__GridAPPSD-Python Library Method:__ The correct topic can also be imported from the GridAPPSD-Python topics library:\n", "\n", "* `from gridappsd import topics as t`\n", "* `gapps.get_response(t.TIMESERIES, message)`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Structure of a Query Message\n", "\n", "Queries passed to Timeseries API are formatted as python dictionaries or equivalent JSON scripts wrapped as a python string. \n", "\n", "The accepted set of key-value pairs for the Timeseries API query message is\n", "\n", "```\n", "message = \"\"\"\n", "{\n", " \"queryMeasurement\": \"INSERT QUERY HERE\",\n", " \"queryFilter\": {\"key1\": \"value1\"\n", " \"key2\": \"value2\"},\n", " \"responseFormat\": \"JSON\"\n", "}\n", "```\n", "\n", "The components of the message are as follows:\n", "\n", "* `\"queryMeasurement\":` -- Specifies the type of measurement being requested. Allowed queryMeasurement values are listed in the next section.\n", "\n", "\n", "* `\"queryFilter\":` -- Filters the measurements to just the set values desired by the application. The set of allowed values depends on the value associated with queryMeasurement.\n", "\n", "\n", "* `\"responseFormat\":` -- Specifies the format of the response, can be `\"JSON\"`, `\"CSV\"`, or `\"XML\"`. (CAUTION: the Timeseries API uses the key _reponseFormat_, while the PowerGridModel API uses the key _resultFormat_. Using the wrong key for either API will result in a java.lang error.)\n", "\n", "\n", "\n", "The usage of each of these message components are explained in detail with code block examples below. \n", "\n", "__Important__: Be sure to pay attention to placement of commas ( __,__ ) at the end of each line. Commas are placed at the end of each line _except_ the last line. Incorrect comma placement will result in a JsonSyntaxException. \n", "\n", "All of the queries are passed to the Timeseries API using the `.get_response(topic, message)` method for the GridAPPS-D platform connection variable." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Specifying the queryMeasurement value\n", "\n", "Below are the allowable values associated with the `queryMeasurement` key, which are used to specify the type of data requested by each query. Executable code block examples are provided for each of the requests in the subsections below.\n", "\n", "\n", "* `\"queryMeasurement\": \"weather\"` -- [Query for weather data](#3.1.-Query-for-Weather-Data)\n", "\n", "\n", "* `\"queryMeasurement\": \"simulation\"` -- [Query for simulation output data](#3.2.-Query-for-Simulation-Output-Data) and [query for simulation intput data](#3.3.-Query-for-Simulation-Input-Data)\n", "\n", "\n", "* `\"queryMeasurement\": \"gridappsd-sensor-simulator\"` -- [Query for sensor service data](#3.4.-Query-for-Sensor-Service-Data)\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Querying for Timeseries Data\n", "\n", "This section outlines the details of key-value pairs for the possible queries associated with each value of the `queryMeasurement` key listed above." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Querying for Weather Data\n", "\n", "#### Interpreting GridAPPS-D Weather Data\n", "\n", "The weather data is based on exported data collected from the Solar Radiation Research Laboratory (39.74N,105.18W,1829 meter elevation) January - December 2013. The original dataset was based in Mountain Standard Time (MST).\n", "\n", "The original column names included engineering units, but could not be included during data import. Below is a mapping between the exported column headers and the fields in the Influx database management system. \n", "\n", "```\n", "Original Exported Data Influx Measurement Field Key Field Type\n", "------------------------------------ ---------------------------- ----------\n", "DATE (MM/DD/YYYY) DATE String\n", "MST MST String\n", "Global CM22 (vent/cor) [W/ft^2] GlobalCM22 Float\n", "Direct CH1 [W/ft^2] DirectCH1 Float\n", "Diffuse CM22 (vent/cor) [W/ft^2] Diffuse Float\n", "Tower Dry Bulb Temp [deg F] TowerDryBulbTemp Float\n", "Tower RH [%] TowerRH Float\n", "Avg Wind Speed @ 42ft [MPH] AvgWindSpeed Float\n", "Avg Wind Direction @ 42ft [deg from N] AvgWindDirection Float\n", "\n", "Original Exported Data Influx Measurement Tag Type\n", "------------------------------------ ---------------------------- ----------\n", "n/a lat String\n", "n/a long String\n", "n/a place String\n", "\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Querying Weather Data\n", "\n", "GridAPPS-D contains one year of weather data (details given in next subsection), which can be obtained with the following query\n", "\n", "Measurement request key-value pair:\n", "\n", "* `\"queryMeasurement\": \"weather\"`\n", "\n", "Allowed key-value pairs for `queryFilter`:\n", "\n", "```\n", "\"queryFilter\": { KEY VALUE\n", " \"startTime\": epoch time number,\n", " \"endTime\": epoch time number,\n", " \"AvgWindDirection\": number,\n", " \"AvgWindSpeed\": number,\n", " \"Diffuse\": number,\n", " \"DirectCH1\": number,\n", " \"GlobalCM22\": number,\n", " \"MST\": number,\n", " \"TowerDryBulbTemp\": number,\n", " \"TowerRH\": number,\n", " \"lat\": string,\n", " \"long\": string,\n", " \"place\": string }\n", " \n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Not all of the `queryFilter` key-value pairs need to be used. These are filters that can be used to restrict the returned data to particular values. \n", "\n", "__Note:__ The Timeseries API currently does not support querying for a range of values (e.g. all data for temperature betweeen 25 and 30C). This functionality will be added in a future release. Queries are currently limited to a single value. \n", "\n", "__Example 1: query for just the data between a given `\"startTime\"` and `\"endTime\"`:__" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "topic = \"goss.gridappsd.process.request.data.timeseries\" # Specify Timeseries API GridAPPS-D topic\n", "\n", "# Use queryFilter of \"startTime\" and \"endTime\"\n", "message = {\n", " \"queryMeasurement\":\"weather\",\n", " \"queryFilter\":{\"startTime\":\"1357048800000000\",\n", " \"endTime\":\"1357048860000000\"},\n", " \"responseFormat\":\"JSON\"\n", "}\n", "\n", "gapps.get_response(topic, message) # Pass API call" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "__Example 2: Query for weather data of particular values:__" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "topic = \"goss.gridappsd.process.request.data.timeseries\" # Specify Timeseries API GridAPPS-D topic\n", "\n", "# Use queryFilter of \"TowerDryBulbTemp\" and \"AvgWindSpeed\"\n", "message = {\n", " \"queryMeasurement\":\"weather\",\n", " \"queryFilter\":{\"TowerDryBulbTemp\": 30.0326,\n", " \"AvgWindSpeed\": 7.4624},\n", " \"responseFormat\":\"JSON\"\n", "}\n", "\n", "gapps.get_response(topic, message) # Pass API call" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Query for Simulation Output Data\n", "\n", "All of the measurement output from a simulation is stored in the Influx Database and can be queried with the Timeseries API\n", "\n", "Measurement request key-value pair:\n", "\n", "* `\"queryMeasurement\": \"simulation\"`\n", "\n", "Allowed key-value pairs for `queryFilter` for output data\n", "\n", "```\n", "\"queryFilter\": { KEY VALUE\n", " \"startTime\": epoch time number ,\n", " \"endTime\": epoch time number ,\n", " \"measurement_mrid\": string OR [array of string values] ,\n", " \"simulation_id\": numeric string ,\n", " \"hasSimulationMessageType\": \"OUTPUT\" OR \"INPUT\" ,\n", " \"angle\": number ,\n", " \"magnitude\": number } \n", " \n", "```\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The query response will be in the form\n", "\n", "```\n", "{'data': [{'hasSimulationMessageType': 'OUTPUT',\n", " 'measurement_mrid': '_meas1-1234-abcd-mrid',\n", " 'angle': number,\n", " 'magnitude': number,\n", " 'simulation_id': 'string',\n", " 'time': number},\n", " {'hasSimulationMessageType': 'OUTPUT',\n", " 'measurement_mrid': '_meas2-2345-bcde-mrid',\n", " 'angle': number,\n", " 'magnitude': number,\n", " 'simulation_id': 'string',\n", " 'time': number},\n", " \n", " ...\n", " ] \n", "}\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In the Getting Started section of this tutorial, we started a demo simulation of the IEEE 123 node model. We can pass queries to the Timeseries API to retrieve measurements from the completed simulation. Note that it sometimes takes up to a minute for simulation data written to the Timeseries Database to be available for queries.\n", "\n", "__Example 1: Query for all measurements between a `\"startTime\"` and `\"endTime\"`__\n", "\n", "It is recommended to use a 3 second time interval between startTime and endTime to ensure that simulation measurements are found by the Timeseries Database." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "topic = \"goss.gridappsd.process.request.data.timeseries\" # Specify Timeseries API GridAPPS-D topic\n", "\n", "# Use queryFilter of \"startTime\" and \"endTime\"\n", "message = {\n", " \"queryMeasurement\": \"simulation\",\n", " \"queryFilter\": {\n", " \"simulation_id\": simulation_id,\n", " \"startTime\": \"1570041114\",\n", " \"endTime\": \"1570041117\"},\n", " \"responseFormat\": \"JSON\"\n", "}\n", "\n", "gapps.get_response(topic, message) # Pass API call" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "__Example 2: Query for all measurements associated with list of measurement mRIDs__" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "topic = \"goss.gridappsd.process.request.data.timeseries\" # Specify Timeseries API GridAPPS-D topic\n", "\n", "# Query for a particular set of measurments\n", "message = {\n", " \"queryMeasurement\":\"simulation\",\n", " \"queryFilter\":{\"simulation_id\": simulation_id,\n", " \"measurement_mrid\":[\"_5e2645c3-df1b-4fe7-919e-e1d5274f96cf\",\"_4c0e6f4f-79c4-47c7-afbc-475fa6a6e8e6\"]},\n", " \"responseFormat\":\"JSON\"\n", "} \n", "\n", "gapps.get_response(topic, message) # Pass API call" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "topic = \"goss.gridappsd.process.request.data.timeseries\" # Specify Timeseries API GridAPPS-D topic\n", "\n", "# Query for a particular set of measurments\n", "message = {\n", " \"queryMeasurement\":\"simulation\",\n", " \"queryFilter\":{\"simulation_id\": simulation_id,\n", " \"measurement_mrid\":\"_e302d534-9a09-461c-b541-a6ec77ec8f5c\"},\n", " \"responseFormat\":\"JSON\"\n", "} \n", "\n", "gapps.get_response(topic, message) # Pass API call" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Query for Simulation Input Data\n", "\n", "All of the equipment control commands and other input data from a simulation is stored in the Influx Database and can be queried with the Timeseries API\n", "\n", "Measurement request key-value pair:\n", "\n", "* `\"queryMeasurement\": \"simulation\"`\n", "\n", "Allowed key-value pairs for `queryFilter` for output data\n", "\n", "```\n", "\"queryFilter\": { KEY VALUE\n", " \"startTime\": epoch time number ,\n", " \"endTime\": epoch time number ,\n", " \"measurement_mrid\": mRID string OR [array of string values] ,\n", " \"simulation_id\": numeric string ,\n", " \"hasSimulationMessageType\": \"OUTPUT\" OR \"INPUT\" ,\n", " hasMeasurementDifference \"FORWARD\" OR \"REVERSE\" ,\n", " attribute string , \n", " difference_mrid mRID string ,\n", " object string ,\n", " value number }\n", "```\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "__Known issue in GridAPPS-D releases_2019.09.0:__ Events passed from the Test Manager or from a Configuration File are not registering correctly in the Influx database. Simulation Input from operator control actions in the viz are being stored correctly. For this example, run a simulation of the 123 Node Model in the Viz and open any switch. Then copy the simulation_id and paste it into the first code block below." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "__Example 1: Query for all simulation input commands__" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "viz_simulation_id = \"2043209294\"" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "topic = \"goss.gridappsd.process.request.data.timeseries\" # Specify Timeseries API GridAPPS-D topic\n", "\n", "# Query for all equipment command inputs passed to simulation\n", "message = {\n", " \"queryMeasurement\": \"simulation\",\n", " \"queryFilter\": {\n", " \"simulation_id\": simulation_id,\n", " \"hasSimulationMessageType\": \"INPUT\"},\n", " \"responseFormat\": \"JSON\"\n", "}\n", "\n", "gapps.get_response(topic, message) # Pass API call" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Query for Sensor Service Data\n", "\n", "#### Using the Sensor Service\n", "\n", "The GridAPPS-D Sensor Service simulates the noise and packet losses of real field device measurements based upon the magnitude of “prestine” simulated values from the GridLab-D simulation. This service has been specifically designed to work within the gridappsd platform container. The GridAPPS-D platform will start the service when it is specified as a dependency of an application or when a service configuration is specified within the GridAPPS-D Visualization. \n", "\n", "__Python Application Usage:__\n", "The python application using this service should require gridappsd-sensor-simulator as a requirement. In addition, the following python code shows how to get the correct topic for the service.\n", "\n", "__Service Configuration:__\n", "The sensor-config in the above image shows an example of how to configure a portion of the system to have sensor output. Each mrid (such as `_99db0dc7-ccda-4ed5-a772-a7db362e9818`) will be monitored by this service and either use the default values or use the specified values during the service runtime.\n", "\n", "The general format for the Sensor Service configuration message is\n", "\n", "```\n", "{\n", " \"_99db0dc7-ccda-4ed5-a772-a7db362e9818\": {\n", " \"nominal-value\": 100,\n", " \"perunit-confidence-band\": 0.01,\n", " \"aggregation-interval\": 30,\n", " \"perunit-drop-rate\": 0.01\n", " },\n", " \"_ee65ee31-a900-4f98-bf57-e752be924c4d\":{},\n", " \"_f2673c22-654b-452a-8297-45dae11b1e14\": {}\n", "}\n", "```\n", "\n", "The other options for the service are:\n", "\n", "* `\"default-perunit-confidence-band\"`\n", "* `\"default-aggregation-interval\"`\n", "* `\"default-perunit-drop-rate\"`\n", "* `\"passthrough-if-not-specified\"`\n", "\n", "These options will be used when not specified within the sensor-config block.\n", "\n", "Note: Currently the nominal-value is not looked up from the database. At this time services aren’t able to tell the platform when they are “ready”. This will be implemented in the near future and then all of the nominal-values will be queried from the database.\n", "\n", "A complete example of Sensor Service configuration is available in the sample python script [Run123NodeSensorServiceDemo.py](Run123NodeSensorServiceDemo.py) for three measurements in the IEEE 123 Node Model generated over a two minute simulation.\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Querying Sensor Service Data\n", "\n", "All of the simulated field measurements created by the Sensor Service are stored in the Influx Database and can be queried using the Timeseries API:\n", "\n", "Measurement request key-value pair:\n", "\n", "* `\"queryMeasurement\": \"gridappsd-sensor-simulator\"`\n", "\n", "Allowed key-value pairs for `queryFilter` for output data\n", "\n", "```\n", "\"queryFilter\": { KEY VALUE\n", " \"startTime\": number ,\n", " \"endTime\": number ,\n", " \"measurement_mrid\": string OR [array of string values] ,\n", " \"simulation_id\": string ,\n", " \"angle\": number ,\n", " \"magnitude\": number ,\n", " \"value\" number } \n", "```\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "__Example 1: Query for all Sensor Service Data between a given `\"startTime\"` and `\"endTime\"`:__" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "topic = \"goss.gridappsd.process.request.data.timeseries\" # Specify Timeseries API GridAPPS-D topic\n", "\n", "# Query for all Sensor Service Data in time range:\n", "message = {\n", " \"queryMeasurement\": \"gridappsd-sensor-simulator\",\n", " \"queryFilter\": {\"simulation_id\": simulation_id,\n", " \"startTime\": \"157004114\",\n", " \"endTime\": \"1570041117\"},\n", " \"responseFormat\": \"JSON\"\n", "}\n", "\n", "gapps.get_response(topic, message) # Pass API call" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "![gridappsd-logo](../images/GridAPPS-D_narrow.png)" ] } ], "metadata": { "celltoolbar": "Tags", "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.5" } }, "nbformat": 4, "nbformat_minor": 4 }