{ "cells": [ { "cell_type": "markdown", "id": "63bf39c1", "metadata": {}, "source": [ "# Using the CIMantic Graphs API" ] }, { "cell_type": "markdown", "id": "7c7fbe82", "metadata": {}, "source": [ "This tutorial provides an introduction to usage of the CIMantic Graphs library (aka CIM-Graph). \n", "\n", "CIMantic Graphs is an open-source library for creating in-memory labeled property graphs for creating, parsing, and editing CIM power system models. It creates Python object instances in memory using a data profile exported from a specified CIM profile (e.g. IEC61970cim17v40 or GridAPPS-D RC4_2021)." ] }, { "cell_type": "markdown", "id": "0d1bbb68", "metadata": {}, "source": [ "To install CIMantic Graphs clone the github repository or use pip install: `pip install cim-graph`" ] }, { "cell_type": "markdown", "id": "5224b4f5", "metadata": {}, "source": [ "----" ] }, { "cell_type": "markdown", "id": "bf3fb7b2", "metadata": {}, "source": [ "## CIMantic Graphs Structure\n", "\n", "CIMantic Graphs uses the layered architecture shown below:" ] }, { "cell_type": "markdown", "id": "e6b13cfc", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "id": "fb5d0b26", "metadata": {}, "source": [ "### Database Layer\n", "CIMantic Graphs currently supports the following databases and interfaces:\n", "\n", "* Blazegraph Database\n", "* GraphDB Database\n", "* Neo4J Database\n", "* MySQL Database (in progress)\n", "* GridAPPS-D Platform\n", "* AVEVA PI Asset Framework (in progress)\n", "* RDFLib File Parser\n", "\n", "\n", "* XML Flat Files\n", "* JSON-LD Flat Files (in progress)\n", "* CSV Flat Files (in progress)\n", "\n", "The library uses a unified syntax for all upper-level calls and routines. The databases can be swapped interchangeably by changing the `ConnectionParameters` created during application startup and no other changes to any other application syntax or methods" ] }, { "cell_type": "markdown", "id": "1cb05db8", "metadata": {}, "source": [ "### Data Profile Layer\n", "\n", "CIMantic Graphs is able to support any standard or custom CIM profile. The CIM profile needs to be exported as an XSD data profile / schema. CIMantic Graphs is then able to ingest the data profile and convert all UML classes and attributes to python dataclasses, which power all of the routines and unified API syntax" ] }, { "cell_type": "markdown", "id": "a4532a4b", "metadata": {}, "source": [ "### API Layer\n", "\n", "CIMantic Graphs offers a breakthrough in terms of ease-of-use through a unified API with two core methods.\n", "\n", "__Access to labeled property graph objects:__\n", "\n", "* `network.graph[cim.ClassName]`: This offers access to a catalog of CIM object instances stored in memory and sorted by class type and mRID forming the named property graph.\n", "\n", "__Universal database query method:__\n", "\n", "* `network.get_all_edges(cim.ClassName)`: This is a universal query method that gets all attributes and all objects one edge away from instances of the specified class. This method works for all CIM classes, CIM profiles, serialization formats, and supported databases." ] }, { "cell_type": "markdown", "id": "0758a905", "metadata": {}, "source": [ "### Knowledge Graph Layer\n", "\n", "CIMantic Graphs offers three core knowledge graph classes for handing various kinds of power system models:\n", "\n", "* `BusBranchModel`: Transmission bus-branch models commonly used for planning and power flow studies\n", "\n", "* `NodeBreakerModel`: Transmission node-breaker models commonly used inside energy management systems\n", "\n", "* `FeederModel`: Distribution feeder models with support for single-phase unbalanced networks used in North America.\n", "\n", "Centralized or distributed representations of the power system network model can be used. Centralized models use a single labeled property graph for the network. Distributed models use nested `DistributedArea` class instances to represent the grouping of equipment inside a Substation, VoltageLevel, Bay, Feeder, and switch-delimited topological area inside a combined T+D model." ] }, { "cell_type": "markdown", "id": "8f417552", "metadata": {}, "source": [ "### Application Layer\n", "\n", "T+D applications are able to access all of the power system objects through knowledge graph, without any need to connect to the database or perform any custom i/o operations. " ] }, { "cell_type": "markdown", "id": "d918bf53", "metadata": {}, "source": [ "----" ] }, { "cell_type": "markdown", "id": "b8318a41", "metadata": {}, "source": [ "## Importing CIMantic Graphs\n", "\n", "CIMantic Graphs contains several modules within the core library:\n", "\n", "* `data_profiles`: This package contains full CIM profiles exported through [CIMTool](https://github.com/CIMug-org/CIMTool/releases) and provides direct access to CIM dataclasses, their attributes, and usage documentation.\n", "\n", "* `databases`: This package contains database i/o connections and backend query handling for multiple databases (e.g. Blazegraph, GraphDB, Neo4J, MySQL, etc.).\n", "\n", "* `models`: This package contains knowledge graph models for transmission node-breaker, transmission bus-branch, and distribution feeder models.\n", "\n", "* `queries`: This package contains generic queries that are built at runtime for any CIM profile and CIM class.\n", "\n", "* `utils`: This package contains shortcut methods for common routines, such as writing out new files." ] }, { "cell_type": "markdown", "id": "e809179b", "metadata": {}, "source": [ "## Specifying Environment Variables\n", "\n", "**IMPORTANT: As of v0.3+, the ConnectionParameters class has been deprecated. Use the environment variables below to authenticate with the database.**\n", "\n", "The CIM-Graph library supports multiple databases. The environment variables specify how to read the CIM model:\n", "\n", "__Required environment variables:__\n", "\n", "* `CIMG_CIM_PROFILE`: This specifies the specific version of CIM to be used, based on the available python data profiles loaded into the library\n", "\n", "__Optional environment variables:__\n", "\n", "If these values are not specified, they will default to the sample values listed in [example.env](https://github.com/PNNL-CIM-Tools/CIM-Graph/blob/develop/example.env)\n", "\n", "* `CIMG_NAMESPACE`: CIM namespace, default is `\"http://iec.ch/TC57/CIM100#\"`\n", "* `CIMG_IEC61970-301`: Serialization version. Versions 7(default) and below use `rdf:ID=`. Version 8 uses `rdf:about=urn:uuid:`\n", "* `CIMG_URL`: URL at which the database can be reached via TCP/IP or other connection\n", "* `CIMG_HOST`: Database host address\n", "* `CIMG_PORT`: Database host port\n", "* `CIMG_DATABASE`: Database name\n", "* `CIMG_USERNAME`: Database username\n", "* `CIMG_PASSWORD`: Database password\n", "\n", "Note that not all parameters are required. Each database connection uses a subset of these arguments depending on the requirements of the database connection driver." ] }, { "cell_type": "markdown", "id": "11cb8710", "metadata": {}, "source": [ "To user the GridAPPS-D platform connection, set the environment variables shown below in your script or in a `.env` file" ] }, { "cell_type": "code", "execution_count": 1, "id": "1322dc63", "metadata": {}, "outputs": [], "source": [ "import os\n", "os.environ['CIMG_CIM_PROFILE'] = 'cimhub_2023'\n", "os.environ['CIMG_URL'] = 'http://localhost:8889/bigdata/namespace/kb/sparql'\n", "os.environ['CIMG_DATABASE'] = 'powergridmodel'\n", "os.environ['CIMG_HOST'] = 'localhost'\n", "os.environ['CIMG_PORT'] = '61613'\n", "os.environ['CIMG_USERNAME'] = 'system'\n", "os.environ['CIMG_PASSWORD'] = 'manager'\n", "os.environ['CIMG_NAMESPACE'] = 'http://iec.ch/TC57/CIM100#'\n", "os.environ['CIMG_IEC61970_301'] = '8'\n", "os.environ['CIMG_USE_UNITS'] = 'False'" ] }, { "cell_type": "markdown", "id": "f2d220b9", "metadata": {}, "source": [ "**IMPORTANT - CHANGES IN SERIALIZATION IN GRIDAPPSD V2025.01.0**\n", "\n", "For GridAPPS-D / Blazegraph tags `v2025.01.0` and later, the following environment variables are required:\n", "\n", "```python\n", "os.environ['CIMG_CIM_PROFILE'] = 'cimhub_2023'\n", "os.environ['CIMG_IEC61970_301'] = '8'\n", "```\n", "\n", "For GridAPPS-D / Blazegraph tags between `v2021.01.0` and `v2024.09.0`, the following environment variables are recommended:\n", "```\n", "os.environ['CIMG_CIM_PROFILE'] = 'rc4_2021'\n", "os.environ['CIMG_IEC61970_301'] = '7'\n", "```\n", "\n", "These changes relate to a change in the export format in the OpenDSS cim100 command which changed serialization versions of mRIDs for `rdf:id=_mrid` to `rdf:about=urn:uuid:mrid`" ] }, { "cell_type": "markdown", "id": "9ef0ba43", "metadata": {}, "source": [ "The ConnectionParameters class has been deprecated and will now throw a warning when called. " ] }, { "cell_type": "code", "execution_count": 2, "id": "7288192a", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "ConnectionParameters class is deprecated and will be deleted in a future release\n", "Set environment variables for required authentication\n" ] } ], "source": [ "from cimgraph.databases import ConnectionParameters # DEPRECATED!\n", "params = ConnectionParameters(url = \"http://localhost:8889/bigdata/namespace/kb/sparql\",\n", " cim_profile='cimhub_2023', iec61970_301=8) # Blazegraph params" ] }, { "cell_type": "markdown", "id": "789d7800", "metadata": {}, "source": [ "----" ] }, { "cell_type": "markdown", "id": "481757d1", "metadata": {}, "source": [ "## Connecting to the Data Source\n", "\n", "The next step is to connect to the database or file source that will be used for CIM model, such as `XMLFile`, `BlazegraphConnection`, or `GridappsdConnection`" ] }, { "cell_type": "code", "execution_count": 3, "id": "6bb2d2a3", "metadata": {}, "outputs": [], "source": [ "from cimgraph.databases.blazegraph import BlazegraphConnection\n", "database = BlazegraphConnection()" ] }, { "cell_type": "markdown", "id": "2c094c24", "metadata": {}, "source": [ "----" ] }, { "cell_type": "markdown", "id": "c864a7db", "metadata": {}, "source": [ "## Importing the CIM Profile\n", "The first step in using CIMantic Graphs is to import the python data profile for desired CIM profile. The data profile provides the ability to invoke CIM classes directly based on their name.\n", "\n" ] }, { "cell_type": "markdown", "id": "2719a373", "metadata": {}, "source": [ "__Method 1:__ Directly import the desired CIM profile:" ] }, { "cell_type": "code", "execution_count": 4, "id": "bc106de3", "metadata": {}, "outputs": [], "source": [ "import cimgraph.data_profile.cim17v40 as cim" ] }, { "cell_type": "markdown", "id": "4db1c156", "metadata": {}, "source": [ "__Method 2:__ Use `importlib`:" ] }, { "cell_type": "code", "execution_count": 5, "id": "099ba244", "metadata": {}, "outputs": [], "source": [ "import importlib\n", "cim_profile = 'cimhub_2023'\n", "cim = importlib.import_module('cimgraph.data_profile.' + cim_profile)" ] }, { "cell_type": "markdown", "id": "3aa6c39d", "metadata": {}, "source": [ "### Invoking CIM classes" ] }, { "cell_type": "markdown", "id": "048e29f9", "metadata": {}, "source": [ "After importing the data profile, it is possible to create an instance of a class or view the attributes of any class. " ] }, { "cell_type": "markdown", "id": "ed729732", "metadata": {}, "source": [ "__Example 1:__ Create a new breaker with a name. If an mRID string is not provided, a UUID is automatically created based on the name or a random seed. Attributes of the class can be assigned " ] }, { "cell_type": "code", "execution_count": 6, "id": "938fb5b4", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\n", " \"@id\": \"7e72cb38-275c-4369-9c3b-e8b5a42a8703\",\n", " \"@type\": \"Breaker\",\n", " \"name\": \"breaker_01\",\n", " \"open\": \"True\",\n", " \"NormalOpen\": false\n", "}\n" ] } ], "source": [ "breaker = cim.Breaker(name = \"breaker_01\", open=True)\n", "breaker.NormalOpen = False\n", "breaker.pprint()" ] }, { "cell_type": "markdown", "id": "9a780d4d", "metadata": {}, "source": [ "__Example 2:__ Associate two CIM objects based on their associations" ] }, { "cell_type": "code", "execution_count": 7, "id": "895e52c8", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\n", " \"@id\": \"7e72cb38-275c-4369-9c3b-e8b5a42a8703\",\n", " \"@type\": \"Breaker\",\n", " \"name\": \"breaker_01\",\n", " \"EquipmentContainer\": {\n", " \"@id\": \"5d58b92a-5734-4ae7-8f3c-1357e662cf16\",\n", " \"@type\": \"Substation\"\n", " },\n", " \"open\": \"True\",\n", " \"NormalOpen\": false\n", "}\n", "{\n", " \"@id\": \"5d58b92a-5734-4ae7-8f3c-1357e662cf16\",\n", " \"@type\": \"Substation\",\n", " \"name\": \"sub_1\",\n", " \"Equipments\": [\n", " {\n", " \"@id\": \"7e72cb38-275c-4369-9c3b-e8b5a42a8703\",\n", " \"@type\": \"Breaker\"\n", " }\n", " ]\n", "}\n" ] } ], "source": [ "substation = cim.Substation(name = \"sub_1\")\n", "breaker.EquipmentContainer = substation\n", "substation.Equipments.append(breaker)\n", "breaker.pprint()\n", "substation.pprint()" ] }, { "cell_type": "markdown", "id": "3e13a53b", "metadata": {}, "source": [ "__Example 3:__ View documentation of the ACLineSegment class" ] }, { "cell_type": "code", "execution_count": 8, "id": "d7af12dd", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", " A wire or combination of wires, with consistent electrical characteristics,\n", " building a single electrical system, used to carry alternating current\n", " between points in the power system.\n", " For symmetrical, transposed 3ph lines, it is sufficient to use attributes\n", " of the line segment, which describe impedances and admittances for the\n", " entire length of the segment. Additionally impedances can be computed by\n", " using length and associated per length impedances.\n", " The BaseVoltage at the two ends of ACLineSegments in a Line shall have\n", " the same BaseVoltage.nominalVoltage. However, boundary lines may have slightly\n", " different BaseVoltage.nominalVoltages and variation is allowed. Larger\n", " voltage difference in general requires use of an equivalent branch.\n", " \n" ] } ], "source": [ "print(cim.ACLineSegment.__doc__)" ] }, { "cell_type": "markdown", "id": "373f3ede", "metadata": {}, "source": [ "-----" ] }, { "cell_type": "markdown", "id": "9e4a7dd6", "metadata": {}, "source": [ "## Creating a Container and Graph Model\n", "\n", "CIMantic Graphs uses an EquipmentContainer class as the starting point for building a knowledge graph of the power system model. The supported classes are `BusBranchModel`, `NodeBreakerModel`, and `FeederModel`." ] }, { "cell_type": "code", "execution_count": 9, "id": "71c425c2", "metadata": {}, "outputs": [], "source": [ "from cimgraph.models import FeederModel" ] }, { "cell_type": "markdown", "id": "720f3cd3", "metadata": {}, "source": [ "The power system network is then created by passing the database connection, container object, and a boolean flag whether a centralized or distributed model should be built." ] }, { "cell_type": "code", "execution_count": 10, "id": "64287711", "metadata": {}, "outputs": [], "source": [ "feeder = database.get_object(mRID=\"49AD8E07-3BF9-A4E2-CB8F-C3722F837B62\") #ieee 13 bus\n", "network = FeederModel(connection=database, container=feeder, distributed=False)" ] }, { "cell_type": "markdown", "id": "06edeadc", "metadata": {}, "source": [ "The network is populated by default with all ConductingEquipment, ConnectivityNode, and Terminal class instances. The knowledge graph is indexed by the class type and then the device mRID.\n", "\n", "\n", "To view all instances of a particular class, use the `.pprint(cim.ClassName)` method. The default python print method and individual .pprint() statements can also be used on `network.graph` and individual objects.\n", "\n", "By default, only equipment classes and basic connectivity information (nodes and terminals) are loaded with just the UUIDs of each class to optimize memory usage." ] }, { "cell_type": "code", "execution_count": 11, "id": "fadccf06", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[\n", " {\n", " \"@id\": \"52de9189-20dc-4c73-bdee-e960fe1f9493\",\n", " \"@type\": \"Breaker\",\n", " \"Terminals\": [\n", " {\n", " \"@id\": \"1d81c7fe-e88f-41e3-a900-476ca6476ccd\",\n", " \"@type\": \"Terminal\"\n", " },\n", " {\n", " \"@id\": \"2847e06b-c8ed-41e6-b515-c61c9e8eb4b4\",\n", " \"@type\": \"Terminal\"\n", " }\n", " ]\n", " }\n", "]\n" ] } ], "source": [ "network.pprint(cim.Breaker) # View all instance data of type Breaker" ] }, { "cell_type": "markdown", "id": "ab5659a7", "metadata": {}, "source": [ "CIM-Graph also supports object visualization using the Mermaid.js library via the `cimgraph.utils` module. Full documentation of visualization is available in the separate documentation via PyPi. The breaker that we just viewed can be visualized by passing the object to the `utils.get_mermaid()` method:" ] }, { "cell_type": "code", "execution_count": 12, "id": "407c8bf9", "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "
\n", " \n", " " ], "text/plain": [ "